Containerizing Yieldbot

by Neil Ouellette, Engineer
December 14th, 2015

Here at Yieldbot we like to embrace new technology. We believe that trying new things is the recipe for creating something useful and powerful. Finding the right tech for the job requires research and mental fortitude to push implementation forward. Our newest journey is to containerize Yieldbot. Containers have so many positive aspects that it wasn’t hard for our engineering team to adopt the idea. Although, changing our infrastructure would involve multiple teams and each team would have a lengthy task list in order to make it happen. So, why would we decide to overhaul our system?

Explaining Containers

Think of containers as those metal boxes on a freight ship and think of the freight ship as a machine or a cluster of machines. Each metal box contains application specific logic and can be loaded onto the freight ship as long as there is enough storage space, processing power and memory.

Each container is isolated from each other and are individually protected. If one were to crash or have an unexpected issue the other containers would be ok because of this isolation.

The containers are also really small in size because they all share the same resources on the ship. This makes life a lot easier when managing those resources (storage, processing and memory) whether you’re scaling up or down depending on traffic load.

What are the Benefits?

  1. Lightweight for Rapid Deployment – containers are reduced in size because it only includes run time dependencies for a small set of code. So it’s faster to build and faster to deploy.
  2. Portability – since the application and all its dependencies are bundled into a single container and run through a common platform, like docker, we remove compatibility issues across operating systems as docker integrates with them. This will also give us reproducible results, what loads in my development environment will load just the same in production.
  3. Version control – is at the container level so no need to roll back to a previous version of a library hoping that will solve the issue. Now you will just roll back the whole container to a known working one.
  4. Sharing – you can use a remote repository to share your container images with others without configuration and environment setup.

How Yieldbot uses Containers

Our production environment is highly distributed. We take advantage of regionalizing our traffic as well as load balancing our clusters to provide our publishers with the fastest response times.

Given our large production environment we really needed to get control of our build and deploy process. We set out to create a system to ensure we are able to deploy without disruption of service as well as provide a stable environment for our developers to push their changes to production.

Containerizing Yieldbot - Code

Containers != Docker

Docker has become synonymous with containers, but there are other providers out there. New players are entering this space like Rocket, LXD, and even Microsoft is showing up to the party with Drawbridge. Yieldbot will continue to evaluate and adjust its stack as this space matures.

Why Yieldbot Made the Switch

Aside from the benefits detailed above, we needed a better way to manage all our components. When creating a large system for a company to run it becomes very complex with each week passing. Mitigating the problem of conceptualization of that system will reap huge rewards if executed properly. The small nature of containers allows someone to understand what that component is doing easily. This allows someone to jump in and create a mental model for themselves of just how that component functions in the larger system. This not only helps engineers find problems and build out functionality faster, but a flowchart and/or design document of the components will also help at the business level to make better choices by understanding the impact of those decisions. For us here at Yieldbot, we’ve simply found that containers are the best solution to help us isolate applications in an easy, buildable and reproducible ways.

Yieldbot's Build / Deploy Pipeline for Containers-

Neil Ouellette is a Yieldbot Engineer on the Frontend Infrastructure Team. Located in our Boston office, Neil works to enable frontend developers to work efficiently by not having to be concerned with how their applications access data through our distributed systems as well as how their application gets built and deployed.

Contact Us

Download Case Study