Posterous theme by Cory Watilo

Filed under: python

See all posts on Posterous with this tag »

One Slick Way Yieldbot Uses MongoDB

As a key/value store. Wait, what?  Yeah, MongoDB as a key/value store.

Why? Because we were already using Mongo and planned to move some of our data to a key/value store. But we also didn't want to wait until we made a decision on a specific solution to make the transitions in our code.

First, as if you wouldn't have guessed, here's how easy a "Python library for MongoDB as a key/value store" is:

 

def kv_put(collection, key, value):
    """key/value put interface into mongo collection"""
    collection.save({'_id': key, 'v': value}, safe=True)

 

def kv_get(collection, key):
    """key/value get interface into mongo collection"""
    v = collection.find_one({'_id':key})
    if v:
        return v['v']

 

Note: kv_get() returns None if nothing found, so technically this doesn't gracefully handle the case where you want None to be a possible value.

What was the pain point?

We basically found that we had collections with nightly analytics results that were really big, and whose indexes were really, really big.  And the index requirements were going way beyond our 70GB RAM server.  We didn't want to shard our Mongo server because of the cost involved, so instead decided to take a different appraoch.  Since this data was read-only results of analytics, where we once had collections that had entries that were multiply indexed, we now have collections that are pages of the old entries in a defined sorted order and are accessed as key/value.

How did the change work out? Great. We still haven't switched from MongoDB for this data. Still plan to, but in a startup once you address a pain point you move on to the next one.

You definitely can't argue that MongoDB isn't flexible.

 

Looking for a Few Good Devs

At Yieldbot we're a small team building incredible technology that's getting major publishers and advertisers hooked. We're always looking for the best technical talent to join our development team and work with us on getting to the next level, and as CTO I think it's only fair that you know what we're looking for. :-)

What you need to know:

  • We're a small talented team looking to become a bigger talented team.
  • We work on tough interesting problems, use cutting-edge "big data" technology, and enjoy winning.
  • We're working on revolutionizing how web advertising works.
  • This is gonna be big.

What we need to know:

  • You like to solve tough problems and have a history of winning.
  • You can code in a few different languages and are expert with one of them.
  • That language is Python or you are py-curious.
  • You want to make big contributions on a small team and build a valuable business.

Some  technology stuff:

  • Python's big here, but we rock Clojure and Javascript too.
  • We use Django. Bonus if you know it, but if you don't it ain't so hard.
  • We wrestle some serious big data, and use Hadoop to do it.
  • MongoDB and HBase compete for our love.
  • Chef and the Vagrant knifed Puppet and we've never been the same

 

If you're a fit, dust off your Python and contact us:

yieldbot=[46,13,-19,10,-44,5,60,-4,-2,68,-4,-6,6,-53,-17,6,-2]

''.join(map(chr,map(lambda x:x[0]+x[1],zip(map(ord,str(dir)),yieldbot))))