The previous post didn’t offer much information on how to actually set everything up, so the documentation has been updated, and can be found here:

http://openplaces.github.com/bigrecord/file.cassandra_install.html

Bigrecord was also updated to version 0.1.1 and changed the Cassandra usage significantly. Hopefully no one was using it for anything important You can always e-mail me with questions at greg [@] openplaces [dot] org, or on IRC to greglu on #cassandra at freenode.net

Despite this blog being inactive, Bigrecord itself has had new versions rolled out every month or so to address bugs. Now that I have some time, I’ll be working on this project more regularly.

Cassandra support has always been planned, and finally work has begun! Before getting into the details, I’d like to address some misconceptions that have risen regarding its integration. First of all, Cassandra was never supported (at all) in Bigrecord. There was some preliminary work done on Bigrecord-Driver to include support back when Cassandra was first open-sourced and client access was done through a native Java API (rather than exclusively through Thrift). We originally created Bigrecord-Driver for Ruby client access to HBase (before its implementation of Thrift was stable) by using JRuby to access HBase’s Java API and start up a DRb server that allowed native Ruby (Matz) clients to connect to it and interact with HBase. When Cassandra was first released, we tried this same approach in Bigrecord-Driver first, but found that it was unstable and could not be integrated well enough. So, development of Cassandra in Bigrecord-Driver (and consequently Bigrecord) halted. Since then, Cassandra’s client access has been done exclusively through Thrift, so Bigrecord-Driver isn’t required at all. It was just a matter of creating a connection adapter in Bigrecord that used Ruby Thrift or some other client library.

Which brings us to Bigrecord’s support of Cassandra. As of version 0.1.0, Cassandra can be used as one of the data stores. This is a first implementation of the adapter, and there will be many updates (probably daily), so check the commit logs for the latest. Documentation is still to be updated, terminology is inconsistent (since Bigrecord was rooted in HBase), and there are some specs that fail sporadically, but those will all be addressed in the coming weeks.

To get the latest, run

sudo gem install bigrecord

And edit your bigrecord.yml like so:

Everything else should work the same.

Much like with beards, when it comes to databases, bigger isn’t always bettter.  But at openplaces, we’re stuck between a rock and hard place.  A big hard place.  With terabytes of data.  With millions of records. Some painful truths:

• Distributed computing is messy.  Nodes go out of sync, data gets lost.
• There is no mature distributed database option.  HBase, Hypertable, Cassandra, CouchDB, MongoDB, Voldemort… They all have sharp edges.  You are going to get cut and you are going to bleed.
• It takes about 12 hours to upload our database to EC2.  But our data is growing.  Soon we’ll have to actually ship hard drives to Seattle: http://aws.amazon.com/importexport/.

On that up-beat note, Ruby coders, you are finally invited to start playing with BigRecord, your window to the world of Bigtable-esque data.  Which, for now, means HBase.  Click here to get started.

If you found this site and you’re wondering what the deal is, all I can say is: we’re working on it.  Basically we have some to strip dependencies to hbase and our application code, implement a driver for Cassandra to prove we’ve abstracting BigRecord properly, write some more tests and then package it all up nicely into a Gem.  Stand by for launch in about a month.

ActiveRecord is pretty awesome.  And column-oriented data storage?  Also awesome.  Relational databases for ambitiously scalable websites serving massive amounts of data?  Not so much.

So we were riding the Rails and we wanted to go big.  But replication and sharding with MySQL made us want to go home.  Then we got seduced by a Google-inspired bleeding-edge project called HBase, a siren who called out ‘distributed!’, ’schema-less!’ and ‘map/reduce!’.  The result?  A rewrite of ActiveRecord for a new kind of non-relational database.

BigRecord is ActiveRecord for web-scale column-oriented systems like HBase (e.g. Cassandra, Hypertable) and with some help, can be adapted for document-oriented databases, too (e.g. CouchDB, Dynomite, ThruDB, Tokyo Cabinet).

We’re putting it out there.