Interesting post on the derwiki blog … Especially the commenting is quite entertaining!
Amazing how ignorance produces patronizing statements (-> Morg).
See belwo the top of teh post …
Background
We have a legacy system in our production environment that keeps track of when a user takes an action on Causes.com (joins a Cause, recruits a friend, etc). I say legacy, but I really mean a prematurely-optimized system that I’d like to make less smart. This 500m record database is split across monthly sharded tables. Seems like a great solution to scaling (and it is) — except that we don’t need it. And based on our usage pattern (e.g. to count a user’s total number of actions, we need to do query N tables), this leads to pretty severe performance degradation issues. Even with memcache layer sitting in front of old month tables, new features keep discovering new N-query performance problems. Noticing that we have another database happily chugging along with 900 million records, I decided to migrate the existing system into a single table setup. The goals were:
reduce complexity. Querying one table is simpler than N tables.
push as much complexity as possible to the database. The wrappers around the month-sharding logic in Rails are slow and buggy.
increase performance. Also related to one table query being simpler than N.
…