From about 2am to 4am EDT this morning, Trello was malfunctioning due to memory issues caused by a bug in an underlying system we use. We have resolved the immediate issue and instituted monitoring to guard against similar issues in the future. The bug in the underlying system (Redis) is due to be fixed soon: https://github.com/antirez/redis/issues/91
Technical Explanation:
A node process stoppped consuming its Redis pubsub last night, and that pushed keys out of Redis, occupying Redis's memory with the output buffer to the node pubsub subscriber. Finally, all of the Redis space was gone, and THAT tripped our monitors because processes started dying and couldn't serve requests. To guard against this in the future we're setting up a monitor for the Redis output buffer list.The Redis bug is supposed to be fixed in 2.6 (release candidate soon).
