In a technology startup speed is very often the enemy of quality.  I tend to prefer speed and thankfully our CTO tends to prefer quality.  He is usually right.  He has a strongly held belief that you can speed up by slowing down.  There are a fair number of variations on this that I heard often as a child, “slow down and do it right the first time” or the oft-quoted Benjamin Franklin favorite, “an ounce of prevention is worth a pound of cure.”

Tonight we are experiencing by far the worst outage we’ve ever experienced in our company’s history as a direct result of us trying to go fast and operating for a short period of time without a proper net.  Due to a unique set of circumstances we’ve ended up with a major database outage and will likely be down for at least 12 hours.  This is maddening, embarrassing and frustrating as hell.  Unfortunately I can trace most of the blame directly back to a variety of my decisions that were supposed to make us go faster.  Startups have an amazing ability to continually provide painful lessons.

This outage is having a huge impact on our customers and we hate that more than just about anything in this world.  Our team is continuing to work through to the night to mitigate the outage and get our systems back online.  We regret the outage and will be taking steps to make sure it doesn’t happen again – just as soon as we triage the situation and get everyone back online.