Today, Brad Feld walked into my office (well, the Techstars office) and yelled “Time to spin up more dynos PivotDesk!” 5 minutes earlier he and Jason Mendelson had tweeted about PivotDesk which drove 50 concurrent users to our Beta site looking for flexible office space in Boulder.
Our site caved, I added more dynos and restarted all processes and we were back (Thanks Ryan Cook). The dumbest 10 minutes of my summer at Techstars Boulder thus far.
2012-07-25T22:45:23+00:00 app[web.1]: Started GET "/" for 220.127.116.11 at 2012-07-25 22:45:23 +0000
2012-07-25T22:45:23+00:00 heroku[router]: Error H12 (Request timeout) -> GET beta.pivotdesk.com/assets/layout/denver-img.png dyno=web.1 queue= wait= service=30000ms status=503 bytes=0
“Going Live” is a tricky thing in the enterprise and in Startups. Leading a large team for the past 2 years at a global enterprise software company, “Going Live” was always about when we felt comfortable telling the disjointed Marketing department that they could make banners for conferences and write whitepapers to put on the website. I heard “When will it be ready” thousands of times. “Being Live” meant a 100% bug free, feature complete product that everyone thought was perfect. Of course this was not realistic and we struggled constantly to come to a shared understanding across the company about this.
In a Startup, “Going Live” means people start tweeting about you.
Here are 2 lessons I reflected on today during my long run up Boulder Canyon to cool down after my fuckup:
1. The day you turn off some kind of basic auth is the day you should increase your infrastructure.
I was kicking back listening to Jeff Clavier drop knowledge on the Techstars teams when my phone started buzzing and the New Relic alerts started arriving. I had no expectation of increased traffic today and we had not had a single infrastructure hiccup in 60 days since deploying to production for the first time. My mistake was not thinking of the site as live even though it absolutely was.
2. “Being Live” is not the same thing as MVP, Alpha, Beta, Prototype
You’ve probably heard the question “Are you live?” and the response “Yes, live with a MVP” or “Yes, live with an Alpha”. That’s all well and good. Getting feedback fast, not being afraid to put things out there and iterating quickly are wonderful things for software development. Powering those early versions with weak infrastructure is not acceptable. Although I had all of the right stuff in place (monitoring, backups, analytics, ability to scale), I was not ready for today’s traffic increase.
So, learn from me and don’t let this happen to you.