Just about every CTO at a growth-stage company that I know is carrying an infrastructure problem they are quietly deciding not to look at. The system works, so the problem stays out of sight right up until something drags it into the open.
That “something,” in our particular case, was a large UK-based bank. The new customer arrived with the expectation of five times our usual traffic. For a nascent startup, the product-market fit really began to feel real. It’s also in this moment that what had previously been seen as a technical chore can morph into a business risk.
We’re a contact center observability platform, so our work runs on processing especially large volumes of real-time interaction data for some of the world’s largest contact centers. A couple years ago we were five engineers, and we ran our own data infrastructure. Frankly we were proud of it, enough that it showed up in our pitch. When that bank prospect came along, our reflex was to treat its scale as one more thing we would, and could, handle in-house. The math, for its part, said otherwise, or at least not without breaking something we had promised our customers.
The bank did not create that risk, which had been quietly compounding for the better part of a year. We were leaning on a hosted Elasticsearch setup that had gotten flaky, from a vendor who was not fast enough on support for us to honor the promises we were making to our own customers. We knew all of this, but by then had gotten used to working around it like you might ignore the rattle in your car hoping it will just work itself out. What the bank deal did, in an instant, was take away our ability to keep ignoring it.
The data infrastructure debt you stop noticing
This is the trap that I now watch other teams wander into constantly. The system keeps humming right up until it doesn’t, ergo the problem never forces its way onto a roadmap. CTOs and other decision makers end up making the call only once the stakes have climbed high enough that a “calm” version of the decision is no longer available. By then there is a customer somewhere watching an onboarding clock, which is just about the worst possible setting for a choice this expensive to get wrong.
For our own part, the math wasn’t great when we actually ran it. The bank onboard and keeping everything else at status quo would have meant three of our five engineers pretty much residing inside our infrastructure (leaving just two people to build the product our customers were paying us for). I began to see the shape of what we were about to turn into, a company that mostly babysits its own plumbing while the real work crawls, and, to be honest, I wanted no part of running it.
Moving before we were forced to
We ended up changing the setup before the bank went live, while we still had enough buffer to do it carefully. We came off the hosted Elasticsearch service and onto open source OpenSearch, and brought in NetApp Instaclustr to run both OpenSearch and Postgres, which underpinned our observability platform. That relationship extended to our ClickHouse analytics clusters not long after, which is where the bulk of our processing actually lives. Importantly, all of these are 100% open source technologies, which could ensure portability without lock-in. The principle behind the move, however, mattered just as much as the technology itself. I wanted my engineers out of the 2 am migration business and the emergency-patch business, and I wanted that time pointed squarely back at the product.
One condition did matter more than the rest, the one where many of these moves quietly die. We sell into regulated industries (like that large bank) and cannot just pour our data into somebody else’s cloud and give up control of where it physically sits. Zero wiggle room, so whatever we picked had to let us keep the data inside our own environment. No amount of “more stable” would have saved us if we flunked that test.
The payoff turned out to be easy to put a number on, and I’ll give you ours because it genuinely changed how I think about all of this. We got back somewhere north of 6,200 engineering hours a year. That was time that used to disappear into the black hole of back-end upkeep and the kind of fire drills that eat a sprint (and whatever weekend you were hoping to have). We have since pushed past 500,000,000 interactions and the trillions of insights our ClickHouse analytics layer chews through, and the performance scares we used to brace for mostly stopped happening. Failovers turned, well, boring, while the upgrades I once lay awake over became just a Tuesday afternoon.
Decide before the deal decides for you
This is not an argument for outsourcing your data infrastructure across the board. Plenty of companies should run their own, and for some it is a real edge. My point is smaller, and I think harder to wriggle out of. The question of who runs your data layer ought to get answered
Read the original article:
How fixing our data infrastructure before it broke us gave back 6,200 engineering hours