← §02 Battle Log§ 03.01 · Healthcare CMS · 2024
Patient-care CMS through the utilization ceiling — without going offline.
A clinic CMS used by hospitals and clinics for daily patient services was running on a single MongoDB ReplicaSet hosted in Azure compute. Headroom had run out — peak operating hours started degrading. The fix needed horizontal scale, but the system was vendor-locked, the network routing had a misconfiguration nobody had noticed, and nobody on the team had hands-on MongoDB sharding experience. I took it solo, learned what I needed, and shipped it without a second of downtime to patient care.
MONGODBATLASSHARDINGAZUREZERO-DOWNTIMEHEALTHCARE CMS
0
Sec downtime
Solo
Operator · 1 human
Vertical → Horizontal
Scale model shift
0 → MongoDB Atlas
Prior team experience
§01
The system
A content-management and operations system used by hospitals and clinics to deliver day-to-day patient services — bookings, records, point-of-care workflows. The platform was vendor-owned, which meant the application layer was effectively a black box. Anything we wanted to change had to be accomplished underneath the application, at the data and network layers, without changing the contract the app code expected.
§02
The signal
The database tier was a single MongoDB ReplicaSet sitting on Azure compute. As the customer base grew, peak-hour load was driving the primary node into a utilization ceiling — CPU and IOPS both climbing into the red, then staying there. Vertical scale-up wasn't going to clear the trajectory; the platform needed horizontal scale or it was going to start failing requests during the worst possible hours of the day.
On the way into the diagnosis, I also found that the Azure network routing between the application tier and the database tier was misconfigured in a way nobody had flagged. Tail latencies were higher than they had any right to be even before MongoDB was saturated, and traffic was taking a longer path than the architecture diagram implied. Fixing that was a free win before any migration planning even started.
§03
The constraints
Three constraints shaped every decision:
- Vendor-owned application. No upstream changes to the app's data-access layer. The migration had to be invisible to the running code — same driver, same connection semantics, same query shapes — even as the topology changed underneath.
- Solo operator. No internal DBA, no external MongoDB consultant on the engagement. I was the planner, the executor, and the rollback path.
- No team experience with MongoDB Atlas or sharded clusters, including me. The honest part of this story. I had to build the expertise before I could build the plan.
§04
The approach
MongoDB's core strength is horizontal scale through sharding, but only if you choose the right shard key and route writes the way the cluster expects. Picking that key wrong is a one-way trip to a hotspot.
The order of operations I gave myself:
- Learn the target topology end-to-end in a sandbox. I built throwaway Atlas clusters, broke them on purpose, ran the production query shapes against them, and read Atlas's own playbooks until I could explain every decision.
- Profile the live workload. Real queries, real hot collections, real write patterns from production-shape data. The shard key candidates came from that profile, not from a generic "use _id" recommendation.
- Lock down the migration strategy on paper — including the dual-write phase, the validation queries, and the rollback path — before any production change was scheduled.
- Rehearse the cutover against a copy of production-shape data, twice, and have the rollback be just as rehearsed as the forward path.
§05
The migration plan
The plan was phased so that an issue at any single phase could be reversed without taking down patient services:
- Stand up the target. A fresh Atlas-managed Sharded Cluster sized off the profile, with the chosen shard key locked in and the network path corrected.
- Sync. Bulk-load existing data into the target, then turn on a continuous sync from the source ReplicaSet so the target stayed in step with live writes.
- Verify. Pre-cutover validation queries compared row counts, checksums, and a sample of live records between source and target until parity was provable.
- Cut over. Connection-string flip in a controlled window with the app's traffic re-pointed at the cluster. Live reads continued; the application never saw an unavailable database.
- Bake. Old source kept warm and reversible for a defined period before final decommission.
§06
The outcome
The Sharded Cluster came up clean. The platform crossed over without dropping a request and the headroom that had been gone for weeks was back. The team could now scale by adding shards rather than chasing a bigger VM, which was the architectural shift the system needed in the first place. And the corrected Azure routing meant tail latencies started lower than they had been before the ceiling problem ever surfaced.
§07
What I took from it
Three things from this engagement that I carry into every one since:
- "No prior experience" is not a disqualifier — but it is a schedule item. Carving out explicit time to study the target topology end-to-end was the single most important pre-condition for shipping this safely. Pretending I already knew it would have lost the engagement.
- Always check the network before you blame the database. The misconfigured Azure routing was not in scope, not on anyone's radar, and would have made the new cluster look bad if I'd left it. Free wins compound.
- Plan the rollback at the same depth as the forward path. On a vendor-locked production system serving patient care, the rollback isn't a footnote — it's the thing that makes the forward path safe to attempt. I now write both before I touch anything.
Challenges like this — high stakes, narrow constraints, a learning curve on top — are the engagements I look for. Architecture and engineering both get sharper when there's no room to guess.
Request a preview of Arky
Arky is in private preview with a small number of internal IT teams. Tell us about your estate and we will walk a real request through it, end to end, against your own scenarios.
What to include
- How many people field the questions and make the changes.
- The systems of record you run — issue tracker, wiki, repos, cloud accounts.
- The environment you would pilot in, and who would hold the approver role.
What happens next
- You get a reply within two business days.
- A 30-minute walkthrough: one question answered, one change gated and approved.
- If it fits, we scope a single-tenant preview deployment against your own accounts.
No form, no sequence, no sales list. The address goes to the operator.
Talk about an engagement
The studio also takes on backend, data, and integration work — the migrations, pipelines, and integration layers that other teams have already tried once. Send a short note about the problem you are fighting and a rough timeline.