When A Bucket Versioning Change got Customer Support inundated

This is a recollection about the afternoon a distributed lock that had worked perfectly for years suddenly turned into a bottleneck, piled up thousands of jobs, and made customer flows that normally took a few seconds hang for the entire morning, all without a single application deploy.

The incident started building quietly through the morning and broke into a full-blown SEV around 3 PM, when customers started complaining that flows which normally completed in seconds had been hanging since morning. By the time it surfaced, the job queue was in tens of thousands.

The Pattern We Trusted

To understand why this hurt so much, you need to know the pattern the team was relying on.

The workload was a set of .NET container app replicas running as background jobs, capped at a maximum of six. Each replica would pick up a row (task) from a MSSQL database, process it, and delete the row. Classic distributed coordination problem, i.e. making sure two replicas don’t grab the same task at the same time.

The team’s solution for tracking job state was a single JSON file on Azure Blob Storage, protected by a blob lease so only one replica could update it at a time.

The flow executed four serialized Azure REST API calls per job: Lease Blob (Acquire), Get Blob, modify the JSON locally, PUT Blob (passing the lease ID header), and Lease Blob (Release). One single blob file, shared across all replicas, acting as a pessimistic concurrency lock.

They had been explicitly asked not to keep locking mechanism on MSSQL. The database was already running hot, north of 70% utilization even with 120 cores provisioned. Adding more writes for status tracking would have made a bad situation worse. So the blob file became the write path for coordination state, and the DB ops stayed at SELECT and DELETE.

It’s a cheap pattern on paper. Blob writes are normally in the ~30ms range. You don’t need to run a separate coordination service like Redis or a consensus system. You just lean on the storage account you already have. For a long time, this worked perfectly. The team had been running it for ages, throughput was fine, latency was fine, nobody thought about it.

The Morning It Started Going Wrong

The first sign was a wave of complaints of slowness from clients via Customer Support. Not a regular in-house alert.

Not a hard crash, not a deployment failure, just a slow, ugly climb in response times. Customers had already been feeling it since morning, but it took until around 3 PM for the volume of complaints to turn it into a declared incident.

By the time platform (I) was pulled in to the incident call, the count of rows on the DB tables had reached 30k. Jobs that should have been picked up and processed in seconds were sitting there waiting and accumulating. Workers were still working as usual, however, as per the team, a 50ms trace is now 3s long, pointing me to their Coralogix dashboard.

Important thing was that the traces had fattened around Blob Storage, whereas CPU, Mem metrics were far underutilised.

At some point during the incident, the devs asked whether they should increase the replica count beyond the six-replica cap. I told them no. It was difficult to establish that the bottleneck was compute, throwing more replicas at an underutilised deployment does not jibe with logic.

It was something else, and clearly on Azure.

The First Hypothesis: It’s Azure’s Fault

Blob Storage has felt unreliable in the past, we’d seen transient throttling, the occasional regional hiccup, the kind of things that makes Azure the usual suspect. So the first hypothesis was: there’s an Azure outage, and Blob Storage is degraded.

I started there too, because you have to rule it out before you go deeper. And the way to rule it out is to look at the other apps on the same Blob service.

We had other applications hitting the same Blob service, the same storage account tier, the same region. They were all healthy. Their latency was normal. Their throughput was normal. No 500s, no throttling, no anomalies.

So it wasn’t a broad Azure outage. Whatever was happening was specific to this team’s usage of Blob, not to Blob itself.

That narrowed the search, but it also made the problem scarier. If it’s not Azure, and it’s not our code, then something in our control plane changed without us knowing. And we had no idea what.

Going By The Traces

Just the PUT Blob call, the same read-modify-write on the shared JSON status file that every replica depended on, that used to take ~30ms was now taking ~2 seconds.

Not 30ms to 60ms. Not 30ms to 300ms. 30ms to two full seconds. That’s roughly a 100x slowdown, on the exact call that every single job was making to update job status in the shared JSON file.

That single number explained everything. With every status-file update taking 2s instead of 30ms, every replica was spending most of its time just waiting to read-modify-write the shared JSON. Throughput collapsed. The queue piled up.

So now the question had a sharp edge: what could make a PUT Blob go from 30ms to 2s, with no code change, on a storage account that was otherwise healthy?

Why A 2 sec PUT Makes Perfect Sense

Going back in time on Coralogix, it was clear that this spike in latency is not new, it also happened day before, since 3 days, however, due to the nature of the app, it was the day when customers most used the settlement feature.

So I started digging around what changed exactly 3 days back.

The team accepted a version upgrade PR from Renovate on the Terraform module they were using for IaC.

I looked at the Apply execution on Octopus, and saw that the Storage Account versioning flipped to enabled.

I then Googled, “versioning, azure, blob storage, latency”. And that was it. The first result itself was a Microsoft link saying:

Having a large number of versions per blob can increase the latency for blob listing operations. Microsoft recommends maintaining fewer than 1000 versions per blob. You can use lifecycle management to automatically delete old versions.

While the doc explicitly called out listing operations (List Blobs), our issue was on the write path (PUT Blob). But the recommendation at the end of that same paragraph was also unmistakable:

“To mitigate these concerns, store frequently overwritten data in a separate storage account with versioning disabled.”

I check next the number of versions of the json file, it was north of 60k. And with every increment it, as per Microsoft, must be getting slower.

I immediately asked the team on the call, do you need versioning on this, they said no.

I announce and I then disable versioning of the Storage Account via az CLI.

Within a few seconds magic happens, traces slim down, from 3s back to 40ms.

With Blob writes cheap again, the job queue started draining. Within about three hours, the queue was empty and we were back to normal throughput.

I declared recovery once the queue was fully drained and the endpoint latency had been back in the normal range for a sustained period. The incident was closed.

The tension had released. It was my time to explain what had really transpired over the previous one week.

The Math of the Queue Collapse

Under normal conditions without versioning, a PUT Blob is an in-place replace on Azure’s storage metadata layer. Cheap. ~20-50ms.

When blob versioning is enabled, every PUT Blob triggers a multi-step metadata sequence on Azure’s storage backend:

1. Version ID allocation: Azure creates a brand-new immutable version metadata object, tagged with a microsecond-precision timestamp.
2. Lease validation: Azure checks the active lease ID against the container/blob lease table.
3. Pointer swap: Azure updates the “current version” pointer to the newly written data while archiving the old metadata entry.

When six replicas are constantly hitting the exact same single blob in a loop, Azure’s internal partition handling that blob’s metadata hits severe contention. The service-side latency spikes, turning what is normally a 30ms API call into a 1.5 to 3-second PUT request. The 2s trace I was seeing in Coralogix was right in the middle of that range.

And because the replicas used a lease lock on a single shared blob, all updates to the JSON file were strictly serialized. Only one replica could hold the lease and execute the PUT at any given moment. The throughput ceiling was thus brought down to 1% of normal state.

Normal state (~30ms PUT):

• Acquire lease + read + write (PUT) + release lease ≈ 50ms total lock duration
• Max system throughput: 1000ms / 50ms = ~20 jobs processed per second across all replicas

Incident state (~2,000ms PUT as seen in Coralogix):

• Acquire lease + read + write (PUT) + release lease ≈ 2,100ms total lock duration
• Max system throughput: 1000ms / 2100ms = ~0.47 jobs per second - less than one job every 2s

Now look at what happens when incoming work outpaces that ceiling. If jobs were arriving into MSSQL at even a modest 10 jobs per second:

• Processing rate during the incident: ~0.5 jobs/second
• Net queue growth: +9.5 jobs piled up in MSSQL every single second
• After one hour: 9.5 × 3,600 = ~34,000 queued jobs
• After a full morning (4-5 hours): tens of thousands of jobs waiting to be picked up

The lock pattern was still working logically. Leases were being acquired, JSON was being updated, no data corruption. But versioning had destroyed the serialized throughput ceiling of the system. The MSSQL job queue didn’t explode because the workers broke. It exploded because each worker could only complete a status update every 2s instead of every 50ms.

The Real Root Cause: A 3 Day Old IaC Change

The versioning change was distributed through a routine version bump PR of our Atlas Terraform module by the Renovate bot. A commit authored by the security team had enabled blob versioning by default, silently applying the setting to any service that upgraded to the new module version.

The change was small and, on the face of it, sensible: it set blob versioning on by default across storage accounts. Versioning was an optional Terraform field, and the security team had flipped it on as a blanket policy, presumably for data protection, recovery, audit reasons. All good intentions.

Downstream repos who do not define the default, would automatically get it enabled.

The Fixes

With the root cause confirmed, the fix had three parts: stop the bleeding now, make sure it didn’t happen again, and do it the right way.

Immediate:

I disabled versioning on the affected storage account via the az CLI rather than Terraform, because we needed speed. A Terraform apply would have gone through the normal review and apply cycle, and we had thousands of jobs piling up, we didn’t have time for that.

Short term:

Once the incident was closed, the team pinned versioning_enabled = false explicitly in their own Terraform repo for that storage account. The security team’s had flipped it on by default; the team’s repo now had an explicit override so the next apply wouldn’t silently re-enable it.

The Right Way:

Once the fire was out, the incident sat with me, because it had surfaced something that had been true for a long time but had never been forced into the open.

Blob-as-coordination-layer is fragile under load. It had worked for years because the contention had never been high enough to expose the fragility. The pattern was always one bad day away from falling over - it just needed the right push. In this case, the push was a silent versioning change that inflated each serialized PUT from 30ms to 2s, collapsing the throughput ceiling from ~20 jobs/second to less than one. In another universe, it could have been a traffic spike, a new high-cardinality job type, a regional Azure degradation, anything that pushed the write contention past the threshold where the cheap-overwrite assumption held.

The right tools for this job are systems built for high-concurrency key-value updates.

For their use case, tracking job IDs and statuses, relatively low cardinality, no need for complex queries, I recommended Cosmos DB. At their volume and access pattern, it would have been effectively free.

If they wanted something simpler to operate and already used to, managed Redis was also available. Either way, the job status writes needed to move off the shared blob file.

What I Took Away From This

A few things stuck with me after this one.

Silent, org-wide policy changes are dangerous in proportion to how many teams are using the affected resource. The security team did something reasonable. Turn on versioning by default for data protection. That’s a defensible policy. But “by default” means it applied to storage accounts being used in ways the policy authors didn’t necessarily anticipate, like a high-write-concurrency coordination file. The more teams using a resource, the more likely it is that someone is using it in a way that a blanket policy will break. Policy changes at the platform level need a blast-radius review, not just a security review.

Scaling replicas is not always the answer. When the devs asked about bumping past the six-replica cap, the instinct was understandable, more workers should drain the queue faster. But without evidence that compute was the constraint, adding replicas would have increased contention on the single JSON file and made things worse. Know what you’re bottlenecked on before you scale.

And finally “works” is not the same as “is right.” The blob-file-as-coordination pattern had worked for years. That’s a very powerful argument for leaving it alone, if it ain’t broke, don’t fix it. But “works” just means the failure conditions haven’t been hit yet. The pattern was always one bad day away from falling over. It didn’t look broken, because the thing that would break it, high write contention on a versioned blob, hadn’t happened yet. When a pattern works by leaning on an assumption (cheap overwrites, low contention, no versioning), the pattern is one silent change away from not working. Architectural doubt that you silence because “it works” is debt you’re going to pay back in an incident one day. We paid it back at 3 PM on a Thursday.

Comments/feedback?

Leave your comments/feedback below. Have you ever been bitten by a silent platform-level policy change that broke an assumption your code was quietly leaning on? Did you find it fast, or did it take a queue in the thousands to surface? And once you found it: did you fix the config, or did you fix the pattern?