The outage that wasn’t our release

Shortly after we deployed one site, a separate commerce site in the same Cloud Manager program started failing on publish. Every page on it failed, not just some. Author was fine, the site we deployed was fine, and the timing pointed straight at our deploy.

A component on the commerce site made a blocking call to its commerce backend while it was being created, with no timeout and no error handling. Our deploy flushed the Dispatcher cache, which is normal. The wave of cache misses created that component on many requests at once, every thread waited on a slow backend, and the server ran out of threads. One component took down every page.

We held the theory to a simple standard: it had to explain every fact, not just the convenient ones.

• Only publish was affected; author stayed healthy
• The flagship site, deployed at the same moment, was unaffected
• Removing the one fragment that held the component restored the whole site
• The logs showed the backend calls failing at exactly that time

We also found a real configuration conflict between the codebases during the investigation. It needed fixing, but it wasn’t the cause, and saying so clearly kept everyone focused on the right fix.

Clear guidance for the owning team: timeouts on every outbound call, graceful failure, and moving network calls out of component initialization. Then we asked the obvious next question: does our own code have the same weakness? That became a whole-codebase audit.

This is one story from a playbook of about twenty diagnoses we’ve made on the same platform. A few of the others:

• A build that failed Adobe’s checks only on dev, traced to a platform-side setting that Adobe confirmed had been switched on by mistake
• Developer disks filling at tens of gigabytes an hour, traced to a connector reinstalling itself in a loop
• Author dialogs that locked the whole page, traced to one case-sensitive class name the editor expects
• Pages that rendered with no styles, traced to a page model resolving back to itself

The pattern is always the same: isolate, find the root cause, explain every symptom, ship a safe workaround, then fix it properly and write down how to spot it next time.