By: Oussama Jarrousse

Yesterday’s emergency PHP security release was another race against the clock.

At Simula, we had prepared for the PHP2Shell vulnerabilities (CVE-2026-109618 and CVE-2026-109619) ahead of the security release scheduled for October 21 at 14:00 UTC.

The time between vulnerability disclosure and exploitation continues to shrink, sometimes to the point where exploitation precedes the official disclosure or patch release. We wanted to be ready. So we scheduled our response for October 21 at 14:01 UTC, just one minute after the announced release.

Thanks to our Infrastructure as Code (IaC), standardized deployments, and centralized observability, our response plan was relatively straightforward: execute a prepared patching and verification script across affected, internet-facing customer deployments as soon as the security updates became available.

Then we encountered something we hadn’t anticipated. Our script couldn’t find the patched packages.

The problem wasn’t the script or the servers themselves. Several systems were configured to use cloud-provider package mirrors and those mirrors hadn’t synchronized the security updates yet.

We quickly switched to Plan B: temporarily restrict public access to the affected sites, replace the cloud-provider mirrors with the official repositories, rerun the patching and verification process, and restore access after confirming the updates had been applied. It worked, but it came at the cost of some downtime.

That wasn’t ideal, especially for customer-facing services. But given the severity of the vulnerabilities and the uncertainty around exploitation, we preferred a controlled interruption over leaving potentially vulnerable services exposed.

Our observability stack also allowed us to review activity over the preceding 30 days. So far, we haven’t identified indicators suggesting successful exploitation of PHP2Shell on the affected systems. That is reassuring, although absence of detected indicators is not proof of absence of compromise.

What we learned

We had standardized deployments, an automated response, and the ability to act across our infrastructure quickly. What we hadn’t adequately accounted for was the propagation delay between upstream security releases and third-party package mirrors. It’s a relatively small detail in normal operations, but during an emergency security release, it can become a critical bottleneck.

More importantly, this incident reinforced the importance of a fallback that remains viable when the assumptions behind the original plan fail.

In our case, Plan B meant temporarily taking services offline until we could patch them. Without that option, we would have faced a much more difficult choice between availability and security.

Infrastructure as Code and automation made our initial response fast and repeatable. Observability helped us assess what had happened. But neither eliminated the need to make operational decisions when something unexpected occurred.

Incident response readiness isn’t just about having automation. It’s about understanding the dependencies that automation relies on, recognizing when those dependencies fail, and having a way to respond safely when they do.

Yesterday was a useful reminder of that.

And it’s an experience we’ll incorporate into how we prepare for the next incident.

Going forward, we’re making fallback strategies an explicit part of every incident response plan. That means identifying critical dependencies, anticipating plausible failure scenarios, and preparing alternative courses of action, including containment measures when remediation isn’t immediately possible. We can’t anticipate every surprise, but we can make sure that a failure in our primary response plan doesn’t leave us without options.

References