Ryan McCurdy, VP with Liquibase, the leader in database DevOps, has just published “The Day a Database Permission Change Broke the Internet: A Cloudflare Story.” His analysis details how:
- A minor adjustment, routine in most organizations, touched a hidden part of Cloudflare’s architecture and awakened a dependency no one had considered dangerous.
- What happened next revealed how modern systems fail today: Nodes that loaded the expanded file went dark. Nodes that loaded the old file continued to serve traffic. The network oscillated, recovering for minutes at a time before failing again, as if trapped between two different realities.
- Once every shard of the ClickHouse cluster adopted new permissions, every file produced was oversized and every proxy that touched it entered the same panic.
The analysis details clearly and compellingly how the cascading failure occurred, and ways in which most data-driven organizations are at risk.
He notes:
“Cloudflare is one of the most capable engineering organizations in the world. Their systems are built to survive pressure that would overwhelm most companies. Their teams live in incident response. Their infrastructure is distributed, hardened, and instrumented with extraordinary detail. Yet the event that brought them down started with a quiet change in who could read what inside a database.”
He concludes by noting that the only real path forward is a new level of discipline at the data layer. Databases must be governed with the same rigor applied to application pipelines, and offers specific vendor-neutral recommendations.
Like this:
Like Loading...
Related
This entry was posted on November 20, 2025 at 4:39 pm and is filed under Commentary with tags Liquibase. You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
The Day a Database Permission Change Broke the Internet: A Cloudflare Story – Liquibase analysis
Ryan McCurdy, VP with Liquibase, the leader in database DevOps, has just published “The Day a Database Permission Change Broke the Internet: A Cloudflare Story.” His analysis details how:
The analysis details clearly and compellingly how the cascading failure occurred, and ways in which most data-driven organizations are at risk.
He notes:
“Cloudflare is one of the most capable engineering organizations in the world. Their systems are built to survive pressure that would overwhelm most companies. Their teams live in incident response. Their infrastructure is distributed, hardened, and instrumented with extraordinary detail. Yet the event that brought them down started with a quiet change in who could read what inside a database.”
He concludes by noting that the only real path forward is a new level of discipline at the data layer. Databases must be governed with the same rigor applied to application pipelines, and offers specific vendor-neutral recommendations.
Share this:
Like this:
Related
This entry was posted on November 20, 2025 at 4:39 pm and is filed under Commentary with tags Liquibase. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.