On April 21, 2026, thousands of cPanel-managed servers running MySQL silently upgraded themselves to a major version they were never supposed to be on. By the time server owners noticed — often because their customers were calling about websites returning database connection errors — the damage was already done. This is a post-mortem of what happened, why cPanel servers were structurally unable to prevent it, and what a different architectural approach looks like.
What Happened
April 21, 2026 was a significant day for MySQL: Oracle published both MySQL 9.7.0 (the latest Innovation release) and MySQL 8.0.46 — the final release of the MySQL 8.0 branch, marking it as End of Life. Under normal circumstances, this would be a routine release announcement. Administrators running 8.0.x would note the EOL, plan a deliberate migration to 8.4 LTS, and move on.
But something went wrong in the official MySQL APT and YUM repository metadata that day. The repository's package priority signaling indicated MySQL 9.7 as the upgrade successor to 8.4 LTS — treating an Innovation track release as though it were a stable LTS continuation. On servers where the package manager was configured with automatic upgrades enabled, apt dist-upgrade and yum update resolved this as a valid upgrade path and proceeded to install MySQL 9.7.0 without any major-version warning being surfaced to the operator.
Reports started appearing in forums and support channels around 19:10 Lisbon time. The trigger, in most confirmed cases, was cPanel's nightly automated update process — a scheduled background job that runs unattended, applies system package updates, and completes without operator review.
The error message that greeted administrators the next morning:
ERROR 2002 (HY000): Can't connect to local MySQL server
through socket '/var/lib/mysql/mysql.sock' (2)Or, from PHP applications:
mysqli_connect(): (HY000/2002): No such file or directory
PDO::__construct(): SQLSTATE[HY000] [2002] No such file or directoryAnatomy of the Upstream Bug
To understand why this happened, it helps to understand how Linux package managers handle version tracks. MySQL publishes packages under two parallel tracks: Innovation releases (rapid feature releases, not intended for production stability) and LTS releases (stable, long-term support). For the package manager to respect this distinction, the repository metadata must correctly set package epochs and priority values that prevent the package manager's dependency resolver from treating an Innovation track package as an upgrade to an LTS track package.
The metadata published on April 21 failed to enforce this boundary. From the package manager's perspective, MySQL 9.7 appeared as the logical next version after 8.4 LTS. The resolver followed the version number — 9.7 is numerically greater than 8.4 — and treated it as an upgrade candidate.
For servers running unattended-upgrades on Debian/Ubuntu, or yum-cron / dnf-automatic on RHEL-based systems with default configurations, this resolved to an automatic major version jump with no human in the loop.
Why cPanel Servers Were Helpless
cPanel ships with nightly automatic updates enabled by default. The rationale is reasonable: security patches should not sit uninstalled on production servers. The problem is that the same mechanism that applies a critical libssl patch is also the one that, in this case, applied a MySQL major version upgrade.
cPanel's official response to the incident was to advise administrators to temporarily disable automatic MySQL updates. There was no automated rollback path. Rolling back a MySQL major version upgrade is not a straightforward operation: the data directory format changes between major versions, mysql_upgrade is not designed to run in reverse, and in-place downgrade is not supported. The only reliable path back is a database dump from before the upgrade — which most affected servers did not have from that specific point in time.
Percona, which maintains parallel MySQL-compatible builds, announced it would not be producing MySQL 9.x Innovation packages. The community signal was clear: 9.7 on a production hosting server was not where anyone wanted to be.
Blast Radius: What Actually Broke
The breakage was not limited to "MySQL won't start." The MySQL 9.7 upgrade introduced changes that cascaded across the entire LAMP stack:
- Authentication plugin change: MySQL 9.x deprecates the
mysql_native_passwordauthentication plugin more aggressively than 8.4 did. Older PHP versions (7.4, 8.0) usingmysqliorPDO_MySQLextensions compiled against older client libraries do not negotiatecaching_sha2_passwordcorrectly in all configurations. Connection failures followed. - Socket path behavior: On some configurations, MySQL 9.7's default socket path differed from the path hardcoded in application configuration files and PHP's
mysql.default_socketini directive. - No
mysql_upgraderun: An automatic package manager upgrade does not runmysql_upgrade— the tool that migrates system tables, user grants, and internal metadata to the new format. Servers that "survived" the initial upgrade were often running with a partially inconsistent data directory. - Moodle and WordPress: Both platforms have documented minimum MySQL version requirements and behavior expectations around transaction isolation levels and full-text index formats. MySQL 9.7 changed default values for several system variables that these platforms relied on implicitly.
What Panelica Does Differently
This incident is worth analyzing not to criticize how cPanel handles MySQL — nightly updates are a reasonable policy decision with real security justification — but because the failure mode reveals a structural assumption: that upstream package repositories can be trusted not to push breaking changes through automated update channels. April 21 demonstrated that this assumption does not always hold.
Panelica's architecture was built around a different assumption: that upstream trust must be verified at multiple checkpoints before reaching production.
1. Version Pinning in Build Recipes
Panelica does not install MySQL from the OS package manager on customer servers. Every managed service is built from a declarative recipe by pn-builder, Panelica's internal build tool. The MySQL recipe specifies the exact download path:
# builder/recipes/mysql.yaml (simplified)
source:
url_template: "https://cdn.mysql.com/Downloads/MySQL-8.0/mysql-8.0.{{version}}-linux-glibc2.28-x86_64.tar.xz"
version: "8.0.46"
build:
health_check: |
mysqld --version
mysqladmin --socket=/opt/panelica/var/run/mysqld/mysqld.sock ping
mysql --socket=/opt/panelica/var/run/mysqld/mysqld.sock -e "SELECT 1"The URL template points to the MySQL-8.0/ directory. Even if the upstream repository reclassifies MySQL 9.7 as an 8.x successor, pn-builder never consults the package manager's dependency resolver. It fetches the binary at the exact path specified by the recipe. Moving to a new MySQL major version requires a deliberate recipe change by a Panelica maintainer — it cannot happen through metadata drift.
2. Operator-Gated Release
When pn-builder builds a new MySQL package and uploads it to Panelica Central, the package is set to status = "testing" and is_published = false. Customer servers only pull packages marked as released. That status transition requires a Panelica maintainer to log into Central admin, review the package, and manually mark it as released.
No amount of upstream metadata confusion can cause a package to go from build output to customer production without a human making a deliberate decision. The automation handles the build and the upload. The release gate requires a human.
3. No Silent Nightly Updates
Panelica customer servers do not run a background cron job that applies updates. The update flow is entirely panel-UI-driven: the administrator opens the panel, sees "Update Available," reviews the changelog, and clicks "Apply." The panel applies the update, shows progress, and reports success or failure. If the administrator never clicks Apply, the server stays on the current version indefinitely.
This means updates do not happen at 3 AM while the administrator is asleep. They happen when the administrator decides they should happen.
4. Automatic Rollback on Health Check Failure
Before applying any service update, pn-builder backs up the current binary to builder/backups/. After installing the new binary, it runs the health check defined in the recipe. For MySQL, this includes a connection test through the socket. If the health check fails, pn-builder automatically restores the backed-up binary and reports the failure. The service continues running on the previous version.
A broken MySQL build cannot silently land on a production server. If it cannot pass its own health check, it does not replace the working version.
5. SHA256 Checksum Verification on Every Pull
When a customer server downloads a package from Central, it verifies the SHA256 checksum of the archive before extracting or applying it. A corrupted download, a CDN-level tampering attempt, or a repository compromise that serves modified binaries will be caught before the binary executes on the customer's server. This is a straightforward defense that adds negligible overhead and eliminates an entire class of supply chain attack.
The Broader Principle
The MySQL 9.7 incident is an example of supply chain trust failure. The affected administrators were not running untrusted software. They were running software from official MySQL repositories, managed by a panel from an established hosting software vendor, with what seemed like a reasonable default: keep software updated automatically.
The failure was upstream — but the exposure was created by the absence of gates between upstream and production. Automation is not the problem. Automation without checkpoints is the problem.
The right model is: automate the fetch, the build, and the health verification. Keep a human in the loop for the release decision. Never let a version number change propagate to production without someone looking at it. Speed and safety are not in conflict here — the build pipeline can be fast, and the release gate can still be deliberate.
Takeaways for Self-Hosted Panel Administrators
Whether you run Panelica or any other panel, the following practices would have prevented or significantly limited the impact of this incident:
- Pin your MySQL major version in whatever package configuration your system uses. On APT systems,
apt-mark hold mysql-server. Do not leave major version selection to the package manager's resolver. - Separate security updates from package upgrades. Security patches for known CVEs are different from new versions. Configure
unattended-upgradesto apply security patches only, not arbitrary version upgrades. - Require manual approval for any service that touches your data layer. Databases and their configuration are not appropriate candidates for silent automatic updates.
- Have a tested rollback procedure for every service on your server before you need it. A database backup from six hours before the incident is not the same as a rollback procedure.
- Know what your panel's nightly update job does and what package sources it consults. If you do not know, assume it can upgrade anything and plan accordingly.
The April 2026 incident was not unique in kind — similar events have occurred with other software components over the years when upstream repository metadata produced unexpected upgrade paths. It will happen again with some other package, from some other vendor. The question is not whether you trust upstream; it is whether you have guardrails that catch the cases where upstream gets it wrong.