MariaDB Foundation launches public library to detect performance regressions

The MariaDB Foundation has launched, within the Test Automation Framework (TAF), a public library of tests for detecting performance changes between database versions. This was announced in a post on the official MariaDB.org blog, written by Jonathan "Jeb" Miller. For anyone running MariaDB instances in production who needs to validate a migration or a release, this is a further point of reference alongside the usual private benchmarks or vendor-supplied figures: here, the test case, the script and the results are public and reproducible.

The first case: MDEV-32750

The first entry in the library is MDEV-32750, a behavioural variation discovered during a migration from MariaDB 10.4.10 to 10.11.4: the Adaptive Hash Index (AHI) behaves differently between the two versions, with effects on throughput at high thread counts. The workload used to highlight the issue is HammerDB TPROC-C, chosen because it mixes CPU-bound and lock-heavy OLTP operations, making the effects of the AHI visible.

The case has been turned into a deterministic, repeatable test consisting of: two properties files (AHI ON / AHI OFF), a multi-version runner script, a script for release testing, a HammerDB/TPROC-C workload calibrated for reproducibility, deterministic CPU behaviour, constant thread counts, checksums for correctness, and flamegraph generation for analysis.

Structure and release-testing scripts

The library lives in the mdevs/ folder of the TAF, with future subfolders planned for daily checks, weekly sweeps, release validation, pre-commit checks, investigation workloads and long-term stability testing. The commit also introduces a release-testing script designed for validating a single version: it automatically detects the TAF root, locates the MDEV properties folder, builds deterministic test tags, runs the selected test cases against the active MariaDB installation, and records the results. This is the script to use to work out whether a behaviour observed on a release candidate is expected or not.

The test infrastructure

According to the post, TAF 4.0 was developed and validated on hosts provided by Hetzner: one system with an Intel i5-13500 CPU (20 vCPUs/14 cores), 62 GB of RAM and AlmaLinux 9.7, and a second with an AMD EPYC 9454 CPU (96 vCPUs/48 cores), 125 GB of RAM and AlmaLinux 9.8.

The Foundation is inviting anyone wishing to contribute their own performance-detection test cases to come forward; it has promised to document the process for community contributions in due course. The MariaDB Foundation's annual survey is also open until 15 October 2026.