
Automated testing to find logic and performance bugs in database systems

SQLancer is a tool to automatically test Database Management Systems (DBMSs) in order to find bugs in their implementation. That is, it finds bugs in the code of the DBMS implementation, rather than in queries written by the user. SQLancer has found hundreds of bugs in mature and widely-known DBMSs.
SQLancer tackles two essential challenges when automatically testing the DBMSs:
Community. We have a Slack workspace to discuss SQLancer, and DBMS testing in general. Previously, SQLancer had an account on Twitter/X @sqlancer_dbms, which is no longer maintained. We have a blog, which, as of now, contains only posts by contributors of the Google Summer of Code project.
Minimum Requirements:
The following commands clone SQLancer, create a JAR, and start SQLancer to test SQLite using Non-optimizing Reference Engine Construction (NoREC):
git clone https://github.com/sqlancer/sqlancer
cd sqlancer
mvn package -DskipTests
cd target
java -jar sqlancer-*.jar --num-threads 4 sqlite3 --oracle NoREC
Running and terminating. If the execution prints progress information every five seconds, then the tool works as expected. The shortcut CTRL+C can be used to terminate SQLancer manually. If SQLancer does not find any bugs, it executes infinitely. The option --num-tries can be used to control after how many bugs SQLancer terminates. Alternatively, the option --timeout-seconds can be used to specify the maximum duration that SQLancer is allowed to run.
Parameters. If you launch SQLancer without parameters, available options and commands are displayed. Note that general options that are supported by all DBMS-testing implementations (e.g., --num-threads) need to precede the name of the DBMS to be tested (e.g., sqlite3). Options that are supported only for specific DBMS (e.g., --test-rtree for SQLite3), or options for which each testing implementation provides different values (e.g. --oracle NoREC) need to go after the DBMS name.
DBMSs. To run SQLancer on SQLite, it was not necessary to install and set up a DBMS. The reason for this is that embedded DBMSs run in the same process as the application and thus require no separate installation or setup. Embedded DBMSs supported by SQLancer include DuckDB, H2, and SQLite. Their binaries are included as JAR dependencies. Note that any crashes in these systems will also cause a crash in the JVM on which SQLancer runs.
Logs. SQLancer stores logs in the target/logs subdirectory. By default, the option --log-each-select is enabled, which results in every SQL statement that is sent to the DBMS being logged. The corresponding file names are postfixed with -cur.log. In addition, if SQLancer detects a logic bug, it creates a file with the extension .log, in which the statements to reproduce the bug are logged, including only the last query that was executed along with the other statements to set up the database state.
Reducing bugs. After finding a bug-inducing test input, the input typically needs to be reduced to be further analyzed, as it might contain many SQL statements that are redundant to reproduce the bug. One option is to do this manually, by removing a statement or feature at a time, replaying the bug-inducing statements, and applying the test oracle (e.g., for test oracles like TLP or NoREC, this would require checking that both queries still produce a different result). This process can be automated using a so-called delta-debugging approach. SQLancer includes an experimental implementation of a delta debugging approach, which can be enabled using --use-reducer. In the past, we have successfully used C-Reduce, which requires specifying the test oracle in a script that can be executed by C-Reduce.
Testing the latest DBMS version. For most DBMSs, SQLancer supports only a previous release version. Thus, potential bugs that SQLancer finds could be already fixed in the latest development version of the DBMS. If you are not a developer of the DBMS that you are testing, we would like to encourage you to validate that the bug can still be reproduced before reporting it. We would appreciate it if you could mention SQLancer when you report bugs found by it. We would also be excited to hear about your experience using SQLancer or related use cases or extensions.