Summary
Materialize has a dedicated QA team, currently consisting of two members. Developers at Materialize are encouraged to write their own tests for the features that they work on. This adds another layer of quality assurance to our process. The QAteam writes tests using the least powerful tool that covers the feature in question.
Unit tests are an area the QA team is rarely involved in, since our approach is to consider all of Materialize as one system. Miri is a Rust interpreter that can run a subset of our unit tests and detect undefined behavior. The rest of this blog will describe tests utilizing a more full-fledged Materialize instance.
Toxiproxy is a TCP proxy that can simulate various network problems. Materialize uses Toxipoxy to verify that it recovers properly from connection problems. The tool is also used to test the reliability of Materialize’s database queries. It is also a way to test Materialize's database queries against other databases.
SQLsmith and SQLancer are both open source tools that work for many different DBMSes. They don’t utilize some of the most interesting parts of Materialize, like sources, sinks and materialized views. Zippy creates random Testdrive fragments which ingest data into Materialize.
Introducing full code coverage for a mature project is difficult, and code coverage can easily be gamed. Instead, we opted for checking the code coverage of risky pull requests. Sanitizers like the Address Sanitizer in all of our code, including the C/C++ libraries, for extended testing.