Test data organization¶
A dataset is the content of the results of multiple test suites. These are sent to calcite for each commit, and multiple datasets can be sent per commit.
Sending multiple datasets for a same commit can be used to enhance the results of the statistical analyses, or simply because you have different configurations.
- Test suite
A test suite regroups multiple tests, and are defined by the users. Depending on the importer used, it will correspond to different things.
This usually regroups runs of a same benchmark with different input data.
- Benchmark configuration
Each dataset is linked to a given configuration. This is how you usually differentiate datasets based on their execution environment (Hardware, OS, …), compiler options, application settings, etc.
Windows-10-64 Linux-x64-clang Linux-x64-gcc Runner-AMD-ryzen-3900X
A test can be anything from a microbenchmark to a performance load test. Those tests usually report multiple datapoints
Fibonacci10 Fibonacci20 canada.json one-json-per-line.jsons
A datapoint is a set of values sampled for a given test. Usually those values represent different iterations of the same test, and describe one of the characteristics of the test.
Execution time (ns) Loading time (ms) Cache miss rate (%) Branch misprediction rate (%) Memory footprint (MB) Executable Size (kB)
A calcite workflow can be triggered to initiate various actions such as sending notifications.