Concepts

Test data organization

In calcite, the performance tests are submitted in the form of a Dataset and organized as Test suite/config/Test/Datapoint.

Dataset

A dataset is the content of the results of multiple test suites. These are sent to calcite for each commit, and multiple datasets can be sent per commit.

Sending multiple datasets for a same commit can be used to enhance the results of the statistical analyses, or simply because you have different configurations.

Test suite

A test suite regroups multiple tests, and are defined by the users. Depending on the importer used, it will correspond to different things.

This usually regroups runs of a same benchmark with different input data.

Examples:

Fibonacci
ParseJSon

Benchmark configuration

Each dataset is linked to a given configuration. This is how you usually differentiate datasets based on their execution environment (Hardware, OS, …), compiler options, application settings, etc.

Examples:

Windows-10-64
Linux-x64-clang
Linux-x64-gcc
Runner-AMD-ryzen-3900X

Test

A test can be anything from a microbenchmark to a performance load test. Those tests usually report multiple datapoints

Examples:

Fibonacci10
Fibonacci20

canada.json
one-json-per-line.jsons

Datapoint

A datapoint is a set of values sampled for a given test. Usually those values represent different iterations of the same test, and describe one of the characteristics of the test.

Examples:

Execution time (ns)
Loading time (ms)
Cache miss rate (%)
Branch misprediction rate (%)
Memory footprint (MB)
Executable Size (kB)

Workflow

A calcite workflow can be triggered to initiate various actions such as sending notifications.