Concepts
========

Test data organization
----------------------

In calcite, the performance tests are submitted in the form of a :term:`Dataset` and organized as :term:`Test suite`/:term:`config <Benchmark configuration>`/:term:`Test`/:term:`Datapoint`.

.. glossary::
    Dataset
        A dataset is the content of the results of multiple test suites.
        These are sent to calcite for each commit, and multiple datasets can be sent per commit.

        Sending multiple datasets for a same commit can be used to enhance the results of the statistical analyses, or simply because you have different configurations.




    Test suite
        A test suite regroups multiple tests, and are defined by the users.
        Depending on the :ref:`importer <usage/upload:Available importers>` used, it will correspond to different things.

        This usually regroups runs of a same benchmark with different input data.

        Examples:
        
        .. code-block::

            Fibonacci
            ParseJSon
  

    Benchmark configuration

        Each dataset is linked to a given configuration.
        This is how you usually differentiate datasets based on their execution environment (Hardware, OS, ...), compiler options, application settings, etc.

        Examples:
        
        .. code-block::

            Windows-10-64
            Linux-x64-clang
            Linux-x64-gcc
            Runner-AMD-ryzen-3900X

    Test

        A test can be anything from a microbenchmark to a performance load test.
        Those tests usually report multiple :term:`datapoints <Datapoint>`

        Examples:
        
        .. code-block::

            Fibonacci10 
            Fibonacci20

            canada.json
            one-json-per-line.jsons
            


    Datapoint

        A datapoint is a set of values sampled for a given test.
        Usually those values represent different iterations of the same test, and describe one of the characteristics of the test.

        Examples:
        
        .. code-block::

            Execution time (ns)
            Loading time (ms)
            Cache miss rate (%)
            Branch misprediction rate (%)
            Memory footprint (MB)
            Executable Size (kB)

    Workflow

        A calcite workflow can be triggered to initiate various actions such as sending notifications.