To upload a dataset to calcite, we recommand using our Command line interface.

Hint

npx @siliceum/calcite-cli [command arguments] let you run the command line directly.

Each project has a unique token, which is used when uploading a dataset. It can be found in the project details page, under the description of the project.

A dataset contains your datapoints and some metadata. It can either be provided in our json format or through a configuration file used to import the data.

Hint

A fake_bench.json file is provided as part of the example, and this is what we will use for our first upload.

Setup your CI so that the CALCITE_TOKEN environment variables contains your project token, and use the following command to upload the ./fake_bench.json dataset:

calcite upload ./fake_bench.json


Note

This should automatically detect the git branch, commit, etc (see the supported CI providers).
We however suggest overriding the default dataset config name.

upload one dataset to a calcite server, by specifiying the config file, the gitRef and the commit id.

-u, --url <server url>

Set the target server url, for preview and enterprise users. The CALCITE_URL environment variable can also be used, otherwise it will be set to the public server.

-t, --token <token id>

Set the project token. By default the CALCITE_TOKEN environment variable is used.

-C, --commit <sha>

Specify the full git commit sha.

-R, --ref <gitref>

Specify the git ref.

-BC, --benchconfig <sha>

Specify the benchmark config name.

-BID, --buildId <id>

Specify the CI build number.

-PR, --pullRequestId <id>

Specify the pull request number.

--dry-run

-h, --help

display help for command

## Dataset formats¶

To understand why the data is organized the following way, please have a look at the Concepts documentation.

### calcite JSon schema¶

calcite uses its own json schema to describe a dataset. Ultimately, this is what will be sent to the server, but you would usually create this file using an importer.

A lot of importers are being added, and will be open-sourced. Do not hesitate to contact us if you created a new one!

The schema is available here.

calcite dataset

A benchmark dataset for use with calcite

metadata:object[optional]
testSuites:array
Items must be of type: #/definitions/TestSuite
definitions
TestSuite

A test suite regroups similar tests, or one test that was ran with different inputs.

name:string
tests:array
Items must be of type: #/definitions/Test
Test

A test can report multiple datapoints, such as time, memory usage, etc. Tests using different input data are usually grouped in the same TestSuite

name:string
dataPoints:array
Items must be of type: #/definitions/Datapoint
Datapoint:object
name:string
values:array

The values that were sampled during the test.

unit:string

The unit of this datapoint (s, ms, ns, Mb, Gb, MB, GB, W, …). This can be used to infer default values for other parameters of the datapoint.

aggregationPolicy

This is the aggregation method used to summarize all values of a given commit and used for diffs between commits.

Values:
• mean
• min
• max
• median
diffPolicy

This describes how to compute the value used for regression detections.

• substraction = target - baseline

• relativeDifference = 100 * (target - baseline)/baseline (percentage)

• none = ignore this datapoint

Values:
• substraction
• relativeDifference
• useTargetValue
• none
regressionPolicy

Describes how fluctuations of values must be interpreted for regressions. absoluteDifferenceThreshold means that changes in any direction will be interpreted as regressions. equal means that any value different than regressionArgument will be treated as a regression.

Values:
• lessIsBetter
• moreIsBetter
• absoluteDifferenceThreshold
• equal
regressionArgument:number[optional]

This is the argument used in conjonction with the diff and regression policies. This could be an absolute value or percentage, based on the diff policy.

Example
{
"testSuites": [
{
"name": "someTestSuite",
"tests": [
{
"name": "someTestName",
"dataPoints": [
{
"displayName": "",
"aggregationPolicy": "avg",
"diffPolicy": "sub",
"regressionPolicy": "lessIsBetter",
"regressionArgument": 0,
"name": "duration",
"values": [0.11,0.128,0.091,0.089,0.102,0.098,0.102,0.101,0.099,0.107],
"unit": "s"
}
]
}
]
}
]
}


### Importing from other formats¶

Importing from other formats is best handled by specifying a .js file (usually calcite.config.js) instead of a .json file to the calcite upload command. The cli expects the configuration file to export a function returning a calcite .json file. This way you can write or reuse any kind of importer.

Example
npx calcite upload ./calcite.config.js

calcite.config.js
// Require modules, create functions, anything you want here...
const getTestDatapoints = (testname) => {
// ...
}

// Export the function to be called by the cli
// It must return a valid calcite JSon object.
module.exports = async () => (
testSuites: [
{
name: 'fake testSuite',
tests: [
{
name: 'sometest',
datapoints: getTestDatapoints('sometest')
}, {
name: 'othertest',
datapoints: getTestDatapoints('othertest')
},
]
},
]
});


#### Available importers¶

The following importers are available:

##### C++¶

• nanobench (WIP, need to upstream)

##### Rust¶
• Official microbenchmarking library (WIP)