Dataset upload

calcite upload

To upload a dataset to calcite, we recommand using our Command line interface.


npx @siliceum/calcite-cli [command arguments] let you run the command line directly.

Uploading with calcite-cli

Each project has a unique token, which is used when uploading a dataset. It can be found in the project details page, under the description of the project.

A dataset contains your datapoints and some metadata. It can either be provided in our json format or through a configuration file used to import the data.


A fake_bench.json file is provided as part of the example, and this is what we will use for our first upload.

Setup your CI so that the CALCITE_TOKEN environment variables contains your project token, and use the following command to upload the ./fake_bench.json dataset:

calcite upload ./fake_bench.json


This should automatically detect the git branch, commit, etc (see the supported CI providers).
We however suggest overriding the default dataset config name.

You can also specify more information via additional arguments:

calcite upload [options] <filePath>

upload one dataset to a calcite server, by specifiying the config file, the gitRef and the commit id.

-u, --url <server url>

Set the target server url, for preview and enterprise users. The CALCITE_URL environment variable can also be used, otherwise it will be set to the public server.

-t, --token <token id>

Set the project token. By default the CALCITE_TOKEN environment variable is used.

-C, --commit <sha>

Specify the full git commit sha.

-R, --ref <gitref>

Specify the git ref.

-BC, --benchconfig <sha>

Specify the benchmark config name.

-BID, --buildId <id>

Specify the CI build number.

-PR, --pullRequestId <id>

Specify the pull request number.


Do not upload, instead print the content of the payload to the console.

-h, --help

display help for command

Dataset formats

To understand why the data is organized the following way, please have a look at the Concepts documentation.

calcite JSon schema

calcite uses its own json schema to describe a dataset. Ultimately, this is what will be sent to the server, but you would usually create this file using an importer.

A lot of importers are being added, and will be open-sourced. Do not hesitate to contact us if you created a new one!

The schema is available here.

calcite dataset

A benchmark dataset for use with calcite

Items must be of type: #/definitions/TestSuite

A test suite regroups similar tests, or one test that was ran with different inputs.

Items must be of type: #/definitions/Test

A test can report multiple datapoints, such as time, memory usage, etc. Tests using different input data are usually grouped in the same TestSuite

Items must be of type: #/definitions/Datapoint

The values that were sampled during the test.


The unit of this datapoint (s, ms, ns, Mb, Gb, MB, GB, W, …). This can be used to infer default values for other parameters of the datapoint.


This is the aggregation method used to summarize all values of a given commit and used for diffs between commits.

  • mean
  • min
  • max
  • median

This describes how to compute the value used for regression detections.

  • substraction = target - baseline

  • relativeDifference = 100 * (target - baseline)/baseline (percentage)

  • none = ignore this datapoint

  • substraction
  • relativeDifference
  • useTargetValue
  • none

Describes how fluctuations of values must be interpreted for regressions. absoluteDifferenceThreshold means that changes in any direction will be interpreted as regressions. equal means that any value different than regressionArgument will be treated as a regression.

  • lessIsBetter
  • moreIsBetter
  • absoluteDifferenceThreshold
  • equal

This is the argument used in conjonction with the diff and regression policies. This could be an absolute value or percentage, based on the diff policy.

  "testSuites": [
      "name": "someTestSuite",
      "tests": [
          "name": "someTestName",
          "dataPoints": [
              "displayName": "",
              "aggregationPolicy": "avg",
              "diffPolicy": "sub",
              "regressionPolicy": "lessIsBetter",
              "regressionArgument": 0,
              "name": "duration",
              "values": [0.11,0.128,0.091,0.089,0.102,0.098,0.102,0.101,0.099,0.107],
              "unit": "s"

Importing from other formats

Importing from other formats is best handled by specifying a .js file (usually calcite.config.js) instead of a .json file to the calcite upload command. The cli expects the configuration file to export a function returning a calcite .json file. This way you can write or reuse any kind of importer.

npx calcite upload ./calcite.config.js
// Require modules, create functions, anything you want here...
const getTestDatapoints = (testname) => {
    // ...

// Export the function to be called by the cli
// It must return a valid calcite JSon object.
module.exports = async () => (
    testSuites: [
            name: 'fake testSuite',
            tests: [
                    name: 'sometest',
                    datapoints: getTestDatapoints('sometest')
                }, {
                    name: 'othertest',
                    datapoints: getTestDatapoints('othertest')

Available importers

The following importers are available:

  • Google benchmark (WIP)

  • nanobench (WIP, need to upstream)

  • Official microbenchmarking library (WIP)

Load testing
  • JMeter (WIP)

  • Gatling (WIP)

  • NeoLoad (WIP)

  • Lighthouse CI (WIP)

Importer user example

@TODO@ How to use importers ? Via npm ? Copy the file directly ?