AI Stress Testing

Configuration of RIME AI Stress Testing is done through a JSON configuration file that you pass as an argument to the RIME CLI.

Template

{
  "run_name": "Run Name",                       (REQUIRED)
  "data_info": {...},                           (REQUIRED)
  "model_info": {...},
  "model_task": "Binary Classification",
  "model_profiling_info": {...},
  "data_profiling_info": {...},
  "tests_config_path": "path/to/tests/config",
  "tests_config": {...},
  "subset_profiling_config": {...},
  "random_seed": 0
}

Arguments

  • run_name: string, required

    The name under which your run will appear in the UI.

  • data_info: DataInfo, required

    Configuration for the datasources to use. For a reference on how to configure a datasource, see Data Configuration.

  • model_info: ModelInfo or null, default = null

    Configuration for the model to use. For a reference on how to configure a model, see Model Configuration. If you are not providing a model, this should be null.

  • model_task: string, default = "Binary Classification"

    Specification of the model task. Current accepted values are "Binary   Classification", "Multi-class Classification", "Regression", and "Ranking".

  • model_profiling_info: ModelProfilingInfo, default = ModelProfilingInfo()

    Specification of how to profile the model. For a guide on how to specify this info, see Model Profiling Configuration.
    NOTE: this argument is optional. If it is not specified default parameters will be used.

  • data_profiling_info: DataProfilingInfo, default = DataProfilingInfo()

    Specification of how to profile the data. For a guide on how to specify this info, see Data Profiling Configuration.
    NOTE: this argument is optional. If it is not specified default parameters will be used.

  • tests_config_path: string or null, default = null

    Path to a TestSuiteConfig JSON file. If both tests_config_path and test_config are null RIME will use the default TestSuiteConfig().
    NOTE: Only one of tests_config_path OR test_config should be specified.

  • tests_config: TestSuiteConfig or null, default = null

    Specification of which tests to run and what test parameters to use. If both tests_config_path and tests_config are null RIME will use the default TestSuiteConfig().
    NOTE: Only one of tests_config_path OR tests_config should be specified.

  • subset_profiling_config: SubsetProfilingConfig or null, default = SubsetProfilingConfig()

    Specification of how to slice and profile subsets of the data. For a guide on how to specify this, see Subset Profiling Configuration.

  • random_seed: int or null, default = null

    Seed all random operations in a test run. A seeded test run should return identical results if executed multiple times. If random_seed is not specified or null then there is no guarantee that two test runs executed from the same config will have identical results.