Configuring Test Categories

This section of the configuration specifies which categories of tests Robust Intelligence runs in Stress Testing. Summary Tests discusses what each of these categories tests for.

  • categories: List[str], default = ["TEST_CATEGORY_TYPE_ADVERSARIAL", "TEST_CATEGORY_TYPE_SUBSET_PERFORMANCE", "TEST_CATEGORY_TYPE_TRANSFORMATIONS"]

    Test categories to run. Options include TEST_CATEGORY_TYPE_ABNORMAL_INPUTS, TEST_CATEGORY_TYPE_ADVERSARIAL, TEST_CATEGORY_TYPE_BIAS_AND_FAIRNESS, TEST_CATEGORY_TYPE_DATA_CLEANLINESS, TEST_CATEGORY_TYPE_DATA POISONING_DETECTION, TEST_CATEGORY_TYPE_DRIFT, TEST_CATEGORY_TYPE_MODEL_PERFORMANCE, TEST_CATEGORY_TYPE_SUBSET_PERFORMANCE, TEST_CATEGORY_TYPE_SUBSET_PERFORMANCE_DEGRADATION, and TEST_CATEGORY_TYPE_TRANSFORMATIONS. If no categories are specified, the default categories listed above are run. The default categories for Stress Testing are TEST_CATEGORY_TYPE_ADVERSARIAL, TEST_CATEGORY_TYPE_SUBSET_PERFORMANCE, and TEST_CATEGORY_TYPE_TRANSFORMATIONS,

{
    #...,
    "categories": [
        "TEST_CATEGORY_TYPE_MODEL_PERFORMANCE",
        "TEST_CATEGORY_TYPE_DATA_CLEANLINESS",
    ]
}