RIME Tests

Main stress tests exposed by RIME library.

class rime.tabular.tests.CategoricalDriftTest(col_name: str, max_sample_size: Optional[int] = None, num_values_for_graph: int = 5, **kwargs: Any)

Run categorical drift comparison test.

Test for comparing the distribution of a categorical feature across datasets.

Parameters:: col_name (string) – Name of the feature column to test.

abstract property data_test_name: str: Return the verbose data test name.

property historic_metric_explanation_suffix: Optional[str]: Return the suffix for the historic metric explanation string.

abstract get_data_impact(ref_col_counts: Dict[Any, int], eval_col_counts: Dict[Any, int]) → dict: Return the data impact vals for this test.

get_drift_statistic(run_helper_dict: dict) → float: Get the drift statistic from the helper dictionary.

property data_drift_template: str: Template for returning info about drift statistic.

abstract property drift_statistic_api_name: str: Return the drift statistic api name for this test. Should be snake_case.

abstract property drift_statistic_display_name: str: Return the drift statistic display name for this test.

get_details_components(impact_val: Optional[Union[DataError, float]], impact_metric_name: Optional[MetricName], is_drift: bool, data_test: str, data_explanation: str) → List[Detail]: Get the details components.

get_drift_stat_detail(is_drift: bool, data_test: str, data_explanation: str) → Detail

Get detail about drift statistic.

Generally shared by all DataDrift tests.

get_table_column_info(run_helper_dict: dict, extra_info: dict) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.Chi2LabelDriftTest(p_value_thresholds: Tuple[float, float, float] = (0.01, 0.05, 0.1), max_sample_size: Optional[int] = None, num_values_for_graph: int = 5)

Run Chi-squared Label Drift test.

get_test_values(ref_col_counts: Dict[Any, int], eval_col_counts: Dict[Any, int]) → dict: Obtain test values.

get_api_output(test_vals: dict) → dict: Get api output.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

property stat_key: str: Get the key which contains the test statistic.

class rime.tabular.tests.MulticlassChi2PredictedLabelDriftTest(p_value_thresholds: Tuple[float, float, float] = (0.01, 0.05, 0.1), max_sample_size: Optional[int] = None, num_values_for_graph: int = 5)

Compute drift in predicted labels for multiclass classification using chi2.

get_api_output(test_vals: dict) → dict: Get api output.

get_test_values(ref_col_counts: Dict[Any, int], eval_col_counts: Dict[Any, int]) → dict: Obtain test values.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

property stat_key: str: Get the key which contains the test statistic.

class rime.tabular.tests.PSILabelDriftTest(psi_thresholds: Tuple[float, float, float] = (0.2, 0.4, 0.6), max_sample_size: Optional[int] = None, num_values_for_graph: int = 5)

Run PSI Label Drift test.

get_test_values(ref_col_counts: Dict[Any, int], eval_col_counts: Dict[Any, int]) → dict: Obtain test values.

get_api_output(test_vals: dict) → dict: Get api output.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

property stat_key: str: Get the key which contains the test statistic.

class rime.tabular.tests.MulticlassPSIPredictedLabelDriftTest(psi_thresholds: Tuple[float, float, float] = (0.2, 0.4, 0.6), max_sample_size: Optional[int] = None, num_values_for_graph: int = 5)

Compute drift in predicted labels for multiclass classification using PSI.

get_api_output(test_vals: dict) → dict: Get api output.

get_test_values(ref_col_counts: Dict[Any, int], eval_col_counts: Dict[Any, int]) → dict: Obtain test values.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

property stat_key: str: Get the key which contains the test statistic.

class rime.tabular.tests.CorrelationDriftTest(col_names: List[str], severity_thresholds: Tuple[float, float, float] = (0.1, 0.2, 0.3), p_value_threshold: Optional[float] = None)

Run correlation drift test.

Test for the correlation drift between numeric features in the dataset.

Parameters:

col_name (string) – Name of the feature column to test.
severity_thresholds (Tuple[float]) – Correlation difference threshold.
p_value_threshold (float) – P-value threshold.

property data_drift_template: str: Template for returning info about drift statistic.

get_details_components(impact_val: Optional[Union[DataError, float]], impact_metric_name: Optional[MetricName], is_drift: bool, data_test: str, data_explanation: str) → List[Detail]: Get the details components.

get_drift_stat_detail(is_drift: bool, data_test: str, data_explanation: str) → Detail

Get detail about drift statistic.

Generally shared by all DataDrift tests.

get_table_column_info(key_details: str, severity_level: ImportanceLevel) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.MutualInformationFeatureDriftTest(col_names: List[str], severity_thresholds: Tuple[float, float, float] = (0.1, 0.2, 0.3), **kwargs: Any)

Run mutual information drift test over features.

Test for the mutual information drift between categorical features in the dataset.

Parameters:

col_names (string) – Name(s) of the feature columns to test.
severity_thresholds (Tuple[float]) – Mutual information difference threshold.

property data_drift_template: str: Template for returning info about drift statistic.

get_details_components(impact_val: Optional[Union[DataError, float]], impact_metric_name: Optional[MetricName], is_drift: bool, data_test: str, data_explanation: str) → List[Detail]: Get the details components.

get_drift_stat_detail(is_drift: bool, data_test: str, data_explanation: str) → Detail

Get detail about drift statistic.

Generally shared by all DataDrift tests.

get_table_column_info(key_details: str, severity_level: ImportanceLevel) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.MutualInformationLabelDriftTest(col_name: str, label_name: str, severity_thresholds: Tuple[float, float, float] = (0.1, 0.2, 0.3), **kwargs: Any)

Run mutual information drift test.

Test for the mutual information drift between features and labels in the dataset.

Parameters:

col_name (string) – Name of the feature column to test.
label_name (string) – Name of the label column to test.
severity_thresholds (Tuple[float]) – Mutual information difference threshold.

property data_drift_template: str: Template for returning info about drift statistic.

get_details_components(impact_val: Optional[Union[DataError, float]], impact_metric_name: Optional[MetricName], is_drift: bool, data_test: str, data_explanation: str) → List[Detail]: Get the details components.

get_drift_stat_detail(is_drift: bool, data_test: str, data_explanation: str) → Detail

Get detail about drift statistic.

Generally shared by all DataDrift tests.

get_table_column_info(key_details: str, severity_level: ImportanceLevel) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.NullProportionDriftTest(col_name: str, p_value_threshold: float = 0.05, **kwargs: Any)

Run null proportion drift test.

Test for drift in the proportion of values within a feature that are null relative to the reference set.

Parameters:

col_name (string) – Name of the feature column to test.
p_value_threshold (float) – P-value threshold.

property drift_statistic_display_name: str: Return the drift statistic name for this test.

property drift_statistic_api_name: str: Return the drift statistic name for this test.

get_drift_statistic(run_helper_dict: dict) → float: Get the drift statistic from the helper dictionary.

property data_drift_template: str: Template for returning info about drift statistic.

get_details_components(impact_val: Optional[Union[DataError, float]], impact_metric_name: Optional[MetricName], is_drift: bool, data_test: str, data_explanation: str) → List[Detail]: Get the details components.

get_drift_stat_detail(is_drift: bool, data_test: str, data_explanation: str) → Detail

Get detail about drift statistic.

Generally shared by all DataDrift tests.

get_table_column_info(run_helper_dict: dict, extra_info: dict) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.JSDivergenceTest(col_name: str, distance_threshold: float = 0.05, num_bins: int = 100, **kwargs: Any)

Run JS-Divergence test.

Test for numeric feature drift from the reference set, using JS-Divergence to measure drift.

Parameters:

col_name (string) – Name of the feature column to test.
distance_threshold (float) – Distance threshold for divergence metric.
num_bins (int) – Number of bins to compute divergence metric.

property drift_statistic_api_name: str: Return a name associated with the distance metric used.

classmethod distance_display_name() → str: Return a name associated with the distance metric used (for frontend).

property data_drift_template: str: Template for returning info about drift statistic.

property drift_statistic_display_name: str: Return the drift statistic name for this test.

get_details_components(impact_val: Optional[Union[DataError, float]], impact_metric_name: Optional[MetricName], is_drift: bool, data_test: str, data_explanation: str) → List[Detail]: Get the details components.

get_drift_stat_detail(is_drift: bool, data_test: str, data_explanation: str) → Detail

Get detail about drift statistic.

Generally shared by all DataDrift tests.

get_drift_statistic(run_helper_dict: dict) → float: Get the drift statistic from the helper dictionary.

get_table_column_info(run_helper_dict: dict, extra_info: dict) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.KLDivergenceTest(col_name: str, distance_threshold: float = 0.1, num_bins: int = 100, **kwargs: Any)

Run KL-Divergence test.

Test for numeric feature drift from the reference set, using KL-Divergence to measure drift.

Parameters:

col_name (string) – Name of the feature column to test.
distance_threshold (float) – Distance threshold for divergence metric.
num_bins (int) – Number of bins to compute divergence metric.

property drift_statistic_api_name: str: Return a name associated with the distance metric used.

classmethod distance_display_name() → str: Return a name associated with the distance metric used (for frontend).

property data_drift_template: str: Template for returning info about drift statistic.

property drift_statistic_display_name: str: Return the drift statistic name for this test.

get_details_components(impact_val: Optional[Union[DataError, float]], impact_metric_name: Optional[MetricName], is_drift: bool, data_test: str, data_explanation: str) → List[Detail]: Get the details components.

get_drift_stat_detail(is_drift: bool, data_test: str, data_explanation: str) → Detail

Get detail about drift statistic.

Generally shared by all DataDrift tests.

get_drift_statistic(run_helper_dict: dict) → float: Get the drift statistic from the helper dictionary.

get_table_column_info(run_helper_dict: dict, extra_info: dict) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.NumericDistributionEqualityTest(col_name: str, p_value_threshold: float = 0.01, **kwargs: Any)

Run numeric distribution equality test.

Test for numeric feature drift from the reference set, using the Kolmogorov–Smirnov (K-S) test to measure drift.

Parameters:

col_name (string) – Name of the feature column to test.
p_value_threshold (float) – P-value threshold.

property drift_statistic_display_name: str: Return the drift statistic name for this test.

property drift_statistic_api_name: str: Return the drift statistic name for this test.

get_drift_statistic(run_helper_dict: dict) → float: Get the drift statistic from the helper dictionary.

property data_drift_template: str: Template for returning info about drift statistic.

get_details_components(impact_val: Optional[Union[DataError, float]], impact_metric_name: Optional[MetricName], is_drift: bool, data_test: str, data_explanation: str) → List[Detail]: Get the details components.

get_drift_stat_detail(is_drift: bool, data_test: str, data_explanation: str) → Detail

Get detail about drift statistic.

Generally shared by all DataDrift tests.

get_table_column_info(run_helper_dict: dict, extra_info: dict) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.PredictionDriftTest

Run Prediction Drift Test.

Test for measuring prediction drift.

abstract compute_drift_metrics(ref_col: NumericColumn, test_col: NumericColumn) → dict: Compute drift metrics associated with this test.

abstract property drift_statistic: NumericDriftStatistic: Get the drift statistic associated with this test.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.RowNullProportionChi2Test(p_value_threshold: float = 0.05, num_values_for_graph: int = 5, drift_scaling_factor: float = 0.005, severity_level_thresholds: Tuple[float, float, float] = (0.01, 0.02, 0.1))

Run chi2 row-wise null proportion comparison test.

Test for drift in the proportion of features within a row that are null.

Parameters:: p_value_threshold (float) – P-value threshold.

property data_drift_template: str: Template for returning info about drift statistic.

get_details_components(impact_val: Optional[Union[DataError, float]], impact_metric_name: Optional[MetricName], is_drift: bool, data_test: str, data_explanation: str) → List[Detail]: Get the details components.

get_drift_stat_detail(is_drift: bool, data_test: str, data_explanation: str) → Detail

Get detail about drift statistic.

Generally shared by all DataDrift tests.

static get_model_impact(ref_pct: Dict[str, float], test_pct: Dict[str, float], ref_subsets_info: SubsetsInfo, impact_metric_name: MetricName) → Optional[float]: Get test impact (magnitude, p-value).

static get_table_column_info(perf_impact: Optional[float], key_details: str, severity: ImportanceLevel) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.RowNullProportionPSITest(psi_threshold: float = 0.2, num_values_for_graph: int = 5, drift_scaling_factor: float = 0.005, severity_level_thresholds: Tuple[float, float, float] = (0.01, 0.02, 0.1))

Run PSI row-wise null proportion comparison test.

Test for drift in the proportion of features within a row that are null.

property data_drift_template: str: Template for returning info about drift statistic.

get_details_components(impact_val: Optional[Union[DataError, float]], impact_metric_name: Optional[MetricName], is_drift: bool, data_test: str, data_explanation: str) → List[Detail]: Get the details components.

get_drift_stat_detail(is_drift: bool, data_test: str, data_explanation: str) → Detail

Get detail about drift statistic.

Generally shared by all DataDrift tests.

static get_model_impact(ref_pct: Dict[str, float], test_pct: Dict[str, float], ref_subsets_info: SubsetsInfo, impact_metric_name: MetricName) → Optional[float]: Get test impact (magnitude, p-value).

static get_table_column_info(perf_impact: Optional[float], key_details: str, severity: ImportanceLevel) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.CapitalizationTest(col_name: str, **kwargs: Any)

Run capitalization test.

Test how the model responds to data containing values that are capitalized differently from those observed in the reference set.

Parameters:: col_name (string) – Name of the feature column to test.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get impute helper.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

property firewall_identifiers: Dict: Return identifiers for firewall id.

property firewall_id: str: Return a unique reproducible ID.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.CapitalizationTransformationTest(col_name: str, **kwargs: Any)

Run capitalization transformation test.

Test how the model responds to transformations of data that are capitalized differently.

Parameters:: col_name (string) – Name of the feature column to test.

property rowwise_test_id: str: Return a unique reproducible ID.

get_table_info(severity: ImportanceLevel, impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.NonParametricOutliersTest(col_name: str, min_normal_prop: float = 0.99, baseline_quantile: float = 0.1, perturb_multiplier: float = 1.0, **kwargs: Any)

Run Outliers tests.

Test whether features in the data have distributional outliers.

Parameters:

col_name (string) – Name of the feature column to test.
min_normal_prop (float) – Minimum proportion of the data that shouldn’t be classified as an outlier.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get impute helper.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

property firewall_identifiers: Dict: Return identifiers for firewall id.

property firewall_id: str: Return a unique reproducible ID.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.NumericOutlierTransformationTest(col_name: str, min_normal_prop: float = 0.99, baseline_quantile: float = 0.1, perturb_multiplier: float = 1.0, **kwargs: Any)

Transformation test that tests numeric outliers.

property rowwise_test_id: str: Return a unique reproducible ID.

get_table_info(severity: ImportanceLevel, impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.DuplicateRowsTest(col_names: Optional[List[str]] = None, severity_thresholds: Tuple[float, float] = (0.01, 0.05))

Run duplicate rows test.

Check that no two rows in the data are duplicates of each other.

Parameters:

col_names (List[str], optional) – Set of column names. If None, runs over all columns.
severity_thresholds (Tuple[float, float]) – Thresholds for different levels of severity.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.FeatureTypeTest(col_name: str, col_type: Type[Column], **kwargs: Any)

Run feature type test.

Check that the specified column contains all values of a given type.

Parameters:

col_name (string) – Name of the feature column to test.
col_type (rime.tabular.profiler.columns.Column) – One of (“BoolCategoricalColumn”, “DomainColumn”, “EmailColumn”, “FloatColumn”, “IntegerColumn”, “NumericCategoricalColumn”, “StringCategoricalColumn”, “UrlColumn”)

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

property firewall_identifiers: Dict: Return identifiers for firewall id.

property firewall_id: str: Return a unique reproducible ID.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get a context dictionary containing impute parameters.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.FeatureTypeTransformationTest(col_name: str, col_type: Type[Column], **kwargs: Any)

Transformation test for feature types.

property rowwise_test_id: str: Return a unique reproducible ID.

get_table_info(severity: ImportanceLevel, impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.FeatureLeakageTest(col_name: str, label_name: str, severity_thresholds: Tuple[float, float, float], min_mutual_info_threshold: float)

Run feature leakage test.

Check if a feature in the reference set has significantly higher mutual information with the label.

Parameters:

col_name (string) – Name of feature column to test
label_name (string) – Name of label column to test
severity_thresholds (Tuple[float, float, float]) – Mutual information difference threshold
min_mutual_info_threshold (float) – threshold that mutual information of ref data must reach before this test would flag it

get_table_column_info(metric_container: PairwiseMetricContainer, severity_level: ImportanceLevel) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.InconsistenciesTest(col_names: List[str], freq_ratio_threshold: float = 0.02, max_unique_pairs_for_firewall: Optional[int] = 15, **kwargs: Any)

Run inconsistencies test.

Test how the model responds to data points with pairs of feature values that are inconsistent with each other.

Parameters:

col_names (List[string]) – List of feature columns that can be used for testing.
freq_ratio_threshold (float) – Ratio of joint frequency to product of marginal frequencies, below which the test raises a warning.

property firewall_identifiers: Dict: Return identifiers for firewall id.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

property firewall_id: str: Return a unique reproducible ID.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get a context dictionary containing impute parameters.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.NullCheckComparisonTest(col_name: str, **kwargs: Any)

Run null check test.

Test how the model responds to data that has nulls in features observed to not have nulls in the reference set.

Parameters:: col_name (string) – Name of the feature column to test.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

property firewall_identifiers: Dict: Return identifiers for firewall id.

property firewall_id: str: Return a unique reproducible ID.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get a context dictionary containing impute parameters.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.NullTransformationTest(col_name: str, **kwargs: Any)

Transformation test that adds nulls.

property rowwise_test_id: str: Return a unique reproducible ID.

get_table_info(severity: ImportanceLevel, impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.OutOfRangeTest(col_name: str, dir_str: str, std_factor: float, **kwargs: Any)

Run out of range test.

Test how the model responds to data whose feature values are outside of the range seen in the reference set.

Parameters:

col_name (string) – Name of the feature column to test.
dir_str (string) – One of (“above”, “below”).
std_factor (float) – Number of std. deviations below/above the feature range with which to test your model.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

property id: str: Return a unique reproducible ID.

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.OutOfRangeTransformationTest(col_name: str, dir_str: str, std_factor: float, **kwargs: Any)

Transformation test for out of range values.

Parameters:

col_name (string) – Name of the feature column to test.
dir_str (string) – One of (“above”, “below”).
std_factor (float) – Number of std. deviations below/above the feature range with which to test your model.

get_table_info(severity: ImportanceLevel, impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

property rowwise_test_id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.OverallMetricsTest(metric_cls: Type[Metric], severity_level_thresholds: Tuple[float, float, float] = (0.03, 0.08, 0.13))

Run overall metrics test.

Test whether overall model performance metrics have changed between reference and evaluation sets.

Parameters:

metric_cls (rime.tabular.metric.Metric) – Metric to measure (“ACCURACY_METRIC”, “AUC_METRIC”, “BALANCED_ACCURACY_METRIC”, “F1_METRIC”, “PRECISION_METRIC”, “PRED_VARIANCE_NEG_METRIC”, “PRED_VARIANCE_POS_METRIC”, “RECALL_METRIC”)
severity_level_thresholds (Tuple[float, float, float]) – Thresholds for performance change.

get_metric_gap_and_stub(ref_metric_val: Optional[float], test_metric_val: Optional[float]) → Tuple[Optional[float], str]: Get the metric gap and explanation stub.

get_explanation_detail(severity_level: ImportanceLevel, metric_stub: str, metric_gap: float) → Tuple[Status, Detail]: Get explanation detail.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.RareCategoriesTest(col_name: str, min_num_occurrences: int = 0, min_pct_occurrences: float = 0, min_ratio_rel_uniform: float = 0.005, **kwargs: Any)

Run rare categories test.

Test whether a categorical feature has rare values.

Parameters:

col_name (string) – Name of the feature column to test.
min_num_occurrences (int) – Minimum number of occurrences for value to not be rare.
min_pct_occurrences (float) – Minimum percent of occurrences for value to not be rare.
min_ratio_rel_uniform (float) – Minimum ratio of occurrences of a uniformly distributed value for it to not be rare.

issue_str(run_helper_dict: dict) → str: Return a description of failing rows.

property firewall_identifiers: Dict: Return identifiers for firewall id.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

property firewall_id: str: Return a unique reproducible ID.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get a context dictionary containing impute parameters.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.RequiredFeaturesTest(required_feats: Optional[List[str]] = None, allowed_feats: Optional[List[str]] = None, ordered: bool = False, required_only: bool = False)

Run required features test.

Check if required features are present in the dataset.

Parameters:

required_feats (list) – List of required feature names.
allowed_feats (list) – List of allowed feature names.
ordered (bool) – Whether columns needed to be in same order as allowed feats or not.
required_only (bool) – Whether to only allow required features.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get a context dictionary containing impute parameters.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

property firewall_identifiers: Dict: Return identifiers for firewall id.

property firewall_id: str: Return a unique reproducible ID.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.MultiFeatSensitivityTest(col_names: List[str], l0_constraint: int = 3, linf_constraint: float = 0.01, sample_size: int = 10, use_tqdm: bool = True, label_range: Tuple[float, float] = (0.0, 1.0), **kwargs: Any)

Run bounded multi-feature changes test.

Check that simultaneously manipulating multiple features by a small amount does not significantly change the model’s predictions.

Parameters:

col_names (List[string]) – List of feature columns that can be used for testing. If None, use all columns.
l0_constraint (int) – Maximum number of feature columns that can be changed per row.
linf_constraint (float) – Percentage of the maximum range of the feature, which serves as the upper bound of the scale of the manipulations.
sample_size (int) – Number of rows within data to run the test over.
label_range (Tuple[float, float]) – The range of the labels (assumed to be [0, 1] by default).

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.MultiFeatVulnerabilityTest(col_names: List[str], l0_constraint: int = 3, sample_size: int = 10, search_count: int = 10, use_tqdm: bool = True, label_range: Tuple[float, float] = (0.0, 1.0), **kwargs: Any)

Run multi-feature changes test.

Check that simultaneously manipulating multiple features does not significantly change the model’s predictions.

Parameters:

col_names (List[string]) – List of feature columns that can be used for testing. If None, use all columns.
l0_constraint (int) – Maximum number of feature columns that can be changed per row.
sample_size (int) – Number of rows within data to run the test over.
search_count (int) – Number of values to consider in each column when making manipulations.
label_range (Tuple[float, float]) – The range of the labels (assumed to be [0, 1] by default).

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.SensitivityTest(col_name: str, linf_constraint: float = 0.01, sample_size: int = 10, label_range: Tuple[float, float] = (0.0, 1.0), **kwargs: Any)

Run bounded single feature changes test.

Check that manipulating a single numeric feature by a small amount does not significantly change the model’s predictions.

Parameters:

col_name (string) – Name of the feature column to test.
linf_constraint (float) – Percentage of the maximum range of the feature, which serves as the upper bound of the scale of the manipulations.
sample_size (int) – Number of rows within data to run the test over.
label_range (Tuple[float, float]) – The range of the labels (assumed to be [0, 1] by default).

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.VulnerabilityTest(col_name: str, sample_size: int = 10, search_count: int = 10, label_range: Tuple[float, float] = (0.0, 1.0), **kwargs: Any)

Run single feature changes test.

Evaluate manipulating a single feature does not significantly change the model’s predictions.

Parameters:

col_name (string) – Name of the feature column to test.
sample_size (int) – Number of rows within data to run the test over.
search_count (int) – Number of values to consider in each column when making manipulations.
label_range (Tuple[float, float]) – The range of the labels (assumed to be [0, 1] by default).

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.UnseenCategoricalTest(col_name: str, **kwargs: Any)

Run unseen categorical test.

Test how the model responds to data that contains categorical values that were not seen in the reference set.

Parameters:: col_name (string) – Name of the feature column to test.

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

property firewall_identifiers: Dict: Return identifiers for firewall id.

property firewall_id: str: Return a unique reproducible ID.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get a context dictionary containing impute parameters.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.UnseenCategoricalTransformationTest(col_name: str, **kwargs: Any)

Class that tests unseen categorical transformations.

property rowwise_test_id: str: Return a unique reproducible ID.

get_table_info(severity: ImportanceLevel, impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.UnseenDomainTest(col_name: str, **kwargs: Any)

Run unseen domain test.

Test how the model responds to data that contains domain values that were not seen in the reference set.

Parameters:: col_name (string) – Name of the feature column to test.

property firewall_id: str: Return a unique reproducible ID.

property firewall_identifiers: Dict: Return identifiers for firewall id.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get a context dictionary containing impute parameters.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

class rime.tabular.tests.UnseenEmailTest(col_name: str, **kwargs: Any)

Run unseen email test.

Test how the model responds to data that contains email values that were not seen in the reference set.

Parameters:: col (string) – Name of the feature column to test.

property firewall_id: str: Return a unique reproducible ID.

property firewall_identifiers: Dict: Return identifiers for firewall id.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get a context dictionary containing impute parameters.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

class rime.tabular.tests.UnseenURLTest(col_name: str, **kwargs: Any)

Run unseen URL test.

Test how the model responds to data that contains URL values that were not seen in the reference set.

Parameters:: col_name (string) – Name of the feature column to test.

property firewall_id: str: Return a unique reproducible ID.

property firewall_identifiers: Dict: Return identifiers for firewall id.

get_context_helper(run_container: TabularRunContainer) → Dict: Get a context dictionary containing test parameters.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_impute_helper(run_container: TabularRunContainer) → Dict[ImputationStrategy, Any]: Get a context dictionary containing impute parameters.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

impute(datapoint: dict, impute_strategy: ImputationStrategy, impute_helper: Dict) → Dict[str, Tuple[Any, Any]]: Impute datapoint.

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

validate_single(datapoint: dict, context_helper: Dict) → Tuple[bool, Dict]: Validate single and return text details.

class rime.tabular.tests.UnseenDomainTransformationTest(col_name: str, **kwargs: Any)

Run unseen domain transformation test.

Test how the model responds to data that contains domain values that were not seen in the reference set.

Parameters:: col_name (string) – Name of the feature column to test.

get_table_info(severity: ImportanceLevel, impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

property rowwise_test_id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.UnseenEmailTransformationTest(col_name: str, **kwargs: Any)

Run unseen email transformation test.

Test how the model responds to data that contains email values that were not seen in the reference set.

Parameters:: col (string) – Name of the feature column to test.

get_table_info(severity: ImportanceLevel, impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

property rowwise_test_id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.UnseenURLTransformationTest(col_name: str, **kwargs: Any)

Run unseen URL transformation test.

Test how the model responds to data that contains URL values that were not seen in the reference set.

Parameters:: col_name (string) – Name of the feature column to test.

get_table_info(severity: ImportanceLevel, impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

property rowwise_test_id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.FeatureSubsetTest(col_name: str, metric_cls: Type[Metric], performance_thresholds: Tuple[float, float, float], min_sample_size: int = 20)

Run feature-based subset performance test.

Test whether a given metric differs significantly across different slices of the evaluation data.

Parameters:

col_name (string) – Name of the feature column to test.
metric (rime.core.metric) – metric class as defined in rime.core.
performance_thresholds (Tuple[float, float, float]) – Thresholds for performance change.
min_sample_size (int) – The minimum number of relevant samples a subset must have in order to be tested.

property subset_mode: SubsetMode: Tell whether the test indexes subsets along a feature axis.

get_table_column_info(worst_subset: Optional[SubsetInfo], severity_level: ImportanceLevel) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.LabelSubsetTest(metric_cls: Type[Metric], performance_change_thresholds: Tuple[float, float, float], min_sample_size: int = 20)

Run feature-based subset performance test.

Test whether a given metric differs significantly across different slices of the evaluation data.

Parameters:

col_name (string) – Name of the feature column to test.
metric (rime.tabular.metric.Metric) – Metric to measure (“ACCURACY_METRIC”, “AUC_METRIC”, “BALANCED_ACCURACY_METRIC”, “F1_METRIC”, “PRECISION_METRIC”, “PRED_VARIANCE_NEG_METRIC”, “PRED_VARIANCE_POS_METRIC”, “RECALL_METRIC”)
performance_change_thresholds (Tuple[float, float, float]) – Thresholds for performance change.

property subset_mode: SubsetMode: Tell whether the test indexes subsets along a feature axis.

get_table_column_info(worst_subset: Optional[SubsetInfo], severity_level: ImportanceLevel) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.NullSubsetTest(metric_cls: Type[Metric], performance_change_thresholds: Tuple[float, float, float], min_sample_size: int = 20)

Run null subset performance test.

Test whether a given metric differs significantly across different slices (null percentages) of the evaluation data.

Parameters:

metric (rime.tabular.metric.Metric) – Metric to measure (“ACCURACY_METRIC”, “AUC_METRIC”, “BALANCED_ACCURACY_METRIC”, “F1_METRIC”, “PRECISION_METRIC”, “PRED_VARIANCE_NEG_METRIC”, “PRED_VARIANCE_POS_METRIC”, “RECALL_METRIC”)
performance_change_thresholds (Tuple[float, float, float]) – Thresholds for performance change.

property subset_mode: SubsetMode: Tell whether the test indexes subsets along a feature axis.

get_table_column_info(worst_subset: Optional[SubsetInfo], severity_level: ImportanceLevel) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.EmptyStringTest(col_name: str, **kwargs: Any)

Run Empty String test.

DataTest for flagging when empty strings and null values are present.

Parameters:: col_name (string) – Name of the feature column to test.

get_data_detail(run_helper_dict: dict, num_failing_rows: int) → Detail: Get data impact details.

get_table_info(severity: ImportanceLevel, num_failing_rows: int, pct_failing_rows: float, observed_impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

issue_str(run_helper_dict: dict) → str: Return a description of the issue.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.EmptyStringSubstitutionTest(col_name: str, **kwargs: Any)

Run Empty String Substitution test.

Simulates how the model reacts when empty strings are substituted into data.

Parameters:: col_name (string) – Name of the feature column to test.

get_table_info(severity: ImportanceLevel, impact: ModelImpactInfo) → dict: Get table info.

property id: str: Return a unique reproducible ID.

property rowwise_test_id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.RegressionLabelDriftComparisonTest(p_value_thresh: float = 0.05, ks_stat_thresholds: Tuple[float, float, float] = (0.1, 0.33, 0.67))

Run Regression Label Drift Comparison test.

Test for comparing the distribution distance of a feature across datasets.

Parameters:

p_value_thresh (float) – Threshold for K-S Test p-value.
ks_stat_threshold (Tuple[float, float, float]) – K-S Stat thresholds.

get_text_details(ks_stat: float, p_value: float, severity_level: ImportanceLevel) → List[Detail]: Compile the TestDetails.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.CalibrationComparisonTest(severity_level_thresholds: Tuple[float, float, float] = (0.02, 0.06, 0.1))

Run calibration comparison test.

Test that the calibration curve has not changed significantly between the reference and evaluation sets.

Parameters:: severity_thresholds (Tuple[float]) – MSE Difference threshold.

get_table_column_info(severity_level: ImportanceLevel, mse_diff: Optional[float] = None) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

class rime.tabular.tests.SelectionRateTest(col_name: str, metric_cls: Type[Metric], performance_thresholds: Tuple[float, float, float] = (0.8, 0.7, 0.6), min_sample_size: int = 20)

Selection Rate test.

get_table_column_info(worst_subset: Optional[SubsetInfo], severity_level: ImportanceLevel) → dict: Get table column info to return.

property id: str: Return a unique reproducible ID.

run_notebook(run_container: TabularRunContainer, show_html: bool = True) → dict: Run test and return notebook friendly output.

property subset_mode: SubsetMode: Tell whether the test indexes subsets along a feature axis.