Custom Metrics

In order to use the custom metrics functionality, you first need to create a python file that contains the function used to calculate the custom metric. This function should be named custom_metric_func, and the interface this function is expected to expose is below:

from typing import Optional

import numpy as np
import pandas as pd


def custom_metric_func(
    df: Optional[pd.DataFrame],
    labels: Optional[pd.Series],
    preds: Optional[np.ndarray],
    preds_index: Optional[pd.Index],
    query_ids: Optional[pd.Series],
) -> Optional[float]:
    """Return a custom metric.

    Args:
        df: Optional pd.DataFrame of data, will be the full data.
        labels: Optional pd.Series of labels, if exists will have the same length
            as df.
        preds: Optional np.array of predictions, if exists may be a subset of the
            full df.
        preds_index: Optional pd.Index of predictions, if exists will have the same
            length as `preds`. Can be used to align the predictions with rows in the
            df.
        query_ids: Optional pd.Series of query ids, if exists will have the same
            length as `df`.

    Returns:
        an optional float corresponding to the metric to be tracked.
    """

After defining this function, you then need to specify it in the CLIConfig. The place to specify it is in the model_profiling_info section. Specifically, it should look like:

  "model_profiling_info": {
    "custom_metrics": {"NAME_OF_METRIC": "path/to/file.py"}
  },

Replace NAME_OF_METRIC with the name of the metric (this is how it will show up in the UI). Replace path/to/file.py with the path to the python file you created in the above step.