Metrics

This module provides a collection of metrics for evaluating model performance. Each metric is a subclass of BaseMetric and can be used to compute specific evaluation scores.

BaseMetric Class

class lis2.metrics.base_metric.BaseMetric(*args, **kwargs)[source]

Bases: ABC, TypedConfigurable

BaseMetric for defining custom metrics.

An abstract base class for defining custom metrics. This class provides a structure for implementing metrics that can be updated with predictions and targets, and then computed to yield a result.

Configuration:
  • name (str): The name of the metric.

Example Configuration:
config = {
    "name": "example_metric"
}
compute() → float[source]

Computes the metric result.

Returns:

The computed metric result.

Return type:

float

Raises:

ValueError – If no data is available to compute the metric.

reset() → None[source]

Resets the metric result and averaging coefficient.

abstractmethod update(pred: ndarray, target: ndarray, idx: ndarray, *args: Any, **kwargs: Any) → None[source]

Updates the metric with current predictions and targets.

Parameters:
  • pred (np.ndarray) – Model predictions.

  • target (np.ndarray) – Ground truth labels.

  • idx (np.ndarray) – Mask or weighting for valid examples.

  • *args – Variable length argument list.

  • **kwargs – Arbitrary keyword arguments.

The BaseMetric class is an abstract base class for all metrics. It provides methods for updating, computing, and resetting metric results.

Example:

class CustomMetric(BaseMetric):
    def __init__(self):
        super().__init__()
        self.name = "custom_metric"

    def update(self, pred, target, idx):
        # Custom update logic
        score = np.sum((pred == target) * idx)
        self.result += score
        self.averaging_coef += idx.sum()

    def compute(self):
        return super().compute()

# Usage
metric = CustomMetric()
metric.update(np.array([0, 1, 0]), np.array([0, 1, 1]), np.array([1, 1, 0]))
print(metric.compute())  # Outputs the computed metric result

Metrics Zoo

  • The MSSIM class uses the structural similarity function from skimage.metrics.

  • The AveragePrecision class uses the average_precision_score function from sklearn.metrics.

  • The ROCAUCScore class uses the ROCAUCScore_score function from sklearn.metrics.

  • The Dice class uses custom logic to compute the Dice coefficient.

  • All metrics inherit from BaseMetric and follow a similar structure for updating and computing results.

MSSIM Metric

class lis2.metrics.mssim.MSSIM(*args, **kwargs)[source]

Bases: BaseMetric

MSSIM for evaluating the Mean Structural Similarity Index.

A metric to compute the Mean Structural Similarity Index (MSSIM) between model predictions and ground truth. This metric is useful for evaluating the similarity between two images, particularly in image segmentation tasks.

Configuration:
  • name (str): The name of the metric. Default is “mean_ssim”.

  • threshold (float): Threshold for binarizing predictions. Default is 0.5.

  • win_size (int): Window size for the structural similarity computation. Default is 7.

Example Configuration:
config = {
    "name": "mean_ssim",
    "threshold": 0.5,
    "win_size": 7
}
Aliases:
  • mssim

  • ssim

aliases = ['mssim', 'ssim']
config_schema: Dict = {'threshold': Schema(type=<class 'float'>, aliases=[], optional=True, default=0.5), 'win_size': Schema(type=<class 'int'>, aliases=[], optional=True, default=7)}
update(pred: ndarray, target: ndarray, idx: ndarray, *args: Any, **kwargs: Any) → None[source]

Updates the MSSIM metric with current predictions and targets.

Parameters:
  • pred (np.ndarray) – Model predictions (as probabilities, already flattened numpy arrays).

  • target (np.ndarray) – Ground truth (already flattened numpy arrays).

  • idx (np.ndarray) – Mask or weighting for valid examples.

  • *args – Variable length argument list.

  • **kwargs – Arbitrary keyword arguments.

The MSSIM class computes the Mean Structural Similarity Index (MSSIM) between model predictions and ground truth. It uses a threshold to binarize predictions and calculates the structural similarity for each segmentation.

Average Precision Metric

class lis2.metrics.average_precision.AveragePrecision(*args, **kwargs)[source]

Bases: BaseMetric

AveragePrecision for evaluating the average precision score.

A metric to compute the average precision score, which summarizes the precision-recall curve as the area under the curve. This metric is particularly useful for evaluating the performance of binary classifiers. It is based on the average_precision_score function from sklearn.metrics.

Configuration:
  • name (str): The name of the metric. Default is “average_precision”.

Example Configuration (YAML):
name: "average_precision"
Aliases:
  • average_precision

  • ap

  • map

aliases = ['average_precision', 'ap', 'map']
update(pred: ndarray, target: ndarray, idx: ndarray, *args: Any, **kwargs: Any) → None[source]

Updates the Average Precision metric with current predictions and targets.

Parameters:
  • pred (np.ndarray) – Model predictions, rounded for binary masks.

  • target (np.ndarray) – Ground truth labels (binary).

  • idx (np.ndarray) – Mask or weighting for valid examples.

  • *args – Variable length argument list.

  • **kwargs – Arbitrary keyword arguments.

The AveragePrecision class computes the average precision score between model predictions and ground truth. It rounds predictions to create binary masks and uses the average precision score from scikit-learn.

ROC AUC Metric

class lis2.metrics.roc.ROCAUCScore(*args, **kwargs)[source]

Bases: BaseMetric

ROCAUCScore for evaluating the area under the ROC curve.

A metric to compute the Area Under the Receiver Operating Characteristic Curve (ROC AUC) score, which is a measure of the ability of a classifier to distinguish between classes. This metric is particularly useful for binary classification problems.

Configuration:
  • name (str): The name of the metric. Default is “roc_auc”.

Example Configuration:
config = {
    "name": "roc_auc"
}
Aliases:
  • roc_auc

  • auc

aliases = ['roc_auc', 'auc']
update(pred: ndarray, target: ndarray, idx: ndarray, *args: Any, **kwargs: Any) → None[source]

Updates the ROCAUCScore metric with current predictions and targets.

Parameters:
  • pred (np.ndarray) – Model predictions.

  • target (np.ndarray) – Ground truth labels.

  • idx (np.ndarray) – Mask or weighting for valid examples.

  • *args – Variable length argument list.

  • **kwargs – Arbitrary keyword arguments.

The ROCAUCScore class computes the Area Under the Receiver Operating Characteristic Curve (ROC AUC) for evaluating binary classification models.

Dice Metric

class lis2.metrics.dice.Dice(*args, **kwargs)[source]

Bases: BaseMetric

Dice Metric for evaluating the similarity between two binary masks.

This metric is commonly used in image segmentation tasks to measure the overlap between predicted and ground truth binary masks. The Dice coefficient is a statistical tool that measures the similarity between two sets.

Configuration:
  • name (str): The name of the metric. Default is “dice”.

  • threshold (float): The threshold value to binarize the prediction. Default is 0.5.

Example Configuration:
config = {
    "name": "dice",
    "threshold": 0.5
}
Aliases:
  • dice

  • dice_index

aliases = ['dice', 'dice_index']
config_schema: Dict = {'threshold': Schema(type=<class 'float'>, aliases=[], optional=True, default=0.5)}
update(pred: ndarray, target: ndarray, idx: ndarray, **kwargs: Any) → None[source]

Updates the Dice metric with current predictions and targets.

Parameters:
  • pred (np.ndarray) – Model predictions.

  • target (np.ndarray) – Ground truth labels.

  • idx (np.ndarray) – Mask or weighting for valid examples.

  • **kwargs – Arbitrary keyword arguments.

The Dice class computes the Dice coefficient, a measure of overlap between two samples. It is often used for evaluating segmentation models.

Metric Manager

class lis2.metrics.metric_manager.MetricManager(metrics_configs: list[dict[str, Any]])[source]

Bases: object

compute() → dict[str, float][source]

Computes and returns the results of all metrics.

Returns:

A dictionary mapping metric names to their computed results.

Return type:

dict[str, float]

reset() → None[source]

Resets all metrics.

to_dict() → dict[str, float][source]

Returns the results of all metrics as a dictionary.

Returns:

A dictionary mapping metric names to their computed results.

Return type:

dict[str, float]

update(*args: Any, **kwargs: Any) → None[source]

Updates all metrics with the given arguments.

Parameters:
  • *args – Variable length argument list.

  • **kwargs – Arbitrary keyword arguments.

The MetricManager class manages multiple metrics, allowing for batch updates, computations, and resets. It provides a convenient way to handle multiple evaluation metrics simultaneously.