Metrics
This module provides a collection of metrics for evaluating model performance. Each metric is a subclass of BaseMetric and can be used to compute specific evaluation scores.
BaseMetric Class
- class lis2.metrics.base_metric.BaseMetric(*args, **kwargs)[source]
Bases:
ABC,TypedConfigurableBaseMetric for defining custom metrics.
An abstract base class for defining custom metrics. This class provides a structure for implementing metrics that can be updated with predictions and targets, and then computed to yield a result.
- Configuration:
name (str): The name of the metric.
- Example Configuration:
config = { "name": "example_metric" }
- compute() float[source]
Computes the metric result.
- Returns:
The computed metric result.
- Return type:
float
- Raises:
ValueError – If no data is available to compute the metric.
- abstractmethod update(pred: ndarray, target: ndarray, idx: ndarray, *args: Any, **kwargs: Any) None[source]
Updates the metric with current predictions and targets.
- Parameters:
pred (np.ndarray) – Model predictions.
target (np.ndarray) – Ground truth labels.
idx (np.ndarray) – Mask or weighting for valid examples.
*args – Variable length argument list.
**kwargs – Arbitrary keyword arguments.
The BaseMetric class is an abstract base class for all metrics. It provides methods for updating, computing, and resetting metric results.
Example:
class CustomMetric(BaseMetric):
def __init__(self):
super().__init__()
self.name = "custom_metric"
def update(self, pred, target, idx):
# Custom update logic
score = np.sum((pred == target) * idx)
self.result += score
self.averaging_coef += idx.sum()
def compute(self):
return super().compute()
# Usage
metric = CustomMetric()
metric.update(np.array([0, 1, 0]), np.array([0, 1, 1]), np.array([1, 1, 0]))
print(metric.compute()) # Outputs the computed metric result
Metrics Zoo
The
MSSIMclass uses the structural similarity function fromskimage.metrics.The
AveragePrecisionclass uses theaverage_precision_scorefunction fromsklearn.metrics.The
ROCAUCScoreclass uses theROCAUCScore_scorefunction fromsklearn.metrics.The
Diceclass uses custom logic to compute the Dice coefficient.All metrics inherit from
BaseMetricand follow a similar structure for updating and computing results.
MSSIM Metric
- class lis2.metrics.mssim.MSSIM(*args, **kwargs)[source]
Bases:
BaseMetricMSSIM for evaluating the Mean Structural Similarity Index.
A metric to compute the Mean Structural Similarity Index (MSSIM) between model predictions and ground truth. This metric is useful for evaluating the similarity between two images, particularly in image segmentation tasks.
- Configuration:
name (str): The name of the metric. Default is “mean_ssim”.
threshold (float): Threshold for binarizing predictions. Default is 0.5.
win_size (int): Window size for the structural similarity computation. Default is 7.
- Example Configuration:
config = { "name": "mean_ssim", "threshold": 0.5, "win_size": 7 }
- Aliases:
mssim
ssim
- aliases = ['mssim', 'ssim']
- config_schema: Dict = {'threshold': Schema(type=<class 'float'>, aliases=[], optional=True, default=0.5), 'win_size': Schema(type=<class 'int'>, aliases=[], optional=True, default=7)}
- update(pred: ndarray, target: ndarray, idx: ndarray, *args: Any, **kwargs: Any) None[source]
Updates the MSSIM metric with current predictions and targets.
- Parameters:
pred (np.ndarray) – Model predictions (as probabilities, already flattened numpy arrays).
target (np.ndarray) – Ground truth (already flattened numpy arrays).
idx (np.ndarray) – Mask or weighting for valid examples.
*args – Variable length argument list.
**kwargs – Arbitrary keyword arguments.
The MSSIM class computes the Mean Structural Similarity Index (MSSIM) between model predictions and ground truth. It uses a threshold to binarize predictions and calculates the structural similarity for each segmentation.
Average Precision Metric
- class lis2.metrics.average_precision.AveragePrecision(*args, **kwargs)[source]
Bases:
BaseMetricAveragePrecision for evaluating the average precision score.
A metric to compute the average precision score, which summarizes the precision-recall curve as the area under the curve. This metric is particularly useful for evaluating the performance of binary classifiers. It is based on the average_precision_score function from sklearn.metrics.
- Configuration:
name (str): The name of the metric. Default is “average_precision”.
- Example Configuration (YAML):
name: "average_precision"
- Aliases:
average_precision
ap
map
- aliases = ['average_precision', 'ap', 'map']
- update(pred: ndarray, target: ndarray, idx: ndarray, *args: Any, **kwargs: Any) None[source]
Updates the Average Precision metric with current predictions and targets.
- Parameters:
pred (np.ndarray) – Model predictions, rounded for binary masks.
target (np.ndarray) – Ground truth labels (binary).
idx (np.ndarray) – Mask or weighting for valid examples.
*args – Variable length argument list.
**kwargs – Arbitrary keyword arguments.
The AveragePrecision class computes the average precision score between model predictions and ground truth. It rounds predictions to create binary masks and uses the average precision score from scikit-learn.
ROC AUC Metric
- class lis2.metrics.roc.ROCAUCScore(*args, **kwargs)[source]
Bases:
BaseMetricROCAUCScore for evaluating the area under the ROC curve.
A metric to compute the Area Under the Receiver Operating Characteristic Curve (ROC AUC) score, which is a measure of the ability of a classifier to distinguish between classes. This metric is particularly useful for binary classification problems.
- Configuration:
name (str): The name of the metric. Default is “roc_auc”.
- Example Configuration:
config = { "name": "roc_auc" }
- Aliases:
roc_auc
auc
- aliases = ['roc_auc', 'auc']
- update(pred: ndarray, target: ndarray, idx: ndarray, *args: Any, **kwargs: Any) None[source]
Updates the ROCAUCScore metric with current predictions and targets.
- Parameters:
pred (np.ndarray) – Model predictions.
target (np.ndarray) – Ground truth labels.
idx (np.ndarray) – Mask or weighting for valid examples.
*args – Variable length argument list.
**kwargs – Arbitrary keyword arguments.
The ROCAUCScore class computes the Area Under the Receiver Operating Characteristic Curve (ROC AUC) for evaluating binary classification models.
Dice Metric
- class lis2.metrics.dice.Dice(*args, **kwargs)[source]
Bases:
BaseMetricDice Metric for evaluating the similarity between two binary masks.
This metric is commonly used in image segmentation tasks to measure the overlap between predicted and ground truth binary masks. The Dice coefficient is a statistical tool that measures the similarity between two sets.
- Configuration:
name (str): The name of the metric. Default is “dice”.
threshold (float): The threshold value to binarize the prediction. Default is 0.5.
- Example Configuration:
config = { "name": "dice", "threshold": 0.5 }
- Aliases:
dice
dice_index
- aliases = ['dice', 'dice_index']
- config_schema: Dict = {'threshold': Schema(type=<class 'float'>, aliases=[], optional=True, default=0.5)}
- update(pred: ndarray, target: ndarray, idx: ndarray, **kwargs: Any) None[source]
Updates the Dice metric with current predictions and targets.
- Parameters:
pred (np.ndarray) – Model predictions.
target (np.ndarray) – Ground truth labels.
idx (np.ndarray) – Mask or weighting for valid examples.
**kwargs – Arbitrary keyword arguments.
The Dice class computes the Dice coefficient, a measure of overlap between two samples. It is often used for evaluating segmentation models.
Metric Manager
- class lis2.metrics.metric_manager.MetricManager(metrics_configs: list[dict[str, Any]])[source]
Bases:
object- compute() dict[str, float][source]
Computes and returns the results of all metrics.
- Returns:
A dictionary mapping metric names to their computed results.
- Return type:
dict[str, float]
The MetricManager class manages multiple metrics, allowing for batch updates, computations, and resets. It provides a convenient way to handle multiple evaluation metrics simultaneously.