Datasets

BaseDataset

BaseDataset for creating datasets.

This abstract base class provides methods for detecting missing data. It is designed to be subclassed for specific dataset implementations.

Dataset Zoo

FilamentsDataset

FilamentsDataset for loading and managing preprocessed data stored in an HDF5 file.

This dataset class loads a dataset from an HDF5 file, extracts relevant patches, encodes specific parameters, and applies optional data augmentation techniques. This dataset class is specifically designed to handle the k-folds methods provided in the Controller file.

Configuration:
  • dataset_path (Union[Path, str]): Path to the HDF5 file containing the dataset.

  • data_augmentations (list[Config]): Type of data augmentation to apply (default: None).

  • toEncode (list[str]): List of parameters to encode (default: []).

  • stride (int): Stride value for mapping patch indices (default: 1).

  • fold_assignments (Optional[dict]): Dictionary defining fold assignments (default: None).

  • fold_list (Optional[list]): List of folds to use in training (default: None).

  • use_all_patches (bool): Whether to include all patches in the dataset (default: False).

Example Configuration (YAML):
dataset_path: "/path/to/dataset.h5"
data_augmentation: [{"type": "ToTensor"},
{"type": "NoiseDataAugmentation", "name": "input", "to_augment": [0, 1], "keys_to_augment":["patch"]}]
toEncode: ["param1", "param2"]
stride: 2
fold_assignments: {"fold1": [0, 2, 4], "fold2": [1, 3, 5]}
fold_list: [[[1, 2], [3], [4]], [[3, 4], [1], [2]]]
use_all_patches: True

Disclamer: In most cases, when they are necessary (i.e k-folds), fold_assignments and fold_list are computed automatically by the Controller and don’t have to be specified.

Note

Torchvision Transforms and Custom Transforms are supported as Data Augmentations. Please refer to Data Augmentation for more details about the data augmentations

The FilamentsDataset class handles the loading, preprocessing, and augmentation of an HDF5 dataset. Below is a structured breakdown of its internal process:

  1. Initialization

    The dataset is loaded from the specified dataset_path, using the HDF5 format. The following dataset components are retrieved:

    • patches: Input data patches.

    • spines: Target labels.

    • labelled: Mask indicating labelled pixels.

    A set of parameters to encode is created based on the toEncode list (i.e. the position of the patch in our example). The create_mapping method is called to organize the dataset indices, i.e. to map with the fold assignment created by the Controller. Data augmentation techniques are initialized based on the data_augmentations configuration.

  2. Preconditions

    The preconditions method ensures consistency in dataset configuration:

    • If fold assignments are provided, both fold_assignments and fold_list must be defined.

    • If neither is provided, all dataset patches are used.

  3. Dataset Mapping (create_mapping)

    This method organizes dataset indices based on:

    • Fold assignments.

    • Stride for sampling (i.e. if stride==2, we take one out of two patches).

    • The use_all_patches flag, which determines whether all patches are included.

    If fold_assignments exists:

    • It ensures only patches that have been assigned to folds are used.

    • If use_all_patches is False, only patches containing at least one labelled pixel are retained.

    This method is particularly useful because some patches may not be included, depending on fold_assignment.

  4. Sample Retrieval (__getitem__)

    The dataset retrieves samples using index mapping (dic_mapping). A patch, its corresponding label, and any encoded parameters are extracted. Data augmentation is applied if configured. The _create_sample method formats the data into tensors.

    Note

    This is the output that will be used by the trainer, so if you make a new dataset or a new trainer, please make sure that they correspond to one another.

  5. Sample Creation (_create_sample)

    Converts the patch, spine labels, and labelled mask into PyTorch tensors. Ensures correct tensor formatting (C, H, W). Encodes additional parameters if specified. Validates that the target tensor is non-empty.

  6. Logging and Debugging

    Logs whether fold assignments are used. Warns if no valid data is found for the given fold configuration. Ensures patches are properly formatted and contain valid labels.

Subtleties:

  • Fold-aware Data Loading: Ensures patches are correctly assigned to training, validation, and test folds.

  • Efficient Sampling: Uses stride-based mapping for scalable dataset handling.

  • Data Integrity Checks: Ensures no empty target tensors are included.

  • Augmentation Support: Applies configurable transformations to enhance training diversity.

class lis2.datasets.fakeData3d.Fake3DDataset(*args, **kwargs)[source]

Bases: BaseDataset

Dataset for creating fake 3D volumes by adding an additional dimension to 2D patches.

Configurations:
  • dataset_path (Union[Path, str]): Path to the HDF5 file containing the dataset.

  • learning_mode (str): Mode of learning strategy. Default is “conservative”.

  • data_augmentation (str): Type of data augmentation applied. Default is “noise”.

  • normalization_mode (str): Normalization mode used for preprocessing. Default is “none”.

  • input_data_noise (float): Amount of noise added to the input data. Default is 0.

  • output_data_noise (float): Amount of noise added to the output data. Default is 0.

  • stride (int): Stride used for indexing patches. Default is 2.

  • fold_assignments (defaultdict, optional): Dictionary mapping folds to dataset indices.

  • fold_list (list, optional): List of folds to include in the dataset.

  • toEncode (list, optional): List of parameters to be encoded. Default is [].

config_schema: Dict = {'data_augmentation': Schema(type=<class 'str'>, aliases=[], optional=True, default=noise), 'dataset_path': Schema(type=typing.Union[pathlib.Path, str], aliases=[], optional=False, default=None), 'fold_assignments': Schema(type=<class 'collections.defaultdict'>, aliases=[], optional=True, default=None), 'fold_list': Schema(type=<class 'list'>, aliases=[], optional=True, default=None), 'input_data_noise': Schema(type=<class 'float'>, aliases=[], optional=True, default=0), 'learning_mode': Schema(type=<class 'str'>, aliases=[], optional=True, default=conservative), 'normalization_mode': Schema(type=<class 'str'>, aliases=[], optional=True, default=none), 'output_data_noise': Schema(type=<class 'float'>, aliases=[], optional=True, default=0), 'stride': Schema(type=<class 'int'>, aliases=[], optional=True, default=2), 'toEncode': Schema(type=<class 'list'>, aliases=[], optional=True, default=[])}
create_mapping()[source]

Creates a mapping of dataset indices based on fold assignments.

Returns:

Dictionary mapping indices from 0 to the total number of patches considering the stride.

Return type:

dict

Adding a Custom Dataset

First, create a class that inherits from BaseDataset, and create its config_schema:

class CustomData(BaseDataset):
    """
    Your custom dataset

    It must implements __len__, _create_sample and __getitem__.
    """
    config_schema = {
        "dataset_path": Schema(Union[Path, str]),
        "data_augmentations": Schema(list[Config], optional=True),
        "toEncode": Schema(list, optional=True, default=[]),
        "stride": Schema(int, default=1),
        "fold_assignments": Schema(dict, optional=True),
        "fold_list": Schema(list, optional=True),
    }

Now you need to implement the methods :

data = # Load your data here

def __len__(self):
    return len(self.data)

def __getitem__(self):
    """
    Important note : This is the output that will be used by the trainer. Make sure that they correspond
    """

    item1 = self.data["item1"]
    item2 = self.data["item2"]

    data_to_augment = {"item1": item1, "item2": item2}
    item1, item2 = self.data_augmentations.compute(data_to_augment)

    sample = self._create_sample(item1, item2)

    return sample

Note

In getitem, please make sure that the name of the variables in data_to_augment corresponds to what you specify in keys_to_encode in data_augmentations.

def _create_sample(self, item1, item2):

    sample = {
        "item1": item1,
        "item2": item2,
    }
    return sample