Datasets
BaseDataset
BaseDataset for creating datasets.
This abstract base class provides methods for detecting missing data. It is designed to be subclassed for specific dataset implementations.
Dataset Zoo
FilamentsDataset
FilamentsDataset for loading and managing preprocessed data stored in an HDF5 file.
This dataset class loads a dataset from an HDF5 file, extracts relevant patches, encodes specific parameters, and applies optional data augmentation techniques. This dataset class is specifically designed to handle the k-folds methods provided in the Controller file.
- Configuration:
dataset_path (Union[Path, str]): Path to the HDF5 file containing the dataset.
data_augmentations (list[Config]): Type of data augmentation to apply (default: None).
toEncode (list[str]): List of parameters to encode (default: []).
stride (int): Stride value for mapping patch indices (default: 1).
fold_assignments (Optional[dict]): Dictionary defining fold assignments (default: None).
fold_list (Optional[list]): List of folds to use in training (default: None).
use_all_patches (bool): Whether to include all patches in the dataset (default: False).
- Example Configuration (YAML):
dataset_path: "/path/to/dataset.h5" data_augmentation: [{"type": "ToTensor"}, {"type": "NoiseDataAugmentation", "name": "input", "to_augment": [0, 1], "keys_to_augment":["patch"]}] toEncode: ["param1", "param2"] stride: 2 fold_assignments: {"fold1": [0, 2, 4], "fold2": [1, 3, 5]} fold_list: [[[1, 2], [3], [4]], [[3, 4], [1], [2]]] use_all_patches: True
Disclamer: In most cases, when they are necessary (i.e k-folds), fold_assignments and fold_list are computed automatically by the Controller and don’t have to be specified.
Note
Torchvision Transforms and Custom Transforms are supported as Data Augmentations. Please refer to Data Augmentation for more details about the data augmentations
The FilamentsDataset class handles the loading, preprocessing, and augmentation of an HDF5 dataset. Below is a structured breakdown of its internal process:
Initialization
The dataset is loaded from the specified dataset_path, using the HDF5 format. The following dataset components are retrieved:
patches: Input data patches.
spines: Target labels.
labelled: Mask indicating labelled pixels.
A set of parameters to encode is created based on the
toEncodelist (i.e. thepositionof the patch in our example). Thecreate_mappingmethod is called to organize the dataset indices, i.e. to map with thefold assignmentcreated by the Controller.Data augmentationtechniques are initialized based on thedata_augmentationsconfiguration.Preconditions
The
preconditionsmethod ensures consistency in dataset configuration:If
fold assignmentsare provided, bothfold_assignmentsandfold_listmust be defined.If neither is provided, all dataset patches are used.
Dataset Mapping (create_mapping)
This method organizes dataset indices based on:
Fold assignments.
Stride for sampling (i.e. if
stride==2, we take one out of two patches).The use_all_patches flag, which determines whether all patches are included.
If
fold_assignmentsexists:It ensures only patches that have been assigned to folds are used.
If
use_all_patchesis False, only patches containing at least one labelled pixel are retained.
This method is particularly useful because some patches may not be included, depending on
fold_assignment.Sample Retrieval (__getitem__)
The dataset retrieves samples using index mapping (
dic_mapping). Apatch, its correspondinglabel, and anyencoded parametersare extracted.Data augmentationis applied if configured. The _create_sample method formats the data into tensors.Note
This is the output that will be used by the
trainer, so if you make a newdatasetor a newtrainer, please make sure that they correspond to one another.Sample Creation (_create_sample)
Converts the
patch,spinelabels, andlabelledmask into PyTorch tensors. Ensures correct tensor formatting(C, H, W). Encodes additional parameters if specified. Validates that the target tensor is non-empty.Logging and Debugging
Logs whether fold assignments are used. Warns if no valid data is found for the given fold configuration. Ensures patches are properly formatted and contain valid labels.
Subtleties:
Fold-aware Data Loading: Ensures patches are correctly assigned to training, validation, and test folds.
Efficient Sampling: Uses stride-based mapping for scalable dataset handling.
Data Integrity Checks: Ensures no empty target tensors are included.
Augmentation Support: Applies configurable transformations to enhance training diversity.
- class lis2.datasets.fakeData3d.Fake3DDataset(*args, **kwargs)[source]
Bases:
BaseDatasetDataset for creating fake 3D volumes by adding an additional dimension to 2D patches.
- Configurations:
dataset_path (
Union[Path, str]): Path to the HDF5 file containing the dataset.learning_mode (
str): Mode of learning strategy. Default is “conservative”.data_augmentation (
str): Type of data augmentation applied. Default is “noise”.normalization_mode (
str): Normalization mode used for preprocessing. Default is “none”.input_data_noise (
float): Amount of noise added to the input data. Default is 0.output_data_noise (
float): Amount of noise added to the output data. Default is 0.stride (
int): Stride used for indexing patches. Default is 2.fold_assignments (
defaultdict, optional): Dictionary mapping folds to dataset indices.fold_list (
list, optional): List of folds to include in the dataset.toEncode (
list, optional): List of parameters to be encoded. Default is [].
- config_schema: Dict = {'data_augmentation': Schema(type=<class 'str'>, aliases=[], optional=True, default=noise), 'dataset_path': Schema(type=typing.Union[pathlib.Path, str], aliases=[], optional=False, default=None), 'fold_assignments': Schema(type=<class 'collections.defaultdict'>, aliases=[], optional=True, default=None), 'fold_list': Schema(type=<class 'list'>, aliases=[], optional=True, default=None), 'input_data_noise': Schema(type=<class 'float'>, aliases=[], optional=True, default=0), 'learning_mode': Schema(type=<class 'str'>, aliases=[], optional=True, default=conservative), 'normalization_mode': Schema(type=<class 'str'>, aliases=[], optional=True, default=none), 'output_data_noise': Schema(type=<class 'float'>, aliases=[], optional=True, default=0), 'stride': Schema(type=<class 'int'>, aliases=[], optional=True, default=2), 'toEncode': Schema(type=<class 'list'>, aliases=[], optional=True, default=[])}
Adding a Custom Dataset
First, create a class that inherits from BaseDataset, and create its config_schema:
class CustomData(BaseDataset):
"""
Your custom dataset
It must implements __len__, _create_sample and __getitem__.
"""
config_schema = {
"dataset_path": Schema(Union[Path, str]),
"data_augmentations": Schema(list[Config], optional=True),
"toEncode": Schema(list, optional=True, default=[]),
"stride": Schema(int, default=1),
"fold_assignments": Schema(dict, optional=True),
"fold_list": Schema(list, optional=True),
}
Now you need to implement the methods :
data = # Load your data here
def __len__(self):
return len(self.data)
def __getitem__(self):
"""
Important note : This is the output that will be used by the trainer. Make sure that they correspond
"""
item1 = self.data["item1"]
item2 = self.data["item2"]
data_to_augment = {"item1": item1, "item2": item2}
item1, item2 = self.data_augmentations.compute(data_to_augment)
sample = self._create_sample(item1, item2)
return sample
Note
In getitem, please make sure that the name of the variables in data_to_augment corresponds to what you specify
in keys_to_encode in data_augmentations.
def _create_sample(self, item1, item2):
sample = {
"item1": item1,
"item2": item2,
}
return sample