summit.multiview_platform.result_analysis package

Submodules

summit.multiview_platform.result_analysis.duration_analysis module

get_duration(results)
plot_durations(durations, directory, database_name, durations_stds=None)

summit.multiview_platform.result_analysis.error_analysis module

gen_error_data(sample_errors)

Used to format the error data in order to plot it efficiently. The data is saves in a .csv file.

Parameters:

sample_errors (dict of dicts of np.arrays) – A dictionary conatining all the useful data. Organized as : sample_errors[<classifier_name>][“error_on_samples”] is a np.array of ints with a - 1 if the classifier <classifier_name> classifier well the sample, - 0 if it fail to classify the sample, - -100 if it did not classify the sample (multiclass one versus one).

Returns:

  • nbClassifiers (int) – Number of different classifiers.

  • nbExamples (int) – NUmber of samples.

  • nbCopies (int) – The number of times the data is copied (classifier wise) in order for the figure to be more readable.

  • classifiers_names (list of strs) – The names fo the classifiers.

  • data (np.array of shape (nbClassifiers, nbExamples)) – A matrix with zeros where the classifier failed to classifiy the sample, ones where it classified it well and -100 if the sample was not classified.

  • error_on_samples (np.array of shape (nbExamples,)) – An array counting how many classifiers failed to classifiy each samples.

gen_error_data_glob(iter_results, stats_iter)
get_sample_errors(groud_truth, results)
Used to get for each classifier and each sample whether the classifier

has misclassified the sample or not.

Parameters:

  • ground_truth (numpy array of 0, 1 and -100 (if multiclass)) – The array with the real labels of the samples

  • results (list of MonoviewResult and MultiviewResults objects) – A list containing all the resluts for all the mono- & multi-view experimentations.

Returns:

sample_errors –

For each classifier, has an entry with a np.array over the samples,

with a 1 if the samples was

well-classified, a 0 if not and if it’s multiclass classification, a

-100 if the samples was not seen during

the one versus one classification.

Return type:

dict of np.array

iter_cmap(statsIter)

Used to generate a colormap that will have a tick for each iteration : the whiter the better.

Parameters:

statsIter (int) – The number of statistical iterations.

Returns:

  • cmap (matplotlib.colors.ListedColorMap object) – The colormap.

  • norm (matplotlib.colors.BoundaryNorm object) – The bounds for the colormap.

plot_2d(data, classifiers_names, nb_classifiers, file_name, dataset_name, labels=None, stats_iter=1, use_plotly=True, sample_ids=None, label_names=None)

Used to generate a 2D plot of the errors.

Parameters:

  • data (np.array of shape (nbClassifiers, nbExamples)) – A matrix with zeros where the classifier failed to classifiy the sample, ones where it classified it well and -100 if the sample was not classified.

  • classifiers_names (list of str) – The names of the classifiers.

  • nb_classifiers (int) – The number of classifiers.

  • file_name (str) – The name of the file in which the figure will be saved (“error_analysis_2D.png” will be added at the end)

  • minSize (int, optinal, default: 10) – The minimum width and height of the figure.

  • width_denominator (float, optional, default: 1.0) – To obtain the image width, the number of classifiers will be divided by this number.

  • height_denominator (float, optional, default: 1.0) – To obtain the image width, the number of samples will be divided by this number.

  • stats_iter (int, optional, default: 1) – The number of statistical iterations realized.

plot_errors_bar(error_on_samples, nb_samples, file_name, dataset_name, use_plotly=True, sample_ids=None)

Used to generate a barplot of the muber of classifiers that failed to classify each samples

Parameters:

  • error_on_samples (np.array of shape (nbExamples,)) – An array counting how many classifiers failed to classifiy each samples.

  • classifiers_names (list of str) – The names of the classifiers.

  • nb_classifiers (int) – The number of classifiers.

  • nb_samples (int) – The number of samples.

  • file_name (str) – The name of the file in which the figure will be saved (“error_analysis_2D.png” will be added at the end)

publish_all_sample_errors(iter_results, directory, stats_iter, sample_ids, labels, data_base_name, label_names)
publish_sample_errors(sample_errors, directory, database_name, label_names, sample_ids, labels)

summit.multiview_platform.result_analysis.execution module

analyze(results, stats_iter, benchmark_argument_dictionaries, metrics, directory, sample_ids, labels, feature_ids, view_names)

Used to analyze the results of the previous benchmarks

analyze_all(iter_results, stats_iter, directory, data_base_name, sample_ids, label_names)

Used to format the results in order to plot the mean results on the iterations

analyze_iterations(results, benchmark_argument_dictionaries, stats_iter, metrics, sample_ids, labels, feature_ids, view_names)

Used to extract and format the results of the different experimentations performed.

Parameters:

  • results (list) –

    The result list returned by the benchmark execution function. For each

    executed benchmark, contains

    a flag & a result element. The flag is a way to identify to which benchmark the results belong, formatted this way : flag = iter_index, [classifierPositive, classifierNegative] with - iter_index the index of the statistical iteration - [classifierPositive, classifierNegative] the indices of the labels considered positive and negative by the classifier (mainly useful for one versus one multiclass classification).

  • benchmark_argument_dictionaries (list of dicts) – The list of all the arguments passed to the benchmark executing functions.

  • statsIter (int) – The number of statistical iterations.

  • metrics (list of lists) – THe list containing the metrics and their configuration.

Returns:

results – The list contains a dictionary for each statistical iteration. This dictionary contains a dictionary for each label combination, regrouping the scores for each metrics and the information useful to plot errors on samples.

Return type:

list of dicts of dicts

format_previous_results(iter_results_lists)

Formats each statistical iteration’s result into a mean/std analysis for the metrics and adds the errors of each statistical iteration.

Parameters:

iter_results_lists (The raw results, for each statistical iteration i) –

contains

  • biclass_results[i][“metrics_scores”] is a dictionary with a

pd.dataframe for each metrics - biclass_results[i][“sample_errors”], a dicaitonary with a np.array for each classifier.

Returns:

  • metrics_analysis (The mean and std dataframes for each metrics)

  • error_analysis (A dictionary containing the added errors) – arrays for each classifier

get_arguments(benchmark_argument_dictionaries, iter_index)

Used to get the arguments passed to the benchmark executing function corresponding to the flag of an experimentation.

Parameters:

  • flag (list) – The needed experimentation’s flag.

  • benchmark_argument_dictionaries (list of dicts) – The list of all the arguments passed to the benchmark executing functions.

Returns:

benchmark_argument_dictionary – All the arguments passed to the benchmark executing function for the needed experimentation.

Return type:

dict

summit.multiview_platform.result_analysis.feature_importances module

get_feature_importances(result, feature_ids=None, view_names=None)

Extracts the feature importance from the monoview results and stores them in a dictionnary : feature_importance[view_name] is a pandas.DataFrame of size n_feature*n_clf containing a score of importance for each feature.

Parameters:

result (list of results)

Returns:

feature_importances – The dictionary containing all the feature importance for each view as pandas DataFrames

Return type:

dict of pd.DataFrame

plot_feature_importances(file_name, feature_importance, feature_std)
plot_feature_relevance(file_name, feature_importance, feature_std, metric_scores)
publish_feature_importances(feature_importances, directory, database_name, feature_stds=None, metric_scores=None)

summit.multiview_platform.result_analysis.metric_analysis module

autolabel(rects, ax, set=1, std=None)

Used to print the score below the bars.

Parameters:

  • rects (pyplot bar object) – THe bars.

  • ax (pyplot ax object) – The ax.

  • set (integer) – 1 means the test scores, anything else means the train score

  • std (None or array) – The standard deviations in the case of statsIter results.

get_fig_size(nb_results, min_size=15, multiplier=1.0, bar_width=0.35)

Used to get the image size to save the figure and the bar width, depending on the number of scores to plot.

Parameters:

  • nb_results (int) – The number of couple of bar to plot.

  • min_size (int) – The minimum size of the image, if there are few classifiers to plot.

  • multiplier (float) – The ratio between the image size and the number of classifiers.

  • bar_width (float) – The width of the bars in the figure. Mainly here to centralize bar_width.

Returns:

  • fig_kwargs (dict of arguments) – The argument restraining the size of the figure, usable directly in the subplots function of matplotlib.pyplot.

  • bar_width (float) – The width of the bars in the figure. Mainly here to centralize bar_width.

get_metrics_scores(metrics, results, label_names)

Used to extract metrics scores in case of classification

Parameters:

  • metrics (dict) – The metrics names with configuration metrics[i][0] = name of metric i

  • results (list of MonoviewResult and MultiviewResults objects) – A list containing all the results for all the monoview experimentations.

Returns:

metricsScores – Regroups all the scores for each metrics for each classifier and for the train and test sets. organized as : -metricScores[metric_name][“classifiers_names”] is a list of all the classifiers available for this metric, -metricScores[metric_name][“train_scores”] is a list of all the available classifiers scores on the train set, -metricScores[metric_name][“test_scores”] is a list of all the available classifiers scores on the test set.

Return type:

dict of dict of list

init_plot(results, metric_name, metric_dataframe, directory, database_name, class_metric_scores)
plot_class_metric_scores(class_test_scores, class_file_name, labels_names, classifier_names, metric_name, stds=None, tag='')
plot_metric_scores(train_scores, test_scores, names, nb_results, metric_name, file_name, dataset_name, tag='', train_STDs=None, test_STDs=None, use_plotly=True)

Used to plot and save the score barplot for a specific metric.

Parameters:

  • train_scores (list or np.array of floats) – The scores of each classifier on the training set.

  • test_scores (list or np.array of floats) – The scores of each classifier on the testing set.

  • names (list or np.array of strs) – The names of all the classifiers.

  • nb_results (int) – The number of classifiers to plot.

  • metric_name (str) – The plotted metric’s name

  • file_name (str) – The name of the file where the figure will be saved.

  • tag (str) – Some text to personalize the title, must start with a whitespace.

  • train_STDs (np.array of floats or None) – The array containing the standard deviations for the averaged scores on the training set.

  • test_STDs (np.array of floats or None) – The array containing the standard deviations for the averaged scores on the testing set.

publish_all_metrics_scores(iter_results, class_iter_results, directory, data_base_name, stats_iter, label_names, min_size=10)
publish_metrics_graphs(metrics_scores, directory, database_name, labels_names, class_metric_scores)

Used to sort the results (names and both scores) in descending test score order.

Parameters:

  • metrics_scores (dict of dicts of lists or np.arrays) – Keys : The names of the metrics. Values : The scores and names of each classifier .

  • directory (str) – The path to the directory where the figures will be saved.

  • database_name (str) – The name of the database on which the experiments where conducted.

  • labels_names (list of strs) – The name corresponding to each numerical label.

Return type:

results

sort_by_test_score(train_scores, test_scores, names, train_STDs=None, test_STDs=None)

Used to sort the results (names and both scores) in descending test score order.

Parameters:

  • train_scores (np.array of floats) – The scores of each classifier on the training set.

  • test_scores (np.array of floats) – The scores of each classifier on the testing set.

  • names (np.array of strs) – The names of all the classifiers.

  • train_STDs (np.array of floats or None) – The array containing the standard deviations for the averaged scores on the training set.

  • test_STDs (np.array of floats or None) – The array containing the standard deviations for the averaged scores on the testing set.

Returns:

  • sorted_names (np.array of strs) – The names of all the classifiers, sorted in descending test score order.

  • sorted_train_scores (np.array of floats) – The scores of each classifier on the training set, sorted in descending test score order.

  • sorted_test_scores (np.array of floats) – The scores of each classifier on the testing set, sorted in descending test score order.

  • sorted_train_STDs (np.array of floats or None) – The array containing the standard deviations for the averaged scores on the training set, sorted in descending test score order.

  • sorted_test_STDs (np.array of floats or None) – The array containing the standard deviations for the averaged scores on the testing set, sorted in descending test score order.

summit.multiview_platform.result_analysis.noise_analysis module

summit.multiview_platform.result_analysis.tracebacks_analysis module

publish_tracebacks(directory, database_name, labels_names, tracebacks, iter_index)
save_dict_to_text(dictionnary, output_file)
save_failed(failed_list, directory)

Module contents