franc.evaluation.evaluation

Collection of tools for the evaluation and testing of filters

Attributes

NDArrayF

NDArrayU

Classes

EvaluationRun

Representation of an evaluation run

Functions

measure_runtime(filter_classes[, n_samples, ...])

Measure the runtime of filers for a specific scenario

residual_power_ratio(target, prediction[, start, ...])

Calculate the ratio between residual power of the residual and the target signal

residual_amplitude_ratio(*args, **kwargs)

Calculate the ratio between residual amplitude of the residual and the target signal

Module Contents

franc.evaluation.evaluation.NDArrayF
franc.evaluation.evaluation.NDArrayU
franc.evaluation.evaluation.measure_runtime(filter_classes, n_samples=int(10000.0), n_channel=1, n_filter=128, idx_target=0, additional_filter_settings=None, repititions=1)

Measure the runtime of filers for a specific scenario Be aware that this gives no feedback upon how much multithreading is used!

Parameters:

  • n_samples (int) – Length of the test data

  • n_channel (int) – Number of witness sensor channels

  • n_filter (int) – Length of the FIR filters / input block size

  • idx_target (int) – Position of the prediction

  • additional_filter_settings (collections.abc.Sequence[dict[str, Any]] | None) – optional settings passed to the filters

  • repititions (int) – how many repititions to perform during the timing measurement

  • filter_classes (collections.abc.Sequence[franc.evaluation.filter_interface.FilterInterface]) –

Returns:

(time_conditioning, time_apply) each in seconds

Return type:

tuple[collections.abc.Sequence, collections.abc.Sequence]

class franc.evaluation.evaluation.EvaluationRun(method_configurations, dataset, optimization_metric, metrics=None, name='unnamed', directory='.', figsize=(10, 4))

Representation of an evaluation run

Parameters:
multi_sequence_support = True
method_configurations
dataset
optimization_metric
metrics = []
name = 'unnamed'
directory
figsize = (10, 4)
all_configurations_list: list | None = None
_check_method_configurations(method_configurations)

Throw meaningful errors for problems with the configurations

Parameters:

method_configurations (collections.abc.Sequence[tuple[type[franc.evaluation.filter_interface.FilterInterface], collections.abc.Sequence]]) –

Return type:

bool

get_all_configurations()

Returns a list of all unique (filter_technique, configuration) pairs.

Return type:

list

_create_folder_structure()

Create standardized folder structure for results

Return type:

None

static save_np_array_list(data, filename)

Save a list of numpy arrays to a .npz file

Parameters:

  • data (collections.abc.Sequence[collections.abc.Sequence[NDArrayF]] | collections.abc.Sequence[NDArrayF] | NDArrayF) –

  • filename (str | pathlib.Path) –

Return type:

None

static load_np_array_list(filename)

Load a list of numpy arrays from a .npz file

Parameters:

filename (str | pathlib.Path) –

Return type:

collections.abc.Sequence[NDArrayF]

static software_version_report()

generate a list of strings indicating the important software versions

Return type:

list[str]

static platform_info_report()

generate a list of strings indicating platform information (cpu, OS, ..)

Return type:

list[str]

generate_overview_plots(results)

Generate overview plots Returns a Report section with the generated plot

Parameters:

results (list[tuple[type[franc.evaluation.filter_interface.FilterInterface], list]]) –

generate_parameter_scan_plots(results)

Generate a plot of the optimization_metric values over all varied parameters Returns a report section with the generated plots

Parameters:

results (list[tuple[type[franc.evaluation.filter_interface.FilterInterface], list[tuple[dict, numpy.typing.NDArray, franc.evaluation.metrics.EvaluationMetricScalar, list[franc.evaluation.metrics.EvaluationMetric], str]]]]) –

Return type:

list[list[franc.evaluation.report_generation.ReportFigure]]

generate_report(results, compile_report=False, report_type=LatexReport)

Generate a report for the given results object from run()

Parameters:
get_prediction(filter_technique, conf)

Load the prediction created by applying the given filter and configuration to the dataset

Parameters:
Return type:

tuple[collections.abc.Sequence[numpy.typing.NDArray] | numpy.typing.NDArray, str, str]

run()

Execute the evaluation run

Returns:

list of (Prediction, optimization_metric, other_metrics) objects

Return type:

list[tuple[type[franc.evaluation.filter_interface.FilterInterface], list]]

hash_str()

returns a hash over the dataset and filtering configurations as a string

Return type:

str

franc.evaluation.evaluation.residual_power_ratio(target, prediction, start=None, stop=None, remove_dc=True)

Calculate the ratio between residual power of the residual and the target signal

Parameters:

  • target (collections.abc.Sequence) – target signal array

  • prediction (collections.abc.Sequence) – prediction array (same length as target

  • start (int | None) – use only a section of the arrays, start at this index

  • stop (int | None) – use only a section of the arrays, stop at this index

  • remove_dc (bool) – if true, the mean is subtracted from each array to remove the DC component before the calculations

Return type:

float

franc.evaluation.evaluation.residual_amplitude_ratio(*args, **kwargs)

Calculate the ratio between residual amplitude of the residual and the target signal

Parameters:

  • target – target signal array

  • prediction – prediction array (same length as target

  • start – use only a section of the arrays, start at this index

  • stop – use only a section of the arrays, stop at this index

  • component (remove DC) – remove DC component before calculation

Return type:

float