BaseScore

class scio.scores.BaseScore(*, act_norm=None)

Bases: ParamClass, Generic

Base class for every scoring algorithms.

A score is defined through:

1. a calibration procedure, during which In-Distribution data and labels may be observed;

2. an inference procedure.

These should respectively be carried out in calibrate() and get_conformity().

Scores are paramclasses and hyperparameters are defined as their parameters. Optionally, scores may override _on_param_will_be_set() and _check_params() (if so, it should call super()._check_params()). Refer to source code for examples.

For development, the score should not assume anything about which layers are recorded. All that is known is that the activations() method returns activations following the layers recording order (see concatenate keyword for scores using only one layer).

It is also possible to use on-the-fly activations postprocessing during inference (see Recorder.forward()) and access the network’s named parameters for recorded layers with Recorder.recorded_params (see e.g. GradNorm).

For usage, instances should be fit() before being called:

score = Score(param="param")
score.fit(rnet, calib_data, calib_labels)
out, conformity = score(test_data)

Parameters:

act_norm (float, optional) – If provided, the recorded activations are normalized for norm act_norm. Should not be 0 or nan.

Useful methods defined here

__call__(inputs)	Return bound net output with conformity estimate.
__repr__()	Add fit/unfit information to paramclass repr.
activations(*[, concatenate])	Return last recorded activations of bound net.
calibrate(calib_data, calib_labels)	Calibrate the score with In-Distribution data.
fit(rnet, calib_data, calib_labels)	Fit the score for given rnet and In-Distribution data.
get_conformity(inputs)	Compute output and associated conformity at inference.
reset_timer()	Reset timer.
unfit()	Unfit the score instance.

Useful attributes defined here

rnet	Recorder net bound by fit().
timer	The ScoreTimer instance for self.

__call__(inputs)

Return bound net output with conformity estimate.

Parameters:

inputs (Tensor) – Input samples to process.

Returns:

• out (Tensor) – The natural output of the bound neural network.

• conformities (Conformities) – The associated conformity.

Raises:

RuntimeError – If the instance is not fit.

__repr__()

Add fit/unfit information to paramclass repr.

Example

>>> KNN()
KNN[unfit](act_norm=2.0, mode='raw', k=?, index_metric='l2')

activations(*, concatenate=False)

Return last recorded activations of bound net.

During a standard forward pass, Recorder nets record the requested activations. This method retrieves those, with additional per-sample normalization if specified by act_norm. The normalization occurs layer-wise, unless concatenate is True.

Parameters:

concatenate (bool) – If True, flattens every sample at every layer and concatenates across layers, in which case out is a \(1\)-tuple of one Tensor with shape (n_samples, tot_data_dim). This option is handy for scoring techniques that only work with one latent space: concatenating across layers mimicks having recorded only one layer. Keep in mind that it might be preferable to only record one layer in the first place, if the technique is designed this way. Defaults to False.

Returns:

out (tuple[Tensor, ...]) – The last recorded activations. It is a tuple of length the number of recorded layers if not contatenate, otherwise one. If not concatenate, the shape of samples is untouched.

abstractmethod calibrate(calib_data, calib_labels)

Calibrate the score with In-Distribution data.

Parameters:

• calib_data (Tensor) – Batched calibration samples. Shape (n_calib, *sample_shape).

• calib_labels (Labels) – Batched labels.

fit(rnet, calib_data, calib_labels)

Fit the score for given rnet and In-Distribution data.

Parameters:

• rnet (Recorder) – Target Recorder net under analysis. It becomes bound to the score as self.rnet until the next fit() call.

• calib_data (Tensor) – Batched calibration samples. Shape (n_calib, *sample_shape).

• calib_labels (Labels) – Batched labels.

Returns:

self (Self) – The fit instance.

abstractmethod get_conformity(inputs)

Compute output and associated conformity at inference.

Parameters:

inputs (Tensor) – Batched data to be processed by the bound neural network. Shape (n_sample, *sample_shape).

Returns:

• out (Tensor) – The neural network output. Shape (n_samples, *output_sample_shape).

• conformity (Conformities) – Associated conformities.

reset_timer()

Reset timer.

unfit()

Unfit the score instance.