Trust#

class scio.scores.Trust(*, act_norm=None, mode='raw', alpha=?, k=10)[source]#

Bases: BaseScoreClassif

Trust for classification.

The score is build by comparing the distances of an input representation to those of calibration samples grouped by class, after having excluded class-wise outliers.

Parameters:

  • alpha (float) – Target (and minimum) proportion of outliers to exclude when building the alpha-high-density-sets.

  • k (int) – The k-th nearest neighbors distance is used to build the alpha-high-density-sets. Defaults to 10.

  • mode – See BaseScoreClassif.

  • act_norm – See BaseScore.

References

[JKGG18] (1,2,3)

Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. In Advances in Neural Information Processing Systems, volume 31. 2018. URL: https://proceedings.nips.cc/paper_files/paper/2018/file/7180cffd6a8e829dacfc2a31b3f72ece-Paper.pdf.

Hint

Below this point, the documentation is meant for development purposes only. Manual use of any listed member is highly discouraged. For usage, see Inferring with Confidence.

Useful methods defined here

alpha_high_density_index(population)

Build index for alpha-high-density-set of population.

trust_score(distances)

Compute trust score [JKGG18, algorithm 2].
alpha_high_density_index(population)[source]#

Build index for alpha-high-density-set of population.

It is built by excluding a proportion \(\alpha\) (rounded up to the nearest fraction possible) of outliers, for \(k\)-th NN distance.

Note

Because of a faiss corner case, we return None to manually handle the case of empty indexes.

static trust_score(distances)[source]#

Compute trust score [JKGG18, algorithm 2].

Parameters:

distances (Tensor) – For the batch, distances to every classes’ alpha-high-density-set. Shape (n_samples, n_classes).

Returns:

inv_preference (Tensor) – Inverse of multiplicative preference, that is the ratio between the the highest other value and the value itself. Same shape as distances.