Calibration

Functionality to calibrate a trained, binary classification model using temperature scaling.

ECELoss

 ECELoss (n_bins=10)

Calculates the Expected Calibration Error of a model.

source

TemperatureSetter

 TemperatureSetter (model, lr=0.01, max_iter=1000, line_search_fn=None,
                    n_bins=10, verbose=True)

Calibrates a binary classification model optimizing temperature

source

ModelWithTemperature

 ModelWithTemperature (model)

A decorator which wraps a model with temperature scaling

source

plot_calibration_curve

 plot_calibration_curve (labels, logits, cal_logits=None, figsize=(6, 6),
                         n_bins=10, strategy='uniform')

source

Learner.calibrate_model

 Learner.calibrate_model (X=None, y=None, lr=0.01, max_iter=10000,
                          line_search_fn=None, n_bins=10,
                          strategy='uniform', show_plot=True, figsize=(6,
                          6), verbose=True)

from tsai.basics import *
from tsai.models.FCNPlus import FCNPlus

X, y, splits = get_UCR_data('FingerMovements', split_data=False)
tfms  = [None, [TSClassification()]]
batch_tfms = TSRobustScale()
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, FCNPlus, metrics=accuracy)
learn.fit_one_cycle(2)

epoch	train_loss	valid_loss	accuracy	time
0	0.724956	nan	nan	00:00
1	0.713688	nan	nan	00:00

/Users/nacho/notebooks/tsai/tsai/data/core.py:648: RuntimeWarning: overflow encountered in scalar add
  b = slice(b[0], min(self.n, b[0] + self.bs))

learn.calibrate_model()
calibrated_model = learn.calibrated_model

Before temperature - NLL: 0.696, ECE: 0.032
Calibrating the model...
...model calibrated
Optimal temperature: 3.641
After temperature  - NLL: 0.693, ECE: 0.018