Calibration

Functionality to calibrate a trained, binary classification model using temperature scaling.


source

ECELoss

 ECELoss (n_bins=10)

Calculates the Expected Calibration Error of a model.


source

TemperatureSetter

 TemperatureSetter (model, lr=0.01, max_iter=1000, line_search_fn=None,
                    n_bins=10, verbose=True)

Calibrates a binary classification model optimizing temperature


source

ModelWithTemperature

 ModelWithTemperature (model)

A decorator which wraps a model with temperature scaling


source

plot_calibration_curve

 plot_calibration_curve (labels, logits, cal_logits=None, figsize=(6, 6),
                         n_bins=10, strategy='uniform')

source

Learner.calibrate_model

 Learner.calibrate_model (X=None, y=None, lr=0.01, max_iter=10000,
                          line_search_fn=None, n_bins=10,
                          strategy='uniform', show_plot=True, figsize=(6,
                          6), verbose=True)
from tsai.basics import *
from tsai.models.FCNPlus import FCNPlus
X, y, splits = get_UCR_data('FingerMovements', split_data=False)
tfms  = [None, [TSClassification()]]
batch_tfms = TSRobustScale()
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, FCNPlus, metrics=accuracy)
learn.fit_one_cycle(2)
epoch train_loss valid_loss accuracy time
0 0.696826 0.706016 0.430000 00:04
1 0.690209 0.699720 0.490000 00:03
learn.calibrate_model()
calibrated_model = learn.calibrated_model
Before temperature - NLL: 0.700, ECE: 0.066
Calibrating the model...
...model calibrated
Optimal temperature: 6.383
After temperature  - NLL: 0.693, ECE: 0.019