from tsai.basics import *
from tsai.models.FCNPlus import FCNPlus
Calibration
Functionality to calibrate a trained, binary classification model using temperature scaling.
ECELoss
ECELoss (n_bins=10)
Calculates the Expected Calibration Error of a model.
TemperatureSetter
TemperatureSetter (model, lr=0.01, max_iter=1000, line_search_fn=None, n_bins=10, verbose=True)
Calibrates a binary classification model optimizing temperature
ModelWithTemperature
ModelWithTemperature (model)
A decorator which wraps a model with temperature scaling
plot_calibration_curve
plot_calibration_curve (labels, logits, cal_logits=None, figsize=(6, 6), n_bins=10, strategy='uniform')
Learner.calibrate_model
Learner.calibrate_model (X=None, y=None, lr=0.01, max_iter=10000, line_search_fn=None, n_bins=10, strategy='uniform', show_plot=True, figsize=(6, 6), verbose=True)
= get_UCR_data('FingerMovements', split_data=False)
X, y, splits = [None, [TSClassification()]]
tfms = TSRobustScale()
batch_tfms = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
dls = ts_learner(dls, FCNPlus, metrics=accuracy)
learn 2) learn.fit_one_cycle(
epoch | train_loss | valid_loss | accuracy | time |
---|---|---|---|---|
0 | 0.696826 | 0.706016 | 0.430000 | 00:04 |
1 | 0.690209 | 0.699720 | 0.490000 | 00:03 |
learn.calibrate_model()= learn.calibrated_model calibrated_model
Before temperature - NLL: 0.700, ECE: 0.066
Calibrating the model...
...model calibrated
Optimal temperature: 6.383
After temperature - NLL: 0.693, ECE: 0.019