Estimate Uncertainty
Synopsis
This operator generates uncertainty estimates based on the conformal prediction method.
Description
In many situations you do not just want to predict a value but also estimate the uncertainty of the prediction. This operator allows you to get an estimate for the uncertainty.
For regression the result are upper and lower bounds which can be seen as confidence bands. The algorithm guarantees that with 1-alpha probability the true value is within the bounds.
For classification the result is a set of predictions. The algorithm guarantees that the real prediction is within the set with a probability of 1-alpha.
For an introduction into the conformal prediction algorithm we recommend "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification" by Angelopoulos and Bates (2022) on which this implementation is loosely based.
Input
calibration
Data which is used calibrating the algorithm
mod
The model whose predictions are to be attached with uncertainties.
Output
mod
The conformal prediction model which can be used with apply model to estimate the uncertainty of forecast for unknown data.
ori
The original calibration data passed through
Parameters
Problem type
What kind of problem to be solved (classification or regression)
Problem type
What kind of problem algorithm you want to use for classification.
Error rate
Error rate for conformal prediction. The result is true to be in 1-alpha% of the cases.
Sample size
Only available for LIME. This is the number of random sample drawn to get a neighborhood for an individual datapoint
Redraw local samples
Only available for LIME. If checked the operator will draw new random samples in each iteration. If deactivated a set of examples is generated once and used for the interpretation of each example.
Locality
Only available for LIME. Locality controls how local the data points are. The lower locality the more you get a general answer and not a specific one. Also called kernel_width.