In practice

In TSB-UAD, we provide a unique fonction to retrieve all evaluation measures.

class TSB_UAD.vus.metrics.get_metrics(score, labels, metric='all', version='opt', slidingWindow=5, thre=250)

Compute all (or some) evaluation measures for a given score and labels.

Parameters

  • score (numpy array of shape (n_samples,)) – The input score to evaluate.

  • labels (numpy array of shape (n_samples,)) – the labels to compare the score with. It have to be composed of 0 and 1 (1 indicating if the point is an anomaly).

  • slidingWindow (int,) – Buffer length for Range-based AUC measures and for VUS-based measures. For Range-AUC, the buffer length is exactly equals to slidingWindow. For VUS-based measures, the buffer length varies from 0 to 2*slidingWindow.

  • version (string, optional, default 'opt') –

    Implementation of VUS.

    • if ‘opt’, run the default implementation

    • if ‘opt_mem’, run the optimized implementation, but more complex in memory

  • thre (int, optional, default 250) – Number of thresholds for VUS

  • metric (string, optional, default 'vus') –

    compute a subset or all metrics:

    • if ‘vus’, compute and store in a dictionary the following measures (the string is the key of each measure).

      • ’R_AUC_ROC’, Range-adapted version of AUC-ROC [Paparrizos et al. 2022]

      • ’R_AUC_PR’, Range-adapted version of AUC-PR [Paparrizos et al. 2022]

      • ’VUS_ROC’, Volume under the surface for ROC [Paparrizos et al. 2022]

      • ’VUS_PR’, Volume under the surface for PR [Paparrizos et al. 2022]

    • if ‘all’, compute all measures and store them in a dictionary (the string is the key of each measure). The threhold based measures are computed with a predifined threshold (score_mu + 3*score_sigma). In total here are the measures:

      • ’AUC_ROC’, Area Under the ROC Curve

      • ’AUC_PR’, Area Under the Precision-Recall Curve

      • ’Precision’, generic Precision

      • ’Recall’, generic Recall

      • ’F’, generic F-score (with beta equals to 1)

      • ’Precision_at_k’, generic precision at k.

      • ’Rprecision’, Time series-adapted Precision [Tatbul et al. 2018].

      • ’Rrecall’, Time series-adapted Recall [Tatbul et al. 2018].

      • ’RF’, Time series-adapted F-score (with beta equals to 1) [Tatbul et al. 2018].

      • ’R_AUC_ROC’, Range-adapted version of AUC-ROC [Paparrizos et al. 2022]

      • ’R_AUC_PR’, Range-adapted version of AUC-PR [Paparrizos et al. 2022]

      • ’VUS_ROC’, Volume under the surface for ROC [Paparrizos et al. 2022]

      • ’VUS_PR’, Volume under the surface for PR [Paparrizos et al. 2022]

      • ’Affiliation_Precision’, Affiliation-based precision [Huet et al. 2022]

      • ’Affiliation_Recall’, Affiliation-based recall [Huet et al. 2022]

Returns

metrics – contains the accuracy values for all evaluation measures.

Return type

dictionary

Example of usage

We depicts below the usage of get_metrics.

import os
import numpy as np
import pandas as pd
from TSB_UAD.utils.visualisation import plotFig
from TSB_UAD.models.sand import SAND
from TSB_UAD.models.feature import Window
from TSB_UAD.utils.slidingWindows import find_length
from TSB_UAD.vus.metrics import get_metrics

#Read data
filepath = 'PATH_TO_TSB_UAD/ECG/MBA_ECG805_data.out'
df = pd.read_csv(filepath, header=None).dropna().to_numpy()
name = filepath.split('/')[-1]

data = df[:,0].astype(float)
label = df[:,1].astype(int)

#Pre-processing
slidingWindow = find_length(data)

# Run SAND (offline)
modelName='SAND (offline)'
clf = SAND(pattern_length=slidingWindow,subsequence_length=4*(slidingWindow))
clf.fit(data,overlaping_rate=int(1.5*slidingWindow))
score = clf.decision_scores_

#Post-processing
score = MinMaxScaler(feature_range=(0,1)).fit_transform(score.reshape(-1,1)).ravel()


#Print accuracy
results = get_metrics(score, label, metric="all", slidingWindow=slidingWindow)
for metric in results.keys():
    print(metric, ':', results[metric])

AUC_ROC : 0.996779310807228
AUC_PR : 0.8942079947725918
Precision : 0.7393483709273183
Recall : 0.9735973597359736
F : 0.8404558404558404
Precision_at_k : 0.9735973597359736
Rprecision : 0.7394705860012913
Rrecall : 0.9790057437116261
RF : 0.8425439890952773
R_AUC_ROC : 0.9996748675955897
R_AUC_PR : 0.9911647851406946
VUS_ROC : 0.9993050973645579
VUS_PR : 0.9802087454821152
Affiliation_Precision : 0.9825340283920497
Affiliation_Recall : 1.0

References

  • [Paparrizos et al. 2022] John Paparrizos, Paul Boniol, Themis Palpanas, Ruey S. Tsay, Aaron Elmore, and Michael J. Franklin. 2022. Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection. Proc. VLDB Endow. 15, 11 (July 2022), 2774–2787.

  • [Tatbul et al. 2018] N. Tatbul, T.J. Lee, S. Zdonik, M. Alam, J. Gottschlich, in Advances in Neural Information Processing Systems, vol. 31

  • [Huet et al. 2022] Alexis Huet, Jose Manuel Navarro, and Dario Rossi. 2022. Local Evaluation of Time Series Anomaly Detection Algorithms. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ‘22).