#1. Why evaluation matters in anomaly detection
Anomaly detection (AD) identifies rare observations or patterns that depart substantially from the normal data distribution. Applications include cybersecurity (detecting network intrusions and malicious activity)12, finance (identifying fraudulent transactions)3, industrial manufacturing and inspection (finding product defects and system faults)45, medicine and medical imaging (identifying signs of disease or abnormal tissue)67, and systems monitoring (detecting performance degradation and predicting failures)8.
Sound evaluation metrics are essential for comparing algorithms, tuning hyperparameters, and establishing how reliably a model performs. In safety-critical applications, evaluation also has to account for the demand to explain individual decisions.910
Several features of anomaly detection make evaluation difficult.
- Class imbalance. By definition, anomalies are rare relative to normal observations.1112 An ordinary classification metric such as accuracy can therefore give a misleading picture. A model can achieve very high accuracy simply by labeling every observation normal.13
- Different types of anomaly. A point anomaly is a single unusual observation. A contextual anomaly is unusual only in a particular setting. A collective anomaly emerges from a group of observations taken together. A pattern-wise anomaly takes the form of a distinctive pattern.31415 Each can call for a different unit of evaluation.
- Domain specificity. A deviation matters for different reasons in different settings. A small change in body temperature can merit attention in medicine, while a fluctuation of comparable relative size may be ordinary in a financial series. The domain affects both the choice of metric and the interpretation of its value.
- Data modality. Images9, time series14, and graphs15 each have structural properties that can warrant specialized metrics.
- Lack of consensus and flawed benchmarks. There is no universal agreement on the best metric.1617 Researchers have also questioned the benchmarks and evaluation procedures used in published comparisons.181920
The consequences of a poor metric can extend beyond an uninformative comparison. In medical diagnosis6 or industrial safety4, a metric that hides class imbalance or overlooks a relevant anomaly type can make a system appear ready for deployment even when it misses critical events. The demand for explainability910 raises a related question. To assess a detector, it helps to know why it flagged a particular observation.
Imbalance, variation in anomaly type, and domain-specific costs make a single best metric unlikely. AUPRC emphasizes a rare positive class2122, MCC measures agreement across both classes2324, and range-based metrics account for the duration of events in a time series1619. No one measure captures robustness to imbalance, sensitivity to particular errors, and temporal structure equally well. A multi-metric approach can give a fuller picture of performance.1125
Critiques of benchmarks and metrics181920 have prompted a closer look at established evaluation practices. Several studies1819 show how commonly reported scores, including pointwise F1 in time series, can create an illusion of progress. This has encouraged work on alternatives such as range-based metrics1619, affiliation metrics26, and PRO-AUC27.
#2. Basic evaluation metrics
Many classification metrics are calculated from the confusion matrix. Comparing the true class and predicted class gives four possible outcomes.28
- True positive (TP). The detector correctly flags an actual anomaly.
- False positive (FP). The detector flags a normal observation as anomalous, a type I error.
- False negative (FN). The detector labels an actual anomaly as normal, a type II error.
- True negative (TN). The detector correctly labels a normal observation.
These four counts define the following basic metrics.
#2.1. Precision, recall, and F1
-
Precision asks what fraction of alerts correspond to actual anomalies:
Precision = TP / (TP + FP). It matters when false alarms are costly, for example when repeated alerts cause alarm fatigue.29 Precision does not count missed anomalies. -
Recall, also called sensitivity or the true positive rate, asks what fraction of actual anomalies were detected:
Recall = TP / (TP + FN). It matters when missing fraud or disease has a high cost.6 Recall does not count false alarms. -
F1 is the harmonic mean of precision and recall:
F1 = 2 * (Precision * Recall) / (Precision + Recall). The score is widely reported, but it depends on the classification threshold30 and can give an incomplete account under severe imbalance.2431 The generalized F-beta score weights precision and recall differently:F_beta = (1 + beta^2) * (Precision * Recall) / ((beta^2 * Precision) + Recall).
Whether to emphasize precision, recall, or their balance in F1 depends on the relative consequences of false positives and false negatives. Precision includes false positives in its denominator, TP + FP. Recall instead includes false negatives in TP + FN. A system intended to minimize false alarms needs high precision29, while one intended to catch every critical event needs high recall.6 Choosing between them requires understanding what happens downstream after each kind of error.
#2.2. Area under the ROC curve (AUROC or AUC)
- Receiver operating characteristic (ROC) curve. As the classification threshold varies, the curve plots the true positive rate (recall) against the false positive rate,
FPR = FP / (FP + TN). - AUROC is the area under this curve. It ranges from 0 to 1, with higher values indicating better discrimination.
- Interpretation. AUROC can be read as the probability that a randomly selected anomaly receives a higher score than a randomly selected normal sample, with half credit for ties.21
- Strength. It measures the detector’s overall ability to distinguish the two classes without committing to one threshold.
- Limitation. When normal samples dominate, a low false positive rate can still yield many false alarms. A high AUROC can therefore coexist with poor performance on the minority class at the chosen operating point.2122
AUROC is useful for comparing ranking ability across thresholds, but its use of FPR can hide poor performance on rare anomalies.21 Consider FPR = FP / (FP + TN). With many true negatives, the detector can produce many false positives while FPR remains small. Precision, TP / (TP + FP), can then be low even for a model with high AUROC. Good ranking over the full ROC curve need not translate into practical usefulness when searching for rare events.2122
#3. Metrics for class imbalance
Anomaly detection datasets are inherently imbalanced, so measures such as accuracy can be unreliable.1113 The following metrics address different aspects of that problem.
#3.1. Area under the precision–recall curve (AUPRC or PR-AUC)
-
Precision–recall (PR) curve. This plots precision on the vertical axis against recall on the horizontal axis as the classification threshold changes.
-
AUPRC is the area under the PR curve. Values range from 0 to 1, and higher is better. Average precision (AP) is a closely related summary widely used in object detection3233 and information retrieval. AP and AUPRC are sometimes used interchangeably, although the numerical result depends on how the curve is integrated.
-
Rationale. AUPRC focuses on the positive class, here the rare anomalies. It can give a more informative view of alert quality than AUROC when true negatives dominate the dataset.2122 The no-skill precision baseline is the prevalence of positives.
-
The AUPRC–AUROC debate
- Common claim. Many publications recommend AUPRC over AUROC for imbalanced datasets.2234
- Counterarguments and qualifications from recent work2135
- AUPRC is sensitive to the fraction of outliers. AUROC is more stable when class proportions change but the within-class score distributions stay fixed.12
- AUROC weights false-positive contributions uniformly. At a threshold τ, AUPRC gives a false-positive contribution a weight proportional to the reciprocal of the probability that the model assigns a score above τ. Errors near the top of the ranking therefore carry more weight.
- AUROC rewards comparable ranking improvements uniformly across positive examples. AUPRC gives more weight to improvements among high-scoring examples than among low-scoring ones.
- The claim that imbalance alone favors AUPRC is often made with little supporting evidence or with citations that do not establish it.
- AUPRC can amplify algorithmic bias against subgroups whose members tend to receive lower scores.
-
Implication. The right choice depends on whether high-confidence errors deserve priority, whether improvements should count uniformly, and whether the weighting has fairness consequences. AUPRC is useful when positive-class performance is paramount, as in retrieval and rare-event detection.21 AUROC can be appropriate when error costs are more balanced or improvements should receive uniform weight.
#3.2. Matthews correlation coefficient (MCC)
- Definition. MCC measures the correlation between the true binary labels and the predicted binary labels. It uses all four entries in the confusion matrix:
MCC = (TP * TN - FP * FN) / sqrt((TP + FP) * (TP + FN) * (TN + FP) * (TN + FN)).2324 - Range and interpretation. MCC ranges from −1 to +1. A value of +1 indicates perfect agreement. Zero indicates no correlation, as expected for random predictions. A value of −1 indicates complete disagreement.
- Main strength. To obtain a high MCC, a detector has to perform well across both classes. True and false decisions about each class enter the calculation in relation to the class totals. This can make MCC more informative than accuracy or F1 on imbalanced datasets.232436
- Relevance to anomaly detection. The imbalance inherent in AD makes MCC useful2336, and an increasing number of studies recommend reporting it.2437
- Limitations. The coefficient can be undefined if an entire row or column of the confusion matrix is zero.24 Its denominator also requires a count of true negatives, which can be difficult to define in an open-world task such as object detection.
#3.3. Other metrics
- Balanced accuracy averages TPR and the true negative rate,
TNR = TN / (TN + FP). It gives the two class-specific rates equal weight.13 - Average precision (AP) summarizes a PR curve and is particularly common in detection tasks.3233
The choice between AUPRC and AUROC hinges on the performance property the task values. Is the priority to find any anomaly, or to make the highest-confidence alerts accurate? AUPRC’s emphasis on high scores suits retrieval tasks in which the top-ranked results matter most.21 AUROC’s uniform treatment of positive–negative ranking errors can be appropriate when each such error matters equally.21 “The data are imbalanced” often stands in for “the rare positive class matters.” Even then, the relative costs of errors at different ranks remain an open question. AUPRC may obscure weaknesses among low-scoring positives or amplify subgroup bias.21
MCC offers another strong candidate when a single number is needed. Its use of the entire confusion matrix gives a more balanced view than F1 or accuracy alone.2324 Several studies explicitly recommend it for imbalanced settings.2436 The formula incorporates TP, TN, FP, and FN symmetrically. High recall with poor precision, or high accuracy from always predicting the majority class, therefore cannot by themselves establish good performance. The correlation measures agreement between predictions and labels while accounting for their marginal totals, which can make it less dominated by class proportions than accuracy or F1.24
AUPRC, MCC, and balanced accuracy each address a different weakness of ordinary classification metrics. Their development and continuing debate112313242122 reflect the shortcomings of measures such as accuracy under imbalance.13 AUPRC omits true negatives, while MCC balances information from all four cells. Evaluating an anomaly detector therefore requires deciding which property matters for the task.
#4. Trends and criticisms in anomaly detection evaluation
#4.1. Metrics in common use
Across the papers and benchmarks reviewed, precision, recall, F1, and AUROC remain widely reported despite their known limitations.1314 AUPRC also appears frequently, especially in recent work and on imbalanced datasets.62122 MCC is less common, though it is receiving more attention and explicit recommendations.232436 Image AD benchmarks commonly report pixel AUROC, image AUROC, and PRO-AUC.383927 For time series, researchers also use range-based metrics161926 and the NAB score4020.
As discussed in Section 3, recent critical analyses2135 have qualified the case for AUPRC. Nevertheless, a general preference for it on imbalanced datasets remains widespread, including in high-stakes domains such as medicine.4134
#4.2. Problems with current evaluation practice
- Flawed metrics. Pointwise precision, recall, and F1 can misrepresent range-based anomalies in time series.1819 Even some range-based metrics can be gamed or award too much credit. Point-adjusted F1 can credit a whole event for one detected point19, and some measures reward inaccurate or incomplete detections.26 In images, large regions can dominate pixel-level metrics.27
- Inconsistent benchmarking. Different datasets and evaluation protocols make fair comparisons across studies difficult.4217 ADGym43, IM-IAD44, TimeSeriesBench45, and ADer46 are efforts to address this.
- Problems with benchmark datasets. Some datasets have trivial anomalies, incorrect annotations, or unrealistic anomaly densities.3920 A shortage of real-world data and privacy constraints present further obstacles.2047
- Excessive emphasis on complexity. Under rigorous evaluation, a new deep learning model does not always outperform a simpler baseline.1920 Apparent progress can therefore be misleading.
- Threshold dependence. Measures such as F1 require a classification threshold. The chosen threshold can change the score even when the underlying ranking stays the same.30
Practice can lag well behind a documented methodological problem. Studies continue to report pointwise scores on time series1819 and AUROC alone on imbalanced data2122 despite extensive discussion of their limitations. Research conventions, the cost of switching metrics, and limited awareness or agreement about alternatives may all contribute. The persistence of the blanket recommendation to use AUPRC for imbalance is another example.2234
The move toward standardized benchmarks46444543 responds directly to the difficulty of comparing inconsistent results. IM-IAD44, TimeSeriesBench45, and ADer46 aim to provide consistent settings, a broader collection of metrics, and fairer comparisons. Standardization makes it easier to tell whether a reported advance survives a common protocol.
Findings that simple models can match more complex ones1920 illustrate how evaluation can guide research priorities. Under rigorous benchmarks19, autoregressive models or linear mappings can be surprisingly effective on certain tasks, including point anomalies. Earlier evaluations may have inflated the apparent advantage of more complex methods. A fair comparison can direct attention to simpler, more interpretable approaches when they perform well.
#5. Evaluating image anomaly detection
Image AD is used in industrial inspection, for example to detect surface defects45, and in medical imaging, for example to detect lesions67. Common benchmark datasets include MVTec AD3948, VisA49, MVTec-3D49, and BTAD49.
#5.1. Levels of evaluation
Image anomaly detection can be evaluated at three levels.
- Image level. The model produces a single score for each image and classifies the entire image as normal or anomalous. Typical metrics include image-level AUROC (I-AUC)5051 and precision, recall, F1, accuracy, or MCC calculated from the image labels.
- Pixel level (localization or segmentation). The model produces an anomaly map or heatmap indicating the affected pixels or regions. Evaluation requires precise pixel-level ground-truth annotations.3852
- Object level. Detections are assessed against anomalous objects or structures in the image. This is less common than pixel-level evaluation, but it is useful when the anomaly has a distinct object boundary.53
#5.2. Key pixel-level metrics
- Pixel AUROC (P-AUC) applies standard AUROC to pixel predictions.385051 It is widely reported. Large anomalous regions contribute many positive pixels, while the many normal pixels can dilute the false positive rate. A good score can conceal poor detection of small defects.5227
- Per-region overlap AUC (PRO-AUC or AUPRO) computes overlap separately for each annotated anomalous region and averages the results. Each region receives the same weight regardless of its size.2754 This addresses the size bias of pixel AUROC. Studies often report both P-AUC and PRO-AUC.55
- Dice similarity coefficient (DSC) / F1 measures overlap between the predicted and annotated masks.3852 Often computed at an optimal threshold, DSC can be sensitive to how that threshold is chosen. Variants include soft Dice. Pixel-level F1 is also used.50
- Intersection over union (IoU) is another common measure of segmentation overlap.38
- Average precision (AP) and mean average precision (mAP) are familiar from object detection.3233565758 AP derived from a pixel-level PR curve also measures anomaly localization38 and is sometimes preferred to pixel AUROC when anomalous pixels are rare.5960 Depending on the evaluation protocol, mAP averages AP across classes or IoU thresholds.33
Detecting that an image is anomalous and locating the anomaly within it are different tasks. Image AUROC tells us whether anomalous images generally receive higher scores.50 It does not show whether the detector found the defect. P-AUC, PRO-AUC, DSC, and IoU assess localization.385227 Many papers accordingly report both levels of evaluation.5051 A model may rank images well while placing its heatmap in the wrong area, or localize selected anomalies well while classifying whole images poorly.
The development of PRO-AUC27 illustrates why a localization metric may need to account for region size. Pixel AUROC treats each pixel as one observation.52 A large, easy defect can contribute many correctly ranked pixels and raise the score even when the model completely misses several small defects. PRO-AUC was designed to address this problem.27 Averaging overlap by region gives a small defect as much influence as a large one.
Using AP or mAP for pixel localization connects object detection and information retrieval3233 to image AD. The detector ranks candidate items, which may be objects, documents, or pixels. Pixel-level AP38 asks how effectively a detector retrieves anomalous pixels from a much larger field of normal ones. As with finding relevant documents in a large collection, a PR curve can be particularly informative when relevant items are scarce.5960
The following table summarizes the main metrics used in image anomaly detection.
Table 1. Key metrics for image anomaly detection
| Metric | Level | Definition | Main focus |
|---|---|---|---|
| Image AUROC (I-AUC) | Image | AUROC over scores for entire images. | Discrimination between normal and anomalous images. |
| Pixel AUROC (P-AUC) | Pixel | AUROC over pixel scores. | Ranking anomalous pixels for localization. |
| PRO-AUC (AUPRO) | Pixel | Aggregate overlap for each ground-truth region and integrate over operating points. | Localization with equal weight for regions of different sizes. |
| Pixel F1 / DSC | Pixel | Overlap between predicted and true masks expressed as F1 or Dice. | Pixel-level segmentation overlap. |
| Pixel AP / mAP | Pixel | Summarize the pixel PR curve or average AP over classes or IoU thresholds, as specified. | Ranking and retrieval of anomalous pixels. |
#6. Evaluating time-series anomaly detection
Time-series AD deals with sequential observations such as sensor readings, financial data, system logs, and network traffic.1461
#6.1. Challenges specific to time series
- Temporal dependence. An observation may look ordinary on its own and become anomalous in light of its history. Standard evaluation often treats samples as independent and identically distributed.1419
- Range-based anomalies. An anomaly often extends across a contiguous interval, or event, rather than a single timestamp.1619
- Timeliness. Detecting an event early often matters.4020
#6.2. Evaluating points and evaluating ranges
- Pointwise metrics evaluate timestamps individually. Examples include standard precision, recall, F1, and pointwise AUROC.
- Limitations. These metrics ignore temporal correlations and can penalize a small shift in timing. A long detected event can contribute many true positives and dominate the result, giving a misleading impression of how many events the system detects.1819
#6.3. Metrics for temporal anomalies
Several metrics have been developed to address the limits of pointwise evaluation.
-
Point-adjusted F1 (PA-F1). If a prediction hits even one point in a labeled anomalous interval, the entire interval is counted as detected for recall.19 The intention is to give partial detections credit. Criticism: a single hit can receive full credit even when the estimated range is inaccurate or incomplete. This can inflate performance and make the metric easy to game.1926
-
Range-based precision, recall, and F1 evaluate overlap or other relationships between predicted and labeled intervals. Implementations differ in how they define a successful detection.
- Tatbul et al. (range-based precision and recall). This NeurIPS 2018 proposal6263 allows the user to specify how much credit an event receives for being detected, how much of it is covered, and which positions matter. Weights and functions such as α, β, and δ make the framework configurable.
- Hwang et al. (time-series-aware precision and recall).64 A prediction must cover at least a specified fraction θ of an anomalous interval to count as a detection. This rule is simpler than the general range-based framework.
- TaPR (time-series-aware precision and recall).64 TaPR treats each attack or anomalous event equally, which can be limiting when events should carry different weights. A later version64 accounts for the interval actually affected by the event.
- PATE (Proximity-Aware Time series anomaly Evaluation).6540 PATE defines buffer zones around anomalous intervals and computes a weighted AUPRC. How close a detection lies to an event affects the credit it receives.
-
Affiliation metrics. Introduced at KDD 202226, these extend precision and recall using a zone of affiliation around each true event. They require no user-selected parameters, are interpretable, apply to both points and ranges, and were designed to resist strategies that inflate scores.
-
NAB score. The Numenta Anomaly Benchmark4020 evaluates real-time streaming detection. Anomaly windows reward early detections. Application profiles give false alarms and missed events different penalties. The scale uses 0 for a null detector and 100 for a perfect detector; scores below the null baseline are possible.
-
Mean time to detection (MTTD) measures how long detection takes, a practical concern in real-time systems.40
The shift from points to ranges reflects the importance of event duration and temporal context. Critiques of pointwise metrics1819 and the development of range-based alternatives1662642665 show what is lost when timestamps are treated as independent events. Two systems can flag a similar number of points while differing substantially in whether they found the events and when they raised the alarm.
Range-based approaches still differ in how they reward partial detections and tolerate timing errors. Consider PA-F119, range-based precision and recall62, affiliation26, and PATE65. PA-F1 is simple, but critics find it too generous.1926 Range-based precision and recall allow customization at the cost of additional parameters.62 Affiliation aims for robustness without parameters.26 PATE adds proximity weighting.65 There is still no consensus on how to compress the quality of a range detection into a single number, and new metrics continue to appear.642665
The NAB benchmark and score4020 specifically address streaming requirements such as timeliness and continuous adaptation. Batch summaries such as AUROC, AUPRC, and standard range-based F1 often overlook these requirements. NAB explicitly rewards an early alert inside an anomaly window and penalizes false alarms in the stream.20 A retrospective score for a complete dataset does not, by itself, establish the performance of an online monitor as data arrive.
The following table summarizes commonly used time-series metrics.
Table 2. Key metrics for time-series anomaly detection
| Metric | Evaluation unit | Main feature |
|---|---|---|
| Pointwise F1 | Point | Scores each timestamp independently and ignores event extent. |
| Point-adjusted F1 (PA-F1) | Event/range | Credits the entire labeled interval if a single point is detected. |
| Range-based P/R/F1 | Event/range | Measures range overlap with configurable credit for existence, size, and position. |
| TS-aware P/R | Event/range | Counts a detection only if the prediction covers a specified fraction θ of the anomalous interval. |
| TaPR | Event/range | Time-series-aware precision and recall. A later version accounts for the interval actually affected. |
| Affiliation P/R | Event/range | Extends precision and recall using affiliation zones around true events. No tunable parameters. |
| PATE | Event/range | Calculates weighted AUPRC using proximity to events and their buffer zones. |
| NAB score | Stream | Rewards early detection in real time and uses profile-specific penalties for false alarms and misses. |
#7. Synthesis and recommendations
Evaluation measures range from basic classification metrics to measures tailored to class imbalance, image regions, or intervals in time. A useful evaluation starts with the basic measures and then accounts for the structure of the observations and the decision the detector must support.
The table brings together the main metrics discussed in this report.
Table 3. Summary of anomaly detection metrics
| Metric | Definition | Main use | Strength | Limitation |
|---|---|---|---|---|
| Precision | Fraction of alerts that are actual anomalies | Costly false positives, such as alarm fatigue | Directly measures alert quality | Ignores false negatives |
| Recall | Fraction of actual anomalies detected | Costly misses, such as critical events | Measures coverage of true anomalies | Ignores false positives |
| F1 | Harmonic mean of precision and recall | Balancing precision and recall | Combines them in one score | Threshold-dependent and can mislead under imbalance |
| AUROC | Area under the ROC curve (TPR against FPR) | Comparing general discrimination | Evaluates ranking across thresholds | Severe imbalance can hide poor alert quality |
| AUPRC (PR-AUC) | Area under the PR curve (precision against recall) | Imbalanced data and rare events | Focuses on the minority positive class | Changes with prevalence and can amplify subgroup bias |
| MCC | Correlation of observed and predicted labels using TP, TN, FP, and FN | Balanced assessment under imbalance | Uses all four confusion-matrix counts and accounts for class totals | Requires TN and can be undefined |
| Image AUROC | AUROC over image scores | Classifying whole images as normal or anomalous | Summarizes image-level detection | Provides no localization information |
| Pixel AUROC | AUROC over pixel scores | Localizing anomalies at pixel level | Measures discrimination among pixels | Large regions can dominate the score |
| PRO-AUC | AUC based on per-region overlap | Localization with equal weight for each region | Gives small defects a measurable contribution | More complex to compute |
| PA-F1 | Point-adjusted F1 for time series | Simple range-based evaluation | Recognizes a partial hit | Can overestimate performance and is easy to game |
| Range-based P/R/F1 (including range and affiliation measures) | Evaluates relationships between predicted and actual intervals | Time-series event detection | Accounts for temporal structure | More complex to implement and interpret, with no consensus on one measure |
| NAB score | Framework for real-time streaming evaluation | Streaming anomaly detection | Rewards early alerts and reflects real-time conditions | Tied to a particular benchmark design |
Choosing evaluation metrics
- Data type. For images, consider pixel AUROC and PRO-AUC. For time series, consider range-based P/R/F1, affiliation, or NAB, as the setting warrants.
- Task. Detecting whether an anomaly is present can be evaluated with image AUROC or standard precision, recall, F1, MCC, and AUPRC. For localization, use pixel AUROC, PRO-AUC, pixel F1, or pixel AP. For real-time streaming, consider NAB and MTTD.
- Class imbalance. If anomalies are rare and their detection is the priority, consider AUPRC, MCC, balanced accuracy, or a task-specific metric alongside accuracy, F1, and AUROC. Choose between AUPRC and AUROC with their different error weightings and the intended use in mind.2122
- Anomaly characteristics. In time series, distinguish isolated points from extended events. In images, decide how much weight small defects should receive relative to large ones.
- Error costs. Emphasize the measure sensitive to the more costly error: precision for false alarms and recall for missed events. F-beta permits an explicit weighting.
- Multiple metrics. Reporting complementary measures, such as AUROC with AUPRC21 or I-AUC with PRO-AUC27, gives a fuller account of performance than a single value.1125
Limitations and future directions. Researchers continue to develop better metrics2665 and call for more reliable benchmarks.454317 In practice, a numerical score also has to be considered alongside explainability910 and computational efficiency.
Conclusion. To evaluate an anomaly detector, connect the metric to the data type, task, prevalence, structure of anomalies, and cost of errors. Standardized benchmarks and new evaluation methods help make progress measurable. Researchers and practitioners can choose an appropriate strategy by examining where established metrics fail and testing whether the alternatives address those failures in their own setting.
#References
#Footnotes
-
Evaluating Deep Learning Algorithms for Log-Based Anomaly Detection - Journal of Information Systems Engineering and Management ↩ ↩2
-
A Survey of Anomaly Detection in Cyber-Physical Systems - arXiv ↩
-
Anomaly Detection for Industrial Applications, Its Challenges, Solutions, and Future Directions: A Review - arXiv ↩ ↩2 ↩3
-
Unsupervised Anomaly Detection on Metal Surfaces Based on Frequency Domain Information Fusion - MDPI ↩ ↩2
-
Contrastive Learning-Based Anomaly Detection for Actual Corporate Environments - PMC ↩ ↩2 ↩3 ↩4 ↩5 ↩6
-
Beyond Heatmaps: A Comparative Analysis of Metrics for Anomaly Localization in Medical Images - OpenReview ↩ ↩2
-
On the Influence of Data Resampling for Deep Learning-Based Log Anomaly Detection: Insights and Recommendations - IEEE Computer Society ↩
-
[2302.06670] Explainable Anomaly Detection in Images and Videos: A Survey - arXiv ↩ ↩2 ↩3 ↩4
-
Evaluating Anomaly Detection Algorithms: A Multi-Metric Analysis Across Variable Class Imbalances - Lamarr Institute ↩ ↩2 ↩3 ↩4 ↩5
-
Exploring the Impact of Outlier Variability on Anomaly Detection Evaluation Metrics - arXiv ↩ ↩2
-
Machine Learning Evaluation of Imbalanced Health Data: A Comparative Analysis of Balanced Accuracy, MCC, and F1 Score | Request PDF - ResearchGate ↩ ↩2 ↩3 ↩4 ↩5
-
Deep Learning for Time Series Anomaly Detection: A Survey - arXiv ↩ ↩2 ↩3 ↩4 ↩5
-
A Comprehensive Survey on Graph Anomaly Detection with Deep Learning - arXiv ↩ ↩2
-
Precision and Recall for Range-Based Anomaly Detection - mlsys.org ↩ ↩2 ↩3 ↩4 ↩5 ↩6
-
(PDF) “Do you know existing accuracy metrics overrate time-series anomaly detections?” - ResearchGate ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7
-
Anomaly Detection in Time Series: A Comprehensive Evaluation - VLDB Endowment ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20
-
The Numenta Anomaly Benchmark - numenta.com ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11
-
A Closer Look at AUROC and AUPRC under Class Imbalance - arXiv ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17
-
A Closer Look at AUROC and AUPRC under Class Imbalance - arXiv (Note: Duplicate arXiv ID) ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10
-
Evaluation metrics: leave your comfort zone and try MCC and Brier Score - Towards Data Science ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7
-
The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation - OUCI ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11
-
A Study on Performance Metrics for Anomaly Detection Based on Industrial Control System Operation Data - MDPI ↩ ↩2
-
Local Evaluation of Time Series Anomaly Detection Algorithms - arXiv ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11
-
PRO scores of various methods on MVTec dataset. - ResearchGate ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9
-
What metrics are used for anomaly detection performance? - Milvus Blog ↩
-
Enhancing Network Anomaly Detection Using Graph Neural Networks - OSTI ↩ ↩2
-
Evaluation of Machine Learning Algorithms for Supervised Anomaly Detection and Comparison between Static and Dynamic Thresholds - Semantic Scholar ↩ ↩2
-
Apart from F-1 measure, which other metrics can one use to evaluate binary rating data? - ResearchGate ↩
-
Mean Precision in Object Detection and Computer Vision - Keylabs ↩ ↩2 ↩3 ↩4
-
Mean Average Precision (mAP) Explained: Everything You Need to Know - V7 Labs ↩ ↩2 ↩3 ↩4 ↩5
-
Imbalanced Data? Stop Using ROC-AUC and Use AUPRC Instead | Towards Data Science ↩ ↩2 ↩3
-
A Closer Look at AUROC and AUPRC under Class Imbalance - arXiv (Note: Duplicate arXiv ID) ↩ ↩2
-
[Q] Precision, recall and Matthews Correlation Coefficient for anomaly detection - Reddit ↩ ↩2 ↩3 ↩4
-
Matthews Correlation Coefficient - why not in wider use? [Discussion] : r/statistics - Reddit ↩
-
Metrics — Anomalib documentation - Read the Docs ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8
-
MVTecAD (MVTEC ANOMALY DETECTION DATASET) - Papers With Code ↩ ↩2 ↩3
-
PATE: Proximity-Aware Time series anomaly Evaluation - arXiv (Note: Duplicate arXiv ID, likely refers to the same paper as 64) ↩ ↩2 ↩3 ↩4 ↩5 ↩6
-
Deep Learning for Anomaly Detection: A Review - ResearchGate ↩
-
ADGym: Design Choices for Deep Anomaly Detection - proceedings.neurips.cc ↩ ↩2 ↩3
-
IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturing - arXiv (Note: Duplicate arXiv ID) ↩ ↩2 ↩3
-
TimeSeriesBench: An Industrial-Grade Benchmark for Time Series Anomaly Detection Models - arXiv ↩ ↩2 ↩3 ↩4
-
ADer: A Comprehensive Benchmark for Multi-class Visual Anomaly Detection - arXiv (Note: Duplicate arXiv ID) ↩ ↩2 ↩3
-
Is there any real world public dataset of anomaly detection example? - HTM Forum ↩
-
Superpixel-based Anomaly Detection for Irregular Textures with a Focus on Pixel-level Accuracy - Toby Breckon ↩
-
Deep Industrial Image Anomaly Detection: A Survey - arXiv (Note: Duplicate arXiv ID) ↩ ↩2 ↩3
-
Image-level and pixel-level anomaly detection results with teacher network using pretraining and nonpretraining. - ResearchGate ↩ ↩2 ↩3 ↩4 ↩5
-
Image-and Pixel-level AUROC performance comparison of anomaly detection… | Download Scientific Diagram - ResearchGate ↩ ↩2 ↩3
-
Anomaly Detection Using Computer Vision: A Comparative Analysis of Class Distinction and Performance Metrics - arXiv ↩ ↩2 ↩3 ↩4 ↩5
-
Advancing Anomaly Detection: The IDW dataset and MC algorithm - BMVA Archive ↩
-
Applying Machine Learning Techniques for Anomaly Detection in Wooden Plank Images - Diva Portal ↩
-
mAP or “mean Average Precision”, a key AI metric - Innovatiana ↩
-
Mean Average Precision Made Simple With A Complete Guide - Spot Intelligence ↩
-
Mean Average Precision (mAP) in Object Detection : r/learnmachinelearning - Reddit ↩
-
How is anomaly detection evaluated? - Zilliz Vector Database ↩ ↩2
-
Creating a time series anomaly prediction experiment (Watson Machine Learning) - IBM ↩
-
Precision and Recall for Time Series - NIPS papers ↩ ↩2 ↩3 ↩4
-
Reviews: Precision and Recall for Time Series - NIPS papers ↩
-
navigating the metric maze:ataxonomy of evaluation metrics for anomaly detection in time series - arXiv ↩ ↩2 ↩3 ↩4 ↩5 ↩6
-
Evaluating Time Series Anomaly Detection: Proximity-Aware Time Series Anomaly Evaluation (PATE) - MarkTechPost ↩ ↩2 ↩3 ↩4 ↩5 ↩6