Real-World Evaluation of AI-Assisted Readout of Skin Prick Automated Test Results

Abstract

To the Editor, Allergy diagnostics has taken an innovative turn with the introduction of the skin prick automated test (SPAT) [1, 2]. For respiratory allergy, skin prick testing (SPT) and serum specific IgE (sIgE) measurement are the gold standard methods to detect IgE-mediated sensitisation [3]. Despite its high sensitivity, manual SPT is subject to operator-dependent variability and prone to human errors. SPAT addresses these issues by standardizing the entire procedure, applying a fixed amount of allergen and a controlled prick force, in combination with AI-assisted readout to improve efficiency [1]. Previous research has demonstrated that SPAT, compared to manual SPT, shows lower intra-subject variability [1], higher consistency, and reduced patient discomfort [4]. When applying the validated 4.5-mm cutoff , SPAT shows equivalent diagnostic accuracy in detecting birch pollen and house dust mite allergies [2]. More recently, AI has been integrated into SPAT to provide wheal measurement suggestions, further increasing standardization and efficiency. Seys et al. described how the AI algorithm was trained (n = 651), validated (n = 217), and independently tested (n = 95) [5]. A strong correlation was observed between the physician-measured and the AI-measured longest wheal diameter (Pearson r = 0.83; p < 0.0001), with 5.8% of AI measurements adjusted by physicians, resulting in a change in test interpretation in 0.5% of cases. To evaluate external validity in clinical practice, real-world SPAT data from 37 hospitals across 5 European countries were assessed. This cohort comprises test results from 10,756 patients (126,526 pricks) with respiratory (96.7%), food (3.3%), insect venom (0.06%) and drug (0.01%) allergens. AI-measured longest wheal diameters were compared with physician-verified AI measurements, confirming a strong correlation (Pearson r = 0.96, p < 0.0001; Figure 1). No wheal was detected by AI in 7.9% (10,031) of pricks, with only 0.3% (380) of pricks corresponding to a physician measurement of ≥ 4.5 mm. In total, 6.1% of the wheals were enlarged by the physician (median (IQR): +1.0 mm (+0.5; +1.6)), and 2.9% reduced (median (IQR): −0.8 mm (−1.8; −0.4)), resulting in a change in test interpretation-from negative to positive or vice versa-in 1.7% and 0.4% of the cases, respectively. An analysis of potential confounders is provided in the Supporting Information (Table S3-S5 and Figure S3). This large real-world analysis demonstrates that the AI model is broadly generalizable and shows acceptable clinical performance. The low rate of interpretation-affecting adjustments is reassuring. The observed high correlation between AI and physician-verified AI measurements suggests that the AI model performs consistently across different hospitals. This is particularly relevant in comparison to manual SPT, where inter-and intra-observer variability has historically been a major limitation [6]. We observed that the total number of physician adjusted wheals (9.0%) in the current study was lower than the inter-observer readout variability (median: 19.8%) reported previously This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.