Please use this identifier to cite or link to this item:
http://hdl.handle.net/1942/43437
Title: | Data Ingestion Validation Through Stable Conditional Metrics with Ranking and Filtering | Authors: | BYLOIS, Niels NEVEN, Frank VANSUMMEREN, Stijn |
Issue Date: | 2024 | Publisher: | SPRINGER | Source: | Information systems frontiers (Print), | Status: | Early view | Abstract: | We introduce an advanced method for validating data quality, which is crucial for ensuring reliable analytics insights. Traditional data quality validation relies on data unit tests, which use global metrics to determine if data quality falls within expected ranges. Unfortunately, these existing approaches suffer from two limitations. Firstly, they offer only coarse-grained assessments, missing fine-grained errors. Secondly, they fail to pinpoint the specific data causing test failures. To address these issues, we propose a novel approach using conditional metrics, enabling more detailed analysis than global metrics. Our method involves two stages: unit test discovery and monitoring/error identification. In the discovery phase, we derive conditional metric-based unit tests from historical data, focusing on stability to select appropriate metrics. The monitoring phase involves using these tests for new data batches, with conditional metrics helping us identify potential errors. We validate the effectiveness of this approach using two datasets and seven synthetic error scenarios, showing significant improvements over global metrics and promising results in fine-grained error detection for data ingestion validation. | Notes: | Bylois, N (corresponding author), Hasselt Univ, Data Sci Inst, Agoralaan Bldg D, B-3590 Diepenbeek, Belgium. niels.bylois@uhasselt.be |
Keywords: | Data monitoring;Data profiling;Dynamic data;Data unit tests | Document URI: | http://hdl.handle.net/1942/43437 | ISSN: | 1387-3326 | e-ISSN: | 1572-9419 | ISI #: | 001262877700001 | Rights: | The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024 | Category: | A1 | Type: | Journal Contribution |
Appears in Collections: | Research publications |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
s10796-024-10504-y.pdf Restricted Access | Early view | 2.59 MB | Adobe PDF | View/Open Request a copy |
Google ScholarTM
Check
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.