Please use this identifier to cite or link to this item:
http://hdl.handle.net/1942/42131
Title: | Data Ingestion Validation through Stable Conditional Metrics with Ranking and Filtering | Authors: | BYLOIS, Niels NEVEN, Frank VANSUMMEREN, Stijn |
Issue Date: | 2023 | Publisher: | Springer | Source: | Abelló, Alberto; Vassiliadis, Panos; Romero, Oscar; Wrembel, Robert (Ed.). Advances in Databases and Information Systems 27th European Conference, ADBIS 2023, Barcelona, Spain, September 4–7, 2023, Proceedings, Springer, p. 210 -223 | Series/Report: | Lecture notes in computer science | Series/Report no.: | 13985 | Abstract: | We present a data ingestion quality validation approach using conditional metrics, a novel form of metrics that compute data quality metrics over specific parts of the ingestion data. We propose a method that automatically derives conditional metrics from historical ingestion sequences, using stability as a selection criterion for implementing these metrics as data unit tests. If an ingestion batch fails any unit tests, we show how conditional metrics can be utilized to identify potential errors. We show the effectiveness of our approach through an evaluation on a real world data set under various error scenarios. | Keywords: | data cleaning;data profiling;dynamic data | Document URI: | http://hdl.handle.net/1942/42131 | ISBN: | 9783031429132 9783031429149 |
ISSN: | 0302-9743 | DOI: | 10.1007/978-3-031-42914-9_15 | Rights: | This version of the contribution has been accepted for publication, after peer review but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1007/978-3-031-42914-9_15. Use of this Accepted Version is subject to the publisher’s Accepted Manuscript terms of use https://www.springernature.com/gp/open-research/policies/accepted-manuscript-terms | Category: | C1 | Type: | Proceedings Paper |
Appears in Collections: | Research publications |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
main.pdf | Peer-reviewed author version | 349.9 kB | Adobe PDF | View/Open |
978-3-031-42914-9_15.pdf Restricted Access | Published version | 517.26 kB | Adobe PDF | View/Open Request a copy |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.