Data Ingestion Validation through Stable Conditional Metrics with Ranking and Filtering

Abstract

We present a data ingestion quality validation approach using conditional metrics, a novel form of metrics that compute data quality metrics over specific parts of the ingestion data. We propose a method that automatically derives conditional metrics from historical ingestion sequences, using stability as a selection criterion for implementing these metrics as data unit tests. If an ingestion batch fails any unit tests, we show how conditional metrics can be utilized to identify potential errors. We show the effectiveness of our approach through an evaluation on a real world data set under various error scenarios.