Mining Data Lineage Patterns Using Machine Learning to Predict Downstream Impact

Abstract

This study investigates the use of machine learning to mine data lineage patterns and predict downstream impact across complex enterprise data ecosystems. As organizations increasingly rely on interconnected data pipelines for analytics, reporting, and regulatory compliance, understanding how changes in upstream datasets affect downstream processes has become critical. Traditional lineage tracking methods often rely on manual documentation or static metadata, which fail to capture evolving pipeline behavior and hidden dependencies. This research proposes a machine learning–driven framework that analyzes historical lineage graphs, transformation logs, schema evolution patterns, and workload metadata to identify recurring dependency structures. By training predictive models on these lineage-derived features, the system forecasts potential downstream impacts resulting from schema changes, data quality anomalies, or pipeline modifications. Experimental evaluation on large-scale enterprise datasets demonstrates that the proposed approach achieves high accuracy in predicting affected downstream tables, workflows, and analytical outputs. The findings highlight the value of AI-enabled lineage intelligence for proactive risk mitigation, automated impact analysis, and improved data governance. This work contributes to the broader field of metadata analytics and presents a scalable, model-driven strategy for enhancing the reliability of modern data platforms