The Quality Rule Maker
Description
This ETL workflow dynamically ingests metadata to identify the column representing business rules, then extracts and filters non-empty business rules from table columns. It enriches these rules by joining with related schema, table, and connection metadata, aggregating column names per table to provide context. The enriched dataset is validated for completeness and persisted as a structured table. Finally, the workflow transforms the enriched business rules into a data quality rule recommendation dataset, including rule definitions and validation queries, and persists this for downstream data governance and quality enforcement. The process includes error handling, schema enforcement, and incremental data validation to ensure data integrity and operational reliability.
Details
Lineage Depth and Coverage
Description
This recipe assesses the depth and completeness of column-level lineage recorded in the catalog for every active table. It classifies each asset by lineage presence, measures lineage coverage across active columns, and traces upstream dependencies to determine how deeply data can be reliably sourced. The recipe identifies orphan and low-coverage assets that require lineage remediation and provides a foundation for future impact-analysis capabilities.Details
Most tables in the catalog appear to have lineage, but it can be difficult to determine how complete or how deeply connected that lineage actually is. This recipe provides visibility into lineage coverage, identifies orphan assets, and determines how many upstream hops can be reliably traced.
- Surfaces orphan assets with no lineage that were previously difficult to identify.
- Quantifies column-level lineage coverage instead of relying on a simple connected/not-connected indicator.
- Helps prioritize lineage remediation based on measurable coverage and depth.
- Improves confidence in downstream impact analysis and data traceability.
- Data governance teams tracking lineage completeness across connected systems.
- Data engineers identifying and prioritizing lineage remediation work.
- Data stewards validating the traceability of critical assets.
- Analysts who depend on reliable lineage for impact analysis and data discovery.
How the recipe works
Identify Active Assets
Retrieves active tables and their associated column-level metadata from the catalog.
Assess Lineage
Determines whether each asset has upstream or downstream lineage and classifies its lineage status.
Measure Coverage
Calculates the percentage of active columns participating in lineage relationships.
Calculate Depth
Traverses upstream lineage for multiple hops while safely handling circular lineage relationships.
Recipe inputs & outputs
Inputs
- Active catalog tables
- Column-level metadata
- Lineage relationships
- Object and column identifiers
Outputs
- Lineage connectivity classification
- Column-level lineage coverage
- Upstream lineage depth
- Orphan and low-coverage asset indicators
Expected outcome
The recipe provides an asset-level view of lineage health, making it easier to identify gaps, prioritize remediation, and establish a reliable foundation for future lineage-driven impact analysis.