Others • Others

Wide Table Risk and Sparsity Assessment

Description

This recipe evaluates the readiness and health of datasets before they enter any governance workflow. It identifies tables that are too wide, sparsely populated, missing profiling, or containing inconsistent metadata. The recipe classifies each table into High, Medium, Low, or Profile-Required risk priority and recommends appropriate next steps such as profiling, cleanup, or governance onboarding. It provides a reliable, objective way to determine which datasets are worth curating and which require remediation, helping teams avoid wasted effort and focus on high-value assets.

Details

Wide Table Health and Readiness Analyzer Based on Sparsity

Early diagnostic to identify wide, sparse, unprofiled, or low-quality tables before governance work begins.

Why this recipe matters

Large, wide tables often contain many empty or unused columns, missing profiling, stale attributes, and technical leftovers. Governing such tables leads to wasted effort, slow profiling, and misleading business interpretations. This recipe surfaces sparsity, profiling gaps, and structural issues early—showing whether a table is even ready for governance.

Business value

Prevents teams from governing tables that contain mostly empty or unused data.
Detects profiling gaps and inconsistencies before stewardship begins.
Improves prioritization by focusing governance on meaningful datasets.
Supports cleanup of legacy, deprecated, or dead attributes.
Enhances user trust by increasing metadata reliability.

Who benefits

Data Governance Teams
Data Stewards
Platform & Profiling Admins
Data Engineering Teams
Analytics & Domain Owners

What you receive

Column sparsity assessment

Profiling gap detection

Table-level health score

Risk classification (High/Medium/Low)

Governance readiness recommendations

How the recipe works (guided flow)

Step 1 — Read metadata from oecolumn, oetable, and oeschema
Load column details, table width, profiling values, and schema metadata.

Step 2 — Compute sparsity per column
Use the formula: (nulls + zeros + empty values) / rowcount to measure how much real data exists.

Step 3 — Detect profiling gaps and inconsistencies
Identify columns with missing profiling, zero rowcount, or inconsistent metrics.

Step 4 — Aggregate to table-level metrics
Compute width, sparse column count, % sparse, inconsistent profiling, and no-profile attributes. :contentReference[oaicite:5]{index=5}

Step 5 — Assign risk categories
PROFILE_REQUIRED / HIGH / MEDIUM / LOW based on sparsity and profiling completeness.

Step 6 — Generate recommendations
Create governance actions such as reprofiling, cleanup, or deprioritization.

Step 7 — Output results
Produce table_sparsity_summary and sample views for UI display.

Sample insights (illustrative)

Insight Category	What the recipe discovered	Business Impact
High-sparsity tables	Several tables contain more than 80% sparse columns.	Indicates poor data quality; governance should deprioritize these datasets.
Missing profiling	10% of columns have NULL rowcount or missing profiling stats.	Profiling must be rerun before assessing readiness or risk.
Inconsistent profiling	Several numeric columns show contradictory profiling indicators.	Signals ingestion or profiling anomalies that need admin attention.

Before you run this recipe

Make sure the following ingredients are available:

Connected datasets are crawled and profiled
Columns have distinct counts and top values generated

Back to Recipes