A prioritized data engineering checklist covering pipeline design, data quality, idempotency, schema management, testing, and observability, with reasons.
Pipeline observability : the three pillars (metrics, logs, lineage), alerting strategies, and building dashboards that catch failures before users do.
What to test in a data pipeline and when: a testing pyramid for data, schema and contract tests, runtime validation, and regression and anomaly detection.
How to partition and organize data for query performance : partition strategies, sort orders, Z-ordering, and common mistakes that kill performance.
