Pipeline observability : the three pillars (metrics, logs, lineage), alerting strategies, and building dashboards that catch failures before users do.
What to test in a data pipeline and when: a testing pyramid for data, schema and contract tests, runtime validation, and regression and anomaly detection.
How to partition and organize data for query performance : partition strategies, sort orders, Z-ordering, and common mistakes that kill performance.
Choose the right processing model : batch, micro-batch, or streaming. Decision framework based on latency requirements, cost, and operational complexity.
