Data Engineering & Delta Lake - Databricks Community

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Community Leaders

Sample_Community_Lea 22

Sample_Member_Analys 12

Sample_Member_Admin 12

Sample_Member_Ops 12

Sample_Product_Exper 12

Recent Discussions

How are teams testing declarative pipeline changes before promoting to production?
Our pipeline definitions are moving toward a declarative model, but our current tests mostly validate notebooks in isolation. I would like a promotion workflow that catches schema drift, expectation failures, and unintended full refreshes before a production run. What does a practical test environment and release gate look…
Best pattern for handling late-arriving CDC events in a multi-table Delta pipeline?
We ingest change events from several operational systems, and a small percentage arrive hours after the normal watermark. The pipeline needs to keep customer and order tables consistent without replaying an entire day. How are teams combining watermarks, MERGE logic, and idempotency when related events arrive out of order?…