-
Strategies for reducing BI query cost without making dashboards feel stale?
We have a set of popular dashboards backed by complex views. Shorter refresh intervals improve trust, but repeated scans are expensive and some dimensions change only a few times per day. What combination of materialized views, result caching, incremental tables, and dashboard refresh policies has worked well without…
-
When does serverless SQL outperform a tuned classic warehouse for bursty dashboards?
Our executive dashboards are quiet most of the day and receive sharp bursts around scheduled meetings. A classic warehouse is predictable but often idle. For teams that moved this pattern to serverless SQL, which workload traits made the biggest difference, and how did you compare startup time, concurrency, and total cost…
-
Separating experiment tracking by team while keeping shared model visibility?
Several teams need independent experiment spaces and retention policies, while reviewers want one place to discover approved models and understand lineage. How would you structure experiments, registered models, permissions, and naming so teams can move independently without creating a fragmented model catalog?
-
Feature freshness checks across batch and streaming tables—what works in practice?
Our models combine daily aggregates with near-real-time behavioral features. Individual pipelines have freshness alerts, but we still miss cases where the final training or serving table is technically updated yet contains stale upstream values. How are teams defining and enforcing end-to-end freshness contracts across…
-
How do you attribute GenAI serving cost across endpoints and experiments?
Several teams share model-serving capacity, and we need a fair way to show cost by application, endpoint, model version, and experiment. Request counts alone hide major differences in token volume and latency. Which usage tags, system tables, and allocation rules have helped you produce actionable chargeback data without…
-
How are you versioning evaluation datasets for retrieval-augmented generation apps?
Our RAG evaluation set changes as product documentation and expected answers evolve. We want to compare prompt, retriever, and model changes without quietly changing the benchmark underneath the experiment. How are you versioning the questions, reference context, grading rubric, and human review notes so results remain…
-
Practical way to audit column-level access without overwhelming data owners?
We need a recurring review of access to sensitive columns, but a raw permission export is too noisy for domain owners. Has anyone built a review process that summarizes effective access by business purpose and highlights only meaningful changes? I am looking for a workflow that is defensible for audit but still manageable…
-
Should external locations be organized by domain or environment in Unity Catalog?
We are redesigning storage credentials and external locations for multiple business domains across development, test, and production. A domain-first model seems easier for ownership, while an environment-first model may simplify operational controls. Which approach has aged better for larger organizations, and where do you…
-
How are teams testing declarative pipeline changes before promoting to production?
Our pipeline definitions are moving toward a declarative model, but our current tests mostly validate notebooks in isolation. I would like a promotion workflow that catches schema drift, expectation failures, and unintended full refreshes before a production run. What does a practical test environment and release gate look…
-
Best pattern for handling late-arriving CDC events in a multi-table Delta pipeline?
We ingest change events from several operational systems, and a small percentage arrive hours after the normal watermark. The pipeline needs to keep customer and order tables consistent without replaying an entire day. How are teams combining watermarks, MERGE logic, and idempotency when related events arrive out of order?…