Analysis of scalability and bottleneck mitigation in ETL with growing data on databricks: the impact of performance techniques on operational efficiency and query quality

Abstract

This study evaluates the scalability of a Databricks Medallion architecture using a 2×2 factorial design across Small (100K) and Large (100M) datasets. The methodology comprised 120 controlled pipeline executions to assess ingestion overhead and 480 query runs (30 replicates across 4 tables and 4 scenarios) to validate read performance. The results reveal that while optimizations introduce a 22.02% ingestion overhead, they achieve a 64.7% reduction in query latency for large volumes. Most significantly, the optimized architecture demonstrated a near-perfect scalability ratio of 1.02x (versus 2.95x in the baseline), proving that shifting computational costs to the write-side is essential for decoupling analytical performance from data growth.