Data Engineering Is Not Software Engineering, Until It Is
Modern Data Engineering: Bridging DevOps, MLOps, and Software Development | This technical session examines how modern data engineering is evolving beyond traditional software engineering practices, focusing on data pipeline architecture, testing frameworks, and deployment strategies. Through real-world case studies from dbt (data build tool) implementations and SQLMesh data transformation workflows, the presentation explores how data teams are adopting GitOps methodologies, continuous integration, and version control for data-centric systems. As artificial intelligence and machine learning operations become central to software development, these emerging data engineering practices are reshaping how teams approach data quality, system validation, and production deployment. The session will demonstrate how differences in ETL pipeline feedback loops and data testing environments are driving new best practices for managing enterprise data systems, while offering insights into the future convergence of CI/CD, data governance, and MLOps practices.