The evolution of software engineering has embraced microservices and modular design, enabling resilience and adaptability in complex systems. However, data engineering remains rooted in monolithic architectures and outdated practices. This talk draws from decades of experience in data and software engineering, advocating for the application of modern software principles to data systems. By embracing modularity, data contracts, and tools like dbt, data professionals can address challenges such as siloed systems, centralized ownership, and inflexible processes. This session will provide practical guidance for shifting toward an anti-monolith architecture and fostering innovation in data engineering.

Key Areas of Focus:

• Data Mesh: An understanding of this socio-technical approach to decentralizing data architecture and enabling interoperability.
• Modularity in Data Engineering: Breaking down monolithic data systems into independent components integrated via data contracts.
• Evolutionary Architecture: Leveraging microservice patterns for low-risk upgrades and adaptability in data systems.
• Adopting Software Principles: Implementing CI/CD, automated testing, and observability in data engineering.
• Tools and Technologies: Utilizing dbt, automated quality checks, and cross-project references to modernize workflows.
• Practical Next Steps: Incremental actions for embracing anti-monolith design and modernizing data practices.