Data mesh is a decentralized approach to managing data where domain teams own the data they need. Rather than having a centralized data team manage all data, data mesh delegates ownership to product teams. This allows them to move faster without waiting on approvals from a central data team. But does this decentralized approach really work in practice? Let’s explore the pros and cons.
What is data mesh?
Data mesh is an architectural paradigm shift in how organizations manage data. The key principles of data mesh are:
- Domain-oriented decentralization – Data is owned by domain teams rather than a central data team
- Data as a product – Treat data as a product with its own lifecycle and roadmap
- Self-serve data infrastructure – Domain teams can access and build on data infrastructure services
- Federated computational governance – Standards and tools for consistency but decentralized data ownership
In plain terms, data mesh gives domain teams increased autonomy and ownership over the data they need to build products and serve customers. The centralized data team shifts towards a platform role, providing self-serve data infrastructure and governing via standards rather than gatekeeping.
The potential benefits
Data mesh offers several potential benefits for organizations:
- Speed – By removing bottlenecks around central data team approvals, domain teams can move faster on data projects.
- Agility – Domain teams have flexibility to evolve data models as business needs change.
- Alignment – Domain data owners are closer to the business needs.
- Developer productivity – Self-serve access to data removes engineering bottlenecks.
- Scalability – Decentralized ownership allows organizations to scale quickly.
In theory, data mesh offers faster access to data, tighter alignment to business goals, and increased productivity for engineers. But does this work in practice?
The potential challenges
While the benefits sound great on paper, implementing data mesh introduces challenges including:
- Coordination – With decentralized ownership, cross-domain data projects become harder.
- Data quality – Maintaining consistency and quality standards can be difficult.
- Security & compliance – More access points increases security and compliance risks.
- Tooling – New technical capabilities needed for discovering and accessing data.
- Governance – Light-touch governance requires organizational maturity.
These challenges introduce new complexities around how teams coordinate, maintain standards, comply with regulations, build new tooling, and evolve governance models. An immature organization may struggle.
When does data mesh work best?
Data mesh is best suited to organizations with:
- Mature engineering culture and practices
- Low need for centralized coordination
- Strong existing data governance foundations
- Sufficient resources to build new tooling
Organizations without these attributes will likely struggle to adopt data mesh. On the other hand, fast moving consumer tech companies like Amazon, Uber, and Airbnb with strong engineering DNA are good candidates.
Here is a comparison of how different organizational attributes affect data mesh suitability:
Attribute | Low Suitability | High Suitability |
---|---|---|
Engineering maturity | Waterfall development processes | Robust CI/CD and testing |
Coordination needs | Highly interdependent systems and data | Loose coupling between systems |
Existing governance | Informal or compliance-focused only | Standards for security, quality, metadata |
Resource availability | Constrained budgets and headcount | Healthy budgets and staffing |
Key takeaways
Some key takeaways on whether data mesh can work:
- Data mesh offers significant benefits around productivity, alignment, and agility but has real implementation challenges.
- It requires mature engineering practices, loose system coupling, strong governance foundations, and ample resources.
- Immature organizations will likely struggle with quality, security, and coordination.
- Mesh aligns well to consumer tech company cultures but may not suit highly regulated sectors.
- Take an eyes wide open approach – expect a major cultural and architectural shift.
Conclusion
Data mesh offers compelling benefits but requires organizational maturity to work well. It is best suited to software-driven companies with strong engineering cultures. Adoption will require significant evolution of skills, platforms, and culture across both central and domain data teams.
With realistic expectations of the challenges, suitable organizations can benefit greatly from the increased agility and alignment of data mesh. But it is not a quick win – expect a multi-year journey requiring vision, investment and expertise.