What is Data Mesh?
Data mesh is a new approach based on a modern, distributed architecture for analytical data management. It enables end users to easily access and query data where it lives without first transporting it to a data lake or data warehouse.
The strategy of data mesh distributes data ownership to domain-specific teams that manage the data as a product.
4 Principles of Data Mesh
Domain-oriented data ownership and architecture: The ecosystem creating and consuming data can scale out as the number of sources of data, number of use cases. So simply increase the autonomous nodes on the mesh.
Data as a product: Data users can easily discover, understand and securely use high quality data with a delightful experience. Data that is distributed across many domains.
Self-serve data infrastructure as a platform: The domain teams can create and consume data products autonomously using the platform abstractions. Hiding the complexity of building, executing and maintaining secure and interoperable data products.
Data governance: Data users can get value from aggregation and correlation of independent data products. The mesh is behaving as an ecosystem following global interoperability standards; standards that are inserted into the platform.
3 Problems of Current Data Platforms
Problem #1: Until now, enterprises used a centralisation strategy to process extensive data with various data sources, types, and use cases.
However, it requires users to import/transport data from edge locations to a central data lake to be queried for analytics, which is time-consuming and expensive.
How Data Mesh Solves It: The distributed architecture of data mesh views data as a
product with separate domain ownership of each business unit.
So this data ownership model reduces the time-to-insights and time-to-value by empowering business units. Operational teams to access and analyze “non-core” data quickly and easily.
Problem #2: As global data volumes continue to increase, the query method in a management model requires changes.
It slows down the response time to new consumers/data sources as the number of sources increases, which negatively affects business agility to get value from data and respond to change.
How Data Mesh Solves It: Data mesh delegates datasets ownership from the central to the domains (individual teams or business users) to enable business agility and change at scale.
Data mesh architecture steers enterprises towards real-time decision-making by closing the time and space gap between an event happening and its consumption/process for analysis.
Problem #3: Data transfer is often susceptible to data residency and privacy guidelines that prohibit data migration if the data is stored in particular geographies; such as data stored in an EU country but needing to be accessed by a user in North America.
Abiding by data governance regulations is time-consuming and tedious, and can significantly delay data processing and analysis. Teams need for critical business intelligence that helps them maintain a competitive advantage.
How Data Mesh Solves It: In data management, the domains are responsible
for the quality, security, and transfer of their data products.
So Data mesh provides a connectivity layer that enables direct access and query capabilities by technical and non-technical users to data sets where they reside. Avoiding costly data transfers and residency concerns.
Conclusion
As you’ve hopefully gathered from reading this post, data mesh in many ways represents a completely new approach to data.
While it certainly is preferred in many ways about how technology should be leveraged to implement data mesh principles. Perhaps the bigger implementation challenge is the organisational changes that are needed in order to implement.
Overcoming the inability to move the decades of monolithic architecture will not be easy for most companies.
Nevertheless, we think that the four principles of data mesh address significant issues that have long plagued data and analytics applications. Therefore there is real value in thinking about them and gleaning what we can regardless of whether your organisation ever goes.
