What is Data Mesh

Data Mesh is an analytical data architecture and operating model where data is treated as a product and owned by teams that most intimately know and consume the data.

Today, data is ubiquitous. Data is the by-product of any and every digital action we take. Everything, every system, every process, every sensor generates data. Technology makes it easier for organizations to collect and store data for businesses to leverage to make better decisions or create more tailored experiences for their customers.

However, organizations are struggling to enable and empower their employees to make the most informed and timely decisions possible. Centralized data platform architectures fail to deliver insights with the speed and flexibility scaling organizations need. Data Mesh serves as a solution to these problems.

Data Mesh applies the principles of modern software engineering and the learnings from building robust, internet-scale solutions to unlock the true potential of enterprise data.

Principles of a Data Mesh

Domain ownership

The domain ownership principle mandates the domain teams to take responsibility for their data. According to this principle, analytical data should be composed around domains, similar to the team boundaries aligning with the system’s bounded context. Following the domain-driven distributed architecture, analytical and operational data ownership is moved to the domain teams, away from the central data team.

Data as a product

The data as a product principle projects a product thinking philosophy onto analytical data. This principle means that there are consumers for the data beyond the domain. The domain team is responsible for satisfying the needs of other domains by providing high-quality data. Basically, domain data should be treated as any other public API.

Self-serve data infrastructure platform

The idea behind the self-serve data infrastructure platform is to adopt platform thinking to data infrastructure. A dedicated data platform team provides domain-agnostic functionality, tools, and systems to build, execute, and maintain interoperable data products for all domains. With its platform, the data platform team enables domain teams to seamlessly consume and create data products.

federated governance

The federated governance principle achieves interoperability of all data products through standardization, which is promoted through the whole data mesh by the governance guild. The main goal of federated governance is to create a data ecosystem with adherence to the organizational rules and industry regulations.

Benefits of Data Mesh

Business Agility and Scalability

Data mesh powers decentralized data operations, independent team performance, and data infrastructure as a service provision, resulting in improved time-to-market, scalability, and business domain agility. It eliminates the process complexities and IT backlog to reduce operating and storage costs.

Faster Access and Accurate Data Delivery

Data mesh offers easily governable and centralized infrastructure based on a self-service model without underlying complexity for faster data access and accurate delivery. Businesses can access data from anywhere with SQL queries with much lower latency. The distributed architecture reduces the processing and intervention layers that delay time to insight.

Flexibility and Independence

Enterprises adopting data mesh architecture are becoming vendor-agnostic businesses that are not locked in with one data platform. The distributed infrastructure allows companies unparalleled flexibility and choices due to connectors to many systems.

Platform Connectivity and Data Security

The decentralized framework allows cloud applications to be connected to on-site sensitive data, which can be live streaming or existing on devices in real-time. Data mesh queries/compiles data analytics where the data resides, instead of requiring users to make a copy and route it through a public network to a data warehouse.

It eliminates the risk of data breach or information loss to improve security and reduces data latency to improve overall performance in various use cases including, live streaming, online gaming, financial trading, etc., through platform connectivity in a distributed model.

Robust Data Governance for End-to-End Compliance

Distributed architecture reconciles data ingestion with its sources, formats, and volumes to allow businesses to control their security at the source system. The decentralized data operations simplify compliance with global data governance guidelines for quality data delivery and ease of data access.

Cross-Functional Teams for Improved Transparency

The centralized data ownership of traditional data platforms isolates expert teams, creates a lack of transparency, and fails to provide contingency against data control/ownership loss. Data mesh decentralizes data ownership by distributing it among cross-functional domain teams, including domain experts, business teams, IT, and agile virtual teams through its domain-oriented approach for improved transparency and data quality.

Data Mesh in Action – Get More Out of Distributed Data

Data mesh unlocks endless possibilities for businesses in various consumption scenarios, including behavior modeling, analytics, and data-intensive applications. It could be core data comprising the business sales data or/and non-core data encompassing web data and clickstream; the distributed architecture enables easy data access and faster delivery without a vendor lock-in with an expensive enterprise warehouse.

Data Mesh Use Cases

n the data mesh implementation, the central IT still exists to build a self-service data platform, but it does not own the data. For instance, a marketing company with the central IT team responsible for delivering an enabling technology, is still responsible for overarching governance and security for connected systems but individual functional teams have responsibility for the data itself

IT and DevOps

Data mesh offers a modern development approach to data analytics and software teams. It reduces data latency by providing instant access to query data from proximate geographies without access limitations.

Sales and Marketing

The distributed data enables sales and marketing teams to curate a 360-degree perspective of consumer behaviors and profiles from various systems and platforms to create more targeted campaigns, increase lead scoring accuracy, and project customer lifetime values (CLV), churn, and other essential performance metrics.

AI and Machine Learning Training

Data mesh enables development and intelligence teams to create virtual data warehouses and data catalogs from different sources to feed machine learning (ML) and artificial intelligence (AI) models to help them learn, without having to consolidate data in a central location.

Loss Prevention

Data mesh implementation in the financial sector creates faster time-to-insight at lower operating costs and operational risks. Distributed data analytics compacts fraudulent behavior modeling to detect and prevent fraud in real-time. It allows international financial bodies to analyze data locally – within any particular country or region, to identify fraud threats without replicating data sets and transporting them to their central database.

Global Business

A decentralized data platform makes it easy to comply with worldwide data governance rules to provide global analytics across multiple regions with end-to-end data sovereignty and data residency compliance.



Previous Story

Three convicted for stealing more than 564 kg of Eskom’s aluminium

Next Story

Hacking Healthcare