If you’re in the data strategy space, you’ve likely heard the term “data mesh” — a new paradigm shift in big data management toward decentralization.
The concept of data mesh was founded in 2018 by Zhamak Dehghani, Director of Emerging Technologies at the software company ThoughtWorks.
Zhamak and Thoughtworks describe data mesh as, “an analytical data architecture and operating model where data is treated as a product and owned by teams that most intimately know and consume the data.”
There are four core concepts of data mesh that serve as the underpinning of its successful adoption: domain ownership, data as a product, self-service data platforms, and federated computational governance.
Let’s take a closer look at how Zhamak describes each principle.
Domain Ownership
In an article for Martin Fowler, Zhamak dove into the logic behind each pillar of data mesh and the architecture that supports it.
She explained how enterprises are broken down based on each business unit, and as a result, the concept of data mesh is rooted in decentralization and domain ownership.
Zhamak utilized the example of a digital media company divided into teams that support different facets of the enterprise. For instance, some employees may be focused on podcast publication, while others are tasked with managing and paying the hosts and artists. Data mesh was designed to respect these individual domains and their goals.
“For example, the teams who manage podcasts, while providing APIs for releasing podcasts, should also be responsible for providing historical data that represents ‘released podcasts’ over time with other facts such as ‘listenership’ over time,” Zhamak said.
The goal is to remove any friction and allow each business domain to serve its own analytical data, independent of other domains.
This model is in stark contrast to the centralized systems most enterprises are leveraging, which consists of a data lake for unstructured data and a data warehouse for structured data that’s managed by a department of engineering and other data specialists.
The software company Ellie outlined some of the issues associated with this model, which primarily revolve around a lack of communication.
Ellie states there is little to no communication between the business domain and the data platform team. So much so that the domain might not know their data is being brought to a data platform in the first place.
Additionally, the data team may not fully understand what’s going on in the business applications, which can lead to a situation where neither group takes ownership over data as a whole.
Many argue data mesh can serve as a solution to this challenge.
Data as a Product
Notably, issues can arise with this level of decentralization, and according to Zhamak, treating data as a product can help mitigate them.
The notion of data as a product aims to address data quality and silo issues, or what Gartner commonly refers to as “dark data” — “the information assets organizations collect, process, and store during regular business activities, but generally fail to use for other purposes.”
Instead, Zhamak argued that analytics data provided by each business domain should be treated as a product, and the consumers of that data should be looked at as customers.
She also highlights the need for domain data product owners to ensure data quality and user satisfaction. This individual should have a deep understanding of the organization’s data users and how the data is used.
Each domain data product owner should be supported by data product developers who will build and maintain the product.
“Such intimate knowledge of data users results in design of data product interfaces that meet their needs,” Zhamak said.
Self-Service Data Platform
One of the main concerns of shifting data ownership to individual business domains is the necessary skills required to manage data throughout its lifecycle and operate data pipelines.
This is where Zhamak introduces her principle of a self-serve data infrastructure. In her article, she illustrates the need for a platform that alleviates overall complexity and friction.
Additionally, this self-service platform should provide a new selection of tools and many capabilities including, data product lineage, monitoring, governance and standardization, and unified data access control, among many others.
“A self-serve data platform must create tooling that supports a domain data product developer’s workflow of creating, maintaining, and running data products with less specialized knowledge than existing technologies assume.”
Zhamak Dehghani
Self-service in itself can be a bit of a buzzword, and the parameters and tools utilized can vary depending on the enterprise industry and the number of users.
During an Enterprise Data Strategy Board panel on leading a data culture transformation, Chris Gifford, USAA Chief Data Officer, spoke on the importance of striking the right balance between enabling enterprise employees to use data to make smarter business decisions without creating a potential for risk.
“You want to give them the ability to run their business on data, but you don’t want to give them the ability to run with scissors.”
Chris Gifford
Federated Computational Governance
With these risks in mind, data mesh requires an effective governance model, and one that embraces decentralization.
Zhamak calls this “federated computational governance,” or “a decision-making model led by the federation of domain data product owners and data platform product owners, with autonomy and domain-local decision-making power, while creating and adhering to a set of global rules.”
However, this is no small task as it’s difficult to maintain an equilibrium between centralization and decentralization. The enterprise must determine which decisions to leave up to each domain, and which decisions should be universally accepted.
While traditional governance and data mesh governance both aim to abstract value from data, traditional data governance accomplishes that through centralized decision-making with minimal changes. In contrast, Zhamak said data mesh governance embraces change.
Will Data Mesh Actually Work for Your Enterprise?
It’s clear there are many benefits to adopting a data mesh model, and as a result, a large number of enterprises are moving in that direction. However, data mesh might not be the right choice for every company — it comes with risks too.
Members of the Enterprise Data Strategy Board — senior data strategy leaders at multibillion-dollar brands — got together in a confidential video call to share their candid advice on data mesh and what structures they’ve implemented to make it work.
