Confused about Data Mesh and Data Fabric? here’s all you need to know

By
IntelligentHQ

Today’s world is a hyper-connected system of the Internet of Things (IoT) and the cloud where a complex interconnected network connects one computing network device to another. This intricate interconnection of networks can pose a serious challenge to the future of data management because the eventual goal of data management is to share business data across different technologies and platforms. As a solution to this challenge, ‘data fabric’ and ‘data mesh’ have been evolving as paradigms to solve significant problems in modern data management.

The problem statements

Centralized data platforms and monolithic architectures can be a deteriorating factor for business agility. This makes it difficult for businesses to adjust to the dynamic environment of data that demands new projections, views, and aggregations of the data. Businesses that are dependent on legacy architectures face challenges in scaling data and analytics, given:

  • Data complexity and proliferation.
  • The absence or lack of collaboration.
  • The inability to adhere to data demands in time.
  • Lack of trustable data given the duplication of the same data.

Consequently, data mesh and data fabric were established to help businesses achieve optimal scalability and agility. On the one hand, the data fabric framework highlights the increasing data management complexity by intelligently connecting, integrating, and organizing data, making data assets eligible for consumption. On the other hand, data mesh is an emerging framework that promotes the objectives of the data fabric.

Now, if you only have a little understanding of these two concepts, they might sound very alike. Nevertheless, they are primarily quite distant techniques making different technical assumptions even though they both are techniques that attempt to solve a mutual problem. This is why it seems appropriate to explain the difference and mitigate the confusion between data mesh and data fabric. However, before moving on to its differences, it is helpful to have an understanding of what each idea stands for.

A brief introduction of data fabric

The term ‘data fabric’ has its roots in the mid-200s, when Noel Yuhanna, a Forrester analyst, first defined it. Conceptually, data fabric can be defined as a way of connecting a huge collection of data tools but in a metadata-driven manner. The architecture and design framework that is focused on addressing the complexities of data management is known as data fabric. Data fabric minimizes the destruction caused to data consumers besides making sure that any available data on a platform at a given location can be combined, shared, governed, and assessed successfully and efficiently. The primary tools that drive the architecture of data fabric are AI/ML automation and augmentation, a strong technology, and an intelligent metadata foundation.

In other words, data fabric can be considered as a logical data framework that serves as a layer for data knowledge and connection. Virtualization of language or rules and data federation is enabled by data fabric. The key component of data fabric is a knowledge graph that serves as the data discovery layer or data orchestration. The ultimate goal of this framework is to unify structured and unstructured data and aggregate it to connect different formats of data for easy understanding of both humans and machines.

A brief introduction of data mesh

The architecture that focuses on organizational changes is referred to as a data mesh. Within a data mesh framework, domain teams typically own the delivery of data products. They do so by taking into account that domain teams and data are in close connection, thereby making the data easier for them to understand. In simpler terms, data mesh can be regarded as a type of data platform that accepts the presence of data in an organization by leveraging a self-serve and a domain-oriented architecture.

Data mesh enables consumers to discover, decipher, trust, and utilize data or data products to drive initiatives and decisions that are data-driven. According to data teams, data mesh also offers a prime opportunity when data platforms are needed to convert into a data microservices architecture. At the focal point of data mesh lies the idea of separating the monolithic data architecture and monolithic ownership or custodianship of data around organizational domains. Data lakes and data warehouses can be present in the data mesh framework. However, both of them become another node rather than being a centralized monolithic.

The primary differences between data mesh and data fabric

Now that you have a surface idea of what data mesh and data fabric are all about, you may be curious to know their key differences. Even though both data mesh and data fabric are providers of a data architecture that promotes a connected and integrated data experience within an intricate data landscape, the data mesh vs. data fabric debate tells us its dissimilarities.

  • Although data mesh and data fabric are architectures to deliver data products, under the core design principle, data mesh governs product thinking for data. Consequently, like any other data mesh product in an organization, data is provisioned and maintained similarly.
  • On the one hand, data mesh is dependent on data domain/product owners to straightforwardly channelize the data product requirements. On the other hand, data fabric is used to leverage mechanization in discovering, recognizing, connecting, suggesting, and offering data assets to consumers based on a metadata foundation.
  • Data mesh promotes the advocacy of decentralizing data practitioners such as data engineers and data scientists, as a part of the data/domain products team. However, management within a data fabric structure is unified and remains distributed. An organization can promote unified management of data sources with a singular data fabric virtually overlaid on top of other data repositories.
  • Another primary difference between data mesh and data fabric structure is in the accessibility of APIs. Unlike data fabric, a data mesh approach is an API-based solution for developers. In a data fabric approach, developers need to write codes specifically for the APIs to the interface. In the data mesh approach, the API integration does not happen inside the fabric as it happens in the data fabric approach.

Summarizing the big differences

 

DATA MESH DATA FABRIC
It is customer-driven It is a vendor-driven
Data is available as APIs, data sets, and  streams Data is available in tabular formats
Data estate is not aggregated Data estate is distributed and physically heterogeneous
Data management and governance is under unified control Data governance is centralized, and control is given to business domains
Enables accessibility of distributed data and centralized policy through a virtualization layer of the data fabric. Data domain management is delegated to organizational units that work with data mesh

 

The pros of data mesh architecture

  • The data mesh architecture allows the delivery of customized data products that adhere to organizational demands by connecting strategic organizational goals to a data product landscape that drives value.
  • Data mesh leverages the delivery of data products through domain-specific expertise and decentralized ownership that transforms organizational culture into data product thinking.
  • Data mesh helps in improving business agility by lowering complexities and breaking down centralized architectures that can be a drawback to complying with organizational data demands promptly.
  • Data mash also enables a data governance operating model that is flexible UN allows organizations to skill the operating model to meet their unique needs.

The pros of the data fabric architecture

  • Data fabric allows the integration and connection of all data in an organization. It also facilitates the easy sharing of data for improving business outcomes.
  • Data fabric enables the acceleration of self-service data analytics and discovery by ensuring that data consumers have faster access to trusted data.
  • Data fabric also reduces the efforts and cost of managing data by using intelligent mechanization of data management activities.
  • Data fabric also promotes real-time insights and analytics through the optimization of the data life cycle, enabling faster and more flexible development of applications that rely on data.

The final verdict

As you can see from the above information, there are quite a few similarities between data mesh and data fabric. However, the differences should be taken into account. In conclusion, both the data mesh and data fabric approaches provide a rigid foundation for the assessment of data across multiple platforms and technologies. But a data mesh addresses organizational change while a data fabric is more focused on technology.

Additionally, data mesh addresses more processes and people rather than architecture. But the data fabric approach is architectural to smartly manage the complexities of both metadata and data. Both data and data fabric have a lot to offer. When it comes to uh architectures that support big data projects and architectural concepts, the choice between the two comes down to discovering what fits your own needs and resources.

RELATED ARTICLESMORE FROM AUTHOR