Data fabric and data mesh are emerging data management concepts that address the needs of effective data management. You shouldn’t consider them as competitors but rather as two different approaches to data decentralization. They are growing on popularity and if used well, they can be a valuable asset that matches your company’s needs. Both are working towards the same goal: to reach people with information wherever they are.
As Gartner mentions in its 2022 report, data fabric is promoted by tool providers as a semantic or virtualisation solution. The solutions often include some form of caching of data and optimization techniques. It begins by accumulating of passive metadata then progresses into active metadata scenario. It is an end-to-end data integration solution consisting of architecture, data management, integration software and shared data to manage your data.
Gartner defines it as “design concept that serves as an integrated layer (fabric) of data and connecting processes.” What it means is that it’s not a single product but is made of a set of integrated technologies that improve the value of enterprise data. It combines multiple technologies to design a meta-driven, AI-driven data platform. Since it is only metadata-driven, the abstraction layer is easier to model, can integrate any data source, even in the real-time.
It integrates data, analytics, and dashboards into one and serves as a management solution, allowing frictionless access in a distributed environment.
Data mesh is an application layer that distributes information to the stakeholders and people effectively and with ease. It basically brings people, technology and processes together. It promotes the adoption of cloud native and cloud platform technologies to scale and achieve the goals of data management.
Data mesh is commonly used within large organisations with many departments. If you are working for a marketing department, you do not need to see data from HR departments or IT departments. Data mesh allows you to see and access only those data resources that are valuable and needed for your work scope. It helps you focus on the important things.
The goal of data mesh is to treat data as a product, with each source having a data product owner who could be part of the team of data engineers. Data mesh reduces the human burden that is spent in data discovery. The machine learning algorithms make recommendations about which datasets are relevant to an analysis task. Then, they can be brought together for particular applications, that takes away the burden on human teams that deploy and redeploy infrastructure of the apps.
Data fabric relies on the efficiency and capabilities of centralised data management tools. Data mesh shifts the architecture design towards distributed data services in business domains to design and create data products. A data fabric is based on the “active metadata” which uses knowledge graph, semantics, and AI / ML technology to discover patterns in various types of metadata and apply this to automate the data value chain. What’s more interesting is that data fabric makes the data mesh even better as e it can automate key parts of the data mesh such as creating data products faster.
They represent the perfect blend of more than 50 years of data management experience. They can work with each other and exploit the practices of one another.
Interested in hearing more? Read our blogs or try our new tool BinarBase as an optimal solution for your data reporting and visualisations.
