An article by Yash Mehta, an IoT, M2M and Big Data technology expert.
In web 3.0, the dynamics of not just the internet but the data streams are undergoing a decentralized transformation. As a first step, thanks to distributed data governance every domain can now manage and govern its data products but at the same time, it also relies on central control of security policies, data modelling, and compliance.
Data mesh distributes data across physical and virtual networks in a decentralized manner. Unlike conventional data integration tools that call for a highly centralized infrastructure, a data mesh instead works across on-premise, multi and single-cloud, edge environments.
In this post, we discuss the real-world applications of mesh in different setups.
Data Mesh: Solving Several Common Problems
As per the findings from MIT, only 13% of surveyed organizations could successfully deliver as per their data strategy. Data Mesh is addressing many responsible root causes.
Using a data mesh can solve several problems seen in data pipelines on a smaller scale. If not addressed, these can quickly become problematic and brittle over time as messy point-to-point systems can create their webs over time.
At the same time, a data mesh also solves bigger issues in an organization like core business facts that different departments in a company may disagree on.
By implementing a data mesh there is a less likely possibility for the system to have copies of facts.
Using a data mesh not only brings order to a system but also gives you a better manageable, mature, and evolved data architecture.
As we see the rise of cloud-based applications, app architectures are moving away and transitioning from conventional centralized IT and are moving towards distributed service mesh or microservices. A real-time data platform called K2view took a step ahead and successfully implemented the use of micro-DB in their fabric and mesh architectures. Every micro-DB stores the data for a particular business partner (customer) only while their mesh platform stores millions of such micro-DBs.
Data Mesh: Use Cases
A data mesh can support several analytical and operational use cases across multiple domains. A few examples include:-
1. Understanding Customer Lifecycle
It provides 360 support for customer care and reduces average customer handling time significantly. It also improves customer satisfaction and increases first contact resolution.
The marketing department can also deploy a single customer view for next-best-offer decision-making or predictive churn modelling.
2. Utility in the Internet of Things (IoT)
With IoT device monitoring, product teams can get insights into edge device usage patterns. They can use this pattern information to iterate and improve their profitability and product adoption.
By adopting a mesh network for IoT devices companies can get several benefits that make it a popular technology when it comes to choosing networks.
Companies can store all their IoT, enterprise, streaming, and 3rd party data collectively together into an S3 data lake at a very low cost.
3. Self-Healing Algorithm
As mentioned before the Shortest Path Bridging, the self-healing algorithm automatically selects the best path for sending data even in a case where some nodes lose their connection.
This algorithm allows the system to use only available and working connections. So, even if some devices stop functioning, the network is still able to send and receive the information needed to maintain or finish a given task.
4. Distributed & More Effective Security
Now when it comes to security, corporates are well prepared and keep updating their protocols. SMEs, however, lack the needed guidance. As per Accenture’s Cybercrime study, 43% of attacks are aimed at smaller organizations while only 14% are able to prevent themselves.
With contemporary data management solutions like Mesh, SMEs have an opportunity to stay in line with the trends.
Security is critical in a situation where the data is highly decentralized and distributed.
Such systems should delegate authorization and authentication activities to different users providing them with different levels of access as needed.
Following key security abilities of data mesh have been identified in the 2022 market premier report:
- Data privacy management in all forms
- Data encryption, be it at rest or in motion
- Data masking, to effectively manage PII obfuscation
- CCPA and GDPR compliances along with other legislations
- Identity management that covers all IAM/LDAP type services
5. Self-Configuration
IoT devices can now self-configure themselves thanks to auto-discovery for mesh networks. It automatically calibrates new nodes and connects them to the desired network without having any previous setup.
With this feature, the network can be expanded and governed easily.
6. Marketing and Sales
The marketing and sales team can easily curate a 360 view of consumer profiles and behaviours from different platforms and systems by using distributed data.
This allows them to create better-targeted campaigns, CLV (customer lifetime values), better lead scoring accuracy and perform several other essential metrics for performance.
Marketing teams use hyper-segmentation to deliver a campaign to the right customer via the right channel at the right time.
7. AI and Machine Learning
Intelligence & development teams can easily create data catalogues and virtual warehouses from several sources to feed AI and machine learning models.
This gives them more insights without having to collect all the data in a given central location.
Teams can also use federated data preparation that enables domains to provide trusted data and quality for data analytic workloads.
8. Loss Prevention
By implementing data mesh in the financial sector companies can develop a quicker time-to-insight with lower operational risks and costs.
This feature enables international financial institutions and organizations to analyze their data locally. This can be done in any region or country and it helps identify any fraud threats without creating any copies of data sets that can be transported to the central database.
Data privacy management allows companies to protect their customer data as they have to comply with evolving regional data and privacy laws like the VCDPA.
A Few Real Implementations of Data Mesh
Financial Services Institution
In one of their blogs, Thoughtworks discussed the impact of data mesh on a financial institution’s data process.
Since such an application deals with large volumes of transactional data in real time, it is important to stream accurate and timely feeds to analytics systems.
In this case, the executives had the flexibility to operationalize data quickly and they were able to access domain-oriented data products.
This allowed them to ask more relevant questions and eventually got them more reliable answers and valuable insights to act on in less time.
Not only this, but the domain team was also able to use analytical data and build it directly onto their users’ digital experience.
AWS S3
There was a drastic change when AWS commoditized its storage layer and replaced it with the AWS S3 object store around 15 years ago.
Because of the affordability and ubiquity of S3 and other cloud storage, companies are now shifting their data to cloud object stores. This allows them to build data lakes where eventually data can be analyzed in different ways.
A Fashion Retailer Brand
Europe’s largest online fashion retailer, Zalando learned that there is a simple way to guarantee access and availability at scale. This can be done by moving more responsibilities to the teams that initially gather this data and also have the required domain knowledge. And also by keeping all metadata information and data governance central.
Trust me, the space is just not enough to cover all use cases. It’s a propelling market and businesses want to extract the most out of it.
What’s Next? Embrace a Data Product Mindset
There are several innovative practices for data products that amalgamate different concepts together like design thinking, the jobs-to-be-done theory, and breaking organizational silos that bar cross-functional innovation. Enterprises, in 2022, should grab the opportunity and revamp their data management strategy keeping web 3.0 in mind.
Author Bio: Yash Mehta is an internationally recognized IoT, M2M and Big Data technology expert. He has written a number of widely acknowledged articles on Data Science, IoT, Business Innovation, and Cognitive intelligence. His articles have been featured in the most authoritative publications and awarded as one of the most innovative and influential works in the connected technology industry by the IBM and Cisco IoT departments.
