Advances in the data management industry have led to a surge in the popularity of areas such as data governance and metadata management. Organisations now have large volumes of information, data collected from different sources, with which to perform analysis and improve productivity in increasingly competitive markets.
To reach this point, the first cultural change has been focused on the collection of data and how to get the most out of it. The next step has come naturally as the need to manage these new volumes of information has arisen: are the data up to date, do I have good quality data for the company's processes, does it satisfy the current regulatory frameworks, and does it meet the needs of the company's business processes?
Trends in data architecture
In this article we will look at the three trends in Data Management that are here to stay and others that are yet to come associated with data architectures. In a following article we will analyse other developments within the area of Data Management, but outside the area of architecture.
Data Fabric
In the beginning it was common to have to choose between a Data Warehouse or a Data Lake. As we grow in data volume or complexity, it becomes more and more common to have a combination of both, and with it the likelihood of a value appearing in several systems with different information.
This architecture model centralises all data sources in a single logical layer ("Enterprise Data Layer") and applies all data management processes: security, validation, predictive analytics, etc., to this layer. This makes it possible to maintain data consistency and ensure that a unique value is delivered for each piece of data.
Data Mesh
There are business models that need to have a distributed database or with different data owners. These models have important advantages in the decentralisation of the data, but a high cost in the management, as it requires a meticulous control to guarantee the order and consistency of the information.
In this regard, Zahmak Dehghani proposed, in his 2019 book, an organisational model to define a roadmap for a decentralised architecture. He based his Data Mesh architecture on compliance with the following principles:
- Domain-driven ownership.
- Data as a product.
- Federated governance.
- Self-service infrastructure.
Companies such as Zalando or Netflix have implemented this type of architecture, but have not managed to exceed expectations. For the time being, they are still working on their implementation in order to take advantage of the benefits of a decentralised data model.
Edge Data Management
In addition to data warehouses located in the Data Center or on external servers, the increase in IoT devices or mobile phones has generated an area known as the "edge" that also needs to be managed. In this area we are talking about devices that suffer or may suffer from latency and connectivity problems, so there is no continuous availability of the information contained in these devices.
Industry has targeted this challenge on the basis of two requirements to be met:
- Focused on response: data must be stored and displayed immediately. An example could be the data produced by a commercial aircraft. Avionics data contains the information that keeps the aircraft in flight and is not required to connect to the tower for decision making. In this case the data is available to the pilot in real time.
- Focus on analytics: the device is provided with sufficient computational capabilities to perform operations of Machine Learning and to run predictive models. For this reason, the data remains inside the device and is used without leaving the device.
