At Bosonit we have kicked off the TechXperience with a talk on 'Layered Architecture'. The TechXperience is a series of events of technological culture, divided by areas of knowledge aimed at creating and enriching the knowledge of our community.
The aim of these is to improve the visibility and skills of the people we work with. Bosonit to share our knowledge and keep abreast of all the latest technological developments and updates, disseminate and enrich our talent through the dissemination of experiences in projects that our colleagues have experienced to serve as inspiration and help us in our day to day. These sessions will be focused on the areas of knowledge that make Bosonit different: Big Data, Machine Learning / Data Science, DevOps & Cloud, Development y Data Management / Business Intelligence.
The first of the sessions was led by our partner David Ortega CruzBI consultant, and dealt with "Experiences in data processing". In it, he discussed the vision of the design, architecture and modelling of databases for Business Intelligence and on architectures, modelling and Data Management in large companies for the creation of databases. David's experience accredits him as an authoritative voice and reference given his experience and training.
Who is David Ortega Cruz?
David Ortega Cruz is part of Bosonit for almost five years. He has a degree in Industrial Engineering with a specialisation in Electronics, a degree in Computer Engineering and a Master's degree in Analysis and Visualisation of Massive Data. In the application of his knowledge, the roles he normally plays in projects are Data Engineer, BI Consultant or Big Data Developer. He has worked on projects for the architecture, modelling and development of databases as well as big data loading processes. Creating structures and processes from scratch.
He has also worked within a team of Data Governancewhere he reviewed processes, documentation, dictionaries, tables, etc. In addition, he has also optimised already created processes and database tables, to improve performance and loading times. He has designed and created visualisations, both the dashboards with all its graphs and KPIs, as well as designing and creating the templates for the visualisation tools.
Layered architecture
During his presentation, David Ortega talked about the importance of taking into account certain aspects, conditions and practices in the architecture of data layers in business projects. Intelligence. Emphasising certain key aspects such as not only the development and its implementation, but also adding easy maintenance that allows improvements to be made or the management of access to data. Emphasising the steps, techniques and appropriate processes to be carried out to achieve this.
Data architecture and modelling in Business Intelligence
When developing and implementing a Business Intelligence project, with all its processes, databases and visualisations, certain aspects, conditions and practices must be taken into account. It is not only to develop it and see that it works, but also that it is easy to maintain, that improvements can be made or that people in the business have access to the data. Mainly there are two architectures, 2-layer and 3-layer, for the development of the modelling of the tables.
The first thing is to create the layers of the data architecture and everything that goes with it. Generally two layers are used in the architecture. A first layer called ODS (Operational Data Store) and a second layer called DWH (Data Warehouse). The ODS layer integrates data from multiple sources, where data is cleansed, operations are performed and integrity is checked. In addition to applying Data Quality and business rules. The DWH layer is the non-volatile, time-varying store, ready for exploitation by visualisation tools and business users.
Between the layers and the sources, temporary tables stored in memory or auxiliary tables stored in the database are often used for data processing. This all depends on various factors such as available memory, complexity of calculations, processing time, etc.
Sometimes a three-layer architecture is used, and a third data layer is usually created in front of these two data layers. This data layer is called Staging and precedes the ODS layer. It stores an exact copy of the source data, with the same schema, tables and names. The objective of this layer is not to interfere with the daily operations of the different applications and users. This has the advantage of being able to examine and reprocess without interfering with the source tables.
