Skip to main content
LTTS

LTTS

Quick Links mini

  • Search
  • info@LTTS.com
  • English
  • 日本語
  • Israel
  • German
  • Contact
  • Engineering the change
  • Industry
    • Communication
    • Media & Entertainment
    • Semiconductors
    • Consumer Electronics
      • Consumer IoT
      • Enterprise Devices
      • Personal Devices
    • Oil & Gas
      • Digitalization
      • Oilfield Services
      • Owners and Operators
    • Software & Platforms
    • Healthcare
      • Medical Device & Life Sciences
      • Pharma & CRO
      • Healthcare Providers
    • Plant Engineering
      • FMCG
      • Chemicals
      • Discrete Manufacturing
      • Metals & Mines
    • Transportation
      • Aerospace Engineering
      • Automotive
      • Rail Transportation
      • Travel & Hospitality
      • Trucks & Off-Highway Vehicles
    • Industrial Products
      • Building Technology & Smart Infrastructure
      • Electrical, Instrumentation and Controls
      • Power & Utilities
    • Public Infrastructure & Smart Cities
  • Services
    • Digital
      Engineering & Consulting
      • Cybersecure
        • Security Monitoring
        • Security Services
        • Security Solutions
      • Immersive Experiences
      • Industry 4.0
      • Product Consulting
      • Sustainability Engineering
      • Sustainable Smart World
      • 5G
    • Product
      Engineering
      • Digital Engineering
        • Cloud Engineering
        • DevOps
        • Immersive Experiences
        • User Experience
      • Embedded Engineering
        • Embedded Systems
        • Sustenance
        • VLSI
        • Wearables Engineering
      • Mechanical Design
        • CAE & CFD
        • CAx Automation
      • Software Engineering
        • Engineering Analytics
        • Sustenance & Maintenance
        • User Experience Design
        • Voice Innovations
      • Testing & Validation
        • Integrated Design, Validation & Testing
        • Lab as a Service
        • Testing
    • Manufacturing
      Engineering
      • Smart Manufacturing
        • Accelerated Operations
        • Digital Factory & Simulations
        • Plant Design & Engineering
      • Manufacturing & Planning
        • Accelerated Operations
        • Digital Factory & Simulations
        • Line Expansion & Transfer
        • Manufacturing Automation
        • New Product Development
        • Plant Design & Engineering
        • PLM on Cloud
      • Manufacturing Execution
        • Agile Supply Chain
        • Content Engineering
        • Material & Parts Management
        • Sourcing & Procurement
    • Operations
      Engineering
      • Connected Product Support
        • DevOps
      • Supply Chain Engineering
        • Sourcing & Procurement
      • Plant Engineering
        • CAPEX Project E/EPCM Services
        • Operational Excellence
        • Plant Sustenance & Management
        • Material & Parts Management
        • Regulatory Compliance Engineering
  • Solutions
    • AiCEArtificial Intelligence Clinical Evaluation
    • AiKno™ Machine Learning, NLP & Vision Computing
    • AnnotAiAI Based Data Annotation Tool
    • ARC Asset Reliability Centre
    • Avertle®AI Predictive Maintenance Solution
    • CHEST-rAi™ AI Chest X-Ray Radiology Assist Suite
    • Cogmation Device Test Automation Framework
    • Connected Security Integrative Zero Trust Architecture
    • ESM Energy and Sustainability Manager
    • FlyBoard®Advanced Digital Signage Solution
    • i-BEMSIntelligent Building Experience Management
    • nBOnnB-IoT Protocol Stack
    • Fusion Pre-integrated City Operating System
    • Semiconductor IP For Security, Communication & Verification
    • UBIQWeise 2.0 Device to Cloud IoT Platform
  • Insights
    • Blogs
      • Embedding Innovation in Semiconductors with Platform Software and Middleware
      • Towards a Connected, Collaborative, & Intuitive Tomorrow: Redefining the Future with a Blink
      • 6 DevOps Trends to Watch Out for in 2023
    • News
      • L&T Technology Services unveils CoE to offer suite of automotive solutions on AWS
      • LTTS Selected as Strategic Engineering Partner to Airbus for Advanced Capabilities and Digital Manufacturing Services
      • LTTS agrees to acquire Smart World & Communication business of L&T
    • POV
      • Hidden Correlations Shaping the Future of European Enterprises
      • From the Super Bowl to 5G
      • Vehicle-to-Everything (V2X) Enabling Smarter, Safer, and Greener Transportation
      • Will 5G Technologies Drive a New Future for Media & Entertainment?
    • eBooks
      • The Art of Cyberwar
      • Digital Twin - The Future of Manufacturing
      • Digitalising Wind Energy Ecosystem
      • INDUSTRY 4.0: The Future Is Now
      • Digital Engineering Explained
      • Sustainability Engineering
  • Explore LTTS
    • About Us
    • Accolades
    • Alliances
    • Analysts
    • Board of Directors
    • Careers
    • CSR
    • Events & Webinars
    • Investors
    • Media Kit
    • Nearshore Centers
    • News & Media
    • Quality Management
    • Resources
    • Sustainability
    • Testimonials
  • Contact
 

Data Mesh

Understanding Data Warehouses, Data Lakes, & Data Mesh: A Quick Primer for Business Success

  1. Home
  2. Blogs
  3. Spotlight
  4. Understanding Data Warehouses, Data Lakes, & Data Mesh

Understanding Data Warehouses, Data Lakes, & Data Mesh: A Quick Primer for Business Success

Data Mesh
Published on: 14 Sep, 2021
3077 Views
0 comments
Share This Article:
  • Twitter
  • Facebook
  • Linked in
Data Warehouse
Data Lakes
Data Mesh
Data
Extract-Transform-Load
ETL

In 2011, James Dixon, the founder and then CTO of Pentaho coined the word “Data Lake.” His objective was to help overcome the issues with traditional data warehouses around the need for pre-categorization at the point of entry. Since then, we have surged ahead from the ideas of data warehouses and data lakes to arrive at the hottest topic on the block today – a “Data Mesh.”

With the growing global importance of data-driven decisions for ensuring organizational success, let us take a quick look at how each of these concepts can enable your operations  

Data Warehouses

A data warehouse supports integration of data from heterogeneous sources, and categorizes and stores it for future use.  The operational schemas here are prebuilt for each relevant business requirement, typically following the ETL (Extract-Transform-Load) process. 

Some of the challenges associated with a data warehouse include:

  • For every new business requirement, we need to identify the related sources and data from it to build the schema and implement the ETL process.
  • When the existing schema needs to be updated, this could pose a challenge in terms of time requirement as the volume of data can be quite large (several terabytes/petabytes)
  • Multiple data warehouses could have been created by business users for maintaining their own raw and processed data for their analytical and BI reports, leading to a duplicity of sources.

Data Lakes

Data lakes have helped solve most of the above-mentioned issue with the help of a schema-less architecture for storing any kind of data in a centralized store.  They are designed with multiple zones, starting from landing zone to receive the data (temporal data store), raw data zone to store the original data, production zone where the cleansed and processed data is stored, sensitive zone to store the sensitive data, and dev zone for the data scientist and engineer to work on. This is controlled via role-based access management. 

With data lakes, the ETL process is now changed to ELT process (Extract-Load-Transform), all data from heterogeneous sources is first collected in a single store (imagine different streams flowing into a lake). The team of data engineers, data scientists, and business analysts can the derive the key results dynamically. 

Its benefits notwithstanding, data lakes have their own set of challenges, including:

  • All data is collected into a centralized store, which can result in a data swamp in the absence of proper cataloging.
  • Data engineers who deal with a data lake are not always equipped with a deep domain understanding for deriving the target results for the business.

Data Mesh

A data mesh similar to its counterpart service. It solves the above problem by breaking the data into business domain where each user will own the relevant data as a product to ensure that each piece of information is:

  • Discoverable
  • Addressable        
  • Trustworthy and Truthful
  • Self-describing
  • Inter-operable, and
  • Secure

 

 

DataMesh

                                                                             Figure 1 Data Mesh architecture from 30K foot view by Martin Flower

                                                                                Ref: https://martinfowler.com/articles/data-monolith-to-mesh.html

 

Data mesh is a new pattern which co-exists with the data warehouse and data lakes. While data warehousing remains the overall activity, data lakes act as a wider information store, with the data mesh enabling a faster access to insights and analytics.

By introducing a virtual separation within a data lake and helping overcome the challenges associated with a data puddle or data pond, a data mesh has therefore emerged as the hottest new topic in the domain.

Authors

Ganesh S K
Ganesh S K
Senior Specialist - Digital Transformation Practices

Related Blogs

Salman Hamza Hussain
Evolution of Data Pipelines
11 Feb, 2022
Divya Bhatt
Harnessing Information to Optimize CAPEX and Asset Utilization
18 Dec, 2017
L&T Technology Services
Wearable Ultrasound Patches: Redefining Real-time Healthcare Interventions
05 Jun, 2023
Leave a Comment
About text formats

Comments

No Comments

×Explore
  • Industry
  • Spotlight
  • ×
  • Automotive
  • Consumer Electronics
  • Industrial Engineering
  • Lighting & Building Solutions
  • Media & Entertainment
  • Medical Devices
  • Oil & Gas
  • Plant Engineering
  • Power Electronics
  • Renewable Energy
  • Semiconductors
  • Industrial Products
  • Transportation
  • Telecommunications
  • 5G
  • Cloud engineering
  • Cyber security
  • Embedded systems
  • Industry 4.0
  • Smart Manufacturing
  • Smart Products
  • VLSI
  • Sustainability
  • Artificial Intelligence
  • AR/VR
  • Image Processing
  • Connected Healthcare
  • Smart Factory
  • Digital Twins
  • Building Automation
  • Autonomous Transport
  • Robotics
  • Digital Entertainment
  • Machine Learning
  • UI/UX
  • Manufacturing Automation
  • Smart Sourcing
  • Simulation
  • Software Defined Networking
  • Telehealth
  • Wearables
  • Design Thinking
  • IoT Security
  • ER&D Hackathon 2019
  • Digital Media
  • The New Normal
  • Data Mesh
  • DevOps
  • Media
  • parent-company-logo.png
  • Need Help
  • Contact Us
  •  

Contact Us

By clicking Submit, you agree to the Privacy Policy

  • Engineering the change
  • Industry
    • Communication
    • Consumer Electronics
    • Healthcare
    • Industrial Products
    • Media & Entertainment
    • Oil & Gas
    • Plant Engineering
    • Public Infrastructure & Smart Cities
    • Semiconductors
    • Software Products
    • Transportation
  • Services
    • Digital
      • Cybersecure
      • Security Monitoring
      • Security Solutions
      • Security Services
      • Immersive Experiences
      • Industry 4.0
      • Product Consulting
      • Sustainable Smart World
      • Sustainability Engineering
      • 5G
    • Products
      • CAE & CFD
      • CAx Automation
      • Cloud Engineering
      • DevOps
      • Embedded Systems
      • Engineering Analytics
      • Integrated Design, Validation & Testing
      • Lab as a Service
      • Sustenance
      • Testing
      • Testing & Validation
      • User Experience
      • VLSI
      • Voice Innovations
      • Wearables Engineering
    • Manufacturing
      • Accelerated Operations
      • Agile Supply Chain
      • Content Engineering
      • Digital Factory & Simulations
      • Line Expansion & Transfer
      • Manufacturing Automation
      • New Product Development
      • Plant Design & Engineering
      • PLM on Cloud
      • Sourcing & Procurement
    • Operations
      • CAPEX Project E/EPCM Services
      • DevOps
      • Material & Parts Management
      • Operational Excellence
      • Plant Sustenance & Management
      • Regulatory Compliance Engineering
      • Sourcing & Procurement
  • Solutions
    • AiCE
    • AiKno™
    • AnnotAi
    • ARC
    • Avertle®
    • Chest-rAi™
    • Cogmation
    • Connected Security
    • ESM
    • FlyBoard®
    • i-BEMS
    • nBOn
    • Fusion
    • Semiconductor IP
    • UBIQWeise 2.0
  • Insights
    • Blogs
    • News
    • POV
    • eBooks
  • Explore LTTS
    • About Us
    • Accolades
    • Alliances
    • Analysts
    • Board of Directors
    • Careers
    • CSR
    • Events & Webinars
    • Investors
    • Media Kit
    • Nearshore Centers
    • News & Media
    • Quality Management
    • Resources
    • Sustainability
    • Testimonials
  •  
  •  
  •  
  •  
  •  
^
  •  
  •  
  •  
  •  
  •  

© 2023 L&T Technology Services Limited. All Rights Reserved.

  • COPYRIGHT & TERMS
  • PRIVACY
  • Site Map
  • info@LTTS.com