Skip to main content
Home

Search

Header (Main)
Industry
Mobility Mobility
Automotive
Vehicle Electrification

Vehicle Electrification

Explore
Aerospace Engineering
Optimized Aircraft Downtime and Overhead Costs for a Leading Global Airline with Automated Analytics

Optimized Aircraft Downtime and Overhead Costs for a Leading Global Airline with Automated Analytics

Explore
Rail Transportation
Railway Track Inspection System

Railway Track Inspection System

Explore
Trucks & Off-Highway Vehicles
On the Road to Connected Mobility: An ER&D Perspective

On the Road to Connected Mobility: An ER&D Perspective

Explore
On the Road to Connected Mobility

On the Road to Connected Mobility: An ER&D Perspective

Explore
Sustainability Sustainability
Discrete Manufacturing & Industrial Products
Building Technology & Smart Infrastructure
Electrical Power and Controls
Industrial Machinery
Unlock the future of manufacturing with Factories of the Future

Unlock the future of manufacturing with Factories of the Future

Explore
Process Manufacturing
Oil & Gas
Chemicals
FMCG
ISG Provider Lens™ 2024: Oil and Gas Industry Services and Solutions

ISG Provider Lens™ 2024: Oil and Gas Industry Services and Solutions

Explore
L&T Technology Services and bp sign multi-year engineering services partnership

L&T Technology Services and bp sign multi-year engineering services partnership

Explore
Tech Tech
HiTech
Consumer Electronics
Media & Entertainment
NexGen Comms
Semiconductors
L&T Technology Services Secures $50 Million Deal as a Strategic Partner from a global Network Provider

L&T Technology Services Secures $50 Million Deal as a Strategic Partner from a global Network Provider

Explore
MedTech
Revolutionizing Endoscopy with Software-Defined Innovation in Collaboration with NVIDIA

Revolutionizing Endoscopy with Software-Defined Innovation in Collaboration with NVIDIA

Explore
Public Infrastructure & Smart Cities
Integrated Smart Surveillance Project

Integrated Smart Surveillance Project

Explore
Software & Platforms
LTTS & SymphonyAI to provide AI-based transformation

LTTS & SymphonyAI to provide AI-based transformation

Explore
Services
Digital Engineering & Consulting Digital Engineering & Consulting
Artificial Intelligence
Cybersecure
Security Monitoring
Security Services
Security Solutions
Immersive Experiences
Industry 4.0
Product Consulting
Sustainability Engineering
Sustainable Smart World
5G
LTTS Completes Acquisition of Intelliswift

LTTS Completes Acquisition of Intelliswift

Explore
Product Engineering Product Engineering
Software Engineering
Cloud Engineering
DevOps
Engineering Analytics
Immersive Experiences
Sustenance & Maintenance
User Experience
Voice Innovations
Embedded Engineering
Embedded Systems
Sustenance
VLSI
Wearables Engineering
Mechanical Design
CAE & CFD
CAx Automation
Testing & Validation
Integrated Design, Validation & Testing
Lab as a Service
Testing
Enabling a Paradigm Shift in Testing An LTTS AI Perspective

Enabling a Paradigm Shift in Testing An LTTS AI Perspective

Explore
Manufacturing Engineering Manufacturing Engineering
Smart Manufacturing
Accelerated Operations
Digital Factory & Simulations
Plant Design & Engineering
Supply Chain Engineering
Sourcing & Procurement
Manufacturing & Planning
Accelerated Operations
Digital Factory & Simulations
Line Expansion & Transfer
Manufacturing Automation
New Product Development
Plant Design & Engineering
PLM on Cloud
Manufacturing Execution
Agile Supply Chain
Content Engineering
Material & Parts Management
Sourcing & Procurement
Asset Reliability Centre

Asset Reliability Centre

Explore
Plant Engineering Plant Engineering
CAPEX Project E/EPCM Services
Operational Excellence
Plant Sustenance & Management
Material & Parts Management
Regulatory Compliance Engineering
ISG Provider Lens™ 2024: Oil and Gas Industry Services and Solutions

ISG Provider Lens™ 2024: Oil and Gas Industry Services and Solutions

Explore
Solutions
AiCE
AiKno®
AnnotAI
ARC
Asset Health Framework
CHEST-rAi™
Connected Security
EDGYneer
ESM
EvQUAL
FlyBoard®
Fusion
i-BEMS
Nliten
nBOn
PSM
SafeX
Semiconductor IP
Sensor & Gateway Solution
UBIQWeise 2.0
Insights
Analyst Reports
Blogs
Brochures
Case Studies
eBooks
Events
PoVs
Webinars
Whitepapers
Careers
About Us
Accolades
Alliances
Analysts
Board of Directors
CSR
Engineer At Heart
Engineering The Change
Investors
Nearshore Centers
News & Media
Quality Management
Corporate Sustainability
Testimonials
Contact
Header (Secondary)
Search
Mail
Country
EN
DE
JP
HE
Contact

Breadcrumb

  1. Blogs
  2. Spotlight
  3. Understanding Data Warehouses, Data Lakes, & Data Mesh

Understanding Data Warehouses, Data Lakes, & Data Mesh: A Quick Primer for Business Success

Ganesh S K
Ganesh S K

Senior Specialist - Digital Transformation Practices

Published on 14 Sep 2021

min read

Data Mesh

In 2011, James Dixon, the founder and then CTO of Pentaho coined the word “Data Lake.” His objective was to help overcome the issues with traditional data warehouses around the need for pre-categorization at the point of entry. Since then, we have surged ahead from the ideas of data warehouses and data lakes to arrive at the hottest topic on the block today – a “Data Mesh.”

With the growing global importance of data-driven decisions for ensuring organizational success, let us take a quick look at how each of these concepts can enable your operations  

Data Warehouses

A data warehouse supports integration of data from heterogeneous sources, and categorizes and stores it for future use.  The operational schemas here are prebuilt for each relevant business requirement, typically following the ETL (Extract-Transform-Load) process. 

Some of the challenges associated with a data warehouse include:

  • For every new business requirement, we need to identify the related sources and data from it to build the schema and implement the ETL process.
  • When the existing schema needs to be updated, this could pose a challenge in terms of time requirement as the volume of data can be quite large (several terabytes/petabytes)
  • Multiple data warehouses could have been created by business users for maintaining their own raw and processed data for their analytical and BI reports, leading to a duplicity of sources.
Data Lakes

Data lakes have helped solve most of the above-mentioned issue with the help of a schema-less architecture for storing any kind of data in a centralized store.  They are designed with multiple zones, starting from landing zone to receive the data (temporal data store), raw data zone to store the original data, production zone where the cleansed and processed data is stored, sensitive zone to store the sensitive data, and dev zone for the data scientist and engineer to work on. This is controlled via role-based access management. 

With data lakes, the ETL process is now changed to ELT process (Extract-Load-Transform), all data from heterogeneous sources is first collected in a single store (imagine different streams flowing into a lake). The team of data engineers, data scientists, and business analysts can the derive the key results dynamically. 

Its benefits notwithstanding, data lakes have their own set of challenges, including:

  • All data is collected into a centralized store, which can result in a data swamp in the absence of proper cataloging.
  • Data engineers who deal with a data lake are not always equipped with a deep domain understanding for deriving the target results for the business.
Data Mesh

A data mesh similar to its counterpart service. It solves the above problem by breaking the data into business domain where each user will own the relevant data as a product to ensure that each piece of information is:

  • Discoverable
  • Addressable        
  • Trustworthy and Truthful
  • Self-describing
  • Inter-operable, and
  • Secure

 

 

DataMesh

                                                                             Figure 1 Data Mesh architecture from 30K foot view by Martin Flower

                                                                                Ref: https://martinfowler.com/articles/data-monolith-to-mesh.html

 

Data mesh is a new pattern which co-exists with the data warehouse and data lakes. While data warehousing remains the overall activity, data lakes act as a wider information store, with the data mesh enabling a faster access to insights and analytics.

By introducing a virtual separation within a data lake and helping overcome the challenges associated with a data puddle or data pond, a data mesh has therefore emerged as the hottest new topic in the domain.

Relevant Blogs

Evolution of Data Pipelines
Harnessing Information to Optimize CAPEX and Asset Utilization
Embedded Engineering Excellence: A Cross-Industry Perspective
Explore All

Stay Relevant With Us

Subscribe to our blogs

Ganesh S K
Ganesh S K

Senior Specialist - Digital Transformation Practices

Ganesh has over 14 years of experience in Software architecture including architecting, designing and implementing various solutions.

He has experience working on Cloud (Azure / AWS), IoT, Big data, Microservices, Enterprise and Mobile applications.

In his current role as Senior Technology specialist in Digital Products and Services (DPS) at LTTS, he is part Digital Architecture practice group predominantly responsible for Digital Productization, App Modernization and Digial Platformization.

Footer Navigation
  • Industry
    • Mobility
      • Aerospace Engineering
      • Automotive
      • Rail Transportation
      • Trucks & Off-Highway Vehicles
    • Sustainability
      • Discrete Manufacturing & Industrial Products
      • Process Manufacturing
    • Tech
      • Consumer Electronics
      • MedTech
      • Media & Entertainment
      • NexGen Comms
      • Semiconductors
      • Software & Platforms
      • Public Infrastructure & Smart Cities
  • Services
    • Digital Engineering
      • Artificial Intelligence
      • Cybersecure
      • Security Monitoring
      • Security Solutions
      • Security Services
      • Immersive Experiences
      • Industry 4.0
      • Product Consulting
      • Sustainability Engineering
      • Sustainable Smart World
      • 5G
    • Product Engineering
      • CAE & CFD
      • CAx Automation
      • Software Engineering
      • Cloud Engineering
      • DevOps
      • Embedded Systems
      • Engineering Analytics
      • Integrated Design, Validation & Testing
      • Lab as a Service
      • Sustenance
      • Testing
      • Testing & Validation
      • User Experience
      • VLSI
      • Voice Innovations
      • Wearables Engineering
    • Manufacturing Engineering
      • Accelerated Operations
      • Agile Supply Chain
      • Content Engineering
      • Digital Factory & Simulations
      • Line Expansion & Transfer
      • Manufacturing Automation
      • New Product Development
      • PLM on Cloud
      • Plant Design & Engineering
      • Sourcing & Procurement
    • Plant Engineering
      • CAPEX Project E/EPCM Services
      • Material & Parts Management
      • Operational Excellence
      • Plant Sustenance & Management
      • Sourcing & Procurement
      • Regulatory Compliance Engineering
  • Engineering The Change
  • Careers
  • Engineer at Heart
  • Resources
  • Solutions
    • AiCE
    • AiKno®
    • AnnotAI
    • ARC
    • Asset Health Framework
    • CHEST-rAi™
    • Connected Security
    • EDGYneer
    • ESM
    • EvQUAL
    • FlyBoard®
    • Fusion
    • i-BEMS
    • Nliten
    • nBOn
    • PSM
    • SafeX
    • Semiconductor IP
    • Sensor & Gateway Solution
    • UBIQWeise 2.0
  • About Us
    • Accolades
    • Alliances
    • Blogs
    • Board of Directors
    • Careers
    • CSR
    • Events & Webinars
    • Investors
    • Media Kit
    • Nearshore Centers
    • News & Media
    • Quality Management
    • Resources
    • Corporate Sustainability
    • Testimonials
LTTS
  •  Twitter
  •  LinkedIn
  •  YouTube
  •  Facebook
  •  Instagram
  • Copyright & Terms
  • Privacy
  • Sitemap
  • info@ltts.com

© 2025 L&T Technology Services Limited. All Rights Reserved.