Skip to main content
Home

Search

Header (Main)
Industry
Mobility Mobility
Automotive
Vehicle Electrification

Vehicle Electrification

Explore
Aerospace Engineering
Optimized Aircraft Downtime and Overhead Costs for a Leading Global Airline with Automated Analytics

Optimized Aircraft Downtime and Overhead Costs for a Leading Global Airline with Automated Analytics

Explore
Rail Transportation
Railway Track Inspection System

Railway Track Inspection System

Explore
Trucks & Off-Highway Vehicles
On the Road to Connected Mobility: An ER&D Perspective

On the Road to Connected Mobility: An ER&D Perspective

Explore
On the Road to Connected Mobility

On the Road to Connected Mobility: An ER&D Perspective

Explore
Sustainability Sustainability
Discrete Manufacturing & Industrial Products
Building Technology & Smart Infrastructure
Electrical Power and Controls
Industrial Machinery
Unlock the future of manufacturing with Factories of the Future

Unlock the future of manufacturing with Factories of the Future

Explore
Process Manufacturing
Oil & Gas
Chemicals
FMCG
ISG Provider Lens™ 2024: Oil and Gas Industry Services and Solutions

ISG Provider Lens™ 2024: Oil and Gas Industry Services and Solutions

Explore
L&T Technology Services and bp sign multi-year engineering services partnership

L&T Technology Services and bp sign multi-year engineering services partnership

Explore
Tech Tech
HiTech
Consumer Electronics
Media & Entertainment
NexGen Comms
Semiconductors
L&T Technology Services Secures $50 Million Deal as a Strategic Partner from a global Network Provider

L&T Technology Services Secures $50 Million Deal as a Strategic Partner from a global Network Provider

Explore
MedTech
Revolutionizing Endoscopy with Software-Defined Innovation in Collaboration with NVIDIA

Revolutionizing Endoscopy with Software-Defined Innovation in Collaboration with NVIDIA

Explore
Public Infrastructure & Smart Cities
Integrated Smart Surveillance Project

Integrated Smart Surveillance Project

Explore
Software & Platforms
LTTS & SymphonyAI to provide AI-based transformation

LTTS & SymphonyAI to provide AI-based transformation

Explore
Services
Digital Engineering & Consulting Digital Engineering & Consulting
Artificial Intelligence
Cybersecure
Security Monitoring
Security Services
Security Solutions
Immersive Experiences
Industry 4.0
Product Consulting
Sustainability Engineering
Sustainable Smart World
5G
LTTS Completes Acquisition of Intelliswift

LTTS Completes Acquisition of Intelliswift

Explore
Product Engineering Product Engineering
Software Engineering
Cloud Engineering
DevOps
Engineering Analytics
Immersive Experiences
Sustenance & Maintenance
User Experience
Voice Innovations
Embedded Engineering
Embedded Systems
Sustenance
VLSI
Wearables Engineering
Mechanical Design
CAE & CFD
CAx Automation
Testing & Validation
Integrated Design, Validation & Testing
Lab as a Service
Testing
Enabling a Paradigm Shift in Testing An LTTS AI Perspective

Enabling a Paradigm Shift in Testing An LTTS AI Perspective

Explore
Manufacturing Engineering Manufacturing Engineering
Smart Manufacturing
Accelerated Operations
Digital Factory & Simulations
Plant Design & Engineering
Supply Chain Engineering
Sourcing & Procurement
Manufacturing & Planning
Accelerated Operations
Digital Factory & Simulations
Line Expansion & Transfer
Manufacturing Automation
New Product Development
Plant Design & Engineering
PLM on Cloud
Manufacturing Execution
Agile Supply Chain
Content Engineering
Material & Parts Management
Sourcing & Procurement
Asset Reliability Centre

Asset Reliability Centre

Explore
Plant Engineering Plant Engineering
CAPEX Project E/EPCM Services
Operational Excellence
Plant Sustenance & Management
Material & Parts Management
Regulatory Compliance Engineering
ISG Provider Lens™ 2024: Oil and Gas Industry Services and Solutions

ISG Provider Lens™ 2024: Oil and Gas Industry Services and Solutions

Explore
Solutions
AiCE
AiKno®
AnnotAI
ARC
Asset Health Framework
CHEST-rAi™
Connected Security
EDGYneer
ESM
EvQUAL
FlyBoard®
Fusion
i-BEMS
Nliten
nBOn
PSM
SafeX
Semiconductor IP
Sensor & Gateway Solution
UBIQWeise 2.0
Insights
Analyst Reports
Blogs
Brochures
Case Studies
eBooks
Events
PoVs
Webinars
Whitepapers
Careers
About Us
Accolades
Alliances
Analysts
Board of Directors
CSR
Engineer At Heart
Engineering The Change
Investors
Nearshore Centers
News & Media
Quality Management
Corporate Sustainability
Testimonials
Contact
Header (Secondary)
Search
Mail
Country
EN
DE
JP
HE
Contact

Breadcrumb

  1. Blogs
  2. Spotlight
  3. Evolution of Data Pipelines

Evolution of Data Pipelines

Salman Hamza Hussain, Practice Head, Digital Architecture & Analytics
Salman Hamza Hussain

Practice Head, Digital Architecture & Analytics

Published on 11 Feb 2022

min read

Evolution of Data Pipelines

In the past, when data had to be updated, operators manually entered it into a data table. This would lead to manual user entry errors and time lag. Since this was majorly done in batches, mostly as a daily job, there was substantial lead time from the time the event occurred to the time it was reported. Decision makers had to live with this time lag and often make decisions on stale data.

Fast forward into the present and now we have real-time updates and insights which are common place requirements. Building data pipelines essentially was with the intent to move data from one layer (transactional or event sources) to data warehouses or lakes where insights where derived.

The question is with these advancements in requirements to support real-time insights, and other quality requirements, are we efficient by using traditional architectures or popularly used ETL approaches. Let’s find out!

Current state of Data Pipeline Architectures and Challenges

Data pipelines is important to any Product Digitization program. Later half of this decade we witnessed immense focus on Digital architecture and technologies being adopted. Adoption of microservices and containerization is only seeing a strong growth trajectory establishes this fact. We also see tech advancements being applied but limited to traditional “OLTP” or core service/business logic.

However, the story is bit different, when one inspects the patterns involved in Data pipelines or “OLAP” side of things. We observe limited adaptation to tech evolution seen in core services space. Most common data pipelines are built using either traditional ETL, or ELTL architectures. These are popular industry de-facto approaches. Though these do solve the larger problem at hand i.e. deriving actionable insights, but it also comes with certain limitations. Let’s explore some of these challenges:

Siloed Teams: The ETL process requires expertise or skills in data extraction or migration. This could mean the technical team is layered or structured to deal with technical nuances of the process. E.g.: An ETL engineer is many a times oblivious to insights being derived and how it is consumed by end users.

Limited Manifestation: The implementation team is now trying to fit any use-case that is desired in to the set structure or pattern. Though this is always not a problem or a wrong thing to do, there are times this can be more in-efficient. E.g.: How does one extract from an unstructured source and deal with modelling the intermediate persistence schema?

Latency: Time taken to process extract, transform and load the data many a times does introduce lags. This lag could be attributed to the fact that data is processed in batches, or the necessary intermediate load steps to persist interim results. In few business scenario, this is not acceptable.  E.g.: Data streams emanating from an IoT service is stored and batch processed at a later scheduled time. Thereby, introducing a lag from data generation to updated insights on dashboards.

Future state of Data Pipeline Architecture and Key considerations

As we see advancements in general software architecture like Microservice, Service Mesh, and so on, there is a need for similar modernization. One key approach emerging is distributing the data pipeline for the domains instead of centralized data pipeline contributing to build multiple such objects resulting in Data Mesh. Data Mesh aims to address these challenges by adopting a different approach:

  • Team or pods that are aligned on functional feature delivery
  • Treat Data as Product (discoverable, self-contained and secure)
  • Polyglot storage and communication facilitate via Mesh

Initial read on Data Mesh can be found here.

Data Mesh can be implemented in various ways. One effective pattern is to use Event driven approach and Event storming to form Data Products. A Domain can comprise of one or more Data Products. This would also mean that data can be redundant and persisted in one or more stores. This is referred to as Polyglot storage. Finally, these data products are consumed via the Mesh APIs designed along the lines of each domain requirement.

Other architectural styles include Data Lake, Data Hub and Data Virtualization. A brief comparison on these can be found here.

Some other considerations that one should evaluate:

  • Facilitate easy data access any time use standard interfaces like SQL. Tech like Snowflake, DBT, Materialize enable such real-time joins which not only enables BI, but also helps in low level plumbing of the pipeline
  • Design Data Pipelines to be robust and fault tolerant, E.g. checkpoint intermediate results where required for further analysis
  • Leverage distributed loosely-couple processing units, scalable to use polyglot technologies e.g. Spark job or Python models
  • Use Data Virtualization to mitigate bottlenecks, E.g. shorten lead time for data availability
  • Use of DataOps effectively to track and evaluate your Data pipeline performance
Conclusion

Finally, I would like to conclude with a disclaimer. This article is not to discard current architectures associated to ETL. In fact, for certain use cases like batch jobs, ETL is still a very good option to adopt. The intent here is more of a realization one would need to have based on the varied requirements and explore further architectures which could suit well for the need. In this article, we looked at few such architectures like Data Mesh and associated areas one needs to consider.

Feel free to drop your comments, feedback, queries on this article, I will try and answer each of those at my earliest convenience.

Relevant Blogs

Understanding Data Warehouses, Data Lakes, & Data Mesh: A Quick Primer for Business Success
Connected Manufacturing: Blurring the Lines with IIOT
Europe’s IIoT Tale: From Islands to a Unified Digital Unit
Explore All

Stay Relevant With Us

Subscribe to our blogs

Salman Hamza Hussain, Practice Head, Digital Architecture & Analytics
Salman Hamza Hussain

Practice Head, Digital Architecture & Analytics

Salman Hamza Hussain is a seasoned software leader with more than 20 years of software engineering experience. Throughout his career, he has led digital transformation initiatives spanning various industry sectors, including Industrial Products, Transportation, Medical Devices and more. He is recognized for his talent in incubating and expanding digital service lines, such as Managed Devices, Engineering Cloud, and Intuitive Products.

Footer Navigation
  • Industry
    • Mobility
      • Aerospace Engineering
      • Automotive
      • Rail Transportation
      • Trucks & Off-Highway Vehicles
    • Sustainability
      • Discrete Manufacturing & Industrial Products
      • Process Manufacturing
    • Tech
      • Consumer Electronics
      • MedTech
      • Media & Entertainment
      • NexGen Comms
      • Semiconductors
      • Software & Platforms
      • Public Infrastructure & Smart Cities
  • Services
    • Digital Engineering
      • Artificial Intelligence
      • Cybersecure
      • Security Monitoring
      • Security Solutions
      • Security Services
      • Immersive Experiences
      • Industry 4.0
      • Product Consulting
      • Sustainability Engineering
      • Sustainable Smart World
      • 5G
    • Product Engineering
      • CAE & CFD
      • CAx Automation
      • Software Engineering
      • Cloud Engineering
      • DevOps
      • Embedded Systems
      • Engineering Analytics
      • Integrated Design, Validation & Testing
      • Lab as a Service
      • Sustenance
      • Testing
      • Testing & Validation
      • User Experience
      • VLSI
      • Voice Innovations
      • Wearables Engineering
    • Manufacturing Engineering
      • Accelerated Operations
      • Agile Supply Chain
      • Content Engineering
      • Digital Factory & Simulations
      • Line Expansion & Transfer
      • Manufacturing Automation
      • New Product Development
      • PLM on Cloud
      • Plant Design & Engineering
      • Sourcing & Procurement
    • Plant Engineering
      • CAPEX Project E/EPCM Services
      • Material & Parts Management
      • Operational Excellence
      • Plant Sustenance & Management
      • Sourcing & Procurement
      • Regulatory Compliance Engineering
  • Engineering The Change
  • Careers
  • Engineer at Heart
  • Resources
  • Solutions
    • AiCE
    • AiKno®
    • AnnotAI
    • ARC
    • Asset Health Framework
    • CHEST-rAi™
    • Connected Security
    • EDGYneer
    • ESM
    • EvQUAL
    • FlyBoard®
    • Fusion
    • i-BEMS
    • Nliten
    • nBOn
    • PSM
    • SafeX
    • Semiconductor IP
    • Sensor & Gateway Solution
    • UBIQWeise 2.0
  • About Us
    • Accolades
    • Alliances
    • Blogs
    • Board of Directors
    • Careers
    • CSR
    • Events & Webinars
    • Investors
    • Media Kit
    • Nearshore Centers
    • News & Media
    • Quality Management
    • Resources
    • Corporate Sustainability
    • Testimonials
LTTS
  •  Twitter
  •  LinkedIn
  •  YouTube
  •  Facebook
  •  Instagram
  • Copyright & Terms
  • Privacy
  • Sitemap
  • info@ltts.com

© 2025 L&T Technology Services Limited. All Rights Reserved.