Skip to main content
LTTS

LTTS

Quick Links mini

  • Search
  • info@LTTS.com
  • English
  • 日本語
  • Israel
  • German
  • Contact
  • Engineering the change
  • Industry
    • Transportation
      • Aerospace
      • Automotive
      • Rail Transportation
      • Travel & Hospitality
      • Trucks & Off-Highway Vehicles
    • Industrial Products
      • Building Solutions
      • Lighting Engineering
      • Power electronics & drives
      • Renewable Energy
      • Test & Measurement
      • Power Generation & Transmission
    • Plant Engineering
      • CAPEX Project E/EPCM Services
      • Digital Engineering Services
      • Engineering Reapplication & Global Rollouts
      • Integrated Asset Management Services
      • Operational excellence
      • Plant sustenance & management
      • Regulatory compliance engineering
    • Semiconductors
      • IP Core Solutions
    • Media & Entertainment
      • Cable & Broadcasting
      • OTT
      • Rdk
      • Set-Top Boxes
      • Smart Home
    • Consumer Electronics
      • Consumer IoT
      • Enterprise Devices
      • Personal Devices
    • Healthcare
      • Healthcare Providers
      • Medical Devices
    • Telecommunications
      • 5G
      • SDN/NFV
      • Wireless Networks
      • Wireline Networks
    • Oil & Gas
      • Digitalization
      • Oil Field Equipment
      • Owners and Operators
    • Software Products
  • Services
    • Product Engineering
      • User Experience
      • Mechanical Design
        • CAE & CFD
        • CAx Automation
        • Integrated Design, Validation & Testing
      • Security
        • Security Monitoring
        • Security Solutions
        • Security Services
      • Software
        • Cloud Engineering
        • DevOps
        • Engineering Analytics
        • Immersive Experiences
        • Voice Innovations
      • Hardware
        • Embedded Systems
        • Wearables Engineering
        • Testing
        • Sustenance
      • VLSI
      • Testing & Validation
    • Manufacturing Engineering
      • PLM on Cloud
      • aspenONE on Cloud
      • Plant Design & Engineering
      • Digital Factory & Simulations
      • Line Expansion & Transfer
      • Frugal Product Design
      • Asset Care
      • Accelerated Operations
      • Manufacturing Automation
      • Agile Supply Chain
      • Content Engineering
    • Operations Engineering
      • Connected Service Support
      • Integrated Content Management
      • Intelligent Building Management
      • Sourcing & Procurement
    • Engineering Consultancy
      • Industry 4.0
      • Product Strategy
      • Security
      • Smart Factory
      • Sustainability
  • Solutions
    • FlyBoard®Advanced Digital Signage Solution
    • Connected Security Integrative Zero Trust Architecture
    • ESM Energy and Sustainability Manager
    • Cogmation Device Test Automation Framework
    • i-BEMSIntelligent Building Experience Management
    • UBIQWeise 2.0 Device to Cloud IoT Platform
    • AiKno™ Machine Learning, NLP & Vision Computing
    • Semiconductor IP For Security, Communication & Verification
    • nBOnnB-IoT Protocol Stack
    • Avertle®AI Predictive Maintenance Solution
    • ARC Asset Reliability Centre
    • CHEST-rAi™ AI Chest X-Ray Radiology Assist Suite
  • Insights
    • Blogs
      • 5G-Enabled Connected Products: Driving Next-Gen Enterprise Communications
      • Evolution of Data Pipelines
      • The Future of Television
      • Flyboard®: Shaping the Digital Signage Landscape of the Future
    • News
      • How L&T Technology Services cracked the talent code for cloud
      • Accelerated momentum seen at transportation clients: LTTS
      • In Conversation with Rajeev Gupta Chief Financial Officer, L&T Technology Services Ltd.
      • Healthcare vertical has brought in $27 millionin revenue, will nurture & grow it: LTTS MD
    • POV
      • Hidden Correlations Shaping the Future of European Enterprises
      • Solving the O-RAN Component Testing Conundrum
      • CX in the Digical Era: What Fortune 500 Companies are Doing About It
      • Re-imagining Cybersecurity through a Blockchain Lens
    • eBooks
      • The Art of Cyberwar
      • Digital Twin - The Future of Manufacturing
      • Digitalising Wind Energy Ecosystem
      • INDUSTRY 4.0: The Future Is Now
      • Digital Engineering Explained
      • Sustainability Engineering
  • Explore LTTS
    • About Us
    • Nearshore Centers
    • Testimonials
    • Events & Webinars
    • News & Media
    • Board of Directors
    • CSR
    • Accolades
    • Quality Management
    • Analysts
    • Careers
    • Investors
    • Media Kit
    • Resources
    • Alliances
    • Sustainability
  • Contact
 

Cloud engineering

Evolution of Data Pipelines

  1. Home
  2. Blogs
  3. Spotlight
  4. Evolution of Data Pipelines

Evolution of Data Pipelines

Evolution of Data Pipelines
Published on: 11 Feb, 2022
313 Views
0 comments
Share This Article:
  • Twitter
  • Facebook
  • Linked in
Data Pipelines
Data Warehouses
Cloud
Data Lakes
Data Pipeline Architectures
Data Extraction
Data Mesh

In the past, when data had to be updated, operators manually entered it into a data table. This would lead to manual user entry errors and time lag. Since this was majorly done in batches, mostly as a daily job, there was substantial lead time from the time the event occurred to the time it was reported. Decision makers had to live with this time lag and often make decisions on stale data.

Fast forward into the present and now we have real-time updates and insights which are common place requirements. Building data pipelines essentially was with the intent to move data from one layer (transactional or event sources) to data warehouses or lakes where insights where derived.

The question is with these advancements in requirements to support real-time insights, and other quality requirements, are we efficient by using traditional architectures or popularly used ETL approaches. Let’s find out!

Current state of Data Pipeline Architectures and Challenges

Data pipelines is important to any Product Digitization program. Later half of this decade we witnessed immense focus on Digital architecture and technologies being adopted. Adoption of microservices and containerization is only seeing a strong growth trajectory establishes this fact. We also see tech advancements being applied but limited to traditional “OLTP” or core service/business logic.

However, the story is bit different, when one inspects the patterns involved in Data pipelines or “OLAP” side of things. We observe limited adaptation to tech evolution seen in core services space. Most common data pipelines are built using either traditional ETL, or ELTL architectures. These are popular industry de-facto approaches. Though these do solve the larger problem at hand i.e. deriving actionable insights, but it also comes with certain limitations. Let’s explore some of these challenges:

Siloed Teams: The ETL process requires expertise or skills in data extraction or migration. This could mean the technical team is layered or structured to deal with technical nuances of the process. E.g.: An ETL engineer is many a times oblivious to insights being derived and how it is consumed by end users.

Limited Manifestation: The implementation team is now trying to fit any use-case that is desired in to the set structure or pattern. Though this is always not a problem or a wrong thing to do, there are times this can be more in-efficient. E.g.: How does one extract from an unstructured source and deal with modelling the intermediate persistence schema?

Latency: Time taken to process extract, transform and load the data many a times does introduce lags. This lag could be attributed to the fact that data is processed in batches, or the necessary intermediate load steps to persist interim results. In few business scenario, this is not acceptable.  E.g.: Data streams emanating from an IoT service is stored and batch processed at a later scheduled time. Thereby, introducing a lag from data generation to updated insights on dashboards.

Future state of Data Pipeline Architecture and Key considerations

As we see advancements in general software architecture like Microservice, Service Mesh, and so on, there is a need for similar modernization. One key approach emerging is distributing the data pipeline for the domains instead of centralized data pipeline contributing to build multiple such objects resulting in Data Mesh. Data Mesh aims to address these challenges by adopting a different approach:

  • Team or pods that are aligned on functional feature delivery
  • Treat Data as Product (discoverable, self-contained and secure)
  • Polyglot storage and communication facilitate via Mesh

Initial read on Data Mesh can be found here.

Data Mesh can be implemented in various ways. One effective pattern is to use Event driven approach and Event storming to form Data Products. A Domain can comprise of one or more Data Products. This would also mean that data can be redundant and persisted in one or more stores. This is referred to as Polyglot storage. Finally, these data products are consumed via the Mesh APIs designed along the lines of each domain requirement.

Other architectural styles include Data Lake, Data Hub and Data Virtualization. A brief comparison on these can be found here.

Some other considerations that one should evaluate:

  • Facilitate easy data access any time use standard interfaces like SQL. Tech like Snowflake, DBT, Materialize enable such real-time joins which not only enables BI, but also helps in low level plumbing of the pipeline
  • Design Data Pipelines to be robust and fault tolerant, E.g. checkpoint intermediate results where required for further analysis
  • Leverage distributed loosely-couple processing units, scalable to use polyglot technologies e.g. Spark job or Python models
  • Use Data Virtualization to mitigate bottlenecks, E.g. shorten lead time for data availability
  • Use of DataOps effectively to track and evaluate your Data pipeline performance

Conclusion

Finally, I would like to conclude with a disclaimer. This article is not to discard current architectures associated to ETL. In fact, for certain use cases like batch jobs, ETL is still a very good option to adopt. The intent here is more of a realization one would need to have based on the varied requirements and explore further architectures which could suit well for the need. In this article, we looked at few such architectures like Data Mesh and associated areas one needs to consider.

Feel free to drop your comments, feedback, queries on this article, I will try and answer each of those at my earliest convenience.

Authors

Salman Hamza Hussain, Practice Head, Digital Architecture & Analytics
Salman Hamza Hussain
Practice Head, Digital Architecture & Analytics

Related Blogs

Ganesh S K
Understanding Data Warehouses, Data Lakes, & Data Mesh: A Quick Primer for Business Success
14 Sep, 2021
Snehal Oza
Connected Manufacturing: Blurring the Lines with IIOT
06 Jun, 2017
Mayank Pandya
Virtualize Manufacturing? – A Binary Choice
07 Jun, 2017
Leave a Comment
About text formats

Comments

No Comments

×Explore
  • Industry
  • Spotlight
  • ×
  • Automotive
  • Consumer Electronics
  • Industrial Engineering
  • Industrial Products
  • Lighting & Building Solutions
  • Media & Entertainment
  • Medical Devices
  • Oil & Gas
  • Plant Engineering
  • Power Electronics
  • Renewable Energy
  • Semiconductors
  • Telecommunications
  • Transportation
  • 5G
  • AR/VR
  • Artificial Intelligence
  • Autonomous Transport
  • Building Automation
  • Cloud engineering
  • Connected Healthcare
  • Cyber security
  • Data Mesh
  • Design Thinking
  • Digital Entertainment
  • Digital Media
  • Digital Twins
  • Embedded systems
  • ER&D Hackathon 2019
  • Image Processing
  • Industry 4.0
  • IoT Security
  • Machine Learning
  • Manufacturing Automation
  • Robotics
  • Simulation
  • Smart Factory
  • Smart Manufacturing
  • Smart Products
  • Smart Sourcing
  • Software Defined Networking
  • Sustainability
  • Telehealth
  • The New Normal
  • UI/UX
  • Wearables
  • Media
  • parent-company-logo.png
  • Need Help
  • Contact Us
  •  

Contact Us

By clicking Submit, you agree to the Privacy Policy

  • Engineering the change
  • Industry
    • Transportation
    • Industrial Products
    • Plant Engineering
    • Semiconductors
    • Media & Entertainment
    • Consumer Electronics
    • Healthcare
    • Telecommunications
    • Oil & Gas
    • Software Products
  • Services
    • Products
      • CAE & CFD
      • CAx Automation
      • Cloud Engineering
      • DevOps
      • Embedded Systems
      • Engineering Analytics
      • Immersive Experiences
      • Integrated Design, Validation & Testing
      • Security Monitoring
      • Security Solutions
      • Security Services
      • Sustenance
      • Testing
      • Testing & Validation
      • User Experience
      • VLSI
      • Voice Innovations
      • Wearables Engineering
    • Manufacturing
      • PLM on Cloud
      • aspenONE on Cloud
      • Plant Design & Engineering
      • Digital Factory & Simulations
      • Line Expansion & Transfer
      • Frugal Product Design
      • Asset Care
      • Accelerated Operations
      • Manufacturing Automation
      • Agile Supply Chain
      • Content Engineering
    • Operations
      • Connected Service Support
      • Integrated Content Management
      • Intelligent Building Management
      • Sourcing & Procurement
    • Consultancy
      • Industry 4.0
      • Product Strategy
      • Security
      • Smart Factory
      • Sustainability
  • Solutions
    • i-BEMS
    • Connected Security
    • nBOn
    • UBIQWeise 2.0
    • ESM
    • AiKno™
    • Cogmation
    • Avertle®
    • ARC
    • Chest-rAi™
  • Insights
    • Blogs
    • News
    • POV
    • eBooks
  • Explore LTTS
    • About Us
    • Nearshore Centers
    • Testimonials
    • Events & Webinars
    • News & Media
    • Board of Directors
    • CSR
    • Accolades
    • Alliances
    • Quality Management
    • Sustainability
  •  
  •  
  •  
  •  
  •  
^
  •  
  •  
  •  
  •  
  •  

© 2022 L&T Technology Services Limited. All Rights Reserved.

  • COPYRIGHT & TERMS
  • Privacy Policy
  • Site Map
  • info@LTTS.com