Skip to main content
LTTS

LTTS

Quick Links mini

  • Search
  • info@LTTS.com
  • English
  • 日本語
  • Israel
  • German
  • Contact
  • Engineering the change
  • Industry
    • Transportation
      • Aerospace
      • Automotive
      • Rail Transportation
      • Travel & Hospitality
      • Trucks & Off-Highway Vehicles
    • Industrial Products
      • Building Solutions
      • Lighting Engineering
      • Power electronics & drives
      • Renewable Energy
      • Test & Measurement
      • Power Generation & Transmission
    • Plant Engineering
      • CAPEX Project E/EPCM Services
      • Digital Engineering Services
      • Engineering Reapplication & Global Rollouts
      • Integrated Asset Management Services
      • Operational excellence
      • Plant sustenance & management
      • Regulatory compliance engineering
    • Semiconductors
      • IP Core Solutions
    • Media & Entertainment
      • Cable & Broadcasting
      • OTT
      • Rdk
      • Set-Top Boxes
      • Smart Home
    • Consumer Electronics
      • Consumer IoT
      • Enterprise Devices
      • Personal Devices
    • Healthcare
      • Healthcare Providers
      • Medical Devices
    • Telecommunications
      • 5G
      • SDN/NFV
      • Wireless Networks
      • Wireline Networks
    • Oil & Gas
      • Digitalization
      • Oil Field Equipment
      • Owners and Operators
    • Software Products
  • Services
    • Product Engineering
      • Mechanical Design
        • CAE & CFD
        • CAx Automation
        • Integrated Design, Validation & Testing
      • Security
        • Security Monitoring
        • Security Solutions
        • Security Services
      • Software
        • Cloud Engineering
        • DevOps
        • Engineering Analytics
        • Immersive Experiences
        • Voice Innovations
      • Hardware
        • Embedded Systems
        • Wearables Engineering
        • Testing
        • Sustenance
      • VLSI
      • Testing & Validation
    • Manufacturing Engineering
      • PLM on Cloud
      • aspenONE on Cloud
      • Plant Design & Engineering
      • Digital Factory & Simulations
      • Line Expansion & Transfer
      • Frugal Product Design
      • Asset Care
      • Accelerated Operations
      • Manufacturing Automation
      • Agile Supply Chain
      • Content Engineering
    • Operations Engineering
      • Connected Service Support
      • Integrated Content Management
      • Intelligent Building Management
      • Sourcing & Procurement
    • Engineering Consultancy
      • Industry 4.0
      • Product Strategy
      • Security
      • Smart Factory
  • Solutions
    • FlyBoard Advanced Digital Signage Solution
    • Connected Security Integrative Zero Trust Architecture
    • WagesApp Plant Utility Monitoring
    • Cogmation Device Test Automation Framework
    • i-BEMSIntelligent Building Experience Management
    • UBIQWeise 2.0 Device to Cloud IoT Platform
    • AiKno™ Machine Learning, NLP & Vision Computing
    • Semiconductor IP For Security, Communication & Verification
    • nBOnnB-IoT Protocol Stack
    • Avertle®AI Predictive Maintenance Solution
  • Insights
    • Blogs
      • The Sino-US Divide – Can India Harness the Opportunity to Become a Manufacturing Hub?
      • Breaking the Mold to Create Next-Gen Leaders
      • Eventual Consistency in Microservices
      • i-BEMS Shield: Building a Future-Proof Digital Workplace
    • News
      • Engineering the future: Life under ‘new normal’
      • India’s healthcare has a new best friend: Robots
      • How Digitalisation will impact marketing in 2021?
      • Robotic Automation & the 5G Revolution
    • POV
      • Managing Business Growth with Innovation Labs
      • Artificial Intelligence: The Next Frontier in Healthcare
      • Transforming Packaging Capabilities: A CPG Perspective
      • Cybersecurity in Healthcare: A Critical Need in the Digital Age
    • eBooks
      • Digitalising Wind Energy Ecosystem
      • Digital Engineering Explained
  • Explore LTTS
    • Company Profile
    • Nearshore Centers
    • Testimonials
    • Events & Webinars
    • News & Media
    • Board of Directors
    • CSR
    • Accolades
    • Quality Management
    • Analysts
    • Careers
    • Investors
    • Media Kit
    • Resources
    • Alliances
  • Contact
 

Smart Products

Eventual Consistency in Microservices

  1. Home
  2. Blogs
  3. Industry
  4. Eventual Consistency in Microservices

Eventual Consistency in Microservices

Eventual Consistency in Microservices
Published on: 10 Aug, 2020
1247 Views
0 comments
Share This Article:
  • Twitter
  • Facebook
  • Linked in
Microservices
Microservices Consistency
Microservices Inconsistency
Eventual Consistency
SAGA pattern
CDC
BASE Vs. ACID

I am sure most of us, in one way or the other have experienced data inconsistency issues in a software system. For instance, while operating an ATM machine for withdrawing money, often people encounter failed transactions due to various reasons. The ATM machine might not dispense the currency, although may proceed to show balance deduction in the accounts. When customers report the failure in the system, the bank usually responds saying that it shall take 60 mins to 24 hours to resolve the erroneous transaction record. This is a typical case of data inconsistency wherein the ATM and the bank server are out of sync for a brief period.

Let us take another case of bill payment through mobile phone apps. The user initiates the payment, the app securely connects to the bank interface, payment is processed and is being handed over to the merchant interface to confirm the payment. Let us say the merchant website is unavailable at this moment. Now, the transaction remains incomplete for a brief period until the app servers sync up with the merchant interface.

Notice, in both the above cases the data inconsistencies are due to certain portions of the system being unavailable, services not responding, network induced errors etc., and these factors are very common in large distributed software systems. Though the system eventually over a period establishes consistency, a seamless and hassle-free end-user experience is not a certainty.

Microservices

Microservices style of building software architecture adopts a distributed approach whereby each service/function of a software is highly independent, has a specific business boundary or a defined context, may or may not know of the existence of other services, is capable of managing its own data, can use different storing mechanisms and is possibly authored in different programming languages, and maintained by separate teams.

Microservices architecture uses a decentralizing approach and is much inspired by the nature. Microservices have become a popular choice for building huge enterprise systems. With the advent of cloud, elastic computing and storage, and container tech and orchestration mechanisms, there is a huge surge in building applications using Microservices architectural style.

Microservices and Eventual Consistency

By now, you might have already figured out that owing to the strong advocacy of decentralized data approach of the microservices architecture, it induces eventual data consistency in the systems. However, it would not be prudent to think that the monolithic software architecture deals with these issues seamlessly. They have their own perils.

It is thus imperative that microservice developers take these issues into consideration and be aware of the challenges that can creep in the eventual data consistency. The below listed techniques helps in managing the eventual consistency in Microservices.

Saga Pattern

Transactions that span multiple services are viewed as a saga/chain of simple atomic local transactions at each service level. Thus, a service completes and commits its transaction, notifies the next service in the chain with an event/message to trigger the next local transaction and so on and so forth. If one transaction in this chain fails, owing to any particular reason, it basically triggers an undo operation coming backward in the chain. It is thus imperative to address pattern failures while designing the architecture.

 

Saga pattern though addresses the eventual consistency issues, is more suitable for a smaller distributed architecture than large scale. The designers also need to consider that compensation calls/transactions can also fail. Thus, making it suitable for simpler and small distributed architecture.

Change Data Capture (CDC)

CDC, although well rooted and practiced extensively in Data Warehousing, can well be adopted to Microservices design effectively to ensure that the transactions span services with consistency.

The basic principle remains same at the core. A service, after its local transaction commits itself to the local DB, triggers a separate process that creates a change capture record and propagates the change capture record to the next service. This service takes in the change capture record from the previous service, processes it, commits to its local DB and in a separate process within, creates its own new change capture record and propagates it to the next set of services.

Thus, this change capture record can be propagated through the distributed services until all services are in sync without any great stress on orchestration.

Ways to implement CDC in Microservices:

Using Database Transaction Logs: Many databases offer operational logs and transactional logs. By scanning the contents of these logs and interpreting the changes, one can identify the changes made to database. This can be the change capture record to be propagated to the next service.

Using Kafka Connect and Apache Kafka: The DB changes captured made in a service can be hooked into a Kafka connecter which effectively stems the changes to the various Kafka topics for the subscribers to consume and act.

CDC in general is suitable for large distributed architecture styles, as it is not performance heavy and it can be easily enabled without wanting to have any extra changes to the schema. CDC inherently carries lower latency and enables downstream databases to quick tail the change. There is also a possibility to run stream processing on the change feed received by services.

However, the most important drawback designers feel in CDC is the inflexibility to change the schema of the services. This blocks the evolution of service DB schema to an extent. Changing schema in a service triggers change in all downstream services.

Besides these two ways, there are several other ways of achieving data sync through CDC which have been implemented by cloud infra providers.

BASE v/s ACID Debate

Having seen two popular approaches to achieve data consistency and integration and their drawbacks, let us now consider a totally new perspective and paradigm to view the issue of data consistency... “Accepting the Inconsistency” - Not all software systems/parts of the system need the data to be synced and be consistent all the time. Let us rewind back to our example of money withdrawal through ATM. We all have accepted the data inconsistency here and wait for the system to become consistent over a period.

Many business systems are more tolerant of the data inconsistencies than usually believed. This is because, businesses stress and monetize more on service availability, leading us to the long-lasting debate on prizing BASE over ACID in many enterprise scale systems.

BASE is an acronym for Basically Available, Soft state and Eventual Consistency

ACID is an acronym for Atomicity, Consistent, Isolated and Durable

This is in line with the popular CAP theorem which states, that it is impossible for a distributed data store to simultaneously provide more than two out of the following three guarantees Consistency, Availability and Partition tolerance.

Thus, software systems always prize either BASE over ACID or vice-versa suiting there need and keep parts of the system consistent.

Feel free to drop your comments, feedback, queries on this article, I will try and answer each of those at my earliest convenience.

Authors

Ashwin Atri
Ashwin Atri
Project Manager, Application Engineering

Related Blogs

Sunil Kumar Rana
DevOps Approach to Microservices Using Kubernetes
07 May, 2020
Suresh Babu P.
Infrastructure as Code: Turning Enterprise IT Architecture into Software
25 Jun, 2020
Amit Chadha
The Sino-US Divide – Can India harness the opportunity to become a manufacturing hub?
09 Nov, 2020
Leave a Comment
About text formats

Comments

No Comments

×Explore
  • Industry
  • Spotlight
  • ×
  • Automotive
  • Consumer Electronics
  • Industrial Engineering
  • Industrial Products
  • Lighting & Building Solutions
  • Media & Entertainment
  • Medical Devices
  • Oil & Gas
  • Plant Engineering
  • Power Electronics
  • Semiconductors
  • Telecommunications
  • Transportation
  • 5G
  • AR/VR
  • Artificial Intelligence
  • Autonomous Transport
  • Building Automation
  • Cloud engineering
  • Connected Healthcare
  • Cyber security
  • Design Thinking
  • Digital Entertainment
  • Digital Media
  • Digital Twins
  • Embedded systems
  • ER&D Hackathon 2019
  • Image Processing
  • Industry 4.0
  • IoT Security
  • Machine Learning
  • Manufacturing Automation
  • Robotics
  • Simulation
  • Smart Factory
  • Smart Manufacturing
  • Smart Products
  • Smart Sourcing
  • Software Defined Networking
  • Sustainability
  • Telehealth
  • The New Normal
  • UI/UX
  • Wearables
  • Media
  • parent-company-logo.png
  • Need Help
  • Contact Us
  •  

Contact Us

By clicking Submit, you agree to the Privacy Policy

  • Engineering the change
  • Industry
    • Transportation
    • Industrial Products
    • Plant Engineering
    • Semiconductors
    • Media & Entertainment
    • Consumer Electronics
    • Healthcare
    • Telecommunications
    • Oil & Gas
    • Software Products
  • Services
    • Products
      • CAE & CFD
      • CAx Automation
      • Cloud Engineering
      • DevOps
      • Embedded Systems
      • Engineering Analytics
      • Immersive Experiences
      • Integrated Design, Validation & Testing
      • Security Monitoring
      • Security Solutions
      • Security Services
      • Sustenance
      • Testing
      • Wearables Engineering
      • Voice Innovations
      • VLSI
      • Testing & Validation
    • Manufacturing
      • PLM on Cloud
      • aspenONE on Cloud
      • Plant Design & Engineering
      • Digital Factory & Simulations
      • Line Expansion & Transfer
      • Frugal Product Design
      • Asset Care
      • Accelerated Operations
      • Manufacturing Automation
      • Agile Supply Chain
      • Content Engineering
    • Operations
      • Connected Service Support
      • Integrated Content Management
      • Intelligent Building Management
      • Sourcing & Procurement
    • Consultancy
      • Industry 4.0
      • Product Strategy
      • Security
      • Smart Factory
  • Solutions
    • i-BEMS
    • Connected Security
    • nBOn
    • UBIQWeise 2.0
    • WagesApp
    • AiKno™
    • Cogmation
    • Avertle®
  • Insights
    • Blogs
    • News
    • POV
  • Explore LTTS
    • Company Profile
    • Nearshore Centers
    • Testimonials
    • Events & Webinars
    • News & Media
    • Board of Directors
    • CSR
    • Accolades
    • Alliances
    • Quality Management
  •  
  •  
  •  
  •  
^
  •  
  •  
  •  
  •  

© 2021 L&T Technology Services Limited. All Rights Reserved.

  • COPYRIGHT & TERMS
  • Privacy Policy
  • Site Map
  • info@LTTS.com