Deleting Data in a Microservices Architecture


(17 comments)
November 5th 2021


For all the advantages of microservices, there seem to be just as many new complexities and complications. One scenario I've hit a lot recently (and haven't found a lot of great resources on) is that of deleting data. Consider a simple example:

There are three services: a Product service which manages data related to products offered, an Order service which tracks purchases of products, and a Catalog service which controls what products are published to different mediums (say, print, digital, etc.).

Now what happens if a given product needs to be deleted? Say "Widget X" is no longer a product to be sold. Or "Widget X" was accidentally added by some sleepy user, and just needs to be removed. What's the best way to handle this?

Solution 1: Just Delete It

In a monolith world, we might be able to just delete the "Widget X" row in the PRODUCT table and that would be that. If there was some foreign key reference to that product row from some row in another table (e.g. because the product was sold in a prior order, or the product was included in an existing catalog), there could either be a delete cascade that would automatically remove those rows too (often a dangerous thing), or at least there would be some foreign key constraint that would throw an error when trying to delete that product. Either way though, because all the data in a monolith resides in one database, the management of referential integrity (i.e. not leaving orphaned data) could be centrally managed.

In a microservices architecture, this obviously doesn't work. Sure, the "Widget X" row could just be thoughtlessly deleted from the PRODUCT table in the Product service, but other services would then have data that's effectively orphaned.

For example, the Order service might have a reference to "Widget X" in its ORDER_DETAILS table for an old order, and might want to fetch information on this product to show full order details. If "Widget X" was deleted from the Product service, the Order service would get a 404 on its GET request for /products/widget-x. In other words, even though each microservice is intended to be de-coupled and autonomous, it doesn't mean that it's completely independent from a data perspective. Coordination and communication is necessary if we want to preserve some level of referential integrity.

Solution 2: Synchronous Coordination

The next possibility is to have the Product service, on delete of a given product, first check with any dependent service to see if it's able to delete that product and then to carry out that "cascade" if necessary. For example, the Product Service could first check with the Order Service and Catalog Service if it's safe to delete "Widget X" (i.e. there are no references to it), and only on confirmation that it's safe would it do its delete from the PRODUCT table. Or alternatively, the Product Service could go forward with its own delete and then make a synchronous call to the Order and Catalog Services, and say "I just deleted my Widget X, so delete any reference to it from your databases".

Even if deleting this way was ok from a business rule perspective, this also is ill-advised. Creating a synchronous dependence from the Product Service to the Catalog and Order Services has major downsides - an increase in complexity, additional latency per request, and compromised resilience/availability (i.e. if the Order Service was down, the Product Service couldn't do a delete). Microservices should strive to be as independent as possible. Binding them together with runtime dependencies should only be done with great consideration, lest we just create a distributed monolith.

Solution 3: Just Don't Delete

At this point, we might observe that it doesn't really make sense to delete a product anyway! As Udi Dahan points out, in the real world there really isn't a notion of "deleting" things. Instead, the data just changes state. Employees aren't deleted, they're fired or they quit. Orders aren't deleted, they're cancelled. And Products aren't deleted, they're no longer sold. In other words, in almost all cases, it's better to support some status field on a given row than to outright delete it.

In the example here then, this would solve some of the problems. Instead of supporting a true "delete", the Product service would just expose a "decommission" (or whatever) endpoint for a product, but preserve the data in its database. In this way, the Catalog and Order Services would no longer have orphaned data - they could both still call back to the Product service to get information on "Widget X", even if it's no longer currently sold.

But this doesn't solve everything. What if the Catalog Service needs to be aware of a Product being decommissioned, so that it doesn't get shown to a customer for a given catalog? Sure, it could query the Product Service to get this information, but this could introduce a synchronous dependence from the Catalog Service to the Product Service, and introduce those same issues of complexity, latency, and resilience discussed above.

Solution 4: Asynchronous Update

To support the Catalog Service's autonomy, instead of relying on the Product Service it could maintain its own local cache of product data, keeping it sync with changes from the Product Service. When a product is decommissioned, the Product Service could emit a ProductDecommissioned event which the Catalog Service would listen for, and then it could update its own local product store. In the case of "Widget X", as soon as it was decommissioned, the Catalog service could know to not show it in its catalog. And this works. Except...

What exactly is this local cache of product data? Is it in-memory? Or a table in the Catalog Service's database? And how do we make sure it's in sync with the source of truth, the Product Service? There are a few options here, each with its own pros and cons.

If the local cache is in-memory, then this is light weight to implement, and potentially easy to synchronize (e.g. maybe there's some reconciliation process that checks the Product Service and ensures that its own data is fresh/up to date). The downside is that it gives up database-level referential integrity. If there's some product_id in a CATALOG table in its database, there would be nothing to enforce (at a database level) that that product_id is in fact valid (i.e. maps to a product_id in the Product Service database). Additionally, the in-memory cache may not tenable if the data to be cached is large.

Alternatively the local cache in the Catalog Service could also be a database table, which would solve the database-level referential integrity and storage issues from above. One problem though is complexity. Writing the SQL to seed and update a local PRODUCT table is not trivial - more code written means more code to maintain. Additionally, there's added complication around the synchronization - if there are multiple instances of a given microservice, but only one database instance, which microservice or what process is responsible for updating and synchronizing the cache?

Probably the better solution is to implement an Event Log (using Kafka, etc.), and let each microservice leverage this to keep in sync with changes to data. There is so much to consider with this option (way beyond the scope of this blog post), and a great place to start is Event Driven Microservices. While this pattern elegantly solves all of the problems above, this is, in my opinion, a level-up architecture, both to build and support, and so only something to jump into without adequate expertise and resources.

In conclusion, though I know this doesn't fully settle the issue on deleting data within Microservices, my hope is that it did highlight some of the salient issues and options. Please comment below with any tips or feedback!

I'm an "old" programmer who has been blogging for almost 20 years now. In 2017, I started Highline Solutions, a consulting company that helps with software architecture and full-stack development. I have two degrees from Carnegie Mellon University, one practical (Information and Decision Systems) and one not so much (Philosophy - thesis here). Pittsburgh, PA is my home where I live with my wife and 3 energetic boys.
I recently released a web app called TechRez, a "better resume for tech". The idea is that instead of sending out the same-old static PDF resume that's jam packed with buzz words and spans multiple pages, you can create a TechRez, which is modern, visual, and interactive. Try it out for free!
Got a Comment?
Comments (17)
TechPhantom
November 05, 2021
The issues highlighted are ofcourse valud but I also see some domain-level concerns here....

Its interesting how the author considers 'ProductsSold' as a separate service to 'Orders'. Maybe I fail to see the use case (and would gladly appreciate learning something new :) ) but I would question that, isn't a 'ProductSold' already an 'Order'? If yes then the 'ProductsSold' microservice would be redundant no? Shouldn't there be just 'Products' and 'Orders'?

Secondly, when it comes to how products are handled in the context of an order, I have seen that once a product is sold, the order microservices contains a subset of the product information as part of order items. Along with order-item details, this would include a product-id, name and other minimal details. This makes the call to '/products/widget-x' redundant as all the required information is available within the orders microservice itself.

Thirdly, I do agree with not completely deleting a product and just decommisioning them. The catalog service could then be an indexed cache that gets invalidated when there is a change in the db.
Ben
Thanks for the comment! Great point. The wording was confusing, so I actually changed it in the post. By "products sold" I meant "products that could be sold", or maybe more clear "products offered". In other words, I was thinking this was the service that has all the production information. It's a fabricated example, so probably not 100% realistic. Thanks for pointing that out though.

Good point about the the Order service including some basic info about the Product. I hit this recently with addresses and orders. The Order service keeps the full address info *at the time of the order* in its database. That way, if the user changed the address, he could still look back at a previous order and see where it was shipped to then, versus what the address is now.
Pirras Torres
December 24, 2021
Muchas gracias!!
Pawel
April 02, 2022
Hi Benn, really good article, I stumbled upon in while googling for soft deletes in microservices. I'll definitely check out your other essays. Cheers!
Ben
Thanks Pawel!
November 26, 2023


StakeOnline Casino
November 28, 2023

November 28, 2023


Discover www.strongbody.uk for an exclusive selection of B2B wholesale healthcare products. Retailers can easily place orders, waiting a smooth manufacturing process. Closing the profitability gap, our robust brands, supported by healthcare media, simplify the selling process for retailers. All StrongBody products boast high quality, unique R&D, rigorous testing, and effective marketing. StrongBody is dedicated to helping you and your customers live longer, younger, and healthier lives.
November 29, 2023

November 29, 2023



Sun52
November 30, 2023


Kantorbola adalah situs slot gacor terbaik di indonesia , kunjungi situs RTP kantor bola untuk mendapatkan informasi akurat slot dengan rtp diatas 95% . Kunjungi juga link alternatif kami di kantorbola77 dan kantorbola99
BrettEscof
December 03, 2023
VPS SERVER
Высокоскоростной доступ в Интернет: до 1000 Мбит/с
Скорость подключения к Интернету — еще один важный фактор для успеха вашего проекта. Наши VPS/VDS-серверы, адаптированные как под Windows, так и под Linux, обеспечивают доступ в Интернет со скоростью до 1000 Мбит/с, что гарантирует быструю загрузку веб-страниц и высокую производительность онлайн-приложений на обеих операционных системах.

December 04, 2023


In recent years, the landscape of digital entertainment and online gaming has expanded, with 'nhà cái' (betting houses or bookmakers) becoming a significant part. Among these, 'nhà cái RG' has emerged as a notable player. It's essential to understand what these entities are and how they operate in the modern digital world.

A 'nhà cái' essentially refers to an organization or an online platform that offers betting services. These can range from sports betting to other forms of wagering. The growth of internet connectivity and mobile technology has made these services more accessible than ever before.

Among the myriad of options, 'nhà cái RG' has been mentioned frequently. It appears to be one of the numerous online betting platforms. The 'RG' could be an abbreviation or a part of the brand's name. As with any online betting platform, it's crucial for users to understand the terms, conditions, and the legalities involved in their country or region.

The phrase 'RG nhà cái' could be interpreted as emphasizing the specific brand 'RG' within the broader category of bookmakers. This kind of focus suggests a discussion or analysis specific to that brand, possibly about its services, user experience, or its standing in the market.

Finally, 'Nhà cái Uy tín' is a term that people often look for. 'Uy tín' translates to 'reputable' or 'trustworthy.' In the context of online betting, it's a crucial aspect. Users typically seek platforms that are reliable, have transparent operations, and offer fair play. Trustworthiness also encompasses aspects like customer service, the security of transactions, and the protection of user data.

In conclusion, understanding the dynamics of 'nhà cái,' such as 'nhà cái RG,' and the importance of 'Uy tín' is vital for anyone interested in or participating in online betting. It's a world that offers entertainment and opportunities but also requires a high level of awareness and responsibility.
December 05, 2023

December 05, 2023


In recent years, the landscape of digital entertainment and online gaming has expanded, with 'nhà cái' (betting houses or bookmakers) becoming a significant part. Among these, 'nhà cái RG' has emerged as a notable player. It's essential to understand what these entities are and how they operate in the modern digital world.

A 'nhà cái' essentially refers to an organization or an online platform that offers betting services. These can range from sports betting to other forms of wagering. The growth of internet connectivity and mobile technology has made these services more accessible than ever before.

Among the myriad of options, 'nhà cái RG' has been mentioned frequently. It appears to be one of the numerous online betting platforms. The 'RG' could be an abbreviation or a part of the brand's name. As with any online betting platform, it's crucial for users to understand the terms, conditions, and the legalities involved in their country or region.

The phrase 'RG nhà cái' could be interpreted as emphasizing the specific brand 'RG' within the broader category of bookmakers. This kind of focus suggests a discussion or analysis specific to that brand, possibly about its services, user experience, or its standing in the market.

Finally, 'Nhà cái Uy tín' is a term that people often look for. 'Uy tín' translates to 'reputable' or 'trustworthy.' In the context of online betting, it's a crucial aspect. Users typically seek platforms that are reliable, have transparent operations, and offer fair play. Trustworthiness also encompasses aspects like customer service, the security of transactions, and the protection of user data.

In conclusion, understanding the dynamics of 'nhà cái,' such as 'nhà cái RG,' and the importance of 'Uy tín' is vital for anyone interested in or participating in online betting. It's a world that offers entertainment and opportunities but also requires a high level of awareness and responsibility.
December 06, 2023
In a world where holiday experiences vary, a group Virtual Christmas Cards stands as a symbol of shared joy, a collective expression of merriment during this festive and magical time.
December 06, 2023
In a world where holiday experiences vary, a group Virtual Christmas Cards stands as a symbol of shared joy, a collective expression of merriment during this festive and magical time.
December 07, 2023



1. C88 Fun - Infinite Entertainment Beckons!
C88 Fun is not just a gaming platform; it's a gateway to limitless entertainment. Featuring an intuitive interface and an eclectic game selection, C88 Fun caters to every gaming preference. From timeless classics to cutting-edge releases, C88 Fun ensures every player discovers their personal gaming haven.

2. JILI & Evo 100% Welcome Bonus - A Grand Welcome Awaits!
Embark on your gaming journey with a grand welcome from C88. New members are greeted with a 100% Welcome Bonus from JILI & Evo, doubling the thrill from the get-go. This bonus acts as a springboard for players to explore the diverse array of games available on the platform.

3. C88 First Deposit Get 2X Bonus - Double the Excitement!
Generosity is a cornerstone at C88. With the "First Deposit Get 2X Bonus" offer, players revel in double the fun on their initial deposit. This promotion enhances the gaming experience, providing more avenues to win big across various games.

4. 20 Spin Times = Get Big Bonus (8,888P) - Spin Your Way to Glory!
Spin your way to substantial bonuses with the "20 Spin Times" promotion. Accumulate spins and stand a chance to win an impressive bonus of 8,888P. This promotion adds an extra layer of excitement to the gameplay, combining luck and strategy for maximum enjoyment.

5. Daily Check-in = Turnover 5X?! - Daily Rewards Await!
Consistency reigns supreme at C88. By simply logging in daily, players not only savor the thrill of gaming but also stand a chance to multiply their turnovers by 5X. Daily check-ins bring additional perks, making every day a rewarding experience for dedicated players.

6. 7 Day Deposit 300 = Get 1,500P - Unlock Deposit Rewards!
For those hungry for opportunities, the "7 Day Deposit" promotion is a game-changer. Deposit 300 and receive a generous reward of 1,500P. This promotion encourages players to explore the platform further and maximize their gaming potential.

7. Invite 100 Users = Get 10,000 PESO - Spread the Joy!
C88 believes in the strength of community. Invite friends and fellow gamers to join the excitement, and for every 100 users, receive an incredible reward of 10,000 PESO. Sharing the joy of gaming has never been more rewarding.

8. C88 New Member Get 100% First Deposit Bonus - Exclusive Benefits!
New members are in for a treat with an exclusive 100% First Deposit Bonus. C88 ensures that everyone kicks off their gaming journey with a boost, setting the stage for an exhilarating experience filled with opportunities to win.

9. All Pass Get C88 Extra Big Bonus 1000 PESO - Unlock Unlimited Rewards!
For avid players exploring every nook and cranny of C88, the "All Pass Get C88 Extra Big Bonus" offers an additional 1000 PESO. This promotion rewards those who embrace the full spectrum of games and features available on the platform.

Ready to immerse yourself in the excitement? Visit C88 now and unlock a world of gaming like never before. Don't miss out on the excitement, bonuses, and wins that await you at C88. Join the community today, and let the games begin! #c88 #c88login #c88bet #c88bonus #c88win
December 08, 2023


tombak118