Deleting Data in a Microservices Architecture


(1 comments)
November 5th 2021


For all the advantages of microservices, there seem to be just as many new complexities and complications. One scenario I've hit a lot recently (and haven't found a lot of great resources on) is that of deleting data. Consider a simple example:

There are three services: a Product service which manages data related to products offered, an Order service which tracks purchases of products, and a Catalog service which controls what products are published to different mediums (say, print, digital, etc.).

Now what happens if a given product needs to be deleted? Say "Widget X" is no longer a product to be sold. Or "Widget X" was accidentally added by some sleepy user, and just needs to be removed. What's the best way to handle this?

Solution 1: Just Delete It

In a monolith world, we might be able to just delete the "Widget X" row in the PRODUCT table and that would be that. If there was some foreign key reference to that product row from some row in another table (e.g. because the product was sold in a prior order, or the product was included in an existing catalog), there could either be a delete cascade that would automatically remove those rows too (often a dangerous thing), or at least there would be some foreign key constraint that would throw an error when trying to delete that product. Either way though, because all the data in a monolith resides in one database, the management of referential integrity (i.e. not leaving orphaned data) could be centrally managed.

In a microservices architecture, this obviously doesn't work. Sure, the "Widget X" row could just be thoughtlessly deleted from the PRODUCT table in the Product service, but other services would then have data that's effectively orphaned.

For example, the Order service might have a reference to "Widget X" in its ORDER_DETAILS table for an old order, and might want to fetch information on this product to show full order details. If "Widget X" was deleted from the Product service, the Order service would get a 404 on its GET request for /products/widget-x. In other words, even though each microservice is intended to be de-coupled and autonomous, it doesn't mean that it's completely independent from a data perspective. Coordination and communication is necessary if we want to preserve some level of referential integrity.

Solution 2: Synchronous Coordination

The next possibility is to have the Product service, on delete of a given product, first check with any dependent service to see if it's able to delete that product and then to carry out that "cascade" if necessary. For example, the Product Service could first check with the Order Service and Catalog Service if it's safe to delete "Widget X" (i.e. there are no references to it), and only on confirmation that it's safe would it do its delete from the PRODUCT table. Or alternatively, the Product Service could go forward with its own delete and then make a synchronous call to the Order and Catalog Services, and say "I just deleted my Widget X, so delete any reference to it from your databases".

Even if deleting this way was ok from a business rule perspective, this also is ill-advised. Creating a synchronous dependence from the Product Service to the Catalog and Order Services has major downsides - an increase in complexity, additional latency per request, and compromised resilience/availability (i.e. if the Order Service was down, the Product Service couldn't do a delete). Microservices should strive to be as independent as possible. Binding them together with runtime dependencies should only be done with great consideration, lest we just create a distributed monolith.

Solution 3: Just Don't Delete

At this point, we might observe that it doesn't really make sense to delete a product anyway! As Udi Dahan points out, in the real world there really isn't a notion of "deleting" things. Instead, the data just changes state. Employees aren't deleted, they're fired or they quit. Orders aren't deleted, they're cancelled. And Products aren't deleted, they're no longer sold. In other words, in almost all cases, it's better to support some status field on a given row than to outright delete it.

In the example here then, this would solve some of the problems. Instead of supporting a true "delete", the Product service would just expose a "decommission" (or whatever) endpoint for a product, but preserve the data in its database. In this way, the Catalog and Order Services would no longer have orphaned data - they could both still call back to the Product service to get information on "Widget X", even if it's no longer currently sold.

But this doesn't solve everything. What if the Catalog Service needs to be aware of a Product being decommissioned, so that it doesn't get shown to a customer for a given catalog? Sure, it could query the Product Service to get this information, but this could introduce a synchronous dependence from the Catalog Service to the Product Service, and introduce those same issues of complexity, latency, and resilience discussed above.

Solution 4: Asynchronous Update

To support the Catalog Service's autonomy, instead of relying on the Product Service it could maintain its own local cache of product data, keeping it sync with changes from the Product Service. When a product is decommissioned, the Product Service could emit a ProductDecommissioned event which the Catalog Service would listen for, and then it could update its own local product store. In the case of "Widget X", as soon as it was decommissioned, the Catalog service could know to not show it in its catalog. And this works. Except...

What exactly is this local cache of product data? Is it in-memory? Or a table in the Catalog Service's database? And how do we make sure it's in sync with the source of truth, the Product Service? There are a few options here, each with its own pros and cons.

If the local cache is in-memory, then this is light weight to implement, and potentially easy to synchronize (e.g. maybe there's some reconciliation process that checks the Product Service and ensures that its own data is fresh/up to date). The downside is that it gives up database-level referential integrity. If there's some product_id in a CATALOG table in its database, there would be nothing to enforce (at a database level) that that product_id is in fact valid (i.e. maps to a product_id in the Product Service database). Additionally, the in-memory cache may not tenable if the data to be cached is large.

Alternatively the local cache in the Catalog Service could also be a database table, which would solve the database-level referential integrity and storage issues from above. One problem though is complexity. Writing the SQL to seed and update a local PRODUCT table is not trivial - more code written means more code to maintain. Additionally, there's added complication around the synchronization - if there are multiple instances of a given microservice, but only one database instance, which microservice or what process is responsible for updating and synchronizing the cache?

Probably the better solution is to implement an Event Log (using Kafka, etc.), and let each microservice leverage this to keep in sync with changes to data. There is so much to consider with this option (way beyond the scope of this blog post), and a great place to start is Event Driven Microservices. While this pattern elegantly solves all of the problems above, this is, in my opinion, a level-up architecture, both to build and support, and so only something to jump into without adequate expertise and resources.

In conclusion, though I know this doesn't fully settle the issue on deleting data within Microservices, my hope is that it did highlight some of the salient issues and options. Please comment below with any tips or feedback!

I'm an "old" programmer who has been blogging for almost 20 years now. In 2017, I started Highline Solutions, a consulting company that helps with software architecture and full-stack development. I have two degrees from Carnegie Mellon University, one practical (Information and Decision Systems) and one not so much (Philosophy - thesis here). Pittsburgh, PA is my home where I live with my wife and 3 energetic boys.
I recently released a web app called TechRez, a "better resume for tech". The idea is that instead of sending out the same-old static PDF resume that's jam packed with buzz words and spans multiple pages, you can create a TechRez, which is modern, visual, and interactive. Try it out for free!
Got a Comment?

Comments (1)
TechPhantom
November 05, 2021
The issues highlighted are ofcourse valud but I also see some domain-level concerns here....

Its interesting how the author considers 'ProductsSold' as a separate service to 'Orders'. Maybe I fail to see the use case (and would gladly appreciate learning something new :) ) but I would question that, isn't a 'ProductSold' already an 'Order'? If yes then the 'ProductsSold' microservice would be redundant no? Shouldn't there be just 'Products' and 'Orders'?

Secondly, when it comes to how products are handled in the context of an order, I have seen that once a product is sold, the order microservices contains a subset of the product information as part of order items. Along with order-item details, this would include a product-id, name and other minimal details. This makes the call to '/products/widget-x' redundant as all the required information is available within the orders microservice itself.

Thirdly, I do agree with not completely deleting a product and just decommisioning them. The catalog service could then be an indexed cache that gets invalidated when there is a change in the db.
Ben
Thanks for the comment! Great point. The wording was confusing, so I actually changed it in the post. By "products sold" I meant "products that could be sold", or maybe more clear "products offered". In other words, I was thinking this was the service that has all the production information. It's a fabricated example, so probably not 100% realistic. Thanks for pointing that out though.

Good point about the the Order service including some basic info about the Product. I hit this recently with addresses and orders. The Order service keeps the full address info *at the time of the order* in its database. That way, if the user changed the address, he could still look back at a previous order and see where it was shipped to then, versus what the address is now.