For all the advantages of microservices, there seem to be just as many new complexities and complications. One scenario I've hit a lot recently (and haven't found a lot of great resources on) is that of deleting data. Consider a simple example:
There are three services: a Product service which manages data related to products offered, an Order service which tracks purchases of products, and a Catalog service which controls what products are published to different mediums (say, print, digital, etc.).
Now what happens if a given product needs to be deleted? Say "Widget X" is no longer a product to be sold. Or "Widget X" was accidentally added by some sleepy user, and just needs to be removed. What's the best way to handle this?
Solution 1: Just Delete It
In a monolith world, we might be able to just delete the "Widget X" row in the PRODUCT table and that would be that. If there was some foreign key reference to that product row from some row in another table (e.g. because the product was sold in a prior order, or the product was included in an existing catalog), there could either be a delete cascade that would automatically remove those rows too (often a dangerous thing), or at least there would be some foreign key constraint that would throw an error when trying to delete that product. Either way though, because all the data in a monolith resides in one database, the management of referential integrity (i.e. not leaving orphaned data) could be centrally managed.
In a microservices architecture, this obviously doesn't work. Sure, the "Widget X" row could just be thoughtlessly deleted from the PRODUCT table in the Product service, but other services would then have data that's effectively orphaned.
For example, the Order service might have a reference to "Widget X" in its ORDER_DETAILS table for an old order, and might want to fetch information on this product to show full order details. If "Widget X" was deleted from the Product service, the Order service would get a 404 on its GET request for /products/widget-x. In other words, even though each microservice is intended to be de-coupled and autonomous, it doesn't mean that it's completely independent from a data perspective. Coordination and communication is necessary if we want to preserve some level of referential integrity.
Solution 2: Synchronous Coordination
The next possibility is to have the Product service, on delete of a given product, first check with any dependent service to see if it's able to delete that product and then to carry out that "cascade" if necessary. For example, the Product Service could first check with the Order Service and Catalog Service if it's safe to delete "Widget X" (i.e. there are no references to it), and only on confirmation that it's safe would it do its delete from the PRODUCT table. Or alternatively, the Product Service could go forward with its own delete and then make a synchronous call to the Order and Catalog Services, and say "I just deleted my Widget X, so delete any reference to it from your databases".
Even if deleting this way was ok from a business rule perspective, this also is ill-advised. Creating a synchronous dependence from the Product Service to the Catalog and Order Services has major downsides - an increase in complexity, additional latency per request, and compromised resilience/availability (i.e. if the Order Service was down, the Product Service couldn't do a delete). Microservices should strive to be as independent as possible. Binding them together with runtime dependencies should only be done with great consideration, lest we just create a distributed monolith.
Solution 3: Just Don't Delete
At this point, we might observe that it doesn't really make sense to delete a product anyway! As Udi Dahan points out, in the real world there really isn't a notion of "deleting" things. Instead, the data just changes state. Employees aren't deleted, they're fired or they quit. Orders aren't deleted, they're cancelled. And Products aren't deleted, they're no longer sold. In other words, in almost all cases, it's better to support some status field on a given row than to outright delete it.
In the example here then, this would solve some of the problems. Instead of supporting a true "delete", the Product service would just expose a "decommission" (or whatever) endpoint for a product, but preserve the data in its database. In this way, the Catalog and Order Services would no longer have orphaned data - they could both still call back to the Product service to get information on "Widget X", even if it's no longer currently sold.
But this doesn't solve everything. What if the Catalog Service needs to be aware of a Product being decommissioned, so that it doesn't get shown to a customer for a given catalog? Sure, it could query the Product Service to get this information, but this could introduce a synchronous dependence from the Catalog Service to the Product Service, and introduce those same issues of complexity, latency, and resilience discussed above.
Solution 4: Asynchronous Update
To support the Catalog Service's autonomy, instead of relying on the Product Service it could maintain its own local cache of product data, keeping it sync with changes from the Product Service. When a product is decommissioned, the Product Service could emit a ProductDecommissioned event which the Catalog Service would listen for, and then it could update its own local product store. In the case of "Widget X", as soon as it was decommissioned, the Catalog service could know to not show it in its catalog. And this works. Except...
What exactly is this local cache of product data? Is it in-memory? Or a table in the Catalog Service's database? And how do we make sure it's in sync with the source of truth, the Product Service? There are a few options here, each with its own pros and cons.
If the local cache is in-memory, then this is light weight to implement, and potentially easy to synchronize (e.g. maybe there's some reconciliation process that checks the Product Service and ensures that its own data is fresh/up to date). The downside is that it gives up database-level referential integrity. If there's some product_id in a CATALOG table in its database, there would be nothing to enforce (at a database level) that that product_id is in fact valid (i.e. maps to a product_id in the Product Service database). Additionally, the in-memory cache may not tenable if the data to be cached is large.
Alternatively the local cache in the Catalog Service could also be a database table, which would solve the database-level referential integrity and storage issues from above. One problem though is complexity. Writing the SQL to seed and update a local PRODUCT table is not trivial - more code written means more code to maintain. Additionally, there's added complication around the synchronization - if there are multiple instances of a given microservice, but only one database instance, which microservice or what process is responsible for updating and synchronizing the cache?
Probably the better solution is to implement an Event Log (using Kafka, etc.), and let each microservice leverage this to keep in sync with changes to data. There is so much to consider with this option (way beyond the scope of this blog post), and a great place to start is Event Driven Microservices. While this pattern elegantly solves all of the problems above, this is, in my opinion, a level-up architecture, both to build and support, and so only something to jump into without adequate expertise and resources.
In conclusion, though I know this doesn't fully settle the issue on deleting data within Microservices, my hope is that it did highlight some of the salient issues and options. Please comment below with any tips or feedback!
