Deleting Data in a Microservices Architecture


(10 comments)
November 5th 2021


For all the advantages of microservices, there seem to be just as many new complexities and complications. One scenario I've hit a lot recently (and haven't found a lot of great resources on) is that of deleting data. Consider a simple example:

There are three services: a Product service which manages data related to products offered, an Order service which tracks purchases of products, and a Catalog service which controls what products are published to different mediums (say, print, digital, etc.).

Now what happens if a given product needs to be deleted? Say "Widget X" is no longer a product to be sold. Or "Widget X" was accidentally added by some sleepy user, and just needs to be removed. What's the best way to handle this?

Solution 1: Just Delete It

In a monolith world, we might be able to just delete the "Widget X" row in the PRODUCT table and that would be that. If there was some foreign key reference to that product row from some row in another table (e.g. because the product was sold in a prior order, or the product was included in an existing catalog), there could either be a delete cascade that would automatically remove those rows too (often a dangerous thing), or at least there would be some foreign key constraint that would throw an error when trying to delete that product. Either way though, because all the data in a monolith resides in one database, the management of referential integrity (i.e. not leaving orphaned data) could be centrally managed.

In a microservices architecture, this obviously doesn't work. Sure, the "Widget X" row could just be thoughtlessly deleted from the PRODUCT table in the Product service, but other services would then have data that's effectively orphaned.

For example, the Order service might have a reference to "Widget X" in its ORDER_DETAILS table for an old order, and might want to fetch information on this product to show full order details. If "Widget X" was deleted from the Product service, the Order service would get a 404 on its GET request for /products/widget-x. In other words, even though each microservice is intended to be de-coupled and autonomous, it doesn't mean that it's completely independent from a data perspective. Coordination and communication is necessary if we want to preserve some level of referential integrity.

Solution 2: Synchronous Coordination

The next possibility is to have the Product service, on delete of a given product, first check with any dependent service to see if it's able to delete that product and then to carry out that "cascade" if necessary. For example, the Product Service could first check with the Order Service and Catalog Service if it's safe to delete "Widget X" (i.e. there are no references to it), and only on confirmation that it's safe would it do its delete from the PRODUCT table. Or alternatively, the Product Service could go forward with its own delete and then make a synchronous call to the Order and Catalog Services, and say "I just deleted my Widget X, so delete any reference to it from your databases".

Even if deleting this way was ok from a business rule perspective, this also is ill-advised. Creating a synchronous dependence from the Product Service to the Catalog and Order Services has major downsides - an increase in complexity, additional latency per request, and compromised resilience/availability (i.e. if the Order Service was down, the Product Service couldn't do a delete). Microservices should strive to be as independent as possible. Binding them together with runtime dependencies should only be done with great consideration, lest we just create a distributed monolith.

Solution 3: Just Don't Delete

At this point, we might observe that it doesn't really make sense to delete a product anyway! As Udi Dahan points out, in the real world there really isn't a notion of "deleting" things. Instead, the data just changes state. Employees aren't deleted, they're fired or they quit. Orders aren't deleted, they're cancelled. And Products aren't deleted, they're no longer sold. In other words, in almost all cases, it's better to support some status field on a given row than to outright delete it.

In the example here then, this would solve some of the problems. Instead of supporting a true "delete", the Product service would just expose a "decommission" (or whatever) endpoint for a product, but preserve the data in its database. In this way, the Catalog and Order Services would no longer have orphaned data - they could both still call back to the Product service to get information on "Widget X", even if it's no longer currently sold.

But this doesn't solve everything. What if the Catalog Service needs to be aware of a Product being decommissioned, so that it doesn't get shown to a customer for a given catalog? Sure, it could query the Product Service to get this information, but this could introduce a synchronous dependence from the Catalog Service to the Product Service, and introduce those same issues of complexity, latency, and resilience discussed above.

Solution 4: Asynchronous Update

To support the Catalog Service's autonomy, instead of relying on the Product Service it could maintain its own local cache of product data, keeping it sync with changes from the Product Service. When a product is decommissioned, the Product Service could emit a ProductDecommissioned event which the Catalog Service would listen for, and then it could update its own local product store. In the case of "Widget X", as soon as it was decommissioned, the Catalog service could know to not show it in its catalog. And this works. Except...

What exactly is this local cache of product data? Is it in-memory? Or a table in the Catalog Service's database? And how do we make sure it's in sync with the source of truth, the Product Service? There are a few options here, each with its own pros and cons.

If the local cache is in-memory, then this is light weight to implement, and potentially easy to synchronize (e.g. maybe there's some reconciliation process that checks the Product Service and ensures that its own data is fresh/up to date). The downside is that it gives up database-level referential integrity. If there's some product_id in a CATALOG table in its database, there would be nothing to enforce (at a database level) that that product_id is in fact valid (i.e. maps to a product_id in the Product Service database). Additionally, the in-memory cache may not tenable if the data to be cached is large.

Alternatively the local cache in the Catalog Service could also be a database table, which would solve the database-level referential integrity and storage issues from above. One problem though is complexity. Writing the SQL to seed and update a local PRODUCT table is not trivial - more code written means more code to maintain. Additionally, there's added complication around the synchronization - if there are multiple instances of a given microservice, but only one database instance, which microservice or what process is responsible for updating and synchronizing the cache?

Probably the better solution is to implement an Event Log (using Kafka, etc.), and let each microservice leverage this to keep in sync with changes to data. There is so much to consider with this option (way beyond the scope of this blog post), and a great place to start is Event Driven Microservices. While this pattern elegantly solves all of the problems above, this is, in my opinion, a level-up architecture, both to build and support, and so only something to jump into without adequate expertise and resources.

In conclusion, though I know this doesn't fully settle the issue on deleting data within Microservices, my hope is that it did highlight some of the salient issues and options. Please comment below with any tips or feedback!

I'm an "old" programmer who has been blogging for almost 20 years now. In 2017, I started Highline Solutions, a consulting company that helps with software architecture and full-stack development. I have two degrees from Carnegie Mellon University, one practical (Information and Decision Systems) and one not so much (Philosophy - thesis here). Pittsburgh, PA is my home where I live with my wife and 3 energetic boys.
I recently released a web app called TechRez, a "better resume for tech". The idea is that instead of sending out the same-old static PDF resume that's jam packed with buzz words and spans multiple pages, you can create a TechRez, which is modern, visual, and interactive. Try it out for free!
Got a Comment?
Comments (10)
TechPhantom
November 05, 2021
The issues highlighted are ofcourse valud but I also see some domain-level concerns here....

Its interesting how the author considers 'ProductsSold' as a separate service to 'Orders'. Maybe I fail to see the use case (and would gladly appreciate learning something new :) ) but I would question that, isn't a 'ProductSold' already an 'Order'? If yes then the 'ProductsSold' microservice would be redundant no? Shouldn't there be just 'Products' and 'Orders'?

Secondly, when it comes to how products are handled in the context of an order, I have seen that once a product is sold, the order microservices contains a subset of the product information as part of order items. Along with order-item details, this would include a product-id, name and other minimal details. This makes the call to '/products/widget-x' redundant as all the required information is available within the orders microservice itself.

Thirdly, I do agree with not completely deleting a product and just decommisioning them. The catalog service could then be an indexed cache that gets invalidated when there is a change in the db.
Ben
Thanks for the comment! Great point. The wording was confusing, so I actually changed it in the post. By "products sold" I meant "products that could be sold", or maybe more clear "products offered". In other words, I was thinking this was the service that has all the production information. It's a fabricated example, so probably not 100% realistic. Thanks for pointing that out though.

Good point about the the Order service including some basic info about the Product. I hit this recently with addresses and orders. The Order service keeps the full address info *at the time of the order* in its database. That way, if the user changed the address, he could still look back at a previous order and see where it was shipped to then, versus what the address is now.
Pirras Torres
December 24, 2021
Muchas gracias!!
Pawel
April 02, 2022
Hi Benn, really good article, I stumbled upon in while googling for soft deletes in microservices. I'll definitely check out your other essays. Cheers!
Ben
Thanks Pawel!
May 22, 2023


Нейросеть рисует по описанию
May 27, 2023

May 28, 2023


A Clínica Dr. Günther Heller é uma referência em tratamentos de Invisalign, ClearCorrect e implantes dentais. Liderada pelo renomado Dr. Heller, a clínica oferece atendimento especializado e personalizado, utilizando tecnologia avançada para criar soluções personalizadas. Com expertise em ortodontia, os tratamentos de Invisalign e ClearCorrect são discretos e eficazes para corrigir problemas de alinhamento dental. Além disso, a clínica é reconhecida pela excelência em implantes dentais, oferecendo resultados duradouros e esteticamente agradáveis. Com uma equipe qualificada e liderança experiente, a Clínica Dr. Günther Heller é a escolha certa para transformar sorrisos e obter uma saúde bucal de qualidade.
May 29, 2023


B52 là một trò chơi đổi thưởng phổ biến, được cung cấp trên các nền tảng iOS và Android. Nếu bạn muốn trải nghiệm tốt nhất trò chơi này, hãy làm theo hướng dẫn cài đặt dưới đây.

HƯỚNG DẪN CÀI ĐẶT TRÊN ANDROID:

Nhấn vào "Tải bản cài đặt" cho thiết bị Android của bạn.
Mở file APK vừa tải về trên điện thoại.
Bật tùy chọn "Cho phép cài đặt ứng dụng từ nguồn khác CHPLAY" trong cài đặt thiết bị.
Chọn "OK" và tiến hành cài đặt.
HƯỚNG DẪN CÀI ĐẶT TRÊN iOS:

Nhấn vào "Tải bản cài đặt" dành cho iOS để tải trực tiếp.
Chọn "Mở", sau đó chọn "Cài đặt".
Truy cập vào "Cài đặt" trên iPhone, chọn "Cài đặt chung" - "Quản lý VPN & Thiết bị".
Chọn "Ứng dụng doanh nghiệp" hiển thị và sau đó chọn "Tin cậy..."
B52 là một trò chơi đổi thưởng đáng chơi và có uy tín. Nếu bạn quan tâm đến trò chơi này, hãy tải và cài đặt ngay để bắt đầu trải nghiệm. Chúc bạn chơi game vui vẻ và may mắn!
May 30, 2023

June 01, 2023




In the ever-evolving realm of online gaming, Jilibet stands as a beacon of excellence, captivating players with its unrivaled offerings and immersive experiences. With its commitment to innovation, a vast array of games, and a user-centric approach, Jilibet has emerged as a trailblazer in the world of online entertainment. In this article, we explore the unique attributes that make Jilibet a force to be reckoned with and why it continues to redefine the landscape of online gaming.

Unparalleled Gaming Variety:
Jilibet takes pride in its diverse and expansive collection of games, designed to cater to the unique preferences and interests of players. From classic slots that evoke a sense of nostalgia to cutting-edge video slots that push the boundaries of creativity, Jilibet offers an extensive library of titles from renowned software providers. Additionally, players can engage in captivating table games, experience the thrill of live dealer interactions, or try their luck with progressive jackpots that promise life-changing wins. With Jilibet, the possibilities for exhilarating gameplay are virtually endless.

Innovative Technological Advancements:
Jilibet stays at the forefront of technological advancements, harnessing the power of state-of-the-art software to enhance the gaming experience. The platform's intuitive interface ensures seamless navigation, allowing players to effortlessly explore the vast gaming landscape. Moreover, Jilibet leverages cutting-edge graphics, animations, and sound effects to create an immersive environment that transcends the boundaries of traditional gaming.

Unmatched Bonuses and Rewards:
Jilibet understands the significance of rewarding its players generously. From the moment you join, you'll be greeted with enticing bonuses that amplify your gaming potential. Whether it's a generous welcome bonus, exciting promotions, or exclusive tournaments, Jilibet consistently goes the extra mile to keep players engaged and rewarded. These bonuses serve as a catalyst for thrilling gameplay and offer the opportunity to win big while enjoying your favorite games.

Commitment to Safety and Security:
Jilibet places paramount importance on player safety and security. With advanced encryption technology and robust security measures, the platform ensures that personal and financial information remains protected at all times. By adhering to strict regulatory standards and employing fair gaming practices, Jilibet guarantees a level playing field, providing players with a secure and transparent gaming environment.

Exceptional Customer Support:
At Jilibet, customer satisfaction is paramount. The platform boasts a dedicated support team that is readily available to assist players with any queries or concerns they may have. Whether it's technical assistance or guidance on navigating the platform, Jilibet's customer support ensures that players receive prompt and personalized attention, ensuring a seamless and enjoyable gaming experience.

Conclusion:

Jilibet stands tall as an epitome of online gaming excellence, offering an unparalleled selection of games, innovative features, and a commitment to player satisfaction. With its diverse gaming variety, technological advancements, generous bonuses, and unwavering focus on security and customer support, Jilibet continues to set new standards in the world of online gaming. Embark on a thrilling journey with Jilibet and discover the limitless possibilities and excitement that await in this exceptional gaming paradise
June 01, 2023




In the ever-evolving landscape of online gaming, Jili Money emerges as a trailblazer, captivating players with its unrivaled combination of immersive gameplay and exceptional rewards. This unique platform has garnered a loyal following by revolutionizing the way gamers experience virtual entertainment. In this article, we delve into the distinctive qualities that make Jili Money stand out, redefining the very essence of online gaming.

Unleashing a World of Endless Adventure:
Jili Money presents an extensive collection of top games that transport players into a realm of boundless excitement. With meticulously crafted visuals, captivating storylines, and interactive gameplay, Jili Money's games transcend mere entertainment, immersing players in unforgettable experiences. From the adrenaline-pumping thrill of high-stakes slot machines to the strategic challenges of skill-based games, Jili Money offers a comprehensive selection that caters to all gaming preferences.

Seamless Access, Unparalleled Convenience:
Jili Money revolutionizes the gaming experience with its state-of-the-art mobile app. Designed with the user in mind, the Jili App seamlessly integrates with players' devices, providing unparalleled convenience and accessibility. Now, gamers can indulge in their favorite Jili Money games on the go, breaking free from the confines of traditional gaming setups. With just a few taps, players can dive into a world of excitement, no matter where they are.

Rewards that Transcend Expectations:
Jili Money believes in rewarding its players generously, elevating the thrill of gaming to new heights. Upon signing up, players are greeted with an extraordinary welcome bonus, designed to amplify their gaming journey from the very beginning. The platform also offers free credit opportunities, enabling players to explore different games and strategies without risking their own funds. This commitment to providing a risk-free environment fosters an atmosphere of exploration and experimentation.

Unlocking the Gateway to Fortune:
Jili Money tantalizes players with its exclusive free spin tickets, opening the doors to captivating bonus rounds and untold riches. With every spin, players have the chance to uncover hidden treasures, trigger exciting features, and amass substantial winnings. The allure of these free spin tickets adds an extra layer of anticipation to the gaming experience, making each moment spent on Jili Money an exhilarating endeavor.

Creating a Community of Champions:
Jili Money not only provides an immersive gaming experience but also cultivates a sense of camaraderie among its players. The platform hosts engaging events and competitions, where gamers can showcase their skills and compete for coveted prizes. These gatherings foster a vibrant community, where players can connect, share experiences, and celebrate victories together. Jili Money believes in nurturing a supportive environment that encourages players to excel and grow.

The Future of Gaming Awaits:
Jili Money stands at the forefront of the online gaming revolution, redefining the landscape with its unique blend of rewards and immersive gameplay. With its unparalleled selection of games, seamless mobile app, and commitment to player satisfaction, Jili Money sets a new standard for the online gaming experience. Prepare to embark on a journey where adventure, rewards, and camaraderie intertwine, as Jili Money paves the way to an extraordinary future in gaming.

Conclusion:
Jili Money has reshaped the online gaming landscape, offering an unparalleled fusion of immersive gameplay and captivating rewards. With its diverse game selection, mobile app convenience, and dedication to player satisfaction, Jili Money provides an unrivaled gaming experience. Whether you're seeking thrilling adventures, substantial rewards, or a vibrant community, Jili Money is the ultimate destination for gamers who dare to redefine what is possible in the realm of online gaming. Embrace the future of gaming with Jili Money, where limitless adventures await.