Building a Deduper, a systematic approach in our system context

If you are a software engineer, you would have heard about deduplication hundreds of times. Every system that wants to scale or focus on scaling should be interested in filtering out duplicate processing as early as and in the shortest possible time in the event processing pipeline. At Walmart Global Tech, we are no different for our enterprise rule engine platform that was built to optimize a specific event (we call it as override on rules) where we needed processing that was created and updated in many different ways by the user (e.g., UI/API/spread-sheets via REST or Kafka).

Assumptions:

• If…