Skip to content
Back to System DesignAll lessons

System Design

Deduplication (Drop Duplicate Events)

O(1) time to check and record one event ID · O(n) space to remember n distinct event IDs seen so far. Real systems expire old IDs after a while, so this doesn't grow forever.

The idea, in plain English

Think of a bouncer at a club with a guest list, checking off names as people enter. If the same name tries to walk in twice — maybe they wandered off and came back — the bouncer sees the name is already checked off. They turn the person away the second time. Only the first entry counts. Deduplication does this for events flowing through a system. It remembers which events — by a unique ID — it has already handled. It silently drops any repeat, instead of processing it again.

How it works

  1. 1Give every event a unique identifier — an event ID — when it's created.
  2. 2Keep a 'seen' set of event IDs the system has already processed.
  3. 3When a new event arrives, check whether its ID is already in the seen set. If yes, drop it and do nothing. If no, process it for real and add its ID to the seen set.

When you'd use it

Use deduplication once your app is popular and events can arrive more than once. A flaky network retries a message, two servers both forward the same notification, or a queue redelivers a message it wasn't sure got processed. Deduplication makes sure handling the same event twice doesn't double-count anything — like counting a 'like' twice, or sending the same email twice.

Common beginner mistakes

  • Deduplicating by the event's content instead of a unique ID. Two genuinely different events can have identical content — like two separate $10 purchases. Content-based dedup would wrongly drop the second, real one.
  • Never expiring old IDs from the seen set. In a real, long-running system, that set grows forever and eventually eats all your memory.

Try it — edit and run

Click the code to edit · press ⌘/Ctrl+↵ to run

Editable code. Tab and Shift+Tab indent. Press Escape, then Tab, to move focus out of the editor.

Expected output — hit Run to try it
evt-1: processed
evt-2: processed
evt-1: dropped
evt-3: processed
evt-2: dropped
Processed payloads: like:post-9, like:post-3, like:post-9
Total processed: 3 out of 5 events received

Not sure this is the right topic? See the learning paths → or where this leads →