Remove Duplicates — Drops items that repeat — in this run, or seen in an earlier one — keeps one item for each value and drops the rest. Use it when a source sends the same thing twice — the same email address on two form answers, the same order in two exports — or turn on Across runs so an item handled in an earlier run is not handled again.

Who can do this

Workspace Admins and Editors, on every plan.

Steps

  1. Select + after the items to check. In the step picker, open Data and select Remove Duplicates.
  2. Choose Compare:
    • All fields (the default) — two items are the same only when every field matches.
    • Chosen fields — in Fields, type the fields that make an item the same, separated by commas, such as email, or order_no.
  3. Choose Keep — The first (the default) or The last of each set of repeats.
  4. Optional: turn on Across runs. The step then remembers the values it has let through, and drops an item whose value came in an earlier run.
  5. Select Run this step, then check Output.

Output

The items that were kept, unchanged and in their order. The step's panel says under Output how many were dropped and why: Dropped none, Dropped 4 · repeated here, or with Across runs on, Dropped 4 · 3 seen in an earlier run, 1 repeated here.

Good to know

  • It runs once for all the items that reach it.
  • Across runs remembers the last 10,000 values for 30 days, per workflow. A value older than 30 days, or pushed out by newer ones, counts as new again. A value seen again is remembered for another 30 days.
  • Values are remembered as fingerprints (hashes), not as they are, and are kept with your company's other workflow data.
  • Only live runs add to the memory. A test checks against what is remembered but adds nothing, so testing does not use up values. A Run again from a past run is a live run: its items are checked, and those seen before are dropped.
  • Changing what is compared starts the memory afresh — other fields, or switching between All fields and Chosen fields.
  • To start afresh yourself, select Forget remembered values on the step — it shows while Across runs is on. Confirm with Forget remembered values in Forget remembered values? — The next run treats every item as new. A toast says Values forgotten. It forgets everything the workflow remembers.
  • With Across runs, prefer Chosen fields. With All fields, an item that carries a time or a run number is never the same as an earlier one.
  • Comparison is exact. Rohan@Consulace.com and rohan@consulace.com are different, and so are the number 1499 and the text "1499". Use Change case first to treat letters the same — see Change, split or clean up text. The order of fields on an item does not matter.

If something goes wrong

What the run says Why What to do
Every item dropped as seen in an earlier run Across runs is on and these values came before — after a Run again from a past run, say. That is the step working. To handle them again, select Forget remembered values on the step, or turn Across runs off.
Fields is empty. Name the fields to compare, separated by commas. Chosen fields is chosen and Fields is empty. Name the fields, or choose All fields.
Needs setup: fill in fields. Chosen fields is chosen and Fields is empty at publishing. Fill it in, or choose All fields.
Remove Duplicates can't remember values here: the hashing key is not set. Tell Bizomate support. Across runs cannot work for a setting on our side. Contact Bizomate support; turn Across runs off meanwhile.