Duplicate Detection

When you save a new idea, NeuralRepo checks your existing ideas for potential duplicates using vector similarity. This keeps your repository clean without requiring you to remember every idea you have ever captured.

How It Works

Duplicate detection runs automatically as part of the idea creation pipeline:

1. A new idea is created and queued for processing.
2. The queue worker generates an embedding using the @cf/baai/bge-m3 model.
3. The embedding is compared against your five nearest ideas in Cloudflare Vectorize. Shelved ideas are excluded from the comparison.
4. Any neighbour scoring above the dedup threshold (default 0.75) gets a duplicate_detections record with status pending.
5. Any neighbour scoring above the related threshold (default 0.5) gets an automatic related relation.

Both thresholds are evaluated independently, so a pair above 0.75 clears 0.50 too: a detected duplicate is also linked with a related relation. No duplicate relation is ever created for you — that type is reserved for links you make yourself.

Note

Detection runs asynchronously, after the save. There is a brief delay — typically a few seconds — between capturing an idea and its pair appearing for review. Nothing warns you during the save itself.

Caution

Editing an idea’s title or body re-runs the whole pipeline: its pending detections and its system-created relations are deleted first, then recomputed against the edited text. Relations you created by hand survive. A duplicate you were about to merge can therefore disappear — or a new one appear — after an edit.

Who Sees Detections

Detections are computed for every account. The Pro plan is what displays them: on the free plan the duplicates list returns empty, and the Duplicates view is replaced by an upgrade panel. Nothing is lost in the meantime — the backlog is recorded and appears in full the moment you upgrade.

Viewing Duplicates

Web

Duplicates in the sidebar carries an amber count when pending detections exist. It opens a list of pairs with their similarity score, each offering Merge or Not a duplicate. The same pairs appear at the top of the Review queue.

CLI

# List all pending duplicate detections
nrepo duplicate list

# Output:
# Detection #1  Score: 0.92
#   Idea 42: "Background job processing system"
#   Idea 58: "Async worker queue for tasks"
#
# Detection #2  Score: 0.88
#   Idea 11: "Mobile habit tracker"
#   Idea 37: "Daily routine tracking app"

API

GET /api/v1/ideas/duplicates

Response:

{
  "duplicates": [
    {
      "id": 1,
      "idea_id": 58,
      "idea_title": "Async worker queue for tasks",
      "duplicate_of_id": 42,
      "duplicate_title": "Background job processing system",
      "similarity_score": 0.92,
      "status": "pending",
      "created_at": "2026-03-20T14:30:00Z"
    }
  ]
}

Resolving Duplicates

Every pending detection must be resolved with one of two actions: dismiss or merge.

Dismiss

If the detection is a false positive — the ideas are similar but distinct — dismiss it. The detection is marked as dismissed and will not appear again.

API

POST /api/v1/ideas/duplicates/1/dismiss

Web

Click the Dismiss button on the detection card in the duplicates panel.

Merge

If the ideas are truly duplicates, merge them. Merging folds one idea into the other and archives the loser.

Both merge paths share the same core:

1. Bodies are concatenated — the absorbed body is appended to the kept one, separated by a --- rule.
2. Tags are unioned — every tag from both ideas ends up on the kept idea. (The 20-tag cap is applied when you submit tags, not on merge, so a merge can leave an idea holding more than 20.)
3. The kept idea keeps the earlier created_at of the two, so it sorts where the original did.
4. The absorbed idea is archived — out of every list, search, and relation query, but retained in the database.

Caution

No supersedes relation is created. Archived ideas are filtered out of relation queries, so the link would be invisible; the merge deliberately skips it. If you want a visible pointer between two ideas, dismiss the detection and create a relation instead of merging.

Where the two paths differ:

	Resolving a detection (web, nrepo duplicate merge, API)	Merging two ideas directly (nrepo merge, API)
Which idea survives	The newer one — the idea that triggered the detection	The one you name first (keep-id)
Which is archived	The older existing idea it matched	The one you name second (absorb-id)
Relations on the absorbed idea	Left behind with the archived idea	Transferred to the kept idea
Status of the absorbed idea	Archived, status unchanged	Archived and moved to shelved

CLI

# Resolve detection #1 from the pending list
nrepo duplicate merge 1

# Or merge two ideas directly: keep 42, absorb 58
nrepo merge 42 58

API

# Resolve a detection — keeps detection.idea_id, archives duplicate_of_id
POST /api/v1/ideas/duplicates/1/merge

# Merge directly — keeps 42, absorbs 58, moves its relations across
POST /api/v1/ideas/42/merge  { "absorb_id": 58 }

Web

Click Merge on the pair, in either the Duplicates view or the Review queue. The detection is marked merged and leaves the queue.

Caution

Merging is not reversible from the UI or the API. Review the two ideas carefully first. If you are unsure, dismiss the detection — you can always create a related relation to keep the connection visible.

Configuring the Threshold

The dedup threshold controls how similar two ideas must be to trigger a duplicate detection. Adjust it in your user settings.

Setting	Range	Default	Effect
dedup_threshold	0.1 - 0.9	0.75	Higher = fewer detections, only near-exact matches
related_threshold	0.1 - 0.9	0.50	Neighbours above this get an automatic related relation

Both comparisons are strictly greater-than, and both apply only to the five nearest neighbours — raising related_threshold above dedup_threshold stops the automatic relations without stopping detection.

curl -X PATCH https://neuralrepo.com/api/v1/user/me \
  -H "X-API-Key: nrp_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"settings_json": "{\"dedup_threshold\": 0.80}"}'

Tip

If you are seeing too many false positives, raise the dedup threshold to 0.90. If real duplicates are slipping through, lower it to 0.80.

Next Steps

Review Queue The other place duplicates surface — alongside quiet ideas and new connections.

Idea Relations Relation types, cycle rules, and which relations NeuralRepo creates for you.

Search The same embeddings power semantic search — and the same threshold settings.

nrepo duplicate List, dismiss, and merge detections from the terminal.