Semantic Search

NeuralRepo goes beyond keyword matching. Every idea is converted into a vector embedding that captures its meaning, enabling search by concept rather than exact words.

Embedding Model

NeuralRepo uses the @cf/baai/bge-m3 model running on Cloudflare Workers AI:

Property	Value
Model	@cf/baai/bge-m3
Dimensions	1024
Multilingual	Yes (100+ languages)
Input truncated at	8,192 characters (~2,048 tokens) before the call
Timeout	30 seconds, after which the queue message is retried
Runtime	Cloudflare Workers AI

The BGE-M3 model was chosen for its strong multilingual performance, high dimension count for nuanced similarity, and native availability on Cloudflare’s infrastructure.

Vector Storage

Embeddings are stored in Cloudflare Vectorize (index neuralrepo-ideas):

Setting	Value
Metric	Cosine similarity
Dimensions	1024
Vector ID	idea_<id> — the database id, not the display number
Metadata	user_id, idea_id, status, tags (comma-joined), source, created_at

Metadata is stored alongside each vector to enable filtered queries — every query filters on user_id, and search adds status and tag filters on top.

Tags get vectors too, under tag-<user_id>-<tag_id>, embedded from the tag name plus the titles of five of its ideas.

Embedding Pipeline

When a new idea is created, the embedding process runs asynchronously:

The order matters: duplicate detection runs against the query embedding before the new vector is stored, so an idea is never its own nearest neighbour, and auto-tagging happens before the upsert so the vector’s tags metadata is complete.

Note

Embedding is asynchronous. The API returns "processing": true immediately. Semantic search results for the new idea are available within a few seconds. Until then the idea’s vectorize_id is null — that field, not processing, is how you tell the pipeline finished.

Search Pipeline

Semantic search is not a hybrid merge of keyword and vector results. It is a vector search with a reranker, and keyword search only appears as a fallback:

In words: the query is embedded, Vectorize returns between 20 and 50 candidates filtered to your own ideas, archived ideas and anything failing your status or tag filter are dropped, and everything scoring at or below min_score (default 0.2, or your search_threshold setting) is discarded. What survives is reranked by @cf/baai/bge-reranker-base and truncated to your limit. If nothing survives — or the vector search errored — keyword search over the FTS5 index answers instead.

Two details worth knowing:

• The reranker is skipped when there is nothing to rerank. With fewer candidates than the requested limit, results are returned in score order without a model call. If the rerank call fails, results also degrade to score order rather than falling back to keyword search.
• The keyword fallback is per-surface. Only the MCP tool and the AI agent fall back automatically. The REST API, the CLI, and the web app return an empty result set instead. See Search for the full table.

Duplicate Detection

When a new idea is embedded, NeuralRepo checks its five nearest neighbours — excluding ideas in shelved status — and acts on each:

Score	Action
> 0.75 (dedup threshold)	A duplicate_detections row with pending status — and, because the score also clears the related threshold, a system-created related relation
> 0.50 (related threshold)	A system-created related relation
≤ 0.50	No action

Note

The bands are not exclusive. A pair above 0.75 gets both a duplicate detection and a related relation — the two checks are independent comparisons against the same match. Both comparisons are strictly greater-than, so a score of exactly 0.75 is not a duplicate.

Both thresholds are per-user settings (dedup_threshold and related_threshold), each clamped to 0.1–0.9. Detections are written with INSERT OR IGNORE against a unique index on the idea pair, so re-running detection cannot duplicate a detection.

Tip

Reviewing detections requires Pro — on Free, GET /ideas/duplicates always returns an empty list even though detection itself still runs and still creates the rows.

Text Normalization

Before embedding, text is normalized:

1. Combine fields. Title and body are joined with a space: {title} {body}.
2. Trim. Leading and trailing whitespace is removed.
3. Collapse whitespace. Runs of spaces, tabs, and newlines collapse to a single space — so Markdown structure contributes nothing to the embedding.
4. Truncate. The result is cut to 8,192 characters. A 50,000-character body is embedded from its first ~8,000 characters only.

Ideas with only a title are still embedded. An idea that normalizes to an empty string throws, which sends the queue message to a retry.

Re-embedding

When an idea’s title or body is updated, a new embedding is generated:

1. An idea_updated queue message is dispatched.
2. The consumer re-normalizes and re-embeds the updated text.
3. System-created relations and pending duplicate detections for the idea are deleted, then re-derived from the new embedding.
4. The existing Vectorize record is upserted (replaced) with the new vector.

Metadata-only changes (status, tags) dispatch an idea_metadata_updated message that rewrites the Vectorize metadata using the stored vector, with no model call.

Archiving an idea deletes its vector (VECTORIZE.deleteByIds(["idea_<id>"])), which is why an archived idea stops appearing as anyone’s neighbour immediately — and why archiving cannot be undone by restoring a row.

Backfill

Two queue messages rebuild vectors:

• backfill_vectors re-enqueues every unarchived idea for a user as idea_created, replaying the whole pipeline.
• backfill_tag_vectors force-re-embeds the user’s tag vectors, 20 at a time.

An hourly cron also repairs individual gaps on its own, re-queueing up to 50 ideas that have no vectorize_id more than five minutes after creation. In practice that catches the failure these backfills exist for, without anyone asking.

Note

Both backfills are triggered by the maintainer, not by an API you can call. The admin surface is gated to a single account id and exposes aggregate statistics only.

Next Steps

Queue Processing The full pipeline this page's embedding step lives in.

Search Searching from each surface, including where the fallback is.

Duplicate Detection Working through detections, merging, and tuning the thresholds.

Search API GET /ideas/search — parameters, scoring, and response shape.