Queue Processing

NeuralRepo processes ideas asynchronously using Cloudflare Queues. When you create or update an idea, the API returns immediately while background processing handles embedding, duplicate detection, auto-tagging, and vector indexing.

Architecture Overview

The API request returns immediately (processing: true) while, on the same request, a queue producer publishes a message to the Cloudflare Queue. A queue consumer — a Workers handler — receives messages in batches of up to 10 and routes each by its message type to one of five pipelines.

Note

processing: true is a constant on the create response, not a stored column. Nothing ever writes processing: false — there is no such field on the idea. The signal that the pipeline finished is the idea’s vectorize_id becoming non-null.

Consumer configuration

Setting	Value
Queue	neuralrepo-processing
Max batch size	10 messages
Max batch timeout	30 seconds
Max retries	3
Dead-letter queue	neuralrepo-processing-dlq

Message Types

Every message has the same envelope: a type and a payload object.

idea_created

Triggered when a new idea is saved from any surface. This is the most comprehensive pipeline.

Payload:

{
  "type": "idea_created",
  "payload": {
    "idea_id": 138,
    "user_id": "9f2c7a1b4e8d40c3ab55e1f0d7c92e64",
    "title": "Browser extension for idea capture",
    "body": "Highlight text on any page and save it..."
  }
}

Processing steps, in order:

1. Fetch the idea from D1 and load the user’s threshold settings.
2. Create the parent relation, if the idea was saved with a parent_id (a branch or a sub-idea). This is why a branch’s relation appears seconds after the branch itself.
3. Normalize text — join title and body with a space, trim, collapse whitespace, truncate to 8,192 characters.
4. Generate the embedding via Workers AI (@cf/baai/bge-m3), with a 30-second timeout.
5. Duplicate detection — query Vectorize for the 5 nearest neighbours, excluding shelved ideas. Anything above the dedup threshold becomes a duplicate_detections row; anything above the related threshold becomes a system-created related relation.
6. Auto-tag, but only if the idea arrived with no tags at all.
7. Upsert the vector into Vectorize as idea_<id>, with metadata for user_id, idea_id, status, tags, source, and created_at.
8. Store vectorize_id on the idea row.
9. Ensure tag embeddings exist for the idea’s tags, creating any that are missing.

Caution

Steps 6 and 7 are in that order deliberately: auto-tags have to be applied before the vector is upserted, or the vector’s tags metadata would be written empty and tag-filtered searches would miss the idea until something else touched it.

Auto-tagging is not a BYOK feature and it does not ask permission. Both plans auto-tag, by different means, and the tags are applied directly to the idea rather than stored as suggestions:

Plan	How tags are chosen
Pro	An LLM classifier sees the title, body, your 20 most-used tags, and tags ranked by frequency across the idea’s nearest neighbours.
Free	No AI call. Tags shared by at least two of the five nearest neighbours are borrowed, up to three — and only when the idea has at least two related neighbours to borrow from.

idea_updated

Triggered when an idea’s title or body changes.

Payload:

{
  "type": "idea_updated",
  "payload": {
    "idea_id": 138,
    "user_id": "9f2c7a1b4e8d40c3ab55e1f0d7c92e64",
    "title": "Browser extension for idea capture",
    "body": "Updated body text..."
  }
}

Processing steps:

1. Reconcile the parent relation — delete existing parent relations for the idea, then re-create one if parent_id is still set.
2. Re-generate the embedding from the new text.
3. Delete system-created relations for the idea. Manual relations are left alone.
4. Delete pending duplicate detections for the idea.
5. Re-run duplicate detection, producing fresh detections and related relations.
6. Auto-tag, if the idea now has no tags.
7. Re-upsert the vector under the same ID and refresh the tag embeddings.

Caution

Steps 3 and 4 mean an edit can make auto-detected relations and pending duplicate flags disappear — they are re-derived from the new text, not merged with the old set. Manual links survive any number of edits.

idea_metadata_updated

Triggered when only status or tags change — no content change.

Payload:

{
  "type": "idea_metadata_updated",
  "payload": {
    "idea_id": 138,
    "user_id": "9f2c7a1b4e8d40c3ab55e1f0d7c92e64"
  }
}

Processing steps:

1. Fetch the existing vector by ID and reuse its values. If no vector exists yet, the message is dropped — an idea whose embedding has not landed cannot have its metadata refreshed.
2. Re-upsert with the same embedding and fresh metadata, so filtered searches (for example, “search within building status”) reflect the change.

backfill_vectors

Full re-indexing of one user’s ideas.

Payload:

{
  "type": "backfill_vectors",
  "payload": { "user_id": "9f2c7a1b4e8d40c3ab55e1f0d7c92e64" }
}

The handler does not embed anything itself. It selects every unarchived idea for the user and re-enqueues each one as an idea_created message, so each goes through the full pipeline — including duplicate detection and auto-tagging.

Caution

Backfill is heavy and it fans out: one message becomes one message per idea. Because each replays idea_created, it also re-runs duplicate detection across the account. It is not triggered by normal usage.

backfill_tag_vectors

Re-embeds a user’s tag vectors — the per-tag embeddings used for tag-similarity features — in batches of 20, each batch queueing the next until every tag is done.

{
  "type": "backfill_tag_vectors",
  "payload": { "user_id": "9f2c7a1b4e8d40c3ab55e1f0d7c92e64", "offset": 20 }
}

Unlike the per-idea path, this one forces re-embedding of tags that already have a vector.

Error Handling

The consumer wraps each message in a try/catch and treats every failure the same way:

Outcome	Behavior
Handler succeeds	message.ack() — the message is done
Handler throws (Workers AI, Vectorize, D1, anything)	message.retry() — Cloudflare redelivers it
Retried more than 3 times	Cloudflare moves it to neuralrepo-processing-dlq
Unknown message type	Logged as an error and acked — the message is discarded, not retried
Idea no longer exists	The handler returns early and acks — archiving an idea mid-flight is not an error

There is no per-error-class strategy and no application-level backoff; retry timing is Cloudflare’s. Failures are logged with the error message, and queue-send failures on the producer side are logged and swallowed rather than failing the API request — which is exactly what the hourly backfill cron below exists to repair.

Cron Triggers

Four Cloudflare Workers Cron Triggers are live in production:

Schedule	Job	Scope
0 18 * * SUN	Weekly digest	Pro users who have not opted out
0 9 * * 1	Stale-idea nudges	All plans, once per user per ~25 days
0 4 * * *	Trial expiry	Users whose trial has ended with no Stripe subscription
0 * * * *	Backfill missed ideas	Up to 50 ideas per run

Caution

docs/CRONS.md in the source repo lists the stale-idea check as disabled with a “placeholder handler”. That is out of date: the trigger is in wrangler.toml, the handler is fully implemented, and the emails send.

Weekly digest

1. Queries Pro users who haven’t opted out of the digest in Settings ▸ Notifications.
2. For each user, gathers the week’s new ideas, pending duplicates, stale ideas, and totals.
3. If the user has a BYOK key, calls the AI provider for a narrative summary; otherwise formats the stats directly. If the AI call fails, it falls back to the stats summary rather than skipping the email.
4. Sends the digest — skipped entirely when there is nothing to report.

Stale-idea nudges

Runs independently of the digest and covers all plans. It checks every user who has not opted out, skips anyone with an active KV cooldown key, and emails a list of ideas sitting in captured past the user’s stale threshold (default 30 days, configurable 7–180). The cooldown is 25 days, so the weekly cron delivers at most one nudge a month per user. The same stale definition feeds the digest’s stale section.

Backfill missed ideas

The self-healing half of the queue. Every hour it selects up to 50 unarchived ideas that have vectorize_id IS NULL and were created more than five minutes ago — the signature of a queue send that failed — and re-enqueues each as idea_created. The five-minute floor keeps it from racing ideas that are simply still in flight.

Monitoring

Queue health can be monitored through the Cloudflare dashboard:

• Messages in queue — should be near zero during normal operation.
• Messages processed per minute — spikes after bulk imports and backfills.
• Dead-letter count — should be zero. Anything in neuralrepo-processing-dlq survived three retries and indicates a processing bug.
• Consumer latency — time from message publish to processing completion.

Next Steps

Semantic Search The embedding model, vector storage, and how search actually resolves.

Duplicate Detection The user-facing side of the dedup step, including thresholds.

Weekly Digest What the two email crons send, and how to opt out.

Tags Auto-tagging from the user's side, on both plans.