Queue Processing

NeuralRepo processes ideas asynchronously using Cloudflare Queues. When you create or update an idea, the API returns immediately while background processing handles embedding, duplicate detection, auto-tagging, and vector indexing.

Architecture Overview

The API request returns immediately (processing: true) while, on the same request, a queue producer publishes a message to the Cloudflare Queue. A queue consumer — a Workers handler — receives messages in batches and routes each by its message type to one of four pipelines.

The queue consumer runs as a Cloudflare Workers handler bound to the queue. Messages are processed in batches for efficiency.

Message Types

idea_created

Triggered when a new idea is saved. This is the most comprehensive pipeline.

Payload:

{
  "type": "idea_created",
  "idea_id": 42,
  "user_id": "user_abc123"
}

Processing steps:

1. Fetch idea from D1 (title + body).
2. Normalize text — combine title and body, trim, collapse whitespace, truncate to 8,192 characters.
3. Generate embedding via Workers AI (@cf/baai/bge-m3).
4. Upsert to Vectorize with metadata (user_id, status, tags, created_at).
5. Duplicate detection — query Vectorize for top 5 nearest neighbors. Flag pairs above 0.75 similarity threshold.
6. Auto-tag (if BYOK key available) — use AI to suggest tags based on content. Store suggestions for user review.
7. Update idea — set processing: false in D1.

idea_updated

Triggered when an idea’s title or body changes.

Payload:

{
  "type": "idea_updated",
  "idea_id": 42,
  "user_id": "user_abc123"
}

Processing steps:

1. Fetch updated idea from D1.
2. Normalize text and re-generate embedding.
3. Upsert to Vectorize — replaces the existing vector with the new one.
4. Re-run duplicate detection against the updated embedding.
5. Update idea — set processing: false.

idea_metadata_updated

Triggered when only metadata changes (status, tags) — no content change.

Payload:

{
  "type": "idea_metadata_updated",
  "idea_id": 42,
  "user_id": "user_abc123",
  "metadata": {
    "status": "building",
    "tags": ["cli", "devtools"]
  }
}

Processing steps:

1. Update Vectorize metadata only. The vector itself is not regenerated since the content has not changed.
2. This ensures filtered searches (e.g., “search within building status”) reflect the latest metadata.

backfill_vectors

Triggered manually for full re-indexing of a user’s ideas.

Payload:

{
  "type": "backfill_vectors",
  "user_id": "user_abc123"
}

Processing steps:

1. Fetch all active ideas for the user from D1.
2. Batch embed — process ideas in batches of 10 to avoid rate limits.
3. Batch upsert to Vectorize.
4. Run duplicate detection across all pairs above the threshold.

Caution

Backfill is a heavy operation. For users with 500+ ideas, it can take several minutes. It is not triggered by normal usage — only by admin action or model migration.

Error Handling

The queue consumer handles errors with a retry strategy:

Scenario	Behavior
Workers AI unavailable	Retry with exponential backoff (3 attempts)
Vectorize write failure	Retry with exponential backoff (3 attempts)
D1 read failure	Retry once, then dead-letter
Invalid message format	Log error, discard message
All retries exhausted	Message moves to dead-letter queue for manual review

Failed messages are logged with the idea ID, user ID, error message, and attempt count for debugging.

Cron Triggers

NeuralRepo uses Cloudflare Workers Cron Triggers for scheduled tasks:

Weekly Digest

Property	Value
Schedule	0 18 * * SUN (Sunday 6:00 PM UTC)
Purpose	Generate and send weekly digest emails

The digest cron:

1. Queries Pro users who haven’t opted out of the digest in Settings > Notifications.
2. For each user, gathers the week’s new ideas, pending duplicates, stale ideas, and totals.
3. If the user has a BYOK key, calls the AI provider for a narrative summary; otherwise formats the stats directly.
4. Formats and sends the digest email — skipped entirely when there is nothing to report.

Stale Idea Nudges

Property	Value
Schedule	0 9 * * 1 (Monday 9:00 AM UTC)
Purpose	Email users about ideas stuck in captured

The stale-check cron runs independently of the digest and covers all plans, not just Pro. It queries ideas in captured status (not archived) that have not been updated past the user’s stale threshold (default 30 days, configurable in Settings > Notifications) and emails a list of them. Although the cron fires weekly, a per-user KV cooldown caps delivery at roughly one nudge email per month. The same stale definition feeds the weekly digest’s stale section.

Note

Cron triggers run as Workers invocations, not as queue messages. They execute at the scheduled time and have a maximum runtime of 30 seconds (Workers CPU limit for cron).

Monitoring

Queue health can be monitored through the Cloudflare dashboard:

• Messages in queue — should be near zero during normal operation.
• Messages processed per minute — spikes after bulk imports.
• Dead-letter count — should be zero. Any messages here indicate a processing bug.
• Consumer latency — time from message publish to processing completion.