Be Production-Ready

Getting to production means surviving crashes, restarts, and flaky dependencies without losing work or corrupting state. A durable task queue gives your code the guarantees it needs: persistence, retries, idempotency, and isolation.

Instead of ad‑hoc crons and in‑process jobs, model background work as durable tasks with clear ownership and observability. This reduces incidental complexity while improving correctness and operability.

Why use a durable task queue?

  • Process crashes happen: Persisted tasks resume safely on healthy workers.
  • APIs are unreliable: Retries with backoff tame transient errors and rate limits.
  • Throughput varies: Queues smooth spikes; priorities protect critical paths.
  • Isolation matters: Route CPU, I/O, and third‑party calls to dedicated worker pools.
  • Operate with confidence: IDs, attempts, and timing for every run—no more guesswork.

Common pitfalls without one

  • In‑process background jobs: Lost on deploys or crashes; no handoff.
  • Ad‑hoc cron scripts: Duplicate runs, race conditions, and manual backoff.
  • Best‑effort logging: Hard to reconstruct failures or prove correctness.
  • Unbounded concurrency: Spikes take down databases or third parties.

Best practices checklist

  • Make external side effects idempotent (natural keys).
  • Set timeouts and configure deterministic retries with backoff.
  • Use queues to isolate heavy or third‑party work.
  • Record task context and return structured results.
  • Monitor attempts, latency, and failure reasons.

Example: document indexing

Fan-out embeddings per chunk and upsert idempotently into your vector store. Use queue isolation for ML/IO, timeouts, and deterministic retries.

src/hyrex/upsert_order.py
1# Python
2from hyrex import HyrexRegistry
3from pydantic import BaseModel
4from typing import List
5
6hy = HyrexRegistry()
7
8class IndexArgs(BaseModel):
9    doc_id: str
10    uri: str
11
12class Chunk(BaseModel):
13    id: str
14    text: str
15
16class EmbedChunk(BaseModel):
17    doc_id: str
18    chunk_id: str
19    text: str
20
21@hy.task(queue="ml", max_retries=5, timeout_seconds=20)
22def embed_chunk(payload: EmbedChunk):
23    vector = embed(payload.text)  # pseudo
24    vector_db_upsert(payload.doc_id, payload.chunk_id, vector, idempotency_key=payload.chunk_id)
25    return {"ok": True}
26
27@hy.task(queue="io", max_retries=3, timeout_seconds=30)
28def index_document(args: IndexArgs):
29    text = fetch_and_extract(args.uri)  # pseudo
30    chunks: List[Chunk] = chunk_text(text)
31    for c in chunks:
32        embed_chunk.send(EmbedChunk(doc_id=args.doc_id, chunk_id=c.id, text=c.text))
33    return {"chunks": len(chunks)}
34
35# Kick off indexing
36index_document.send(IndexArgs(doc_id="doc-123", uri="https://example.com/doc.pdf"))

At‑least‑once delivery pairs with idempotent side effects to guarantee correctness during retries and failovers.

Learn more