High-Available Workers

Keep your worker fleet available during deploys, failures, and traffic spikes. Hyrex workers are stateless, self-healing, and rebalance work automatically to sustain throughput—ideal for document parsing, ingestion, and backfills.

No per-queue sharding or external brokers to babysit. Scale horizontally by starting more workers— Hyrex coordinates safe ownership and fair distribution.

What high-availability means

  • No single point of failure: Any worker can disappear; others take over in-flight or pending work.
  • Zero-downtime deploys: Graceful drains and short leases ensure smooth handoff during rolling restarts.
  • Fair distribution: Workers rebalance claims to avoid hotspots and keep queues flowing.
  • Elastic scale-out: Add more workers to increase throughput immediately.

How Hyrex keeps workers available

  • Leased reservations: Tasks are claimed with short, renewable leases stored in Postgres.
  • Heartbeats: Healthy executors refresh leases; if heartbeats stop, ownership safely expires.
  • Automatic takeover: Expired leases are reacquired by healthy workers with recorded attempt/backoff.
  • Periodic rebalancing: Workers periodically redistribute claims to maintain fairness during spikes.

These mechanics provide at-least-once delivery with bounded duplication and strong operational ergonomics.

Example: scale-out document parsing

Parse and index documents at scale; Hyrex handles safe ownership, fail-over, and rebalancing.

src/hyrex/workers.py
1# Python
2from hyrex import HyrexRegistry
3
4hy = HyrexRegistry()
5
6@hy.task(queue="documents", max_concurrency=1)
7def parse_document(payload: dict):
8    """
9    Stateless and idempotent task. Hyrex provides at-least-once delivery
10    with short leases + heartbeats, so another worker can safely resume
11    after crashes or deploy restarts.
12    """
13    text = extract_text_from_document(payload["uri"])
14    index_into_search(text)
15
16# Scale by starting more processes/containers.
17# Hyrex rebalances claims to keep throughput high during spikes.

Start more workers to raise throughput; Hyrex will distribute and reclaim work automatically across your documents backlog.