Scale to 100+ Workers
Hyrex scales horizontally by simply adding more workers. No external brokers or custom sharding— workers coordinate via Postgres and rebalance claims to keep throughput high.
Throughput grows with worker count. Tasks remain stateless and easy to parallelize across machines.
Why horizontal scale is simple
- Stateless workers: Any worker can process any task; no sticky partitions.
- Postgres coordination: Short, renewable leases and heartbeats manage safe ownership.
- No extra infra: Skip message brokers and per-queue sharding; use the database you already run.
Fair distribution at scale
- Balanced claims: Workers share the backlog; periodic rebalancing avoids hotspots.
- Self-healing: If a worker crashes, tasks fail-over after short lease expiry.
- Backlog friendly: Large queues are processed steadily without manual sharding.
When to add more workers
- Backlog growth: Pending tasks trend upward for sustained periods.
- Dequeue latency: Time-to-start rises, indicating contention during spikes.
- Arrival rate > completion rate: Scale workers until steady state returns.