Crash-Proof

Hyrex makes background work resilient to process crashes, deploy restarts, and node failures. Tasks are durably persisted and resume safely with deterministic retries and backoff. In checkout flows, this means a payment attempt is never lost.

Keep pipelines correct during outages—use idempotency keys to protect external side effects while relying on at-least-once delivery and bounded duplication.

Why it matters

  • Payment never lost: Payment attempts are persisted and retried with backoff until captured or surfaced for action.
  • Deploys and restarts: Rolling updates won’t drop work; in-flight tasks resume after short leases expire.
  • Crashes and OOMs: Sudden exits are recovered automatically with the recorded attempt/backoff policy.
  • Network blips: Ownership is tied to renewable leases, preventing split-brain processing.

How Hyrex prevents data loss

  • Durable enqueue: Tasks are written to Postgres before execution begins.
  • Atomic state transitions: Attempts, errors, and next-run times are recorded transactionally.
  • Short leases + heartbeats: Stale ownership expires quickly; healthy workers safely take over.
  • Deterministic retries: Exponential backoff avoids thundering herds and preserves throughput.

Together these provide strong crash resilience with clear, auditable history for every run.

Correctness guarantees

  • At-least-once processing: Tasks are retried until success or an explicit terminal state.
  • Idempotent payments: Use PSP idempotency keys (e.g., paymentIntentId) to avoid double charges during retries/fail-overs.
  • Bounded duplication: Short leases and detection minimize overlaps during recovery.

Example: resilient ecommerce checkout workflow

Define a checkout workflow with retries, backoff, and idempotent payment. Hyrex guarantees a payment attempt is never lost—even across crashes and deploys.

src/hyrex/checkout.py
1# Python
2from hyrex import HyrexRegistry, get_hyrex_workflow_context
3
4hy = HyrexRegistry()
5
6@hy.task
7def create_order():
8    ctx = get_hyrex_workflow_context()
9    args = ctx.workflow_args if ctx else None
10    order_id = create_pending_order(args.cart_id, args.user_id)
11    return { 'order_id': order_id }
12
13@hy.task(
14    max_retries=5,                 # deterministic retries
15    retry_backoff=lambda a: 2**a   # exponential backoff
16)
17def process_payment():
18    """
19    Payment is never lost: Hyrex persists attempts and resumes after crashes.
20    Use payment_intent_id as the idempotency key with your PSP to avoid double-charges.
21    """
22    ctx = get_hyrex_workflow_context()
23    args = ctx.workflow_args if ctx else None
24    create_run = ctx.durable_runs.get('create_order') if ctx else None
25    create_run.refresh() if create_run else None
26    order_id = (create_run.result or {}).get('order_id') if create_run else None
27
28    intent = psp_charge(
29        idempotency_key=args.payment_intent_id,
30        amount=calculate_cart_total(args.cart_id),
31        currency='usd',
32    )
33    mark_payment_captured(order_id, intent.id)
34
35@hy.task
36def finalize_order():
37    ctx = get_hyrex_workflow_context()
38    create_run = ctx.durable_runs.get('create_order') if ctx else None
39    create_run.refresh() if create_run else None
40    order_id = (create_run.result or {}).get('order_id') if create_run else None
41    confirm_order(order_id)
42    send_order_confirmation_email(ctx.workflow_args.customer_email)
43
44@hy.workflow()
45def checkout():
46    create_order >> process_payment >> finalize_order

If a worker crashes mid-checkout, another resumes once the lease expires—capturing or surfacing the payment without duplicates when using idempotency.