xVault Docs
Operations

Keeper and oracle operations

Keeper job graph, NAV freshness, quote handling, and high-availability posture.

The keeper is the operational core of xVault. It maintains fresh oracle data, scans for drift, executes rebalances, watches corporate actions, and closes weekly reward epochs. Oracles are the bridge between off-chain price / multiplier data and on-chain NAV calculations; keeper correctness and oracle freshness are the two pillars users rely on.

Jobs

JobPurposeCadence
nav-pushFetch prices + multipliers, write oracle PDA snapshots15 s market hours / 60 s off-hours
drift-scanCompute per-vault drift vs. target weights, enqueue rebalance jobs60 s
rebalance-execRequest xChange RFQ (or Jupiter quote), co-sign, submit, confirmTriggered by drift
corp-action-watchPoll upcoming corporate actions, alert / auto-pause on severe eventsHourly
deposit-routerOn deposit event: buy basket, validate via rebalance_legEvent-driven
withdrawal-routerOn USDC withdrawal event: sell xStocks, refill cash bufferEvent-driven
collect-mgmt-feePermissionless trigger of management-fee collectionDaily
epoch-distributorSweep Pump.fun fees, build Merkle distribution, post epoch rootWeekly (Sunday 00:00 UTC)

All jobs run through a Bull queue on Redis. Each job is idempotent on job ID so retries never double-execute a rebalance.

MAX_STALE_SECS = 60    // market hours
MAX_STALE_SECS = 300   // off-hours

NAV-sensitive instructions (deposit, withdraw_*, collect_perf_fee) revert with StaleNav if the oracle snapshot is older than MAX_STALE_SECS. The nav-push job targets a p95 ≤ 30 s latency during market hours.

Sources, in priority order:

  1. Primary: /v2/oracles — Backed's on-chain oracle feed.
  2. Secondary: /public/assets/{symbol}/price-data — indicative quote only, marked source=Indicative.
  3. Tertiary: Pyth equity feed for overlapping tickers (cross-check only).

Indicative prices are never treated as authoritative. If the primary oracle is stale and only the indicative quote is available, NAV-sensitive ix continue to revert — better to block for a few minutes than mint at a wrong price.

Rebalance execution

Drift scan. drift-scan reads vault_state.holdings[] and the latest oracle snapshot, computes scaled weights, and emits a job if any leg exceeds its drift threshold or the time-based trigger fires.

Route decision. rebalance-exec checks tradingHoursMode and isAtomicTradingHalted for each leg. MarketHours assets outside market hours must use Jupiter or skip.

Quote. xChange RFQ or Jupiter v6 swap. Prefer cashAmount (USD notional) over quantity so sizing stays NAV-anchored.

Validate. Prepend vault::rebalance_leg(expected_mint, expected_amount_raw) which introspects sibling ixs via SysvarInstructions to confirm the quoted transfer matches expectations.

Co-sign and submit before the blockhash window expires (xChange: 60–90 s, enforced per-asset via /trades/xchange/assets).

Confirm. Poll getSignatureStatuses with finalized commitment. For xChange also poll GET /trades/xchange/quote/{id} until generalStatus == Completed and blockchainStatus == Executed.

Retry or fall back. On ExpiredExecution: re-quote up to 3 times, then switch to Jupiter. On persistent Jupiter failure: skip the leg and alert.

Retry and backoff policy

  • API retries: exponential backoff 250 ms → 4 s, max 5 attempts, ±20% jitter. Honor Retry-After on 429; never retry 4xx on /rfq.
  • Rebalance retries: max 3 per leg before falling back to the alternate route or skipping. Alert on skip.
  • Blockchain retries: rebuild the tx with a fresh blockhash before each retry.

High-availability posture

  • Leader election via Redis Redlock; two keeper replicas in separate AZs.
  • Only the leader signs and submits; the follower maintains warm state and takes over within 30 s on leader loss.
  • Signing keys live in an HSM/KMS; keepers carry per-slot transaction rate limits and canary alerts for unexpected volume.

Availability target: 99.5%.

Observability

  • Structured logs shipped to the ops sink; one log line per job iteration with job, sku, leg, route, result.
  • Metrics (Prometheus): nav_push_latency_seconds, drift_scan_legs_flagged, rebalance_slippage_bps, xchange_quote_expiry_ratio, jupiter_fallback_count.
  • Alerts: NAV staleness > MAX_STALE_SECS for > 2 min, rebalance skip on any leg, Jupiter fallback rate > 5% over 1 h.

Corporate-action automation

corp-action-watch triggers set_paused(true) automatically for severe caType values within 48 h of activation. Full mapping: Corporate actions.

Infrastructure budget (Helius Developer tier)

  • nav-push at 30 s market / 60 s off = ~100 k RPC / month per vault, comfortably under 1 M credits for 4 vaults.
  • Webhook events ~5 k/day across three programs = 150 k credits / month.
  • Frontend DAS calls (getAssetsByOwner) SWR-cached at 60 s TTL, budgeted ~2 M credits / month at 5 k MAU.
  • Never loop Enhanced Transactions (100 credits each). Use raw getSignaturesForAddress + getTransaction + Anchor IDL decoding.
  • Never use getTransactionsForAddress for backfills.
  • Batch keeper reads via getMultipleAccounts.

Operational invariants

  • MarketHours assets route through Jupiter (or skip) outside market hours.
  • Rebalance legs reject quotes whose mint or raw-amount shape doesn't match rebalance_leg expectations.
  • NAV-sensitive ix revert on any stale snapshot.
  • Keeper signatures are valid only from wallets in the authorized keeper set on the oracle program.

On this page