Keeper and oracle operations
Keeper job graph, NAV freshness, quote handling, and high-availability posture.
The keeper is the operational core of xVault. It maintains fresh oracle data, scans for drift, executes rebalances, watches corporate actions, and closes weekly reward epochs. Oracles are the bridge between off-chain price / multiplier data and on-chain NAV calculations; keeper correctness and oracle freshness are the two pillars users rely on.
Jobs
| Job | Purpose | Cadence |
|---|---|---|
nav-push | Fetch prices + multipliers, write oracle PDA snapshots | 15 s market hours / 60 s off-hours |
drift-scan | Compute per-vault drift vs. target weights, enqueue rebalance jobs | 60 s |
rebalance-exec | Request xChange RFQ (or Jupiter quote), co-sign, submit, confirm | Triggered by drift |
corp-action-watch | Poll upcoming corporate actions, alert / auto-pause on severe events | Hourly |
deposit-router | On deposit event: buy basket, validate via rebalance_leg | Event-driven |
withdrawal-router | On USDC withdrawal event: sell xStocks, refill cash buffer | Event-driven |
collect-mgmt-fee | Permissionless trigger of management-fee collection | Daily |
epoch-distributor | Sweep Pump.fun fees, build Merkle distribution, post epoch root | Weekly (Sunday 00:00 UTC) |
All jobs run through a Bull queue on Redis. Each job is idempotent on job ID so retries never double-execute a rebalance.
NAV freshness
MAX_STALE_SECS = 60 // market hours
MAX_STALE_SECS = 300 // off-hoursNAV-sensitive instructions (deposit, withdraw_*, collect_perf_fee) revert with StaleNav if the oracle snapshot is older than MAX_STALE_SECS. The nav-push job targets a p95 ≤ 30 s latency during market hours.
Sources, in priority order:
- Primary:
/v2/oracles— Backed's on-chain oracle feed. - Secondary:
/public/assets/{symbol}/price-data— indicativequoteonly, markedsource=Indicative. - Tertiary: Pyth equity feed for overlapping tickers (cross-check only).
Indicative prices are never treated as authoritative. If the primary oracle is stale and only the indicative quote is available, NAV-sensitive ix continue to revert — better to block for a few minutes than mint at a wrong price.
Rebalance execution
Drift scan. drift-scan reads vault_state.holdings[] and the latest oracle snapshot,
computes scaled weights, and emits a job if any leg exceeds its drift threshold or the time-based
trigger fires.
Route decision. rebalance-exec checks tradingHoursMode and isAtomicTradingHalted for
each leg. MarketHours assets outside market hours must use Jupiter or skip.
Quote. xChange RFQ or Jupiter v6 swap. Prefer cashAmount (USD notional) over quantity so
sizing stays NAV-anchored.
Validate. Prepend vault::rebalance_leg(expected_mint, expected_amount_raw) which introspects
sibling ixs via SysvarInstructions to confirm the quoted transfer matches expectations.
Co-sign and submit before the blockhash window expires (xChange: 60–90 s, enforced per-asset
via /trades/xchange/assets).
Confirm. Poll getSignatureStatuses with finalized commitment. For xChange also poll GET /trades/xchange/quote/{id} until generalStatus == Completed and blockchainStatus == Executed.
Retry or fall back. On ExpiredExecution: re-quote up to 3 times, then switch to Jupiter. On
persistent Jupiter failure: skip the leg and alert.
Retry and backoff policy
- API retries: exponential backoff 250 ms → 4 s, max 5 attempts, ±20% jitter. Honor
Retry-Afteron 429; never retry 4xx on/rfq. - Rebalance retries: max 3 per leg before falling back to the alternate route or skipping. Alert on skip.
- Blockchain retries: rebuild the tx with a fresh blockhash before each retry.
High-availability posture
- Leader election via Redis Redlock; two keeper replicas in separate AZs.
- Only the leader signs and submits; the follower maintains warm state and takes over within 30 s on leader loss.
- Signing keys live in an HSM/KMS; keepers carry per-slot transaction rate limits and canary alerts for unexpected volume.
Availability target: 99.5%.
Observability
- Structured logs shipped to the ops sink; one log line per job iteration with
job,sku,leg,route,result. - Metrics (Prometheus):
nav_push_latency_seconds,drift_scan_legs_flagged,rebalance_slippage_bps,xchange_quote_expiry_ratio,jupiter_fallback_count. - Alerts: NAV staleness >
MAX_STALE_SECSfor > 2 min, rebalance skip on any leg, Jupiter fallback rate > 5% over 1 h.
Corporate-action automation
corp-action-watch triggers set_paused(true) automatically for severe caType values within 48 h of activation. Full mapping: Corporate actions.
Infrastructure budget (Helius Developer tier)
nav-pushat 30 s market / 60 s off = ~100 k RPC / month per vault, comfortably under 1 M credits for 4 vaults.- Webhook events ~5 k/day across three programs = 150 k credits / month.
- Frontend DAS calls (
getAssetsByOwner) SWR-cached at 60 s TTL, budgeted ~2 M credits / month at 5 k MAU. - Never loop Enhanced Transactions (100 credits each). Use raw
getSignaturesForAddress+getTransaction+ Anchor IDL decoding. - Never use
getTransactionsForAddressfor backfills. - Batch keeper reads via
getMultipleAccounts.
Operational invariants
MarketHoursassets route through Jupiter (or skip) outside market hours.- Rebalance legs reject quotes whose mint or raw-amount shape doesn't match
rebalance_legexpectations. - NAV-sensitive ix revert on any stale snapshot.
- Keeper signatures are valid only from wallets in the authorized keeper set on the
oracleprogram.