> For the complete documentation index, see [llms.txt](https://docs.keystonefi.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.keystonefi.xyz/run-it/keeper-bot.md). # Keeper Bot The keeper is an off-chain bot that cranks settlement, lending, mode transitions, and withdrawals on a schedule. Mode transitions are automatic: the bot opens or closes the basis when the on-chain smoothed funding signal and guardrails permit, with no manual intervention. The position, NAV-attestation, and lending instructions require the vault's `authorized_keeper` to sign — so the keeper is a permissioned *executor*, bounded by the on-chain rules, not a decision-maker. `settle`, `process_withdrawal`, and `emergency_close` are permissionless, so anyone can keep the vault safe and redeemable if the keeper goes offline. {% hint style="warning" %} v1 hedges on Phoenix Perps (USDC margin via Ember). The keeper assembles the Phoenix and Ember account groups with the Rise SDK. Mode transitions run automatically off the on-chain funding signal; the keeper bot executes them and cranks settlement, lending, and withdrawals. {% endhint %} > Reference implementation: [the keeper](https://github.com/kamwithak/keystone-contracts/tree/main/scripts/keeper/README.md) — `index.ts` is the loop, `duties/` are the individual cranks, shared plumbing is in [its shared libraries](https://github.com/kamwithak/keystone-contracts/tree/main/scripts/lib/README.md). TypeScript + Anchor + Jupiter API. Run with `npm run keeper` (`KEEPER_MODE=dry-run` to plan only, `KEEPER_MODE=simulate` to build and simulate without sending). `npm run config:check` validates the configuration first and refuses anything that would run but not work. *** ## Cadence The keeper ticks every `KEEPER_INTERVAL` seconds — 300 by default, and **hard capped at 900 on mainnet**, because deposits and withdrawals revert `NavStale` once NAV is 30 minutes old. That is a liveness constraint, not a preference: the keeper refuses to start with a longer interval. | Instruction | Frequency | Trigger | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `settle` | Every tick | Refresh funding EMA, drawdown peak, LST depeg check, `settle_pnl` if open. While parked it is the only thing keeping NAV fresh. | | `rebalance_hedge` | When \|delta\| / NAV clears `delta_band_bps` (25 = \~every other week) | Sweep the unhedged jitoSOL accrual back onto the short. Bounded by `max_rebalance_clip_lots`, which admin must set — see [the clip](#the-rebalance-clip). | | `attest_nav(new_nav_usdc)` | After `settle` | Snap NAV to live position value when on-chain reads can't capture unrealized PnL. Bounded by `max_nav_change_bps_per_hour`. | | `open_position` | When the smoothed funding signal clears the threshold and dwell has elapsed | One-shot per mode entry | | `close_position` | When the signal falls back through the threshold | One-shot per mode exit | | `lend_idle_usdc` / `unlend_usdc` | Whenever idle vault USDC exceeds the buffer target by > a margin (or buffer is short) | Keep USDC productive without breaching `liquidity_buffer_target` | | `process_withdrawal` | When the queue has unprocessed requests AND vault holds enough idle USDC for the **next** request (FIFO) | Permissionless — anyone can crank | | `emergency_close` | When drawdown guard trips OR vault is paused with a position open | Permissionless once tripped | | `add_margin` | Every tick while a position is open, when margin health drops below 2.5 | **The keeper moves funds on its own here** — see below | | `reduce_position` | When the withdrawal queue still falls short after redeeming the whole Kamino position, and that shortfall persists across ticks | De-lever both legs to fund the queue: buy back part of the short and sell the matching slice of jitoSOL, so net exposure never moves. It waits deliberately — tearing down part of the hedge realises slippage on both legs and permanently shrinks the position, so a queue one deposit would satisfy must not trigger it. Also the margin backstop, and **permissionless below `hard_margin_floor_bps`** | **Every perp fill is priced against the oracle, whoever sent it.** `open_position`, `close_position`, `rebalance_hedge`, `reduce_position` and `emergency_close` all band the realized fill against the pinned SOL/USD mark by `max_swap_slippage_bps`, computed from the position delta the trader account records. The keeper's `min_base_lots_to_fill` / `min_quote_lots_to_fill` are its own floor on top; the program's bound does not depend on the keeper getting them right, which matters most on `emergency_close`, where the caller is a stranger. **Margin only ever moves onto the venue to defend the position.** Nothing the keeper does reduces collateral on a short that stays the same size — the only way collateral comes back is if the short itself gets smaller, which `reduce_position` does by shrinking both legs together. Surplus margin raises a `margin-surplus` warning so an operator can judge whether the drag is worth closing early, and when idle USDC and the Kamino balance can no longer fund a top-up, `reduce_position` de-levers both legs proportionally rather than pulling collateral out from under the short. ### The rebalance clip `initialize` turns rebalancing **on** — `delta_band_bps = 25` — but leaves `max_rebalance_clip_lots` at `0`, which means *unclipped*: one order may take the entire correction. That is deliberate, because the right ceiling is a function of the venue's book depth and the program never sees the book. It is also the one thing that must not be left as it ships. It matters because `rebalance_hedge` places an IOC market order with `min_base_lots_to_fill = 0` and no `price_in_ticks`. The fill price is bounded after the fact — `assert_fill_price_within` bands it against the pinned SOL/USD mark — so the clip is what bounds how much size one order pushes into the book, and how far that order walks the book. The keeper knows and will warn (`clip-unset`, `clip-oversized`) but cannot fix it: `update_params` is admin-gated. Measure and set it: ```bash npm run phoenix:capacity # prints the measured clip npm run params:set -- --clip=auto # or --clip= npm run freeze:check # "rebalance bounds" should be green ``` Re-measure after any large move in Phoenix volume — the clip is 1% of a typical day, and a book that halves makes yesterday's clip twice as aggressive. *** ## Signal pipeline (open / close decisions) *The normal↔parked transition is decided by the on-chain funding signal; the keeper bot assembles and submits the switch automatically whenever the rules below permit. The pipeline is that funding logic.* 1. **Read funding.** `settle` parses the last funding rate + timestamp from the perp venue's SOL-PERP market and updates the on-chain EMA (`funding_apr_smoothed_bps`). Sanity-bounded — absurd values revert with `FundingRateInsane`. 2. **Stale check.** If `now - funding_smooth_last_ts > funding_max_staleness_seconds`, opens revert with `FundingSignalStale`. Run `settle` first. 3. **Threshold + dwell + drawdown latency.** * If `funding_apr_smoothed_bps ≥ funding_threshold_normal_bps` AND `now − last_mode_change_ts ≥ min_dwell_seconds` → call `open_position`. * Else if in the active mode but the signal decayed below threshold + hysteresis → call `close_position`. * Drawdown trips require `consecutive_dd_settles_observed ≥ consecutive_dd_settles_required` (default 2) before `emergency_close` can fire on a NAV breach — single-tick noise is rejected with `DrawdownTriggerLatent`. 4. **Build the transaction.** Assemble per-instruction `remaining_accounts` groups (see [Instructions](/build-on-it/quick-start/instructions.md)) and the Jupiter swap data via the Jupiter `/swap-instructions` endpoint. 5. **Submit.** Wrap with priority-fee compute budget instructions. Retry on slot-out / blockhash-not-found. *** ## Assembling `remaining_accounts` * Strategy instructions pack multiple CPIs into one transaction * Each group needs a precise account list — wrong order or missing accounts reverts the CPI * Use the venue's official client / API to build each group: * **Phoenix margin (Ember) / place\_perp\_order / settle funding** — use the **Rise SDK** (`github.com/Ellipsis-Labs/rise-public`) instruction builders and extract the `keys` array. * **Kamino lending (`lend_idle_usdc` / `unlend_usdc`)** — use the Kamino IDL and PDA derivation utilities to build the USDC reserve deposit/withdraw account groups. * **Jupiter route\_swap** — call Jupiter's `/swap-instructions` HTTP endpoint with `userPublicKey = vaultPda`, then pass the returned `swapInstruction.data` as `jupiter_swap_data` and the returned `keys` as the relevant remaining-accounts group. > Pass per-group counts as the u8 instruction args (`jupiter_account_count`, `phoenix_deposit_account_count`, etc.) so the on-chain handler can split safely. *** ## Process-withdrawal cranking **Strict FIFO:** * Next-in-line request\_id is `vault.queue_processed_through` itself (both counters start at 0, and `process_withdrawal` requires `request_id == queue_processed_through`) * Any other request errors with `WithdrawalNotNextInQueue` * Anyone can crank (including the original requester) * Rent refunds to the original requester even if a different wallet pays the tx **Liquidity shortfall:** * Reverts with `InsufficientLiquidityBuffer` if vault USDC is short of the next `usdc_owed` * Keeper should `unlend_usdc` first if Kamino USDC can cover the shortfall * Otherwise wait for the next `close_*` to free up USDC *** ## Failure modes | Symptom | Cause | Resolution | | ---------------------------------------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `MinDwellNotElapsed` | Tried to switch modes too soon | Wait for `last_mode_change_ts + min_dwell_seconds` | | `FundingThresholdNotMet` | Smoothed funding hasn't crossed the threshold | Re-read `funding_apr_smoothed_bps` | | `FundingSignalStale` | EMA older than `funding_max_staleness_seconds` | Run `settle` first | | `SlippageExceeded` | Jupiter route filled at worse price than expected | Refresh the quote, narrow the bound, retry | | `PhoenixCpiFailed` | Phoenix returned an error — usually margin or oracle | Inspect program logs; may need to settle funding first | | `InsufficientLiquidityBuffer` (`process_withdrawal`) | Not enough idle USDC for the next request | `unlend_usdc` to refill, or wait for next `close_*` | | `LendingBreachesBuffer` | `lend_idle_usdc` would drop the buffer below `liquidity_buffer_target` | Reduce amount or run `unlend_usdc` instead | | `NavChangeExceedsCap` | `attest_nav` delta > `max_nav_change_bps_per_hour` | Stage with smaller deltas over multiple hours, or admin raises the cap if the position genuinely moved that much | | `DrawdownTriggerLatent` | Drawdown observed but not yet across consecutive settles | Wait one more settle cycle; this is by design | | `LstDepeg` | jitoSOL/SOL deviation > `lst_depeg_bps` | `settle` auto-pauses; admin investigates and either unpauses or emergency-closes | *** ## Idempotency * All open / close instructions assert `position_mode` invariants on entry * Duplicate calls during a retry storm revert cleanly without state damage * `settle` is fully idempotent within an hour (no state change beyond the EMA refresh) *** ## The keeper moves margin autonomously Unlike every other duty, `margin-health` can move vault funds without a human. That is deliberate: liquidation on a delta-hedged short can arrive in minutes, and an alert you read an hour later is not a defence. The mechanism it defends against is unintuitive — the jitoSOL leg is in the vault and the margin is on Phoenix, and **Phoenix can only see its own side**. On a rally the short loses margin while the offsetting jitoSOL gain sits somewhere Phoenix cannot count, so a hedged position gets liquidated anyway. `emergency_close` will not save it, because NAV doesn't fall in a hedged rally. Bounds on that autonomy: * it can never draw on the withdrawal buffer * total posted margin can never exceed **half of effective NAV** (enforced on-chain, not just in the keeper) * collateral never falls on a short that stays the same size; de-levering closes part of the position and the matching spot leg together * `KEEPER_AUTO_MARGIN=0` reduces it to alert-only — but then you must be reachable within minutes If the keeper is down, or auto-margin is off, the thin-margin policy (`min_margin_bps = 900`) is no longer justified — close the position rather than leave it sitting. Note the 10× this implies is the venue's view of the perp leg, not portfolio leverage: the vault's total exposure equals its NAV. *** ## Related * [Admin operations](/run-it/keeper-bot/admin-ops.md) * [Monitoring](/run-it/keeper-bot/monitoring.md) * **Deploying it** — `scripts/keeper/deploy/README.md`: Railway setup, dead-man's switch, kill switch (`KEEPER_READ_ONLY=1`), key rotation, and the keeper-down procedure * [Instructions reference](/build-on-it/quick-start/instructions.md) — every entry point and who may call it * [Errors](/build-on-it/quick-start/errors.md) — what a revert means