# Monitoring & Alerts The ksUSD vault is one PDA — most monitoring reduces to watching that account plus a few external signals. *** ## Key metrics | Metric | Source | Alert when | | --------------------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | Share price (`share_price_1e9`) | Derived from `effective_nav_usdc / total_shares` | Drops > 2% in 24h, or > `emergency_close_dd_bps` from `peak_share_price_1e9` | | `cached_nav_usdc` vs. computed-live NAV | On-chain vs. off-chain reconciliation against Drift / Kamino positions | Discrepancy > 1% | | Pyth SOL/USD staleness | Last-update timestamp on the Pyth feed | > 5 min | | Pyth confidence | `confidence / price` | > 1.5% (program reverts at 2%) | | jitoSOL/SOL depeg | Derived from `jitosol_oracle.price / sol_oracle.price` | > 80% of `lst_depeg_bps` | | `cached_nav_usdc` cap utilization | `cached_nav_usdc / deposit_cap_usdc` | > 90% (raise cap or pause new deposits) | | `position_mode` | `vault.position_mode` | `Idle` for > 24h while funding signal is active | | `position_base_amount` | `vault.position_base_amount` | Drifts > 1% from expected (1× of jitoSOL collateral leg) | | `funding_apr_smoothed_bps` staleness | `now - funding_smooth_last_ts` | Approaching `funding_max_staleness_seconds` | | `consecutive_dd_settles_observed` | `vault.consecutive_dd_settles_observed` | ≥ 1 (single-tick warning before lethal) | | Drift health factor | Drift sub-account margin | < 1.5 | | Drift SOL-PERP funding rate | `last_funding_rate` on Drift PerpMarket | Sustained outside `[reverse_thresh, normal_thresh]` past dwell | | Pending withdrawals | `queue_next_id − queue_processed_through` | > 0 for more than one keeper cycle | | Reserve fund | `reserve_fund_usdc` and `reserve_lent_kamino` | Reserve ATA empty while `reserve_fund_usdc > 0` (means `unlend_reserve` needed before any payout) | | Lending balances | `usdc_lent_kamino`, `reserve_lent_kamino` | Sudden drop without a matching `unlend_*` event = Kamino-side anomaly | | Kamino obligation health (reverse only) | LTV from Kamino refresh | > 80% of liquidation LTV | | `vault.paused` | bool | Unexpected `true` = investigate `LstDepeg` or admin action | | `vault.position_mode == WindDown` | enum | Terminal — only `claim_wind_down` should run | *** ## Reconciliation Once per cycle, an off-chain script should compute the "true" NAV by summing: 1. Vault USDC ATA balance 2. `usdc_lent_kamino × kamino USDC reserve cToken exchange rate` (lent vault USDC, current value) 3. `reserve_fund_usdc` (already includes `reserve_lent_kamino` per the vault's accounting) 4. `(jitoSOL collateral on Drift) × Pyth SOL price × jitoSOL/SOL ratio` (normal basis only) 5. Drift sub-account quote balance (settled USDC + accrued PnL) 6. `usdc_collateral_kamino − jitosol_borrowed_kamino × Pyth jitoSOL price` (reverse basis only) Compare against `vault.cached_nav_usdc`. A discrepancy > 1% means: * `settle` is overdue, **or** * `attest_nav` is needed to capture unrealized PnL *** ## Emergency conditions | Signal | Response | | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | Pyth SOL/USD stale > 10 min OR confidence > 2% | Pause new deposits + positions; manual `emergency_close` if a position is open | | jitoSOL depeg > `lst_depeg_bps` | `settle` auto-pauses; investigate and either unpause or emergency-close | | Drift outage / market halted | Same — pause and investigate | | NAV drawdown ≥ `emergency_close_dd_bps` for `consecutive_dd_settles_required` settles | Anyone can call `emergency_close` permissionlessly; keeper should do it first | | `position_mode == WindDown` | Stop the keeper; broadcast `claim_wind_down` instructions to depositors | *** ## Suggested external infra * A serverless function (Cloudflare Workers / AWS Lambda) running every 5 min: fetch `vault` + Pyth + Drift account, compute the deltas, push to Grafana / Datadog / PagerDuty. * A `program.addEventListener` consumer that subscribes to `DepositEvent`, `WithdrawEvent`, `WithdrawalRequested`, `WithdrawalProcessed`, `FeesCollected`, `PositionOpened`, `PositionClosed`, `NavAttested`, `IdleLent`, `IdleUnlent`, `ReserveLent`, `ReserveUnlent`, `WindDownInitiated`, and writes to a database for audit / reporting. * A page in the operator dashboard tying `cached_nav_usdc` to the reconciliation script's live computation, with a visible delta and an "attest now" button that calls `attest_nav` when the delta crosses an internal threshold. *** ## Related * [Keeper bot](/for-operators/keeper-bot.md) · [Admin operations](/for-operators/admin-ops.md) * [Events](/for-developers/events.md) · [Errors](/for-developers/errors.md) --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.keystonefi.xyz/for-operators/monitoring.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.