Session Recovery

Servers restart. Power goes out, operating systems update, processes crash. When Botmarley runs trading sessions that manage real money (or paper money you care about tracking), it needs to handle restarts gracefully. This chapter explains what happens when the server goes down and comes back up.

Automatic Recovery

When the Botmarley server starts, one of the first things it does is check the database for trading sessions that were active at the time of the last shutdown. Specifically, it looks for sessions with a status of Running or Paused.

For each session it finds, the server:

Logs a server_restart event in the session's event history. This creates an audit trail so you can see exactly when gaps occurred.
Parses the strategy TOML snapshot stored when the session was originally started.
Creates a new control channel (the mechanism used to send pause/resume/stop commands).
Restores the portfolio state from the last saved snapshot in the database (positions, balances, fire counts).
Spawns a new trading engine task that picks up where the previous one left off.

The entire process is automatic. You do not need to manually restart sessions after a server restart.

flowchart TD
    A["Server starts"] --> B["Query DB for sessions<br/>with status Running or Paused"]
    B --> C{"Sessions found?"}
    C -- No --> D["Nothing to recover<br/>Normal startup continues"]
    C -- Yes --> E["For each session:"]
    E --> F["Log server_restart event"]
    F --> G["Parse strategy TOML<br/>from stored snapshot"]
    G --> H["Restore portfolio state<br/>from DB snapshot"]
    H --> I["Spawn trading engine<br/>async task"]
    I --> J["Engine backfills missed<br/>candles from exchange"]
    J --> K["Normal operation resumes"]

Tip

Recovery is fully automatic. If your server restarts unexpectedly, just wait for it to come back up. Your trading sessions will resume on their own within seconds of the server starting.

What Gets Recovered

Component	Recovery method
Session status	Read from database. Both "running" and "paused" sessions are recovered. Paused sessions resume as running.
Strategy rules	Parsed from the TOML snapshot stored at session creation time. This means the strategy used is the one that was active when the session started, not the current version of the strategy file.
Portfolio state	Deserialized from the `last_portfolio_snapshot` JSON field in the database. This includes USD balance, crypto balances, open positions, entry prices, and action fire counts.
Trade action history	Loaded from the database. All previously executed actions are intact.
Session events	All previous events (started, paused, resumed) remain in the database, plus the new `server_restart` event.
Counters	`total_actions` and `candles_processed` are restored from the database, preventing duplicate counting.

What Does Not Recover

Component	What happens instead
Candle buffer	The in-memory rolling buffer of recent candles is lost on shutdown. On recovery, the engine rebuilds it by fetching historical candles from the exchange API (Kraken or Binance, based on the account type) -- typically the last 12-24 hours of 1-minute data. This usually takes a few seconds.
Indicator cache	Rebuilt automatically from the refetched candle buffer. No data loss -- indicators are always recalculated from raw candle data.
Progress store	The in-memory progress map (current price, live PnL display) is empty on restart. It repopulates after the first poll cycle.
SSE connections	Browser SSE connections are dropped when the server stops. Browsers will automatically reconnect when the server comes back up. You may see a brief "connection lost" state in the UI.

The Gap Period

Between the server stopping and restarting, the engine is not running. During this gap:

No candles are fetched. The market continues to move, but Botmarley is not watching.
No triggers are evaluated. Trading opportunities may be missed.
Open positions remain open. If you have positions on the exchange, they persist on the exchange regardless of whether Botmarley is running.

When the engine resumes, it backfills missed candles from the exchange API. This means it fetches the candles that were produced during the downtime, appends them to the buffer, and processes them. However, the backfilled candles are processed in bulk -- triggers that might have fired at specific moments during the gap may not fire in the same way when processed after the fact.

Note

For short gaps (a few minutes), the impact is negligible. For longer gaps (hours), you may miss trading signals that occurred during the downtime. The server_restart event in the session log tells you exactly when the gap happened, so you can review the price action during that period manually.

The `server_restart` Event

Every recovered session gets a server_restart event recorded in its event log. This event:

Has a timestamp of when the server restarted (not when it shut down).
Can include metadata about the recovery process (e.g., number of candles backfilled).
Is visible in the session detail page's event history.

You can use this event to:

Audit gaps -- know exactly when the bot was offline.
Correlate missed opportunities -- compare the gap period's price action with what the strategy would have done.
Monitor stability -- frequent server_restart events may indicate infrastructure issues.

Edge Cases

What if the strategy TOML snapshot is invalid?

If the stored TOML snapshot cannot be parsed (e.g., due to a bug in a previous version), the session is marked as Failed instead of being recovered. An error is logged explaining the parse failure. You will need to manually create a new session with the corrected strategy.

What if the portfolio snapshot is missing or corrupt?

If the last_portfolio_snapshot field is empty or cannot be deserialized, the engine starts with a fresh portfolio using the session's initial capital. This means positions tracked in the previous run are lost. A warning is logged so you can investigate.

What about exchange orders placed just before shutdown?

If the engine sent an order to the exchange and the server crashed before recording the result, there may be a discrepancy between the exchange account's actual balance and what Botmarley thinks the balance is. After recovery, check your exchange account balance manually to confirm alignment.

Warning

If you suspect a state mismatch between Botmarley and your exchange account after a crash, stop the session, verify your exchange balance, and start a new session. Do not continue trading with potentially incorrect state.

Botmarley User Manual