Monte Carlo session stays 'running' forever when the worker dies before the runner starts

Summary

When run_monte_carlo raises before the runner is constructed, the session has already been written with status='running' and nothing ever changes it. The worker dies, no session log file is created, and the API keeps reporting the session as running indefinitely. There is no way to tell from the API that it failed.

Version: jesse 2.5.0.

Where

jesse/modes/monte_carlo_mode/__init__.py. The status is persisted first:

    else:
        store_monte_carlo_session(
            id=session_id,
            status='running',
            ...
        )

    # Load historical candles AFTER session is persisted
    start_date_timestamp = jh.arrow_to_timestamp(arrow.get(start_date, 'YYYY-MM-DD'))
    finish_date_timestamp = jh.arrow_to_timestamp(arrow.get(finish_date, 'YYYY-MM-DD'))
    warmup_candles, candles = load_candles(start_date_timestamp, finish_date_timestamp)

    # Create and run Monte Carlo runner
    runner = MonteCarloRunner(...)
    runner.run()

MonteCarloRunner.run() is what logs "Monte Carlo session started" and therefore what creates storage/logs/monte-carlo-mode/<session>.txt. Anything that raises between store_monte_carlo_session and runner.run() leaves the session running with no log file at all. MonteCarloRunner.run() has its own try/except that marks the session stopped, but it is never reached.

Reproduction

Start a candles Monte Carlo whose warmup window extends past the earliest candle in the database. The mode reserves 240 days of warmup, so a start date less than 240 days after the first available candle is enough:

exchange:    Bitfinex Spot
symbol:      BTC-USD
start_date:  2017-08-01     (earliest candle in db: 2017-01-01)
run_candles: true

The container log shows the worker dying:

Unhandled exception in the process:
Traceback (most recent call last):
  File "/jesse/services/multiprocessing.py", line 23, in run
    mp.Process.run(self)
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/jesse/modes/monte_carlo_mode/__init__.py", line 103, in run
    warmup_candles, candles = load_candles(start_date_timestamp, finish_date_timestamp)
  ...
jesse.exceptions.CandleNotFoundInDatabase: Missing candles for BTC-USD on Bitfinex Spot.
Requested data from 2016-12-04, but earliest available candle is from 2017-01-01.
==> Removed finished worker <session_id> from active workers

The exception itself is correct and useful — the data genuinely is not there. The problem is everything after it:

I left one of these sitting at running for over two hours before checking the container log. From the API alone there is nothing to distinguish it from a long simulation — the trades mode legitimately takes a while, so "still running" is not suspicious on its own.

The same start date works fine through jesse.research.backtest, which reserves 210 warmup candles rather than 240, so the failure only appears in the Monte Carlo path and looks like the mode itself is broken.

Suggested fix

Wrap the body of run_monte_carlo from store_monte_carlo_session onward so any exception marks the session as failed and records the message, mirroring what MonteCarloRunner.run() already does for exceptions raised after it starts. Storing the exception text on the session would make it visible in the UI, which is where a user would look first.

Validating the requested range against the available candles before writing status='running' would also avoid creating the row at all in this particular case, though the general guard is what matters — the same window exists for any failure in load_candles or in MonteCarloRunner.__init__.

Related

Same shape as #603, where an invalid route timeframe leaves a backtest session in draft forever after the API returns 202. In both cases the request is accepted, the worker dies, and the session status is never reconciled.

Also opened: #606 and #607 on the trades Monte Carlo metrics, with #608 as the PR for #607.