Skip to content
Last updated

Recovery and Gap Fill

Recovery depends on the channel. A message ID is useful for logging, but is not an account replay cursor.

ChannelReconnect behavior
strategy.signalResubscribe with this strategy's last successfully processed signal seq_id
strategy.live_fillsResubscribe with this strategy's last successfully processed fill seq_id
strategy.positionsResubscribe and replace the full position collection with the fresh snapshot
All four account.* channelsResubscribe without resume_from, then replace account state from fresh snapshots

Strategy Stream Cursors

Signals and completed live fills have independent cursor domains. Keep a durable cursor per (channel, strategy_id) and advance it after processing a message successfully. Sequence IDs are monotonic but need not be consecutive; a numeric gap alone does not prove message loss. Do not share the signal cursor with fills or compare cursors across strategies.

For either replayable channel, put a positive integer resume_from in the subscription payload. The server acknowledges, replays available messages after that cursor up to its configured limit, then delivers live messages. Omitted or zero cursors request future delivery without historical replay. Replays use the same envelope and seq_id as live delivery, so processing should be idempotent within that stream.

Default replay limits are independent:

  • Signals: 1,000 messages.
  • Live fills: 500 messages.

A replay error or truncation can produce a gap message. A representative signal truncation is:

{
  "msg_type": "gap",
  "id": "msg_125e9b9d52774f32b56425970aa0",
  "timestamp": "2026-10-26T15:00:00Z",
  "v": 4,
  "payload": {
    "channel": "strategy.signal:12345678",
    "from": "98765",
    "message": "Replay limit of 1000 messages exceeded"
  }
}

The channel includes the strategy, such as strategy.signal:12345678 or strategy.live_fills:12345678. from is a string containing the last replayed cursor (or retry boundary for an unavailable replay). The optional strategy_id is currently absent because null values are omitted. A gap means additional recovery is needed; advancing a cursor alone does not recover missed activity.

Recovering Snapshot-based Channels

Account channels reject any resume_from, including zero. Reconnect, mark cached state stale, resend each account subscription and replace local state when its is_snapshot: true message arrives. An empty order/position array or empty strategies collection clears the old collection. Do not merge the new initial array into the state left over from the old socket.

This small handler shows account recovery for orders; use the corresponding replacement rules for positions, balance and state from Subscribing:

orders = {}
ready_accounts = set()

def on_reconnect():
    orders.clear()
    ready_accounts.clear()
    # Resend account subscriptions without resume_from using the live receive loop.

def on_order_message(message):
    account = (message["account"], message["broker_id"])
    if message["is_snapshot"]:
        orders[account] = {row["id"]: dict(row) for row in message["payload"]}
        ready_accounts.add(account)  # An empty snapshot is ready too.
    else:
        if account not in ready_accounts:
            raise RuntimeError("A fresh account snapshot is required")
        for row in message["payload"]:
            if row["orderStatus"] in {"V", "R", "0", "1", "6", "A", "E"}:
                orders[account][row["id"]] = dict(row)
            else:
                orders[account].pop(row["id"], None)

For strategy.positions, each received array is already a full replacement snapshot. There is no replay cursor and no need to infer zero-quantity tombstones for that channel.

Reconciling After a Strategy Stream Gap

REST endpoints do not replay individual signal/fill events by WebSocket seq_id. Use current strategy state to reconcile the application view, and treat unavailable historical events as a genuine gap. These authenticated GET endpoints provide current snapshots:

  • /Strategies/GetStrategyOpenPositions?StrategyIds=12345678
  • /Strategies/GetStrategyActiveOrders?StrategyIds=12345678

The following helper performs the actual GETs off the receive loop. It returns the two REST response documents; it does not claim to reconstruct historical events or advance a cursor.

import asyncio
import json
from urllib.parse import urlencode
from urllib.request import Request, urlopen

async def fetch_strategy_snapshots(api_url, api_key, strategy_id):
    def get(path):
        query = urlencode({"StrategyIds": strategy_id})
        request = Request(f"{api_url.rstrip('/')}{path}?{query}",
                          headers={"Authorization": f"Bearer {api_key}"})
        with urlopen(request, timeout=20) as response:
            return json.load(response)
    positions, orders = await asyncio.gather(
        asyncio.to_thread(get, "/Strategies/GetStrategyOpenPositions"),
        asyncio.to_thread(get, "/Strategies/GetStrategyActiveOrders"))
    return {"positions": positions, "orders": orders}

The application must supply its reconciliation policy:

  1. Pause decisions based on the affected derived state. Keep the sole WebSocket receive loop running and acknowledging heartbeats; buffer affected live events with their stream cursors while a background task fetches the snapshots.
  2. Check successful REST responses and reconcile their Results collections by the relevant order/position identities. REST fields use REST casing. The two GETs can observe state at different times, and neither returns a WebSocket cursor boundary.
  3. Apply buffered events idempotently using their identities/statuses, with the application's rules for observations already represented by a snapshot. Do not blindly add fill quantities to positions; fills exclude partial fills and can overlap the snapshot. A fresh strategy.positions subscription supplies a current replacement position snapshot.
  4. If a concurrent change leaves the view uncertain, fetch/reconcile again before resuming decisions. Advance each cursor only after that stream's messages were successfully handled. Recording gap.from alone is not evidence that the missing interval was recovered.
  5. Bound the live-event buffer. If it fills, restart recovery with fresh state instead of silently discarding data. Reconnect with exponential backoff and jitter when the transport fails.

These steps restore a current state view. They do not turn time-based REST snapshots into an exact historical event replay. Account channels recover through their own fresh snapshots and must not be sent a strategy replay cursor.