The pump thread and sim time

One thread owns the link

pymavlink is not thread-safe, so each session has exactly one thread that touches its connection. It starts SITL, then loops: adjust the pace, run the next queued command if there is one, otherwise pump one message.

Pumping is what keeps the flight healthy between tool calls. Every pump sends the session’s RC override, so a held stick stays held and the RC failsafe never sees silence. Every half second of autopilot time it also sends a GCS heartbeat. That period is counted in sim seconds on purpose: ArduPilot’s GCS failsafe times out in sim seconds, so a heartbeat paced by the wall clock would arrive ten times too rarely at speedup 10.

Tools never touch the link. A tool posts a small function to the session’s queue and awaits a future; the pump thread runs the function between messages and resolves the future. set_param, arm, set_mode and upload_mission are all functions run that way, and every wait inside them goes through the same pump, so RC and heartbeats keep flowing while a mission uploads or an arm retries.

If a request is still queued when its call’s time runs out, it is cancelled and the error says it did not run. If it is already running, the error says so, and snapshot shows where it got to. Nothing is ever cancelled halfway through talking to the autopilot.

Waits are evaluated in the tick

wait_until does not block the pump. It registers a waiter, and the pump thread checks every waiter after every message it receives. That tick is where a wait settles: the condition is tested against the latest message of each type, the then_rc, then_mode or then_speedup reaction is applied, and the snapshot returned to the caller is taken, all before the next message is read.

Because the thread is never blocked by a wait, snapshot and other read-only calls keep answering during one, and several callers may wait on the same session at once.

Step programs run in the same place. A program’s on_tick runs after the waiters, applies a step’s actions the tick it starts, tests its condition, and starts the next step in the tick the previous one settled. Session wind is advanced there too, once per SYSTEM_TIME message. Nothing that runs in the tick may pump or block, which is why step actions are fire and forget.

Sim time and wall time

Two clocks matter, and the tools keep them apart. Conditions and timeouts are in sim seconds, read from the autopilot’s own SYSTEM_TIME. The cap on how long one call may block, wait_cap_s, is in wall seconds, because it protects the caller’s protocol connection, not the flight.

The gap between them is the speedup, and it cuts both ways. At speedup 10 a minute of flight takes six seconds to wait for. It also means every second the caller spends between calls is ten seconds of flight:

your turn4 s

the aircraft40 sim s, last RC still held

At speedup 10, a 4 second pause between two tool calls is 40 seconds of flight. A climb at 2.9 m/s that you meant to stop at 5 m is past 100 m by the time the next call lands.

SITL cannot pause. It ignores SIM_SPEEDUP 0, and below about 0.05 its 20 Hz clock messages arrive so rarely that the link looks dead. So the tools offer three ways to keep a caller’s pace out of the flight: react inside the wait with then_ actions, schedule against the autopilot clock with sim_time, and slow down to think with set_speedup(0.05), which SITL applies live.

Idle pacing

open_sitl(idle_speedup=0.05) automates the last of those. The session counts calls in flight. While one is in flight, or a step program is running, SITL runs at the session’s speed; otherwise it runs at the idle speed.

The switch to idle happens in the tick a wait settles, not when the reply reaches the caller. If the settling wait is the only call in flight, the pump sets the idle speed in that same tick, then holds it until a new call starts, so the sim does not flick back to full speed for the milliseconds the reply spends in transit. In the tutorial’s hop, every settle snapshot reads speedup: 0.05 for this reason, and the time a reader spends on each result costs the aircraft about a twentieth of it.

When SITL stops answering

Every pump that times out, and at least once a second regardless, checks that SITL is alive: the process has not exited, its log has no PANIC (SITL stays up after one), any bridge process lives, and, in a server session, the autopilot clock has moved within stall_s, scaled up for a deliberately slowed sim. Any failure kills the process group and ends the session with state died; waiters settle with reason died and the error in snapshot.error.

Sessions nobody touches for session_idle_s, 30 minutes by default, are closed by a reaper, so an agent that wanders off does not hold an instance and a SITL forever.