Lessons from the SITL harness

mcardupilot was pulled out of a quadplane test harness whose SITL scripts each solved the same problems by hand. Most of its odd-looking code is there because something below cost a debugging session once. The reasons sit in comments next to the code; these are the ones worth knowing even if you never use the package.

SITL ignores SIGTERM, and stays up after a PANIC

Send SITL a SIGTERM and nothing happens. Hit an internal error and it prints PANIC to its log and keeps the process, and its ports, alive. A script that “stopped” SITL politely, or waited for it to exit after a fault, left it running, and the next run on that instance found its port taken. One leftover was found a day and a half after the run that started it.

So SITL is launched as the leader of its own session (start_new_session), which makes its pid the process group id, and stopped with SIGKILL to the whole group. That also takes any child it started, and a physics bridge launched through uv is killed as a group for the same reason: its Python outlives uv. After the kill, the launcher polls the instance’s ports until they are free, and only then releases the lease. Liveness checks read SITL’s log for PANIC as well as polling the process, because a PANIC alone never ends it.

recv_match(type=...) silently drops every other message

pymavlink’s recv_match(type="HOME_POSITION", blocking=True) is the obvious way to wait for home. It reads messages off the link until one matches, and throws the rest away: heartbeats, the EKF’s status texts, parameter replies, everything. Worse, while it waits nothing else runs, so nothing sends RC overrides or a GCS heartbeat, and a long enough wait trips the autopilot’s failsafes.

So nothing in mcardupilot reads the link directly. Every wait, including the home wait at start, parameter reads, arming and mission uploads, goes through pump(), which sends RC and heartbeat, keeps the latest message of each type, records every status text, and calls tick(). A wait is a loop of pumps with a test.

On a dataflash log the same call is the right one. pymavlink’s log reader indexes the file by message type when it opens it, so a typed recv_match jumps from record to record instead of reading and discarding. The log tools use it that way, which is why a 45 MB log summarizes in about a third of a second.

A new log starts with startup records

When ArduPilot opens a dataflash log, it writes a block of startup records: the firmware banner, the build hash, every parameter, and the current flight mode as a MODE record, all stamped with the log’s first timestamp. A MODE record at that timestamp is the mode in force when the log opened. It is not a mode change, and reading it as one invents a switch that never happened.

The summary treats the log’s first timestamp as the start of the first phase rather than as a boundary, and log_compare takes the last PARM value per name rather than the first, so the startup copy of a parameter never hides a change made in flight.

MAVLink parameter names are 16 characters

A parameter’s name travels in MAVLink’s param_id field, which is exactly 16 bytes, and ArduPilot’s names are at most 16 characters to fit. A name of exactly 16 characters has no terminating NUL in the message, so anything that reads it as a C string must stop at 16.

The trap is a longer name. pymavlink packs param_id into its 16 bytes and truncates the rest without complaint. The set then lands on whatever parameter has those first 16 characters, or on nothing, and a readback that waits for a PARAM_VALUE carrying the full name never sees one: get() times out with no PARAM_VALUE for <name>. When a parameter seems to be ignored, count its characters before you suspect the autopilot.

Smaller ones the package absorbs

RC overrides beat Lua. A MAVLink RC override replaces the channel on every message, so a Lua script driving a channel with set_override is overwritten each pump. The override sends 65535 (UINT16_MAX), which ArduPilot ignores, on the channels a script drives: rc_override(script_channels=[...]).

Old Lua scripts load unannounced. SITL loads every script in its run directory’s scripts/. A script left by a previous run on the same instance flies along uninvited, so the folder is wiped before each run’s scripts are copied in, and open_sitl reports lua_loaded.

A stale eeprom carries parameters over. The previous flight’s eeprom.bin would bring its parameters into this one, so it is deleted at launch.

SITL strips a leading / from paths relative to its run directory. Parameter and model files are copied into the run directory and passed by bare name.

Consumed charge accumulates across arm cycles. SITL’s battery counts down over flights until BATT_LOW_MAH refuses to arm. disarm(reset_battery=True) fits a fresh pack: SITL rebuilds its battery only when the requested voltage changes, so the voltage dips 0.2 V for a sim second and returns to exactly its original value, then MAV_CMD_BATTERY_RESET zeroes the count.

ArduPilot ignores mission traffic for a while after boot. Mission uploads wait for home and retry up to six times.

QRTL before home is a PANIC. A parameter set that boots straight into QRTL (INITIAL_MODE) panics before home exists. Nothing outside SITL can prevent that, but the PANIC is caught in its log, SITL is killed, and open_sitl fails with the reason instead of hanging. open_sitl returns only once home is set, so modes chosen through the tools come after it.

A rebuild during a flight must not relabel it. Binaries get rebuilt in place while flights run. open_sitl stamps the binary’s build time, the tree’s commit and a hash of any uncommitted patch in provenance, and lists patched files newer than the build as stale_binary, so a result always says which code flew it.