Clean up a stray SITL

A stray is a SITL nobody is flying: left by a script that was SIGKILLed, by a crashed run of make fly, by a test that hung. It holds its instance’s ports, so every launcher skips that instance. mcardupilot finds strays by their ports and stops them only when no lease covers them.

Find it

A launch that skips a busy instance says why, and list_instances shows every instance that is leased or has its ports taken:

list_instances()

[
  { "instance": 44, "in_pool": true, "mavlink_port": 6200,
    "holder": "mcardupilot session f391b1 (tools-hop-test) pid 728436",
    "session_id": null, "leased_here": false, "busy_ports": [] },
  { "instance": 47, "in_pool": true, "mavlink_port": 6230,
    "holder": null,
    "session_id": null, "leased_here": false, "busy_ports": ["tcp:6230 (arduplane pid 51234)"] }
]

Instance 44 is held: someone is flying it, and it is not yours to stop. Instance 47 has an arduplane on its MAVLink port and no holder. That is the stray. (The second entry shows the shape of a stray; the pid is illustrative.)

Stop it

kill_instance(47)

{ "instance": 47, "closed_session": null, "killed": ["killed arduplane pid 51234 (its group)"], "ports_still_taken": [] }

What kill_instance does, in order:

1. Your own session on that instance is closed the normal way, logs kept per its keep_logs, and nothing else happens.
2. Someone else’s lease stops it. The call is refused with the holder named: only the holder may stop it.
3. Outside the pool, or a reserved instance (0 belongs to autotest.py) needs force=true, because scripts that take no leases run there.
4. Otherwise it takes the instance’s lease itself for the whole kill, so nothing can start there between the check and the kill. Then it kills the SITL a pid file names, if one is left under the sessions directory, and any process on the instance’s ports whose name looks like a SITL binary (ardu*, antennatracker, blimp). Anything else on those ports is reported and left alone.
5. A stray that leads its own process group is killed with its group; one started from a shell, which is not a group leader, is killed alone so the shell survives.

ports_still_taken lists anything still bound five seconds later.

Without an agent

uv run python -m mcardupilot.leases status      # who holds what

A script that was SIGKILLed while using Sitl left <run_root>/<n>/sitl.pid, and the library can act on it:

from pathlib import Path
from mcardupilot.sitl import kill_instance

kill_instance(47, Path("~/.local/share/mcardupilot/run").expanduser())   # True if it killed something

It kills the group only if its leader is still the process that was launched, checked by its start time, so a recycled pid is safe.

Not pkill -f

SITL ignores SIGTERM, so it needs SIGKILL, and pkill -f arduplane is the tempting way. Don’t. It kills every SITL on the machine, other people’s included, and pkill -f <pattern> also matches the shell running it, whose own command line contains the pattern. Kill by pid or process group (kill -9 -<pgid>), or use kill_instance.

Strays left by a dead server

You rarely need any of this for sessions. A server that starts looks through the sessions directory for pid files, and for each instance whose lease it can take (meaning nothing alive owns it) it kills what the pid file names. server_info lists what it did under reconciled. The lease decides, not the pid file, because every agent’s server shares the same directory: a pid file alone may belong to another live server’s session.