Run a gusty A/B test

An A/B test flies the same thing twice with one difference and asks what changed. Here the difference is gusts: the same 6 m/s mean wind, steady in flight A and with Dryden turbulence in flight B. Everything else is held equal by flying both as the same step program.

Fly A and B

The comparison works with current and energy. The stock quadplane parameters configure no battery, so open_sitl turns on SITL’s simulated one (BATT_MONITOR 4) unless you set it; the set below only makes that explicit.

STEPS = [
    {"label": "climb", "rc": {3: 2000}, "until": "altitude", "value": 5},
    {"label": "hover", "rc": {3: 1500}, "until": "sim_time", "value": 20, "relative": True},
    {"label": "land",  "mode": "QLAND", "until": "disarmed", "timeout_s": 120},
]

# A: steady wind
a = open_sitl(model="quadplane", label="calm-a", set={"BATT_MONITOR": 4},
              wind={"speed_mps": 6, "from_deg": 270, "gusts": "off"})
arm(a.session_id, mode="QLOITER")
fly_steps(a.session_id, steps=STEPS)
close_sitl(a.session_id)          # kept: 20261003T202301Z-calm-a.BIN

# B: the same mean wind with seeded Dryden gusts
b = open_sitl(model="quadplane", label="gusty-b", set={"BATT_MONITOR": 4},
              wind={"speed_mps": 6, "from_deg": 270, "gusts": "dryden", "seed": 7})
arm(b.session_id, mode="QLOITER")
fly_steps(b.session_id, steps=STEPS)
close_sitl(b.session_id)          # kept: 20261003T202311Z-gusty-b.BIN

The seed makes B repeatable: another flight with seed 7 meets the same gusts, because real SITL ticks on an exact 20 Hz sim clock and the gust model draws its random numbers in the same order. During the flight, every snapshot shows the gust as it is now:

"wind": { "speed_mps": 6.0, "from_deg": 270.0, "gusts": "dryden", "seed": 7,
          "gust_mps": [-1.478, -0.052, 1.107],
          "sitl": { "SIM_WIND_SPD": 4.656, "SIM_WIND_DIR": 269.3, "SIM_WIND_DIR_Z": 13.75 } }

wind_history(session_id, seconds=60) returns up to two minutes of it while the session is open.

Compare them

log_compare(a="20261003T202301Z-calm-a.BIN", b="20261003T202311Z-gusty-b.BIN")

Every number comes as {a, b, delta, pct}, with delta = b - a and pct as a percentage of |a|. Abridged:

{
  "same_firmware": true,
  "totals": {
    "duration_s": { "a": 83.368, "b": 82.792, "delta": -0.576, "pct": -0.69 },
    "energy_wh":  { "a": 5.6531, "b": 5.4932, "delta": -0.1599, "pct": -2.83 },
    "alt_max_m":  { "a": 7.43,   "b": 7.57,   "delta": 0.14,    "pct": 1.88 },
    "amps_peak":  { "a": 68.037, "b": 68.012, "delta": -0.025,  "pct": -0.04 }
  },
  "params": {
    "changed": {
      "Q_M_THST_HOVER": { "a": 0.5196732, "b": 0.5079439 },
      "SIM_WIND_DIR":   { "a": 270.0, "b": 281.2 },
      "SIM_WIND_DIR_Z": { "a": 0.0,   "b": -0.99 },
      "SIM_WIND_SPD":   { "a": 6.0,   "b": 5.577 }
    },
    "only_in_a": {}, "only_in_b": {},
    "bookkeeping": { "STAT_RUNTIME": { "a": 92.0, "b": 91.0 }, "BARO1_GND_PRESS": { "a": 101325.4, "b": 101323.2 } },
    "same": 1618
  },
  "phases": {
    "matched": [
      { "mode": "QLOITER", "armed": true, "a_index": 1, "b_index": 1,
        "metrics": {
          "duration_s":       { "a": 23.451, "b": 23.404, "pct": -0.2 },
          "energy_wh":        { "a": 3.52,   "b": 3.3607, "pct": -4.53 },
          "amps":             { "a": 44.572, "b": 42.518, "pct": -4.61 },
          "lift_pct":         { "a": 75.005, "b": 73.131, "pct": -2.5 },
          "roll_abs_max_deg": { "a": 4.49,   "b": 3.29,   "pct": -26.73 }
        } }
    ],
    "only_in_a": [], "only_in_b": []
  },
  "texts": {
    "only_in_a": [], "only_in_b": [],
    "numbers_differ": [ { "text": "SIM Hit ground at {n} m/s",
                          "a": ["SIM Hit ground at 0.496668 m/s"], "b": ["SIM Hit ground at 0.490383 m/s"] } ]
  }
}

Read it in this order

Parameters first. They are often the whole point of a comparison, and here they confirm the setup. The SIM_WIND_* values differ because the log keeps the last value sent, and in B that was a gusting one. Q_M_THST_HOVER differs because the autopilot learns its hover throttle in flight. Values ArduPilot rewrites in every flight, STAT_* and ground pressure, are set apart under bookkeeping so they do not hide a real change. 1618 parameters are the same.

Then the phase alignment. Phases are keyed by mode, armed state and mission item, and matched by longest common subsequence. Both flights have the same four phases, so only_in_a and only_in_b are empty. Had B needed an extra mode change, it would be listed there on its own and every later pair would still line up.

Then the numbers, with their phases in mind. The steady wind shows the larger peak roll, 4.49° against 3.29°, which looks backwards until you remember where phases split: at mode changes, arms and disarms, not at your steps. The armed QLOITER phase holds the takeoff into a 6 m/s crosswind and the climb as well as the hover. To compare the hover alone, read the hover step’s own window, using its t_start_s and t_end_s from the step results:

log_fields(log="20261003T202311Z-gusty-b.BIN", msg="ATT", fields=["Roll", "Pitch"],
           t0=52.9, t1=72.9, step=0.5)

Last, the texts. Messages that differ only in their numbers are grouped under a template, so a touchdown speed of 0.497 against 0.490 m/s is one entry, not two unrelated ones.

One seed is one sample

B met one realization of the gust field. A 4.5 % energy difference between one steady and one gusty flight says little on its own. Fly B with several seeds, and A more than once, before reading a difference of a few percent. Because seeds repeat exactly, a later change can be tested against the very same set of gusts.

A cleaner B

Wind is one variable among many. The same recipe compares a parameter change (set={"Q_A_RAT_RLL_P": 0.3} in B only), a model file for a frame that reads one (model="<frame>:/path/to/b.json"), or a firmware rebuild: same_firmware and the banners show which build flew each log, and open_sitl stamps the binary’s build time and tree SHA in its provenance so a rebuild mid-campaign cannot relabel a flight.