Test record
We test at three levels: an automated suite, simulated missions against the real firmware, and sessions with the hardware in the water and in the air. This page lists what has been run, its objective and its result.
Test levels
Hardware time is limited. A pool booking takes a morning, a boat session takes a day, and the aircraft needs a licensed pilot present. We find as many problems as possible in software and simulation first, and use field sessions for the tests that need water or air.
Automated suite, every commit
The suite runs in about a minute and covers the mission logic, the protocol and the course model.
Simulated mission, before a hardware session
The mission runs end to end against the real firmware over an emulated radio, with a pass or fail per stage.
Hardware sessions
Autonomy, safety systems and failsafes are tested on the water and in the air. Each session is recorded from start to finish, and the footage is used as readiness evidence.
Software suite
The suite has 154 tests. The last full run took 53 seconds and passed 153. The remaining test is the software in the loop round trip, which is opt in because it builds and boots firmware.
| Area | Tests | What it proves |
|---|---|---|
| Readiness rehearsal | 24 | The 3.1 requirement list, the course dimensions and the rehearsal criteria match the handbook |
| Course model | 22 | Course layout, the three live feeds and the browser scene build correctly |
| Suspend and resume | 21 | Task 4 interrupts a task and the task resumes at the step it stopped on |
| RoboCommand protocol | 21 | Envelopes, topics, sequence numbers and the run start handshake, end to end |
| MAVLink adapter | 21 | Message parsing, mode tables per firmware and stream setup |
| State machines | 16 | Every legal and illegal transition in the three machines |
| Software in the loop manager | 11 | Firmware process management, parameters and run directories |
| HaLow link emulator | 8 | Latency, loss, bandwidth cap and outage behaviour |
| Orchestrator | 5 | Startup, dispatch, shutdown |
| Embedded broker | 4 | Retained messages and subscription semantics for offline rehearsal |
| Software in the loop round trip | 1 | Opt in: boot, filter convergence, arm, a 40 m guided transit and an emergency stop |
Simulated missions
The validation scenario is our simulated competition run. It launches the three firmware builds, drives them with the same ground station that drives the hardware, and checks 25 stages from a cold start, through a Task 4 interruption, to an emergency stop.
| Session | Objective | Duration | Result |
|---|---|---|---|
| Validation scenario, software in the loop | Cold start to emergency stop with a Task 4 interruption in the middle, against the real firmware | about 5 min per run | 24 of 25 on the run of 21 September 2026. The open item is a guided waypoint that timed out on the boat; see below |
| Fringe link soak | The same mission at 8 percent packet loss with a two second fade every 25 seconds | per run | Health monitor escalates to an emergency stop on heartbeat loss, as designed. Led to trimming the telemetry streams |
| Headless scene check | Build the course in a browser engine with no screen, fly the mission, verify message order and camera behaviour | seconds | Pass in the suite whenever Node is installed |
| Readiness rehearsals | The four handbook 3.1 demonstrations flown on the specified courses and judged criterion by criterion | about 30 s each | All criteria pass in rehearsal. See proof of readiness |
| Communications rehearsal | The run start sequence of 3.4.10 against RoboNation's test server | 1 min 11 s | Run started with three vehicles declared |
Field and water sessions
| Session | System | Objective | Result | Evidence |
|---|---|---|---|---|
| Autonomous operations, August 2026 | BlueBoat | Fully autonomous run with the status light showing the mode throughout | Pass, submitted for readiness | Video, 21 s |
| Autonomous flights, August 2026 | Quadrotor | Takeoff above 3 m, the published pattern and a landing, flown in Auto | Pass, three missions recorded | Video, 1 min 25 s |
| Manual flights, August 2026 | Quadrotor | The same pattern flown by the pilot, with mode proof on the ground station | Pass, three missions recorded | Video, 2 min 15 s |
| Geofence and ceiling, August 2026 | Quadrotor | Command the aircraft past the fence and above the declared ceiling | Held at both limits | Video, 36 s |
| Link loss failsafe, August 2026 | Quadrotor | Switch the transmitter off in flight and observe the response | Landed five seconds after the link dropped | Video, 42 s |
| Pool buoyancy, August 2026 | Quadrotor | Ditch the aircraft and recover it | Floats upright and is recoverable by swimmer | Video, 7 s |
| Bench motor test | Quadrotor | Motor order and direction before first flight | Pass | Video |
| Communications rehearsal, 31 August 2026 | Ground station | Run the handbook start sequence against RoboNation's test server | Run 1 started, declaration sequence verified | Log and screencast |
| Pipeline survey in water | BlueROV2 | Depth hold survey along a pipeline mockup with the magnetic probe | Scheduled | — |
| Three vehicle integration | All | One run with all three vehicles autonomous and reporting together | Scheduled | — |
What testing changed
These test results led to design changes.
Underwater vehicle position
ArduSub does not allow guided modes without a position estimate, and the vehicle has no position estimate underwater. Depth hold is now used as the autonomous mode for that vehicle, and the survey is flown on heading and time with visual reacquisition.
Telemetry load on the radio
The default telemetry set overloaded the emulated link under the fringe profile. The adapter now requests position at 4 Hz and status at 1 Hz, and requests again if telemetry stops. This is a few packets per second per vehicle.
Arming delay
The filter needs 20 to 40 seconds after boot before prearm passes, and the position prearm check is the usual failure on the boat. Arming now retries for 90 seconds instead of failing the run.
Guided waypoint timeout
One check in the validation run fails: a guided waypoint on the boat does not report arrival inside its window, usually after the vehicle has been through an interruption. The mission still completes, so the cause is the acceptance window and retry logic, not navigation. This is the next item being worked on.
Reuse of the 2024 detector
The trained buoy model still works for the colour classes at 0.945 mAP@0.5. The shape classes do not transfer, because the 2026 buoys carry light beacons instead of shapes.
Planned tests before November
- Pipeline survey in water with the magnetic probe, including a repair hold on a mock light node.
- Docking approach and the water jet against a built docking bay.
- Delivery mechanism flights: pick a tin, fly a route, release on a circle.
- One session with all three vehicles autonomous at the same time, reporting through the ground station to a local broker.
- Validation scenarios for Tasks 2 and 3 alongside the existing Task 1 run.
- Fix the guided waypoint timeout so the validation run passes 25 of 25.