Add hardware-in-the-loop and fault injection tests

[usmanmehmood55]

Summary

This PR adds a hardware-in-the-loop (HIL) test suite for universal_ble.

The fixture is a physical nRF52 DK running purpose-built Zephyr firmware. The firmware lives in the separate universal_ble_hil_firmware repository.

The Windows suite currently has 64 working tests:

• 25 baseline tests for ordinary BLE behavior
• 39 fault injection tests (FIT) for errors, awkward timing, disconnects, stale operations, and recovery

The host controls the fixture over BLE. USB serial is used for firmware logs.

What the tests cover

The baseline suite covers:

• filtered scanning and advertisement data
• service, characteristic, and descriptor discovery
• exact characteristic and descriptor reads and writes
• writes with and without response
• notifications and indications
• subscription, unsubscribe, and resubscribe behavior
• notification ordering and binary integrity
• MTU reporting
• peripheral and host disconnects
• reconnects and repeated connection cycles
• concurrent operations
• connected-device enumeration
• ordinary GATT service changes

The FIT suite arms the firmware with a fault plan before starting an ordinary BLE operation. The fixture can then:

• return ATT read, write, or descriptor errors
• return empty values and values around ATT payload boundaries
• delay an operation beyond the host timeout
• disconnect while an operation is pending
• complete an old operation after the host has reconnected
• race host and peripheral disconnects
• force rapid disconnect and reconnect cycles
• change the GATT service database during reads, writes, and subscription changes
• send notifications with gaps, duplicates, reordered sequence numbers, and mixed payload sizes
• disconnect during notification bursts
• keep notifications queued while the host tears down the connection
• emit a notification while the CCC enable operation is still completing

Most tests also verify recovery. After the injected failure, the test performs another read, write, reconnect, or subscription. This catches cases where the original call fails correctly but leaves the connection or operation queue in a broken state.

These tests go through the complete path: Dart API, operation queue, Pigeon, Windows C++, WinRT, Windows Bluetooth stack, the radio link, and finally the Zephyr GATT server. This gives us coverage which mocks and software-only tests cannot provide.

Running the suite

These are manual hardware tests. They are not intended to run in normal pull-request CI because they require an nRF52 DK and exclusive access to a Bluetooth adapter.

Platform runners are generated locally and are not committed:

cd hil
flutter create --platforms=android,web,windows .
flutter pub get

Flash HIL fixture firmware, and run the tests:

flutter test integration_test/baseline_hil_test.dart -d windows
flutter test integration_test/fault_injection_hil_test.dart -d windows

[fotiDim]

@usmanmehmood55 thanks for the PR. Would https://www.adafruit.com/product/3406 or https://www.ebay.de/itm/406283688259 work?

[usmanmehmood55]

@fotiDim technically yes, but both of these boards have different ways of installing/flashing the firmware onto the boards, compared to the standard nRF development kit.

They use UART over the USB port, and need the firmware to be partitioned in their specific bootloader's styles, so I'd have to get one myself and test out the flashing process and do some trial and error on the partition sizes etc.

They also might require a physical button to be pressed during every installation/flashing, which would make CI automation difficult.

Standard nRF52 devkits have onboard flasher/debugger, so it'll work right out of the box, and won't require physical button presses.

I will look into the Adafruit board, it might be easier than the ebay link one. But again, the standard devkit would be the easiest.

[fotiDim]

HIL has analysis errors:

flutter analyze
Upgrading analysis_options.yaml to exclude build and platform directories.
Analyzing hil...                                                        

  error • The method 'read' can't be unconditionally invoked because the receiver can be 'null'. Try making the call conditional
         (using '?.') or adding a null check to the target ('!') • lib/main.dart:80:40 • unchecked_use_of_nullable_value
  error • The method 'setReadValue' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:84:26 • unchecked_use_of_nullable_value
  error • The method 'read' can't be unconditionally invoked because the receiver can be 'null'. Try making the call conditional
         (using '?.') or adding a null check to the target ('!') • lib/main.dart:86:35 • unchecked_use_of_nullable_value
  error • The method 'write' can't be unconditionally invoked because the receiver can be 'null'. Try making the call conditional
         (using '?.') or adding a null check to the target ('!') • lib/main.dart:93:26 • unchecked_use_of_nullable_value
  error • The method 'read' can't be unconditionally invoked because the receiver can be 'null'. Try making the call conditional
         (using '?.') or adding a null check to the target ('!') • lib/main.dart:95:35 • unchecked_use_of_nullable_value
  error • The method 'readState' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:99:29 • unchecked_use_of_nullable_value
  error • The method 'write' can't be unconditionally invoked because the receiver can be 'null'. Try making the call conditional
         (using '?.') or adding a null check to the target ('!') • lib/main.dart:106:26 • unchecked_use_of_nullable_value
  error • The method 'read' can't be unconditionally invoked because the receiver can be 'null'. Try making the call conditional
         (using '?.') or adding a null check to the target ('!') • lib/main.dart:114:30 • unchecked_use_of_nullable_value
  error • The method 'subscribe' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:122:26 •
         unchecked_use_of_nullable_value
  error • The method 'values' can't be unconditionally invoked because the receiver can be 'null'. Try making the call conditional
         (using '?.') or adding a null check to the target ('!') • lib/main.dart:123:37 • unchecked_use_of_nullable_value
  error • The method 'requestNotification' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:125:26 •
         unchecked_use_of_nullable_value
  error • The method 'unsubscribe' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:133:26 •
         unchecked_use_of_nullable_value
  error • The method 'subscribe' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:137:26 •
         unchecked_use_of_nullable_value
  error • The method 'values' can't be unconditionally invoked because the receiver can be 'null'. Try making the call conditional
         (using '?.') or adding a null check to the target ('!') • lib/main.dart:138:37 • unchecked_use_of_nullable_value
  error • The method 'requestIndication' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:140:26 •
         unchecked_use_of_nullable_value
  error • The method 'unsubscribe' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:148:26 •
         unchecked_use_of_nullable_value
  error • The method 'subscribe' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:152:26 •
         unchecked_use_of_nullable_value
  error • The method 'values' can't be unconditionally invoked because the receiver can be 'null'. Try making the call conditional
         (using '?.') or adding a null check to the target ('!') • lib/main.dart:153:37 • unchecked_use_of_nullable_value
  error • The method 'requestNotificationBurst' can't be unconditionally invoked because the receiver can be 'null'. Try making the
         call conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:154:26 •
         unchecked_use_of_nullable_value
  error • The method 'unsubscribe' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:167:26 •
         unchecked_use_of_nullable_value
  error • The property 'deviceId' can't be unconditionally accessed because the receiver can be 'null'. Try making the access
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:176:22 •
         unchecked_use_of_nullable_value
  error • The method 'readState' can't be unconditionally invoked because the receiver can be 'null'. Try making the call
         conditional (using '?.') or adding a null check to the target ('!') • lib/main.dart:180:29 •
         unchecked_use_of_nullable_value

Also since the PR is large do you think it could be broken down into smaller ones?

[usmanmehmood55]

It seems I was on Flutter 3.44.3 (Dart 3.12.2), and you were on the latest Flutter 3.47.0 (Dart 3.13.0). I've upgraded my side and fixed the issues.

Also since the PR is large do you think it could be broken down into smaller ones?

Yes, I agree. Maybe I should split the PR into HIL tests, and production code improvements.

[usmanmehmood55]

@fotiDim The windows production code changes have been moved out of this PR, so now only 15 changed files remain.

I also had an idea about the nRF52 devkits. Perhaps we could contact Nordic Semi about providing you the devkits. They often send out free devkits for evaluations and are very supportive of open source work related to their products. I think they'll find this use case worthwhile.

We can ask them for 3-4 devkits, and can scale the tests to use 3-4 BLE instances. This will allow us to test concurrency in a nice way.

[fotiDim]

I also had an idea about the nRF52 devkits. Perhaps we could contact Nordic Semi about providing you the devkits. They often send out free devkits for evaluations and are very supportive of open source work related to their products. I think they'll find this use case worthwhile.

We can ask them for 3-4 devkits, and can scale the tests to use 3-4 BLE instances. This will allow us to test concurrency in a nice way.

Good thinking. I contacted them already...

[fotiDim]

@usmanmehmood55 should we expect another PR for Windows hardening?

[usmanmehmood55]

@fotiDim Yes. I'll create it today.
Edit: please see #284