https://toitlang.org/ logo
Powered by
Extended Scan Response with BLE
# help
t
AdvertisementData
is limited to 31 bytes in 
ble.toit
. How does one get extended scan responses? I am unable to see the 
name
or 
manufacturer-data
of BLE beacons I am scanning. Here's a short code snippet I am using and the result.
Copy code
toit
import ble
import encoding.hex


BEACON-SERVICE ::= ble.BleUuid "FEAA"
SCAN-DURATION   ::= Duration --s=3

main:
  adapter := ble.Adapter
  central := adapter.central

  while true:
    addresses := Set
    data := []

    central.scan --duration=SCAN-DURATION: | device/ble.RemoteScannedDevice |
      if device.data.service-classes.contains BEACON-SERVICE:
        address := hex.encode device.address
        addresses.add (address.to-ascii-upper.trim --left "00")
        data.add device.data.manufacturer-data

    print addresses
    print data
    sleep Duration/5
Result:
Copy code
{BC5729059011}
[#[]]
Env:
Copy code
bash
[tej@a1278 temperature_beacon]$ jag firmware
Device 'eloquent-fortune' is running Toit SDK v2.0.0-alpha.164

Running on ESP32-Wroom-32 (Olimex ESP32 Gateway)
What am I missing? I have looked through 
ble.toit
and 
remote.toit
and can't see any alternative scan methods nor any flags I can set.
f
I'm currently traveling but will take a look next week (Tuesday or later). If you have esp-idf code (in C) that does the right thing that would help, as I could compare it to what we do at the moment. (But not a requirement).
t
I think I have found the issue. Nimble does passive scans by default. I also see this in [
resources/ble_esp32.cc
](https://github.com/toitlang/toit/blob/6a2f27006dd25601046c7c996bbc3e159b10842c/src/resources/ble_esp32.cc#L2444):
Copy code
/**
   * Perform a passive scan.  I.e., don't send follow-up scan requests to
   * each advertiser.
   */
  disc_params.passive = 1;
I have been using Bluedroid - let me compile a working example in Nimble and share it tomorrow.
f
Great feedback. Thanks
t
Here's a minimal working example using Nimble with ESP-IDF. As I suspected we are not getting the scan response unless we set 
disc_params.passive = 0
https://github.com/tejpochiraju/nimble_beacon_scanner/blob/main/main/main.c
f
Thanks! I will probably add a flag to the scan method, or, if I don't find any reason against it, enable active by default.
t
I would recommend keeping 
passive
as the default and exposing a flag instead - consumes less energy in 
passive
. And keeps the defaults consistent with upstream Nimble configuration. I was trying to do just that locally and build a custom envelope but am currently running into network issues while running 
git submodule init
- will try again later.
f
When building locally (or even on GitHub builders) set the 
TOIT_GIT_VERSION
to the version Jaguar/Artemis use. Otherwise the tooling will complain that the versions don't match. (Normally that's not completely safe, but I don't remember any recent breaking changes that would make the bytecodes incompatible)
t
I am not getting 
name
or advertisement data on the Toit side of things even after changing 
disc_params.passive = 0;
in 
src/resources/ble_esp32.cc
. I verified that the modified ble_esp32.cc is being compiled by adding a few 
ESP_LOGE
statements. Specifically, I changed the 
BleCentralManagerResource::_on_discovery
function to print the raw event data (see attached file and output below). To be clear, 
disc_params.passive = 0;
is working but the extra data is not being used. I can see that the data is received in this function but is not being passed further? Also, not sure how to handle the raw advertisement data on the Toit side of things - this is not 
manufacturer-data
but raw bytes instead.
Copy code
bash
E (7363) BLE: Discovery Event:
E (7363) BLE:   Address: 11:90:05:29:57:bc
E (7363) BLE:   RSSI: -61
E (7363) BLE:   Event Type: 0
E (7363) BLE:   Data Length: 29
E (7363) BLE:   Advertisement Data:
  02 01 06 03 03 aa fe 15 16 aa fe 21 04 0f 0b d4
  18 f8 43 0c 00 d6 ff f0 03 c4 01 1a 2e
E (7373) BLE: Discovery Event:
E (7383) BLE:   Address: 11:90:05:29:57:bc
E (7383) BLE:   RSSI: -62
E (7383) BLE:   Event Type: 4
E (7393) BLE:   Data Length: 24
E (7393) BLE:   Advertisement Data:
  09 16 80 20 66 09 00 00 00 00 0d 09 42 4c 45 30
  30 30 30 30 30 30 31 31
Steps:
Copy code
TOIT_GIT_VERSION=v2.0.0-alpha.164 make esp32
# beacon.toit is the example code from the first post
toit compile --snapshot -o beacon.snapshot beacon.toit
toit tool firmware -e build/esp32/firmware.envelope container install beacon beacon.snapshot
toit tool firmware -e build/esp32/firmware.envelope flash --port /dev/ttyUSB0 --baud 921600
jag monitor
https://cdn.discordapp.com/attachments/1307381608537657415/1308139646815047790/ble_esp32.cc?ex=673cdb7b&is=673b89fb&hm=11a66ee06e29d2795dd0d4339417a08043ed11b68c47ad8a7feeb245ce073221&
f
I will try to repro tomorrow. Thanks
Started to look into this. I have a meeting in a few minutes, but here is my current status: - I was easily able to run your modified code. Thanks! - the Event Type for active scans seems to be 4. - the received data (that has the debug prints) is stored in a 
DiscoveredPeripheral
object and then put into a queue. - the 
scan_next
function (2480) then takes the next entry and passes it on to the Toit code. - there is a 
ble_hs_adv_parse_fields
(ble_esp32.cc:2502) that decomposes the received data into what is usually expected from advertisement packets. If I understand you correctly, we shouldn't call 
ble_hs_adv_parse_fields
for active-scan responses but pass the data directly to the Toit code. If that's the case I just need to store the event-type in the 
DiscoveredPerihperal
object and then simply avoid the parsing-call.
t
Let me dig a bit more into ble_hs_adv_parse_fields before I comment on the behaviour. I am afraid I haven't had to dig into the details previously so I am also figuring out the "correct" behaviour.
f
Looking too.
After some reading, I'm pretty sure that the scan response packet follows the same structure as the advertisement packet. In your example, there is a 
09 16 80 20 66 09 00 00 00 00
which seems to be a service data - 16-bit UUID. It seems like the 16-bit UUID is 0x2080 and the data is 
66 09 00 00 00 00
. After that, the 
0d 09 42 4c 45 30 30 30 30 30 30 30 31 31
decodes as name 
BLE000000011
. It looks like we don't copy these fields to the Toit side yet. If the macos-side allows it I will probably just move the decoding of the frame into Toit code. That would give access to the raw data, and also make it easier to implement the missing fields.
@Tej you changed the interval and the window. Do you have some information on whether that's something that should be by default? Should we expose it to the Toit level?
t
I did a literal translation of the parameters from our current code base to Nimble. I just checked with interval and window = 0 (default) and there's no difference in the result. I doubt we need those and I suspect these are left over from some old experimentation. In terms of exposing it, maybe make these optional with defaults as 0? Where does Toit sit on simplicity vs power?
f
Exposing it with defaults sounds good.
We are not opposed to expose complicated things as long as it starts simple.
t
I am not clear on what you mean by moving the decoding of the frame into Toit code? Would a developer have access to the raw bytes? Reason I ask is, there's an increasing number of BLE sensors and I am not sure all of them allow standard decoding.
f
Yes. I have started to refactor the code so that the packet data (31 bytes) is given to the Toit code (the BLE library). I can now implement the decoding there.
And I would expose the raw data. Most users wouldn't need it, but it shouldn't hurt.
That said. I'm actually splitting it up into fields right now. So basically a partial decoding.
t
Not sure this is directly useful here but here's an Aruba document that details how they handle BLE advertisement. https://www.arubanetworks.com/techdocs/Instant_87_WebHelp/Content/instant-ug/custom-iap-param/ble-iot.htm I get why they added support for "known" beacons (business partnerships) but that's a losing battle as the number of beacons increases daily. We find their raw frames more useful.
f
The content of a ble advertising packed it well defined. So we could decode it in C and send the decoded data to Toit. I find it just more cumbersome
t
This is the bit I am unclear on the boundary between standard and common practice. Were you able to find a standard on the fields?
f
Yes
In the Bluetooth spec
t
Cool. I had assumed the payloads from a couple of our vendors were not standards compliant but reading up, looks like they are.
I have been a bit tied up so haven't had a chance to dig further. Hoping for some time tomorrow.
f
No need. I'm busy refactoring that code. I'm confident it will solve the case you showed earlier.
t
Hi @floitsch - I have time this weekend so happy to support with this as you need.
f
Thanks for the offer. I'm making good progress and I think I will have some PRs ready on Monday. Just writing the tests took much longer than I anticipated.
t
No worries, I have handed off the primary code migration tasks to one of our engineers so my hands are bit freer to tackle general/common needs like BLE, WiFi (AP-STA switchover) and OTA. By the end of the year, I expect we would have moved 3 if not 4 of our products to Toit. That's 3 years of C compressed into 1.5 months of Toit!
f
Let me know if they solve your issues. (Once they have been committed).
t
Thanks! Will give it a try.
k
Jaguar v1.45.0 comes with the new SDK that bundles all the changes.
@Tej Did you get a chance to try the changes?
t
@kasperl will test this tomorrow. Got tied up in something.
Have set aside time to do more thorough testing tomorrow as I couldn't quite get the scan working correctly with:
Copy code
import ble
import encoding.hex


BEACON-SERVICE ::= ble.BleUuid "FEAA"
SCAN-DURATION   ::= Duration --s=-1

main:
  adapter := ble.Adapter
  central := adapter.central

  while true:
    addresses := Set
    data := []

    central.scan --duration=SCAN-DURATION 
                 --active=true 
                 --limited-only=false
                 --interval=160 
                 --window=158: 
                 | device/ble.RemoteScannedDevice |
      if device.data.services.contains BEACON-SERVICE:
        address := hex.encode device.address-bytes.stringify
        print device.identifier
        print device.address-bytes
        print device.address-type
        print device.data
    sleep Duration/5
Output:
Copy code
an instance with class-id 55
#[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
#[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
0
## Notes 1. 
--limited-only=true
does not give a result at all - whereas this parameter did not seem to have any effect in my Nimble example. 2. Changing 
interval
and 
window
did not yield data either. Tomorrow I will build from source and print logs from the C layer to dig deeper.
f
Interesting. Curious about your findings. Thanks for testing.
t
I think I misinterpreted the data structure. I am infact getting the data correctly but not the name.
Copy code
// From C
E (6538) BLE: Discovery Event:
E (6538) BLE:   Address: 11:90:05:29:57:bc
E (6538) BLE:   RSSI: -89
E (6538) BLE:   Event Type: 0
E (6538) BLE:   Data Length: 29
E (6538) BLE:   Advertisement Data:
  02 01 06 03 03 aa fe 15 16 aa fe 21 04 0f 0b d4
  17 29 45 b6 ff e1 ff d2 03 f2 01 29 db
E (6548) BLE: Discovery Event:
E (6558) BLE:   Address: 11:90:05:29:57:bc
E (6558) BLE:   RSSI: -86
E (6558) BLE:   Event Type: 4
E (6568) BLE:   Data Length: 24
E (6568) BLE:   Advertisement Data:
  09 16 80 20 66 09 00 00 00 00 0d 09 42 4c 45 30
  30 30 30 30 30 30 31 31

// From Toit
ID: #[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address bytes: #[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address type: 0
Name: null
Data: #[0x02, 0x01, 0x06, 0x03, 0x03, 0xaa, 0xfe, 0x15, 0x16, 0xaa, 0xfe, 0x21, 0x04, 0x0f, 0x0b, 0xd4, 0x17, 0x29, 0x45, 0xb6, 0xff, 0xe1, 0xff, 0xd2, 0x03, 0xf2, 0x01, 0x29, 0xdb]
The ESP_LOGE statements are the same as my 
ble_esp32.cc
attachment from earlier. The Toit code is:
Copy code
print "ID: " + device.identifier.stringify
print "Address bytes: " + device.address-bytes.stringify
print "Address type: " + device.address-type.stringify
print "Name: " + device.data.name.stringify
print "Data: " + device.data.to-raw.stringify
Given the name comes from the second scan event (type 4), is it possible that the response from this event is not being copied over?
f
I will try to have a look later today or early tomorrow. If there isn't any update tomorrow please remind me.
Starting to look at this. In theory you should get 2 different 
RemoteScannedDevice
objects in Toit. It would be up to you to merge them.
You can use 
is-scan-response
to distinguish between the two: https://libs.toit.io/ble/remote/class-RemoteScannedDevice#is-scan-response
k
You may want to get used to 
print "ID: $device.identifier"
instead of the manual stringification and concatenation with 
+
🙂
t
Got it. I think I missed that in the docs. Unclear to me how I would go about the merge, though. This is probably mostly because I am yet to internalise the Toit way of working (see above!). Will I get multiple 
RemoteScannedDevice
entries. i.e. a 
List
? Or will the 
block
get called on each response, i.e. once for the 
Discovery
event and once for the 
Scan Response
? If it's the latter, I should have seen the data in my prints at least once but I didn't. I will refactor the code to do the merge and try again.
f
In theory your block should have been called twice; once for each response.
It's not documented, but you can easily merge two advertisements by just merging their data-block lists. From memory this should be something like:
Copy code
scanned-device1 := ...
scanned-device2 := ...
merged-blocks := scanned-device1.data.data-blocks + scanned-device2.data.data-blocks
merged-advertisement := Advertisement merged-blocks --no-check-size
print merged-advertisement.name
...
And I will update the doc (both for the scan-result as well as how to merge them).
t
Definitely getting multiple callbacks (count depends on 
duration
) but only for 
Discovery
.
Copy code
ID: #[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address bytes: #[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address type: 0
Name: null
Data: #[0x02, 0x01, 0x06, 0x03, 0x03, 0xaa, 0xfe, 0x11, 0x16, 0xaa, 0xfe, 0x20, 0x00, 0x0b, 0xd2, 0x17, 0x33, 0x00, 0x4b, 0xe8, 0x01, 0x
01, 0xf8, 0x3e, 0x16]
Is Scan Response: false

ID: #[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address bytes: #[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address type: 0
Name: null
Data: #[0x02, 0x01, 0x06, 0x03, 0x03, 0xaa, 0xfe, 0x15, 0x16, 0xaa, 0xfe, 0x21, 0x04, 0x0f, 0x0b, 0xd2, 0x17, 0x33, 0x47, 0xf3, 0xff, 0x
e1, 0xff, 0xd2, 0x03, 0xf2, 0x01, 0x2b, 0x06]
Is Scan Response: false

// Repeats...
I simplified the code to do a single scan:
Copy code
import ble
import encoding.hex


BEACON-SERVICE ::= ble.BleUuid "FEAA"
SCAN-DURATION   ::= Duration --s=1

main:
  adapter := ble.Adapter
  central := adapter.central

  central.scan --duration=SCAN-DURATION 
                --active=true 
                --limited-only=true
                --interval=160 
                --window=158: 
                | device/ble.RemoteScannedDevice |
    if device.data.services.contains BEACON-SERVICE:
      address := hex.encode device.address-bytes.stringify
      print "ID: $device.identifier"
      print "Address bytes: $device.address-bytes"
      print "Address type: $device.address-type"
      print "Name: $device.data.name"
      print "Data: $device.data.to-raw"
      print "Is Scan Response: $device.is-scan-response"
Can you point me to where you pass the data to Toit from 
ble_esp32.cc
- I can try and debug further from there.
f
Interesting. The discovered peripheral is created here: https://github.com/toitlang/toit/blob/9bc1f09a2da28139d598162f17182479a9351e0c/src/resources/ble_esp32.cc#L1725 We then take them out of the queue in the 
scan_next
function here: https://github.com/toitlang/toit/blob/9bc1f09a2da28139d598162f17182479a9351e0c/src/resources/ble_esp32.cc#L2485 I think the reason it doesn't work for you is that the scan-response doesn't have the service (
BEACON-SERVICE
) anymore. If you look at the raw data, you can see that the first advertisement data contains the service (
aa fe
), but the scan response doesn't. So you would need to collect the identifier (
discovered-device.identifier
) and then allow also scanned-device objects that don't have the 
BEACON-SERVICE
but where you know that they have that service. Something like:
Copy code
found-devices := {}
central.scan ...
  if device.data.contains-service BEACON-SERVICE or found-devices.contains device.identifier:
    found-devices.add device.identifier
    ...
t
Well, that was daft. You were right about 
BEACON-SERVICE
being missing from the scan response.
Copy code
ID: #[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address bytes: #[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address type: 0
Name: null
Data: #[0x02, 0x01, 0x06, 0x03, 0x03, 0xaa, 0xfe, 0x11, 0x16, 0xaa, 0xfe, 0x20, 0x00, 0x0b, 0xd2, 0x18, 0x02, 0x00, 0x4f, 0xa9, 0x65, 0x02, 0x0b, 0x0f, 0xd7]
Is Scan Response: false

ID: #[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address bytes: #[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address type: 0
Name: BLE000000011
Data: #[0x09, 0x16, 0x80, 0x20, 0x66, 0x09, 0x00, 0x00, 0x00, 0x00, 0x0d, 0x09, 0x42, 0x4c, 0x45, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x31, 0x31]
Is Scan Response: true
Happy to report this is working as needed. Thanks again for your patience and support!
2 Views
PreviousNext
Powered by