https://toitlang.org/ logo
Powered by
Jaguar routing problem, followed by container stop...
# help
a
I have added a watchdog to the container now so I expect it wont cause me an issue again, however I'm confused about what has happeend here The script was running, as shown by the INFO log line Then jaguar seems to have a panic, 
Routing problem
and tries to reconnect? fails? and backs off? Then I see my container print 
udptocsv
after which it tries to emit this over UDP Then there are logs to do with wifi reconnecting? After which jaguar seems to be back up? However the container is then stuck? I guess it is stuck here
Copy code
socket.send
    udp.Datagram row.to-byte-array address
Logs for the story above...
Copy code
INFO: Took 43.358ms, Sleeping for additional 1.956642s
[jaguar] WARN: running Jaguar failed due to 'Routing problem' (1/3)
[jaguar.http] INFO: running Jaguar device 'clever-weather' (id: '510b3170-cbd5-4a05-a82a-709b975e45bc') on 'http://192.168.68.71:9000'
[jaguar] WARN: running Jaguar failed due to 'Routing problem' (2/3)
[jaguar.http] INFO: running Jaguar device 'clever-weather' (id: '510b3170-cbd5-4a05-a82a-709b975e45bc') on 'http://192.168.68.71:9000'
[jaguar] WARN: running Jaguar failed due to 'Routing problem' (3/3)
[jaguar] INFO: backing off for 5s
udptocsv,1,2c54204d,2025-06-16 13:51:58,0, 36.799999999999997158, 3.6335000000000001741, 0.17612500000000000377

Decoding by jag, device has version <2.0.0-alpha.174>
EXCEPTION error.
  0: udp-send_                 <sdk>\net\modules\udp.toit:167:37
  2: Socket.send               <sdk>\net\modules\udp.toit:88:12
  3: emitData                  udpAds.toit:194:10
  4: innerMain                 udpAds.toit:170:7
  5: main                      udpAds.toit:119:3
******************************************************************************

[wifi] DEBUG: closing
E (7210621) wifi:NAN WiFi stop
[wifi] DEBUG: connecting
[wifi] DEBUG: connected
[wifi] INFO: network address dynamically assigned through dhcp {ip: 192.168.68.71}
[wifi] INFO: dns server address dynamically assigned through dhcp {ip: [8.8.8.8, 1.1.1.1]}
[jaguar.http] INFO: running Jaguar device 'clever-weather' (id: '510b3170-cbd5-4a05-a82a-709b975e45bc') on 'http://192.168.68.71:9000'
f
My initial guess is that the script also hit a WiFi issue, and then crashed when it couldn't send the UDP.
a
So my initial thought (as this is a container) would have been that the 
EXCEPTION error.
would trigger a crash and for the container to restart?
f
Jaguar doesn't automatically restart containers iirc
a
oh right 😛 So to acheive that ultimately I need something like this..? https://docs.toit.io/tutorials/containers/toit-sdk#starting-containers And to continually check my containers are running? I have potentially been overlooking something this whole time!
And or this can in another way be done via watchdogs I guess? as if each container always has a watchdog that should be expected, then it would always restart on any container failure?
f
You don't need to continuously check: you are already waiting for it (
container.wait
), so you can just put that code into a loop. Alternatively, you can also provide a patch to Jaguar to do that for you. If it's small enough I would accept it. Watchdogs would work too. If your program is supposed to check in every few seconds/minutes, but doesn't the device will eventually reset.
a
yeah, watchdogs feel a little fragile, as for example if a crash happens in a brief moment that you happen to have stopped then, that isnt ideal and I guess an expected state would then never recover unless you have something else checking?
What does the 
--critical
part of a 
container install
result in?
f
Not sure I understand. You generally start a watchdog at a good moment, and then let some important part of your code feed the dog. Depending on the needs of your device the interval at which the dog needs to be fed can be relatively large. The main thing is that you want to recover from a stuck device (or a device where your program isn't running). For example, if you are supposed to offload data every hour, then you can set a watchdog and set it to 5h and feed it whenever data has been offloaded. If that doesn't happen you will reset and get back into a fresh state.
If your program crashes and you restart the container, that's fine. Watchdogs are name-based. So if you reopen the same watchdog and feed it again, the dog is happy. It doesn't need to be the same original container that does the feeding.
a
So far our usecases are more on the side of, a device should be actively working constantly, multiple things happening a second, and if it breaks for some unexpected reason, we really want that to restart asap
f
iirc: 'critical' means that the program should run all the time.
you shouldn't rely on watchdogs alone.
in fact they are typically your last recourse.
a
> If your program crashes and you restart the container, that's fine. Watchdogs are name-based. So if you reopen the same watchdog and feed it again, the dog is happy. It doesn't need to be the same original container that does the feeding. Right, but that implies that you then already have something else watching to restart the container? If so, I think I might look at adding that to jag, seems like it might make sense
for our usecase right now, we count all containers as critical, as when they shouldnt be running, the second processor turns off the ESP anyway 😄
f
If you install containers without Jaguar, then 
--critical
should do. I think.
Otherwise, it's Jaguar that needs to do the loop.
a
aaah right, so perhaps there is another thing I misunderstood. When you said > Jaguar doesn't automatically restart containers iirc Does that mean normally they do restart on a system without jaguar? (Or is that then when 
--critical
comes into play?
f
Note that you sometimes need to be careful: having a 
hello world
container as critical would lead to infinite restarts of the container with "hello world" flooding your output.
On other systems (like Artemis), and maybe with the envelope installation there are other means, but none automatically restart without additional flags.
In Artemis I tend to do an 
interval: 1s
.
If the container is unable to run (for example, because some peripheral isn't working), then it has a delay of 1s before it tries again, instead of thousands of attempts.
a
I'm looking for where this 
interval
goes in artemis now, as I havnt use it yet, and we also might not!
f
It's a trigger.
Last line of this document: https://docs.toit.io/getstarted/fleet/pods
a
Right, yes, so that seems like the kind of behaviour I want, but possibly without needing / using 
artemis
?
Or perhaps I should be using 
artemis
even if I am not using it for OTA updates etc?
f
I have been planning, for a long time, to extract the pod-specification from Artemis and move/copy it to Toit. If you don't use its updating mechanism, I would probably not use it. Feels like unnecessary resources. But a lot of the ideas of Artemis are really nice. I will think a bit more about it. Maybe I will change my opinion. My biggest concern is that Artemis connects to the server when it boots. So you kind-of run into the issue that you need some kind of server.
Btw how do you do firmware updates?
a
So we have our primary processor put the ESP in bootloader, flash the stub, then flash the fw the second processor retrieves if over its cellular connection, and send to esp over i2c
f
I see.
Funny. Usually I would have expected it to be the other way round.
a
Yeah, for various reasons the primary processor for now will remain in ultimate control, and ultimately its its jobs to make sure that connectivity works, and also manage battery usage etc So a couple of common usecases are 1) Having the ESP powered on and actively working for lets say an 8h period, where its sending messages back and forth over i2c 10s a second, including sending messages that get forwarded over cellular 2) Having the ESP mostly powered off, powering it on at a set interval, having container start, run a bunch of customer define logic within a sec time period and power window, and then turning it off again There will be / is also then a base firmware that is on the ESP prior to client code living there that does a baseline set of messaging, to do with wifi scans, BT scans etc
So for now likely a "good" pattern for me to try, might be a management container that is run with 
--critical
which in turn just slowly loops making sure containers are running, very similar to what artemis is doing with its interval
f
If you are using Jaguar, I would probably try to add an interval flag. If you are using envelopes, then I would probably just install the container with 
--critical
.
a
Is the interval part of the container install command? Or jag.interval?
f
I would make it part of the 
container install
a
oh, or you mean I add it 😄
yeah, ok!
f
If it is too complicated, don't bother.
But might not be hard to add.
a
and 
--critical
already works with 
container install
correct? Its just undocumented in the CLI?
f
It works for the envelope container install.
a
that explains it, im looking at the wrong command again
f
it is documented:
Copy code
ᐅ toit tools firmware container install -h
Add a container to the envelope.

Usage:
  toit tool firmware --envelope=<file> container install [<options>] [--] <name> <image:file>

Aliases:
  add

Options:
      --assets file        Add assets to the container.
      --critical           Reboot system if the container terminates.
  -h, --help               Show help for this command.
  -o, --output file        Set the output envelope.
      --trigger none|boot  Trigger the container to run automatically. (default: boot)

Rest:
  image file   (required)
  name string  (required)

Global options:
  -e, --envelope file                                    Set the envelope to work on. (required)
      --output-format human|plain|json                   Specify the format used when printing to the console. (default: human)
      --verbose                                          Enable verbose output. Shorthand for --verbosity-level=verbose.
      --verbosity-level debug|info|verbose|quiet|silent  Specify the verbosity level. (default: info)
That said: apparently it's "reboot system if the container terminates".
Would need to experiment if that's the case or if it just restarts the container.
a
Yeah, a slightly different option would be another flag entirely something similar to https://docs.docker.com/engine/containers/start-containers-automatically/#use-a-restart-policy
which feels slightly different to trigger or critical right now
f
Would work too. We should probably use that for Artemis to make it more similar.
a
So, whats the right way to develop jag both in terms of the CLI but then also the jag container on a device Should I just be updating the container on the ESP by "installing" my modified container over the top of it?
well, I guess that won't work, as I'd be using jag to update jag 🤣 SO I guess it is re flash each time?
AAah, so probably it makes the most sense to use use the emulated device stuff
f
I typically 
jag firmware update
. Could be that there are better ways.
a
I assume I need to do a full 
make
for deving both the cli and container But again running into https://pastebin.com/v9azKjKJ Which I worked around last time bu just doing 
make jag
instead
so, that's fixable after this hack?
Copy code
(master) cd toit/tools && jag toit pkg install
(master) cd toit/system/extensions/host && jag toit pkg install
(master) make
Then toit makes, and then 
make
works in the jaguar repo again too Now sure how to correctly put a fix in for that, or document it though
f
So we are missing the install-packages when doing 
make
? Sounds easy to fix
a
i'd be happy to implement some sort of restart policy too, however interval probably works well enough for me right now 😄
f
Edit: removed message that was for another thread.
a
I was about to say that this device isn't doing any I2C stuff, but it is! The only E: related message we saw is the wifi one
Copy code
[wifi] DEBUG: closing
E (7210621) wifi:NAN WiFi stop
[wifi] DEBUG: connecting
f
my mistake. I answered to the wrong thread.
2 Views
PreviousNext
Powered by