Figuring out where some toit code is locking up
# helpa
addshore
05/14/2025, 1:31 PMSomeone has written some code, and I suspect that it is locking up somewhere blocking on some task / channel etc, and then a watchdog is triggering.
I have a core dump, stack memory, and also the code that is being run.
Is there any way to go from this to try and figure out rougly where it might have been blocked?
k
kasperl
05/14/2025, 1:52 PMGood question. I don't think we've done much post-mortem debugging of this sort up until now.
kasperl
05/14/2025, 1:53 PMI assume it is likely hard to reproduce?
a
addshore
05/14/2025, 1:53 PMI believe it takes about an hour 😛
addshore
05/14/2025, 1:53 PMand yes involves custom hardware, talking to custom services etc
k
kasperl
05/14/2025, 1:53 PMThat doesn't sound too bad.
kasperl
05/14/2025, 1:53 PMOh, wait. Now it sounds bad.
kasperl
05/14/2025, 1:53 PM🙂
a
addshore
05/14/2025, 1:54 PMI remember from some of my early "deadlocks" i seemed to always trace it back to some loop that would never yeield
I always managed to find them with more verbos logging within the loops and then figurign it out from that
addshore
05/14/2025, 1:54 PMso perhaps that is still the best approach for now
k
kasperl
05/14/2025, 1:55 PMProbably. I wonder if we could make the missed watchdog start with a soft signal that dumps a stack trace for all tasks or some such.
kasperl
05/14/2025, 1:59 PMYou could possibly also run with automatic logging of all task block/resume operations.
kasperl
05/14/2025, 2:00 PMThat would not really help if it is just forgotten yields, but if you're blocked on something, you should be able to tell.
kasperl
05/14/2025, 2:02 PMThis is where a task is suspended: https://github.com/toitlang/toit/blob/eb2f42303fad36ccd54da56b83ad9e0b2f564322/lib/core/task.toit#L340
This is where it is resumed later:
https://github.com/toitlang/toit/blob/eb2f42303fad36ccd54da56b83ad9e0b2f564322/lib/core/task.toit#L351
Using something like
print_ thisthiskasperl
05/14/2025, 2:04 PMYou may also be able to find the problematic task by having more than one watchdog.
a
addshore
05/14/2025, 2:04 PMThanks for all the pointers, I like the print_ one and might take a look at that if i go ahead and reproduce it tommorrow
k
kasperl
05/14/2025, 2:09 PMYou may only learn that a specific task is blocked through that. Next step might involve getting some kind of stack trace dumped - but it might be too verbose.