https://toitlang.org/ logo
Powered by
MALLOC Failed Re-Creating SPI Device
# help
z
Copy code
Heap report @ out of memory in primitive 8:2:
  ┌───────────┬──────────┬─────────────────────────────────────────────────────┐
  │   Bytes   │  Count   │  Type                                               │
  ├───────────┼──────────┼─────────────────────────────────────────────────────┤
  │    5648   │    513   │  heap overhead                                      │
  │  173528   │    467   │  untagged                                           │
  └───────────┴──────────┴─────────────────────────────────────────────────────┘
  Total: 179176 bytes in 467 allocations (53%), largest free 116k, total free 152k

******************************************************************************
MALLOC_FAILED error.
  0: S6d05a1.draw-test-pattern s6d05a1.toit:560:20
  1: main.<block>              display_new.toit:43:12
  2: SmallInteger_.repeat      <sdk>/core/numbers.toit:1288:3
  3: main                      display_new.toit:42:5
******************************************************************************
I can reliably make this memory allocation fail by commenting and uncommenting a line in this code:
Copy code
main:
  print "initializing display..."
  driver := S6d05a1 bus cs 320 320 --reset=reset 
  5.repeat:
    driver.draw-test-pattern
    sleep --ms=600
... in draw-test-pattern
Copy code
draw-test-pattern:
    start:= Time.monotonic-us
  
    left := random 0 250
    right := left + 75
    top := random 0 250
    bottom := top + 75
    set-range_ CMD-CASET_ top bottom
    set-range_ CMD-RASET_ left right

    color := random 255
    print "Color: $color"
    bluary := #[color, color, color, color, color, color, 0,0]
    chomps := #[]
    times := 12
    times.repeat:
      chomps += chomp bluary

    device_.transfer --command=0 #[0x2C]
    device_ = bus_.device --cs=cs_ --frequency=6_000_000
    //Here is where we shift into manually having calculated the 9-bit arrays called Chomps
    //Set the device back
    device_ = bus_.device --cs=cs_ --command-bits=1 --frequency=100_000
    end := Time.monotonic-us
    print "Duration: $((end - start) / 1_000) ms"
The line that I can turn on that causes the out of memory is:
Copy code
device_ = bus_.device --cs=cs_ --frequency=6_000_000
Copy code
MALLOC_FAILED error.
  0: S6d05a1.draw-test-pattern s6d05a1.toit:560:20
  1: main.<block>              display_new.toit:43:12
  2: SmallInteger_.repeat      <sdk>/core/numbers.toit:1288:3
  3: main                      display_new.toit:42:5
why I need to do this: - I need to support 3-wire SPI which uses 9-bit "bytes" that I call "chomps" <- slightly bigger than a bite
- the SPI primitives allow for this using the 
--command
flag but it slows down drawing significantly vs packing my own byte arrays and using device.transfer
so I switch back and forth between a "convenient" spi device, where I use the --command-bits flag; and a "fast" spi device, where I insert my own command bits as needed
f
z
ok that fits what I am seeing where if I add in the lines that re-set 
device_
to a new instance of the SPI primitive then it runs OOM
f
Now the question is why there is an OOM error.
It looks like there is still quite a lot of free memory.
z
I mean I guess technically it's not an OOM error but a MALLOC_FAILED error.
so allocating memory could fail for other reasons apart from there not being enough?
like are SPI instances special because of DMA
f
I don't think DMA is relevant, but we have a separation of Toit heap, and C-heap.
Also, there could be fragmentation.
There are a few functions that help debug memory issues. I'm searching in the discord history to find them.
z
if it helps, here is what I get on a regular ESP32 instead of the past traces which have been ESP32-C6
Copy code
Heap report @ out of memory in primitive 8:2:
  ┌───────────┬──────────┬─────────────────────────────────────────────────────┐
  │   Bytes   │  Count   │  Type                                               │
  ├───────────┼──────────┼─────────────────────────────────────────────────────┤
  │    4112   │      3   │  external byte array                                │
  │   45056   │     13   │  toit processes                                     │
  │     16384 │        4 │    system    0 c6e1701e-f05b-c078-c7ca-86c80db24c59 │
  │     20480 │        5 │    other     1 e83e33a6-0922-fccf-aef3-e6d2215dfaef │
  │      8192 │        2 │    current   3 512c9a75-e49f-7a4b-0f71-b2f7206ac0bb │
  │   16384   │      1   │  heap metadata                                      │
  │    4096   │      1   │  spare new-space                                    │
  │    6280   │     30   │  lwip                                               │
  │    6224   │    476   │  heap overhead                                      │
  │    2192   │     33   │  event source                                       │
  │    7848   │    129   │  thread/other                                       │
  │   26400   │     21   │  thread/spawn                                       │
  │   50944   │    161   │  untagged                                           │
  │   11608   │     50   │  wifi                                               │
  └───────────┴──────────┴─────────────────────────────────────────────────────┘
  Total: 181144 bytes in 440 allocations (75%), largest free 32k, total free 58k

***********************************************************
MALLOC_FAILED error.
  0: S6d05a1.draw-test-pattern s6d05a1.toit:555:20
  1: main.<block>              display_new.toit:43:12
  2: SmallInteger_.repeat      <sdk>/core/numbers.toit:1288:3
  3: main                      display_new.toit:42:5
f
- 
system.print-objects
. Makes it easier to see whether there are leaks. - 
esp32.memory-page-report
. Makes it easier to detect fragmentation. - 
esp32.dump-heap
. Big fat dump of how Toit sees the memory. And 
system.serial-print-heap-report
prints something similar to what happens when there is an OOM.
So my suggestion is: try to run these functions when you get close to an OOM. It might help us determine what the reason is (fragmentation, no malloc-heap, ...).
There is also the possibility, that the primitive reports the wrong error. Where it claims that there is no memory, but in reality something else is failing. This has happened in i2c, where we just assumed that a failed operation was due to OOM. iirc.
And sometimes there is memory that is independent. Like for BLE where there is "special" BLE memory. But that's unlikely to be the case here.
also: make sure to close the devices when you don't use them anymore.
The GC should get rid of them eventually, but for hw-resources an explicit 
close
is preferred.
z
welp, adding in 
close
did the trick.
f
might still mean that we are missing a finalizer.
but nice that it solved the issue.
z
Copy code
Heap report @ out of memory in primitive 8:2:
  ┌───────────┬──────────┬─────────────────────────────────────────────────────┐
  │   Bytes   │  Count   │  Type                                               │
  ├───────────┼──────────┼─────────────────────────────────────────────────────┤
  │    4192   │     12   │  external byte array                                │
  │   45056   │     13   │  toit processes                                     │
  │     16384 │        4 │    system    0 c6e1701e-f05b-c078-c7ca-86c80db24c59 │
  │     20480 │        5 │    other     1 e83e33a6-0922-fccf-aef3-e6d2215dfaef │
  │      8192 │        2 │    current  12 3aa61f86-aee4-8c91-982a-f143894a47d0 │
  │   16384   │      1   │  heap metadata                                      │
  │    4096   │      1   │  spare new-space                                    │
  │    6256   │     29   │  lwip                                               │
  │    6368   │    494   │  heap overhead                                      │
  │    2192   │     33   │  event source                                       │
  │    8032   │    136   │  thread/other                                       │
  │   26400   │     21   │  thread/spawn                                       │
  │   50912   │    160   │  untagged                                           │
  │   11608   │     50   │  wifi                                               │
  └───────────┴──────────┴─────────────────────────────────────────────────────┘
  Total: 181496 bytes in 454 allocations (76%), largest free 28k, total free 58k

******************************************************************************
MALLOC_FAILED error.
  0: S6d05a1.draw-test-pattern s6d05a1.toit:563:20
  1: main.<block>              display_new.toit:45:12
  2: SmallInteger_.repeat      <sdk>/core/numbers.toit:1288:3
  3: main                      display_new.toit:43:6
******************************************************************************
well now I can make it run out of memory allocation again if I try to allocate too big of a buffer to send over SPI
Chops array size: 576 Heap report @ out of memory in primitive 8:2:
@floitsch here's the new area where commenting it in or out causes the crash:
Copy code
device_.close
    device_ = bus_.device --cs=cs_ --frequency=6_000_000
    //((left - right) * (top - bottom) / 8 * 3   / times).repeat:
    //  device_.transfer chomps
if I comment those lines back in then I get a MALLOC_FAILED. I will add in those 3 lines and try to get a file attached here
f
Can you try to figure what kind of memory issues we are having by using the functions mentioned above? I will try to repro tomorrow, but more information is always welcome
z
started a GH issue at https://github.com/toitlang/toit/issues/2900 so that I can attach bigger files
f
Didn't have the time today, but I'm optimistic for tomorrow
2 Views
PreviousNext
Powered by