https://toitlang.org/ logo
Join Discord
Powered by
Use of Bucket.open --ram ?
# help
d
Values stored in RAM are retained between reboots, so long as power is maintained, right? I have tried this code on two ezsbc ESP32 boards with fully charged LiPo's and JAG monitoring via USB. If I use --flash, the storage count increases. I I use --ram, the storage count is always 1, nothing is retained after deep-sleep. Using JAG v1.9.1, I just can't see what I am doing wrong.
Copy code
import esp32
import system.storage as storage


// RAM_STORE_       ::= storage.Bucket.open --flash   "/admin"
RAM_STORE_       ::= storage.Bucket.open --ram   "/admin"


main:

  RAMBuffer.add Time.now.s_since_epoch

  print "size after $RAMBuffer.size"

  esp32.deep_sleep (Duration --s=5)


class RAMBuffer:

  static exists -> bool:
    return null != (RAM_STORE_.get "buffer")

  static get_buff_ -> List:
    buffer := RAM_STORE_.get "buffer"
    if not buffer: buffer = []
    return buffer

  static add entry/any -> none:
    buffer := List.from get_buff_
    buffer.add entry
    RAM_STORE_["buffer"] = buffer

  static clear -> none:
    RAM_STORE_["buffer"] = []

  static size -> int:
    return get_buff_.size

  static remove_first -> any:
    buffer := get_buff_
    entry := buffer.first
    RAM_STORE_["buffer"] = buffer[1..]
    return entry
k
I will take a look first thing Monday morning if not before! Thanks 🙏
d
thx, https://github.com/davidg238/test_bucket (To recreate issue. Always helpful to actually push)
k
I can (trivially) reproduce the issue. Will look into a fix 🙂
Found the problem. Pretty embarassing 😉
We store the key/value mapping as a encoding.tison encoded message, but the decoder rejects messages that have extra bits at the end and I always pass it the full buffer.
The fix is to store the encoded message size in addition to the encoded message. Pretty simple.
Will get this rolled out into Jaguar v1.9.2 early next week.
Thanks for reporting this, @davidg238!
Fix landed on main.
Fix shipped in Jaguar v1.9.3. Give it a spin, @davidg238 🙂
d
Thx. I don't know what to tell you. Using https://github.com/davidg238/sleepy_sensor and your Makefile. The test.toit works:
Copy code
rst:0x5 (DEEPSLEEP_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12700
ho 0 tail 12 room 4
load:0x40080400,len:2916
entry 0x400805c4
E (53) psram: No PSRAM detected
E (53) spiram: SPI RAM supported, but not found.
[toit] INFO: starting <v2.0.0-alpha.55>
starting
.... ezSBC Feather 09c2e1345a79 started
storing {"ti": "273"}
size 3 after storing
Entering deep sleep for 15000ms
ets Jul 29 2019 12:21:46

rst:0x5 (DEEPSLEEP_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12700
ho 0 tail 12 room 4
load:0x40080400,len:2916
entry 0x400805c4
E (53) psram: No PSRAM detected
E (53) spiram: SPI RAM supported, but not found.
[toit] INFO: starting <v2.0.0-alpha.55>
starting
.... ezSBC Feather 09c2e1345a79 started
storing {"ti": "288"}
size 4 after storing
data: {"ti": "243"}
data: {"ti": "258"}
data: {"ti": "273"}
data: {"ti": "288"}
buffer emptied, size now 0
Entering deep sleep for 15000ms
ets Jul 29 2019 12:21:46
k
That's good, no? I expect there is a follow up 🙂
d
but when I try to run sleepy_sensor.toit, when it attempts to unload the buffer, I get an exception (then loop):
error
(hit discord limit, apologies for delay)
don't waste your time, I did something dumb, will advise tomorrow
k
Looking forward to it. For what it's worth, you don't have to store string values in the buckets, so you should be able to just store the kind of maps you're using to structure your data.
🙂
d
note the "clearing RTC memory: invalid checksum". Is the following a normal boot sequence?
Copy code
david@MSI-7D43:~$ jag monitor
Starting serial monitor of port '/dev/ttyUSB0' ...
ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12700
ho 0 tail 12 room 4
load:0x40080400,len:2916
entry 0x400805c4
E (586) psram: No PSRAM detected
E (586) spiram: SPI RAM supported, but not found.
[toit] INFO: starting <v2.0.0-alpha.55>
[toit] DEBUG: clearing RTC memory: invalid checksum
[wifi] DEBUG: connecting
[wifi] DEBUG: connected
[wifi] INFO: network address dynamically assigned through dhcp {ip: 192.168.0.244}
[wifi] INFO: dns server address dynamically assigned through dhcp {ip: [192.168.0.1]}
[jaguar] INFO: running Jaguar device 'hungry-egg' (id: '2a1e7668-614e-4ff0-aa36-6b047b432d52') on 'http://192.168.0.244:9000'
k
Yeah, that looks pretty normal.
You shouldn't see that after deep sleep.
d
I am focussed on test2.toit in the repo , which excludes deep sleep
for speed/simplicity
I am getting errors like:
Copy code
david@MSI-7D43:~$ jag monitor -a
Starting serial monitor of port '/dev/ttyUSB0' ...
[jaguar] INFO: program e033b5d1-6478-5145-a783-6b674c445811 started
starting
.... ezSBC Feather 09c2e1345a79 started
stored {"ti": 1197, "v": 3.0}, size now 1
stored {"ti": 1202, "v": 4.0}, size now 2
stored {"ti": 1207, "v": 3.0}, size now 3
stored {"ti": 1212, "v": 4.0}, size now 4
stored {"ti": 1217, "v": 4.0}, size now 5
stored {"ti": 1222, "v": 4.0}, size now 6

******************************************************************************
EXCEPTION error. 
WRONG_OBJECT_TYPE
  0: decode                    <sdk>/encoding/tison.toit:41:3
  1: Bucket.get                <sdk>/system/storage.toit:84:29
  2: Bucket.get                <sdk>/system/storage.toit:73:12
  3: BufferStore.get_buff_     admin.toit:24:22
  4: BufferStore.size          admin.toit:37:12
  5: main                      test2.toit:45:19
******************************************************************************

[jaguar] ERROR: program e033b5d1-6478-5145-a783-6b674c445811 stopped - exit code 1
I thought maybe it was to do with floating point battery voltage, formatting the float ... so I switched to a random number
still unpredictable
k
I can take a look in 20 minutes or so!
d
again, will keep working to isolate for you, just an fyi
there is a clear.toit to re-initialize the buffer?
k
List.from get_buff looks odd.
Is that to copy it?
d
that was the issue the tison delivered non-resizable Lists
k
Ah. Got it.
d
(think I got the evaluation paths right, so that is the only place it is needed)
if you look at the first "test" in test2.toit, if I uncomment that + change loop to >=3, it work fine:
Copy code
[jaguar] INFO: program cdca5a03-c7ca-52f0-88bf-e727749a4e0c started
starting
.... ezSBC Feather 09c2e1345a79 started
stored {"ti": 2077, "v": 3}, size now 1
stored {"ti": 2082, "v": 3}, size now 2
stored {"ti": 2087, "v": 4}, size now 3
buffer full, emptying
data: {"ti": 2077, "v": 3}
data: {"ti": 2082, "v": 3}
data: {"ti": 2087, "v": 4}
buffer emptied, size now 0
stored {"ti": 2092, "v": 3}, size now 1
stored {"ti": 2097, "v": 3}, size now 2
stored {"ti": 2102, "v": 3}, size now 3
buffer full, emptying
data: {"ti": 2092, "v": 3}
data: {"ti": 2097, "v": 3}
data: {"ti": 2102, "v": 3}
buffer emptied, size now 0
and there is no way 10 iterations is exhausting RTC memory
k
No, it looks odd. You can do the tison encoding without storing in RTC memory and then we can look at the sizes.
d
... so just changing to 10 iterations, blew out:
Copy code
[jaguar] INFO: program bbd19ddf-d50c-543e-9731-fc0be3e86827 started
starting
.... ezSBC Feather 09c2e1345a79 started
stored {"ti": 2250, "v": 4}, size now 1
stored {"ti": 2255, "v": 3}, size now 2
stored {"ti": 2260, "v": 3}, size now 3
stored {"ti": 2265, "v": 4}, size now 4
stored {"ti": 2270, "v": 3}, size now 5
stored {"ti": 2275, "v": 4}, size now 6

******************************************************************************
EXCEPTION error. 
WRONG_OBJECT_TYPE
  0: decode                    <sdk>/encoding/tison.toit:41:3
  1: Bucket.get                <sdk>/system/storage.toit:84:29
  2: Bucket.get                <sdk>/system/storage.toit:73:12
  3: BufferStore.get_buff_     admin.toit:24:22
  4: BufferStore.size          admin.toit:37:12
  5: main                      test2.toit:45:19
******************************************************************************

[jaguar] ERROR: program bbd19ddf-d50c-543e-9731-fc0be3e86827 stopped - exit code 1
k
It feels like a platform bug.
d
will make some direct tison encoding tests, as you suggest
... maybe it is in the 
remove_first
, since I am storing a 
ListSlice_
... some flakiness ? I found this when I recoded using a List as a buffer, the first loop of 10 worked, but on the 11th add I got:
Copy code
buffer emptied, size now 0

******************************************************************************
EXCEPTION error. 
SLICE_CANNOT_CHANGE_SIZE
  0: ListSlice_.resize         <sdk>/core/collections.toit:1792:5
  1: List.add                  <sdk>/core/collections.toit:289:7
  2: main                      tison.toit:34:15
******************************************************************************
Copy code
if my_buffer.size >= 10:
      print "buffer full, emptying"
      entry := null
      while my_buffer.size > 0:
        entry = my_buffer.first
        print "data: $entry"
        my_buffer = List.from my_buffer[1..]
      print "buffer emptied, size now $my_buffer.size"
in a test case seems more stable
k
Yeah. I thought about that too. You can try doing buffer[1..].copy.
Let me look into this.
I can reproduce on my Mac.
d
fwiw, v1.0.5 of the repo has the buffer[1..].copy change. So you can reproduce an issue, outside my code?
small numbers of iterations (5) "seem" to work ... 10, no go
k
Yeah. I know what the problem is now 🙂
It's pretty weeeeeiiiiirrrrd.
But totally reproducible.
And very much fixable.
When we send byte arrays across the messaging boundary, we sometimes steal the memory and transfer it without copying. That does mean that we end up dropping the cached byte arrays service side, which means that they turn up as empty arrays (non-decodeable) after that.
d
ah, ok ... np, none of this is urgent. I was just updating an article on power-saving. Can wait til next release
thx for your time
k
Thank you for the report! It is a really easy fix, but I just need to think a bit about how to make it efficient.
I've improved the testing and fixed the issue you were bitten by in Jaguar v1.9.4. I hope things work better for you with the latest bits 🤞
d
I am using flash for testing now, I cannot get 
--ram
to retain values between 
deep_sleep
(using LiPo backed ESPs). This is the simplest example I could think of, to reproduce:
Copy code
import system.storage
import esp32

// with ram it fails, flash it works
bucket := storage.Bucket.open --ram "/admin"

main:

  list := bucket.get "mail"  --if_absent= (: bucket["mail"] = [ ])
  
/* the alternative loop, not using deep sleep, works for ram and flash
  5.repeat:
    print "list  $bucket["mail"]"
    add Time.now.s_since_epoch
    sleep --ms= 5_000
*/
  print "list  $bucket["mail"]"
  add Time.now.s_since_epoch
  esp32.deep_sleep (Duration --s=10)

add entry/any -> none:
    buffer := List.from bucket["mail"]
    buffer.add entry
    bucket["mail"] = buffer
k
Can you show me the message you get on the serial output when it boots after the deep sleep?
I am a bit surprised that this is so hard to get to work, but we will get there. We hope to spin up more on-device testing (maybe emulated using qemu) to catch these issues quicker.
d
Copy code
david@MSI-7D43:~$ jag monitor -a
Starting serial monitor of port '/dev/ttyUSB0' ...
[wifi] INFO: network address dynamically assigned through dhcp {ip: 192.168.0.240}
[wifi] INFO: dns server address dynamically assigned through dhcp {ip: [192.168.0.1]}
[jaguar] INFO: running Jaguar device 'dev04' (id: '1e74b0ec-6156-4672-b7ba-a53ba5c8b18c') on 'http://192.168.0.240:9000'
[jaguar] INFO: denied request, header: 'X-Jaguar-Device-ID' was '288f65b5-7b34-450f-938e-04f19e99029d' not '1e74b0ec-6156-4672-b7ba-a53ba5c8b18c'
[jaguar] INFO: container 'sleep2' installed and started
list  []
Entering deep sleep for 10000ms
ets Jul 29 2019 12:21:46

rst:0x5 (DEEPSLEEP_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12700
ho 0 tail 12 room 4
load:0x40080400,len:2916
entry 0x400805c4
E (53) psram: No PSRAM detected
E (53) spiram: SPI RAM supported, but not found.
[toit] INFO: starting <v2.0.0-alpha.58>
[jaguar] INFO: container 'sleep2' started
list  []
Entering deep sleep for 10000ms
ets Jul 29 2019 12:21:46

rst:0x5 (DEEPSLEEP_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12700
ho 0 tail 12 room 4
load:0x40080400,len:2916
entry 0x400805c4
E (53) psram: No PSRAM detected
E (53) spiram: SPI RAM supported, but not found.
[toit] INFO: starting <v2.0.0-alpha.58>
[jaguar] INFO: container 'sleep2' started
list  []
Entering deep sleep for 10000ms
k
👍
d
just swapping --flash yields:
Copy code
david@MSI-7D43:~$ jag monitor -a
Starting serial monitor of port '/dev/ttyUSB0' ...
[jaguar] INFO: container 'sleep2' installed and started
list  []
Entering deep sleep for 10000ms
ets Jul 29 2019 12:21:46

rst:0x5 (DEEPSLEEP_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12700
ho 0 tail 12 room 4
load:0x40080400,len:2916
entry 0x400805c4
E (53) psram: No PSRAM detected
E (53) spiram: SPI RAM supported, but not found.
[toit] INFO: starting <v2.0.0-alpha.58>
[jaguar] INFO: container 'sleep2' started
list  [86]
Entering deep sleep for 10000ms
ets Jul 29 2019 12:21:46

rst:0x5 (DEEPSLEEP_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12700
ho 0 tail 12 room 4
load:0x40080400,len:2916
entry 0x400805c4
E (53) psram: No PSRAM detected
E (53) spiram: SPI RAM supported, but not found.
[toit] INFO: starting <v2.0.0-alpha.58>
[jaguar] INFO: container 'sleep2' started
list  [86, 97]
Entering deep sleep for 10000ms
ets Jul 29 2019 12:21:46

rst:0x5 (DEEPSLEEP_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12700
ho 0 tail 12 room 4
load:0x40080400,len:2916
entry 0x400805c4
E (53) psram: No PSRAM detected
E (53) spiram: SPI RAM supported, but not found.
[toit] INFO: starting <v2.0.0-alpha.58>
[jaguar] INFO: container 'sleep2' started
list  [86, 97, 107]
Entering deep sleep for 10000ms
k
Ugh. That was entirely my fault. We ended up using the non-RTC backed implementation also on the ESP32 due to a mistyped C++ define: https://github.com/toitlang/toit/pull/1462.
I found this version a bit easier to test with Jaguar, because it sometimes stops the deep sleep and lets Jaguar back in control. Also notice the use of 
bucket.get "mail" --init=: []
which is a nice pattern for buckets and maps.
Copy code
import system.storage
import esp32

bucket := storage.Bucket.open --ram "/admin"

main:
  list := bucket.get "mail" --init=: []
  print "list = $bucket["mail"]"
  add Time.now.s_since_epoch
  if (random 100) < 90: esp32.deep_sleep (Duration --s=1)

add entry/any -> none:
  buffer := List.from bucket["mail"]
  buffer.add entry
  bucket["mail"] = buffer
Will send out a new release within a few hours with the (somewhat embarrasing) fix.
d
thx. saw the 
--init
, the 
--ifAbsent
has been so common you don't have to think about it. will use
pls, there is 3 of you ... the amount of code you folk move is amazing
k
Fix is included in Jaguar v1.9.6 which should be available shortly.
It is out. Fingers really crossed 🤞
d
works for me !! thx for grinding that. and yes 
random
is a clever alternative to a button in many test situations
k
Finally. Thank you for reporting back 🎆
d
there does appear to be an issue the very first time after flashing, only in --ram. Until you write to the slot explicitly, when you read it, you get 
null
. Can't see why the Bucket 
--init
or 
--if_absent
code is not triggered. Take a look at the attached to reproduce, see if it makes sense.
issue
k
Will do!
Isn't it just because 'List.from null' isn't legal?
I guess you're saying that init should kick in.
d
look, it is really hard to describe
yeah, I can't see why the init does not kick in. thx for looking
I have tried probing the difference between flash & ram ... I don't see it
yet, specifically after you first flash, it happens
there is something there, I just can't put my finger on it ... hopefully the instructions are clear enough to reproduce
k
Will try to reproduce and fix! Thanks 🙏
d
(lot easier to stay on track when sitting in a remote debugger, rather than print/run cycles, but Florian has indicated that a remote debugger is low priority. We had one at OTI for embedded smalltalk, I loved it. You could edit/drop the stack/go. Just a comment )
k
FWIW, 
list := bucket.get "mail" --init= (: bucket["mail"] = [])
should be 
list := bucket.get "mail" --init=(: [])
(or just 
list := bucket.get "mail" --init=: []
)
Oh, I think that's the problem.
d
let me retry ... does not explain diff behavior ram & flash
k
The bug is that the []= operator doesn't return the value. It returns null.
(also for flash)
But maybe you pretty much always have something in flash left over.
I'll fix the bug.
So you do initialize the bucket, but the return value is null on the first run.
d
... you mean when I reflashed, maybe there was "enough" leftover in flash, to cause different behavior than ram
ok
k
It is the same bug in both ram and flash (and it is one line fix), so it should fail for the --flash case too if you start with an empty flash.
d
(see my bug, and the bug in []=. was just trying to understand the observed difference, but without a raw memory clear function, not possible to user test. np, likely enough)
k
You can remove a key from your bucket from another program and try again?
d
oh remove ... ok
k
So my conclusion is that for now the "workaround" is to use 
--init=: []
and then the next release will fix it so the original code using 
--if_absent
also works as expected.
d
yeah. sorry for the buggy report, but serendipitously it found the []= bug .. ok.
k
Yeah, it is pretty awesome. Thank you again!
Jaguar v1.9.7 is out with a fix for the 
[]=
issue.
9 Views
PreviousNext
Powered by