https://openobserve.ai/ logo
Join Slack
Powered by
Sorry if this is spam but our instance developed s...
# openobserve
m
Sorry if this is spam but our instance developed some problem overnight and is broken in a way I cant figure out. I made an issue on the GH but figured Id reach out here as well. 🧵
This morning we got an alert that our O2 instance was restarting continuously. When I got the logs they were as follows:
Copy code
thread '<unnamed>' (72) panicked at src/job/mod.rs:502:31:
stream cache failed: Error parsing schema, key: default/metrics/container_memory_usage_total, error: EOF while parsing a value at line 1 column 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: backend job init failed: channel closed
Im not sure why this started happening, but it has completely downed our instance. Is there anything I can do to get it running again? Losing historical data for this metric would be fine.
I ran with 
RUST_BACKTRACE=full
just for kicks but it wasnt helpful:
Copy code
thread '<unnamed>' (72) panicked at src/job/mod.rs:502:31:
stream cache failed: Error parsing schema, key: default/metrics/container_memory_usage_total, error: EOF while parsing a value at line 1 column 0

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: clone
stack backtrace:
   0:     0x5557b4aa43f9 - <unknown>
   1:     0x5557adb48efa - <unknown>
   2:     0x5557b4aaeea6 - <unknown>
   3:     0x5557b4a7d486 - <unknown>
   4:     0x5557b4a99746 - <unknown>
   5:     0x5557b4a999d2 - <unknown>
   6:     0x5557b4a7d542 - <unknown>
   7:     0x5557b4a71c19 - <unknown>
   8:     0x5557b4a7ec6d - <unknown>
   9:     0x5557adb4965c - <unknown>
  10:     0x5557adb49432 - <unknown>
  11:     0x5557b3771177 - <unknown>
  12:     0x5557b3647e30 - <unknown>
  13:     0x5557b3107477 - <unknown>
  14:     0x5557b3a035fd - <unknown>
  15:     0x5557b4aa2d14 - <unknown>
  16:     0x7fdf7e8f7b7b - <unknown>
  17:     0x7fdf7e975630 - clone
  18:                0x0 - <unknown>
Error: backend job init failed: channel closed

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: __libc_start_main
   9: <unknown>
p
What version are you on?
m
v0.91.1
Also I think I found the issue, but am not sure how it happened or how to fix it. in the 
metadata.sqlite
db there is an entry in the 
meta
table for a schema (
metrics/container_memory_usage_total
) that has an empty value column
p
a bug might have caused it. We will check the github issue. but you can try deleting the empty value row. That should help resolve
🙌 1
It will get auto created
m
Excellent, thats what we were just trying and it worked! Thanks so much for the speedy response
👍 1
3 Views
Open in Slack
PreviousNext
Powered by