We're seeing timeouts in functions writing to the state store on our 3.1.0 instances:

[2023-11-13 13:58:20 +0000] [ERROR] log.py: bookkeeper.common.exceptions.RetryError: Deadline of 60.0s exceeded while calling functools.partial(<function Table.__do_get__ at 0x7fca668da700>, b'last_item', stream_id: 1028
[2023-11-13 13:58:20 +0000] [ERROR] log.py: r_key: "last_item"
[2023-11-13 13:58:20 +0000] [ERROR] log.py: , [('bk-rt-sid-bin', b'\x00\x00\x00\x00\x00\x00\x04\x04'), ('bk-rt-key-bin', b'last_item')]), last exception: grpc_status_code = StatusCode.NOT_FOUND, bk_status_code = None

Bookkeeper seems to have issues writing as well, see attached log. Any ideas? Couldn't find any prior issues regarding

org.apache.distributedlog.exceptions.UnexpectedException: Didn't find matched inprogress log segments when completing inprogress

...