This message was deleted.

Hi @Terry Kong - for your “automatic speech recognition” use case, are you thinking about an Client-Server architecture that communicate via HTTP streaming?

hi @Chaoyu. i’m not familiar with all the ways to stream but what little I do know is if grpc is an option, that also supports bidirectional streaming (over http2 i think). i was thinking of a transaction with the bento server where the session is long lived and i want to return partial transcripts over time. i could wait until the entire input is ready, but by that time the input might be quite long so if my inference is exactly real-time, i need to wait

2*real_time

to report the result since i wait

real_time

to record the audio and

real_time

to perform the inference

“anything involving unidirectional RNNs or stateful architectures (where only part of the input is available at any given time)” - that is supported for sure

Cool, perhaps I missed this in the documentation. If you have pointers to something in the docs, please let me know. FWIW, i know if i accept RNN states as inputs in, say the

JsonInput

, and emit the RNN states in the

JsonOutput

, i can effectively have a stateful transaction with a bento service, but I was curious if there was something like a cache in the bento service that we could store the states in so that the RNN states don’t need to be serialized/deserialized all the time. Perhaps this isn’t the best design choice since if we had multiple bentos behind a load balancer, there’s technically only one bento service with the correct cache that can respond to this particular query. But if you guys have already thought about this use case I’d love to hear what you guys suggest here

Your model states or BentoService’s in-memory attributes does not get serialized or deserialized while the API server is up and running, it only happen during

save

or

load

Your model states or BentoService’s in-memory attributes does not get serialized or deserialized while the API server is up and running, it only happen during 

save

  or 

load

Perhaps this was already clear, but the what I meant by serializing and deserializing was something like this, which I know is already possible under the bento API (allow me to write some pseudocode):

class MyBento(...):
   @api(...)
   def do_inference(inputs: JsonInput):
       inputs_json = inputs["inputs"]
       inputs_tensor = tf.convert_to_tensor(inputs_json)
       lstm_states = inputs["lstm_states"]
       lstm_states_tensor = tf.convert_to_tensor(lstm_states)  # "deserializing" from json -> tf.Tensor
       outputs, lstm_cell_and_hidden_states = self.artifacts.model(inputs=inputs_tensor, states=lstm_states_tensor)
       return {
           "outputs": serialize_tensor_to_json(outputs),
           "lstm_cell_and_hidden_states": serialize_tensor_to_json(lstm_cell_and_hidden_states),  # extra "serialize" from tf.Tensor -> json
       }

i was just wondering if bentoml had a way to avoid us having to send and receive those states because they can be quite large (tens of thousands of elements for large models)