This message was deleted.

@Xipeng Guan what is our plans for this?

cc @Sean

@Aleks Smechov Yes, the support for async endpoint is in the works. However, async endpoint implementation requires multiple distributed components, e.g. distributed queue, blob storage, etc. We will first release it to BentoCloud.

Got it, thanks, is there an ETA for that @Sean? Have you checked out KServe's proposition or Cortex's diagram? We'd love to switch over from Cortex but are figuring out ways around the async endpoints at the moment.

@Aleks Smechov Thanks for sharing the designs of KServe and Cortex. We estimate to have async endpoint in BentoCloud before the end of September. Our design very similar to Cortex’s diagram except BentoCloud won’t be specific to AWS. Caller can either check back for status at a later time or be notified by a callback. Would you like us notify you when the feature is available for testing in BentoCloud?

Sure 🙂