http://coldfusion.com logo
Join Slack
Powered by
Anyone have any luck/ideas for handling an http re...
# cfml-general
m
Anyone have any luck/ideas for handling an http response that is returned as a stream? I'm making an API request to Bedrock on AWS and the response is a stream that CF is just barfing on. The content-type coming back is 
application/vnd.amazon.eventstream
and the transfer-encoding is chunked.
The filecontent I'm getting is a native bytearray, but when I run it through toString, the result is garbled - the actual response is there, but with a lot of garbled characters.
Here's what I mean:
Obviously I'm missing something with the proper way to parse the response - possibly because it's streaming and I'm not handling that correctly
d
I've been able to successfully streams from ChatGPT, but it does take some work. I wish I could share the code, but alas I cannot. However, to get this working I had to do a number of things: • I had to build a solution around java.net.http.HttpClient & java.net.http.HttpRequest which I wrapped in a CFC. • This will give you access to raw streams, so you can process the stream as it's populated. • I then wrote a StreamProcesser to act as a handler for the stream. This is passed in to my CFC which is process the raw stream and reads in the stream and watches for commands. • Once it gets a full line, it then streams the required results out to my CFML template (essentially making my page a proxy). I did it that way because I needed to capture information from my stream and do some logging/parse before sending to the client. If I was just basically echoing all the content, I would have looked at just use Nginx as a proxy for the HTTP request, so that I could hide the raw HTTP request details, but did not need to build all the custom code. So it's definitely do-able, but it's not simple.
🙌 1
m
got it - that's super helpful to know - definitely in a similar boat - I guess it'll be a project for next week
any other tips you can share? If not, no worries - but I might ping you before I bang my head against the wall too hard 🤪
d
Not sure what Bedrock uses, but ChatGPT uses a bastardized version of SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events So you will want to read up on that. Basically what you need to do is have your code monitor the incoming string for the events. In my code, I have it respond as soon as an event has been received.
👍 1
m
got it - I think it might be a similar situation then - thank you again!
b
According to ChatGPT, 
application/vnd.amazon.eventstream
is a binary protocol, so converting it to text won't be useful
You'll need to parse it byte-by-byte following their format
Pass the bytes into a byte array input stream and then read them as you parse
Like Dan said, it's meant to be more of a messaging protocol which is sent in chunks that you read as they come in. The very nature of CFHTTP doesn't allow for this as it reads the incoming bytes until the stream is closed and then gives it all back to you at once
in order to keep the connection open and process the events as they come in, you'd need to use a Java-based client.
I have a simple one in CommandBox, where I made my own downloader that's interruptible and updates the status as it downloads, but the newer HTTP client stuff build into the JDK is probably the best way to go these days.
m
Thanks so much @bdw429s - sounds like I have my work cut out for me, but it could be fun. Appreciate the insight!
s
@mjclemente have a look into using an AWS lambda to process it
you can use python etc with that so it might end up easier
m
that's going to be my fallback
s
you might also look into dipping into the jvm to handle it
I also found this with some google-fu that might help or if not might give some inspiration: https://gist.github.com/65/2012978
m
awesome - thanks man! REally appreciate it - I'll take a look!
s
I don’t envy you!
I’ve never seen that mimetype before either so thanks for posting the problem
I also found this in case you go the lambda route - a node.js implementation for processing that mimetype https://github.com/lifion/lifion-aws-event-stream
m
bookmarking that approach right now!
thank you!
s
Keep us updated please if/when you solve it! I’m very interested in the result
m
🤞🙏
Just as an update, for anyone interested. I ended up going the Lambda approach. It was very quick/easy to use the AWS Python library to handle passing the request to/from Bedrock. I dug a bit into translating that node library for parsing the eventstream, but it just wasn't worth the headache/time investment.
👍 1
b
One final thought on this: think about whether or not streaming responses are really necessary for your use case. If the LLM API can return output in a single response, that may be just fine for your use case. As Matt experienced, dealing with streaming responses can be a major hurdle. If it's not a requirement (eg; you're not trying to build a ChatGPT-like interface), it's likely not worth the engineering effort.
Open in Slack
PreviousNext
Powered by