I just set up a pipeline pulling instagram account data into BigQuery a couple of weeks ago, but today all of the instagram story thumbnails disappeared and say “URL signature expired”. Obviously this makes sense, but I wasn’t sure when the thumbnail images expire until now. Does anyone know how to programmatically scrape/copy an image as a file from the thumbnail url while it’s still reachable in order to copy it into cloud storage or google drive?

Hey there, this channel is to discuss #advice-data-transformation, if you’re having issues with Airbyte we recommend posting in Discourse to receive support.

Well I’m talking about transforming the thumbnail url into an actual image file to store on my own server.

@Jordan Scott take a look at this SO answer (https://stackoverflow.com/a/68065104) for a potential alternative to the typical thumbnail URLs

I'll try that. I’m looking for a solution for story thumbnails so I'll check, thanks!