Is it possible to hide the Crawlee request queue to Apify users?

message has been deleted

Is there a use case for it? I guess if you don't wanna show the url you can perform the request from inside the request handler but you will lose all the commodities that the request queue entails

Yes, you described my use case. My URL contains sensitive information but I want to retain the request queue features. Do you know if request headers are visible to users?

Yeah, it's possible to see them. If you need to produce them in runtime you can do it in the request handler and then perform the request with

sendRequest

(you will have to set

skipNavigation: true

when creating the request).

Would it be possible to send you a DM, given the sensitive nature?

Hey @Louis Deconinck, I can try to help you. Right now it acutally is possible to restrict access to some specific storages. So what you could do is to restrict access to the request queue (e.g. access it under your own token hardcoded in the actor). There is an issue with that. To see the run, you need to be able to access all 3 default storages, so you will have to create additional request queue if you would like to hide it (and keep the default empty, but shared with the user). Other possibilities: keep your headers in runtime memory only and add them to the request in the preNavigationHook (similar to what @luigi.ruocco described above). I hope it help you with your case, let me know if you need some more help.

I can confirm that the headers are visible in the request queue. So we should not use secrets when working with the default request queue. It was a bit inconsistent though. Sometimes the headers showed up and sometimes they didn't, even though I always used custom headers for the requests.