https://linen.dev logo
#ask-community-for-troubleshooting
Title
# ask-community-for-troubleshooting
r

Rodrigo Morais

08/06/2021, 2:44 PM
Hello! I’m new to Airbyte. I’m using the
HTTP Request
source but I’d like to have dynamic query parameters on it. Is that possible? For example, my url has
start_date
and
end_date
. I’m executing it once per day with
start_date
= d - 1 and
end_date
= current day. How can I have that in Airbyte?
g

George Claireaux (Airbyte)

08/06/2021, 2:52 PM
Hey @Rodrigo Morais, the recommended approach for syncing data via an API would be to build a dedicated connector for it. Our CDK offers a great template for doing this and we'll be happy to help support! Which API are you getting data from out of curiosity 🙂 ?
r

Rodrigo Morais

08/06/2021, 3:33 PM
Hello George, Thanks for answering. They are many APIs. My company has a product where our clients can inform the APIs and we collect data for them. We don’t know which APIs and what will be the format of them. We decide to use Airbyte behind of the scene. For sure the APIs will have query parameters and we need that flexible. We choose Airbyte because the connectors already exist.
u

[DEPRECATED] Marcos Marx

08/06/2021, 8:23 PM
Unfortenaly the HTTP is very simple and doesn't support any additional params in the query 😞
j

Jeff Crooks

08/07/2021, 9:40 PM
I've got an open issue you can vote for to add this! 😃
r

Rodrigo Morais

08/09/2021, 7:04 AM
Unfortunately, I can’t wait for someone to develop that. Our product needs that right now. I’ll develop a connector with query parameters. Should I develop a new one or extend the
HTTP Request
?
u

[DEPRECATED] Marcos Marx

08/09/2021, 5:16 PM
If is something easy and you can control on your side, probably extending the HTTP Request connector can be more straightforward
r

Rodrigo Morais

08/10/2021, 11:01 AM
For us the only dynamic query parameter are dates. We have other query parameters as strings, arrays, and numbers but they are not dynamic. This is the idea that I’ll follow: Query parameters: • Date ◦ Day ▪︎ Current ▪︎ Current day - n ▪︎ Current day + n ◦ Month ▪︎ Current ▪︎ Current month - n ▪︎ Current month + n ◦ Year ▪︎ Current ▪︎ Current year - n ▪︎ Current year + n • Period ◦ Start date ▪︎ Day • Current • Current day - n • Current day + n ▪︎ Month • Current • Current month - n • Current month + n ▪︎ Year • Current • Current year - n • Current year + n ◦ End date ▪︎ Start date + Day, Month, Year Type: [Date, Period] Format: [Day, Month, Year] Value: [Current, Current - n, Current + n] Examples:
Copy code
<http://abc.gov/data?date={{date_1}}>
{"type": "date", "format": "day", "variable": date_1, "value": "current - 1"}
<http://abc.gov/data?from={{start_date}}&to={{end_date}}>
{"type": "period", "format": "day", "start_date": {"variable": "start_date", "value": "current - 1"}, "end_date": {"variable": "end_date", "value": "start_date + 1"}}
My idea is to have another parameter as
headers
. Every time that Airbyte executes the stream it has to get the correct dates. Does that is possible?
u

[DEPRECATED] Marcos Marx

08/11/2021, 2:16 AM
I think is possible, you probably can add this parameters in the spec.json