https://linen.dev logo
Join Slack
Powered by
Hi all, i am trying to use the connector builder ...
# help-connector-development
j
Hi all, i am trying to use the connector builder to build a connector for https://docs.custify.com/#tag/Company/paths/~1company~1all/get when i define a query parameter like this: sorting={"field": "updated_at", "direction": "desc"} This does not get encoded and sent as expected and when i urlencode myself i get an error. Is this related to this issue? https://github.com/airbytehq/airbyte/issues/18784
j
The issue you linked is only about parameters in the body of the request
What error are you getting when url-encoding yourself?
This does not get encoded and sent as expected
What are you seeing exactly?
j
Copy code
{
  "url": "<https://api.custify.com/company/all>",
  "parameters": {
    "sorting": [
      "field",
      "direction"
    ]
  },
  "headers": {
    "User-Agent": "python-requests/2.31.0",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Authorization": "Bearer ****"
  },
  "http_method": "GET"
}
this is how the query parameter is shown in the request tab
When i urlencode {"field": "created_at", "direction": "asc"} and put that into the value field for the query parameter, that gets urlencoded again. Resulting in this error: ERROR
Copy code
Request to <https://api.custify.com/company/all?sorting=%257B%2522field%2522%253A%2520%2522created_at%2522%252C%2520%2522direction%2522%253A%2520%2522asc%2522%257D> failed with status code 404 and error message Page not found
j
Just because it's shown like this doesn't mean it won't work - the connector builder is putting this debug info together before actually sending out the request. Did you test whether that first variant works?
j
yes, the response data is not sorted according to the value given, thats why i started investigating. I just accidentally added a space before {"field": "created_at", "direction": "asc"} and now its properly encoded!
j
Ah interesting, I think I know what this is about, I will investigate and report back. Does this workaround work for you?
j
Copy code
{
  "url": "<https://api.custify.com/company/all>",
  "parameters": {
    "sorting": [
      " {\"field\": \"created_at\", \"direction\": \"asc\"}"
    ]
  },
  "headers": {
    "User-Agent": "python-requests/2.31.0",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Authorization": "Bearer ****"
  },
  "http_method": "GET"
}
yes this causes the response to be sorted according to the value given
due to the limited date filter on that api i dont see a way to use incremental sync with the ui builder. It would have to be something like this: filters=[{"fieldName": "last_updated_at", "fieldType": "date", "filterType": "before", "filterValue": slice.enddate+1}, {"fieldName": "last_updated_at", "fieldType": "date", "filterType": "after", "filterValue": slice.startdate-1}] Which is ... i dont know who designed this API but this usecase (incremental sync) was not part of the design spec i think ... Is there a workaround using direct yaml definitions with https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#/variables/stream_interval somehow?
j
yes, you are able to use the stream_interval variable for these cases. Just don't inject start and end date from the "incremental sync" section, but use interpolation and 
stream_interval
from the generic request params section
Also see the info box at the bottom of the example section here: https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#example
j
Ah that makes sense, thank you.
Since the filter only supports date input (no time), does the destination have to support deduplicated_history transformations to deal with duplicates in the response for the current day when I run this every hour?
j
Couldn't you simply run it every day in this case?
Otherwise yes, there is currently no way to dynamically filter records this way before they are sent to the destination
j
Can i use https://docs.airbyte.com/connector-development/config-based/tutorial/incremental-reads#supporting-incremental-syncs somehow with the connector builder ui/yaml or do i have to restart using the low-code / config based method?
j
you can do this in the builder ui, just enable the "incremental sync" section
j
alright, i was wondering how that deals with running it multiple times a day but i will just try and see 🙂
specifically, how "destination_sync_mode": "overwrite" relates to the ui builder
but i can see that using lockback window and then deduplicating in the destination post sync would also work
j
the destination sync mode and so is taken care of behind the scenes
4 Views
Open in Slack
PreviousNext
Powered by