https://linen.dev logo
Join Slack
Powered by
Hello Everyone, I have build a connector using the...
# help-connector-development
s
Hello Everyone, I have build a connector using the builder UI to read and extract data from an HTTP API. Where i have different API’s to be done. When i the give the api explicitly like for example 
<https://registre-national-entreprises.fr/api/companies/diff>
i got the records as a json
Copy code
[
  {
    "company": {
      "updatedAt": "2023-07-05T01:28:21+02:00",
      "id": "64a4ab15779d4bbf370ceb23",
      "formality": {
         }
      }
  }
]
Some data like this and its working well. ** I have some other API where i should give the value implicitly and read the id (Because i have lot of id’s) and extract the data. 
<https://registre-national-entreprises.fr/api/companies/{{> config['id'] }}/attachments
this says the request failed. I hope someone might tried this earlier and please give suggesstions and solution. That would be really helpful. Thanks in advance !!
k
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet. However, based on the information provided, it seems like you're trying to dynamically inject an ID into the API endpoint URL. In the Airbyte connector builder UI, you can use variables to inject values into the path of the request. For example, you can change the "Path URL" input to 
/exchangerates_data/{{ stream_interval.start_time }}
to inject the date to fetch data for into the path of the request. This is mentioned in the [tutorial](https://docs.airbyte.com/connector-development/connector-builder-ui/tutorial) on the Airbyte documentation. However, the specific issue you're facing might be due to how you're trying to inject the 'id' into the URL. The correct syntax should be 
/api/companies/{{ config['id'] }}/attachments
. If you're still facing issues, it might be due to how the 'id' is being passed or if it's available in the context. For more detailed help, I would recommend reaching out on the [Airbyte community Slack Channel](https://airbyte.io/community) or posting a question on [Airbyte’s discourse](https://discuss.airbyte.io/).
j
Can you share the manifest for the connector that doesn't work?
s
Do you mean this
Copy code
retriever:
      type: SimpleRetriever
      requester:
        type: HttpRequester
        url_base: <https://registre-national-entreprises.inpi.fr/api/>
        path: '"/companies/{{ config[''siren''] }}/attachments"'
        http_method: GET
        request_parameters: {}
        request_headers: {}
        authenticator:
          type: OAuthAuthenticator
          client_id: '{{ config[''client_id''] }}'
          grant_type: refresh_token
          client_secret: '{{ config[''client_secret''] }}'
          refresh_token: '{{ config[''client_refresh_token''] }}'
          refresh_request_body: {}
j
Please share the whole manifest
s
Please check this
inpi_testing_api.yaml
j
You did not create a user input for your config variable
Screenshot 2023-07-05 at 13.53.29.png
Go to "User inputs" and create a new user input under the name you are referencing in the path (
siren
in this case)
oh, wait, is 
siren
something from the records of the other stream?
s
Yes !!
j
Got it - in this case what you need is not a config variable, but substream partitioning: https://docs.airbyte.com/connector-development/connector-builder-ui/partitioning#substream-partition-router
s
I will try with this and update you !! Thank you
Still getting this error
How can give the nested json value ?
j
Try adding a transformation that "lifts" the nested siren value to the top level of the parent stream record
and then use it from there
I'm not 100% sure whether this works (@Maxime Carbonneau-Leclerc (Airbyte) is this also affected by the incremental sync problem?), but it's worth a try
m
As a workaround, what @Joe Reuter (Airbyte) is proposing should work. Feature request here would be to support deeply nested partition fields for substream partition routing. I don’t think we have an issue for that so I can create one
j
Sounds great, thanks @Maxime Carbonneau-Leclerc (Airbyte)!
s
Sure thats great will try once and update if it works !!
Can you suggest me an example case
j
Screenshot 2023-07-05 at 15.07.57.png
Then, in the child stream, just set parent key to 
siren
m
@Sai Kumar Devulapelli if you want to subscribe to the issue: https://github.com/airbytehq/airbyte/issues/27972
s
That’s great it worked but still the records is not recevied
j
Check the response and the record selector
s
please check this once
j
Seems like an API issue - there is no body returned for the path you are sending
s
Okay thats great !! Thanks a lot for your help !!
Will update you if required thanks a lot @Joe Reuter and @Maxime Carbonneau-Leclerc (Airbyte)
Open in Slack
PreviousNext
Powered by