https://linen.dev logo
#connector-development
Title
# connector-development
o

Oliver Meyer

03/21/2022, 2:30 PM
Hi šŸ‘‹ I’m thinking of extending the BambooHR source connector to support the custom report endpoint. The main complexity I see is that the endpoint takes a POST request with a list of fields, and returns only those fields. Has something like this already been implemented in another connector? One solution would be to add this list of field as a property in
spec.json
, and use that in the request. Maybe there’s a better way 🤷
a

Augustin Lafanechere (Airbyte)

03/21/2022, 3:29 PM
Hi @Oliver Meyer, if the field names are custom, providing these in the
spec.json
is the way to go indeed.
o

Oliver Meyer

03/21/2022, 3:30 PM
šŸ‘ I’ll take a shot at it then, thanks!
a

Augustin Lafanechere (Airbyte)

03/21/2022, 3:31 PM
If you want to make
POST
requests you need to set an
http_method = "POST"
in your stream's class.
s

Sherif Nada

03/21/2022, 4:41 PM
is there a reason you can’t just ask for all fields?
o

Oliver Meyer

03/21/2022, 5:16 PM
is there a reason you can’t just ask for all fields by default?
Not sure I get it. You mean ask for all fields in the request? If that’s what you mean, then by design the endpoint is made to return only specific fields and that’s how we want to use it
is there a reason you can’t just ask for all fields by default?
We could, but specifying fields seems to be more aligned with the way the endpoint works. Also, in our case ā€œall fieldsā€ is 552 fields (bamboohr allows for custom attributes…), so if we can avoid pulling all 552 I’m happy
@Sherif Nada do you see any issues with specifying fields upfront?
s

Sherif Nada

03/21/2022, 6:21 PM
just that it’s arduous for the user to input field by field what they want to get. If there is no API cost, it’s probably better to by default show them all the fields and once https://github.com/airbytehq/airbyte/issues/2227 is implemented, they can just choose which columns they want
o

Oliver Meyer

03/21/2022, 6:29 PM
Perhaps it could be optional - in terms of effort, I think I’d rather input 10 fields manually, than sort through 552 fields downstream
a

Augustin Lafanechere (Airbyte)

03/21/2022, 6:31 PM
@Oliver Meyer, are you mentioning these fields?
o

Oliver Meyer

03/21/2022, 6:32 PM
Those are the default fields in BambooHR, but users can add as many custom fields as they want and they will all appear in the
discover
step
Our users seem to like custom fields a lot and we have 552 fields in total
a

Augustin Lafanechere (Airbyte)

03/21/2022, 6:33 PM
Ok then what do you think of always retrieving all the default fields + allowing users to add the name of their custom fields in the
spec.json
?
o

Oliver Meyer

03/21/2022, 6:40 PM
I’m still not sure that should be the expected behaviour. By design this endpoint is made to pull only specific fields from Bamboo (see the doc). If I give the connector a list of 5 fields I want, and it returns 85 fields instead (there are 80 default fields), I would say that’s more of a bug than a feature
a

Augustin Lafanechere (Airbyte)

03/21/2022, 6:44 PM
It depends on the wording šŸ˜„ Something like internal custom fields with a good description would allow users with this use case to add their fields and others to always have a default custom report filled with data. Do you know if the Bamboo API rates is impacted by the number of requested fields?
o

Oliver Meyer

03/21/2022, 6:53 PM
Can’t say, I’m not familiar with their rate limiting
a

Augustin Lafanechere (Airbyte)

03/21/2022, 6:56 PM
All right, feel free to go with the approach you prefer and we'll keep the fine-tuning session during PR reviews. Thanks for considering contributingšŸ˜„!
o

Oliver Meyer

03/21/2022, 6:56 PM
Sounds good I’ll take a shot and see what happens
Thanks for the input
@Augustin Lafanechere (Airbyte) one last thing - are there any resources around migrating from an old version of the CDK to the latest? The bamboo connector seems to be using a version of the CDK I’m not familiar with, would be easier to extend in the latest version
4 Views