im building a custom connector in the CDK. to successfully use the incremental strategy, i need to pass a timestamp into the query params as a unix timestamp. however, the cursor field is a datetime string. i thought i could use jinja to manipulate the data type, but im getting an error in the connector.

time data '1621468800.0' does not match format '%Y-%m-%d %H:%M:%S.%f+00:00'

the CDK seems to break when the date types dont match between the cursor field and the query param.

Hey David! Is ‘1621468800.0’ a string or a double in your case? Unix timestamp as strings should be supported with datetime format

%s

. However, I see there is an issue with our code where everything after the

.

would be dropped. If you need this part, let us know and we’ll prioritize the fix

hiya @Maxime Carbonneau-Leclerc (Airbyte) .. that value is a float. im converting an ISO 8601 string into an integer with the timestamp jinja function:

{{ timestamp(config['start_dt']) }}

. the issue is that the CDK is expecting a different format. the cursor itself is formatted as

%Y-%m-%dT%H:%M:%S.%fZ

. if i leave the format field blank for the start datetime section, per the docs it assumes the format should be the same as the cursor field. but, im unable to set the format as

%0.2f

very likely user error on my part here.. perhaps im getting the formatting for the query param portion wrong?

Ok! So if I understand this properly, the issue is with the start datetime only. What is the format of

config['start_dt']

? Why do you parse it as a Unix timestamp? Given you require a consistent format for

config['start_dt']

, I would put this format in the field

Format

under

Start Datetime

. If you still need to cast this as a int for whatever reason,

%s

should do the trick considering everything more granular than a second will be ignored

still no joy. let me try to re-state the issue. • the cursor field in the response data is formatted as an ISO 8601 string .. specifically

%Y-%m-%dT%H:%M:%S.%fZ

. ive set the format for the user-defined

config['start_dt']

to be the same. • the query parameter for datetime filtering requires unix timestamps in milliseconds formatted as integers. the api endpoint im working with is here if the docs are helpful at all. • so, i need to convert the value passed into the query parameter from the datetime string into an integer, both for the initial sync when using the user-supplied

config['start_dt']

value as well as for subsequent syncs when using the bookmark for the stream. i think im close with the following setup, but im getting an

ERROR: unconverted data remains

message. is this related to your note re: not handling inputs more granular than seconds?

Oh, yes! We don’t support different datetime format between the cursor field values and the query params. This is something we saw before in the API doc of The Guardian but in practice the API actually supported both so we decided not to go forward with this feature. That being said, you mentioning the Rutter API makes this feature relevant. I can’t provide any timeline on this today but I’ll bring this feature request back to my team and see what is our next step

cool! i built a custom version of this connector previously, but only for a few streams. its working okay, but i was trying out the CDK to re-build the connector, add a few new streams, fix some of the schema definitions, etc. .. will re-visit if / when you all tackle this.

Upping this one as well, for a (defo smaller) usecase I need to set my start/ end date to

%Y-%m-%d

while the api response returns

%Y-%m-%dT%H:%M:%S.%f%z

. I was assuming that the formatting of the start/ end datetime query params would be leading but the format of the cursor field determines the test params. This breaks any second sync with the same errors as mentioned here.

This will also be helpful for me for a custom connector build for Tradedoubler. The date parameter in the response is formated as

%Y-%m-%dT%H:%M:%S.%f%z

but the parameter in the request should look like

%Y%m%d

so it would be ver useful to be able to define the format of the dates in the request.

Just wanted to share the GitHub ticket for this one: https://github.com/airbytehq/airbyte/issues/27153 It is one of our priority but I can’t put a specific date right now. We’ll keep you posted