https://linen.dev logo
#ask-community-for-troubleshooting
Title
# ask-community-for-troubleshooting
r

Rahul Padhy

05/27/2021, 10:10 AM
Hey everyone, I’ve been trying to make a connection using Airbyte API as listed here -> https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create However, if I mention the field ‘operationIds’ in the input schema, then it errors out saying that ‘operationIds’ is not a valid field-name, so I proceeded ahead without the field ‘operationIds’ and when I posted the data, I got status_code 200 for this. When I went back to the Airbyte UI, the sync operation had started by itself, and it was failing - citing the reason that Unexpected character (‘’' (code 39)): was expecting double-quote to start field name - if there were an error, then how come I got the response code 200? I’m attaching the logs below. Can anyone please suggest as to what I’m missing here?
1
I’ve followed the exact schema as given in the link above for passing the JSON input, please do tell if there’s anything incorrect or changed from there. :)
s

Subodh (Airbyte)

05/27/2021, 10:20 AM
hey Rahul can you share the curl request for the API call with your input. Also which version of Airbyte are you using?
r

Rahul Padhy

05/27/2021, 10:20 AM
Curl request as in? The input JSON to the POST method?
s

Subodh (Airbyte)

05/27/2021, 10:21 AM
yes correct.
r

Rahul Padhy

05/27/2021, 10:21 AM
Ok
c

Chris (deprecated profile)

05/27/2021, 10:23 AM
It would be safer to have
"operationsIds" : []
instead of empty strings
1
r

Rahul Padhy

05/27/2021, 10:23 AM
0.22.2-alpha -> my current airbyte version
@Chris (deprecated profile) It isn’t even taking the field “operationIds” - just errors out saying that its an invalid field.
c

Chris (deprecated profile)

05/27/2021, 10:25 AM
it’s normally optional so you can completely omit it otherwise we’re still working on this part of the API
r

Rahul Padhy

05/27/2021, 10:25 AM
Okay, yes, I’ve omitted it as of now 😅
c

Chris (deprecated profile)

05/27/2021, 10:26 AM
instead of using empty strings in the field, you should leave them empty arrays instead
like
cursorField
,
primaryKey
etc
r

Rahul Padhy

05/27/2021, 10:27 AM
Okay, as of now, just using “operationIds” gives out an error, saying that its an invalid field.
c

Chris (deprecated profile)

05/27/2021, 10:28 AM
that field does not exist for 0.22.2
r

Rahul Padhy

05/27/2021, 10:29 AM
Okay, anyways, I’m omitting that
r

Rahul Padhy

05/27/2021, 10:32 AM
I’ve seen the fields that are to be there for that table and I’ve made sure that I’ve included all the fields with the correct inputs and types for those fields, still I’m stuck as to why in the logs, JSON field-name error is coming up
s

Subodh (Airbyte)

05/27/2021, 10:34 AM
Can you retry excluding the
operationsIds
and using empty array (without empty string) for fields like
cursorField
r

Rahul Padhy

05/27/2021, 10:35 AM
But for 0.22.2 (my version of Airbyte), the field operationIds wasn’t even included, referring to the yaml file link sent by @Chris (deprecated profile) just above.
s

Subodh (Airbyte)

05/27/2021, 10:37 AM
Am a bit confused, here https://airbytehq.slack.com/archives/C021JANJ6TY/p1622110968116000?thread_ts=1622110220.112700&cid=C021JANJ6TY in your request it seems like you did put `operationsIds`as part of the request. Am curious about the result when you make the POST request without
operationsIds
and do not use empty string in array for other fileds
r

Rahul Padhy

05/27/2021, 10:38 AM
Aah sorry, sending you the right one
This is the response that I get after doing the POST request without using operationIds
Status code : 200
{
"connectionId": "8c456147-405c-4079-a292-be1d84a6b77d",
"name": "default",
"prefix": "",
"sourceId": "61952097-5cea-46bd-96f8-90fc4cd2290e",
"destinationId": "c89a4973-2bfd-4273-a72d-58754549cae2",
"syncCatalog": {
"streams": [
{
"stream": {
"name": "api_oppo_hist",
"jsonSchema": {
"$schema": "<http://json-schema.org/draft-07/schema#>",
"type": "object",
"properties": {
"ACV": {
"type": "number"
},
"CS Revenue": {
"type": "number"
},
"CVR SQL to Customer": {
"type": "number"
},
"CVR from MQL to SQL": {
"type": "number"
},
"Churn ARR": {
"type": "number"
},
"Churned Customers": {
"type": "number"
},
"Closed Date": {
"type": "string"
},
"Id": {
"type": "number"
},
"Invoiced Revenue": {
"type": "number"
},
"MQL BDR": {
"type": "number"
},
"MQL Marketing": {
"type": "number"
},
"MQL Marketing - Adwords": {
"type": "number"
},
"MQL Marketing - Digital": {
"type": "number"
},
"MQL Marketing - Events, Roundtable & Webinars": {
"type": "number"
},
"MQL Marketing - X": {
"type": "number"
},
"MQL Partnerships": {
"type": "number"
},
"New ARR": {
"type": "number"
},
"Number of MQL": {
"type": "number"
},
"Number of New Customers": {
"type": "number"
},
"Number of SQL": {
"type": "number"
},
"Old ARR to go Live": {
"type": "number"
},
"Region": {
"type": "string"
},
"Renewal": {
"type": "number"
},
"Sales Driven Expansion Revenue": {
"type": "number"
},
"Total ARR": {
"type": "number"
},
"Total Customer": {
"type": "number"
},
"Usage expansion": {
"type": "number"
},
"__sdc_row": {
"type": "number"
},
"__sdc_sheet_id": {
"type": "number"
},
"__sdc_spreadsheet_id": {
"type": "string"
},
"_sdc_batched_at": {
"type": "string"
},
"_sdc_extracted_at": {
"type": "string"
},
"_sdc_received_at": {
"type": "string"
},
"_sdc_sequence": {
"type": "number"
},
"_sdc_table_version": {
"type": "number"
}
}
},
"supportedSyncModes": [
"full_refresh"
],
"sourceDefinedCursor": false,
"defaultCursorField": [
""
],
"sourceDefinedPrimaryKey": [
[
""
]
],
"namespace": ""
},
"config": {
"syncMode": "full_refresh",
"cursorField": [
""
],
"destinationSyncMode": "overwrite",
"primaryKey": [
[
""
]
],
"aliasName": "api_oppo_hist",
"selected": true
}
}
]
},
"schedule": {
"units": 24,
"timeUnit": "hours"
},
"status": "active"
}
s

Subodh (Airbyte)

05/27/2021, 10:43 AM
did you use empty string inside array (
[""]
) for fields like
primaryKey
in your post request?
r

Rahul Padhy

05/27/2021, 10:45 AM
Yes for primaryKey and cursorField
s

Subodh (Airbyte)

05/27/2021, 10:46 AM
can retry without using empty string? just use empty array (
[]
)
r

Rahul Padhy

05/27/2021, 10:46 AM
Okay, sure, give me a sec, empty array for both primaryKey and cursorField?
sourceDefinedPrimaryKey
defaultCursorKey
For all these 4 fields, do I stop using empty string inside array?
s

Subodh (Airbyte)

05/27/2021, 10:49 AM
yes correct. where ever you are using empty string inside array, just use an empty array instead
r

Rahul Padhy

05/27/2021, 10:50 AM
Okay, just 1 more doubt here. Should I keep the empty arrays as 2-D arrays ( [ [ ] ] ) or make them as 1-D empty array ( [ ] ) ?
c

Chris (deprecated profile)

05/27/2021, 10:51 AM
empty array of
[]
is probably preferable
r

Rahul Padhy

05/27/2021, 10:51 AM
Ok
s

Subodh (Airbyte)

05/27/2021, 10:53 AM
it depends on the field, for
primaryKey
you use 2-D arrays. for
cursorField
you use 1-D. Refer to the example in the doc https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create
r

Rahul Padhy

05/27/2021, 10:54 AM
Status code : 200
{
“connectionId”: “dabb363f-1d6f-4ee7-90d2-f97d1d15762f”,
“name”: “default”,
“prefix”: “”,
“sourceId”: “61952097-5cea-46bd-96f8-90fc4cd2290e”,
“destinationId”: “c89a4973-2bfd-4273-a72d-58754549cae2",
“syncCatalog”: {
“streams”: [
{
“stream”: {
“name”: “api_oppo_hist”,
“jsonSchema”: {
“$schema”: “<http://json-schema.org/draft-07/schema#>”,
“type”: “object”,
“properties”: {
“ACV”: {
“type”: “number”
},
“CS Revenue”: {
“type”: “number”
},
“CVR SQL to Customer”: {
“type”: “number”
},
“CVR from MQL to SQL”: {
“type”: “number”
},
“Churn ARR”: {
“type”: “number”
},
“Churned Customers”: {
“type”: “number”
},
“Closed Date”: {
“type”: “string”
},
“Id”: {
“type”: “number”
},
“Invoiced Revenue”: {
“type”: “number”
},
“MQL BDR”: {
“type”: “number”
},
“MQL Marketing”: {
“type”: “number”
},
“MQL Marketing - Adwords”: {
“type”: “number”
},
“MQL Marketing - Digital”: {
“type”: “number”
},
“MQL Marketing - Events, Roundtable & Webinars”: {
“type”: “number”
},
“MQL Marketing - X”: {
“type”: “number”
},
“MQL Partnerships”: {
“type”: “number”
},
“New ARR”: {
“type”: “number”
},
“Number of MQL”: {
“type”: “number”
},
“Number of New Customers”: {
“type”: “number”
},
“Number of SQL”: {
“type”: “number”
},
“Old ARR to go Live”: {
“type”: “number”
},
“Region”: {
“type”: “string”
},
“Renewal”: {
“type”: “number”
},
“Sales Driven Expansion Revenue”: {
“type”: “number”
},
“Total ARR”: {
“type”: “number”
},
“Total Customer”: {
“type”: “number”
},
“Usage expansion”: {
“type”: “number”
},
“__sdc_row”: {
“type”: “number”
},
“__sdc_sheet_id”: {
“type”: “number”
},
“__sdc_spreadsheet_id”: {
“type”: “string”
},
“_sdc_batched_at”: {
“type”: “string”
},
“_sdc_extracted_at”: {
“type”: “string”
},
“_sdc_received_at”: {
“type”: “string”
},
“_sdc_sequence”: {
“type”: “number”
},
“_sdc_table_version”: {
“type”: “number”
}
}
},
“supportedSyncModes”: [
“full_refresh”
],
“sourceDefinedCursor”: false,
“defaultCursorField”: [],
“sourceDefinedPrimaryKey”: [
[]
],
“namespace”: “”
},
“config”: {
“syncMode”: “full_refresh”,
“cursorField”: [],
“destinationSyncMode”: “overwrite”,
“primaryKey”: [
[]
],
“aliasName”: “api_oppo_hist”,
“selected”: true
}
}
]
},
“schedule”: {
“units”: 24,
“timeUnit”: “hours”
},
“status”: “active”
}
Does the aliasName field need to contain some specific value?
c

Chris (deprecated profile)

05/27/2021, 11:29 AM
it’s not used
r

Rahul Padhy

05/27/2021, 11:30 AM
Oh okay, but still the same error exists, I mean even after using some of the recommendations suggested by @Subodh (Airbyte) above
s

Subodh (Airbyte)

05/27/2021, 3:36 PM
Hey @Rahul Padhy I think we would need to debug this together. Am not sure why this might be happening just by looking at the logs.
r

Rahul Padhy

05/27/2021, 3:36 PM
Yeah sure, I’m there now
s

Subodh (Airbyte)

05/27/2021, 3:38 PM
I am in another call meeting right now, and it will be on for another 1 and half hour
r

Rahul Padhy

05/27/2021, 3:38 PM
Okay, no problem.
message has been deleted
s

Subodh (Airbyte)

05/27/2021, 3:41 PM
We can debug this tomorrow if you want. Regarding the stream name, its the table name that we are syncing.
r

Rahul Padhy

05/27/2021, 3:42 PM
Table name in the destination, right? Its fine, we can debug tomorrw 🙂
👍 1
Hi Subodh, I’m there, if and when you get time, we can debug! 🙂
s

Subodh (Airbyte)

05/28/2021, 3:58 PM
Hey @Rahul Padhy apologies I could not get back to you today. I am happy to schedule a call on Monday to help you out with this.
r

Rahul Padhy

05/28/2021, 3:58 PM
Ok
s

Subodh (Airbyte)

05/31/2021, 6:40 AM
I and Rahul had a call and the issue was mainly with the way we were making API requests. We fixed the API request and everything worked fine.
r

Rahul Padhy

05/31/2021, 6:41 AM
Yeah thanks a lot @Subodh (Airbyte)
👍 1
s

Sunil Kumar B

06/24/2021, 1:42 PM
Hi Rahul Can you post the json which worked for you
t

takan

06/28/2021, 9:39 AM
Hi @Rahul Prasad or @Subodh (Airbyte) could you post the working json in case other guys having the similar issue?
s

Subodh (Airbyte)

06/30/2021, 6:17 PM
@takan @Sunil Kumar B hey guys sorry for the late response, the JSON is specific to each connector based on your configuration. https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create this document might be of help
2 Views