Hey everyone I ve been trying to make a connection using Air Airbyte #ask-community-for-troubleshooting

Hey everyone, I’ve been trying to make a connectio...

Rahul Padhy

05/27/2021, 10:10 AM

Hey everyone, I’ve been trying to make a connection using Airbyte API as listed here -> https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create However, if I mention the field ‘operationIds’ in the input schema, then it errors out saying that ‘operationIds’ is not a valid field-name, so I proceeded ahead without the field ‘operationIds’ and when I posted the data, I got status_code 200 for this. When I went back to the Airbyte UI, the sync operation had started by itself, and it was failing - citing the reason that Unexpected character (‘’' (code 39)): was expecting double-quote to start field name - if there were an error, then how come I got the response code 200? I’m attaching the logs below. Can anyone please suggest as to what I’m missing here?

logs-160-2.txt

✅ 1

Rahul Padhy

05/27/2021, 10:12 AM

I’ve followed the exact schema as given in the link above for passing the JSON input, please do tell if there’s anything incorrect or changed from there. :)

Subodh (Airbyte)

05/27/2021, 10:20 AM

hey Rahul can you share the curl request for the API call with your input. Also which version of Airbyte are you using?

Rahul Padhy

05/27/2021, 10:20 AM

Curl request as in? The input JSON to the POST method?

Subodh (Airbyte)

05/27/2021, 10:21 AM

yes correct.

Rahul Padhy

05/27/2021, 10:21 AM

Rahul Padhy

05/27/2021, 10:22 AM

json_input.json

Chris (deprecated profile)

05/27/2021, 10:23 AM

It would be safer to have

"operationsIds" : []

instead of empty strings

➕ 1

Rahul Padhy

05/27/2021, 10:23 AM

• 0.22.2-alpha -> my current airbyte version

Rahul Padhy

05/27/2021, 10:24 AM

@Chris (deprecated profile) It isn’t even taking the field “operationIds” - just errors out saying that its an invalid field.

Chris (deprecated profile)

05/27/2021, 10:25 AM

it’s normally optional so you can completely omit it otherwise we’re still working on this part of the API

Rahul Padhy

05/27/2021, 10:25 AM

Okay, yes, I’ve omitted it as of now 😅

Chris (deprecated profile)

05/27/2021, 10:26 AM

instead of using empty strings in the field, you should leave them empty arrays instead

Chris (deprecated profile)

05/27/2021, 10:27 AM

cursorField

primaryKey

etc

Rahul Padhy

05/27/2021, 10:27 AM

Okay, as of now, just using “operationIds” gives out an error, saying that its an invalid field.

Chris (deprecated profile)

05/27/2021, 10:28 AM

that field does not exist for 0.22.2

Rahul Padhy

05/27/2021, 10:29 AM

Okay, anyways, I’m omitting that

Chris (deprecated profile)

05/27/2021, 10:30 AM

you can check here: https://github.com/airbytehq/airbyte/blob/v0.22.2-alpha/airbyte-api/src/main/openapi/config.yaml whereas in 0.23.0, it’s there: https://github.com/airbytehq/airbyte/blob/v0.23.0-alpha/airbyte-api/src/main/openapi/config.yaml

Rahul Padhy

05/27/2021, 10:32 AM

I’ve seen the fields that are to be there for that table and I’ve made sure that I’ve included all the fields with the correct inputs and types for those fields, still I’m stuck as to why in the logs, JSON field-name error is coming up

Subodh (Airbyte)

05/27/2021, 10:34 AM

Can you retry excluding the

operationsIds

and using empty array (without empty string) for fields like

cursorField

Rahul Padhy

05/27/2021, 10:35 AM

But for 0.22.2 (my version of Airbyte), the field operationIds wasn’t even included, referring to the yaml file link sent by @Chris (deprecated profile) just above.

Subodh (Airbyte)

05/27/2021, 10:37 AM

Am a bit confused, here https://airbytehq.slack.com/archives/C021JANJ6TY/p1622110968116000?thread_ts=1622110220.112700&cid=C021JANJ6TY in your request it seems like you did put `operationsIds`as part of the request. Am curious about the result when you make the POST request without

operationsIds

and do not use empty string in array for other fileds

Rahul Padhy

05/27/2021, 10:38 AM

Aah sorry, sending you the right one

Rahul Padhy

05/27/2021, 10:41 AM

This is the response that I get after doing the POST request without using operationIds

Rahul Padhy

05/27/2021, 10:42 AM

Status code : 200
{
"connectionId": "8c456147-405c-4079-a292-be1d84a6b77d",
"name": "default",
"prefix": "",
"sourceId": "61952097-5cea-46bd-96f8-90fc4cd2290e",
"destinationId": "c89a4973-2bfd-4273-a72d-58754549cae2",
"syncCatalog": {
"streams": [
{
"stream": {
"name": "api_oppo_hist",
"jsonSchema": {
"$schema": "<http://json-schema.org/draft-07/schema#>",
"type": "object",
"properties": {
"ACV": {
"type": "number"
},
"CS Revenue": {
"type": "number"
},
"CVR SQL to Customer": {
"type": "number"
},
"CVR from MQL to SQL": {
"type": "number"
},
"Churn ARR": {
"type": "number"
},
"Churned Customers": {
"type": "number"
},
"Closed Date": {
"type": "string"
},
"Id": {
"type": "number"
},
"Invoiced Revenue": {
"type": "number"
},
"MQL BDR": {
"type": "number"
},
"MQL Marketing": {
"type": "number"
},
"MQL Marketing - Adwords": {
"type": "number"
},
"MQL Marketing - Digital": {
"type": "number"
},
"MQL Marketing - Events, Roundtable & Webinars": {
"type": "number"
},
"MQL Marketing - X": {
"type": "number"
},
"MQL Partnerships": {
"type": "number"
},
"New ARR": {
"type": "number"
},
"Number of MQL": {
"type": "number"
},
"Number of New Customers": {
"type": "number"
},
"Number of SQL": {
"type": "number"
},
"Old ARR to go Live": {
"type": "number"
},
"Region": {
"type": "string"
},
"Renewal": {
"type": "number"
},
"Sales Driven Expansion Revenue": {
"type": "number"
},
"Total ARR": {
"type": "number"
},
"Total Customer": {
"type": "number"
},
"Usage expansion": {
"type": "number"
},
"__sdc_row": {
"type": "number"
},
"__sdc_sheet_id": {
"type": "number"
},
"__sdc_spreadsheet_id": {
"type": "string"
},
"_sdc_batched_at": {
"type": "string"
},
"_sdc_extracted_at": {
"type": "string"
},
"_sdc_received_at": {
"type": "string"
},
"_sdc_sequence": {
"type": "number"
},
"_sdc_table_version": {
"type": "number"
}
}
},
"supportedSyncModes": [
"full_refresh"
],
"sourceDefinedCursor": false,
"defaultCursorField": [
""
],
"sourceDefinedPrimaryKey": [
[
""
]
],
"namespace": ""
},
"config": {
"syncMode": "full_refresh",
"cursorField": [
""
],
"destinationSyncMode": "overwrite",
"primaryKey": [
[
""
]
],
"aliasName": "api_oppo_hist",
"selected": true
}
}
]
},
"schedule": {
"units": 24,
"timeUnit": "hours"
},
"status": "active"
}

Rahul Padhy

05/27/2021, 10:42 AM

logs-161-0.txt

Subodh (Airbyte)

05/27/2021, 10:43 AM

did you use empty string inside array (

[""]

) for fields like primaryKey
in your post request?

Rahul Padhy

05/27/2021, 10:45 AM

Yes for primaryKey and cursorField

Subodh (Airbyte)

05/27/2021, 10:46 AM

can retry without using empty string? just use empty array (

[]

)

Rahul Padhy

05/27/2021, 10:46 AM

Okay, sure, give me a sec, empty array for both primaryKey and cursorField?

Rahul Padhy

05/27/2021, 10:47 AM

sourceDefinedPrimaryKey

Rahul Padhy

05/27/2021, 10:47 AM

defaultCursorKey

Rahul Padhy

05/27/2021, 10:47 AM

For all these 4 fields, do I stop using empty string inside array?

Subodh (Airbyte)

05/27/2021, 10:49 AM

yes correct. where ever you are using empty string inside array, just use an empty array instead

Rahul Padhy

05/27/2021, 10:50 AM

Okay, just 1 more doubt here. Should I keep the empty arrays as 2-D arrays ( [ [ ] ] ) or make them as 1-D empty array ( [ ] ) ?

Chris (deprecated profile)

05/27/2021, 10:51 AM

empty array of

[]

is probably preferable

Rahul Padhy

05/27/2021, 10:51 AM

Subodh (Airbyte)

05/27/2021, 10:53 AM

it depends on the field, for

primaryKey

you use 2-D arrays. for

cursorField

you use 1-D. Refer to the example in the doc https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create

Rahul Padhy

05/27/2021, 10:54 AM

Status code : 200
{
“connectionId”: “dabb363f-1d6f-4ee7-90d2-f97d1d15762f”,
“name”: “default”,
“prefix”: “”,
“sourceId”: “61952097-5cea-46bd-96f8-90fc4cd2290e”,
“destinationId”: “c89a4973-2bfd-4273-a72d-58754549cae2",
“syncCatalog”: {
“streams”: [
{
“stream”: {
“name”: “api_oppo_hist”,
“jsonSchema”: {
“$schema”: “<http://json-schema.org/draft-07/schema#>”,
“type”: “object”,
“properties”: {
“ACV”: {
“type”: “number”
},
“CS Revenue”: {
“type”: “number”
},
“CVR SQL to Customer”: {
“type”: “number”
},
“CVR from MQL to SQL”: {
“type”: “number”
},
“Churn ARR”: {
“type”: “number”
},
“Churned Customers”: {
“type”: “number”
},
“Closed Date”: {
“type”: “string”
},
“Id”: {
“type”: “number”
},
“Invoiced Revenue”: {
“type”: “number”
},
“MQL BDR”: {
“type”: “number”
},
“MQL Marketing”: {
“type”: “number”
},
“MQL Marketing - Adwords”: {
“type”: “number”
},
“MQL Marketing - Digital”: {
“type”: “number”
},
“MQL Marketing - Events, Roundtable & Webinars”: {
“type”: “number”
},
“MQL Marketing - X”: {
“type”: “number”
},
“MQL Partnerships”: {
“type”: “number”
},
“New ARR”: {
“type”: “number”
},
“Number of MQL”: {
“type”: “number”
},
“Number of New Customers”: {
“type”: “number”
},
“Number of SQL”: {
“type”: “number”
},
“Old ARR to go Live”: {
“type”: “number”
},
“Region”: {
“type”: “string”
},
“Renewal”: {
“type”: “number”
},
“Sales Driven Expansion Revenue”: {
“type”: “number”
},
“Total ARR”: {
“type”: “number”
},
“Total Customer”: {
“type”: “number”
},
“Usage expansion”: {
“type”: “number”
},
“__sdc_row”: {
“type”: “number”
},
“__sdc_sheet_id”: {
“type”: “number”
},
“__sdc_spreadsheet_id”: {
“type”: “string”
},
“_sdc_batched_at”: {
“type”: “string”
},
“_sdc_extracted_at”: {
“type”: “string”
},
“_sdc_received_at”: {
“type”: “string”
},
“_sdc_sequence”: {
“type”: “number”
},
“_sdc_table_version”: {
“type”: “number”
}
}
},
“supportedSyncModes”: [
“full_refresh”
],
“sourceDefinedCursor”: false,
“defaultCursorField”: [],
“sourceDefinedPrimaryKey”: [
[]
],
“namespace”: “”
},
“config”: {
“syncMode”: “full_refresh”,
“cursorField”: [],
“destinationSyncMode”: “overwrite”,
“primaryKey”: [
[]
],
“aliasName”: “api_oppo_hist”,
“selected”: true
}
}
]
},
“schedule”: {
“units”: 24,
“timeUnit”: “hours”
},
“status”: “active”
}

Rahul Padhy

05/27/2021, 10:54 AM

logs-162-0.txt

Rahul Padhy

05/27/2021, 10:57 AM

Does the aliasName field need to contain some specific value?

Chris (deprecated profile)

05/27/2021, 11:29 AM

it’s not used

Rahul Padhy

05/27/2021, 11:30 AM

Oh okay, but still the same error exists, I mean even after using some of the recommendations suggested by @Subodh (Airbyte) above

Subodh (Airbyte)

05/27/2021, 3:36 PM

Hey @Rahul Padhy I think we would need to debug this together. Am not sure why this might be happening just by looking at the logs.

Rahul Padhy

05/27/2021, 3:36 PM

Yeah sure, I’m there now

Subodh (Airbyte)

05/27/2021, 3:38 PM

I am in another call meeting right now, and it will be on for another 1 and half hour

Rahul Padhy

05/27/2021, 3:38 PM

Do tell me 1 thing though, in here -> https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create, what should be the stream name?

Rahul Padhy

05/27/2021, 3:39 PM

Okay, no problem.

Rahul Padhy

05/27/2021, 3:39 PM

Screenshot 2021-05-27 at 9.09.08 PM.png

Subodh (Airbyte)

05/27/2021, 3:41 PM

We can debug this tomorrow if you want. Regarding the stream name, its the table name that we are syncing.

Rahul Padhy

05/27/2021, 3:42 PM

Table name in the destination, right? Its fine, we can debug tomorrw 🙂

👍 1

Rahul Padhy

05/28/2021, 9:52 AM

Hi Subodh, I’m there, if and when you get time, we can debug! 🙂

Subodh (Airbyte)

05/28/2021, 3:58 PM

Hey @Rahul Padhy apologies I could not get back to you today. I am happy to schedule a call on Monday to help you out with this.

Rahul Padhy

05/28/2021, 3:58 PM

Subodh (Airbyte)

05/31/2021, 6:40 AM

I and Rahul had a call and the issue was mainly with the way we were making API requests. We fixed the API request and everything worked fine.

Rahul Padhy

05/31/2021, 6:41 AM

Yeah thanks a lot @Subodh (Airbyte)

👍 1

Sunil Kumar B

06/24/2021, 1:42 PM

Hi Rahul Can you post the json which worked for you

takan

06/28/2021, 9:39 AM

Hi @Rahul Prasad or @Subodh (Airbyte) could you post the working json in case other guys having the similar issue?

Subodh (Airbyte)

06/30/2021, 6:17 PM

@takan @Sunil Kumar B hey guys sorry for the late response, the JSON is specific to each connector based on your configuration. https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create this document might be of help

2 Views

Open in Slack

Previous Next