Where can I get help about the API?
# replication-troubleshooting
i
Where can I get help about the API?
I'm struggling to setup a connection using the API
i
@Akilesh V I'm aware of the API documentation I've been using it to create workspaces, sources and destinations for a while now. I'm just struggling with creating a connection and the documentation is lacking in that regard
a
easy step would be create connection from ui and then fetch all required api request from network tab
@Ismail Tigrek if you need more help let me know
i
@Akilesh V the problem is that the connection create process in the ui is somewhat dynamic. Meaning, when you select the source for the connection, the ui automatically fetches the schema of the sources and then you can play around with that before actually creating the connection. Therefore, when the ui actually sends the connection/create request to the API, it has a jsonSchema it populates the payload with. However, if you are trying to build a connection simply with the API from code, you don't have access to that jsonSchema
a
i am using airbyte api to create connection
i
When I create a connection from the UI and observe the request from the network tab, I notice an "operations" parameter in the payload. However, the API documentation doesn't have this parameter so when I try to recreate the request it doesn't work @Akilesh V
a
@Ismail Tigrek operations: [ { "name": "Normalization", "workspaceId": workspaceId, "operatorConfiguration": { "operatorType": "normalization", "normalization": { "option": "basic" } } } ] you can pass this if you want use basic normalization
i
That's what I was already trying to pass because that's what shows up in the network tab, but it's not working
When I remove this, the connection is successfully created but the data isn't normalized
n
What are the connectors in this connection? Are you getting any logs/output from the Airbyte instance?
a
@Ismail Tigrek please share api request payload
i
@Nataly Merezhuk (Airbyte) I'm trying to connect S3 to Postgres
@Akilesh V here is the api request payload that I am recreating )with the operations parameter_ just as I see it in the network tab:
Copy code
{
  "name": "EXAMPLE NAME",
  "prefix": "",
  "sourceId": "e9929431-22df-419a-a2c0-19f4165cb452",
  "destinationId": "9eaa2a2a-811b-4f1e-8297-8e685c406e1d",
  "namespaceDefinition": "source",
  "namespaceFormat": "${SOURCE_NAMESPACE}",
  "syncCatalog": {
    "streams": [
      {
        "stream": {
          "name": "testt",
          "jsonSchema": {
            "type": "object",
            "properties": {
              "name": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "rank": {
                "type": [
                  "null",
                  "number"
                ]
              },
              "type": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "uuid": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "price": {
                "type": [
                  "null",
                  "number"
                ]
              },
              "cb_url": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "permalink": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "price_usd": {
                "type": [
                  "null",
                  "number"
                ]
              },
              "created_at": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "updated_at": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquired_on": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquiree_city": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquiree_name": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquiree_uuid": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquirer_city": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquirer_name": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquirer_uuid": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquiree_cb_url": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquiree_region": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquirer_cb_url": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquirer_region": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquisition_type": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "_ab_source_file_url": {
                "type": "string"
              },
              "acquiree_state_code": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquirer_state_code": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "price_currency_code": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquiree_country_code": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "acquirer_country_code": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "_ab_additional_properties": {
                "type": "object"
              },
              "_ab_source_file_last_modified": {
                "type": "string",
                "format": "date-time"
              }
            }
          },
          "supportedSyncModes": [
            "full_refresh",
            "incremental"
          ],
          "sourceDefinedCursor": true,
          "defaultCursorField": [
            "_ab_source_file_last_modified"
          ],
          "sourceDefinedPrimaryKey": []
        },
        "config": {
          "syncMode": "full_refresh",
          "cursorField": [
            "_ab_source_file_last_modified"
          ],
          "destinationSyncMode": "overwrite",
          "primaryKey": [],
          "aliasName": "testt",
          "selected": true
        }
      }
    ]
  },
  "scheduleType": "basic",
  "scheduleData": {
    "basicSchedule": {
      "timeUnit": "hours",
      "units": 24
    }
  },
  "operations": [
    {
      "name": "Normalization",
      "workspaceId": "25971c1e-9467-4936-adeb-63f691d4379a",
      "operatorConfiguration": {
        "operatorType": "normalization",
        "normalization": {
          "option": "basic"
        }
      }
    }
  ],
  "status": "active"
}
It gives me the following error:
Copy code
Error in resp_abort(resp, error_body(req, resp)) : 
  HTTP 422 Unprocessable Entity.
But if I remove the
operations
parameter, it works. However, then the data is not normalized and only gets loaded as JSON objects
@Nataly Merezhuk (Airbyte) @Akilesh V Basically, I need documentation on how to setup
operations
in
connections
payloads when using the Airbyte API
In summary: When I create an S3 to Postgres connection with Normalized tabular data (basic normalization), Airbyte sends a
connections/create
payload to the API that I observe in the network tab that has an
operations
parameter. When I try to send that exact same payload to the API directly without the UI, it gives me an
HTTP 422
error. When I remove the
operations
parameter and send the payload, it works (but obviously without normalization).
a
@Ismail Tigrek please share error log you are getting while call the api
i
@Akilesh V where can I access the error log?
a
you will find in response if not look inside airbyte server log
i
@Akilesh V this is the error
HTTP 422 Unprocessable Entity.
a
can you share the full log
n
I'm not as familiar with this part of the API, but perhaps you could try creating an operation before applying it? https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/operations/create
i
@Nataly Merezhuk (Airbyte) but as you can see in the example payload above, Airbyte is creating the operation with the connection.
@Akilesh V where can I find the full log?