Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hi, i am developing custom connector. Let’s say I have surveys entities and stream for it.
For each entity I need:
• call POST request to start export surveys responses
• call POST request to download the csv/json file contains responses
• import that responses from file
how is it better to do? thanks!

<@U0239L8MA76> I would recommend extending the Stream class from the CDK (instead of httpstream) and overriding the read_records method to perform what you need

I have a survey item with its own id-related json schema returned by API (fields are dynamic for each survey).
I am trying to use `get_json_schema` method, but seems it isn’t called.
• For dynamic schema I shouldn’t create a `stream_name.json` file?
• what to define in properties in `configured_catalog.json`?
• are there an examples of dynamic schema? thanks!

<@U0239L8MA76> which connector are you building out of curiosity?

<@U01A6TEQXRU> connector to Qualtrics and destination to BigQuery, here is an endpoint to get survey item <https://api.qualtrics.com/guides/reference/singleResponses.json/paths/~1surveys~1%7BsurveyId%7D~1response-schema/get|JSON schema>. So surveys are the same item, but with different structure for each of them, behave like a different entities

Is it the right way to override `get_json_schema` and return right json schema based on survey_id?

Also, as a workaround it would be ok to have one struct field in BigQuery and put all survey data here. Are there any recommendations how to do this way?

Is it possible to specify field to be `record` in BigQuery? I am configuring like this
```    "fields": {
      "properties": {},
      "type": ["null", "object"],
      "additionalProperties": true
    }```
but in table it is still `string`

It is possible to query by string field using `JSON_EXTRACT_SCALAR(fields, "$.ResponseId")` but not sure it is efficient

&gt;  Is it the right way to override `get_json_schema` and return right json schema based on survey_id?
Yes that is the correct way — is that something you can do on your end?

&gt;  would be ok to have one struct field in BigQuery
Airbyte’s normalization currently doesn’t support writing STRUCTs to Bigquery natively. We have an issue open for this and expect to support it in 3-4 weeks: <https://github.com/airbytehq/airbyte/issues/1927>

For the time being the workaround is to use JSON_EXTRACT_SCALAR (which is not great, but we’re working on it :slightly_smiling_face: )