Moving to this thread... Under the Source Settings, the docs warn us about this:

image.png

The trouble is, I don't understand what

data

is being discussed here. Any hints? How / where do I get or set this information?

You can see at the bottom of the image, the only thing it is showing me is the name of the first column in the Azure table: PartitionKey. When uploading a table into Azure from CSV, you are required to have 'PartitionKey' and "RowIndex" columns in your CSV file. Outside of Azure, these columns are meaningless. The point is, the Azure connector is having trouble generating a schema from the Azure table. The connector documentation says that this is a challenge:

This Source have generic schema for all streams. Azure Table storage is a service that stores non-relational structured data (also known as structured NoSQL data). There is no efficient way to read schema for the given table.

The author of the documentation suggests that there is a way to work around this problem:

We use data property to have all the properties for any given row.

data - This property contain all values
additionalProperties - This property denotes that all the values are in data property.
    {     "$schema": "<http://json-schema.org/draft-07/schema#>",     "type": "object",     "properties": {         "data": {             "type": "object"         },         "additionalProperties": {             "type": "boolean"         }     } }

The trouble is that the author provides no context for someone like me. Where is this

data

property? It seems to be related to the source???? And what does this sentence mean:

data - This property contain all values

?

I think the author is suggesting that I can create

data

which is a schema to represent the table. How do I do this? Where does such a definition live?

I figured out how to make this work. I used the AirByte API to create the connection and passed in a schema that corresponded to my table. http://localhost:8000/api/v1/connections/create { "name": "Azure Table Storage 4 <> data-ingestor-dest", "namespaceDefinition": "source", "namespaceFormat": "${SOURCE_NAMESPACE}", "prefix": "", "sourceId": "13150ab9-9a02-47a5-8e0a-a48d3d54cde3", "destinationId": "8fac964e-42c4-4a21-8c75-77ab95756627", "operationIds": [], "status": "active", "syncCatalog": { "streams": [ { "stream": { "name": "swayairbytetesttable", "jsonSchema": { "type": "object", "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "PartitionKey": { "type": "string" }, "RowKey": { "type": "string" }, "CRIM": { "type": "number" }, "ZN": { "type": "number" }, "INDUS": { "type": "number" }, "CHAS": { "type": "number" }, "NOX": { "type": "number" }, "RM": { "type": "number" }, "AGE": { "type": "number" }, "DIS": { "type": "number" }, "RAD": { "type": "number" }, "TAX": { "type": "number" }, "PTRATIO": { "type": "number" }, "B": { "type": "number" }, "LSTAT": { "type": "number" }, "target": { "type": "number" } } }, "supportedSyncModes": [ "full_refresh", "incremental" ], "sourceDefinedCursor": true, "defaultCursorField": [ "PartitionKey" ], "sourceDefinedPrimaryKey": [] }, "config": { "syncMode": "full_refresh", "cursorField": [ "PartitionKey" ], "destinationSyncMode": "overwrite", "primaryKey": [], "aliasName": "swayairbytetesttable", "selected": true } } ] } }