hello everyone! when instantiating an airbyte record message is it mandatory to specify the data as a dictionary as

{"key": "value"}

or is it possible to pass multiple records in a single message using a list of dictionaries

[{"key": "value1"}, {"key": "value2"}]

? It is taking a lot of time to iterate over every single record in the dataset when to yield it as a record message, if I could have a list of 100000 records in a single message it would take a lot less time since the data I am trying to collect has millions of records

Hey @Gustavo Maia, are you still stuck on this? I found this example of a nested schema, does that look similar to what you're going for? There's also more info in the Data Types in Records doc, the section under Arrays looks similar to your idea:

Arrays contain 0 or more items, which must have a defined type. These types should also conform to the type system. Arrays may require that all of their elements be the same type (

"items": {whatever type...}

), or they may require specific types for the first N entries (

"items": [{first type...}, {second type...}, ... , {Nth type...}]

, AKA tuple-type)

Tuple-typed arrays can configure the type of any additional elements using the

additionalItems

field; by default, any type is allowed. They may also pass a boolean to enable/disable additional elements, with

"additionalItems": true

being equivalent to

"additionalItems": {"$ref": "WellKnownTypes.json#definitions/String"}

and

"additionalItems": false

meaning that only the tuple-defined items are allowed.

Destinations may have a difficult time supporting tuple-typed arrays without very specific handling, and as such are permitted to somewhat loosen their requirements. For example, many Avro-based destinations simply declare an array of a union of all allowed types, rather than requiring the correct type in each position of the array.

Hey @Sunny Hashmi (Airbyte) thanks for the the reply. I don't think this is the solution for my problem. A single record of the data I am collecting is a simple non-nested object, I wanted to send multiple different objects in the same airbyte message, which I don't think using nested object structures would be a good option. I could put multiple records into a single record but that would not represent the reality of the data and I would need to unnest it after to make it useful

Sorry to hear it Gustavo, if you haven't already you're welcome to create a feature request in case something can be implemented

Sure, I will