👋 Is there an easy way to validate incoming data with the stream's json schema when building a destination using the python cdk? Can Airbyte be instructed to drop a record or key that isn't part of the json schema? Or is that logic that needs to be implemented in the destination, when desired?
m
Marcos Marx (Airbyte)
11/02/2022, 7:11 PM
Destinations don’t support this today, today the low-code CDK allow to filter records based in rules.
Destinations try to ingest generic data and don’t transform data.
h
Henri Blancke
11/02/2022, 7:53 PM
Thanks @Marcos Marx (Airbyte)! So in the example of a record containing additional properties that are not defined in the json schema, what's the ideal way for a destination to handle those?
m
Marcos Marx (Airbyte)
11/02/2022, 8:03 PM
The destination must handle the AirbyteMessage(airbyte_id, emitted_at, data which is a json object)
h
Henri Blancke
11/02/2022, 8:05 PM
Gotcha, if that json object contains a key that is not in the json schema (and the json schema was used to create a database table schema by the destination), is it acceptable to drop the key that wasn't originally in the json schema from the record/json object?