Hello, is it possible to load JSON files from S3 a...
# ask-community-for-troubleshooting
r
Hello, is it possible to load JSON files from S3 and output it to Iceberg?
k
A message from kapa.ai
Yes, it is possible to load JSON files from S3 and output them to Iceberg using Airbyte. You can use the S3 source connector to read JSON files and the Iceberg destination connector to write the data to Iceberg tables. For the S3 source connector, you can use the JSON Lines format, which is a text format with one JSON object per line. You can find more information about the JSON Lines format in the [S3 destination documentation](https://docs.airbyte.com/integrations/destinations/s3#supported-output-schema-json-lines). For the Iceberg destination connector, the incoming Airbyte data is structured in keyspaces and tables, and is partitioned and replicated across different nodes in the cluster. The connector maps an incoming
stream
to an Iceberg
table
and a
namespace
to an Iceberg
database
. You can find more information about the Iceberg destination connector in the [Iceberg destination documentation](https://docs.airbyte.com/integrations/destinations/iceberg). To set up the data pipeline, follow these steps: 1. Set up the S3 source connector with the JSON Lines format. 2. Set up the Iceberg destination connector. 3. Configure the connections and sync settings between the source and destination connectors. Please note that the JSON schema from the source will be converted to an Avro schema, and then the JSON object will be converted to an Avro record before being written to the Iceberg table. You can learn more about this conversion process in the [JSON to Avro conversion documentation](https://docs.airbyte.com/understanding-airbyte/json-avro-conversion).
r
JSON connector mentions it is able to load pure JSON from S3, but S3 connector mentions only JSONL