Finn Frotscher
09/23/2021, 12:30 PMAnna Mariscotti
09/23/2021, 3:43 PMNick Akincilar
09/24/2021, 3:19 PMTomas Peluritis
09/25/2021, 6:03 PMCheck their health status
gets me to "Page not found" is that expected?Preetam Balijepalli
09/27/2021, 1:25 PMAmeya Bapat
09/28/2021, 9:50 AMCurrently, each data sync will only create one file per stream. In the future, the output file can be partitioned by size. Each partition is identifiable by the partition ID, which is always 0 for now.
Ref: https://docs.airbyte.io/integrations/destinations/s3
When will we get this feature? Parsing/reading from S3 files of size 10s of GBs would be complicated.Ameya Bapat
09/28/2021, 10:01 AMThe current behavior of Incremental is not able to handle source schema changes yet, for example, when a column is added, renamed or deleted from an existing table etc. It is recommended to trigger a Full refresh - Overwrite to correctly replicate the data to the destination with the new schema changes.
Ref: https://docs.airbyte.io/understanding-airbyte/connections/incremental-append
What is the workaround that you propose to identify changes in schemas ? Do we need to handle schema change detection logic out side of airbyte and then trigger full refresh?Jeff Crooks
09/28/2021, 12:43 PMKriti (Postman)
09/29/2021, 9:17 AMssl=true
. How can I do this in Airbyte?Zach Brak
09/30/2021, 1:22 AMStefan Otte
10/01/2021, 4:18 PMdump-airbyte-configuration-to-file
and b) the option to load that file when running airbyte via docker-compose up
Or maybe you folks have another nice workaround.Milad saidi
10/03/2021, 1:49 PM2021-10-03 13:47:15 ERROR i.a.s.a.JobScheduler(run):86 - {workspace_app_root=/workspace/scheduler/logs} - Job Scheduler Error
java.lang.RuntimeException: io.airbyte.validation.json.JsonValidationException: json schema validation failed.
errors: $.workspaceId: is missing but it is required
Kriti (Postman)
10/04/2021, 8:06 AMSetup destination
button. The process Testing connection
is running
Description: I am trying to setup Redshift destination. The cluster IP is whitelisted for Redshift access. The Testing connection
process keeps running for 10 mins (maybe more) and then stops with this message.Namer Medina
10/04/2021, 2:26 PMKriti (Postman)
10/04/2021, 5:00 PMSetup destination
Ā button.
Description:Ā Destination setup fails because I get permission deny on the database. The credentials I am using do have permission to access the database.Jeff Crooks
10/05/2021, 1:05 PMPhilippe Boyd
10/05/2021, 7:39 PM2021-10-05 19:38:22 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 67000
2021-10-05 19:38:22 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 68000
2021-10-05 19:38:23 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 69000
2021-10-05 19:38:23 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 70000
2021-10-05 19:38:24 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 71000
When dealing with tables that have millions of records, it will have a good performance effect to increase batch sizeJorge Castillo
10/05/2021, 8:56 PMBoopathy Raja
10/06/2021, 1:59 AMZach Brak
10/06/2021, 2:04 AMSiavoush Mohammadi
10/07/2021, 10:28 AMTuhin Banerjee
10/08/2021, 4:53 AMVlĆ”Äa Macek
10/09/2021, 6:53 PMBoopathy Raja
10/10/2021, 2:54 AMERROR () LineGobbler(voidCall):65 - Exception in thread "main" java.lang.RuntimeException: com.google.cloud.bigquery.BigQueryException: Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.
Bruno Quinart
10/10/2021, 8:03 PMJulien Bovet
10/11/2021, 12:43 PMMatt Arderne
10/11/2021, 4:29 PMPatrick McKinley
10/11/2021, 6:59 PMTuhin Banerjee
10/12/2021, 8:09 AMSathiya Sarathi Gunasekaran
10/12/2021, 10:52 AM