Dejan Antonic
09/21/2021, 11:36 AMOleksandr Shevchenko
09/21/2021, 1:42 PMdocker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-file:0.2.6 read --config /secrets/config_csv.json --catalog /sample_files/configured_catalog_csv.json > messages.json
cat messages.json | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/destination-csv:0.2.8 write --config /secrets/config_d_csv.json --catalog /sample_files/configured_catalog.json
or just
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-file:0.2.6 read --config /secrets/config_csv.json --catalog /sample_files/configured_catalog_csv.json | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/destination-csv:0.2.8 write --config /secrets/config_d_csv.json --catalog /sample_files/configured_catalog.json
But I don’t see that destination-csv
is consuming AirbyteMessages
cat messages.json | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/destination-csv:dev write --config /secrets/config_d_csv.json --catalog /sample_files/configured_catalog.json
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationRunner(run):96 - {} - Running integration: io.airbyte.integrations.destination.csv.CsvDestination
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=/sample_files/configured_catalog.json, write=null, config=/secrets/config_d_csv.json}
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationRunner(run):100 - {} - Command: WRITE
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationRunner(run):101 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='/secrets/config_d_csv.json', catalogPath='/sample_files/configured_catalog.json', statePath='null'}
2021-09-21 13:29:01 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2021-09-21 13:29:02 INFO i.a.i.d.c.CsvDestination$CsvConsumer(<init>):157 - {} - initializing consumer.
2021-09-21 13:29:02 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):80 - {} - Airbyte message consumer: succeeded.
2021-09-21 13:29:02 INFO i.a.i.d.c.CsvDestination$CsvConsumer(close):198 - {} - finalizing consumer.
2021-09-21 13:29:02 INFO i.a.i.d.c.CsvDestination$CsvConsumer(close):214 - {} - File output: /local/tmp/out/_airbyte_raw_dmesh.csv
2021-09-21 13:29:02 INFO i.a.i.b.IntegrationRunner(run):153 - {} - Completed integration: io.airbyte.integrations.destination.csv.CsvDestination
Could you please give me a hint of what am I doing wrong? Is there some problem with config.json?Kriti (Postman)
09/21/2021, 9:01 PMAirbyteMessage
(image for reference, line 34). It is not available at io.airbyte.protocol.models
What am I missing here?Mihir Kanzariya
09/22/2021, 4:37 AMMané Rom
09/22/2021, 1:38 PMBlake Enyart
09/22/2021, 5:55 PMJonathan Stacks
09/23/2021, 2:06 AMsocat
pods that get spun up in the k8s cluster. I wonder if there could be a strategy that is could hosted for metadata, logs, RBAC, SSO/SAML, etc but an agent is deployed locally that establish a TCP connection with an Airbyte endpoint. The agent would then spin up the pods for moving data the same as it does today and collect logs and other metadata and ship it to Airbyte.tharaka prabath
09/23/2021, 9:48 AMDustin Salmons
09/23/2021, 2:09 PMArtem Astapenko
09/23/2021, 8:57 PMworkTime: {
type: "array",
title: "Work time",
items: {
type: "string",
enum: ["day", "night"],
},
},
Edit:
Ok, looks like we do not have such specs. Figured out how it should look like.Jonathan Stacks
09/24/2021, 3:18 AM/api/v1/health
? I'm seeing that they are listening on port 9000 for heartbeating, but I'm not sure if that is valid to use for a liveness or readiness probeAndrey Morskoy
09/24/2021, 12:20 PMRecords read: 29000
• then just disabled WiFi
• job remains in Running state
- leaving it for couple of minutes to simulate network issue
• then I've re-enabled WiFi
• Nothing changed, job is still in the same Running
state, distination file is not updated since then.
Are there any retry policies for such network/data availability issues? For me it looks like there is couple of issues:
• I am not notified on sync error
• there is no job continuation - seems that I need manually reset connection?Michel
airbyte-queue
?Martin Larsson
09/24/2021, 9:00 PMequests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: <https://provetcloud.com/{clinic_ID}/api/0.1/consultation/>
Artem Astapenko
09/26/2021, 5:01 PM.
in github repo will open this repo in web version of VS-code. Pressing it in PR - will open this PR in web version on VS-codeAkilesh V
09/27/2021, 8:52 AMSawyer Waugh
09/27/2021, 2:23 PMairbyte/airbyte
from git, and then make a new branch for each connector?
There is a lot written in the docs about connector dev, but i’m not finding anything specific for managing git (and the section on Monorepo Python Development + Gradle is a bit intimidating for airbyte novices looking to experiment with the platform).
If someone from the dev team can confirm or correct the below, I am happy to record my process and contribute to the docs
I understand that new connectors get containerized. If I am new to Airbyte, looking to quickly get started with a custom connector, and don’t intend to contribute to main in the immediate future…am I right to think that -
1. my fork with custom connector(s) never needs to be deployed to production and should only exist locally
2. my production airbyte instance should be deployed directly from airbyte/airbyte
3. locally-developed custom connectors should be containerized and added to prod via the UI
a. can private container images be added via the UI? I have only been able to add publicAmeya Bapat
09/28/2021, 5:50 PMThe current behavior of Incremental is not able to handle source schema changes yet, for example, when a column is added, renamed or deleted from an existing table etc. It is recommended to trigger a Full refresh - Overwrite to correctly replicate the data to the destination with the new schema changes.
Ref: https://docs.airbyte.io/understanding-airbyte/connections/incremental-append
What is the workaround that you propose to identify changes in schemas ? Do we need to handle schema change detection logic out side of airbyte and then trigger full refresh?Marc García
09/30/2021, 9:24 AMDon H
10/01/2021, 6:21 PMMartin Larsson
10/02/2021, 9:11 PM9 38.65 running build_ext
#9 38.65 building 'multidict._multidict' extension
#9 38.65 creating build/temp.linux-x86_64-3.7
#9 38.65 creating build/temp.linux-x86_64-3.7/multidict
#9 38.65 gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/local/include/python3.7m -c multidict/_multidict.c -o build/temp.linux-x86_64-3.7/multidict/_multidict.o -O2 -std=c99 -Wall -Wsign-compare -Wconversion -fno-strict-aliasing -pedantic
#9 38.65 error: command 'gcc' failed with exit status 1
#9 38.65 ----------------------------------------
#9 38.66 ERROR: Failed building wheel for multidict
#9 38.66 Successfully built source-provet pyrsistent PyYAML wrapt pendulum
#9 38.66 Failed to build yarl multidict
#9 38.66 ERROR: Could not build wheels for yarl, multidict which use PEP 517 and cannot be installed directly
------
executor failed running [/bin/sh -c pip install --prefix=/install .]: exit code: 1
I am running on a mac which points python by default to 2.7. If I run python I use python3 and pip3. Can that be the problem here?Akilesh V
10/04/2021, 5:57 AMpip is looking at multiple versions
because of this i am unable complete the docker build, i tried docker build for existing singer tap quickbook
for this also i am facing same issue.Cristiano Sarmento
10/04/2021, 5:37 PMMartin Larsson
10/04/2021, 8:39 PMdocker build . -t airbyte/source-my-connector:dev
but now what? I have tried adding my connector in the source_definitions.yaml and restarted all local airbyte docker containers, but it does not appear among the sources in the UI. Any tips?Dmytro Rezchykov
10/05/2021, 7:13 AMMadhup Sukoon
10/05/2021, 12:28 PMtharaka prabath
10/06/2021, 5:39 AMMadhup Sukoon
10/06/2021, 3:26 PMIntercom
source:
(Maybe an issue with pagination?)Eugene Kulak
Mohammad Safari
10/07/2021, 11:28 AM