Hello! I am a newbie when it comes to Airbyte and AWS. I have followed the instructions to deploy Airbyte on an EC2 instance. When I SSH tunnel, I can access the instance through localhost. When I try to access it through the public IP address, I get 'This site can't be reached'. I feel like I need to map the docker port to the EC2 port (80) but not sure how to do it. Can somebody help!

[Asking kapa.ai] Do any airbyte connectors support incremental sync by calendar month?

Hi Team, How we can get source CatalogId Any Idea ????

Hi Team, How to generate catalogId from configuration API??

how to add ca certificate in destination of Elasticsearch through API

Hi Team, we are performing sync with huge amount of data ~ 400 GB from mySql to S3 within AWS EC2 instance having 4 core CPU and 16 GB RAM. While performing sync, data upto 175 GB was read successfully, but suddenly we saw issue with source connector something like below.

2023-04-03 11:23:14 [44msource[0m > ERROR i.a.i.b.AirbyteExceptionHandler(uncaughtException):26 Something went wrong in the connector. See the logs for more details. java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Streaming result set com.mysql.cj.protocol.a.result.ResultsetRowsStreaming@2a49fe is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.

After this sync worker fails. Surprisingly, all the files which was written to S3 also get removed and empties the bucket. It used all the three attempts. For the third attempt data persists in S3, however sync got failed. Please find attached log files. Let us know if you need any further information. @Vijaya Kotwani @Shiraj Gagneja @Mehak Kapoor

Hi Team, I'm trying to connect to my instance using this line gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME -- -L 8000localhost8000 -N, however after adjusting the project_id and instance_name AND making sure I am connected to the cloud project, it only gives me the following error: ERROR: (gcloud.beta.compute.ssh) unrecognized arguments: -L 8000localhost8000 -N

Hi team, is there any way I can change the text that is generated by Airbyte Slack webhook? My use case is that I want to replace

connectionId

with

stream name

.

when i run python main.py write --config secrets/config.json --catalog sample_files/configured_catalog.json i am following steps as per documentation https://docs.airbyte.com/connector-development/tutorials/building-a-python-destination/ i am getting below exception for input_message variable , { "type": "LOG", "log": { "level": "FATAL", "message": "'NoneType' object is not iterable\nTraceback (most recent call last):\n File \"/home/chetanm/Documents/airbyte/airbyte-integrations/connectors/destination-dest-test/main.py\", line 11, in <module>\n DestinationDestTest().run(sys.argv[1:])\n File \"/home/chetanm/Documents/airbyte/airbyte-integrations/connectors/destination-dest-test/.venv/lib/python3.10/site-packages/airbyte_cdk/destinations/destination.py\", line 119, in run\n for message in output_messages:\n File \"/home/chetanm/Documents/airbyte/airbyte-integrations/connectors/destination-dest-test/.venv/lib/python3.10/site-packages/airbyte_cdk/destinations/destination.py\", line 113, in run_cmd\n yield from self._run_write(config=config, configured_catalog_path=parsed_args.catalog, input_stream=wrapped_stdin)\n File \"/home/chetanm/Documents/airbyte/airbyte-integrations/connectors/destination-dest-test/.venv/lib/python3.10/site-packages/airbyte_cdk/destinations/destination.py\", line 49, in _run_write\n yield from self.write(config=config, configured_catalog=catalog, input_messages=input_messages)\nTypeError: 'NoneType' object is not iterable" } }

Hello guys, I'm trying to script Airbyte with the config API but I'm getting this error when triggering a sync:

2023-04-04 12:13:11 ERROR i.a.w.g.DefaultReplicationWorker(replicate):280 - Sync worker failed.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.NullPointerException: Cannot invoke "java.util.Set.contains(Object)" because "fieldsInCatalog" is null
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]

It will be really helpful if someone can drop me a hint on what may be going wrong when sending sources/connections parameters?

This message was deleted.

Hello Everyone octavia wave, Just deployed Airbyte 0.41.0. Whenever trying to update a source or destination, I am getting Server Error: Get Spec job failed. Can anyone please give me some pointer regarding the root cause and potential fix? Screenshot attached for reference.

Hey, I was using the airbyte kustomize deployment method so far and would like to migrate to helm - as kustomize was sunset. Is there a migration path provided? Or how can one update from kustomize versoin 0.40.25 to the latest airbyte version using helm?

Hey 👋 anyone familiar with the following setup: Instagram Source -> Google Big Query Destination and able to help me out quickly, please? 1. I’ve issues to understand the issue (e.g. is it the one in the screenshot) that causes the connection to fail? 2. I had issues creating the API access token (it was not 100% clear from the docs), e.g. I have no clue if “Instagram Public Content Access” is enabled. a. I use a user token now with the following permissions (2nd screenshot) thanks, and btw nice to meet you all 👋

Ugg, hey folks. I am helping out a client on analytics of their Slack. I just realized that it seems like the Slack connector doesn't pull data from private channels, which is exactly the data they need to analyze. Does this mean I need to build my own Slack connector now? It seems like Airbyte used to offer this, but now doesn't. Is that correct?

Hi, I'm new to Airbyte, and I haven't tested it yet, but can someone please further explain how the free Open Source works? Is it completely free to set up? Or are there other side-costs? I wouldn't mind paying, but it's not clear enough to me.

hey folks, a question regarding Amplitude connector: I've spotted there are some events in the

events

table that appear in Amplitude, but not in my dwh (Snowflake), and the pattern they seem to have in common is that

event_time

is several days apart from the

server_upload_time

(which is the parameter I suspect Airbyte is passing to Amplitude API). I have

incremental sync - dedup

mode selected, and the cursor field is hardcoded to

event_time

. Can someone confirm if the issue is caused by the fact that

server_upload_time

is several days after the

event_time

? Example: • event "uuid": "78843499-c242-11ed-9219-3f84661c369d" • "event_time": "2023-03-13 163040.368000" • "server_upload_time": "2023-03-14 083025.193000" • Amplitude sync ran on 2023-03-13 224209 and 2023-03-14 104209, but didn't pick up this event

We are trying to implement CDC using Airbyte for SQL Server to Snowflake. We have added a new column to a table and reran the sync process, the column shows up in Snowflake, but it has a value of NULLS. Does anyone might know why this is happening? We would like to have the new columns with CDC enabled have their respective values in source to target.

Hi Team , I am new to Airbyte

When I am installing , and following the below steps

git clone <https://github.com/airbytehq/airbyte.git>  
cd airbyte  
./run-ab-platform.sh

The git is not getting cloned properly. Neither I have the run-ab-platform.sh file present or the docker compose yml file.? Does any one facing the same issue?

Hi Team, what's the public IP address of Airbyte? I need to whitelist it for my AWS SSH tunnel server to setup AWS RDS connector

Hello guys,

I am new to airbyte and I have a question regarding data normalisation. If let's say that I connect jira as source and as destination s3 and I want my data to be normalised when it will take place this step? All data from jira it will be first loaded on s3 and saved (because airbyte is ELT ) and after that airbyte will read again the file from s3 and do further processing? Thanks

Hey guys, how is it going? I've been having a problem with Airbyte deployed in Kubernetes lately. I want to scale it, since we have a lot of CPU/Memory demanding jobs. It would be better if we could set a different CPU/Memory per job, but since it isn't possible I want to always schedule at least 2 cores of CPU and 1 Gb of memory per job execution and I know there are the variables

global.jobs.resources.memory

and

global.jobs.resources.cpu

, but even after setting them into helm (as mentioned in the docs) and even changing them directly into the config map the job pods return with lower requests, like:

requests:
        cpu: 100m
        memory: 25Mi

Do you guys have any idea on what could I be doing wrong? Airbyte is running on 0.41.0 version

We have provisioned a small DB for Airbyte initially: 5 GB. We’re running low on space, and I’m wondering whether we need to increase storage or if there are any Airbyte caches I can clear to maintain the same amount of storage. Any advice is appreciated.

Hi Team, I'm trying to change the default Airbyte configuration database to an external PostgreSQL database(Aurora) Followed these instructions https://docs.airbyte.com/operator-guides/configuring-airbyte-db I was able to change the database of Airbyte instance running on my local system. But I'm facing issues while trying to do the same on EC2 hosted Airbyte and Aurora. Am I missing something? PS: I'm able to use this same database as a destination to store the extracted data, and the connection is fine.

airbyte-temporal                  | Unable to start server. Error: sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "<http://192.168.xxx.xxx|192.168.xxx.xxx>", user "sampleairbyte", database "temporal", no encryption 
airbyte-temporal exited with code 1 
  
airbyte-server                    | 2023-04-04 15:46:27 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):259 - Waiting for namespace default to be initialized in temporal... 
airbyte-server                    | Apr 04, 2023 3:46:27 PM io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext 
airbyte-server                    | WARNING: [Channel<15>: (airbyte-temporal:7233)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host airbyte-temporal, cause=java.lang.RuntimeException: java.net.UnknownHostException: airbyte-temporal 

airbyte-temporal                  | {"level":"error","ts":"2023-04-04T17:17:28.435Z","msg":"Unable to read dynamic config  
file. Continue with default settings but the ERROR MUST BE FIXED before the next upgrade", 
"error":"unable to validate dynamic config: dynamic config: config/dynamicconfig/development.yaml: 
stat config/dynamicconfig/development.yaml: no such file or directory","logging-call-at":"main.go:140", 
"stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error|go.temporal.io/server/common/log.(*zapLogger).Error>\n\t/temporal/common/log/zap_logger.go: 
142\nmain.buildCLI.func2\n\t/temporal/cmd/server/main.go:140\<http://ngithub.com/urfave/cli/v2.(*Command)|ngithub.com/urfave/cli/v2.(*Command)>. 
Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/command.go:163\<http://ngithub.com/urfave/cli/v2|ngithub.com/urfave/cli/v2>. 
(*App).RunContext\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:313\<http://ngithub.com/urfave/cli/v2.(*App)|ngithub.com/urfave/cli/v2.(*App)>. 
Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224\nmain.main\n\t/temporal/cmd/server/main.go: 
50\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255"}

Is there any published info on the new (?)

FEATURE_FLAG_CLIENT

variable present in the .env file?

Hey folks. Is there a way to customize the notification we get on slack through airbyte? We currently get a connection failure notification but it gives an id instead of the name of connection. Any way we can get the name in the notification?

Hey 👋 can somebody please help me with Instagram? https://airbytehq.slack.com/archives/C021JANJ6TY/p1680619288998829

Hello Airbyte community, Don't know where to say it, but I really appreciate the UI improvements brought in the last months and especially in the latest version 0.42.1! It feels nicer and smoother with each version, congrats and thanks Airbyte team octavia thanksoctavia muscle