👋 Hi everyone!

I'm evaluating using airbyte first to pull data using the linkedin ads connector and use a BQ as a destination, I just created a local instance of airbyte with docker-compose to test it and I can't see the source Linkedin asd connector. Do I need to do any additional installation? or the connector is not available yet? Thanks in advance

Hi, I faced same issue when I sync connection (source: json file, destination: postgres) with noramlization.

airbyte-worker      | 2022-06-07 03:50:22 normalization > Processing destination_catalog.json...
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_columns_format (#44) to ny_p1_json_22_proper_s_view_columns_format (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_columns_cachedContents (#52) to ny_p1_json_22_proper_olumns_cachedContents (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_columns_cachedContents_top (#56) to ny_p1_json_22_proper_ns_cachedContents_top (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_metadata_attachments (#50) to ny_p1_json_22_proper__metadata_attachments (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_metadata_custom_fields (#52) to ny_p1_json_22_proper_etadata_custom_fields (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_metadata_custom_fields_Update (#59) to ny_p1_json_22_proper__custom_fields_Update (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_metadata_custom_fields_Data_Collection (#68) to ny_p1_json_22_proper_ields_Data_Collection (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_metadata_custom_fields_Dataset_Information (#72) to ny_p1_json_22_proper_s_Dataset_Information (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_metadata_renderTypeConfig (#55) to ny_p1_json_22_proper_data_renderTypeConfig (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_metadata_renderTypeConfig_visible (#63) to ny_p1_json_22_proper_derTypeConfig_visible (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_metadata_availableDisplayTypes (#60) to ny_p1_json_22_proper_availableDisplayTypes (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_approvals_submitter (#49) to ny_p1_json_22_proper_w_approvals_submitter (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_approvals_submissionDetails (#57) to ny_p1_json_22_proper_als_submissionDetails (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_approvals_submissionOutcomeApplication (#68) to ny_p1_json_22_proper_ionOutcomeApplication (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Truncating ny_p1_json_22_properties_view_tableAuthor_flags (#47) to ny_p1_json_22_proper_iew_tableAuthor_flags (#43)
airbyte-worker      | 2022-06-07 03:50:22 normalization > Traceback (most recent call last):
airbyte-worker      | 2022-06-07 03:50:22 normalization >   File "/usr/local/bin/transform-catalog", line 8, in <module>
airbyte-worker      | 2022-06-07 03:50:22 normalization >     sys.exit(main())
airbyte-worker      | 2022-06-07 03:50:22 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/transform.py", line 82, in main
airbyte-worker      | 2022-06-07 03:50:22 normalization >     TransformCatalog().run(args)
airbyte-worker      | 2022-06-07 03:50:22 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/transform.py", line 35, in run
airbyte-worker      | 2022-06-07 03:50:22 normalization >     self.process_catalog()
airbyte-worker      | 2022-06-07 03:50:22 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/transform.py", line 62, in process_catalog
airbyte-worker      | 2022-06-07 03:50:22 normalization >     processor.process(catalog_file=catalog_file, json_column_name=json_col, default_schema=schema)
airbyte-worker      | 2022-06-07 03:50:22 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/catalog_processor.py", line 63, in process
airbyte-worker      | 2022-06-07 03:50:22 normalization >     for conflict in tables_registry.resolve_names():
airbyte-worker      | 2022-06-07 03:50:22 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/table_name_registry.py", line 157, in resolve_names
airbyte-worker      | 2022-06-07 03:50:22 normalization >     conflicts = self.resolve_table_names()
airbyte-worker      | 2022-06-07 03:50:22 normalization >   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/table_name_registry.py", line 196, in resolve_table_names
airbyte-worker      | 2022-06-07 03:50:22 normalization >     assert (table_count * 2) == registry_size, f"Mismatched number of tables {table_count * 2} vs {registry_size} being resolved"
airbyte-worker      | 2022-06-07 03:50:22 normalization > AssertionError: Mismatched number of tables 64 vs 62 being resolved
airbyte-worker      | 2022-06-07 03:50:22 normalization >
airbyte-worker      | 2022-06-07 03:50:22 normalization > Showing destination_catalog.json to diagnose/debug errors (1):

Why this issue continues to occur when discover_schema works well?

AssertionError: Mismatched number of tables 64 vs 62 being resolved

Hi Team, We are having issue with source mssql connector. we have enabled CDC and sometime, it stuck in running status for more than 2 hours but. retry complete in couple of minutes. Could you shed some light on CDC connector internal ? is there any configuration for fast fail?

Hey Team, My Team and I are looking into adding airbyte into our pipelines and I wanted to know if there is a possibility to forward alerts into opsgenie?

my connection with Amazon RDS MySQL is failing while running the sync

2022-06-07 154929 WARN i.a.w.g.DefaultReplicationWorker(run):276 - State capture: No new state, falling back on input state: io.airbyte.config.State@38c15ccc[state={}] 2022-06-07 154929 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling... 2022-06-07 154929 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):157 - sync summary: io.airbyte.config.StandardSyncOutput@2c64408a[standardSyncSummary=io.airbyte.config.StandardSyncSummary@6966db62[status=failed,recordsSynced=0,bytesSynced=0,startTime=1654616624667,endTime=1654616969404,totalStats=io.airbyte.config.SyncStats@32829964[recordsEmitted=0,bytesEmitted=0,stateMessagesEmitted=0,recordsCommitted=0],streamStats=[]],normalizationSummary=<null>,state=io.airbyte.config.State@38c15ccc[state={}],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@2e62fe4f[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@58c533c3[stream=io.airbyte.protocol.models.AirbyteStream@562b4841[name=fragment,jsonSchema={“type”“object”,“properties”{“id”{“type”“string”},“ts”{“type”“string”},“data”{“type”“string”},“type”{“type”“string”},“status”{“type”“string”},“tenant”{“type”“string”},“retries”{“type”“number”},“priority”{“type”“number”},“information”{“type”“string”}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[[id], [tenant]],namespace=csi,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id], [tenant]],additionalProperties={}]],additionalProperties={}],failures=[io.airbyte.config.FailureReason@949242d[failureOrigin=source,failureType=<null>,internalMessage=io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!,externalMessage=Something went wrong within the source connector,metadata=io.airbyte.config.Metadata@30e033cd[additionalProperties={attemptNumber=0, jobId=10}],stacktrace=java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:351) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ... 3 more Caused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:136) at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:349) ... 4 more ,retryable=<null>,timestamp=1654616968681]]] 2022-06-07 154929 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):236 - Stopping temporal heartbeating... 2022-06-07 154929 INFO i.a.c.p.ConfigRepository(updateConnectionState):775 - Updating connection f4c4d3cc-feab-4b7e-9910-350e31925a31 state: io.airbyte.config.State@d0759c5[state={}]

I am coming back from testing out Airbyte a while back. Pulled the newest changes from

master

and when I attempt to start Airbyte using

docker-compose up

(intel mac) the web app says "Something went wrong... the server may still be starting up". Looking through the logs it looks like there are a couple errors that I haven't been able to resolve. Firstly, the bootloader wants me to upgrade to 0.32.0-alpha before using the current version (but I cannot find this version on DockerHub). Secondly, the server times-out "while waiting for database to fulfill minimum flyway migration version..", only thing I can see in the logs for the database is that

temporal

already exists and it stops logging after this command (maybe its stuck?)

2022-06-07 16:05:40.186 UTC [39] STATEMENT:  INSERT INTO

namespaces (partition_id, id, name, is_global, data, data_encoding, notification_version)

VALUES($1, $2, $3, $4, $5, $6, $7)

Any help is appreciated. Ive attempted to change the mounts in

.env

to a new local dir, but I get the same error. Thanks!

I got this error when running sync from postgres to redshift. The initial load running with no problem, but in the next incremental sync this error will appear Airbyte Version: 0.39.7-alpha Source: Postgres (0.4.21) Destination: Redshift (0.3.37)

2022-06-08 04:58:49 source > 2022-06-08 04:58:49 ERROR i.a.i.b.AirbyteExceptionHandler(uncaughtException):26 - Something went wrong in the connector. See the logs for more details.
2022-06-08 04:58:49 source > java.lang.IllegalArgumentException: TIMESTAMP_WITH_TIMEZONE is not supported.

Hello, we're getting duplicate rows by using Mixpanel source after schema change. From a quick look, Mixpanel api returns correct data but for some reason it's split into several on our end. Resetting data or creating a new connection doesn't solve the problem. Can't really tell the issue from the logs, but there are failed schema validation messages. First time seeing Airbyte, can you guys give pointers or something? How can we debug this?

Hi, I am trying to get data from restful api using airbyte. But I don't know how to filter duplicate parent id in HttpSubStream to avoid duplicate request. For example, the data obtained by parent HttpStream is

{
  "records": [
    {
      "id": 1,
      "group_id": 1
    },
    {
      "id": 2,
      "group_id": 1
    },
    {
      "id": 3,
      " group_id": 1
    },
    {
      "id": 4,
      "group_id": 2
    },
    {
      "id": 5,
      "group_id": 2
    }
  ]
}

The request path of HttpSubStream is

f"detail/{group_id}"

. Can anyone give some advice or use cases? Thanks！

Hi folks! Is there any way to modify the webhook (success/fail notification) message to include the connection id? Slack Conversation

Hi folks, Looking for some help setting up airbyte in kubernetes (using helm). The environment has not connectivity to public internet. I am trying to understand how I can build the required containers in our registry. Any details on how this can be done is appreciated.

Hi, I am trying to set up s3 as the destination, but I'm getting the below error -

The connection tests failed.
Could not connect to the S3 bucket with the provided configuration. Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler

I see a test connection file created in s3 which says "check content" - _airbyte_connection_test_51a17e652bc74d5881f6535f8a90be1c I have created a user specific to Airbyte with the below perms and able to access that s3 bucket from my local using python boto3 -

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::airbyte-poc/*",
        "arn:aws:s3:::airbyte-poc"
      ]
    }
  ]
}

Would anyone know this issue?

hi

Hey guys, I've a question about restricted networks and airbyte. Let's say, I've a postgres database which is only accessible from an internal company network. Furthermore, I've set up airbyte on a k8s cluster OUTSIDE this company network. Are there any best practices for how I could use airbyte to access this postgres db as a source? Basically I can install an "agent" or worker or connector on the internal network while the airbyte server is on my outside cluster. However I was not able to find out, whether this sort of "remote-worker" or "remote-connector" setup is possible... Thanks for any input there. Maybe as an addition: I can communicate from and to my internal network via HTTPS to a potential "remote worker or connector"

Good morning everybody! when pulling data from postgres to snowflake, is it possible to detect foreign keys on the source table and bring all referenced tables along with the main one?

Hi, did and filled out the application according to the instructions from Airbyte documentation, still got the rejection for the token( Google Ads.

We are unable to grant your request for a developer token to be used with a third-party tool (Airbyte) that you do not manage. This is a violation of the Google Ads API Policies, and is not a permissible use case for API access.

A developer token is a means to enable your application to connect to the Google Ads API. This token must belong to and be managed by the developer responsible for building and maintaining the codebase of the app. You should not share your developer token with another entity whose codebase you don't manage. Additionally you should not build your app in a manner that requires clients to obtain a developer token for it to be used.

hey team, anyone with experience using the

Notion

connector? I know it 'alpha' so i'm not expecting support, just wondering if anyone was able to get it to work. I'm currently able to only retrieve records from the 'users' table, but actually need the 'database' table for my reporting. Let me know, thanks!

Anyone?

quick question - i'm attempting to update a connector through the interface but i'm getting an error that shows "get spec job failed" when inspected in the dev tools, but running

docker run --rm -i <image_name>

on my machine produces output. what could the problem be?

Has anyone in the community made a python library for the Airbyte API yet?

Is there one person in the community that got their Facebook app approved? The documentation mentions it like it's just "ask and they give", but I don't think it is possible. The connector should be configured in a way that works with "standard" api limit.

Linkedin 0 Records Error Linkedin Ads -> Airbyte -> Redshift ================= Is this your first time deploying Airbyte: Yes OS Version / Instance: MacOS, Airbyte Version: 0.35.30 Source name/version: Linkedin 0.1.7 Destination name/version: 0.3.23 source test: successful Step: Running first sync Error message:

2022-06-09 17:29:40 [44msource[0m > Read 0 records from campaigns stream

Full log attached Any idea why it's reading zero records?

Hi ... New to Airbyte, trying to connect to salesforce instance but cant generate a refresh token using the provided documentation, possible to get some help

Hi. Is there an example for setting up a GitHub Action for running PyTest on a new source connector?

For me, logical replication for postgres incremental load some fields are sometimes empty in the output json even though source system has the value. Is there a way to fix it these kinds of issues ? And which version of open source airbyte is stable one or I need to wait for some time to get painless migration

Anyone can help? https://github.com/airbytehq/airbyte/issues/13661

Hi all. Can someone tell me why in

source Jira: User

I always get only 50 users? How can I get everyone?

Docs mention setting these two paramaters when running on an ec2 to maximise memory usage.

JOB_MAIN_CONTAINER_MEMORY_REQUEST=

JOB_MAIN_CONTAINER_MEMORY_LIMIT=

What should the values look like? String of memory size e.g. “28GB”? Is it in bytes as an int? Something else?