Hello guys,

And, another one...After getting successfully connection, I got this error:

any help or advice is greatly appreciated !

I realize this might have been better placed in the troubleshooting channel 🤷

Hi everyone! My team is looking to use Airflow (via GCP composer) with our Airbyte instance. It looks like there was a talk on this a few months back but I'm having trouble finding a recording or documentation about how to set this up. Can anyone point me to some resources to get started? (leaving this here for others who might search this slack in the future 👀 )

Hi All,

I am trying to setup airbyte on huawei cloud ubuntu instance but received following error : ERROR: Invalid interpolation format for "environment" option in service "worker": "CONFIG_DATABASE_USER=${CONFIG_DATABASE_USER:-}" Could you please help. I am following this documentation : https://docs.airbyte.io/deploying-airbyte/on-aws-ec2

image.png

does it need any extra configuration?

i am connecting MySQL to local database

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9.991091900s. [closed=[], open=[[remote_addr=airbyte-temporal/172.18.0.9:7233]]]

Whilst the non typed struct does?

Is one not maintained to same degree?

Hey all, I'm trying to use the API on an open source local deployment. Based on the docs, I've set my tool to test the api to http://localhost:8000/v1/source_definitions/list and POST but am getting back a 405 Not Allowed error. What am I doing wrong and how to resolve? NM: correct URL is http://localhost:8000/api/v1/source_definitions/list

Hey, is there a Google Drive source in Airbyte ?

Hi All, I am trying to load S3 csv file to Database. Currently running airbyte on kubernetes. And getting following exception.

2022-01-24 19:31:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):97 - timed out while running infer_schema_process after 4 seconds, retrying...
2022-01-24 19:31:28 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):97 - timed out while running infer_schema_process after 8 seconds, retrying...
2022-01-24 19:31:44 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):97 - timed out while running infer_schema_process after 16 seconds, retrying...
2022-01-24 19:32:16 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):97 - timed out while running infer_schema_process after 32 seconds, retrying...
2022-01-24 19:33:47 INFO i.a.w.p.KubePodProcess(getReturnCode):682 - Exit code for pod source-s3-sync-139-0-vsmpu is 1
2022-01-24 19:33:47 INFO i.a.w.p.KubePodProcess(exitValue):697 - Closed all resources for pod source-s3-sync-139-0-vsmpu
2022-01-24 19:33:47 INFO i.a.w.p.KubePodProcess(exitValue):697 - Closed all resources for pod source-s3-sync-139-0-vsmpu
.
.
  File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 229, in _get_master_schema
    this_schema = file_reader.get_inferred_schema(f)
  File "/airbyte/integration_code/source_s3/source_files_abstract/formats/csv_parser.py", line 110, in get_inferred_schema
    schema_dict = self._get_schema_dict(file, infer_schema_process)
  File "/airbyte/integration_code/source_s3/source_files_abstract/formats/csv_parser.py", line 127, in _get_schema_dict
    self._convert_options(),
  File "/airbyte/integration_code/source_s3/utils.py", line 31, in run_in_external_process
    raise TimeoutError(f"Timed out too many times while running {fn.__name__}, max timeout of {max_timeout} seconds reached.")
TimeoutError: Timed out too many times while running infer_schema_process, max timeout of 60 seconds reached.
2022-01-24 19:33:16 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):97 - Finished syncing SourceS3
2022-01-24 19:33:16 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):97 - SourceS3 runtimes:
2022-01-24 19:33:16 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - Timed out too many times while running infer_schema_process, max timeout of 60 seconds reached.

Above error is not consistent

we are running 15+ such airbyte connection concurrently on airbyte

and any random connections are failing with this error

Is this your first time deploying Airbyte: Yes OS Version / Instance:Windows 10 Memory / Disk: 1Tb SSD Deployment: Docker Airbyte Version: https://github.com/airbytehq/airbyte.git Step: Python CDK Speedrun: Creating a Source Description: When I'm trying to run ./generate.sh command I'll get "While trying to generate a connector, an error occurred on line 38 of generate.sh and the process aborted early. This is probably a bug."

Hallo Guys, quick poll. Which Database did you choice for your Data warehouse ? 😉PostgreSQL? 😚 MySQL ? 🤗MSSQL ? 😎MariaDB? react with an emoji!

No error or nothing my source and destination connections are good

I am running ec2 - t2.xlarge - 4core - 16GB

I am having 300files and 100MB data

How airbyte usets go about GDPR requests for users to be deleted. Specifically for CDC where SCD tables would persist a record of a user being deleted.

Hello, I am trying to implement incremental streams or slides to improve data recovery performance. since pulling everything in a single block takes a lot of resources and the aws instance crash I followed other sources and set state_checkpoint_interval but it keeps bringing back all the logs in one go, it should be 100 by 100 I think so

class ServicesnowApi(HttpStream):
    url_base = "<https://xxx.service-now.com/api/now/v1/>"

    # Set this as a noop.
    primary_key = None
    # Save the state every 100 records
    state_checkpoint_interval = 100
    page_size = 100
    cursor_field = "sys_updated_on"

    def __init__(self, limit: str, sys_created_from: str, **kwargs):
        super().__init__(**kwargs)
        # Here's where we set the variable from our input to pass it down to the source.
        self.limit = limit
        self.sys_created_from = sys_created_from


    def path(self, **kwargs) -> str:
        # This defines the path to the endpoint that we want to hit.
        limit = self.limit
        sys_created_from = self.sys_created_from
        return f"table/incident?sysparm_offset=0&sysparm_limit={limit}&sysparm_query=sys_created_on>={sys_created_from} 00:00^active=ISNOTEMPTY"


    def request_params(
            self,
            stream_state: Mapping[str, Any],
            stream_slice: Mapping[str, Any] = None,
            next_page_token: Mapping[str, Any] = None,
    ) -> MutableMapping[str, Any]:
        limit = self.limit
        sys_created_from = self.sys_created_from
        return {"limit": limit, "sys_created_from":sys_created_from}


    def parse_response(
            self,
            response: requests.Response,
            stream_state: Mapping[str, Any],
            stream_slice: Mapping[str, Any] = None,
            next_page_token: Mapping[str, Any] = None,
    ) -> Iterable[Mapping]:
        result = response.json()['result']
        return result


    def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
        return None

Thread in Slack Conversation

image.png

Hello, I started using Airbyte (v 0.35.14-alpha) to get data from Bigquery to Kafka in a secure way. Now, while trying so, If I use PLAINTEXT Security protocol for data transfer with Kafka as destination, then data gets transferred. But when I use SASL_SSL protocol. And SASL Mechanism is SCRAM-SHA-256. Now as you can see in the screenshot attached, there’s a field to enter the JAAS config but there’s no place to add SSL Certificate. And I went through Airbyte Docs and found that Airbyte transfers the data securely.* *https://docs.airbyte.com/project-overview/security

All Cloud Connectors (APIs, files, databases) pull data through encrypted channels (SSL, SSH tunnel, HTTPS) and the data transfer between our clients' infrastructure and Airbyte infrastructure is fully encrypted. Some users may elect to whitelist our server IPs to allow them to access their DB server behind a firewall. This is an optional opt-in process. In OSS, there are some Connectors that allow unencrypted data transfer (ex. when pulling from a local database) where data is never transiting the public internet or transiting to Airbyte.

So I need some clarity over here, 1. If I use PLAINTEXT protocol, will the data transfer be encrypted?? or will PLAINTEXT protocol overwrite the internal encryption protocol? i.e. Will my data transfer with PLAINTEXT encrypted internally?? 2. And If I use SASL_SSL protocol with SCRAM-SHA-256 SASL Mechanism and pass JAAS config, transfer fails with AUTHENTICATION error. Is it because of missing SSL Cert? If so, Is there a workaround for it? Can someone please help me with this.

I have no issues for Composer 1 (VPC - Firewall) using this guide but we want to upgrade to Composer 2 that use auto-scaling: I am having trouble connecting Composer 2 (VPC -Firewall) with Airbyte (VM). I am not so familar with kubernetes autopilot with composer 2. Unable to find the external IP. Anyone can assist in the VPC firewall for autoscaling kubernetes clusters connection setting?

Screen Shot 2022-02-09 at 5.05.03 PM.png