This is my first testing using Airbyte. I am loading a file in GS to BQ

hello, i’ve tried to run integration test on my PR, but it’s been more than an hour with no update. is this normal?

Where does

icon:

come from in source_definitions.yaml?

Is it possible to have Airbyte send any sort of global config or environment variable(s) to every connector process, independent of the normal source/destination/connection configs?

I'm running the acceptance tests for my connector and I'm getting an attribute error in

_scan_schema

because the subschema is being read as a list. Not sure if the required key should be included but it is.

subschema = [{'properties': {'Connector': {'type': 'string'}, 'Current': {'type': 'string'}, 'Description': {'type': ['null', 'str... ...}, 'required': ['portNumber', 'stationName', 'Geo', 'Description', 'Reservable', 'Status', ...], 'type': 'object'}]
path = '/Port/[]'

    def _scan_schema(subschema, path=""):
        if "oneOf" in subschema or "anyOf" in subschema:
            if annotate_one_of:
                return [
                    _scan_schema({"type": "object", **s}, path + f"({num})")
                    for num, s in enumerate(subschema.get("oneOf") or subschema.get("anyOf"))
                ]
            return [_scan_schema({"type": "object", **s}, path) for s in subschema.get("oneOf") or subschema.get("anyOf")]
>       schema_type = subschema.get("type", ["null"])
E       AttributeError: 'list' object has no attribute 'get'

../../bases/source-acceptance-test/source_acceptance_test/utils/json_schema_helper.py:194: AttributeError

Is this an issue with my configured catalog?

Hello, I'm developing a custom source connector for my company. When I run this command

python main.py read --config secrets/config.json --catalog source_bubble/schemas/bub.json

it gives the following error:

{"type": "LOG", "log": {"level": "FATAL", "message": "1 validation error for ConfiguredAirbyteCatalog\nstreams\n field required (type=value_error.missing)

Any idea what's wrong ?

Hi, I'm wondering how often the state gets written to the

state

table in the

airbyte

database? My understanding was that this happens every

state_checkpoint_interval

, however, this does not appear to be the case. • I can see in the logs that the state gets set every `state_checkpoint_interval`; • If I check the state in the database after this log appears, no update is made; • Several

STATE

logs appear, yet the database is still not updated. It seems to only happen on job completion, or after some very large number of records have been processed. Can you please provide some further information on how this exactly works, and if this can be configured?

Hello everybody, I am trying to follow the tutorial on creating a http api source from the echangerate api, but apparently the api is requiring an access api key. When I try to call https://api.exchangeratesapi.io/latest?base=USD I got the reponse:

{
  "success": false,
  "error": {
    "code": 101,
    "type": "missing_access_key",
    "info": "You have not supplied an API Access Key. [Required format: access_key=YOUR_ACCESS_KEY]"
  }
}

I didn't find in the toturial any information about the access_key for this API, how this could be?

Hi Community, I am trying to make a connector. I am facing one issue. Upon testing in our Airbyte server, we are getting the following error :

errors: $.type: is missing but it is required
2022-02-17 11:25:10 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: {}
2022-02-17 11:25:14 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $.type: is missing but it is required
2022-02-17 11:25:14 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: {}
2022-02-17 11:25:18 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $.type: is missing but it is required
2022-02-17 11:25:18 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: {}
2022-02-17 11:25:22 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $.type: is missing but it is required

I have attached the logs for reference. I have already validated the schema file and it is a valid json. Also there is no field as test. Please have a look,

Hi community, I am trying to setup a Python HTTP Source and I need to pass a Basic Authentication with username and password. I tried to use the HttpBasicAuth from requests.auth module as suggested but still does not get any permission authentication in the GET request. Here is the Source class definition

class SourcePythonHttpTest(AbstractSource):
    def check_connection(self, logger, config) -> Tuple[bool, any]:
        return True, None

    def streams(self, config: Mapping[str, Any]) -> List[Stream]:
        auth = HTTPBasicAuth(username=config["username"], password="password")
        return [DerivedStream(authenticator=auth)]

Should I use any airbyte module for that?

If I set a connector version to

latest

, does Airbyte always check for a new

latest

image before each sync?

Hello 🙂 I’l trying to run

/test connector=connectors/source-chargebee

in this PR but I keep on getting an error in

:airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles

(see details on screenshot). I pulled lates changes from

master

but still getting this error. Here is the link to failed check. Also I have few checks failing on that PR: • https://github.com/airbytehq/airbyte/runs/5245257875?check_suite_focus=true

Execution failed for task ':airbyte-cdk:python:generateProtocolClassFiles'.
3727
> Process 'command 'bin/generate-protocol-files.sh'' finished with non-zero exit value 1

• https://github.com/airbytehq/airbyte/runs/5245259204?check_suite_focus=true

Execution failed for task ':buildDockerImage-reporter'.
1312
> Could not build image: ADD failed: file not found in build context or excluded by .dockerignore: stat bin/airbyte-metrics-reporter-0.35.30-alpha.tar: file does not exist

• https://github.com/airbytehq/airbyte/runs/5245259013?check_suite_focus=true

Execution failed for task ':buildDockerImage-reporter'.
940
> Could not build image: ADD failed: file not found in build context or excluded by .dockerignore: stat bin/airbyte-metrics-reporter-0.35.30-alpha.tar: file does not exist

• https://github.com/airbytehq/airbyte/runs/5245262023?check_suite_focus=true

E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 3660 (unattended-upgr)
29
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

• https://github.com/airbytehq/airbyte/runs/5245262222?check_suite_focus=true

Execution failed for task ':buildDockerImage-reporter'.
933
> Could not build image: ADD failed: file not found in build context or excluded by .dockerignore: stat bin/airbyte-metrics-reporter-0.35.30-alpha.tar: file does not exist

Will I be able to test my new source connector in the UI once it is added to source_definitions.yaml?

Hello 👋 Does anyone know when we can expect Google Cloud Storage (GCS) connector as source? Thanks.

Hello experts, I'm trying to build a custom connector using python http api template that implements PUT http method. But, I always get a bad request 400 error. Can you tell which part I might be doing wrong? If there's any similar implementation, I'd love to read and do a test myself. Thanks.

Guys a quick question… it’s possible to build a connector just for myself? It’s a private application and doesn’t makes much sense to share with the community. I just want to build the connector and keep using the latest versions of airbyte.

Does anyone know if there's work being done on an Azure Synapse connector with datalake gen2 staging? Been looking around, haven't seen any.

Hi Guys, I actually develop solutions for smart buildings focused on energy. Do you know if there are connector for example to get information from energy suppliers like Total Energy or EDF (french examples) ? Or do you know someone that could be interested for this kind of connector ?

Hey guys, is anyone working on a Pipefy connector?

Hello team, would Yahoo Finance price API be interesting enough to be added to the source connectors? I developed it today and I (should) have it ready, is it worth polishing and documenting for an "official" integration? I'm asking because it's quite small and the Yahoo Finance APIs are not documented to be explicitly publicly available (I think). It's based on the "Price" section of this SO Answer

I’m currently locally seeing around 1-3k rows/sec being replicated to a local csv file, is this in line with the performance you’d expect?

Hi there, I have. A few question around MongoDB connector. So I’m trying to use MongoDB connector to integrate with Amazon DocumentDB (it supports MongoDB compatibility), however Amazon only allows connection from within the same VPC to do that over TLS.  To accomplish this, I set up Airbyte dev instance on the same VPC but the connection still wasn’t going though.  After some research I found out that in order to connect, the default connection string needs a bit more information in addition to

tls=true

and the full connection string needs to have this format:

mongodb://{username}:{password}@{host}:{port}/{database}?tls=true&tlsAllowInvalidHostnames=true&tlsAllowInvalidCertificates=true

I’ve tried to test my theory by creating JDBC source, but in the end I could dockerize the container. Seems like the dependency on

airbyte/integration-base-java:dev

image is not permitted. So I guess question/request, is this possible to add to the existing MongoDB connector; and if yes, what would be the expected time frame for this? It would also be great to add ability to use SSH with this connecotr.

I am trying to update the mongodb-v2 connector to support connections to TLS enabled Amazon DocumentDB. I have a version working locally, but I am concerned by the design. I am installing the required CA certs into the connector's docker image. I think this means the certs will become stale with no way to refresh except rebuild the docker image. Is this a showstopper? If yes I have a more dynamic option that installs the certs at runtime, but this would require the connector to be able to connect to the internet. I have described the pros and cons in more detail on the issue page. https://github.com/airbytehq/airbyte/issues/10388 . Which option would be better, or is there an option I am missing? Please let me know if you need any more background on the issue.

Hey everyone, Can someone please let me know the steps to recompile airtbyte code after making changes to it. So here is the problem. There seems to be a bug in smartsheets connector where the data is scrambled. I think I did get access to the corrected source.py from (Nathan Nowack). I want to be able to use it in my airbyte deployment (deployed on a docker container). How do I make changes to the code and compile the application again? Can someone please point me in the right direction or tell me the steps to do it?

What is required for my custom destination to be able to support dbt transformations?

Hi there, does BigQuery connector still need dataEditor permission? Is dataViewer permission enough? I wonder why it needs the permission as it's only reading the metadata.

Hi there, there is a way to connect GCS as a source?

I see AirByte has a JavaScript CDK now, and I see async in the code. Does the JavaScript CDK support full async for streams? I also see it is on the Coming Soon list for the Python CDK.

Hey team, have seen issues with Shopify refunds being unrolled, any help on this guy would be much appreciated: https://github.com/airbytehq/airbyte/issues/7577

Hey is there a supported way to make additional API calls based on the data returned from one stream. Example stream returning a list of IDs to be queried for subform data.