https://linen.dev logo
Join Slack
Powered by
# help-connector-development
  • a
    Hello All, trying to build a python source connector following these instructions: https://docs.airbyte.com/connector-development/tutorials/building-a-python-source/ When I run ./generate.sh, it just opens the file. What am I missing?
    d
    g
    • 3
    • 2
  • s
    hello All, trying to build a python destination connector in windows environment by following instructions https://docs.airbyte.com/connector-development/tutorials/building-a-python-destination. When I run ./generate.sh in windows power shell, it doing some process in git bash and exiting but it's not asking for selecting Python Destination
  • s
    am I missing something here
    c
    • 2
    • 4
  • c
    I'm sure I'm missing something simple here. I'm writing my first connector. It's a python HTTP API source connector. I'm in the source.py streams method, trying to pass in the api token. When I write my code normally, I use (in my header) 'Authorization': 'my_api_token'. In my airbyte connector code, I try to use the auth = TokenAuthenticator(token=api_token), it returns a 401 because the default is "Bearer". This makes the header become: "Authorization": "Bearer my_api_token". I have tried changing the auth_method to be "token", "Bearer", and "access_token". All fail with the 401 error. I tried to use auth_method="". but then the base code converts it to "Authorization" : " my_api_token" and I get an error that my token can't begin with a space. Anyone know how I can get it to just send "Authorization":"my_token"?
    m
    a
    • 3
    • 2
  • c
    Same connector as above (a python HTTP API source connector). I have an API the able to grab the 1st stream okay. But what I need is to take each record of the first stream and use the id for a 2nd stream. Can anyone provide me direction on how to do that?
    m
    a
    • 3
    • 4
  • s
    hey all, i have been trying to access this link "https://docs.airbyte.com/connector-development/tutorials/cdk-tutorial-python-http/getting-started for the past two days, it says the link is broken. Any help or update?
    a
    a
    • 3
    • 4
  • n
    Hi all, I have built a private connector for an API (internal use only) by cloning the airbyte repo locally and following the instructions in the CDK tutorials (thanks, these are very clear). The connector is running well locally but I now want to add that connector to my production version of airbyte which is running on AWS EC2. Are there any instructions/advice on how to do this?
    m
    a
    • 3
    • 3
  • g
    Has anybody had success using the CDK for a SOAP api? I’m looking to help a client implment this
    Kareo_IntegrationAPI_TechnicalGuide.pdf
    j
    • 2
    • 1
  • m
    Hi all. Wondering if anyone has built a source connector for Dynamodb.
    c
    • 2
    • 2
  • a
    Hi All just wondering has anyone ever built a source connector for Xero ?
    m
    • 2
    • 1
  • r
    Hello team, I am new to airbyte and we are planning to use airbyte to export data from postgres table to s3 bucket. I wanted to know how can I integrate airbyte with spring boot application. Any help is appreciated!
    m
    • 2
    • 1
  • g
    Hey, for #C0450PCSDB2 I’m trying to use the Low Code CDK for an API that has OAuth 2.0 authentication. I’ve looked in the docs and there’s a link that promises info on this setup, but the page doesn’t exists yet. I’ve looked at examples and pending PRs that might show that off, but didn’t see anything super relevant, everything is using BearerAuth or similar (as the CDK template is generated as well). Is there any more info on this? Or for interactive OAuth 2.0 one should still use the Python SDK?
    j
    a
    • 3
    • 4
  • n
    Hola, Is there a way to use the low-code CDK with an API that returns XML data? I’ve seen the JSON requirement within the docs but this is the only thing that my API is missing. It’d be great if there was a flag or something to make it work with XML data.
    u
    a
    • 3
    • 7
  • j
    I'm trying to do the cdk tutorial on a Windows machine, which is proving to be a challenge. (Had to figure out wsl2, had to figure out how to get docker up on wsl2, just so I could run 
    generate.sh
    . ) Now I'm back in the Windows environment and I'm at the 
    pip install -r requirements.txt
    step, but I get this:
    Copy code
    ERROR: Could not find a version that satisfies the requirement pywin32==227; sys_platform == "win32" (from docker) (from versions: 302, 303, 304)
ERROR: No matching distribution found for pywin32==227; sys_platform == "win32"
    • should I not try to continue the steps in Windows? • Is there going to be better support for developing on Windows in the future? • Any advice on the error Thanks!
    m
    s
    s
    • 4
    • 7
  • a
    Hi, I hope everyone doing fine. I'm working on connector development for source which return compressed(tar.gz) file which have multiple .csv file as a response. Does anyone know how we can develop connector in this scenario.
    s
    • 2
    • 5
  • s
    Hello everyone! I’m trying to use the low-code CDK for EmailOctopus, but have been getting 403 errors with simple requests — for debugging, how do I print the API request that’s generated by
    Copy code
    python main.py check --config secrets/config.json
    ? EmailOctopus ’s API docs say, “If you’re making a JSON request, include a 
    Content-Type: application/json
    header.”     so I want to check what the actual requests being sent look like… 😄
    s
    d
    • 3
    • 4
  • h
    Hello Everyone
  • h
    How can i find APIs for the contest
    m
    • 2
    • 1
  • s
    Hi Team, I build destination connector for teradata. When I tried to use Normalized tabular data option in Transformation phase, getting exception "io.airbyte.workers.exception.WorkerException: Normalization Failed" 
    Caused by: io.airbyte.workers.exception.WorkerException: Normalization Failed.
    at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:93) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
    at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:27) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
    ... 1 more
    2022-10-31 05:02:04 [42mnormalization[0m > usage: transform-config --config CONFIG --integration-type
    2022-10-31 05:02:04 [42mnormalization[0m >                         {DestinationType.BIGQUERY,DestinationType.CLICKHOUSE,DestinationType.MSSQL,DestinationType.MYSQL,DestinationType.ORACLE,DestinationType.POSTGRES,DestinationType.REDSHIFT,DestinationType.SNOWFLAKE,DestinationType.TIDB}
    2022-10-31 05:02:04 [42mnormalization[0m >                         --out OUT
    2022-10-31 05:02:04 [42mnormalization[0m > transform-config: error: argument --integration-type: invalid DestinationType value: 'teradata'
    2022-10-31 05:02:04 [42mnormalization[0m > Traceback (most recent call last):
    2022-10-31 05:02:04 [42mnormalization[0m >   File "/usr/local/bin/transform-catalog", line 8, in <module>
    2022-10-31 05:02:04 [42mnormalization[0m >     sys.exit(main())
    2022-10-31 05:02:04 [42mnormalization[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py", line 104, in  main
    2022-10-31 05:02:04 [42mnormalization[0m >     TransformCatalog().run(args)
    2022-10-31 05:02:04 [42mnormalization[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py", line 35, in run
    2022-10-31 05:02:04 [42mnormalization[0m >     self.parse(args)
    2022-10-31 05:02:04 [42mnormalization[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py", line 46, in parse
    2022-10-31 05:02:04 [42mnormalization[0m >     profiles_yml = read_profiles_yml(parsed_args.profile_config_dir)
    2022-10-31 05:02:04 [42mnormalization[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py", line 75, in read_profiles_yml
    2022-10-31 05:02:04 [42mnormalization[0m >     with open(os.path.join(profile_dir, "profiles.yml"), "r") as file:
    2022-10-31 05:02:04 [42mnormalization[0m > FileNotFoundError: [Errno 2] No such file or directory: '/data/21/2/normalize/profiles.yml'
    Can anyone help to resolve this issue.
  • s
    What is the process to add support of Basic Normalization to Java Custom Destination connector?
    m
    r
    • 3
    • 4
  • h
    👋 Is there an easy way to validate incoming data with the stream's json schema when building a destination using the python cdk? Can Airbyte be instructed to drop a record or key that isn't part of the json schema? Or is that logic that needs to be implemented in the destination, when desired?
    m
    • 2
    • 4
  • m
    Hello. Building a destination connector, how can I know which type of normalization was configured? (Raw/Basic normalization) Well, it seems I need to study more. I have this error
    Copy code
    Requested normalization for marlon/mydestconn:v1.0.0, but it is not included in the normalization mappings.
    Can anyone point me to the right direction to support normalization in my destination connector?
  • r
    Hello everyone 👋 I am trying to use a 
    field_pointer
    with DpathExtractor, but my data is nested inside a list like this:
    Copy code
    testdata = { 'data': [
                    {'actual_data': { 'lotsofdata' },
                    {'metadata': 2},
                    {'null': 3},
                ],
               }
    If I set 
    field_pointer: ["data"]
    it works fine but I actually only want the value of 
    "actual_data"
    , which is inside a list. I've tried 
    field_pointer: ["data[0]", "actual_data"]
    but it doesn't work, and it won't allow any string interpolation. Is there a way to retrieve data within a list with DpathExtractor? 😅 Thanks for your help 🙏
    s
    • 2
    • 4
  • j
    Just wondering, I want to update an existing source connector to include more streams - is it okay to migrate from the Python CDK to low code one if possible since it’s easier for me to work with or would you prefer to leave it as a Python CDK?
    a
    • 2
    • 2
  • r
    Hello! Me again 🙂 I am trying to find a way to 
    increment - append
    data instead of 
    full refresh - overwrite/append
    with the low-code CDK. Here's how the data is structured:
    Copy code
    testdata = { 
  'rows': [
    {
      'data_row1': [{ 'kpi': 2, 'date': "2022-11-07" }],
      'metadata_row1': {'id': 1}
    },
    {
      'data_row2': [{ 'kpi': 8, 'date': "2022-11-07" }],
      'metadata_row2': {'id': 2}
    },
  ]
}
    • To get all rows, I need to set 
    record_selector
    on 
    rows
    field • But to get the date field (enabling increment with slicing) I need to set the 
    cursor_field
    to something like that: 
    ["data_row1", "0", "date"]
    or 
    data_row1[0]['date']
    . ◦ With 1st solution, I have an error because it won't accept a list ◦ With 2nd solution, the script runs but the cursor is not working and displays older data than what's in 
    sample_state.json
    I'm wondering, is there actually a way to activate increment on this kind of data? It's the last step before I can contribute for a new connector 🙏
    ✅ 1
    b
    • 2
    • 8
  • a
    Hi team, I need to access a field for pagination from the json response. I tried using body but it outputs to showing that it is not defined.
    s
    m
    • 3
    • 4
  • s
    Hi team, does anyone facing the below issue while building connector/airbyte platform
    Copy code
    2022-11-10 14:49:33 INFO o.t.i.RemoteDockerImage(resolve):75 - Pulling docker image: postgres:13-alpine. Please be patient; this may take some time but only needs to be done once.
2022-11-10 14:49:34 ERROR c.g.d.a.a.ResultCallbackTemplate(onError):52 - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: <https://www.docker.com/increase-rate-limit%22}|https://www.docker.com/increase-rate-limit"}>
    Copy code
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247) ~[testcontainers-1.17.3.jar:?]
        at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269) ~[testcontainers-1.17.3.jar:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2022-11-10 14:49:34 WARN o.t.i.RemoteDockerImage(resolve):105 - Retrying pull for image: postgres:13-alpine (119s remaining)
2022-11-10 14:49:35 ERROR c.g.d.a.a.ResultCallbackTemplate(onError):52 - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: <https://www.docker.com/increase-rate-limit%22}|https://www.docker.com/increase-rate-limit"}>
    Copy code
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247) ~[testcontainers-1.17.3.jar:?]
        at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269) ~[testcontainers-1.17.3.jar:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2022-11-10 14:49:35 WARN o.t.i.RemoteDockerImage(resolve):105 - Retrying pull for image: postgres:13-alpine (118s remaining)
    Copy code
    > Task :airbyte-api:compileJava
Note: /root/airbyte/airbyte-api/build/generated/api/client/src/main/java/io/airbyte/api/client/invoker/generated/JSON.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
    Copy code
    > Task :airbyte-db:jooq:generateConfigsDatabaseJooq
2022-11-10 14:49:35 ERROR c.g.d.a.a.ResultCallbackTemplate(onError):52 - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: <https://www.docker.com/increase-rate-limit%22}|https://www.docker.com/increase-rate-limit"}>
  • r
    Hello 👋 On my final tests (low-code connector built, imported in an Airbyte instance, sync with increment and dedup OK, normalization OK), I realised the 
    lookback_window
    was not working as expected for incremental syncs 😬 • it's only set on the original 
    start_date
    but not in the 
    cursor_field
    used to do the increment • so it's never doing any lookback after the first sync • after some research, I've stumbled upon this issue that was open 3mo ago Since it's a marketing-related connector (with an attribution window) it's a bit painful because we can't use Incremental syncs in that case. By any chance, has anyone encountered a similar issue and found a potential workaround?...
    m
    s
    • 3
    • 4
  • m
    Hello. I've been working for these past few months on a CDK for Go, trying to make it easy and efficient to use. If there are any Golang devs, please feel free to review it and use it. I'd like any feedback. If you want to contribute, I really appreciate it. This is the repo for the CDK, called Kankuro. I think I still need to understand Airbyte's protocol a little bit better. But at least it's a start With this CDK, I created a Mongo destination connector. I compared it with the one is available right now and it seems the one that I built it's faster. But most likely it's because the one in Mongo is still in alpha Right now I'm working ona CLI to make the CDK really to use And finally, I want to thank Saif Abid. All this work was inspired on his work
    🔥 2
    s
    • 2
    • 2
  • k
    Hi guys, i am pleased to meet you.
12345...21Latest