https://linen.dev logo
Join Slack
Powered by
# help-connector-development
  • k
    i managed to create my first airbyte connector
  • k
    it works as expected
  • k
    however i am having some minor technical problems with deployment, i have published docker image with connector to my private mexus and i dont know how to configure airbyte to connect to nexus and use it
    m
    • 2
    • 19
  • k
    thanks in advance for any advice
  • c
    Hey folks! Quick question, is it possible to write an airbyte destination for Hubspot API using the CDK?
  • r
    Hi everyone 👋 I am preparing a PR for a new connector using the low-code CDK, but I'm stuck at the acceptance tests: • 
    secrets/config.json
    exists and is at root of connector folder • I build the image using 
    docker build . -t airbyte/connector-name:dev
    • Then I'm running the tests successfully with:
    Copy code
    docker run --rm airbyte/connector-name:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/connector-name:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/connector-name:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/connector-name:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
    • Then I run 
    python -m pytest integration_tests -p integration_tests.acceptance
    ◦ Everything is fine until 50%, when I get to the check --config part: 
    FAILED ../../bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection::test_check[inputs0] - docker.errors.ContainerError: Command 'check --config /data/tap_config.json'
    ◦ Then all following tests fail with 
    FileNotFoundError: [Errno 2] No such file or directory: '/data/tap_config.json'
    It looks like the 
    secrets/config.json
    file is not correctly detected. I've found someone had the exact same issue here but I don't have details regarding how it's been solved. Has anyone else encountered this issue? 🙏
    t
    • 2
    • 3
  • a
    Hi All, I am trying to create a custom connector using ./generate.sh command but when I execute the command I am getting an error. I am not sure how to resolve this error. Thank you
    m
    • 2
    • 2
  • l
    Hi there, For the lowcode CDK: I managed to be able to query a GraphQL endpoint, however I need to use GraphQL variables for pagination purposes. Is there a way to define and pass nested json? {"variables": {"limit": 123, "offset": 456}} I tried to do this in
    Copy code
    page_token_option:
  field_name: "variables.offset"
  inject_into: "body_json"
page_size_option:
  field_name: "variables.limit"
  inject_into: "body_json"
    But it doesn't nest in the debug of the outbound API request ("variables.limit": 123)
    c
    • 2
    • 4
  • a
    Hi there. What's the process of adding an icon to a new connector? Can I simply add a Pull-Request and place an svg-icon in "airbyte-config/init/src/main/resources/icons/" and add the icon location in the source_specs.yaml?
    s
    • 2
    • 1
  • s
    This message was deleted.
    m
    • 2
    • 1
  • s
    Hi all, I created my own connector for apple search ads reports, everything works well in the main table and the data arrives correctly, but! since i have 2 objects returned in the JSON and my schema is mapping them, I have 2 extra tables (that I could ignore, but still want to makes things correctly and understand the way airbyte works) my record looks like that:
    Copy code
    {
  "type": "RECORD",
  "record": {
    "stream": "campaigns",
    "data": {
      "campaignId": 1043888667,
      "startTime": "2022-11-25",
      "primary_key": "2a5ab6e9b247d79699392f1a6dd87150",
      "metadata": {
        "campaignId": 1043888667,
        "campaignName": "TP_US_generic",
        "deleted": false,
        "campaignStatus": "ENABLED",
        "app_appName": "Asana Rebel: Get in Shape",
        "app_adamId": 1067860796,
        "servingStatus": "RUNNING",
        "servingStateReasons": null,
        "countriesOrRegions": [
          "US"
        ],
        "modificationTime": "2022-05-02T14:25:17.001",
        "totalBudget_amount": "100000",
        "totalBudget_currency": "EUR",
        "dailyBudget_amount": "250",
        "dailyBudget_currency": "EUR",
        "displayStatus": "RUNNING",
        "supplySources": [
          "APPSTORE_SEARCH_RESULTS"
        ],
        "adChannelType": "SEARCH",
        "orgId": 51370,
        "billingEvent": "TAPS",
        "countryOrRegion": "US"
      },
      "metrics": {
        "impressions": 25,
        "taps": 0,
        "installs": 0,
        "newDownloads": 0,
        "redownloads": 0,
        "latOnInstalls": 0,
        "latOffInstalls": 0,
        "ttr": 0,
        "avgCPA_amount": "0",
        "avgCPA_currency": "EUR",
        "avgCPT_amount": "0",
        "avgCPT_currency": "EUR",
        "avgCPM_amount": "0",
        "avgCPM_currency": "EUR",
        "localSpend_amount": "0",
        "localSpend_currency": "EUR",
        "conversionRate": 0,
        "date": "2022-11-25"
      }
    },
    "emitted_at": 1669492309596
  }
}
    As you can see i have 
    metadata
    and 
    metrics
    as objects and they are mapped in the schema. In the main table: 
    campaigns
    there are not duplicates, but! in the 2 extra tables made for the 2 objects (
    metadata
    and 
    metrics
    ) there are duplicates     Could someone please give me a hint here and direct me to where there problem lies? this is my schema:
    Copy code
    {
  "$schema": "<http://json-schema.org/draft-04/schema#>",
  "type": "object",
  "properties": {
    "campaignId": {
      "type": "integer"
    },
    "startTime": {
      "type": "string"
    },
    "primary_key": {
      "type": "string"
    },
    "metadata": {
      "type": "object",
      "properties": {
        "campaignId": {
          "type": "integer"
        },
        "campaignName": {
          "type": "string"
        },
        "deleted": {
          "type": "boolean"
        },
        "campaignStatus": {
          "type": "string"
        },
        "app_appName": {
          "type": "string"
        },
        "app_adamId": {
          "type": "integer"
        },
        "servingStatus": {
          "type": "string"
        },
        "servingStateReasons": {
          "type": "null"
        },
        "countriesOrRegions": {
          "type": "array",
          "items": [
            {
              "type": "string"
            }
          ]
        },
        "modificationTime": {
          "type": "string"
        },
        "totalBudget_amount": {
          "type": "string"
        },
        "totalBudget_currency": {
          "type": "string"
        },
        "dailyBudget_amount": {
          "type": "string"
        },
        "dailyBudget_currency": {
          "type": "string"
        },
        "displayStatus": {
          "type": "string"
        },
        "supplySources": {
          "type": "array",
          "items": [
            {
              "type": "string"
            }
          ]
        },
        "adChannelType": {
          "type": "string"
        },
        "orgId": {
          "type": "integer"
        },
        "billingEvent": {
          "type": "string"
        },
        "countryOrRegion": {
          "type": "string"
        }
      }
    },
    "metrics": {
      "type": "object",
      "properties": {
        "impressions": {
          "type": "integer"
        },
        "taps": {
          "type": "integer"
        },
        "installs": {
          "type": "integer"
        },
        "newDownloads": {
          "type": "integer"
        },
        "redownloads": {
          "type": "integer"
        },
        "latOnInstalls": {
          "type": "integer"
        },
        "latOffInstalls": {
          "type": "integer"
        },
        "ttr": {
          "type": "number"
        },
        "avgCPA_amount": {
          "type": "string"
        },
        "avgCPA_currency": {
          "type": "string"
        },
        "avgCPT_amount": {
          "type": "string"
        },
        "avgCPT_currency": {
          "type": "string"
        },
        "avgCPM_amount": {
          "type": "string"
        },
        "avgCPM_currency": {
          "type": "string"
        },
        "localSpend_amount": {
          "type": "string"
        },
        "localSpend_currency": {
          "type": "string"
        },
        "conversionRate": {
          "type": "number"
        },
        "date": {
          "type": "string"
        }
      }
    }
  }
}
    s
    e
    • 3
    • 5
  • r
    Hello! A few weeks ago I've opened a PR for a new Source: Apple Search Ads, using the low-code CDK. Is anyone familiar with the low-code CDK available to review it please? 🙏
    a
    m
    • 3
    • 6
  • m
    Hello, I have opened this PR on Airbyte for Source Amplitude to resolve issue #19605 to enable data server region selection. Can someone give a review on it ? 🙏 Thanks
    m
    • 2
    • 9
  • l
    Hello everyone, I'm new in this channel. I need to add one field to a schema in google ads source connector and one to bing ads source connector. Can someone help me on that?
    j
    • 2
    • 2
  • l
    Hi all, I´m really lost. I cloned the airbyte git repo locally. There I duplicated the source-google-ads folder and the source-bing-ads folder. Called them source-google-ads-new and source-bing-ads-new. Inside each one of them, I added a field to the schema of one stream. Now, what should I do to upload those new connectors for my use? In this tutorial https://docs.airbyte.com/connector-development/tutorials/cdk-speedrun/ I see these steps because actually, I´m not touching any other definition in the source connectors. how should I proceed?? It must sound dumb.. but I´m really lost here
    a
    • 2
    • 8
  • s
    Writing a destination connector and wondering if there is a way to have stream specific configuration? It seems like config is at the connector level for all streams. Some more context, the destination I have has a vector field that requires being set via special API call, however I don't know which field of stream contains the vector data. So I would like a per stream configuration that tells me which field contains the vector. Is this possible?
  • s
    Is it fine to launch a docker container with the database during integration tests? I plan to add this to the python integration test so it can be run with a real database
    m
    • 2
    • 2
  • s
    I want to test my connector and have published on docker.io as 
    myuser/destination-xyz
    . What's the easiest way to e.g. test loading data from mongoDB and pushing it to destination-xyz?
    • 1
    • 1
  • s
    Did the connector integration test actually get run? https://github.com/airbytehq/airbyte/pull/20094#issuecomment-1338117954
  • s
    I'm ready for a review for a new destination: Weaviate: https://github.com/airbytehq/airbyte/pull/20094 I was able to test the destination end-to-end as well locally
    m
    • 2
    • 1
  • s
    I wrote a new connector using Python however I just heard that I would still need to write acceptance tests in Java. The docs did mention this: https://docs.airbyte.com/connector-development/tutorials/building-a-python-destination/#step-6-set-up-acceptance-tests So how soon is very soon for python acceptance tests? I rather not use Java for this since everything is in Python right now.
    m
    • 2
    • 2
  • s
  • s
    did this github action break due to deprecation of a github api endpoint? if so is someone merging a fix into master? https://github.com/airbytehq/airbyte/actions/runs/3697308626/jobs/6262129523#step:4:90
    • 1
    • 2
  • e
    hey y’all! I’m having trouble getting the platform to compile. I’m running 
    SUB_BUILD=PLATFORM ./gradlew build
    and getting:
    Copy code
    Execution failed for task ':airbyte-webapp:npmInstall'.
> Process 'command '/Users/emmaling/Projects/airbyte/airbyte-webapp/.gradle/nodejs/node-v16.15.1-darwin-arm64/bin/npm'' finished with non-zero exit value 1
    with python 3.9.12, java 17.0.5, and node 16.15.1. anyone seen this or have any advice for debugging?
    • 1
    • 1
  • l
    Hi all, I have the same issue when running ./generate.sh command as this git issue #9810. I followed their solution but still getting same error message Should I run it in bash? Any ideas what might be the issue?
  • l
    Hi, Found my error, in case it helps anyone. I went to localhost 8000 and in username typed airbyte and password: password and only then it loaded the airbyte connection http://localhost:8000/workspaces/d2955c90-fe17-45c4-805c-469fcea87aa3/connections. Then I ran again the ./generate.sh and it worked
  • l
    I´m stuck defining my connector. A WISE API. I manage to create a GET req via postman, but I´m not defining it right in the connector development yml file
    s
    a
    • 3
    • 11
  • l
    Hi all, another question.. as the PR reviews are going slowly due to holidays, and the modifications I need to do to the google ads and bing ads sources, what is the best way to manage it? Should I copy them so there would be one mine and the original one or should I just modify the existing one? and after I modify it, how I load it to my client´s airbyte open source?
    s
    • 2
    • 7
  • l
    Hi all again.. I created a WISE (transfer wise) source connector with the sdk. I experience problems while trying to set the stream_slicer. If I add the following and run it, it just loops until I stop it, although there are only 5 records.
    Copy code
    stream_slicer:
      $ref: "*ref(definitions.stream_slicer)"
    I´ll attach the yaml file. Another issue is that I need to set limit to the maximum, which is 200 records and I don´t know where to put it. And last question, is how to set the pagination. I receive an offset parameter with the starting record number. https://api-docs.transferwise.com/#transfers-list
    wise
  • l
    Hi all, thanks to your support I managed to set my custom sources in Airbyte!! Thanks! A question: I think that my custom source is a version before I modified it. In my local environment is is ok (no limit filter) but the one in airbyte filters with limit=20. Hoy can I modify it to be the latest? I uploaded a new one with other tag, but it created a new source and it behaves the same. So I need to delete one custom source and to see hoy to load to the one that stays, the correct limit
12345...21Latest