https://linen.dev logo
Join SlackCommunities
Powered by
# ask-community-for-troubleshooting
  • m
    Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.
  • r
    Hello. I am trying to understand how airbyte deal with memory. Every time a conector reads a lot records, I need to increase the vm memory and cpu. I found the JOB_MAIN_CONTAINER_MEMORY_REQUEST and JOB_MAIN_CONTAINER_MEMORY_LIMIT. But Id like to understand how this works from source to dest. Example, if I set request and memory to 2gb but the first sync is 10gb of rows, how it will be handled? Rows goes to disk? And when airbyte goes to send this to dest, it will be batches of 2gb, or something like that? I really appreciate any link to docs, discussion or article that answers things like that. 🙂
    • 1
    • 1
  • j
    so i'm just reading through https://docs.airbyte.com/understanding-airbyte/beginners-guide-to-catalog/ and trying to understand how i generate a sample file for the configured airbyte catalog. is that something i'd have to put together manually using jq or is there a script somewhere or something else i'm not seeing?
    🙏 1
  • p
    Hello Is it possible to add custom connector while deploying airbyte using kubernetes / helm charts?
    j
    a
    • 3
    • 10
  • b
    Trying to use an external DB, but the docs still mention an outdated ticket that is Complete Do we have a ETA on when the KB will be updated? Resources: https://docs.airbyte.com/deploying-airbyte/on-kubernetes/#using-an-external-db https://github.com/airbytehq/airbyte/issues/3605 @Harvey Marshall
  • d
    I'm trying to get the metrics in Prometheus But the metrics aren't showing up I have followed this doc https://docs.airbyte.com/operator-guides/collecting-metrics/ I have deployed Airbyte in AWS Ec2 Should I have to deploy an 
    airbyte-metrics
    service ?... If YES Can anyone help in how to deploy it. docs.airbyte.com Collecting Metrics | Airbyte Documentation Airbyte supports two ways to collect metrics - using datadog or open telemetry.
  • d
    doesnt look like customer service is much help with this company.
  • j
    Actually customer support is mostly provided on Discourse: discuss.Airbyte.io
    ✅ 1
  • d
    your bot on your homepage sent me here
    airbyte heart 1
    j
    • 2
    • 1
  • d
    ill head there thanks
  • d
    Real quick, Im trying to add your source into our dashboards. We are building a full ERP to compete with sales force and Oracle. Is there any way to schedule a call with my team?
  • m
    Hello everybody. I’ve just realised that there are a few modifications and fixes related to the woocommerce connector, for example which were merged in with this PR: https://github.com/airbytehq/airbyte/pull/10529 Can I ask you guys to publish these changes to the docker hub? Thank you in advance.
  • j
    Hello, I’m creating a source connector for a schemaless database (Firestore). How is the 
    get_json_schema
    generally implemented in this case? Should I try to guess the schema from a few documents? It could be wrong. Or should I instead just encode everything to a string and leave it there? I thought also about asking for the schema in a connector input maybe?
  • u
    The latest pull the master branch, run 
    ./ gradlew airbyte - integrations: connectors: destination - mysql: build
     error 
    FAILURE: Build failed with an exception.
    * What went wrong:
    Execution failed for task ':airbyte-integrations:bases:base-normalization:flakeCheck'.
    > Python call failed: .venv/bin/python -m pflake8 --config /root/airbyte/pyproject.toml ./
    I need some help. Who can help me look into the problem
    🙏 1
    m
    • 2
    • 1
  • j
    Hey everyone, we have a multi tenant system where our users have already stored their OAuth access tokens for Facebook, Google etc in our database. Is there a way to dynamically create/destroy jobs in Airbyte from the API, and pass through access tokens? I’m not quite sure which terminology I should be hunting for. Thank you!
    m
    z
    • 3
    • 4
  • a
    Hey folks! If I already have developer/refresh tokens for a custom app (Google Ads, Bing Ads, Facebook Ads), would I be able to re-use the same credentials for an Airbyte connection? Otherwise, does anyone know if it is possible to have multiple developer/refresh tokens actively running in parallel?
    m
    j
    j
    • 4
    • 3
  • m
    Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.
  • z
    Hi guys, We found possible bug for amazon ads source connector. https://github.com/airbytehq/airbyte/issues/15571
  • d
    Trying to understand the Embed flow use case. We want to use Airbyte so that our customers will connect their platforms through Airbyte and the data will sync into our DB.
    plus1 1
  • t
    how to fix the storage full issue ??? when I use airbyte 3 4 months my VM storage is full. any solutions for that
  • e
    Hi. I`ve built a new source and followed step by step with this video :

    https://www.youtube.com/watch?v=kJ3hLoNfz_E▾

    and this doc: https://docs.airbyte.com/connector-development/tutorials/cdk-speedrun/ As described in the video and docs - I`ve passed all the tests and ran the docker build command and when I go to the UI to try to add the new connector I`m getting this error:
    Internal Server Error: Get Spec job failed.
    I`m not sure why I`m getting this error since I`ve passed all the tests i.e. the check, discover and read. These are the test commands: 
    py -m main check --config sample_files/config.json
    py -m main discover --config sample_files/config.json
    py -m  main read --config sample_files/config.json --catalog sample_files/configured_catalog.json
    This is the build command: 
    docker build . -t airbyte/source-my-new-source:dev
    What am I missing? Thank you.
    • 1
    • 1
  • p
    Can someone help me if there is any available example of Python Destnation custom connector?
    e
    • 2
    • 1
  • p
    @Eli Sigal You can try docker commands to debug..gives more detailed error messages
    e
    • 2
    • 1
  • k
    Hey folks, QQ: If I want to locally test a new http connector that I made and have the destination be the local json option, would I have to do anything else other than 
    docker compose up
    and spin up the local instance and test? Of do i have to do anything in addition to it ie mounting FS etc. Thanks!
  • g
    Hi all, im a begineer here and just deployed newest airbyte version 0.39.42-alpha on k8s. i found that select/unsellect all button disappeared in the UI. does anyone had same problem? i just want to unselect all first and then select few tables for test.
    ✅ 1
    • 1
    • 1
  • v
    Hello guys, what does it mean that error ?
    Copy code
    2022-08-15 08:04:15 - Additional Failure Information: message='activity timeout', timeoutType=TIMEOUT_TYPE_HEARTBEAT
    Source S3 bucket Destination Postgres Data - .csv files
  • k
    Hey Folks, Has anyone seen this error during a sync before?
    Copy code
    2022-08-14 23:40:48 destination > 2022-08-14 23:40:48 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
    Also it seems that the sync is stuck at and growing:
    Copy code
    2022-08-15 04:31:36 INFO i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 45000 (951 MB)
    Anyone have any ideas?
    👀 1
  • m
    Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.
  • m
    Hi team, is there a way to set sync frequency for a custom connector greater than 24 hours? For example 36 or 48 hours?
  • l
    Where does airbyte open source store credentials once they are entered in the UI?
1...596061...245Latest