https://linen.dev logo
Join Slack
Powered by
# ask-community-for-troubleshooting
  • g
    Hey Airbyte <!subteam^S01M74AUQSE>! Is there a way to import configuration from a lower version (
    0.19.0-alpha
    to 
    0.26.1-alpha
    , to be precise)? I read on your docs that
    This step will throw an exception if the data you are trying to upload does not match the version of Airbyte that is running.
    However, I wanted to check if there is a way around, for example downgrading, importing, and then upgrading etc. Cheers!
    c
    • 2
    • 12
  • m
    Good afternoon! We are making advances, but we are stuck now with
    Copy code
    POST /v1/connections/reset
    We are obtaining an error about the 'configId'
    Copy code
    data: {
      message: 'Could not find configuration for STANDARD_SYNC_OPERATION: [uuid]'
}
    Where can I retrieve the proper id? We also receive a similar error on sync-connection query. Thanks in advance
    u
    • 2
    • 61
  • j
    hi people, please what is the recommended volume size to run airbyte on a ec2 instance?
    m
    u
    • 3
    • 8
  • f
    Hello team ! I just configured Airbyte to copy data from a PostgreSQL to another one. In my destination PostgreSQL database I want the data to be copy in a specific schema, So I have used the "Default Schema" parameters... but the data has been copy in the dbo schema. Any idea ?
    ✅ 1
    u
    • 2
    • 6
  • j
    Good morning! Has anyone run into trouble using nginx as a reverse-proxy for airbyte? I am able to get to airbyte if I bypass nginx just fine but when I try to access via nginx I am seeing 
    400 Bad Request
    errors. The configuration is fairly simple in that I am `proxy_pass`'ing everything to http://localhost:8000
    u
    • 2
    • 3
  • t
    Hi, are there any configuration api tutorials available somewhere?
    u
    • 2
    • 3
  • s
    Hi team, I’m looking to make API requests on behalf of users (capturing their third party API keys in a form, then using them in Airbyte connectors). Does anyone recommend a way to integrate with a key vault / secrets manager, for this use case? The equivalent out of the box in Fivetran would be their “powered by Fivetran” product. I’d rather build that myself and use Airbyte though. Thanks!
    u
    • 2
    • 2
  • m
    Good afternoon. Has anyone front this problem at 
    /v1/web_backend/connections/list
    query on 0.26.2-alpha version? 
    data: { message: 'Could not find configuration for STANDARD_SYNC_OPERATION: c2c7578b-f37b-45b5-b803-9e87c2f53f36.',                  exceptionClassName: 'io.airbyte.server.errors.IdNotFoundKnownException', exceptionStack: [Array], rootCauseExceptionClassName: 'java.lang.Class', rootCauseExceptionStack: [Array] }
    u
    • 2
    • 5
  • n
    Hi Team, I was able to successfully set up airbyte on my local machine. Just curious when we deploy this to EC2 instance how can we setup user authentication for airbyte (self hosted )
    d
    o
    j
    • 4
    • 12
  • t
    Hi all, I’m learning Airbyte and I’m trying to connect bigquery destination but so far no luck. I’m wondering how I should put the Service Account Key into 
    connectionConfiguration
    . More specifically this part:
    Copy code
    "credentials_json": "{I don't know what to put here}"
    I’m consistently getting 
    Invalid Json
    or 
    The provided configuration does not fulfill the specification
    . Should I just copy&paste all of the service account key json contents? I’m using the config API. Thanks 🙂 I referenced this comment (https://airbytehq.slack.com/archives/C01A4CAP81L/p1611632587033400?thread_ts=1611630873.029800&amp;cid=C01A4CAP81L) but I still have the same issue 😞
    ✅ 1
    d
    • 2
    • 5
  • t
    Hi all, I’m learning Airbyte config API and creating a new source with http request. I tried to create a source with below
    Copy code
    {
  "sourceDefinitionId": "9845d17a-45f1-4070-8a60-50914b1c8e2b",
  "sourceId": "my sourceId",
  "workspaceId": "my workspaceId",
  "connectionConfiguration": {
    "url": "<https://api.data.metro.tokyo.lg.jp/v1/Covid19Patient?limit=1>",
    "http_method": "GET"
  },
  "name": "string",
  "sourceName": "HTTP Request"
}
    But I am getting this error:
    Copy code
    {
  "status": "failed",
  "message": "<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n  <title>Error 400 (Bad Request)!!1</title>\n  <style>\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n  </style>\n  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n  <p><b>400.</b> <ins>That's an error.</ins>\n  <p>Your client has issued a malformed or illegal request.  <ins>That's all we know.</ins>\n",
  "jobInfo": {...
    "logs": {
      "logLines": [
        "2021-06-28 01:07:55 INFO () TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.26.4-alpha",
        "2021-06-28 01:07:55 INFO () LineGobbler(voidCall):85 - Checking if airbyte/source-http-request:0.2.4 exists...",
        "2021-06-28 01:07:55 INFO () LineGobbler(voidCall):85 - airbyte/source-http-request:0.2.4 was found locally.",
...        "2021-06-28 01:07:57 ERROR () DefaultAirbyteStreamFactory(internalLog):108 - Check failed",
        "2021-06-28 01:07:57 INFO () TemporalAttemptExecution(get):133 - Stopping cancellation check scheduling..."
      ]
    }
  }
}
    I have no problem getting response with curl. Thank you!🙏
    d
    u
    • 3
    • 4
  • t
    Log for `connections/sync`:
    Copy code
    2021-06-28 23:34:45 INFO  REQ 172.18.0.6 GET 200 /api/v1/health
2021-06-28 23:35:19 INFO  REQ 172.18.0.6 OPTIONS 200 /api/v1/connections/sync
2021-06-28 23:35:19 INFO  enqueuing pending job for scope: 5ca84ed3-72d0-4c40-bfc1-55ebef37e72c
2021-06-28 23:35:19 INFO  REQ 172.18.0.6 POST 200 /api/v1/connections/sync - {"connectionId":"5ca84ed3-72d0-4c40-bfc1-55ebef37e72c"}
2021-06-28 23:35:33 INFO  REQ 172.18.0.6 OPTIONS 200 /api/v1/logs/get
2021-06-28 23:35:33 INFO  WORKER_ENVIRONMENT not found, defaulting to DOCKER
    ✅ 1
    u
    • 2
    • 2
  • d
    hello! New to airbyte, great tool btw. I do have a question. I have setup a connection from a online csv to a local postgres. I am unsure how to change the data type and field names of the source before it hits the destination?
    u
    c
    c
    • 4
    • 11
  • q
    Hi guys, I want to connect to my google cloud postgres instance, but have had no luck so far, i think I would need to whitelist airbyte's IP addresses to allow connections, can anyone help me with this?
    ✅ 1
    c
    d
    • 3
    • 36
  • j
    Hi, we're building a SaaS platform for automotive logistics. As part of the daily routines we need a way to • import files in misc formats, e.g. XML, csv or industry-standard formats, such as EDI (EDIFACT, ODETTE, X12) • perform some field/value mappings • Send the result as a JSON payload to our Web API for processing. A key feature is being able to know when processing of a file has failed and having a mechanism to either retry or cancel the import Do you think Airbyte could be a good fit for what we are trying to do? My areas of concern are not having connectors for every format we want to support. Is there a way to import parsers that we could implement ourselves when we miss a format, e.g. XML? Would it be feasible to do the data mapping/transforms in a easy way for non-developers? I have not seen http as possible destination, is this something that you would plan to do? if not, could we add a plugin so we can perform those http requests? thank you!
    ✅ 1
    c
    d
    • 3
    • 8
  • s
    #C021JANJ6TY #C01AHCD885S # Hi Guys !!! New members to the airbyte . I am trying to find a document to transfer data to Google Cloud Bucket as my destination, but haven't had any luck yet . As I don't see GSC bucket as destination in connector list , Could you please direct me to right direction. Appreciated in advance. Thank you :)
    c
    l
    u
    • 4
    • 11
  • q
    Hi folks, new on airbyte and tried tweaking some configurations within the connection settings, but how do I make sure that only one normalized table that matches with the original table format, gets dumped in bigquery instead of the airbyte raw data table coming along with it?
    ✅ 1
    d
    c
    • 3
    • 13
  • r
    Hi everyone ,new on airbyte and i am using airbyte as EL part and for transformation i am using DBT .......i took postgress as source and destination mysql after that i get json blob ........now our aim to transform json format into table struture so can anyone tell me here how to connect dbt with airbyte
    c
    • 2
    • 1
  • n
    Is there documentation for the CLI-Tool?
    c
    • 2
    • 1
  • m
    Hi guys - got a couple of specific requirements which have disrupted our ticketing system integrations; we've been looking at Airbyte for a while and this might be the "argh" moment which pushes us to standing it up. Can we confirm whether it can: • Translate HTTP content types from source to destination - one of our ticketing systems cannot support GZIP compression, our vendor has turned it on and won't turn it off - need something to translate to application/json • Translate from modern to less secure cipher suites - one of our legacy systems can only support older cipher suites, our vendor has disabled support for them (good) but we still need them to be able to talk to our legacy system while we migrate from it
    c
    • 2
    • 5
  • u
    hello everyone, I discovered airbyte to update my local data from aws server. I don’t know exactly where to start. What I want is a comparison of my local data with aws database. then I want my local data to be updated with aws only the changed data. I need your help, it would be great if there is a document to start with. And sorry for my bad english.
    u
    • 2
    • 1
  • m
    Dear Team, How I can install airbyte using portrainer. Please let me know. Note: I have installed using command line successfully.
    ✅ 1
    d
    j
    • 3
    • 4
  • y
    Hello Guys, I would like to check if someone use airbyte with mongodb as source? What's the limitation to sync data between mongodb and snowflake?
    ✅ 1
    d
    • 2
    • 2
  • a
    Hey Guys, trying to install airbyte via docker on Mac (M1 chip) using the instructions on: https://docs.airbyte.io/quickstart/deploy-airbyte. Cant get it to work. Can see airbyte starting on the terminal but the front end page wither says "cannot reach server. The server may still be starting up." and turns blank after a while. But in the docker cli for the server it says "docker-compose-wait - Everything's fine, the application can now start!"
    ✅ 1
    d
    • 2
    • 3
  • m
    Good morning, mates. My job now is to be able to see the local-csv files content set by the users as Destination. Have I got to raise a dedicated API to cover this? I saw the uploaded files, but I didn't find any solution for this at your docs. Thank you
    ✅ 1
    c
    a
    • 3
    • 7
  • q
    I've currently deployed an airbyte instance on gcp after using it locally for a while, and am setting up my connectors again, host.docker.internal isn't working anymore when im setting up a postgres instance, is there any way to restore all the pipelines/sources/destinations I had in my local deployment?
    a
    c
    • 3
    • 12
  • m
    Hi, is it possible to have an arbitrary SQL query as a data source? I want to move a subset of a MySQL database (i.e. only certain tables and columns) to s3 and/or redshift
    ✅ 1
    c
    • 2
    • 2
  • a
    Hello Everyone
    c
    • 2
    • 1
  • a
    Anyone can provide me Airbyte Tutorials. I am new in Airbyte.
    c
    • 2
    • 2
  • e
    Hi guys, I need help to verify my facebook app and be able to use Facebook Marketing as a data source in airbyte. Do you have any advice?
    d
    u
    • 3
    • 3
12345...245Latest