https://linen.dev logo
Join Slack
Powered by
# ask-community-for-troubleshooting
  • a
    Hi team šŸ™‚ I'm using Airbyte open-source, and I'm trying to import a csv file from s3 to Redshift. I currently have a problem that I can't figure out: one of my fields is a date in the CSV and I would like to ingest it as it in my Redshift table. But I can't find the right way to do it šŸ˜•, any help would be appreciated! And example of the input file:
    Copy code
    ID,USER_ID,ORDER_DATE,STATUS
1,1,2018-01-01,returned
2,3,2018-01-02,completed
3,94,2018-01-04,completed
4,50,2018-01-05,completed
5,64,2018-01-05,completed
    n
    • 2
    • 3
  • p
    Hi Folks, ā€¢ is it possible to upvote connector work somehow? ā€¢ is it possible to sponsor connector work somehow? Background: weā€™d love to use Airbyte, but our main DB is a MongoDB and the connector is not yet stable enough. We played around with it. Thanks for the info / link / help!
    u
    a
    • 3
    • 4
  • c
    we're running 4 syncs every couple minutes, so our airbyte_workspace volume is getting filled with a bunch of metadata that we don't need. Is there an env var or config option for how many syncs' worth of data to keep?
    āœ… 1
    a
    • 2
    • 5
  • p
    Hi team - thanks in advance for your suggestions on this issue. Problem: Airbyte has stopped logs for a given sync, and is in some sort of hung state. Environment: Airbyte self-hosted on GC, standard docker container Background: I have successfully synced a dozen <100MB Postgres tables on this instance (to BigQuery). I attempted syncing a 350GB table. The process ran for many hours, then stopped logging. I have confirmed that the OS clock is accurate, so it seems like Airbyte stopped logging about 2 days ago, on Jan 17. Last few lines of logs, posting in thread below (I can attach entire log file if helpful)
    a
    n
    u
    • 4
    • 14
  • m
    Hi!octavia wave Iā€™m new so Iā€™m sorry if there are any rules to follow for writing here. I have been searching for quite a while but I havenā€™t found anything regarding my problem. I am doing some testing on getting data from Mixpanel to an S3 bucket. Itā€™s working fine but Iā€™m having the following problem. If I use 
    Root level flattening
    the 
    insert_id
    field (which is the unique _id for the event) disappears. Is there any way to create the 
    insert_id
    as a column as well? Thanks in Advance pepe happy
    āž• 2
  • r
    Hi Guys, how are you? Someone here is using the Twitter Source Connector? It's working?
    s
    u
    • 3
    • 7
  • j
    Wondering how hard it is to adjust a source like the Google Sheets source. Our sheets have 1 or more ā€œpretty rowsā€, and then a header row as expected. Iā€™d like to add the ability to say ā€œconsider row 3 the headerā€
  • m
    Hi everyone, I am testing connection between postresql and yugabyte. I am able to establish connection between Postresql as source and Yugabyte as destination but replication yo Yugabyte only creates _raw table storing data in json format but unable to create the main table. Any help really appreciated and let me know if I need to share sync logs.
    a
    • 2
    • 2
  • i
    Hi, I'm getting this error on airbyte star, afer add a destination
  • i
    Copy code
    airbyte-server                    | 2023-01-19 18:26:13 ERROR i.a.s.e.UncaughtExceptionHandler(handle):27 - Uncaught exception
airbyte-server                    | java.lang.IllegalStateException: Duplicate key af2c2d70-8e9c-44e2-bc65-b0f00b5a30da (attempted merging values io.airbyte.config.ActorCatalogFetchEvent@cec51bd[id=<null>,actorId=af2c2d70-8e9c-44e2-bc65-b0f00b5a30da,actorCatalogId=b202e206-de55-4dfd-9e3c-56f39b0a97ad,configHash=<null>,connectorVersion=<null>,createdAt=1674151525] and io.airbyte.config.ActorCatalogFetchEvent@77389684[id=<null>,actorId=af2c2d70-8e9c-44e2-bc65-b0f00b5a30da,actorCatalogId=b202e206-de55-4dfd-9e3c-56f39b0a97ad,configHash=<null>,connectorVersion=<null>,createdAt=1674151525])
  • i
    which makes imposible to open sources or connections, tried to do a docker system prune, but still same error
  • i
    anyway to restart everything?
  • s
    Can someone point me to where the script is to generate a configured catalog? I found it once but it was almost by happenstance.
    u
    • 2
    • 2
  • m
    Hello everyone, Has anyone had the following happen: I am trying to export the configuration of my airbyte to look at it in another installation, I am doing this through the api with this endpoint curl -u -H ā€œContent-Type: application/jsonā€ -X POST localhost:8000/api/v1/deployment/export and i am getting this error message {ā€œmessageā€:ā€œObject not found. HTTP 404 Not Foundā€,ā€œexceptionClassNameā€ā€œio.airbyte.server.errors.IdNotFoundKnownExceptionā€,ā€œexceptionStackā€[ā€œio.airbyte.server.errors.IdNotFoundKnownException: Object not found. Does anyone know what it is and how I can correct it? I really appreciate your help, I have already searched the internet for the error but I canā€™t find any information
    n
    a
    • 3
    • 4
  • c
    is the /v1/jobs/list the right endpoint if I want to check if a connection is currently syncing?
    s
    • 2
    • 4
  • b
    Hi everyone, I am creating a POC. My source is openweather and the destination is MSSQL. Iā€™ve created a dbt project with custom normalization based on the normalization that airbyte did in default sync. The normalization scripts sit on the synced data in mssql and it works perfectly when I run it locally. My problem is that Iā€™d like to use it from airbyte, but it says that ā€œNo module named ā€˜dbt.adapters.sqlserverā€™. I assume that the dbt image should contain the required packages for sql server because that was needed for dbt to run locally too. 1. pic error 2. pic setup of the transformation 3. pic transformation steps Could you please give me some advice on that? Should I add the necessary packages to the dbt image dockerfile or what Iā€™m missing?
    a
    • 2
    • 3
  • g
    Is anyone aware of issues in Airbyte >=0.40.26 on Kubernetes (GKE) preventing the airbyte-worker from launching new pods?
    n
    u
    l
    • 4
    • 8
  • m
    Hey there. I need to look at conversions by keyword on the Google Ads connector. I see the display_keyword_performance_report; this seems to only contain "display" type ads. I need to also see SEARCH type ads. The Keyword report does not have a conversions metric. Am I missing something? Is there a way to add a custom report? Thank you!
    u
    • 2
    • 2
  • s
    I want Airbyte to use 100GB of my RAM out of 124GB, how do I set it up in the .env file?
    s
    • 2
    • 1
  • s
    Please help
    u
    • 2
    • 1
  • m
    hello everyone I am trying to configure a s3 destination, but when i test the destination it return the error ā€œnon-json responseā€, somebody know why happen it this error. I already had a destination s3 configured in the same airbyte and it works correctly. Thanks a lot for your help!
    n
    • 2
    • 3
  • p
    Does Airbyte support event driven connectors? according to this article: https://airbyte.com/blog/airbyte-strategy-to-commoditize-all-data-integration itā€™s on the roadmap. Would love to see an example if itā€™s available
  • s
    This slack group is not really useful in my own view
    āž• 2
    u
    • 2
    • 1
  • j
    https://docs.airbyte.com/deploying-airbyte/on-aws-ec2/ I'm having some difficulty following the directions here to connect to an Airbyte container running on EC2. 1. "_If you want to use different ports, modify 
    API_URL
    in your .env file and restart Airbyte_" This does not appear to be correct, in the airbyte folder's .env file freshly installed, 
    API_URL=/api/v1/
    u
    • 2
    • 1
  • j
    2.
    Copy code
    ssh -i $SSH_KEY -L 8000:localhost:8000 -N -f ec2-user@$INSTANCE_IP
    which port is local and which is remote? do they have to be equal? e.g. can I run port 8000 on EC2 but map it to port 8888 locally?
    • 1
    • 1
  • n
    Hello everyone. Does Airbyte support automatically pulling docker images (for custom connectors) from remote locations? say GCP Artifact Registry. According to the documentation I should manually pull the image before trying to create the connector, but I was wondering if this could be done automatically.
    n
    • 2
    • 1
  • w
    Trying to deploy a python source connector that uses pyodbc. I am getting the following error: Error loading shared library libodbc.so.2: No such file or directory (needed by /usr/local/lib/python3.9/site-packages/pyodbc.cpython-39-x86_64-linux-gnu.so) I believe this file is a part of the unixodbc package which I have as a line in the Dockerfile: RUN apk --no-cache add unixodbc-dev unixodbc Is there something I am missing here?
    n
    • 2
    • 1
  • s
    I messed something up during an AirByte upgrade and am now getting the following output from temporal. Is there a way to resolve this, or do I need to reinstall AirByte altogether?
    Untitled
    u
    • 2
    • 4
  • a
    Does anyone know how I can find the directory for custom sources created within the temporal docker container?
    n
    u
    • 3
    • 6
  • u
    Hi everyone I have tried Airbyte from MySQL to Snowflake transformation. Data successfully synced to destination even Full Sync and CDC. But VARCHAR that has collation UTF8_bin columns transform to base64 string. I know that is correct work. I want to transform VARCHAR(collation UTF8_bin) to String or VARCHAR. But can i change the mapping type or another way? My English is really bad. sry :(
    n
    u
    • 3
    • 3
1...125126127...245Latest