https://linen.dev logo
Join Slack
Powered by
# help-connector-development
  • t
    I have a problem about config partitioning. Acording to the api request, I need the value of id which is parent id inside the body of the request. Do you have any idea to do this? Please help me
    👀 1
    c
    s
    t
    • 4
    • 4
  • t
    can I use a list of id from a parent stream for value list in Partition router? I need an answer for this, please
  • t
    can I use a list of id from a parent stream for value list in Partition router? I need an answer for this, please
    k
    • 2
    • 2
  • t
    how can I set the id from using Substream to request's body data
    👀 1
    k
    • 2
    • 2
  • t
    how to add a parent_stream to YAML on airbyte
    👀 1
    k
    • 2
    • 2
  • c
    I want to remove some items in the UI of AirByte web. How can i do that?
    k
    • 2
    • 2
  • m
    Hi all, I am trying to develop a custom connector for the Missive api but for some reason I can't get pagination to work. The Missive Api docs say: "To paginate, pass an 
    until
    param equal to the 
    last_activity_at
    of the oldest conversation returned in the previous page. The last page is reached when fewer conversations than 
    limit
    are returned." This is my outgoing request.
    Copy code
    {
  "url": "<https://public.missiveapp.com/v1/conversations>",
  "parameters": {
    "until": [
      "1646310992"
    ],
    "team_inbox": [
      "6f1ddba3-***"
    ],
    "limit": [
      "50"
    ]
  },
  "headers": {
    "User-Agent": "python-requests/2.31.0",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Authorization": "Bearer ****"
  },
  "http_method": "GET"
}
    Attached a screenshot of the Cursor Pagination. I get the same result of just the first 50 records using
    Copy code
    response['data']['items'][-1]['last_activity_at'] }}
    Any hint what I am doing wrong here? Api References: https://missiveapp.com/help/api-documentation/rest-endpoints#list-conversations
    s
    • 2
    • 3
  • s
    This message was deleted.
    k
    • 2
    • 2
  • c
    Copy code
    airbyte-proxy                     | nginx version: nginx/1.25.0
airbyte-proxy                     | nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (2: No such file or directory)
airbyte-proxy                     | 2023/06/12 08:48:09 [emerg] 13#13: host not found in upstream "airbyte-webapp" in /etc/nginx/nginx.conf:17
airbyte-proxy exited with code 1
airbyte-webapp                    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
airbyte-webapp                    | /docker-entrypoint.sh: Configuration complete; ready for start up
airbyte-webapp                    | nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (2: No such file or directory)
airbyte-webapp                    | 2023/06/12 08:48:57 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (2: No such file or directory)
airbyte-webapp exited with code 1
    k
    • 2
    • 3
  • s
    Hello everyone, I'm new to Airbyte and I've encountered an issue with my custom connector for a sample API. I followed the "Building a HTTP source with the CDK" guide, but I'm facing a problem when attempting to retrieve data into a local JSON file as the destination. Specifically, the 
    _airbyte_data
    field is empty, despite verifying the JSON schema in the API's response. I would greatly appreciate any suggestions or troubleshooting methods to resolve this issue. Thank you in advance for your assistance.
    k
    • 2
    • 2
  • t
    Hi! in snapchat-marketing I want get data from basic_stats schema but I don't see any stream that implement it.
    k
    • 2
    • 2
  • m
    is there any way to adhere to rate limits by an api? The api I am trying to integrate has the following rate limits: - 5 concurrent requests at any given time - 300 requests per minute (5 requests per second) - 900 requests per 15 minutes (1 request per second) So far i only was able to add a backoff strategy but that doesn't help me not running into the rate limits
    k
    • 2
    • 3
  • c
    I need to import the yaml from the outside into AirByte Builder, besides the import UI, is there any way to import the yaml file? like using an API or something
    k
    • 2
    • 3
  • n
    Does Airbyte have some sort of async mechanism to check connection status after creation? I'm looking to implement this whenever the connection takes too much time and there is a need to retry it automatically. @kapa.ai
    k
    • 2
    • 2
  • o
    🔥 Community Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1pm PDT click here to join us on Zoom!
  • t
    I need a little help with the Iceberg connector, When I Build the connector it cant find the class: import io.airbyte.integrations.base.AirbyteStreamNameNamespacePair;
    k
    i
    • 3
    • 5
  • l
    Hi. I was checking the Airbyte API documentation, and I noticed that there is no payload example to create a 
    source-kafka
    source. What are the requirements to enable a source in the Airbyte API?
    k
    • 2
    • 5
  • t
    Heya team, I'm building a connector for an api that pulls different energy production data (Netanders) for the Dutch market. The api produces volume (in mwh) per 15 min segments of the data and uses a date filter with a before and after date (2023-06-13 is captured by before=2023-06-13 and after=2023-06-12) and only allows for a full date of data. I want to incrementally add rows for yesterday but struggling with the incremental config. My idea is to inject the before/ after date dynamically into the url with end date set to today (as date) and use a step window of P1D. This however produces the same start/ end date in the test string. Any clues?
    👀 1
    k
    ø
    • 3
    • 5
  • o
    🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom!
  • j
    Hey everyone, I'm trying to get Airbyte deployed in GCP virtual machine as per this guide here and have reached the final step where I run this command 
    gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME -- -L 8000:localhost:8000 -N -f
    , but am faced with these errors every time: bind [:1]8000: Cannot assign requested address _channel_setup_fwd_listener_tcpip: cannot listen to port: 8000_ Could not request local forwarding. I've tried other ports with no luck... Has anyone dealt with this before? Any ideas? Thanks.
    k
    • 2
    • 3
  • a
    Hi, we are looking to ingest a bunch of files from GCS which are 
    ^
    -delimited and compressed with GZIP. I notice that the current GCS connector only processes CSV files. I was wondering what the roadmap is for GCS to support a similar level of functionality as the S3 connector?
    k
    • 2
    • 2
  • f
    Hello! When I am trying to bring up a S3 destination connector via UI, the progress bar gets stuck after the process is completed (or so it seems). It shows a message around the lines "few more minutes" and those minutes never get exhausted. Seems like something goes wrong in the background. Is there a way I can see logs of this event? I am not sure where would I see the live or saved logs when connectors are being brought up via GUI. Please help!
    k
    • 2
    • 2
  • e
    HI all! I've written two custom HTTP connectors at this point as I knew they has specialized semantics that wouldn't be a fit for the low-code / no-code world. I'm now trying to add a new API that seemed like a better fit, but I'm running into limits (or lack of knowledge) of the low-code/no-code world and want to make sure I'm not missing something. 1. For pagination, the API returns a "next" token that is a full URL to the next page to fetch (e.g. "next": "https://api.foo.com/thing?page=2"). This "next link" is pretty common in APIs I've seen, but the cursor pagination seems to expect that the field is just a number, not a full blown link. Do the built-in paginators support this URL-based cursor format? 2. For objects, there are additional API calls to get more details about them. So I might do a 
    GET /api/foo
    to get a list of objects, but then 
    GET /api/foo/{id}
    will return more information about them (such as additional fields or modification history). In a custom HTTP connector I can easily chain slices to have a stream based off another, is there a way to do this in the low-code/no-code world? Thanks!
    👀 1
    k
    a
    • 3
    • 8
  • f
    Hi there - I am looking for deleting a custom connector built with the Builder UI but there is a little message saying "It's not possible to delete a builder connector after being published". Can't I delete my own custom connector, ever? Thanks!
    k
    • 2
    • 3
  • o
    🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1PM PDT click here to join us on Zoom!
  • y
    where can i find the list of definition ids for all the available sources and destinations , also the schema for that particular definitions
    k
    • 2
    • 3
  • y
    how to get the available apis in the airbyte
    k
    • 2
    • 2
  • y
    what are the available endpoints in the airbye
    k
    • 2
    • 1
  • y
    api/v1/source_definition_specifications/get is this a valid request in airbyte?
    k
    • 2
    • 2
  • m
    Hi all, We are trying out Quickbooks source connector which is in alpha stage and developed using low code cdk. It has integrated all the main streams in Quickbooks. Unfortunately, they missed the "change data capture" api in quickbooks which provides information on updated and deleted entities. change data capture api: /v3/company/<realmID>/cdc?entities=<entityList>&changedSince=<dateTime> invoice api: /v3/company/<realmID>/invoice Can you please help or advice on syncing the cdc as a separate table for each streams for incremental sync. example : invoice is a main stream, invoice_cdc is a cdc table
    k
    • 2
    • 3
1...131415...21Latest