Damien Busch
04/10/2024, 8:37 AMstatus
& the type
from the status container
{
"status": 200,
"body": {
"tasks": [
{
"id": "86by328yq",
"custom_id": null,
"custom_item_id": 0,
"name": "Ou avoir un filtre dans la recherche avancé",
"text_content": "",
"description": "",
"status": {
"status": "to do",
"color": "#87909e",
"type": "open",
"orderindex": 0
Is there a way for the transformation step to extract 2 fields and put them in 2 different columns ?
It looks like the 1st transfo is avoiding the 2nd one to work properly, and the original response have disappearedChristy Bolland
04/10/2024, 3:45 PM"status": 200,
"body": {
"reviews": [
{
"subscriberId": "f44fe56a-5af5-402f-8d17-2ab3fe436a07",
"reviewId": "305d5b42-3066-4c61-8c93-dc6799baa6ed",
"productId": "shopify-7533586710723",
"attributesWithRating": [
{
"minLabel": "Poor",
"title": "Quality",
"type": "range",
"value": 5,
"maxLabel": "Excellent"
},
{
"minLabel": "Runs Small",
"midLabel": "Just Right",
"title": "How It Fits",
"type": "centered-range",
"value": 0,
"maxLabel": "Runs Large"
}
],
"body": "I was looking for an extra long insulated coat as I'm tall; 6ft1. Ideally I was looking for waterproof also but it's virtually impossible to find. This coat is such a great option. It's shower proof and sooooo cosy. I actually can't believe how warm it is. It's a nice fit too as I tried another coat that was a bit bulky and oversized. It's a great length; just around mid- lower calf length. Would definitely recommend. ",
"containsProfanity": false,
"dateCreated": "2024-04-10T14:44:24.661Z",
"helpfulCount": 0,
"isRecommended": true,
"order": {
"orderId": "5542043746499",
"orderNumber": "1156777"
},
Any help on how to access the whole record using record selector without the body field or to build the record selector from response level?Fernando Lopez
04/10/2024, 8:56 PMairbyte_source_custom
but that doesn't seem to be what we need at all. I was expecting to automate CI process for connector build and deploy to our self-hosted airbyte instance on EC2.Arnaud Selva
04/11/2024, 10:19 AMAziza Alkuatova
04/11/2024, 10:46 AMYepher
04/11/2024, 7:19 PMpage_content
that references a "parent stream" spaces
. The spaces parent stream has a structure like this:
{
:
"pages": [
{
:
"pages": [
...
]
}
]
}
This is the Gitbook API I am trying to sync:
https://developer.gitbook.com/gitbook-api/reference/spaces/content
See <https://api.gitbook.com/v1/spaces/{spaceId}/content>
I want the "page_content" to get called for each page
of the parent. The pages array nests pages that can have pages array for any number of levels.
I basically want to sync the markdown of all pages and nested pages.
original question here https://airbytehq.slack.com/archives/C027KKE4BCZ/p1712674001234469Aziza Alkuatova
04/11/2024, 7:54 PMGergely Imreh
04/12/2024, 7:51 AMGergely Imreh
04/12/2024, 8:48 AMJose Ignacio Fernandez
04/12/2024, 8:13 PM“options”:{“cursor”:”value”}
), which is unsupported right now (tried using options.cursor
with no luck)
since the Python CDK is being phased out, what’s the timeline for enabling nested cursors to support apis like Plaid?Jack Hulbert
04/12/2024, 9:36 PMparent
). What's the best way to acheive this?
I've been attempting to use transformations here, but have not had any success.Gavin Wong
04/15/2024, 10:11 AM/services/api/Recruiting/JobApplicant?CurrentStatus={Status}&StatusDate={Date}
this works perfectly fine when i'm using Parameterized Requests for the Status
and Date
- where I hardcode a list of Parameter Values via a Value List
however - i'd like to automate the Date
to extract only the previous day's data. for this i've used the User Input and a code value of {{ day_delta(1) }}
this runs, however, the request resolves to:
"url": "https://url.com/services/api/Recruiting/JobApplicant?PageSize=100&StatusDate=2&CurrentStatus=New+Submission",
where the StatusDate
is truncated to 2
(not the full date)
i'm using airbyte cloud - so wondering if anyone has also come up against problems with User Input values?Francisco Carneiro
04/15/2024, 1:00 PMmanifest.yaml
file:
version: "0.29.0"
definitions:
selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path: []
requester:
type: HttpRequester
url_base: "<https://customapi.goldcast.io>"
http_method: "GET"
authenticator:
type: ApiKeyAuthenticator
header: "Authorization"
api_token: "Token {{ config['access_key'] }}"
retriever:
type: SimpleRetriever
record_selector:
$ref: "#/definitions/selector"
paginator:
type: NoPagination
requester:
$ref: "#/definitions/requester"
base_stream:
type: DeclarativeStream
retriever:
$ref: "#/definitions/retriever"
events_stream:
$ref: "#/definitions/base_stream"
name: "events"
description: "Events stream. Events are the base object of Goldcast."
primary_key: "id"
$parameters:
path: "/event/"
webinars_stream:
$ref: "#/definitions/base_stream"
name: "webinars"
description: "Webinars stream. Webinares are one of the objects that compose an event."
primary_key: "id"
parent_stream_configs:
- type: ParentStreamConfig
stream: "#/definitions/events_stream"
parent_key: "id"
partition_field: "id"
$parameters:
name: "webinars"
primary_key: "id"
path: "/event/webinars/{{ stream_slice.id }}"
transformations:
- type: AddFields
fields:
- path: ["event_id"]
value: "'{{ stream_slice.id }}'"
retriever:
$ref: "#/definitions/retriever"
record_selector:
$ref: "#/definitions/retriever/record_selector"
extractor:
field_path: ["id", "id"]
requester:
$ref: "#/definitions/requester"
error_handler:
type: DefaultErrorHandler
description: "405 - cMethod \"GET\" not allowed."
response_filters:
- http_codes: [405]
action: IGNORE
streams:
- "#/definitions/events_stream"
check:
type: CheckStream
stream_names:
- "events"
spec:
documentation_url: <https://docs.airbyte.com/integrations/sources/exchangeratesapi>
connection_specification:
$schema: <http://json-schema.org/draft-07/schema#>
title: goldcast.io Source Spec
type: object
required:
- access_key
additionalProperties: true
properties:
access_key:
type: string
description: >-
Your API Access Key. See <a
href="<https://help.goldcast.io/hc/en-us/articles/22931655725723-How-To-Create-an-API-Token-in-Goldcast>">here</a>. The key is
case sensitive.
airbyte_secret: true
As I hope you can understand from my yaml file 😄, for each record of the events stream, I want to make a call to the webinars stream to the path "/event/webinars/{{ stream_slice.id }}"
where {{ stream_slice.id }}
is the id of the event (primary key of the events stream). Could you please tell me what is wrong / missing in my file, currently the connector tries to call the endpoint "/event/webinars/"
which is not allowed. I am guessing it because the placeholder {{ stream_slice.id }}
is not correct ?
Many thanks in advance for your help 🙏Jack Hulbert
04/15/2024, 4:24 PMAbhijith Selvakumar
04/18/2024, 8:31 AMBill Lunney
04/18/2024, 9:16 AMQuentin Moret
04/18/2024, 9:47 AMDiscovering schema failed
common.error
There is no issue when testing in the connector builder, nor when testing the source, everything works well there. The error message does not give more information so I’m really stuck...
Any idea ? Thanks !Siddhant Singh
04/18/2024, 12:52 PMWilliam Guicheney
04/18/2024, 5:31 PMMatheus Dantas
04/19/2024, 7:55 AMBernardo Cardozo
04/19/2024, 2:25 PM"body": {
"call": "ListarMovimentos",
"app_key": "x",
"app_secret": "x",
"nRegPorPagina": 1,
"nPagina": 5
}
Is ther a way to get this into the "param" field inside the request body?Antonio Grass
04/19/2024, 8:09 PMAmit Bose
04/19/2024, 10:42 PMMichal Krawczyk
04/22/2024, 8:58 AMincremental_sync
yaml node should be used in a Full Refresh mode? I'd expect that when the Full Refresh mode is used the incremental_sync
yaml node is just ignored and the stream would just fetch all data using the defined pagination strategy without using incremental parameters. Aparently this seems to not to be the case and from what I observed the Full Refresh mode just doesn't record the final stream state. That said, is there a way to implement an incremental stream (let's say with updated_at
cursor), but initially when a stream is created run a full refresh of all objects regardless when they were last updated?Christian Schmid
04/23/2024, 10:39 AMMateo James
04/24/2024, 10:19 AMAmit Bose
04/24/2024, 5:01 PMJustin Lemmon
04/25/2024, 4:16 AMairbyte-ci connectors --name=source-quickbooks build
to test some connector updates and I'm getting the error:
failed to start daemon: Error initializing network controller: error obtaining controller instance: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain: (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.10 (nf_tables): RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1
Any thoughts? I've already tried the suggestions of sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
that I've seen in a few threads. Running Airbyte on Docker via WSL2 + Ubuntu.Yeongbae Jeon
04/25/2024, 6:17 AMYeongbae Jeon
04/25/2024, 6:25 AM