Krzysztof
11/16/2022, 12:13 PMKrzysztof
11/16/2022, 12:13 PMKrzysztof
11/16/2022, 12:15 PMKrzysztof
11/16/2022, 12:15 PMCesar Santos
11/16/2022, 6:56 PMRachel RIZK
11/17/2022, 11:25 AMsecrets/config.json
exists and is at root of connector folder
• I build the image using docker build . -t airbyte/connector-name:dev
• Then I'm running the tests successfully with:
docker run --rm airbyte/connector-name:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/connector-name:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/connector-name:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/connector-name:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
• Then I run python -m pytest integration_tests -p integration_tests.acceptance
◦ Everything is fine until 50%, when I get to the check --config part: FAILED ../../bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection::test_check[inputs0] - docker.errors.ContainerError: Command 'check --config /data/tap_config.json'
◦ Then all following tests fail with FileNotFoundError: [Errno 2] No such file or directory: '/data/tap_config.json'
It looks like the secrets/config.json
file is not correctly detected. I've found someone had the exact same issue here but I don't have details regarding how it's been solved.
Has anyone else encountered this issue? 🙏Akash Ghadge
11/17/2022, 2:11 PMLenny C
11/21/2022, 1:03 AMpage_token_option:
field_name: "variables.offset"
inject_into: "body_json"
page_size_option:
field_name: "variables.limit"
inject_into: "body_json"
But it doesn't nest in the debug of the outbound API request ("variables.limit": 123)Andreas Nigg
11/25/2022, 2:26 PMSlackbot
11/26/2022, 4:24 PMSharon Lavie
11/26/2022, 8:18 PM{
"type": "RECORD",
"record": {
"stream": "campaigns",
"data": {
"campaignId": 1043888667,
"startTime": "2022-11-25",
"primary_key": "2a5ab6e9b247d79699392f1a6dd87150",
"metadata": {
"campaignId": 1043888667,
"campaignName": "TP_US_generic",
"deleted": false,
"campaignStatus": "ENABLED",
"app_appName": "Asana Rebel: Get in Shape",
"app_adamId": 1067860796,
"servingStatus": "RUNNING",
"servingStateReasons": null,
"countriesOrRegions": [
"US"
],
"modificationTime": "2022-05-02T14:25:17.001",
"totalBudget_amount": "100000",
"totalBudget_currency": "EUR",
"dailyBudget_amount": "250",
"dailyBudget_currency": "EUR",
"displayStatus": "RUNNING",
"supplySources": [
"APPSTORE_SEARCH_RESULTS"
],
"adChannelType": "SEARCH",
"orgId": 51370,
"billingEvent": "TAPS",
"countryOrRegion": "US"
},
"metrics": {
"impressions": 25,
"taps": 0,
"installs": 0,
"newDownloads": 0,
"redownloads": 0,
"latOnInstalls": 0,
"latOffInstalls": 0,
"ttr": 0,
"avgCPA_amount": "0",
"avgCPA_currency": "EUR",
"avgCPT_amount": "0",
"avgCPT_currency": "EUR",
"avgCPM_amount": "0",
"avgCPM_currency": "EUR",
"localSpend_amount": "0",
"localSpend_currency": "EUR",
"conversionRate": 0,
"date": "2022-11-25"
}
},
"emitted_at": 1669492309596
}
}
As you can see i have metadata
and metrics
as objects and they are mapped in the schema.
In the main table: campaigns
there are not duplicates, but! in the 2 extra tables made for the 2 objects (metadata
and metrics
) there are duplicates
Could someone please give me a hint here and direct me to where there problem lies?
this is my schema:
{
"$schema": "<http://json-schema.org/draft-04/schema#>",
"type": "object",
"properties": {
"campaignId": {
"type": "integer"
},
"startTime": {
"type": "string"
},
"primary_key": {
"type": "string"
},
"metadata": {
"type": "object",
"properties": {
"campaignId": {
"type": "integer"
},
"campaignName": {
"type": "string"
},
"deleted": {
"type": "boolean"
},
"campaignStatus": {
"type": "string"
},
"app_appName": {
"type": "string"
},
"app_adamId": {
"type": "integer"
},
"servingStatus": {
"type": "string"
},
"servingStateReasons": {
"type": "null"
},
"countriesOrRegions": {
"type": "array",
"items": [
{
"type": "string"
}
]
},
"modificationTime": {
"type": "string"
},
"totalBudget_amount": {
"type": "string"
},
"totalBudget_currency": {
"type": "string"
},
"dailyBudget_amount": {
"type": "string"
},
"dailyBudget_currency": {
"type": "string"
},
"displayStatus": {
"type": "string"
},
"supplySources": {
"type": "array",
"items": [
{
"type": "string"
}
]
},
"adChannelType": {
"type": "string"
},
"orgId": {
"type": "integer"
},
"billingEvent": {
"type": "string"
},
"countryOrRegion": {
"type": "string"
}
}
},
"metrics": {
"type": "object",
"properties": {
"impressions": {
"type": "integer"
},
"taps": {
"type": "integer"
},
"installs": {
"type": "integer"
},
"newDownloads": {
"type": "integer"
},
"redownloads": {
"type": "integer"
},
"latOnInstalls": {
"type": "integer"
},
"latOffInstalls": {
"type": "integer"
},
"ttr": {
"type": "number"
},
"avgCPA_amount": {
"type": "string"
},
"avgCPA_currency": {
"type": "string"
},
"avgCPT_amount": {
"type": "string"
},
"avgCPT_currency": {
"type": "string"
},
"avgCPM_amount": {
"type": "string"
},
"avgCPM_currency": {
"type": "string"
},
"localSpend_amount": {
"type": "string"
},
"localSpend_currency": {
"type": "string"
},
"conversionRate": {
"type": "number"
},
"date": {
"type": "string"
}
}
}
}
}
Rachel RIZK
11/28/2022, 9:21 AMMariam Thiam
11/30/2022, 9:41 AMlaila ribke
12/01/2022, 3:12 PMlaila ribke
12/01/2022, 8:46 PMSam Stoelinga
12/02/2022, 5:45 PMSam Stoelinga
12/02/2022, 6:19 PMSam Stoelinga
12/05/2022, 8:08 PMmyuser/destination-xyz
. What's the easiest way to e.g. test loading data from mongoDB and pushing it to destination-xyz?Sam Stoelinga
12/05/2022, 8:35 PMSam Stoelinga
12/09/2022, 6:14 PMSam Stoelinga
12/10/2022, 6:03 AMSam Stoelinga
12/13/2022, 11:14 PMSam Stoelinga
12/14/2022, 5:42 PMEmma Forman Ling
12/16/2022, 11:37 PMSUB_BUILD=PLATFORM ./gradlew build
and getting:
Execution failed for task ':airbyte-webapp:npmInstall'.
> Process 'command '/Users/emmaling/Projects/airbyte/airbyte-webapp/.gradle/nodejs/node-v16.15.1-darwin-arm64/bin/npm'' finished with non-zero exit value 1
with python 3.9.12, java 17.0.5, and node 16.15.1.
anyone seen this or have any advice for debugging?laila ribke
12/19/2022, 8:45 AMlaila ribke
12/20/2022, 3:45 AMlaila ribke
12/20/2022, 3:50 AMlaila ribke
12/20/2022, 3:24 PMlaila ribke
12/21/2022, 12:46 AMstream_slicer:
$ref: "*ref(definitions.stream_slicer)"
I´ll attach the yaml file.
Another issue is that I need to set limit to the maximum, which is 200 records and I don´t know where to put it.
And last question, is how to set the pagination. I receive an offset parameter with the starting record number.
https://api-docs.transferwise.com/#transfers-listlaila ribke
12/21/2022, 12:56 PM