Channels
j
Jared Rhizor (Airbyte)
04/02/2021, 8:51 PM
On the demo I upgraded to the new version of the exchange rate api and I’m running into a weird catalog configuration error.
u
user
04/02/2021, 8:54 PMI upgraded the version for the demo instance
u
user
04/02/2021, 8:54 PMAnd then re-ran with the postgres sync https://demo.airbyte.io/source/connection/09235736-d54d-4405-8362-46232c8a1ba3
u
user
04/02/2021, 8:54 PMEverything went perfectly first time
u
user
04/02/2021, 8:54 PMHowever, for the sync to bigquery https://demo.airbyte.io/destination/connection/3c633094-2ecf-4989-b1ad-a64e272c4cb2
u
user
04/02/2021, 8:55 PMI’m getting JSON decoding issues for the catalog.
u
user
04/02/2021, 8:55 PMI tried resetting data/rediscovering schemas/etc and couldn’t get anything to work yet
u
user
04/02/2021, 8:56 PMThis is the configured catalog at
/tmp/workspace/115/0/source_catalog.jsonu
user
04/02/2021, 8:56 PM Copy code
{
"streams": [
{
"stream": {
"name": "exchange_rate",
"json_schema": {
"type": "object",
"properties": {
"AUD": {
"type": [
"null",
"number"
]
},
"BGN": {
"type": [
"null",
"number"
]
},
"BRL": {
"type": [
"null",
"number"
]
},
"CAD": {
"type": [
"null",
"number"
]
},
"CHF": {
"type": [
"null",
"number"
]
},
"CNY": {
"type": [
"null",
"number"
]
},
"CZK": {
"type": [
"null",
"number"
]
},
"DKK": {
"type": [
"null",
"number"
]
},
"EUR": {
"type": [
"null",
"number"
]
},
"GBP": {
"type": [
"null",
"number"
]
},
"HKD": {
"type": [
"null",
"number"
]
},
"HRK": {
"type": [
"null",
"number"
]
},
"HUF": {
"type": [
"null",
"number"
]
},
"IDR": {
"type": [
"null",
"number"
]
},
"ILS": {
"type": [
"null",
"number"
]
},
"INR": {
"type": [
"null",
"number"
]
},
"ISK": {
"type": [
"null",
"number"
]
},
"JPY": {
"type": [
"null",
"number"
]
},
"KRW": {
"type": [
"null",
"number"
]
},
"MXN": {
"type": [
"null",
"number"
]
},
"MYR": {
"type": [
"null",
"number"
]
},
"NOK": {
"type": [
"null",
"number"
]
},
"NZD": {
"type": [
"null",
"number"
]
},
"PHP": {
"type": [
"null",
"number"
]
},
"PLN": {
"type": [
"null",
"number"
]
},
"RON": {
"type": [
"null",
"number"
]
},
"RUB": {
"type": [
"null",
"number"
]
},
"SEK": {
"type": [
"null",
"number"
]
},
"SGD": {
"type": [
"null",
"number"
]
},
"THB": {
"type": [
"null",
"number"
]
},
"TRY": {
"type": [
"null",
"number"
]
},
"USD": {
"type": [
"null",
"number"
]
},
"ZAR": {
"type": [
"null",
"number"
]
},
"date": {
"type": "string",
"format": "date-time"
}
}
},
"supported_sync_modes": [
"incremental"
],
"source_defined_cursor": true,
"default_cursor_field": [
"date"
],
"source_defined_primary_key": []
},
"sync_mode": "incremental",
"cursor_field": [
"date"
],
"destination_sync_mode": "append_dedup",
"primary_key": [
[
"date"
]
]
}
]
}u
user
04/02/2021, 9:00 PMWhich is the same exact configured catalog from before, when it was working properly.
u
user
04/02/2021, 9:04 PMHere’s the error:
u
user
04/02/2021, 9:04 PM Copy code
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - AirbyteEntrypoint(source).start(args)
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "/usr/local/lib/python3.7/site-packages/base_python/entrypoint.py", line 110, in start
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - generator = self.source.read(logger, config, catalog, state)
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "/usr/local/lib/python3.7/site-packages/base_singer/source.py", line 125, in read
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - masked_airbyte_catalog = ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "pydantic/main.py", line 455, in pydantic.main.BaseModel.parse_obj
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "pydantic/main.py", line 346, in pydantic.main.BaseModel.__init__
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - pydantic.error_wrappers.ValidationError: 1 validation error for ConfiguredAirbyteCatalog
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - streams -> 0 -> primary_key -> 0
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - str type expected (type=type_error.str)u
user
04/02/2021, 9:09 PMall you did was upgrade the exchange rates api source, not the whole demo app, right?
u
user
04/02/2021, 9:09 PMyep
u
user
04/02/2021, 9:09 PMi want to say that the schema of the catalog may have changed in between 0.18.1 and now.
u
user
04/02/2021, 9:10 PM(or the migration to 0.18.1 was wrong)
u
user
04/02/2021, 9:10 PMthis is an error within the source though?
u
user
04/02/2021, 9:10 PMso the same CAC goes in before and after
u
user
04/02/2021, 9:11 PMthe last protocol changes were on 3/26
u
user
04/02/2021, 9:11 PMnot necessarily. the API might modify it?
u
user
04/02/2021, 9:11 PM Copy code
pydantic.error_wrappers.ValidationError: 1 validation error for ConfiguredAirbyteCatalog
streams -> 0 -> primary_key -> 0
str type expected (type=type_error.str)u
user
04/02/2021, 9:12 PMIf I run the source locally with the catalog I pasted it shows the same error
u
user
04/02/2021, 9:12 PMoh
u
user
04/02/2021, 9:12 PMkk
u
user
04/02/2021, 9:13 PMi'll just try to debug locally then!
u
user
04/02/2021, 9:13 PMfrom the integration dir on master
u
user
04/02/2021, 9:13 PMdocker run --rm -i -v $(pwd)/sample_files:/files airbyte/source-exchangeratesapi-singer:0.2.1 read --config /files/config.json --catalog /files/cac.jsonu
user
04/02/2021, 9:13 PMif you put the catalog I pasted in
sample_files/cac.jsonu
user
04/02/2021, 9:18 PMInterestingly the airbyte protocol in
./base_python_structs/airbyte_protocol/models/airbyte_protocol.pyu
user
04/02/2021, 9:18 PM Copy code
class ConfiguredAirbyteStream(BaseModel):
class Config:
extra = Extra.allow
stream: AirbyteStream
sync_mode: Optional[SyncMode] = "full_refresh"
cursor_field: Optional[List[str]] = Field(
None,
description="Path to the field that will be used to determine if a record is new or modified since the last sync. This field is REQUIRED if `sync_mode` is `incremental`. Otherwise it is ignored.",
)
destination_sync_mode: Optional[DestinationSyncMode] = "append"
primary_key: Optional[List[str]] = Field(
None,
description="Paths to the fields that will be used as primary key. This field is REQUIRED if `destination_sync_mode` is `*_dedup`. Otherwise it is ignored.",
)u
user
04/02/2021, 9:18 PMwhich is weird since this is a new branch from this morning?
u
user
04/02/2021, 9:19 PMunless there’s some sort of build error
u
user
04/02/2021, 9:24 PMremoved misleading messages
u
user
04/02/2021, 9:24 PMthere is a difference between the protocol / base that was generated on my build on github and latest local
u
user
04/02/2021, 9:26 PMfwiw it seems to work on the latest verison of airbyte for me locally.
u
user
04/02/2021, 9:28 PMfrom scratch, right?
u
user
04/02/2021, 9:28 PMyes
u
user
04/02/2021, 9:29 PMI couldn’t reset this source to fix the problem either in the UI on the demo
u
user
04/02/2021, 9:29 PMDid you match the configuration in the given catalog on the UI? And if so, did it generate a different catalog?
u
user
04/02/2021, 9:32 PMhow do i "log into" the demo again?
u
user
04/02/2021, 9:33 PMi think reset is failing because the UI is supposed to be readonly
u
user
04/02/2021, 9:33 PMhttps://demo.airbyte.io/hack/auth + lastpass
u
user
04/02/2021, 9:33 PMno this was after logging it
u
user
04/02/2021, 9:33 PMit’s how I upgraded the source from 0.2.0 -> 0.2.1
u
user
04/02/2021, 9:33 PMand ran the manual syncs
u
user
04/02/2021, 9:38 PMu
user
04/02/2021, 9:38 PMi just created a new one and it is working 🤷♀️
u
user
04/02/2021, 9:38 PMfor that one I’m going to update to dedupe
u
user
04/02/2021, 9:39 PMkk
u
user
04/02/2021, 9:39 PMwomp womp
u
user
04/02/2021, 9:40 PMyou didn't publish the connector from your local machine, right? you did it via github?
u
user
04/02/2021, 9:40 PMyeah from github
u
user
04/02/2021, 9:41 PMbut building from scratch locally I end up with the same behavior now
u
user
04/02/2021, 9:41 PMand if you look at the protocol files they don’t look updated
u
user
04/02/2021, 9:43 PMwant to zoom?
u
user
04/02/2021, 9:43 PMsure
u
user
04/02/2021, 9:49 PMCharles found it,
Copy code
airbyte/integration-base-singer:0.1.0u
user
04/02/2021, 9:50 PMPrimary key related configuration in the UI will cause failures in sources
u
user
04/02/2021, 9:55 PMIt looks like the
airbyte/integration-base-singer:0.1.0u
user
04/02/2021, 9:56 PMI think what we have to do is bump the version for internal packages, release versions for all python sources, and then document in the FAQ this error, and encourage people to upgrade to the most recent connector versions before using PK incremental
u
user
04/02/2021, 9:56 PMAnd then add testing for primary key configuration
u
user
04/03/2021, 6:10 AMTo close the loop on this, there were two problems:
1. The python data model generator used an old version that couldn’t handle lists of lists which were part of the new protocol version.
2. The second is that python sources hadn’t been updated to include the upgraded protocol code anyways.
u
user
04/03/2021, 6:11 AMhttps://github.com/airbytehq/airbyte/pull/2726 corrected both of these issues. We’ll still need to encourage upgrading connector versions to take advantage of primary key selection.