https://linen.dev logo
j

Jared Rhizor (Airbyte)

04/02/2021, 8:51 PM
On the demo I upgraded to the new version of the exchange rate api and I’m running into a weird catalog configuration error.
u

user

04/02/2021, 8:54 PM
I upgraded the version for the demo instance
u

user

04/02/2021, 8:54 PM
u

user

04/02/2021, 8:54 PM
Everything went perfectly first time
u

user

04/02/2021, 8:54 PM
u

user

04/02/2021, 8:55 PM
I’m getting JSON decoding issues for the catalog.
u

user

04/02/2021, 8:55 PM
I tried resetting data/rediscovering schemas/etc and couldn’t get anything to work yet
u

user

04/02/2021, 8:56 PM
This is the configured catalog at
/tmp/workspace/115/0/source_catalog.json
u

user

04/02/2021, 8:56 PM
Copy code
{
  "streams": [
    {
      "stream": {
        "name": "exchange_rate",
        "json_schema": {
          "type": "object",
          "properties": {
            "AUD": {
              "type": [
                "null",
                "number"
              ]
            },
            "BGN": {
              "type": [
                "null",
                "number"
              ]
            },
            "BRL": {
              "type": [
                "null",
                "number"
              ]
            },
            "CAD": {
              "type": [
                "null",
                "number"
              ]
            },
            "CHF": {
              "type": [
                "null",
                "number"
              ]
            },
            "CNY": {
              "type": [
                "null",
                "number"
              ]
            },
            "CZK": {
              "type": [
                "null",
                "number"
              ]
            },
            "DKK": {
              "type": [
                "null",
                "number"
              ]
            },
            "EUR": {
              "type": [
                "null",
                "number"
              ]
            },
            "GBP": {
              "type": [
                "null",
                "number"
              ]
            },
            "HKD": {
              "type": [
                "null",
                "number"
              ]
            },
            "HRK": {
              "type": [
                "null",
                "number"
              ]
            },
            "HUF": {
              "type": [
                "null",
                "number"
              ]
            },
            "IDR": {
              "type": [
                "null",
                "number"
              ]
            },
            "ILS": {
              "type": [
                "null",
                "number"
              ]
            },
            "INR": {
              "type": [
                "null",
                "number"
              ]
            },
            "ISK": {
              "type": [
                "null",
                "number"
              ]
            },
            "JPY": {
              "type": [
                "null",
                "number"
              ]
            },
            "KRW": {
              "type": [
                "null",
                "number"
              ]
            },
            "MXN": {
              "type": [
                "null",
                "number"
              ]
            },
            "MYR": {
              "type": [
                "null",
                "number"
              ]
            },
            "NOK": {
              "type": [
                "null",
                "number"
              ]
            },
            "NZD": {
              "type": [
                "null",
                "number"
              ]
            },
            "PHP": {
              "type": [
                "null",
                "number"
              ]
            },
            "PLN": {
              "type": [
                "null",
                "number"
              ]
            },
            "RON": {
              "type": [
                "null",
                "number"
              ]
            },
            "RUB": {
              "type": [
                "null",
                "number"
              ]
            },
            "SEK": {
              "type": [
                "null",
                "number"
              ]
            },
            "SGD": {
              "type": [
                "null",
                "number"
              ]
            },
            "THB": {
              "type": [
                "null",
                "number"
              ]
            },
            "TRY": {
              "type": [
                "null",
                "number"
              ]
            },
            "USD": {
              "type": [
                "null",
                "number"
              ]
            },
            "ZAR": {
              "type": [
                "null",
                "number"
              ]
            },
            "date": {
              "type": "string",
              "format": "date-time"
            }
          }
        },
        "supported_sync_modes": [
          "incremental"
        ],
        "source_defined_cursor": true,
        "default_cursor_field": [
          "date"
        ],
        "source_defined_primary_key": []
      },
      "sync_mode": "incremental",
      "cursor_field": [
        "date"
      ],
      "destination_sync_mode": "append_dedup",
      "primary_key": [
        [
          "date"
        ]
      ]
    }
  ]
}
u

user

04/02/2021, 9:00 PM
Which is the same exact configured catalog from before, when it was working properly.
u

user

04/02/2021, 9:04 PM
Here’s the error:
u

user

04/02/2021, 9:04 PM
Copy code
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - AirbyteEntrypoint(source).start(args)
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "/usr/local/lib/python3.7/site-packages/base_python/entrypoint.py", line 110, in start
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - generator = self.source.read(logger, config, catalog, state)
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "/usr/local/lib/python3.7/site-packages/base_singer/source.py", line 125, in read
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - masked_airbyte_catalog = ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "pydantic/main.py", line 455, in pydantic.main.BaseModel.parse_obj
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "pydantic/main.py", line 346, in pydantic.main.BaseModel.__init__
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - pydantic.error_wrappers.ValidationError: 1 validation error for ConfiguredAirbyteCatalog
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - streams -> 0 -> primary_key -> 0
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - str type expected (type=type_error.str)
u

user

04/02/2021, 9:09 PM
all you did was upgrade the exchange rates api source, not the whole demo app, right?
u

user

04/02/2021, 9:09 PM
yep
u

user

04/02/2021, 9:09 PM
i want to say that the schema of the catalog may have changed in between 0.18.1 and now.
u

user

04/02/2021, 9:10 PM
(or the migration to 0.18.1 was wrong)
u

user

04/02/2021, 9:10 PM
this is an error within the source though?
u

user

04/02/2021, 9:10 PM
so the same CAC goes in before and after
u

user

04/02/2021, 9:11 PM
the last protocol changes were on 3/26
u

user

04/02/2021, 9:11 PM
not necessarily. the API might modify it?
u

user

04/02/2021, 9:11 PM
Copy code
pydantic.error_wrappers.ValidationError: 1 validation error for ConfiguredAirbyteCatalog
streams -> 0 -> primary_key -> 0
str type expected (type=type_error.str)
u

user

04/02/2021, 9:12 PM
If I run the source locally with the catalog I pasted it shows the same error
u

user

04/02/2021, 9:12 PM
oh
u

user

04/02/2021, 9:12 PM
kk
u

user

04/02/2021, 9:13 PM
i'll just try to debug locally then!
u

user

04/02/2021, 9:13 PM
from the integration dir on master
u

user

04/02/2021, 9:13 PM
docker run --rm -i  -v $(pwd)/sample_files:/files airbyte/source-exchangeratesapi-singer:0.2.1 read --config /files/config.json --catalog /files/cac.json
u

user

04/02/2021, 9:13 PM
if you put the catalog I pasted in
sample_files/cac.json
u

user

04/02/2021, 9:18 PM
Interestingly the airbyte protocol in
./base_python_structs/airbyte_protocol/models/airbyte_protocol.py
is out of date
u

user

04/02/2021, 9:18 PM
Copy code
class ConfiguredAirbyteStream(BaseModel):
    class Config:
        extra = Extra.allow

    stream: AirbyteStream
    sync_mode: Optional[SyncMode] = "full_refresh"
    cursor_field: Optional[List[str]] = Field(
        None,
        description="Path to the field that will be used to determine if a record is new or modified since the last sync. This field is REQUIRED if `sync_mode` is `incremental`. Otherwise it is ignored.",
    )
    destination_sync_mode: Optional[DestinationSyncMode] = "append"
    primary_key: Optional[List[str]] = Field(
        None,
        description="Paths to the fields that will be used as primary key. This field is REQUIRED if `destination_sync_mode` is `*_dedup`. Otherwise it is ignored.",
    )
u

user

04/02/2021, 9:18 PM
which is weird since this is a new branch from this morning?
u

user

04/02/2021, 9:19 PM
unless there’s some sort of build error
u

user

04/02/2021, 9:24 PM
removed misleading messages
u

user

04/02/2021, 9:24 PM
there is a difference between the protocol / base that was generated on my build on github and latest local
u

user

04/02/2021, 9:26 PM
fwiw it seems to work on the latest verison of airbyte for me locally.
u

user

04/02/2021, 9:28 PM
from scratch, right?
u

user

04/02/2021, 9:28 PM
yes
u

user

04/02/2021, 9:29 PM
I couldn’t reset this source to fix the problem either in the UI on the demo
u

user

04/02/2021, 9:29 PM
Did you match the configuration in the given catalog on the UI? And if so, did it generate a different catalog?
u

user

04/02/2021, 9:32 PM
how do i "log into" the demo again?
u

user

04/02/2021, 9:33 PM
i think reset is failing because the UI is supposed to be readonly
u

user

04/02/2021, 9:33 PM
u

user

04/02/2021, 9:33 PM
no this was after logging it
u

user

04/02/2021, 9:33 PM
it’s how I upgraded the source from 0.2.0 -> 0.2.1
u

user

04/02/2021, 9:33 PM
and ran the manual syncs
u

user

04/02/2021, 9:38 PM
i just created a new one and it is working 🤷‍♀️
u

user

04/02/2021, 9:38 PM
for that one I’m going to update to dedupe
u

user

04/02/2021, 9:39 PM
kk
u

user

04/02/2021, 9:39 PM
womp womp
u

user

04/02/2021, 9:40 PM
you didn't publish the connector from your local machine, right? you did it via github?
u

user

04/02/2021, 9:40 PM
yeah from github
u

user

04/02/2021, 9:41 PM
but building from scratch locally I end up with the same behavior now
u

user

04/02/2021, 9:41 PM
and if you look at the protocol files they don’t look updated
u

user

04/02/2021, 9:43 PM
want to zoom?
u

user

04/02/2021, 9:43 PM
sure
u

user

04/02/2021, 9:49 PM
Charles found it,
Copy code
airbyte/integration-base-singer:0.1.0
is out of date
u

user

04/02/2021, 9:50 PM
Primary key related configuration in the UI will cause failures in sources
u

user

04/02/2021, 9:55 PM
It looks like the
airbyte/integration-base-singer:0.1.0
version is different than the version on master which is also different from the version in 0.2.0
u

user

04/02/2021, 9:56 PM
I think what we have to do is bump the version for internal packages, release versions for all python sources, and then document in the FAQ this error, and encourage people to upgrade to the most recent connector versions before using PK incremental
u

user

04/02/2021, 9:56 PM
And then add testing for primary key configuration
u

user

04/03/2021, 6:10 AM
To close the loop on this, there were two problems: 1. The python data model generator used an old version that couldn’t handle lists of lists which were part of the new protocol version. 2. The second is that python sources hadn’t been updated to include the upgraded protocol code anyways.
u

user

04/03/2021, 6:11 AM
https://github.com/airbytehq/airbyte/pull/2726 corrected both of these issues. We’ll still need to encourage upgrading connector versions to take advantage of primary key selection.