https://linen.dev logo
Join Slack
Powered by
On the demo I upgraded to the new version of the e...
# contributing-to-airbyte
j
On the demo I upgraded to the new version of the exchange rate api and I’m running into a weird catalog configuration error.
u
I upgraded the version for the demo instance
u
u
Everything went perfectly first time
u
u
I’m getting JSON decoding issues for the catalog.
u
I tried resetting data/rediscovering schemas/etc and couldn’t get anything to work yet
u
This is the configured catalog at 
/tmp/workspace/115/0/source_catalog.json
u
Copy code
{
  "streams": [
    {
      "stream": {
        "name": "exchange_rate",
        "json_schema": {
          "type": "object",
          "properties": {
            "AUD": {
              "type": [
                "null",
                "number"
              ]
            },
            "BGN": {
              "type": [
                "null",
                "number"
              ]
            },
            "BRL": {
              "type": [
                "null",
                "number"
              ]
            },
            "CAD": {
              "type": [
                "null",
                "number"
              ]
            },
            "CHF": {
              "type": [
                "null",
                "number"
              ]
            },
            "CNY": {
              "type": [
                "null",
                "number"
              ]
            },
            "CZK": {
              "type": [
                "null",
                "number"
              ]
            },
            "DKK": {
              "type": [
                "null",
                "number"
              ]
            },
            "EUR": {
              "type": [
                "null",
                "number"
              ]
            },
            "GBP": {
              "type": [
                "null",
                "number"
              ]
            },
            "HKD": {
              "type": [
                "null",
                "number"
              ]
            },
            "HRK": {
              "type": [
                "null",
                "number"
              ]
            },
            "HUF": {
              "type": [
                "null",
                "number"
              ]
            },
            "IDR": {
              "type": [
                "null",
                "number"
              ]
            },
            "ILS": {
              "type": [
                "null",
                "number"
              ]
            },
            "INR": {
              "type": [
                "null",
                "number"
              ]
            },
            "ISK": {
              "type": [
                "null",
                "number"
              ]
            },
            "JPY": {
              "type": [
                "null",
                "number"
              ]
            },
            "KRW": {
              "type": [
                "null",
                "number"
              ]
            },
            "MXN": {
              "type": [
                "null",
                "number"
              ]
            },
            "MYR": {
              "type": [
                "null",
                "number"
              ]
            },
            "NOK": {
              "type": [
                "null",
                "number"
              ]
            },
            "NZD": {
              "type": [
                "null",
                "number"
              ]
            },
            "PHP": {
              "type": [
                "null",
                "number"
              ]
            },
            "PLN": {
              "type": [
                "null",
                "number"
              ]
            },
            "RON": {
              "type": [
                "null",
                "number"
              ]
            },
            "RUB": {
              "type": [
                "null",
                "number"
              ]
            },
            "SEK": {
              "type": [
                "null",
                "number"
              ]
            },
            "SGD": {
              "type": [
                "null",
                "number"
              ]
            },
            "THB": {
              "type": [
                "null",
                "number"
              ]
            },
            "TRY": {
              "type": [
                "null",
                "number"
              ]
            },
            "USD": {
              "type": [
                "null",
                "number"
              ]
            },
            "ZAR": {
              "type": [
                "null",
                "number"
              ]
            },
            "date": {
              "type": "string",
              "format": "date-time"
            }
          }
        },
        "supported_sync_modes": [
          "incremental"
        ],
        "source_defined_cursor": true,
        "default_cursor_field": [
          "date"
        ],
        "source_defined_primary_key": []
      },
      "sync_mode": "incremental",
      "cursor_field": [
        "date"
      ],
      "destination_sync_mode": "append_dedup",
      "primary_key": [
        [
          "date"
        ]
      ]
    }
  ]
}
u
Which is the same exact configured catalog from before, when it was working properly.
u
Here’s the error:
u
Copy code
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - AirbyteEntrypoint(source).start(args)
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "/usr/local/lib/python3.7/site-packages/base_python/entrypoint.py", line 110, in start
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - generator = self.source.read(logger, config, catalog, state)
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "/usr/local/lib/python3.7/site-packages/base_singer/source.py", line 125, in read
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - masked_airbyte_catalog = ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "pydantic/main.py", line 455, in pydantic.main.BaseModel.parse_obj
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - File "pydantic/main.py", line 346, in pydantic.main.BaseModel.__init__
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - pydantic.error_wrappers.ValidationError: 1 validation error for ConfiguredAirbyteCatalog
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - streams -> 0 -> primary_key -> 0
2021-04-02 20:39:11 ERROR (/tmp/workspace/115/2) LineGobbler(voidCall):69 - str type expected (type=type_error.str)
u
all you did was upgrade the exchange rates api source, not the whole demo app, right?
u
yep
u
i want to say that the schema of the catalog may have changed in between 0.18.1 and now.
u
(or the migration to 0.18.1 was wrong)
u
this is an error within the source though?
u
so the same CAC goes in before and after
u
the last protocol changes were on 3/26
u
not necessarily. the API might modify it?
u
Copy code
pydantic.error_wrappers.ValidationError: 1 validation error for ConfiguredAirbyteCatalog
streams -> 0 -> primary_key -> 0
str type expected (type=type_error.str)
u
If I run the source locally with the catalog I pasted it shows the same error
u
oh
u
kk
u
i'll just try to debug locally then!
u
from the integration dir on master
u
docker run --rm -i  -v $(pwd)/sample_files:/files airbyte/source-exchangeratesapi-singer:0.2.1 read --config /files/config.json --catalog /files/cac.json
u
if you put the catalog I pasted in 
sample_files/cac.json
u
Interestingly the airbyte protocol in 
./base_python_structs/airbyte_protocol/models/airbyte_protocol.py
is out of date
u
Copy code
class ConfiguredAirbyteStream(BaseModel):
    class Config:
        extra = Extra.allow

    stream: AirbyteStream
    sync_mode: Optional[SyncMode] = "full_refresh"
    cursor_field: Optional[List[str]] = Field(
        None,
        description="Path to the field that will be used to determine if a record is new or modified since the last sync. This field is REQUIRED if `sync_mode` is `incremental`. Otherwise it is ignored.",
    )
    destination_sync_mode: Optional[DestinationSyncMode] = "append"
    primary_key: Optional[List[str]] = Field(
        None,
        description="Paths to the fields that will be used as primary key. This field is REQUIRED if `destination_sync_mode` is `*_dedup`. Otherwise it is ignored.",
    )
u
which is weird since this is a new branch from this morning?
u
unless there’s some sort of build error
u
removed misleading messages
u
there is a difference between the protocol / base that was generated on my build on github and latest local
u
fwiw it seems to work on the latest verison of airbyte for me locally.
u
from scratch, right?
u
yes
u
I couldn’t reset this source to fix the problem either in the UI on the demo
u
Did you match the configuration in the given catalog on the UI? And if so, did it generate a different catalog?
u
how do i "log into" the demo again?
u
i think reset is failing because the UI is supposed to be readonly
u
u
no this was after logging it
u
it’s how I upgraded the source from 0.2.0 -> 0.2.1
u
and ran the manual syncs
u
u
i just created a new one and it is working 🤷‍♀️
u
for that one I’m going to update to dedupe
u
kk
u
womp womp
u
you didn't publish the connector from your local machine, right? you did it via github?
u
yeah from github
u
but building from scratch locally I end up with the same behavior now
u
and if you look at the protocol files they don’t look updated
u
want to zoom?
u
sure
u
Charles found it,
Copy code
airbyte/integration-base-singer:0.1.0
is out of date
u
Primary key related configuration in the UI will cause failures in sources
u
It looks like the 
airbyte/integration-base-singer:0.1.0
version is different than the version on master which is also different from the version in 0.2.0
u
I think what we have to do is bump the version for internal packages, release versions for all python sources, and then document in the FAQ this error, and encourage people to upgrade to the most recent connector versions before using PK incremental
u
And then add testing for primary key configuration
u
To close the loop on this, there were two problems: 1. The python data model generator used an old version that couldn’t handle lists of lists which were part of the new protocol version. 2. The second is that python sources hadn’t been updated to include the upgraded protocol code anyways.
u
https://github.com/airbytehq/airbyte/pull/2726 corrected both of these issues. We’ll still need to encourage upgrading connector versions to take advantage of primary key selection.
Open in Slack
PreviousNext
Powered by