https://linen.dev logo
Join Slack
Powered by
Hi! Im getting a validation error when trying out ...
# contributing-to-airbyte
m
Hi! Im getting a validation error when trying out “read” in the terminal.
Copy code
virt_env) martin@MacBook-Pro source-provet % python3 main.py read --config secrets/config.json --catalog integration_tests/catalog.json
Logg:
Copy code
Traceback (most recent call last):
  File "main.py", line 33, in <module>
    launch(source, sys.argv[1:])
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 125, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 113, in run
    config_catalog = self.source.read_catalog(parsed_args.catalog)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/source.py", line 48, in read_catalog
    return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))
  File "pydantic/main.py", line 578, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 3 validation errors for ConfiguredAirbyteCatalog
streams -> 0 -> stream
  field required (type=value_error.missing)
streams -> 0 -> sync_mode
  field required (type=value_error.missing)
streams -> 0 -> destination_sync_mode
  field required (type=value_error.missing)
u
As you can see im using a json file as catalog
u
Copy code
{
  "streams": [
    {
      "name": "consultation",
      "supported_sync_modes": [
        "full_refresh"
      ],
      "source_defined_cursor": false,
      "json_schema": {
        "type": "object",
        "properties": {
          "id": {
            "type": "number"
          },
          "url": {
            "type": "string"
          }
        }
      }
    }
  ]
}
u
The stream class:
Copy code
class Consultation(ProvetStream):
    """
    TODO: Change class name to match the table/data source this stream corresponds to.
    """

    # TODO: Fill in the primary key. Required. This is usually a unique field in the stream, like an ID or a timestamp.
    primary_key = "id"

    def path(
        self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
    ) -> str:
    
        return "consultation"
u
Where should I start looking for the error?
m
@Martin Larsson I think you are running into the issue of configured catalog vs. catalog you are inputting a catalog when you need to be inputting a configured catalog https://docs.airbyte.io/understanding-airbyte/beginners-guide-to-catalog
u
u
OK, so I used catalog protocol and not the configured catalog protocol?
u
Yep, now at least I got a different validation error. 🙂
u
It wants destination_sync_mode but I dont know what to put in?
u
Now in my catalog.json:
Copy code
{
  "streams": [
    {
      "sync_mode": "full_refresh",
      "stream": {
        "name": "consultation",
        "supported_sync_modes": [
          "full_refresh"
        ],
        "destination_sync_mode": [
          "full_refresh"
        ],
        "source_defined_cursor": false,
        "json_schema": {
          "type": "object",
          "properties": {
            "id": {
              "type": "number"
            },
            "url": {
              "type": "string"
            }
          }
        }
      }
    }
  ]
}
m
Copy code
ydantic.error_wrappers.ValidationError: 1 validation error for ConfiguredAirbyteCatalog
streams -> 0 -> destination_sync_mode
s
@Martin Larsson this is a bit of tech debt we need to clean up from the configured catalog since this config is related only to destinations but for now just put something like “`destination_sync_mode="overwrite"` to make the validation stop failing
u
It didn’t help 😞
d
Copy code
{
  "streams": [
    {
      "sync_mode": "full_refresh",
      "stream": {
        "name": "consultation",
        "supported_sync_modes": [
          "full_refresh"
        ],
        "destination_sync_mode": "overwrite",
        "source_defined_cursor": false,
        "json_schema": {
          "type": "object",
          "properties": {
            "id": {
              "type": "number"
            },
            "url": {
              "type": "string"
            }
          }
        }
      }
    }
  ]
}
u
Copy code
pydantic.error_wrappers.ValidationError: 1 validation error for ConfiguredAirbyteCatalog
streams -> 0 -> destination_sync_mode
Open in Slack
PreviousNext
Powered by