https://linen.dev logo
Join Slack
Powered by
``` features_stream: $ref: "*ref(definitions....
# help-connector-development
k
Copy code
features_stream:
    $ref: "*ref(definitions.base_stream)"
    retriever:
      $ref: "*ref(definitions.retriever)"
      record_selector:
        $ref: "*ref(definitions.selector_features)"
I want this to accept 403 response as INFO/WARNING - in aha.io it's not required to have features with products. It's possible to have only one but source forces both
I've check here and I understand what to do for requester, but does it apply for retriever too?
Is it possible to mark stream as optional in catalog?
a
all streams are optional. the user can specify which streams to sync in the configured catalog https://docs.airbyte.com/understanding-airbyte/beginners-guide-to-catalog/
and you can add error handling to ignore 401 errors
k
@Alexandre Girard (Airbyte) Thanks. I noticed that but when I ignore the error in error handling then I receive 
len()
issue. Is it possible to set the default empty list in this case?
Copy code
ERROR    root:connector_runner.py:164 Docker container failed, code 1, error:
{"type": "TRACE", "trace": {"type": "ERROR", "emitted_at": 1675328754268.001, "error": {"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "object of type 'NoneType' has no len()", "stack_trace": "Traceback (most recent call last):\n  File \"/airbyte/integration_code/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 131, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 122, in run\n    for message in generator:\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py\", line 168, in read\n    yield from super().read(logger, config, catalog, state)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 134, in read\n    raise e\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 120, in read\n    yield from self._read_stream(\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 189, in _read_stream\n    for record in record_iterator:\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 307, in _read_full_refresh\n    for record_data_or_message in record_data_or_messages:\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py\", line 115, in read_records\n    for record in self.retriever.read_records(sync_mode, cursor_field, stream_slice, stream_state):\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 376, in read_records\n    for record in records_generator:\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 432, in _read_pages\n    next_page_token = self.next_page_token(response)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 359, in next_page_token\n    return self.paginator.next_page_token(response, self._last_records)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/paginators/default_paginator.py\", line 102, in next_page_token\n    self._token = self.pagination_strategy.next_page_token(response, last_records)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/paginators/strategies/page_increment.py\", line 31, in next_page_token\n    if len(last_records) < self.page_size:\nTypeError: object of type 'NoneType' has no len()\n", "failure_type": "system_error"}}}
I got this when ignored wrong code with 
error_handler
in `requester`:
Copy code
error_handler:
      response_filters:
        - http_codes: [ 403 ]
          action: IGNORE
3 Views
Open in Slack
PreviousNext
Powered by