Hello Airbyte community I m having some problems when creati Airbyte #ask-community-for-troubleshooting

Hello Airbyte community, I’m having some problems ...

Ricardo de Deijn

09/19/2022, 2:02 PM

Hello Airbyte community, I’m having some problems when creating my own Airbyte Connector. I got an error message saying:

The requested stream categories was not found in the source. Available streams: dict_keys(['accounts'])

. As can be seen in the picture, it happened during the

python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json

command and after it finished syncing the accounts endpoint of my source API. What does this FATAL message mean and how can I fix this problem?

👀 1

Harshith (Airbyte)

09/19/2022, 2:37 PM

Hey could you check if you have added the stream in the source.py

Ricardo de Deijn

09/19/2022, 2:40 PM

Hello @Harshith (Airbyte). Thank you for your answer. You mean this right?

Copy code

class Accounts(RobinOdataStream):
    primary_key = "UniqueId"
    use_cache = True

    def path(self, **kwargs) -> str:
        return ""

Ricardo de Deijn

09/19/2022, 2:42 PM

This was included in my source.py file

Harshith (Airbyte)

09/19/2022, 2:43 PM

Hey in the

def streams

function you have to add this stream

Harshith (Airbyte)

09/19/2022, 2:43 PM

Hey you can also go through this doc https://airbyte.gitbook.io/airbyte/connector-development/tutorials/cdk-speedrun

Ricardo de Deijn

09/19/2022, 2:49 PM

Thank you for your answer @Harshith (Airbyte) Oh sorry I see what you mean now. I feel like I have already add this to the def streams function. This is my current def streams function:

Copy code

def streams(self, config: Mapping[str, Any]) -> List[Stream]:
    gateway_url = self.gateway_url(config)
    client = RobinAuthenticator(config["username"], config["password"], gateway_url)
    kwargs = {
        "gateway_url": gateway_url,
        "client": client,
        "start_date": config.get("start_date"),
    }
    return [
        Accounts(**kwargs),
    ]

Is this right?

Ricardo de Deijn

09/19/2022, 2:50 PM

This is my entire source.py, that might be easier for communication. @Harshith (Airbyte)

source.py

Ricardo de Deijn

09/19/2022, 6:22 PM

Hello, this is my full log file before the error message saying the requested stream categories was not found in the source. Available streams: dict_keys([‘accounts’]). Would any of you be able to help me out? I can’t figure out why it’s not working. It looks like it can find the stream at first, but after recording everything I get the error that the stream category was not found. More information can be found in the thread this message is part of.

Copy code

{"type": "LOG", "log": {"level": "INFO", "message": "Starting syncing SourceRobinOdata"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: accounts "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Backing off _send(...) for 5.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: <https://api.robinhq.com/odata/accounts>, Response Code: 571, Response Text: )"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Caught retryable error 'Request URL: <https://api.robinhq.com/odata/accounts>, Response Code: 571, Response Text: ' after 1 tries. Waiting 5 seconds then retrying..."}}
{"type": "RECORD", "record": {"stream": "accounts", "data": {[deleted for privacy purposes]}, "emitted_at": 1663603385702}}
[240 more records]
Not more than 1000 rows found
{"type": "LOG", "log": {"level": "INFO", "message": "Read 241 records from accounts stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing accounts"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceRobinOdata runtimes:\nSyncing stream accounts 0:00:07.228217"}}
{"type": "LOG", "log": {"level": "FATAL", "message": "\"The requested stream categories was not found in the source. Available streams: dict_keys(['accounts'])\"\nTraceback (most recent call last):\n  File \"main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/Users/ricardo/airbyte/airbyte-integrations/connectors/source-robin-odata/.venv/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py\", line 127, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/Users/ricardo/airbyte/airbyte-integrations/connectors/source-robin-odata/.venv/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py\", line 118, in run\n    for message in generator:\n  File \"/Users/ricardo/airbyte/airbyte-integrations/connectors/source-robin-odata/.venv/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 108, in read\n    f\"The requested stream {configured_stream.stream.name} was not found in the source.\"\nKeyError: \"The requested stream categories was not found in the source. Available streams: dict_keys(['accounts'])\""}}

Harshith (Airbyte)

09/20/2022, 10:54 AM

Hey as in the source.py I can see that Accounts is the only added stream and categories stream is not present

Ricardo de Deijn

09/20/2022, 3:52 PM

Ohh, alright, I get it now. I fixed the problem, thank you!!

15 Views

Open in Slack

Previous Next