https://linen.dev logo
Channels
Powered by
m

Martin Larsson

09/24/2021, 9:00 PM
Hi! I just started a new connector and ran the code generator for an API connector. I didnt get far before I got an error I cant get past alone.
Copy code
equests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: <https://provetcloud.com/{clinic_ID}/api/0.1/consultation/>
u

user

09/24/2021, 9:01 PM
The full error message:
u

user

09/24/2021, 9:01 PM
Copy code
{"type": "LOG", "log": {"level": "ERROR", "message": "Encountered an exception while reading stream SourceProvet\nTraceback (most recent call last):\n  File \"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 117, in read\n    internal_config=internal_config,\n  File \"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 147, in _read_stream\n    for record in record_iterator:\n  File \"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 219, in _read_full_refresh\n    for record in records:\n  File \"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 318, in read_records\n    response = self._send_request(request, request_kwargs)\n  File \"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 295, in _send_request\n    return backoff_handler(user_backoff_handler)(request, request_kwargs)\n  File \"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/backoff/_sync.py\", line 94, in retry\n    ret = target(*args, **kwargs)\n  File \"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/backoff/_sync.py\", line 94, in retry\n    ret = target(*args, **kwargs)\n  File \"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 261, in _send\n    response.raise_for_status()\n  File \"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/models.py\", line 940, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: <https://provetcloud.com/3577/api/0.1/consultation/>\n"}}
Traceback (most recent call last):
  File "main.py", line 33, in <module>
    launch(source, sys.argv[1:])
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 125, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 116, in run
    for message in generator:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 121, in read
    raise e
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 117, in read
    internal_config=internal_config,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 147, in _read_stream
    for record in record_iterator:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 219, in _read_full_refresh
    for record in records:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py", line 318, in read_records
    response = self._send_request(request, request_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py", line 295, in _send_request
    return backoff_handler(user_backoff_handler)(request, request_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py", line 261, in _send
    response.raise_for_status()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: <https://provetcloud.com/.../api/0.1/consultation/>
u

user

09/24/2021, 9:02 PM
I can inspect with print that the token is inserted in the constructor to my stream
Copy code
def streams(self, config: Mapping[str, Any]) -> List[Stream]:
        
        api_token = config['api_token']
        auth = TokenAuthenticator(token=api_token)

        return [Consultation(authenticator=auth)]
m

Martin Larsson

09/24/2021, 9:02 PM
What am I doing wrong?
u

user

09/24/2021, 9:10 PM
This is the command I am running 
python3 main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
m

Martin Larsson

09/24/2021, 9:24 PM
is 
Consultation
passing up the authenticator in an 
super.__init__
call?
u

user

09/24/2021, 9:43 PM
No
Copy code
class Consultation(ProvetStream):
    primary_key = "id"

    def path(
        self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
    ) -> str:
   
        return "consultation"
u

user

09/24/2021, 9:43 PM
I thought that stuff was magic in the CDK
u

user

09/24/2021, 9:55 PM
Looking at the Stripe code I see now code that is passing the authenticator to a super class
u

user

09/24/2021, 9:59 PM
Tried adding
Copy code
def __init__(self, **kwargs):
        super().__init__(**kwargs)
to 
class ProvetStream(HttpStream, ABC)
but no luck
u

user

09/24/2021, 10:30 PM
yeah sorry was in a call, passing is only relevant if the subclass 
ProvetStream
separately overrides 
__init__
u

user

09/24/2021, 10:31 PM
if no class in the hierarchy overrides it, then it’s not needed to explicitly pass the params. they get passed by default
m

Martin Larsson

09/25/2021, 4:45 AM
Is there anything else I can try?
a

Artem Astapenko

09/27/2021, 8:24 AM
hard to say without seeing the full code but I would debug it by: 1. writing a curl request that works 2. compare the params of that curl request with what I’m passing in the python code or until we have something like https://github.com/airbytehq/airbyte/issues/3078 you could insert a bunch of print statements in the CDK to see what’s being passed
u

user

09/27/2021, 9:15 PM
I have tried curl and it gives me the desired response
s

Sawyer Waugh

09/27/2021, 9:16 PM
Somehow I suspect that the api token is not used in the request
u

user

09/27/2021, 9:19 PM
Copy code
class ProvetStream(HttpStream, ABC):


    url_base = "<https://provetcloud.com/xxxx/api/0.1/>"


    def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:

       decoded_response = response.json()
        if bool(decoded_response.get("results", [])):
            return None

        return {'page': response.json()['page'] + 1}

    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
        
        response_json = response.json()
        print(response_json)
        yield from response_json.get("results", [])


class Consultation(ProvetStream):
    primary_key = "id"

    def path(
        self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
    ) -> str:
    
        return "consultation"


class SourceProvet(AbstractSource):
    def check_connection(self, logger, config) -> Tuple[bool, any]:
       
        clinic_id = config['clinic_id']
        if clinic_id is None:
            return False, f"No clinic id is given. Should be a four digit number like 9999"
        elif clinic_id.isdigit() != True:
            return False, f"Clinic id is not numeric. Should be a four digit number like 9999"

        api_token = config['api_token']
        if api_token is None:
            return False, f"No API token is given. Should be a 40 character hexadecimal (0-9, a-f) string"
        if len(api_token) != 40:
            return False, f"API token is not 40 characters. Should be a 40 character hexadecimal (0-9, a-f) string"

        return True, None


    def streams(self, config: Mapping[str, Any]) -> List[Stream]:
        
        api_token = config['api_token']
        auth = TokenAuthenticator(token=api_token)

        return [Consultation(authenticator=auth)]
u

user

09/27/2021, 9:19 PM
Thats all the code
u

user

09/27/2021, 9:21 PM
I can print the api_toke in the def streams method so it is read from the secrets folder alright.
u

user

09/27/2021, 9:41 PM
is the auth header supposed to be 
Authorization: Bearer <token>
?
u

user

09/27/2021, 9:41 PM
can you share what the curl request look slike?
u

user

09/28/2021, 5:39 PM
In curl I use 
Authorization: Token <token>
which works
u

user

09/28/2021, 5:40 PM
curl -H "Authorization: Token <token>" <https://provetcloud.com/xxx/api/0.1/consultation/>
u

user

09/28/2021, 5:42 PM
When trying Bearer instead of Token in the curl command I get: 
{"detail":"Authentication credentials were not provided."}
u

user

09/28/2021, 5:52 PM
This woked! 🙂
Copy code
api_token = config['api_token']
        auth = TokenAuthenticator(token=api_token, auth_method='Token')

        return [Consultation(authenticator=auth)]
a

Ameya Bapat

09/28/2021, 5:57 PM
💪🏼
7 Views
Powered by