Looking for some clarity on the dbt integration We use dbt C | Secoda-support

Docs

Join the conversation Join Slack

Channels

# customers

# discussion

# intros

# secoda-feature-requests

# secoda-support

Channels

Thread

#secoda-support

future-eye-6606

1 month ago

Looking for some clarity on the dbt integration. We use dbt Core and are on prem customers. It says we need to upload a manifest file, which I have. Does this file typically any contain connection details? How does it support continuous integration? If not, am I to assume with dbt core it only updates as we reupload the file? there is also a warning that

Whitelisting
Secoda will connect to your resource from the following IPs:  Make sure to allow inbound traffic from these IPs in your firewall, and include it in any database grants.

I assume this does not apply to on prem customers

elegant-house-93198

1 month ago

A couple of other self-hosted dbt core users upload continuously via the API @future-eye-6606. I’ll send over some instructions on how to do that

future-eye-6606

1 month ago

Cheers!

@plain-balloon-39336 we probably want to look at the api for this use case at least then

👍 1

@elegant-house-93198 so without the api its just reads the file on an ad-hoc basis. I inspected the file and i dont think they are any connection details and on prem so not worried about the IP address issues. But i did try uploading it and got some errors 😛

Dbt manifest extractor: 

'NoneType' object is not subscriptable

Traceback (most recent call last):

  File "/app/databuilder/job/job.py", line 75, in _launch_job

    self.task.run()

  File "/app/databuilder/task/task.py", line 67, in run

    record = self.extractor.extract()

  File "/app/databuilder/extractor/database/dbt/dbt_core_extractor.py", line 149, in extract

    return next(self._extract_iter)

  File "/app/databuilder/extractor/database/dbt/dbt_core_extractor.py", line 225, in _get_extract_iter

    manifest_list = get_manifest_list_from_machine_iam_role(

  File "/app/databuilder/utils/s3.py", line 78, in get_manifest_list_from_machine_iam_role

    if bucket_name[-1] != "/":

TypeError: 'NoneType' object is not subscriptable

Initialization: 

Job extraction failed, Error msg: 'NoneType' object is not subscriptable

Traceback (most recent call last):

  File "/app/databuilder/job/job.py", line 75, in _launch_job

    self.task.run()

  File "/app/databuilder/task/task.py", line 67, in run

    record = self.extractor.extract()

  File "/app/databuilder/extractor/database/dbt/dbt_core_extractor.py", line 149, in extract

    return next(self._extract_iter)

  File "/app/databuilder/extractor/database/dbt/dbt_core_extractor.py", line 225, in _get_extract_iter

    manifest_list = get_manifest_list_from_machine_iam_role(

  File "/app/databuilder/utils/s3.py", line 78, in get_manifest_list_from_machine_iam_role

    if bucket_name[-1] != "/":

TypeError: 'NoneType' object is not subscriptable



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "/app/databuilder/models.py", line 105, in process

    getattr(Extractors, integration_type)(

  File "/app/databuilder/extract/database/dbt/dbt_extract.py", line 117, in _extract_dbt_core

    metadata_job.launch()

  File "/app/databuilder/job/job.py", line 123, in launch

    self._launch_job(

  File "/app/databuilder/job/job.py", line 87, in _launch_job

    raise JobFailedError(f"Job extraction failed, Error msg: {e}")

databuilder.utils.exceptions.JobFailedError: Job extraction failed, Error msg: 'NoneType' object is not subscriptable

elegant-house-93198

1 month ago

@future-eye-6606 it’s likely because there hasn’t been a private bucket where the file gets uploaded to, but it’s much easier to just use the API

plain-balloon-39336

1 month ago

please hide the IP in the original post @future-eye-6606

✅ 1 :thankyou: 1

View count: 1

Join thread in Slack