Thread
#secoda-support
    f

    future-eye-6606

    1 month ago
    Looking for some clarity on the dbt integration. We use dbt Core and are on prem customers. It says we need to upload a manifest file, which I have. Does this file typically any contain connection details? How does it support continuous integration? If not, am I to assume with dbt core it only updates as we reupload the file? there is also a warning that
    Whitelisting
    Secoda will connect to your resource from the following IPs:  Make sure to allow inbound traffic from these IPs in your firewall, and include it in any database grants.
    I assume this does not apply to on prem customers
    e

    elegant-house-93198

    1 month ago
    A couple of other self-hosted dbt core users upload continuously via the API @future-eye-6606. I’ll send over some instructions on how to do that
    f

    future-eye-6606

    1 month ago
    Cheers!
    @plain-balloon-39336 we probably want to look at the api for this use case at least then
    @elegant-house-93198 so without the api its just reads the file on an ad-hoc basis. I inspected the file and i dont think they are any connection details and on prem so not worried about the IP address issues. But i did try uploading it and got some errors 😛
    Dbt manifest extractor: 
    
    'NoneType' object is not subscriptable
    
    Traceback (most recent call last):
    
      File "/app/databuilder/job/job.py", line 75, in _launch_job
    
        self.task.run()
    
      File "/app/databuilder/task/task.py", line 67, in run
    
        record = self.extractor.extract()
    
      File "/app/databuilder/extractor/database/dbt/dbt_core_extractor.py", line 149, in extract
    
        return next(self._extract_iter)
    
      File "/app/databuilder/extractor/database/dbt/dbt_core_extractor.py", line 225, in _get_extract_iter
    
        manifest_list = get_manifest_list_from_machine_iam_role(
    
      File "/app/databuilder/utils/s3.py", line 78, in get_manifest_list_from_machine_iam_role
    
        if bucket_name[-1] != "/":
    
    TypeError: 'NoneType' object is not subscriptable
    Initialization: 
    
    Job extraction failed, Error msg: 'NoneType' object is not subscriptable
    
    Traceback (most recent call last):
    
      File "/app/databuilder/job/job.py", line 75, in _launch_job
    
        self.task.run()
    
      File "/app/databuilder/task/task.py", line 67, in run
    
        record = self.extractor.extract()
    
      File "/app/databuilder/extractor/database/dbt/dbt_core_extractor.py", line 149, in extract
    
        return next(self._extract_iter)
    
      File "/app/databuilder/extractor/database/dbt/dbt_core_extractor.py", line 225, in _get_extract_iter
    
        manifest_list = get_manifest_list_from_machine_iam_role(
    
      File "/app/databuilder/utils/s3.py", line 78, in get_manifest_list_from_machine_iam_role
    
        if bucket_name[-1] != "/":
    
    TypeError: 'NoneType' object is not subscriptable
    
    
    
    During handling of the above exception, another exception occurred:
    
    
    
    Traceback (most recent call last):
    
      File "/app/databuilder/models.py", line 105, in process
    
        getattr(Extractors, integration_type)(
    
      File "/app/databuilder/extract/database/dbt/dbt_extract.py", line 117, in _extract_dbt_core
    
        metadata_job.launch()
    
      File "/app/databuilder/job/job.py", line 123, in launch
    
        self._launch_job(
    
      File "/app/databuilder/job/job.py", line 87, in _launch_job
    
        raise JobFailedError(f"Job extraction failed, Error msg: {e}")
    
    databuilder.utils.exceptions.JobFailedError: Job extraction failed, Error msg: 'NoneType' object is not subscriptable
    e

    elegant-house-93198

    1 month ago
    @future-eye-6606 it’s likely because there hasn’t been a private bucket where the file gets uploaded to, but it’s much easier to just use the API
    p

    plain-balloon-39336

    1 month ago
    please hide the IP in the original post @future-eye-6606