set the channel description: Channel dedicated to design & development for a Vertica integration

Hi folks! @breezy-controller-54597 reached out to me today - we have a draft PR for a Vertica integration! https://github.com/datahub-project/datahub/pull/4555

@breezy-controller-54597 it sounds like you need some support figuring out how to extract column-level detail from Vertica, correct?

Yes, I implemented get_view_definition so I can ingest the view, but I am unable to get the column of the view.

I added

get_columns

and

_get_column_info

, but VerticaDialect object has been recognized as str...

datahub/ingestion/source/sql/vertica.py, line 149, in _get_column_info  
    self.ischema_names["UUID"] = UUID  
AttributeError: 'str' object has no attribute 'ischema_names'

An instance that should be listed in get_columns was missed. I committed a fixed version and confirmed that the View column information can be retrieved.

I am confirming ingestion because the error occurs when the column type is DATE.

I have looked at the code from other sources and it seems that the column type needs to be converted so that sqlalchemy can handle it, but I don't know how. I am getting the following error.

[2022-04-07 04:57:15,056] WARNING {datahub.ingestion.source.sql.sql_common:1066] - Unable to ingest view SCHEMA.VIEWNAME
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/sql/sql_common.py", line 1058, in loop_views
    yield from  self._process_view(
  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/sql/sql_common.py", line 1084, in_process_view
    columns = inspector.get_columns(view, schema)
  File: "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 390, in get_columns
    col_defs = self.dialect.get_columns(
  File "<string>", line 2, in get_columns
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 52, in cache
    ret = fn(self, can, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/sql/vertica.py", line 97, in get_columns
    column_info = self._get_column_info(name, type, default, nullable, schema)
  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/sql/vertica.py", line 155, in _get_column_info
    coltype = coltype(*args, **kwargs)
TypeError: 'DATE' object is not callable

@dazzling-judge-80093 @helpful-optician-78938 @mammoth-bear-12532 any ideas/suggestions for Ebu? (not sure if this is Vertica-specific or sqlalchemy-specific)

Hello, I have created a recipe file based on Vertica as a source (https://datahubproject.io/docs/generated/ingestion/sources/vertica/) and when i execute the receipe file, i get the below error for views: vertica_python.errors.MissingRelation: Severity: ERROR, Message: Relation "pg_class" does not exist, Sqlstate: 42V01, Routine: throwRelationDoesNotExist, File: /data/jenkins/workspace/RE-ReleaseBuilds/RE-Jackhammer/server/vertica/Catalog/CatalogLookup.cpp, Line: 4108, Error Code: 4566, SQL: "SELECT pg_get_viewdef(c.oid) view_def FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'cognos_schema' AND c.relname = 'VNUM' AND c.relkind IN ('v', 'm')" .

We are trying to develop vertica connector and right now datahub ingests objects one after the other hence increasing the ingestion time. Is there a way we can ingest all the metadata at once? (The goal is to make the connector performant and reduce the ingestion time).

Hello, we are on the verge of submitting a new PR for the enhanced Vertica connector. We would like to have the support status of our connector to be "Certified". Apart from adding this variable in the code, what is the criteria to change the status on the official Datahub docs.

has renamed the channel from "integration-vertica-datahub" to "integrate-vertica-datahub"