Hi - I came back to check out datahub after a hiatus. I pulled the latest docker image and did a quickstart. When I tried an ingestion (Snowflake source) that used to work earlier, it fails with the following error. Any leads? As such I verified the ingestion configuration with the examples and nothing jumps out.

2022-02-17 10:54:02,004] ERROR    {datahub.entrypoints:119} - Stackprinter failed while formatting <FrameInfo /usr/local/lib/python3.9/site-packages/datahub/ingestion/source/sql/sql_common.py, line 221, scope SQLAlchemyConfig>:
  File "/usr/local/lib/python3.9/site-packages/stackprinter/frame_formatting.py", line 224, in select_scope
    raise Exception("Picked an invalid source context: %s" % info)
Exception: Picked an invalid source context: [221], [192], dict_keys([192, 193])

So here is your original traceback at least:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/datahub/cli/ingest_cli.py", line 77, in run
    pipeline = Pipeline.create(pipeline_config, dry_run, preview)
  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 175, in create
    return cls(config, dry_run=dry_run, preview_mode=preview_mode)
  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 120, in __init__
    source_class = source_registry.get(source_type)
  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/api/registry.py", line 126, in get
    tp = self._ensure_not_lazy(key)
  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/api/registry.py", line 84, in _ensure_not_lazy
    plugin_class = import_path(path)
  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/api/registry.py", line 32, in import_path
    item = importlib.import_module(module_name)
  File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/source/sql/snowflake.py", line 29, in <module>
    from datahub.ingestion.source.sql.sql_common import (
  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/source/sql/sql_common.py", line 206, in <module>
    class SQLAlchemyConfig(StatefulIngestionConfigBase):
  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/source/sql/sql_common.py", line 221, in SQLAlchemyConfig
    from datahub.ingestion.source.ge_data_profiler import GEProfilingConfig
  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 27, in <module>
    from great_expectations.core.util import convert_to_json_serializable
  File "/usr/local/lib/python3.9/site-packages/great_expectations/__init__.py", line 7, in <module>
    from great_expectations.data_context import DataContext
  File "/usr/local/lib/python3.9/site-packages/great_expectations/data_context/__init__.py", line 1, in <module>
    from .data_context import BaseDataContext, DataContext, ExplorerDataContext
  File "/usr/local/lib/python3.9/site-packages/great_expectations/data_context/data_context.py", line 23, in <module>
    from great_expectations.rule_based_profiler.config.base import (
  File "/usr/local/lib/python3.9/site-packages/great_expectations/rule_based_profiler/__init__.py", line 1, in <module>
    from .rule_based_profiler import RuleBasedProfiler
  File "/usr/local/lib/python3.9/site-packages/great_expectations/rule_based_profiler/rule_based_profiler.py", line 16, in <module>
    from great_expectations.rule_based_profiler.domain_builder.domain_builder import (
  File "/usr/local/lib/python3.9/site-packages/great_expectations/rule_based_profiler/domain_builder/__init__.py", line 1, in <module>
    from .domain_builder import DomainBuilder  # isort:skip
  File "/usr/local/lib/python3.9/site-packages/great_expectations/rule_based_profiler/domain_builder/domain_builder.py", line 6, in <module>
    from great_expectations.rule_based_profiler.types import (
  File "/usr/local/lib/python3.9/site-packages/great_expectations/rule_based_profiler/types/__init__.py", line 3, in <module>
    from .domain import (  # isort:skip
  File "/usr/local/lib/python3.9/site-packages/great_expectations/rule_based_profiler/types/domain.py", line 8, in <module>
    from great_expectations.execution_engine.execution_engine import MetricDomainTypes
  File "/usr/local/lib/python3.9/site-packages/great_expectations/execution_engine/__init__.py", line 4, in <module>
    from .sqlalchemy_execution_engine import SqlAlchemyExecutionEngine
  File "/usr/local/lib/python3.9/site-packages/great_expectations/execution_engine/sqlalchemy_execution_engine.py", line 97, in <module>
    import pybigquery.sqlalchemy_bigquery
  File "/usr/local/lib/python3.9/site-packages/pybigquery/sqlalchemy_bigquery.py", line 32, in <module>
    from google.cloud.bigquery import dbapi
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/__init__.py", line 35, in <module>
    from google.cloud.bigquery.client import Client
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 70, in <module>
    from google.cloud.bigquery import _pandas_helpers
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 67, in <module>
    from google.cloud.bigquery import schema
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/schema.py", line 20, in <module>
    from google.cloud.bigquery_v2 import types
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery_v2/__init__.py", line 18, in <module>
    from .types.encryption_config import EncryptionConfiguration
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery_v2/types/__init__.py", line 16, in <module>
    from .encryption_config import EncryptionConfiguration
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery_v2/types/encryption_config.py", line 26, in <module>
    class EncryptionConfiguration(proto.Message):
  File "/usr/local/lib/python3.9/site-packages/proto/message.py", line 200, in __new__
    file_info = _file_info._FileInfo.maybe_add_descriptor(filename, package)
  File "/usr/local/lib/python3.9/site-packages/proto/_file_info.py", line 42, in maybe_add_descriptor
    descriptor=descriptor_pb2.FileDescriptorProto(
TypeError: descriptor to field 'google.protobuf.FileDescriptorProto.name' doesn't apply to 'FileDescriptorProto' object

This seems to be a bigquery source stacktrace. Are you sure this from snowflake source?

Yes. I dont have any biqquery configs. Made sure to pass a filename that had the following config.

source:
  type: snowflake
  config:
    host_port: blah
    warehouse: "BLAH"
    username: BLAH
    password: blah
    role: "BLAH"
    database_pattern:
        allow: 
            - "BLAH"
    include_table_lineage: True

sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"

Is this the only error message you can see?

Yes. That's the whole stacktrace that I see.

It should be fine if you delete your virtualenv and recreate it

Thanks. Figured out the issue. I had previously not used a virtualenv based setup. So the install script that time had the

datahub

cli entry point command copied to the generic

bin

directory. I guess this was perhaps an older version of the command and caused such errors. I removed it after uninstalling everything and started fresh. No issues like the above now.

Awesome, thanks for letting know what was issue