Hi all, I’m new to the DataHub world and am trying to deploy v0.8.35 using Helm charts via Terraform. I’m deploying on an EKS cluster, using Confluent Cloud + MySQL + Elasticsearch (for both search & graph). I keep hitting errors on datahub-gms, specifically on readiness & liveness probe failures (details below). I might just be missing something basic, but could someone help me figure out what else I can try? I tried setting the following thresholds but they didn’t seem to change anything:

datahub-gms:
  livenessProbe:
    initialDelaySeconds: 60
    periodSeconds: 30
    failureThreshold: 80
  readinessProbe:
    initialDelaySeconds: 60
    periodSeconds: 30
    failureThreshold: 80

Thanks in advance!

Can you attach the gms logs? Seems like it is struggling

Would you want the k8s pod logs for the gms deployment? Or are there other logs I can share?

Pod logs for gms!

Actually, I’m no longer hitting this issue - but I’m getting stuck at the same place as this thread where I can’t update the Kafka SASL configuration. Full logs here:

[2022-06-02 14:01:04,762] ERROR    {datahub.entrypoints:138} - File "/usr/local/lib/python3.9/site-packages/datahub/cli/ingest_cli.py", line 77, in run
    67   def run(config: str, dry_run: bool, preview: bool, strict_warnings: bool) -> None:
 (...)
    73       pipeline_config = load_config_file(config_file)
    74   
    75       try:
    76           logger.debug(f"Using config: {pipeline_config}")
--> 77           pipeline = Pipeline.create(pipeline_config, dry_run, preview)
    78       except ValidationError as e:

File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 176, in create
    172  def create(
    173      cls, config_dict: dict, dry_run: bool = False, preview_mode: bool = False
    174  ) -> "Pipeline":
    175      config = PipelineConfig.parse_obj(config_dict)
--> 176      return cls(config, dry_run=dry_run, preview_mode=preview_mode)

File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 122, in __init__
    106  def __init__(
    107      self, config: PipelineConfig, dry_run: bool = False, preview_mode: bool = False
    108  ):
 (...)
    118      )
    119  
    120      source_type = self.config.source.type
    121      source_class = source_registry.get(source_type)
--> 122      self.source: Source = source_class.create(
    123          self.config.source.dict().get("config", {}), self.ctx

File "/usr/local/lib/python3.9/site-packages/acryl_action_fwk/source/datahub_streaming.py", line 185, in create
    180  def create(cls, config_dict, ctx):
    181      config = DataHubStreamSourceConfig.parse_obj(config_dict)
    182      assert (
    183          "mae" in config.topic_routes
    184      ), "topic_routes must contain an entry for mae (the metadata audit event topic)"
--> 185      return cls(config, ctx)

File "/usr/local/lib/python3.9/site-packages/acryl_action_fwk/source/datahub_streaming.py", line 142, in __init__
    136  def __init__(self, config: DataHubStreamSourceConfig, ctx: PipelineContext):
 (...)
    138      self.source_config: DataHubStreamSourceConfig = config
    139      self.schema_registry_client = SchemaRegistryClient(
    140          {"url": self.source_config.connection.schema_registry_url}
    141      )
--> 142      self.consumer: confluent_kafka.Consumer = confluent_kafka.DeserializingConsumer(
    143          {

File "/usr/local/lib/python3.9/site-packages/confluent_kafka/deserializing_consumer.py", line 103, in __init__
    98   def __init__(self, conf):
    99       conf_copy = conf.copy()
    100      self._key_deserializer = conf_copy.pop('key.deserializer', None)
    101      self._value_deserializer = conf_copy.pop('value.deserializer', None)
    102  
--> 103      super(DeserializingConsumer, self).__init__(conf_copy)

---- (full traceback above) ----
File "/usr/local/lib/python3.9/site-packages/datahub/cli/ingest_cli.py", line 77, in run
    pipeline = Pipeline.create(pipeline_config, dry_run, preview)
File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 176, in create
    return cls(config, dry_run=dry_run, preview_mode=preview_mode)
File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 122, in __init__
    self.source: Source = source_class.create(
File "/usr/local/lib/python3.9/site-packages/acryl_action_fwk/source/datahub_streaming.py", line 185, in create
    return cls(config, ctx)
File "/usr/local/lib/python3.9/site-packages/acryl_action_fwk/source/datahub_streaming.py", line 142, in __init__
    self.consumer: confluent_kafka.Consumer = confluent_kafka.DeserializingConsumer(
File "/usr/local/lib/python3.9/site-packages/confluent_kafka/deserializing_consumer.py", line 103, in __init__
    super(DeserializingConsumer, self).__init__(conf_copy)

KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create consumer: No provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN SASL_SCRAM OAUTHBEARER"}

Can you try following this thread and see if this helps? https://datahubspace.slack.com/archives/CUMUWQU66/p1651531196305699

So our datahub-actions container will not support custom Kafka properties by default… however you can mount an actions file into the container which has any arbitrary Kafka configs

Got it, thanks @brainy-shampoo-83265 for confirming. Is what you’re suggesting the same as what you shared on the thread that Dexter mentioned above?

@breezy-agent-56904 any chance you can open source/share your Terraform? We're about to embark on the same course and this would save us a ton of time, as I'm sure you know! Would be cool to have a community created/managed terraform to quickly get a production instance of Datahub up

Hi @breezy-agent-56904, we are having the same issue with

daathub-gms

node (readiness & liveness probe failures). can you please share your helm chart values file