https://datahubproject.io logo
Join Slack
Powered by
Hi, We have deployed the Datahub on GKE and flagge...
# troubleshoot
q
Hi, We have deployed the Datahub on GKE and flagged log4j vulnerability on the Neo4j : The library 
org.apache.logging.log4j:log4j-core
version 
2.14.0
was detected in 
Maven library manager
located at 
/var/lib/neo4j/lib/neo4j-logging-4.2.4.jar
and is vulnerable to 
CVE-2021-45105
, which exists in versions 
>= 2.13.0, < 2.17.0
. The vulnerability was found in the Github Security Advisory with vendor severity: 
High
(NVD severity: 
Medium
). This vulnerability has a known exploit available. Source: Github. The vulnerability can be remediated by updating the library to any of the following versions 
2.3.1, 2.12.3, 2.17.0
or higher, using 
mvn versions:use-latest-releases -Dincludes=org.apache.logging.log4j:log4j-core
. The library is associated with the technology 
Apache Log4j
. Is there a fix as part of deployment ?
b
Can you share the version and which docker image?
q
The Version of datahub is V0.9.5 with image as image: neo4j:4.2.4, deployed by following https://datahubproject.io/docs/deploy/kubernetes
b
DataHub itself doesn’t produce or maintain the neo4j image and it is optional (since Elasticsearch can be used instead). The helm chart includes the latest version of the community helm chart, here (4.2.4) as you mentioned. This hasn’t been updated since June 2021 and may not be maintained. You have the option to either use Elasticsearch as the graph database or to pursue the enterprise Neo4j which may have the security updates and can be used if you have the license with our charts here.
If you know of a neo4j community supported image that is fixed, we could look at replacing the one linked to in our helm charts.
q
From performance perspective, what will be difference if we use ES rather than no4j community ? Also also do you have reference of changes that are required to be made to use ES instead of neo4j ?
b
We only run Elasticsearch backed instances in our environments. I can assure you that ES scales to millions of entities. The steps would be to disable neo4j in your prerequisites by setting the value here and selecting 
elasticsearch
here for the graph implementation. This configuration is documented here.
q
let us enable the ES, thanks for the input
Hi @brainy-tent-14503, We followed the documentation and changed it from Neo4j to ES. However still the Neo4j container is showing up. We already have data present on the datahub. If we delete the the neo4j, will it impact the existing data. What is the standard practice ? cc: @bland-artist-80619
b
Please follow the instructions in the previously linked documentation to update Elasticsearch, specifically the link to restoreIndices. Ultimately the data is in your sql instance, however it does need to be created in Elasticsearch, specifically the graph_service index. It is ok to remove neo4j instance after moving to ES for the graph implementation.
b
Hey @brainy-tent-14503, Thanks for the Doc, As per the Doc, 4 steps 1. To turn off neo4j in your prerequisites file, set `neo4j-community`’s 
enabled
property to 
false
in this values.yaml. (screenshot-1) 2. Then, set 
graph_service_impl
to 
elasticsearch
in the values.yaml of datahub. (screenshot-2) 3. re run helm commands a. helm upgrade prerequisites datahub/datahub-prerequisites -n datahub (screenshot-3) b. helm upgrade datahub datahub/datahub -n datahub (screenshot-3) 4. Finally, follow the restore-indices helm guide to re-build your graph index. a. kubectl get cronjobs -n datahub (screenshot-4) b. Also tried to create new job to test the restoration, which did not help though. All the steps are performed and then after deleting prerequisites-neo4j-community datahub is down
b
Can you get the logs from GMS? Let’s start there. GMS needs to be running before you start the restoreIndices job. If GMS is healthy, then lets look at the restoreIndices job’s logs.
b
Attaching the logs of GMS
downloaded-logs-20230120-203115.json
The datahub-datahub-restore-indices-job-template seems having issue
b
GMS logs look ok, I don’t see any errors and looks like normal ingestion proposals. The error is non-fatal it continues to log to the console. Is there any further logs?
Is there a failed job with additional details around? I think it keeps the last failed job so you can inspect it. What you’ve got in the screenshot is the template and it must be fetching that log from an actual job that the template ran.
Also what do you mean by 
datahub is down
? Is the UI not responding or missing datasets?
b
Hi @brainy-tent-14503, GMS seems to be fine. attaching all the logs, pls check
downloaded-logs-20230123-142224.csv
Also attaching the job logs here
downloaded-logs-20230123-142435.csv
Datahub is down
- I mean the UI is fine, but the datasets and pipelines are giving error, when the neo4j job is deleted. If I redeploy them using helm, I get those back.
b
Definitely, GMS looks fine. The logs for the job just stops there is odd. The status on the pod for the job? Like from the cli `kubectl -n <namespace> describe pod <pod name for job> might have some insights into what happened. Perhaps an OOM that killed it?
b
Copy code
HXA5QOE@PX7KG2QXL9MBP devops % kubectl describe pod datahub-datahub-restore-indices-job-template-manual-kqz5k-94qlz -n datahub
W0123 15:35:12.225390   10351 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult <https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke>
Name:             datahub-datahub-restore-indices-job-template-manual-kqz5k-94qlz
Namespace:        datahub
Priority:         0
Service Account:  default
Node:             gke-gke-ml-dev-default-gke-pool-26c067d2-emye/10.1.0.64
Start Time:       Fri, 20 Jan 2023 20:31:52 -0400
Labels:           controller-uid=a4594fa5-49f6-477b-b60c-328cf627cb16
                  job-name=datahub-datahub-restore-indices-job-template-manual-kqz5k
Annotations:      <http://cni.projectcalico.org/containerID|cni.projectcalico.org/containerID>: e5e8365d1e894bf97e1065f01c67afbd644065363a855fdb8de0d396d3b51976
                  <http://cni.projectcalico.org/podIP|cni.projectcalico.org/podIP>: 
                  <http://cni.projectcalico.org/podIPs|cni.projectcalico.org/podIPs>: 
Status:           Succeeded
IP:               10.1.41.59
IPs:
  IP:           10.1.41.59
Controlled By:  Job/datahub-datahub-restore-indices-job-template-manual-kqz5k
Containers:
  datahub-upgrade-job:
    Container ID:  <containerd://6a2bd485a7ecf54d241ac3e39e3c0c261b6b46732e93c08d99ffaa436fb4504>c
    Image:         acryldata/datahub-upgrade:v0.9.6.1
    Image ID:      <http://docker.io/acryldata/datahub-upgrade@sha256:7d3f1d8b91d9a61b132894221395903403c42b907c9ebcde826eb3b9188373fb|docker.io/acryldata/datahub-upgrade@sha256:7d3f1d8b91d9a61b132894221395903403c42b907c9ebcde826eb3b9188373fb>
    Port:          <none>
    Host Port:     <none>
    Args:
      -u
      RestoreIndices
      -a
      batchSize=1000
      -a
      batchDelayMs=100
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 20 Jan 2023 20:31:53 -0400
      Finished:     Fri, 20 Jan 2023 20:33:47 -0400
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     300m
      memory:  256Mi
    Environment:
      ENTITY_REGISTRY_CONFIG_PATH:  /datahub/datahub-gms/resources/entity-registry.yml
      DATAHUB_GMS_HOST:             datahub-datahub-gms
      DATAHUB_GMS_PORT:             8080
      DATAHUB_MAE_CONSUMER_HOST:    datahub-datahub-mae-consumer
      DATAHUB_MAE_CONSUMER_PORT:    9091
      EBEAN_DATASOURCE_USERNAME:    root
      EBEAN_DATASOURCE_PASSWORD:    <set to the key 'mysql-root-password' in secret 'mysql-secrets'>  Optional: false
      EBEAN_DATASOURCE_HOST:        prerequisites-mysql:3306
      EBEAN_DATASOURCE_URL:         jdbc:<mysql://prerequisites-mysql:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2>
      EBEAN_DATASOURCE_DRIVER:      com.mysql.cj.jdbc.Driver
      KAFKA_BOOTSTRAP_SERVER:       prerequisites-kafka:9092
      KAFKA_SCHEMAREGISTRY_URL:     <http://prerequisites-cp-schema-registry:8081>
      ELASTICSEARCH_HOST:           elasticsearch-master
      ELASTICSEARCH_PORT:           9200
      SKIP_ELASTICSEARCH_CHECK:     false
      ELASTICSEARCH_INSECURE:       false
      ELASTICSEARCH_USE_SSL:        false
      GRAPH_SERVICE_IMPL:           neo4j
      NEO4J_HOST:                   prerequisites-neo4j-community:7474
      NEO4J_URI:                    <bolt://prerequisites-neo4j-community>
      NEO4J_USERNAME:               neo4j
      NEO4J_PASSWORD:               <set to the key 'neo4j-password' in secret 'neo4j-secrets'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-84dd9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-84dd9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:                      <none>
HXA5QOE@PX7KG2QXL9MBP devops %
From the describe, I see its still connecting to mysql, which i have disabled in values to use the postgres:
b
Yes, I see the restore indices job doesn’t use the global settings. While it is still possible to override the via environment variables, created a PR to fix this here
b
Hello Good Morning @brainy-tent-14503, The neo4j issue is resolved, Thanks. But now Im trying to connect to Google Cloud SQL instead of using the - mysql pod in k8. There is an issue with helm I suppose, whenever I try to change the variables in values.yaml in 1. datahub-helm/charts/datahub/values.yaml 2. datahub-helm/charts/datahub/subcharts/datahub-gms/values.yaml (Also tried changing in datahub-helm/charts/datahub/subcharts/datahub-mce-consumer/values.yaml) When I apply helm upgrade it takes the original prerequisite-mysql values and prerequisite-mysql statefulset and upgrade job will get deployed in GKE. I tried changing the values in the above files and do a “Helm Template” to confirm, the output yaml’s also have the default values. Attaching the screenshots. (Tried with ip address and connection ID in values.yaml, both did not apply - however, when i try to apply the connection id directly on GKE yaml, GMS yaml has taken, but it wont be possible for the other components to edit) Could you check the helm please! Need help!
b
I can’t tell from the screenshot, however the top level values.yaml (#1) should be nested under 
global:
, no need to edit #2. Also please share your helm command or check to make sure you are using the 
-f
flag on the file with your changes. 
-f myvals.yaml
or 
-f values.yaml
if you just copied the default file and edited it.
Also ensure you don’t have a cached older version of the chart
b
helm install prerequisites datahub/datahub-prerequisites -n datahub OR helm upgrade prerequisites datahub/datahub-prerequisites -n datahub AND helm install datahub datahub/datahub -n datahub OR helm upgrade datahub datahub/datahub -n datahub
I made sure the old helm is deleted before every new implementation
cc: @quaint-barista-82836
b
To use your specific configuration, 
-f/--values myvalues.yaml
flag, per the helm docs here. The name of the yaml file doesn’t matter, however if you are not specifying your overrides, then it will use the default values.yaml that is published with the chart.
b
Yep, tried with
Copy code
helm upgrade -n datahub prerequisites datahub/datahub-prerequisites -f /Users/HXA5QOE/Downloads/datahub-helm/charts/datahub/values.yaml


helm upgrade -n datahub datahub datahub/datahub -f /Users/HXA5QOE/Downloads/datahub-helm/charts/datahub/values.yaml
b
Chart version from 
helm list -n datahub
?
To update the chart repo, if it is an older chart version: 
helm repo update
b
image.png
b
You’re up to date then, so that’s good. You can check the rendered manifests with 
helm template
with the values flag, will print the raw manifests. If you’re seeing defaults then note which manifest(s), but the mce template is pulling from the global parameter now.
b
Yes David. it is working fine now. thank!
Open in Slack
PreviousNext
Powered by