Hi team I’m trying to deploy a slim NS from Datahub - just with Elasticsearch & Kafka When deploying Datahub charts, I’m facing the following error, from

datahub-datahub-upgrade-job-xxxxx:

Error: secret "mysql-secrets" not found

I presume this error occurs because I need a DB for the app(pgSQL / MySQL). I want to use a DB that is provided by my cloud vendor(AWS) & not from the app deployment Obviously, I’m customizing the

values.yaml

both for prereq & datahub Pls. advise 10x 🙏

Hi Aviv, have you created the secret within DataHub itself?

Hi Paul No, I didn’t create the secret in DH. I am using the pre-requisites helm chart, but with no mysql/pgsql

Hi @powerful-cat-68806, what is the datastore you are doing? You’re going to need to manually create that secret. In this values file you can overwrite the secret name to not say “mysql” https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml

@astonishing-answer-96712 hi 🙂 I want to have an external link to pgSQL DB I’m managing from RDS(aws). I don’t want to use the default DBs from the chart itself Possible?

@bulky-electrician-72362 ^

hi @powerful-cat-68806, Sure, you can use your own DB instance. Here is an example configuration: https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L180 If you run the prerequisite chart, you can disable the mysql instance there, as it won't be in use.

@bulky-electrician-72362 @loud-island-88694 @astonishing-answer-96712 hi guys On DH deployment, I’m getting the following error:

Warning  Failed     18m (x12 over 20m)    kubelet            Error: secret "neo4j-secrets" not found

This is for the pods

datahub-datahub-gms-xxxxx-xx

&

datahub-datahub-upgrade-job-xxxx

Where should I remove it in the code, to avoid this error? 10x guys 😉

@orange-night-91387, any ideas what might be the problem here?

Hi Aviv, have you followed the instructions in the README.md for your deployment? Specifically:

The dependencies must be deployed before deploying Datahub. We created a separate chart for deploying the dependencies with example configuration. They could also be deployed separately on-prem or leveraged as managed services. To remove your dependency on Neo4j, set enabled to false in the datahub-kubernetes/prerequisites/values.yaml file. Then, override the graph_service_impl field in datahub-kubernetes/datahub/values.yaml to have the value elasticsearch instead of neo4j.

If you are deploying your own ElasticSearch & Neo4J you will need to set up relevant secrets for the credentials required to connect to those services. For ElasticSearch we have a guide for setting up the secrets here, for Neo4J you will need to set up the secrets in a similar manner, but using the name you got in the above error:

neo4j-secrets

. The quickstart doc has an example specific for Neo4J using the default value for non-prod deploys.

Hi @orange-night-91387, how are you? I did disable Neo4J on the prerequisites file I am using the chart for Elasticsearch, but I don’t want to have Neo4j

Doing well thanks 🙂 Have you also disabled it on the main values.yaml? Specifically

global.neo4j.enabled=false

on both prerequisites values.yaml and datahub values.yaml. The only reference to neo4j-secrets should be locked behind that section, so with it disabled it shouldn't try to access that secret.

global.neo4j.enabled=false

I don’t see this in the values.yaml file In prerequisite I see:

Official neo4j chart uses the Neo4j Enterprise Edition which requires a license
neo4j:
  enabled: false  # set this to true, if you have a license for the enterprise edition
  acceptLicenseAgreement: "yes"
  defaultDatabase: "graph.db"
  neo4jPassword: "datahub"
  # For better security, add password to neo4j-secrets k8s secret and uncomment below
  # existingPasswordSecret: neo4j-secrets
  core:
    standalone: true

# Deploys neo4j community version. Only supports single node
neo4j-community:
  enabled: false   # set this to false, if you have a license for the enterprise edition
  acceptLicenseAgreement: "yes"
  defaultDatabase: "graph.db"
  # For better security, add neo4j-secrets k8s secret with neo4j-password and uncomment below
  existingPasswordSecret: neo4j-secrets

Instead of commenting it out do:

neo4j:
  enabled: false

Like you have in the prerequisites

@orange-night-91387 Still the same

Looks like you still have neo4j as the graph_service_impl above, is that intended? Or are you using ElasticSearch for your graph service?:

Then, override the graph_service_impl field in datahub-kubernetes/datahub/values.yaml to have the value elasticsearch instead of neo4j.

From docs linked above

I’m less familiar with this What graph service?

global:
  graph_service_impl: neo4j
  datahub_analytics_enabled: true
  datahub_standalone_consumers_enabled: false

Should I comment it out, or add a line enabled: false?

DataHub uses a metadata graph to back some of the queries we execute. This graph can either be implemented in ElasticSearch or Neo4J, but it's not an optional component. It can use the same existing ElasticSearch deployment that is used for the entity/timeseries search indices.

Ok So - how can I make sure it’s not breaking the deployment?

If you are intending to only deploy with ElasticSearch as a dependency and not Neo4J, you will need to change the graph service implementation field to "elasticsearch"

global:
  graph_service_impl: elasticsearch
  datahub_analytics_enabled: true
  datahub_standalone_consumers_enabled: false

?

Yep

10x Trying

Neo4J is a Graph database that serves as one implementation of a graph store. We also support using ElasticSearch as a graph database.

datahub-datahub-upgrade-job-z4d9j                  0/1     CreateContainerConfigError   0             14m

My values.yaml is set to a pgSQL db that I’m managing

That's the same error as the original post, did you make the recommended modification from Paul and Peter above?

@orange-night-91387 it’s the same. just for MySQL https://datahubspace.slack.com/archives/C029A3M079U/p1671613258996999?thread_ts=1669890797.442779&cid=C029A3M079U

I'm talking about the initial thread post: https://datahubspace.slack.com/archives/C029A3M079U/p1669890797442779 which Paul & Peter gave suggestions on how to solve above, did you take action on those? Specifically:

Paul Logan
  20 days ago

You can change the name/info on line 175
Paul Logan
  20 days ago

password:
        secretRef: mysql-secrets
        secretKey: mysql-root-password
Paul Logan
  20 days ago

there's also more info in the comments under those lines
Paul Logan
  20 days ago

<https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L175>

and

Peter Szalai
  17 days ago

hi 
@Aviv Cohen
,
Sure, you can use your own DB instance. Here is an example configuration: <https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L180>
If you run the prerequisite chart, you can disable the mysql instance there, as it won't be in use.

If you have done so, please replace the secret reference in the main datahub chart to the secret name you have manually created.

hey Aviv, have you run

helm upgrade

? yes,

datahubUpgrade

shouldn't be disabled

Hi @bulky-electrician-72362 🙂 Yes I did, but it keeps failing for the

mysql-secret

you can specify in this part how you want to provide the mysql secret: https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L177 you can use value or a k8s secret ref.

@bulky-electrician-72362 I’ve added my own pgSQL configuration in

values.yaml

you need to create a secret on k8s manually or add the password to the values.yaml

I did it @bulky-electrician-72362 When adding my pgSQL conf(AWS RDS), I set it from line 204

password:
#         secretRef: postgresql-secrets
#         secretKey: postgres-password
#       --------------OR----------------
         value: 'xxxxxxxxxxxxx'

in the datahubUpgrade job you have sqlDbType: Postgres, but you job is missing mysql-secret? which one are you using?

You need to do the same replacement of "mysql-secret" for DataHub Upgrade that you did for PostgreSQL Setup. Rather than comment it out, change the name to your actual secret you created per the previous instructions.

@bulky-electrician-72362 @orange-night-91387 hi guys Can we Zoom? It’ll ease the process a lot 🙂

sure, we can