Hello slightly smiling face l m tyring to use ClouSQL as dat DataHub #troubleshoot

Hello :slightly_smiling_face: l'm tyring to use Cl...

nice-planet-17111

10/21/2021, 8:26 AM

Hello 🙂 l'm tyring to use ClouSQL as datahub storage.. I was trying to connect via CloudSQL auth proxy (i edited

datahub-gms

and

datahub-upgrade-job

deployment, and when i helm update, upgrade job fails and i get

Error creating bean with name 'upgradeCli': Unsatisfied dependency expressed through field 'noCodeUpgrade'

error 😞 Is there something i can do? (error log and helm chart is in the thread because it's too long)

✅ 1

nice-planet-17111

10/21/2021, 8:28 AM

Copy code

ERROR SpringApplication Application run failed
 org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'upgradeCli': Unsatisfied dependency expressed through field 'noCodeUpgrade'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanAspectDao' defined in class path resource [com/linkedin/gms/factory/entity/EbeanAspectDaoFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.entity.ebean.EbeanAspectDao]: Factory method 'createInstance' threw exception; nested exception is java.lang.NullPointerException

nice-planet-17111

10/21/2021, 8:30 AM

This is the edited datahub-upgrade-job deployment,

Copy code

env:
            - name: ENTITY_REGISTRY_CONFIG_PATH
              value: /datahub/datahub-gms/resources/entity-registry.yml
            - name: DATAHUB_GMS_HOST
              value: {{ printf "%s-%s" .Release.Name "datahub-gms" }}
            - name: DATAHUB_GMS_PORT
              value: "{{ .Values.global.datahub.gms.port }}"
            - name: DATAHUB_MAE_CONSUMER_HOST
              value: {{ printf "%s-%s" .Release.Name "datahub-mae-consumer" }}
            - name: DATAHUB_MAE_CONSUMER_PORT
              value: "{{ .Values.global.datahub.mae_consumer.port }}"
            - name: EBEAN_DATASOURCE_USERNAME
              valueFrom:
                secretKeyRef:
                  name: cloudsql-secrets
                  key: username
            - name: EBEAN_DATASOURCE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql-secrets
                  key: password
            - name: EBEAN_DATASOURCE_HOST
              value: <INSTANCE_CONNECTION_NAME>
            - name: EBEAN_DATASOURCE_URL
              value: <INSTANCE_CONNECTION_NAME>
            - name: EBEAN_DATASOURCE_DRIVER
              value: "{{ .Values.global.sql.datasource.driver }}"

nice-planet-17111

10/21/2021, 9:51 AM

nvm i solved the problem ( i used private ip instead of auth proxy)

✅ 1

red-waitress-53338

01/13/2023, 2:22 AM

Hi @nice-planet-17111 Can you please tell me how you solved this issue with private IP?

nice-planet-17111

01/13/2023, 3:20 AM

Hi @red-waitress-53338 , It goes something like this. Private ip address goes in

sql.datasource.host

(In the real chart, it references secret that is deployed separately, but I simplified the chart to help understading.

Copy code

# charts/datahub/values.yaml
sql:
    datasource:
      host: <cloudsql_host_address>:<port>
      url: "jdbc:mysql://<cloudsql_host_address>:<port>/<cloudsql_db_name>?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2"
      hostForMysqlClient: <cloudsql_host_address>
      port: <port>
      driver: "com.mysql.jdbc.Driver"
      username: <username>
      password:
        secretRef: <secret-name>
        secretKey: <secret-key>

Copy code

# charts/datahub/subcharts/datahub-gms/values.yaml
sql:
  datasource:
    host: <cloudsql_host_address>:<port>
    url: "jdbc:mysql://<cloudsql_host_address>:<port>/datahub?verifyServerCertificate=false&useSSL=true"
    driver: "com.mysql.jdbc.Driver"
    username: <username>
    password:
      secretRef: <secret-name>
      secretKey: <secret-key

You can refer the blog post i wrote on deploying datahub on GKE with CloudSQL. (It’s in Korean, but i think you can still check the code blocks 🙂 ) - Hope this helps!

red-waitress-53338

01/13/2023, 5:50 PM

Thanks much @nice-planet-17111 for sharing all the information. Actually I am not using Kuburnetes for the DataHub installation. I am using datahub docker images to let them run on CloudRun. I think I have to make some changes the DataHub GMS env file but I am not sure what changes do I need to make. Any idea where should I make the changes in the below GMS env config file?

Copy code

EBEAN_DATASOURCE_USERNAME=datahub
EBEAN_DATASOURCE_PASSWORD=datahub
EBEAN_DATASOURCE_HOST=mysql:3306
EBEAN_DATASOURCE_URL=jdbc:<mysql://mysql:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8>
EBEAN_DATASOURCE_DRIVER=com.mysql.jdbc.Driver

nice-planet-17111

01/17/2023, 7:40 AM

Hmmm.. Have you tried replacing

mysql

to private IP address in EBEAN_DATASOURCE_HOST / EBEAN_DATASOURCE_URL? Maybe it should be something like this.

Copy code

EBEAN_DATASOURCE_USERNAME=<username>
EBEAN_DATASOURCE_PASSWORD=<password>
EBEAN_DATASOURCE_HOST=<private ip>:3306
EBEAN_DATASOURCE_URL=jdbc:mysql://<private ip>:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8
EBEAN_DATASOURCE_DRIVER=com.mysql.jdbc.Driver

red-waitress-53338

01/22/2023, 2:12 AM

thanks @nice-planet-17111

Open in Slack

Previous Next