Hi team, The security team of my company recently asked a question whether the link should be public: Open the following link in an incognito session: https://datahub-test.company.com/assets/static/js/main.322141e4.chunk.js Can you please confirm: 1. If there is any sensitive information? (Ex. passwords, keys, customer or user data, etc) 2. If this is indeed supposed to be publicly available or not? Thanks!

Hi all, none of my users are able to login. GMS logs reveal this. Running v0.9.5 deployed with k8s. No major changes recently all of a sudden just stopped working

21:56:03.265 [ForkJoinPool.commonPool-worker-5] ERROR c.d.a.a.AuthServiceController:314 - Failed to verify credentials for native user urn:li:corpuser:jbolesjc
java.lang.RuntimeException: Failed to decrypt value using provided secret!
	at com.linkedin.metadata.secret.SecretService.decrypt(SecretService.java:80)
	at com.datahub.authentication.user.NativeUserService.doesPasswordMatch(NativeUserService.java:200)
	at com.datahub.auth.authentication.AuthServiceController.lambda$verifyNativeUserCredentials$3(AuthServiceController.java:310)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
	at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:975)
	at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1056)
	at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853)
	at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
	at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2202)
	at com.linkedin.metadata.secret.SecretService.decrypt(SecretService.java:78)
	... 9 common frames omitted

Hey everyone, we want to run the datahub frontend on the URL https://<<dns-name>>*/datahub* Can someone help me to setup the ingress so this can work out? We tried setting up our own ingress which defines the /datahub path to point to our datahub-frontend which is running in a kubernetes cluster. When doing this, we get the following error message: Ingress config looks like this:

- path: /datahub
	pathType: Prefix
	backend:
	  service:
		name: datahub-service
		port:
		  number: 9002

Hey , please help me!!!!!!!!!!!!!!!!!

Is there any way to find back up versions of datahub?

please help me

😞

i am using datahub quickstart and have backed up datahub before nuke, but unfortunately after nuke i back up again. Is there any way i can restore the old backup before nuke??? Please help me

Does data still exist in mysql,kafka,es...??

Please help me if i can't solve this i will be layed off

Hey everyone. Why is Datahub failing now that you can’t even install a version from scratch. Are there problems with the repositories?

There is no internet problem and it is a clean installation. It is an instance with 32GB of RAM on AWS.

reboot and try:

🧵 Hello everyone, we are trying to deploy DataHub on a VM and have followed the steps listed in the documentation, but have been stuck at the same point. I can get it installed on the VM without any issues, but when I try to deploy the DataHub instance locally, it fails with the following error -

Unable to run quickstart - the following issues were detected:
- quickstart.sh or dev.sh is not running

If you think something went wrong, please file an issue at <https://github.com/datahub-project/datahub/issues>
or send a message in our Slack <https://slack.datahubproject.io/>
Be sure to attach the logs from /tmp/tmpbsyeu5qv.log
apuranik@cvia1dct001:~$ less /tmp/tmpbsyeu5qv.log
apuranik@cvia1dct001:~$ less /tmp/tmpbsyeu5qv.log
apuranik@cvia1dct001:~$ ls -l /tmp/tmpbsyeu5qv.log
-rw------- 1 apuranik docker 0 May 24 16:33 /tmp/tmpbsyeu5qv.log
apuranik@cvia1dct001:~$

Unfortunately, nothing is captured in the log. I was not able to find anything online that could help me with this issue. I have tried this twice (rebuilt the VM) and landed in the same spot. Any help with this will be greatly appreciated!

We are trying to deploy an ingest cron and this is our values for envFromSecrets Here are our assumptions:

envFromSecrets:
  BIGQUERY_PRIVATE_KEY: <--- environment variable
    key: some-privatekey <-- the key in the k8s-secret data file
    secret: k8s-secret <--- the name of the k8s-secret
  BIGQUERY_PRIVATE_KEY_ID:
    key: ...
    secret: k8s-secret

We also use a configmap for the recipe where we then specify

source:
      type: bigquery
      config:
          include_table_lineage: true
          include_usage_statistics: true
          include_tables: true
          include_views: true
          profiling:
              enabled: true
              profile_table_level_only: false
          stateful_ingestion:
              enabled: true
          credential:
              project_id: project_id
              private_key: ${BIGQUERY_PRIVATE_KEY}
              private_key_id: ${BIGQUERY_PRIVATE_KEY_ID}
              ...

When the pod runs we get

UnboundVariable: 'BIGQUERY_PRIVATE_KEY: unbound variable'

Can someone who knows more about this than myself advice or if you have set something like this up before any advice is welcome

Has anyone deployed datahub using ECR aand ECS in aws? Is there any resource I can look at for doing this?

Hi Datahub community friends! I am trying to deploy self hosted datahub integrating with our instance of confluent cloud. While trying to create the topics specified in the documentation in confluent via terraform, our terraform is enforcing for all topic names to be

Data name must: be all lowercase, start and end with a letter or number, and may contain letters, numbers, dashes, underscores, and dots.

I was wondering if it is okay if I created all the topics specified in the documentation as lower case and would I be able to configure Datahub to refer to the lower cased topics in the integration?

hi all, we're continuing to see

OSError: [Errno 24] Too many open files\n']

errors in the

actions

container once it has been online for a while ingesting from a glue data source. there seems to be a connection/file reference leak somewhere, any thoughts?

Hi @here. We are getting this alert with datahub 0.10.0

PyPi package potentially vulnerable to dependency confusion attack | acryl-datahub-actions

Anyone recieved this warning before?

Hello Can someone please suggest here: https://datahubspace.slack.com/archives/CV2UVAPPG/p1684831131067579

Why am I suddenly getting this error??? Please help me . The table's description page can't load anything

Hello! I am fairly new to datahub. I've installed datahub using docker-compose on an EC2 instance. I shut down this EC2 instance daily due to cost issues. So I stop datahub in the evening and start it back in the morning. However when I start datahub it always shows error in starting some service and then process goes into wait state undefinitely. I've to exit the instance and login again. It always takes so many tries to get datahub instance up and running. How can I resolve this issue?

I want to use cp-schema-registry url which is on https, but want to skip verifying the ssl for now. How should I configure in my recipe file and MCE consumer pod to skip verifying the ssl for cp-schema registry?

hi, is there a way to easily do a command that launches a datahub with a particular version? At the moment datahub docker quickstart will download and the latest images. I would like to be able to fix the deployment version, without having to handle the docker-compose files (and tag each image everytime I want to change or update)? thanks! Pablo M.

Hi all. I am currently migrating some stateful dependencies of Datahub (Postgres, Kafka, Neo4j, ES) from Kubernetes to VM and I have a few questions. My first question is about the

neo4j.uri

config in values.yaml. I see that using Kubernetes, it can be specified as

<bolt://prerequisites-neo4j-community>

without specifying port 7687. How about using host from VM which is typically an IP address? Do we need to explicitly specify the port? Something like

<bolt://172.32.31.18:7687>

Hi, what is the minimum permissions needed for Datahub to work with the Postgres/MySQL databases?

We have installed datahub on AWS using https://datahubproject.io/docs/deploy/aws. I can access the UI. But would like to use CLI and python. Where do i get the GMS url and token as there is no server.

Hello datahub team i am facing below error when i am creating ingestion with Trino. Steps I am following up from this link: https://datahubproject.io/docs/ui-ingestion 1. I have created a policy. 2. I have created a ingestion with below details

source:
    type: starburst-trino-usage
    config: null
    host_port: 'datamesh.conci**<http://usquest.com:443|usquest.com:443>'
    database: tpch
    username: ds-starburst
    password: 'lnBs****6Up'
    email_domain: ankit.rawat@co***<http://rrus.com|rrus.com>
    audit_catalog: tiny
    audit_schema: customer
sink:
    type: datahub-rest
    config:
        server: '<http://localhost:8080>'

Now i am receiving this error

~~~~ Execution Summary - RUN_INGEST ~~~~
Execution finished with errors.
{'exec_id': '85e65980-bd26-416d-9fbd-bb15840a12d3',
 'infos': ['2023-05-30 12:32:00.237740 INFO: Starting execution for task with name=RUN_INGEST',
           "2023-05-30 12:32:04.279953 INFO: Failed to execute 'datahub ingest'",
           '2023-05-30 12:32:04.280103 INFO: Caught exception EXECUTING task_id=85e65980-bd26-416d-9fbd-bb15840a12d3, name=RUN_INGEST, '
           'stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/default_executor.py", line 122, in execute_task\n'
           '    task_event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete\n'
           '    return future.result()\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 231, in execute\n'
           '    raise TaskError("Failed to execute \'datahub ingest\'")\n'
           "acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"],
 'errors': []}

~~~~ Ingestion Logs ~~~~
Obtaining venv creation lock...
Acquired venv creation lock
venv setup time = 0
This version of datahub supports report-to functionality
datahub  ingest run -c /tmp/datahub/ingest/85e65980-bd26-416d-9fbd-bb15840a12d3/recipe.yml --report-to /tmp/datahub/ingest/85e65980-bd26-416d-9fbd-bb15840a12d3/ingestion_report.json
[2023-05-30 12:32:02,233] INFO     {datahub.cli.ingest_cli:165} - DataHub CLI version: 0.10.0
7 validation errors for PipelineConfig
source -> audit_catalog
  extra fields not permitted (type=value_error.extra)
source -> audit_schema
  extra fields not permitted (type=value_error.extra)
source -> database
  extra fields not permitted (type=value_error.extra)
source -> email_domain
  extra fields not permitted (type=value_error.extra)
source -> host_port
  extra fields not permitted (type=value_error.extra)
source -> password
  extra fields not permitted (type=value_error.extra)
source -> username
  extra fields not permitted (type=value_error.extra)

Screenshot 2023-05-30 at 6.05.15 PM.png

could anyone help me with this above issue https://datahubspace.slack.com/archives/CV2UVAPPG/p1685450099916369