Source (kafka) report: {'failures': {}, 'filtered': ['__transaction_state', '_confluent-telemetry-metrics', '_confluent_balancer_partition_samples', '_confluent-metrics', '_confluent_balancer_api_state', '__consumer_offsets', '_confluent_balancer_broker_samples', '_confluent-license'], 'topics_scanned': 21, 'warnings': {'topic1': ['failed to get value schema: Subject not found. (HTTP status code 404, SR code 40401)'], 'topic2': ['failed to get value schema: Subject not found. (HTTP status code 404, SR code 40401)'], 'topic3': ['failed to get value schema: Subject not found. (HTTP status code 404, SR code 40401)']}, 'workunit_ids': ['kafka-topic1', 'kafka-topic2', 'kafka-topic3', ], 'workunits_produced': 13} Sink (datahub-kafka) report: {'downstream_end_time': None, 'downstream_start_time': None, 'downstream_total_latency_in_seconds': None, 'failures': [], 'records_written': 13, 'warnings': []}

I run 'datahub ingest -c kafka_to_datahub_ex.yml' and fail

anyone know this error, please?

Moved to #troubleshoot

Hi guys, I'm doing DataHub quickstart to ingest some data from AWS Athena and test functionalities, but after trying to ingest data I have this error multiple times over:

[2021-11-29 13:26:43,382] INFO     {datahub.cli.ingest_cli:73} - Starting metadata ingestion
[2021-11-29 13:26:51,316] ERROR    {datahub.ingestion.run.pipeline:85} - failed to write record with workunit curated.part_00002_4c6e811b_04cc_4189_80d7_ca7c0c18b959_c000_snappy_parquet with ('Unable to emit metadata to DataHub GMS', {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'stackTrace': "com.linkedin.restli.server.RestLiServiceException [HTTP Status:500]: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException\n\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:42)\n\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)\n\tat com.linkedin.metadata.resources.entity.EntityResource.ingest(EntityResource.java:178)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.linkedin.restli.internal.server.RestLiMethodInvoker.doInvoke(RestLiMethodInvoker.java:172)\n\tat 
...

I used this command to start ingesting:

datahub ingest -c ./awsathena_to_datahub.yml

Can anyone help? The UI on localhost works properly though

I Tried removing the

env

field from config and now it works:

source:
  type: athena
  config:
    aws_region: "us-west-2"
    env: "staging"

and this is the deployment .

Hi

Tengo un error, cuando ejecuto "docker quickstart" va todo bien pero, cuando quiero ver en el navegador la pagina esta off

image.png

image.png

I wonder if datahub supports AWS EMR spark3.0

loving the platform btw

sometimes i see demo.datahubproject.io having "waiting for socket" messages while waiting to load a page. just wondering if that is a sign that GMS is busy?

Hi guys , i try to resintall datahub using helm

ehlm install datahub  datahub/datahub  -f helm_custom_settings_custom_helm.yaml

but i have this error

Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition

Thread in Slack Conversation

hello, deploy datahub by helm charts

This content can't be displayed.

<!here> Hi folks! We’re eager to get feedback about how much time/effort is required to run DataHub in Prod — please take a few minutes to fill out the Polly, above! teamwork BONUS! We’ll choose 3 random respondents to receive a DataHub Swag box whoop datahubbbb We’ll be accepting responses until March 17th; feel free to DM me with any questions!

FYI, I use mysql cli tool to access my MySQL instance, it works. One week ago I switch my mysql instance to quickstart and it worked. Today it does not work for some reasons.

can anyone help in this

Hi team, I have enabled the data hub ingestion cron job still it is not coming in cron jobs

I have a fork of the Datahub project, and it always seem to be quite a pain to get through the build-and-test workflow when we commit our own changes. Is there anything we need to change (or set up our own runner?)

my EKS consists of 2 t3.meditum.

Log is like this.

hi, is there any way to convert helm charts to kustomize? I tried to use helm convert but it seems stable/datahub has been deprecated?

I'd like to upgrade this to a newer version without impacting the data present ; any pointers / takeaways please?

I think you can modify the docker image. Try take a look at the helm chart

Just curious, for teams with very granular datahub rights, how do you guys keep track of various policies? Policy implemented on a per person basis?

Hi team, I am doing custom source ingestion so I have created a custom source package and install that over my image Using the same example https://github.com/acryldata/meta-world/tree/master/custom_sources/ And my py file in created in Home/datahub/.local/bin/custom ingestion source.py But when I am trying execute my recipe file by giving the absolute path it is not running

Or even I gave the full path of python file in source It's not working