rapid-house-76230
08/02/2022, 1:38 AM2022/08/02 01:34:44 Received 400 from <http://search-datahub-elasticsearch-foobar.us-west-2.es.amazonaws.com:443>. Sleeping 1s
I used ELASTICSEARCH_HOST
and ELASTICSEARCH_PORT
env variables in the GMS deployment (I used Kustomize). Any idea?steep-winter-48926
07/22/2022, 5:43 AMbillions-twilight-48559
12/23/2021, 7:40 AMancient-apartment-23316
07/29/2022, 5:15 PMaws-load-balancer-internal: "true"
here https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/datahub-frontend/values.yaml#L48 and tried to install from local folder helm install datahub .
But I am getting an error message Error: INSTALLATION FAILED: Chart.yaml file is missing
Chart.yaml file is in place, I downloaded the full repository acryldata/datahub-helm locally
Please advise how best to install datahub-frontend k8s service with enabled annotation <http://service.beta.kubernetes.io/aws-load-balancer-internal|service.beta.kubernetes.io/aws-load-balancer-internal>: "true"
lemon-terabyte-66903
07/19/2022, 2:57 AMacryl-datahub[s3]
on a databricks cluster and hence got timed out.
Can the devs make a fix to this?average-vr-23088
08/02/2022, 4:48 PMfaint-translator-23365
08/02/2022, 7:10 PMbland-orange-13353
08/02/2022, 10:27 PMwonderful-author-3020
08/03/2022, 12:00 PMdatahub-upgrade
container to upgrade to the newest version?quiet-wolf-56299
08/03/2022, 5:41 PMrapid-house-76230
08/03/2022, 6:02 PM2022/08/03 21:35:16 Connected to <tcp://boot-foobar.c1.kafka-serverless.us-west-2.amazonaws.com:9098>
2022/08/03 21:35:16 Connected to <tcp://datahub.cluster-foobar.us-west-2.rds.amazonaws.com:5432>
2022/08/03 21:35:16 Received 200 from <https://datahub:Datahub%21123@search-datahub-elasticsearch-foobar.us-west-2.es.amazonaws.com:443>
2022/08/03 21:35:17 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:18 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:19 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:20 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:21 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:22 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:23 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:24 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:25 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
I have the envs variables on my GMS deployment as follows
- KAFKA_SCHEMAREGISTRY_URL=<http://schema-registry:8081>
- KAFKA_BOOTSTRAP_SERVER=<http://boot-foobar.kafka-serverless.us-west-2.amazonaws.com:9098|boot-foobar.kafka-serverless.us-west-2.amazonaws.com:9098>
- ALLOW_PLAINTEXT_LISTENER=yes
- ZOOKEEPER_CLIENT_PORT=2181
- ZOOKEEPER_TICK_TIME=2000
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=<PLAINTEXT://broker:29092>,PLAINTEXT_<HOST://localhost:9092>
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
- KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0
- KAFKA_HEAP_OPTS=-Xms256m -Xmx256m
Is there any config error or anything that you could think of on my end?average-vr-23088
08/03/2022, 7:13 PMdatahub.metadata_aspect_v2
• restore tables to new instance
• run restore indices on new instance
Would be nice if there was a backup / restore high-level command.high-summer-78960
08/03/2022, 8:56 PMmicroscopic-mechanic-13766
08/04/2022, 8:09 AMjolly-yacht-10587
08/05/2022, 3:39 AMaverage-vr-23088
08/05/2022, 5:03 PMUnable to emit metadata to DataHub GMS
401 Client Error: Unauthorized for url: <http://datahub-gms.datahub.local:8080/aspects?action=ingestProposal>
I’m running DataHub v0.8.40 and DataHub Actions v0.0.4. The above log is showing up in the DataHub actions container.
I have configured the DataHub Actions container with the client id and secret for GMS as well.
It is worth noting that this setup was working previously but I migrated data from one DataHub deployment to another deployment. They both had the same version of containers but the client id / secrets and token generation secrets differed between the two. After the migration, which involved copying the metadata v2 table and ReIndexing, i’m getting ingestion failures.
I’ve also tried deleting the ingestion as well as the secrets it references and recreating them with no success.
I noticed another warning in the logs:
❗Client-Server Incompatible❗ Your client version 0.8.38.2 is older than your server version 0.8.40. Upgrading the cli to 0.8.40 is recommended
I’m guessing the latest release of the actions container doesn’t use the up to date CLI version? Would i need to set the UI_INGESTION_DEFAULT_CLI_VERSION=0.8.38.2
in GMS?microscopic-mechanic-13766
08/08/2022, 9:50 AMDATAHUB_APP_VERSION
, DATAHUB_SECRET
, DATAHUB_SYSTEM_CLIENT_SECRET
, ....
I have been trying to look in the site of the project, but haven't been lucky.
Any help would be apreciated!acceptable-baker-8114
08/08/2022, 11:11 AMdatahub:
metadata_service_authentication:
enabled: true
I can see this when I run helm get values datahub but it still comes up with the error below when I try and generate a token, is there something else I need to do?
Token based authentication is currently disabled. Contact your DataHub administrator to enable this feature.
microscopic-mechanic-13766
08/09/2022, 12:22 PMacryldata/datahub-actions:v0.0.4
. I haven't specify any type of limit either to the CPU they can use or to the memory they can use.bland-orange-13353
08/10/2022, 9:04 AMdamp-greece-27806
08/10/2022, 6:34 PMastonishing-lizard-90580
08/10/2022, 8:20 PMancient-apartment-23316
08/11/2022, 12:52 PMTo do so, you must update the datahub-frontend docker.env file with the values received from your identity provider:the thing is I have installed the datahub on kubernetes (EKS) using helm and AWS managed services I can’t edit the docker.env file Instead, I manually edit k8s deployment.apps/datahub-datahub-frontend and put the envs (AUTH_OIDC_ENABLED, AUTH_OIDC_CLIENT_ID, etc.) there. The new frontend pod is ready and contains all the envs. But redirection to okta does not work, I still have access to the datahab, w/o okta
fancy-thailand-73281
08/11/2022, 8:20 PMv0.8.36
in AWS EKS cluster with helm charts, we are using AWS MSK (KAFKA), with as a Bootstrap servers :*SASL/SCRAM* and ZooKeeper TLS. Everything works fine till here.
But we are not able to Ingest data(SNOWFLAKE) from UI , I see 'N/A' when I try to run ingestion.
We found in Datahub docs(https://datahubproject.io/docs/ui-ingestion/) saying that we need to enable datahub-actions
, then we deployed the public.ecr.aws/datahub/acryl-datahub-actions pods.
The pod is not running(CrashLoopBackOff) and we see error logs shows as below
[2022-08-11 19:56:59,004] ERROR {datahub.entrypoints:138} - File "/usr/local/lib/python3.9/site-packages/datahub/cli/ingest_cli.py", line 77, in run │
│ 67 def run(config: str, dry_run: bool, preview: bool, strict_warnings: bool) -> None:
│
│ KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create consumer: No provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN SASL_SCRAM OAUTHBEARER"} │
│ 2022/08/11 19:56:59 Command exited with error: exit status 1
could someone please help us and thanks in Advanceelegant-article-21703
08/12/2022, 10:04 AM--debug
message extract the following:
wait.go:225: [debug] Deployment is not ready: datahub/datahub-datahub-frontend. 0 out of 1 expected pods are ready
upgrade.go:360: [debug] warning: Upgrade "datahub" failed: timed out waiting for the condition
upgrade.go:378: [debug] Upgrade failed and atomic is set, rolling back to last successful release
history.go:53: [debug] getting history for release datahub
rollback.go:64: [debug] preparing rollback of datahub
rollback.go:112: [debug] rolling back datahub (current: v14, target: v13)
rollback.go:71: [debug] creating rolled back release for datahub
and the command I'm using is (I'm updating the root password):
helm upgrade -n datahub --atomic --debug datahub ./helm/datahub --values .\helm\datahub\charts\datahub-frontend\values.yaml
Thank you in advance!great-account-95406
08/16/2022, 5:05 AMeager-terabyte-73886
08/16/2022, 9:00 PMsilly-oil-35180
08/17/2022, 2:51 AMthousands-solstice-2498
08/17/2022, 7:22 AMthousands-solstice-2498
08/17/2022, 7:22 AM