handsome-football-66174
11/22/2021, 10:34 PMagreeable-hamburger-38305
11/23/2021, 1:27 AM[2021-11-23 00:56:07,285] INFO {datahub.cli.ingest_cli:57} - Starting metadata ingestion
/usr/local/lib/python3.8/site-packages/google/cloud/bigquery/client.py:513: UserWarning: Cannot create BigQuery Storage client, the dependency google-cloud-bigquery-storage is not installed.
warnings.warn(
[2021-11-23 01:00:09,074] INFO {datahub.cli.ingest_cli:59} - Finished metadata ingestion
Source (bigquery) report:
{'failures': {}, 'filtered': [], 'tables_scanned': 0, 'views_scanned': 0, 'warnings': {}, 'workunit_ids': [], 'workunits_produced': 0}
Sink (datahub-rest) report:
{'downstream_end_time': None,
'downstream_start_time': None,
'downstream_total_latency_in_seconds': None,
'failures': [],
'records_written': 0,
'warnings': []}
Pipeline finished successfully
millions-notebook-72121
11/23/2021, 3:27 PMbroad-sandwich-74544
11/24/2021, 11:25 AMplain-farmer-27314
11/24/2021, 2:30 PMhelm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "datahub" chart repository
Update Complete. ⎈Happy Helming!⎈
helm upgrade datahub datahub/datahub --namespace=datahub --values values.yaml
W1124 09:17:37.881889 30156 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
W1124 09:17:38.022348 30156 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
W1124 09:17:38.200681 30156 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
W1124 09:17:38.365292 30156 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
W1124 09:17:38.505592 30156 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
W1124 09:17:38.675228 30156 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
Release "datahub" has been upgraded. Happy Helming!
NAME: datahub
LAST DEPLOYED: Wed Nov 24 09:15:46 2021
NAMESPACE: datahub
STATUS: deployed
REVISION: 4
Are there any further steps that need to be taken? Thanks!nice-planet-17111
11/25/2021, 12:58 AMaloof-forest-55926
11/25/2021, 6:40 AMsome-cricket-23089
11/25/2021, 8:40 AMquaint-lighter-4354
11/30/2021, 12:36 PMcurved-jordan-15657
12/02/2021, 11:30 AMcontainer "datahub-gms" in pod "datahub-dev-datahub-gms-57594565c8-887jj" is waiting to start: trying and failing to pull image
And also not only in gms image, kafka and mysql pods also have some errors like:
Failed to pull image "acryldata/datahub-mysql-setup:v0.8.17.0": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: <https://www.docker.com/increase-rate-limit>
I don’t really know how to resolve the issue.early-lawyer-39982
12/03/2021, 6:22 AMaloof-forest-55926
12/06/2021, 10:04 AMbillions-receptionist-60247
12/07/2021, 8:39 PMkubectl logs -f pods/datahub-elasticsearch-setup-job-x4j48
2021/12/07 20:07:48 Waiting for: <http://elasticsearch-master:9200>
2021/12/07 20:07:48 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp 10.97.31.240:9200: connect: connection refused. Sleeping 1s
2021/12/07 20:07:49 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp 10.97.31.240:9200: connect: connection refused. Sleeping 1s
2021/12/07 20:07:50 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp 10.97.31.240:9200: connect: connection refused. Sleeping 1s
2021/12/07 20:07:51 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp 10.97.31.240:9200: connect: connection refused. Sleeping 1s
2021/12/07 20:07:52 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp 10.97.31.240:9200: connect: connection refused. Sleeping 1s
2021/12/07 20:07:53 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp 10.97.31.240:9200: connect: connection refused. Sleeping 1s
ambitious-cartoon-15344
12/09/2021, 2:18 AMcalm-airplane-47634
12/09/2021, 6:05 AMnice-country-99675
12/12/2021, 11:37 PMorange-flag-48535
12/13/2021, 6:30 AMaloof-airline-3441
12/13/2021, 4:42 PMbest-planet-6756
12/13/2021, 9:12 PMevents {}
http {
server {
listen 443 ssl;
ssl_certificate /etc/nginx/certs/certificate.crt;
ssl_certificate_key /etc/nginx/certs/server.key;
location / {
proxy_pass <http://datahub-frontend-react:9002>;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Forwarded-Proto https;
}
}
}
nginx:
image: nginx:latest
container_name: nginx
volumes:
- ../nginx.conf:/etc/nginx/nginx.conf
- ../certs:/etc/nginx/certs
ports:
- 443:443
depends_on:
- datahub-frontend-react
Im getting this error when I go to the url, which shows what I believe is a datahub error. Has anyone faced this when using nginx to serve the page on port 443?billions-receptionist-60247
12/14/2021, 7:20 AMacceptable-architect-70237
12/14/2021, 5:31 PMprefix
for ES index names? We have the cases that the company requires a certain prefix
for ES index name. It seems I can change the code to do that, but ask if it's a configuration option.calm-airplane-47634
12/16/2021, 4:42 PMnutritious-bird-77396
12/16/2021, 6:34 PM# KAFKA_PROPERTIES_SECURITY_PROTOCOL=SSL
# KAFKA_PROPERTIES_SSL_KEYSTORE_LOCATION=
# KAFKA_PROPERTIES_SSL_KEYSTORE_PASSWORD=
# KAFKA_PROPERTIES_SSL_KEY_PASSWORD=
# KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION=
# KAFKA_PROPERTIES_SSL_TRUSTSTORE_PASSWORD=
# KAFKA_PROPERTIES_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=
Are there any options for these to be set thru MSK IAM jar?
I have seen quite a few ppl have made MSK work..trying to understand this a little better....plain-farmer-27314
12/16/2021, 7:46 PMAUTH_OIDC_ENABLED=true
AUTH_OIDC_CLIENT_ID=your-client-id
AUTH_OIDC_CLIENT_SECRET=your-client-secret
AUTH_OIDC_DISCOVERY_URI=<https://your-okta-domain.com/.well-known/openid-configuration>
AUTH_OIDC_BASE_URL=your-datahub-url
AUTH_OIDC_SCOPE="openid profile email groups"
plain-farmer-27314
12/16/2021, 9:31 PMbillions-receptionist-60247
12/17/2021, 6:16 AMTimeout after 2m0s waiting on dependencies to become available: [<http://elasticsearch-master:9200>]
i'm able to curl elastic search on port forward
->curl localhost:9200
{
"name" : "elasticsearch-master-0",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "s-2yYjHTTgy4c9p2I6w0Ug",
"version" : {
"number" : "7.9.3",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "c4138e51121ef06a6404866cddc601906fe5c868",
"build_date" : "2020-10-16T13:34:25.304557Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
bland-orange-13353
12/23/2021, 6:39 AMbillions-receptionist-60247
12/27/2021, 5:15 AM2021-12-27 04:59:13.599:INFO::main: Logging initialized @4991ms to org.eclipse.jetty.util.log.StdErrLog
WARNING: jetty-runner is deprecated.
See Jetty Documentation for startup options
<https://www.eclipse.org/jetty/documentation/>
2021-12-27 04:59:14.192:INFO:oejr.Runner:main: Runner
2021-12-27 04:59:16.986:INFO:oejs.Server:main: jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_302-b08
2021-12-27 05:00:03.685:INFO:oeju.TypeUtil:main: JVM Runtime does not support Modules
2021-12-27 05:00:13.284:WARN:oeja.AnnotationParser:main: Unrecognized ASM version, assuming ASM7
2021/12/27 05:01:14 Command exited with error: signal: killed
breezy-camera-11182
01/04/2022, 5:37 AMadamant-sugar-28445
01/04/2022, 2:05 PM