wooden-arm-26381
08/05/2022, 7:56 AMUnboundLocalError: local variable 'datahub_corp_group_snapshot' referenced before assignment
The variable is accessed outside this for loop: https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/identity/azure_ad.py#L280
This happens on line #297 and #305jolly-traffic-67085
08/05/2022, 7:58 AMhallowed-lawyer-5424
08/05/2022, 10:01 AM<http://localhost:8080/entities?action=search>
with this body as payload
{
"input": "",
"entity": "dataset",
"start": 9999,
"count": 10000
}
I could able to fetch only upto 10,000 records. Is there any way to get entire records based on entity type and environment. If I give start as '0' and end as '10000', getting the response. Not getting the remaining records.brave-tomato-16287
08/05/2022, 10:04 AMdatahub_actions.pipeline.pipeline.PipelineException: Failed to log failed event to file! EventEnvelope(event_type='MetadataChangeLogEvent_v1', event=MetadataChangeLogEvent({'auditHeader': None, 'entityType': 'dataset', 'entityUrn': 'urn:li:dataset:(urn:li:dataPlatform:dbt,dev.analytics_dbt_test__audit.not_null_dim_customers_customer_id,PROD)', 'entityKeyAspect': None, 'changeType': 'RESTATE', 'aspectName': 'upstreamLineage', 'aspect': GenericAspectClass({'value': b'{"upstreams":[{"auditStamp":{"actor":"urn:li:corpuser:unknown","time":0},"type":"TRANSFORMED","dataset":"urn:li:dataset:(urn:li:dataPlatform:redshift,dev.analytics.dim_customers,PROD)"}]}', 'contentType': 'application/json'}), 'systemMetadata': None, 'previousAspectValue': None, 'previousSystemMetadata': None, 'created': AuditStampClass({'time': 1659685194401, 'actor': 'urn:li:corpuser:__datahub_system', 'impersonator': None})}), meta={'kafka': {'topic': 'MetadataChangeLog_Versioned_v1', 'offset': 162992, 'partition': 0}})
little-army-38555
08/05/2022, 10:48 AMmssql
plugin installation in Kubernetes with Helm Chart? Are there any ways to do this without pip install
? Or should I build a docker container with it?square-solstice-69079
08/05/2022, 1:52 PMgray-nest-42961
08/05/2022, 6:08 PMFailed to instantiate [com.linkedin.metadata.kafka.hook.UpdateIndicesHook]: Constructor threw exception; nested exception is java.lang.RuntimeException: Could not configure system metadata index
22:15:58 [main] INFO c.l.r.t.h.c.c.AbstractNettyClient - Shutdown requested
22:15:58 [main] INFO c.l.r.t.h.c.c.AbstractNettyClient - Shutting down
...
Caused by: java.net.ConnectException: Connection refused
any idea on what might cause it and how to fix it👀? thanks! cc @bitter-lizard-32293 @numerous-byte-87938numerous-account-62719
08/08/2022, 4:56 AMechoing-farmer-38304
08/08/2022, 8:01 AM\datahub\ingestion\source\powerbi.py", line 588, in get_data_source
id=datasource_dict["datasourceId"],
KeyError: 'datasourceId'
Dictionary with data (datasource_dict)
{'datasourceType': 'Sql', 'connectionDetails': {'server': 'server-name', 'database': 'db-name'}}
dataset_type_mapping
dataset_type_mapping:
PostgreSql: postgres
Oracle: oracle
Is there problem in my config or something goes wrong in module, any ideas?salmon-rose-54694
08/08/2022, 10:10 AMorg.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition MetadataChangeLog_Versioned_v1-0
Or
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema not found; error code: 40403
at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)
at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:351)
at io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:659)
at io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:641)
Just want to know anything on the new aspect maybe miss regarding MAE and MCL consumer? Thank you.numerous-account-62719
08/08/2022, 11:29 AMRequirement already satisfied: pip in /opt/conda/lib/python3.9/site-packages (22.0.4)
Collecting install
Using cached install-1.3.5-py3-none-any.whl (3.2 kB)
Collecting datahub
Using cached datahub-0.8.90dev.tar.gz (11 kB)
Preparing metadata (setup.py) ... done
Collecting pastescript>=1.0
Using cached PasteScript-3.2.1-py2.py3-none-any.whl (73 kB)
Collecting cheetah>=2.0
Using cached Cheetah-2.4.4.tar.gz (190 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [9 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-install-qqhf77ju/cheetah_8ebe7010343c4b18a855f6ae5ba8cd7b/setup.py", line 10, in <module>
import SetupTools
File "/tmp/pip-install-qqhf77ju/cheetah_8ebe7010343c4b18a855f6ae5ba8cd7b/SetupTools.py", line 50
except DistutilsPlatformError, x:
^
SyntaxError: invalid syntax
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
gentle-camera-33498
08/08/2022, 3:13 PMfaint-translator-23365
08/08/2022, 3:13 PMshy-parrot-64120
08/08/2022, 7:13 PMdatahub-gms
failed to start:
ERROR: No such classes directory file:///etc/datahub/plugins/auth/resources
bright-diamond-60933
08/08/2022, 9:05 PMrapid-house-76230
08/08/2022, 9:10 PMbright-diamond-60933
08/08/2022, 8:57 PMpurple-analyst-83660
08/09/2022, 4:13 AMfailed to write record with workunit urn:li:dashboard:(tableau,10625179-12ce-63aa-edce-798f6a70d9f6) with ('Unable to emit metadata to DataHub GMS', {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'stackTrace': 'com.linkedin.restli.server.RestLiServiceException [HTTP Status:422]: com.linkedin.metadata.entity.ValidationException: Failed to validate record with class com.linkedin.entity.Entity: ERROR :: /value/com.linkedin.metadata.snapshot.DashboardSnapshot/aspects/0/com.linkedin.dashboard.DashboardInfo/datasets :: unrecognized field found but not allowed
My datahub GMS version is 0.8.40 and CLI is 0.8.42.faint-translator-23365
08/08/2022, 2:02 PMechoing-farmer-38304
08/09/2022, 7:17 AMfrom datahub.metadata.schema_classes import (
E ModuleNotFoundError: No module named 'datahub.metadata'
Also, I tried to build a whl package and run, but getting the same error.
If I install with ( pip install acryl-datahub==0.8.41 ), tests run without that error, but in that case it wouldn't see my local changes. Is there any solutions for that?agreeable-belgium-70840
08/09/2022, 9:50 AMstraight-agent-79732
08/09/2022, 11:39 AMdatahub docker quickstart
I got port conflicts with elastic search, schema registry and datahub-gms. I see we can pass different ports for elastic search and schema registry but no documentation available for datahub-gms. leaving datahub-gms aside, I tried passing different ports for elastic search and schema registry like this, datahub docker quickstart --elastic-port 7310 --schema-registry-port 7311 But, no luck datahub is using same old ports. Can anyone help me out herewooden-pencil-40912
08/09/2022, 12:39 PMv0.8.42
? Any estimate would be really helpful. 🙂stale-printer-44316
08/09/2022, 3:32 PMmost-nightfall-36645
08/09/2022, 3:42 PMlog_level
option we can use to reduce log verbosity.
I tried searching around documentation but I cant seem to find anything.kind-whale-32412
08/09/2022, 4:59 PMconfluent-kafka
in acryl-datahub-actions pip package? Currently it's not compatible with Macbook M1bland-stone-30401
08/09/2022, 11:49 AMOops, an error occurred
This exception has been logged with id *7ohk695mj*.
PS: We have also disabled the OIDC authentication.steep-finland-24780
08/10/2022, 12:17 AMCaused by: com.nimbusds.oauth2.sdk.ParseException: The scope must include an "openid" value
at com.nimbusds.openid.connect.sdk.AuthenticationRequest.parse(AuthenticationRequest.java:1378)
at com.nimbusds.openid.connect.sdk.AuthenticationRequest.parse(AuthenticationRequest.java:1312)
at org.pac4j.oidc.redirect.OidcRedirectActionBuilder.buildAuthenticationRequestUrl(OidcRedirectActionBuilder.java:110)
It seems it's not parsing properly the AUTH_OIDC_SCOPE
env var.
I opened an interactive shell inside the container and it seems the variables are being set accordingly.
Here's the output from the front-end container:
ubuntu@host-name:~$ docker exec -it <CONTAINER_ID_FRONT-END> /bin/sh
/ $ env
ELASTIC_CLIENT_HOST=elasticsearch
HOSTNAME=datahub-frontend-react
SHLVL=1
HOME=/home/datahub
AUTH_OIDC_DISCOVERY_URI=<https://accounts.google.com/.well-known/openid-configuration>
ELASTIC_CLIENT_PORT=9200
AUTH_OIDC_CLIENT_ID=<correct_OIDC_CLIENT>
AUTH_OIDC_CLIENT_SECRET=<correct_OIDC_SECRET>
AUTH_OIDC_ENABLED=true
AUTH_OIDC_USER_NAME_CLAIM=email
AUTH_OIDC_SCOPE="openid profile email"
TERM=xterm
Does anyone had a similar problem?
How are you guys setting those variables?numerous-account-62719
08/10/2022, 5:46 AMhelm dependency build
failed exit status 1: Error: can't get a valid version for repositories datahub-gms, datahub-mae-consumer, datahub-mce-consumer. Try changing the version constraint in Chart.yamlbulky-jordan-44775
08/10/2022, 6:28 AM[Thread-481] WARN n.g.e.SimpleDataFetcherExceptionHandler - Exception while fetching data (/corpUser) : java.lang.RuntimeException: Failed to retrieve entities of type CorpUser
Can anybody help me?