adamant-pharmacist-61996
08/18/2021, 8:32 AMMetadataChangeProposal_v1
don’t seem to be respected in the gms process. Is this a known problem? I’ve tried poking through the code but can’t immediately see the sourcecurved-jordan-15657
08/19/2021, 1:47 PMProgrammingError: (psycopg2.errors.InsufficientPrivilege) permission denied for relation campaign_retention_order_segment
[SQL: CREATE TEMPORARY TABLE "ge_tmp_61606ece" AS SELECT *
FROM dev.campaign_retention_order_segment
LIMIT 10]
blue-megabyte-68048
08/20/2021, 1:58 PMELASTICSEARCH_USE_SSL=true
as well as all the various ELASTICSEARCH_SSL_*
env vars specified by the docker env file. Any suggestions or troubleshooting I can do? I have confirmed that the certs work to connect to ES.handsome-football-66174
08/20/2021, 5:38 PMrhythmic-london-44496
08/23/2021, 10:59 AMv0.8.10
) on Kubernetes using https://github.com/acryldata/datahub-helm/tree/master/charts/datahub
But both GMS
and upgrade-job
pods have troubles running -> they both log errors showing that the main process cannot find entity-registry.yml
file, e.g.:
10:46:49.006 [main] ERROR o.s.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'metadataAuditEventsProcessor' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8080-war_war-_-any-1391980781119123614.dir/webapp/WEB-INF/lib/mae-consumer.jar!/com/linkedin/metadata/kafka/MetadataAuditEventsProcessor.class]: Uns
atisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'searchServiceFactory': Unsatisfied dependency expressed through field 'elasticSearchService'; nested exception is org.springframework.beans
.factory.UnsatisfiedDependencyException: Error creating bean with name 'elasticSearchServiceFactory': Unsatisfied dependency expressed through field 'entityRegistry'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'entityRegistryFactory': Unsatisfied
dependency expressed through field 'configEntityRegistry'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configEntityRegistry' defined in com.linkedin.gms.factory.entityregistry.ConfigEntityRegistryFactory: Bean instantiation via factory method failed; nes
ted exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.models.registry.ConfigEntityRegistry]: Factory method 'getInstance' threw exception; nested exception is java.io.FileNotFoundException: ../../metadata-models/src/main/resources/entity-registry.yml (No su
ch file or directory)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:798)
(...)
Caused by:
org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.models.registry.ConfigEntityRegistry]: Factory method 'getInstance' threw exception; nested exception is java.io.FileNotFoundException: ../../metadata-models/src/main/resources/entity-registry.yml (No such file or direct
ory)
Are published images broken and we should maintain our own builds?handsome-belgium-11927
08/24/2021, 11:16 AMpython -m datahub docker quickstart
is no longer working, help says that there is no "docker" command anymore. Though running docker/quickstart.sh works well.
Is it a bug? Docs at https://datahubproject.io/docs/quickstart have not been modified yet.adamant-pharmacist-61996
08/24/2021, 11:10 PMbland-salesmen-77140
08/25/2021, 8:25 AMsquare-activity-64562
08/25/2021, 11:54 AMSELECT * from metadata_aspect_v2
where aspect = 'corpGroupInfo'
members are present for the groups. But they are not shown on the groups pagesquare-activity-64562
08/25/2021, 5:18 PMSELECT urn AS urn,
count(*) AS count
FROM
(SELECT urn
from metadata_aspect_v2
where aspect = 'tagKey') AS expr_qry
GROUP BY urn
ORDER BY count DESC
But it is missing some tags which I know are applied on datasets. What am I doing wrong?witty-actor-87329
08/25/2021, 8:21 PMRemoving network datahub_network
ERROR: error while removing network: network datahub_network id c2c6739a3ce536f9d1b091c0ba24df7d1584dbeb20dc1cd0b913d3 has active endpoints
should anything be done before running the command. Doing this EC2. Thanksclever-river-85776
08/26/2021, 8:19 AM:9002/api/v2/graphql
. I tried :9002/api/v2/graphql/schema
, (just guessing), but that didn't work.square-activity-64562
08/26/2021, 8:23 AMupstreamLineage
aspects in which it is present in the database.square-activity-64562
08/26/2021, 9:45 AMwonderful-quill-11255
08/26/2021, 2:15 PMdocker/elasticsearch-setup/create-indices.sh
script to update indices but after 2840 that script only creates the usage analytics indices. I'm wondering, how are the indices created these days? @early-lamp-41924 Perhaps you know?fresh-carpet-31048
08/26/2021, 2:18 PMhandsome-belgium-11927
08/27/2021, 2:26 PMgentle-father-80172
08/27/2021, 7:56 PMIdentity Provider: Unknown
Error Code: invalid_request
Description: The 'redirect_uri' parameter must be a Login redirect URI in the client app settings: <https://admin.settings/example>
Not sure how to troubleshoot this... My IT department says they don't see any Okta requests from Datahub. No logs of my request are appearing in docker logs -f datahub-frontend-react
either... Config in thread below:some-microphone-33485
08/30/2021, 3:20 AMsome-microphone-33485
08/30/2021, 6:51 PMfresh-carpet-31048
08/30/2021, 8:27 PMadamant-pharmacist-61996
08/30/2021, 11:30 PMhandsome-football-66174
08/31/2021, 3:20 PMcurved-jordan-15657
08/31/2021, 9:02 PMKafkaException: KafkaError{code=MSG_SIZE_TOO_LARGE,val=10,str="Unable to produce message: Broker: Message size too large"}
I did some research about it and people say that i need to increase some of the properties like message.max.byte, max.request.byte etc. from broker,producer and consumer sides. I updated the server.properties,consumer.properties and producer.properties files inside the k8s kafka pod, but i couldn’t solve the issue. Can anybody help me about kafka and k8s?
Note: I think i need to restart the kafka broker to apply server.properties changes somehow, but i don’t know how.high-hospital-85984
09/03/2021, 6:38 AMsquare-activity-64562
09/07/2021, 7:02 AMsquare-activity-64562
09/07/2021, 7:06 AMcalm-sunset-28996
09/07/2021, 8:54 AMexception: java.util.concurrent.CompletionException: java.lang.RuntimeException: Failed to retrieve entities of type Dataset
Caused by: java.lang.RuntimeException: Failed to batch load Datasets
Caused by: com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <https://datahub-gms.net:443/entities>
at com.linkedin.restli.internal.client.ExceptionUtil.wrapThrowable(ExceptionUtil.java:135)
Caused by: io.netty.handler.codec.TooLongFrameException: Response entity too large: HttpObjectAggregator$AggregatedFullHttpResponse(decodeResult: success, version: HTTP/1.1, content: CompositeByteBuf(ridx: 0, widx: 2096929, cap: 2096929, components=335))
So the entities are too large, causing. the lookup to fail. To give a bit of context: this is only happening with really specific searches, where it has to retrieve multiple datasets which have a huge amount of columns (1000k+). Then it times out. If I search for these individual entities it's fine, the same when I go to their respective pages. Any idea on how to fix this? I'm looking for some netty settings atm like maxResponseKB
which I could potentially set.
It's probably the same error as https://github.com/linkedin/datahub/issues/3106square-activity-64562
09/07/2021, 9:52 AM$ datahub init
Configure which datahub instance to connect to
Enter your DataHub host [<http://localhost:8080>]: <http://datahub-datahub-gms.apps.svc.cluster.local:8080>
Enter your DataHub access token (Supports env vars via `{VAR_NAME}` syntax) []:
Written to /home/datahub/.datahubenv
$ datahub delete --urn "urn:li:corpuser:aseem.bansal"
This will permanently delete data from DataHub. Do you want to continue? [y/N]: y
Successfully deleted urn:li:corpuser:aseem.bansal. 0 rows deleted
It says
0 rows deleted
I have noticed that delete API works for anything that was created after the delete API was released but does not seem to work for things created before that versionhigh-hospital-85984
09/07/2021, 10:10 AMerror in Flask-OpenID setup command: use_2to3 is invalid.
when installing dev dependencies for metadata-ingestion. Most likely cause is this: https://setuptools.readthedocs.io/en/stable/history.html#breaking-changes