• f

    fresh-cricket-75926

    2 days ago
    Hi All , i am trying fetch Metadata from Bigquery and save it to a file . The issue is , i cant read Entities and Table from the source . In shell console , simple the connection get timeout . we are using rancher here to deploy datahub . Any suggestion will be helpful here.
    f
    h
    2 replies
    Copy to Clipboard
  • g

    green-hamburger-3800

    2 days ago
    Hey folks, how are you?! I'm having some issue trying to ingest data from
    trino
    We're using
    Starburst
    with a
    glue catalog
    and when trying to ingest data using the
    Trino Source
    we're encountering the following error:
    'message="Table \'"schema".table$properties\' not found"
    It seems that in fact that doesn't exist and it's not the way to query for properties in this case. Any ideas?
    g
    h
    2 replies
    Copy to Clipboard
  • m

    mammoth-air-95743

    2 days ago
    Hi everyone! I am using ingestion from S3 bucket, and json files within, and in logger of ingestion task I get the message that it’s extracting table schema but there’s nothing actually there, it doesn’t infer the schema. Here’s logger output:
    '[2022-09-20 09:41:44,078] INFO     {datahub.ingestion.source.s3.source:519} - Extracting table schema from file: '
               '<s3://path/to/file.json>\n'
    '[2022-09-20 09:41:44,078] INFO     {datahub.ingestion.source.s3.source:527} - Creating dataset urn with name: '
               'path/to/file.json\n'
    m
    h
    3 replies
    Copy to Clipboard
  • g

    gorgeous-dinner-4055

    3 days ago
    👋🏽 Hi All! I am seeing weird behavior with lineage, and wondering if it's a bug, or something I'm miss-understanding. We have a couple of datasets with multiple versions of Lineage that we have ingested over time. When looking at a urns history, we see that there's 4 versions(details in thread), and the latest version is not == to the largest version number. Is that perhaps a bug with how we're ingesting data?
    g
    7 replies
    Copy to Clipboard
  • c

    clever-garden-23538

    1 week ago
    is it possible to define policies through a configuration file?
    c
    b
    3 replies
    Copy to Clipboard
  • b

    big-carpet-38439

    4 days ago
    It might be that you datahub-frontend container is on a previous version of DataHub
    b
    e
    2 replies
    Copy to Clipboard
  • a

    adamant-rain-51672

    2 days ago
    Is there a guide on datahub version upgrade on EKS/Kubernetes? I upgraded but there seems to be an error with defined ingestions.
    a
    b
    2 replies
    Copy to Clipboard
  • m

    mammoth-bear-12532

    15 hours ago
    Hi @early-oil-62555 would suggest upgrading your datahub version, as this bug has likely been fixed in newer versions.
  • m

    many-keyboard-47985

    4 days ago
    Hi! I am trying to datahub-gms component mysql error log. I’m set up datahub-component and using datahub. Most of the functions are working. However, the error log of datahub-gms is repeatedly output. Can you help me with this error log?
    16:36:52.931 [pool-7-thread-1] ERROR c.d.authorization.DataHubAuthorizer:229 - Failed to retrieve policy urns! Skipping updating policy cache until next refresh. start: 0, count: 30
    javax.persistence.PersistenceException: Query threw SQLException:vtgate: ${my mysql address} : code = Aborted desc = transaction 1663147082760249589: ended at 2022-09-21 16:36:10.929 KST (exceeded timeout: 1m0s) (CallerID: datahub admin db) Bind values:[urn:li:dataHubPolicy:15b29d13-53ad-44d0-a006-a33a9550ee77, dataHubPolicyInfo, 0, urn:li:dataHubPolicy:15b29d13-53ad-44d0-a006-a33a9550ee77, dataHubPolicyKey, 0] Query was:select urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ?
    	at io.ebean.config.dbplatform.SqlCodeTranslator.translate(SqlCodeTranslator.java:52)
    	at io.ebean.config.dbplatform.DatabasePlatform.translate(DatabasePlatform.java:219)
    	at io.ebeaninternal.server.query.CQueryEngine.translate(CQueryEngine.java:149)
    	at io.ebeaninternal.server.query.DefaultOrmQueryEngine.translate(DefaultOrmQueryEngine.java:43)
    	at io.ebeaninternal.server.core.OrmQueryRequest.translate(OrmQueryRequest.java:102)
    	at io.ebeaninternal.server.query.CQuery.createPersistenceException(CQuery.java:702)
    	at io.ebeaninternal.server.query.CQueryEngine.findMany(CQueryEngine.java:411)
    	at io.ebeaninternal.server.query.DefaultOrmQueryEngine.findMany(DefaultOrmQueryEngine.java:133)
    	at io.ebeaninternal.server.core.OrmQueryRequest.findList(OrmQueryRequest.java:459)
    	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1596)
    	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1574)
    	at io.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:1481)
    	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGetUnion(EbeanAspectDao.java:359)
    	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:279)
    	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:260)
    	at com.linkedin.metadata.entity.EntityService.exists(EntityService.java:1309)
    	at com.linkedin.metadata.resources.entity.ResourceUtils.lambda$validateSearchResult$0(ResourceUtils.java:52)
    	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
    	at java.util.Iterator.forEachRemaining(Iterator.java:116)
    	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
    	at com.linkedin.metadata.resources.entity.ResourceUtils.validateSearchResult(ResourceUtils.java:53)
    	at com.linkedin.entity.client.JavaEntityClient.search(JavaEntityClient.java:297)
    	at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:50)
    	at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:42)
    	at com.datahub.authorization.DataHubAuthorizer$PolicyRefreshRunnable.run(DataHubAuthorizer.java:222)
    	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    	at java.lang.Thread.run(Thread.java:748)
    Caused by: com.mysql.cj.jdbc.exceptions.MySQLQueryInterruptedException: ${my mysql address} : rpc error: code = Aborted desc = transaction 1663147082760249589: ended at 2022-09-21 16:36:10.929 KST (exceeded timeout: 1m0s) (CallerID: datahub admin db)
    	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:126)
    	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    	at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
    	at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1003)
    	at io.ebean.datasource.pool.ExtendedPreparedStatement.executeQuery(ExtendedPreparedStatement.java:136)
    	at io.ebeaninternal.server.query.CQuery.prepareResultSet(CQuery.java:376)
    	at io.ebeaninternal.server.query.CQuery.prepareBindExecuteQueryWithOption(CQuery.java:324)
    	at io.ebeaninternal.server.query.CQuery.prepareBindExecuteQuery(CQuery.java:319)
    	at io.ebeaninternal.server.query.CQueryEngine.findMany(CQueryEngine.java:384)
    	... 30 common frames omitted
    m
    m
    5 replies
    Copy to Clipboard
  • s

    silly-oil-35180

    54 minutes ago
    Hello team. I have trouble using GMS api. I read guide about getting aspect from urn(https://github.com/datahub-project/datahub/blob/master/metadata-service/README.md?plain=1#L1372). However, when I sent api request to GMS, I got this error.
    'com.linkedin.restli.server.RestLiServiceException', 'stackTrace': 'com.linkedin.restli.server.RestLiServiceException [HTTP Status:404]\n\tat com.linkedin.metadata.restli.RestliUtil.resourceNotFoundException(RestliUtil.java:79)\n\tat com.linkedin.metadata.restli.RestliUtil.resourceNotFoundException(RestliUtil.java:74)\n\tat com.linkedin.metadata.resources.entity.AspectResource.lambda$get$0(AspectResource.java:81)\n\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:30)\n\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)\n\tat com.linkedin.metadata.resources.entity.AspectResource.get(AspectResource.java:78)\n\tat sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.linkedin.restli.internal.server.RestLiMethodInvoker.doInvoke(RestLiMethodInvoker.java:177)\n\tat com.linkedin.restli.internal.server.RestLiMethodInvoker.invoke(RestLiMethodInvoker.java:333)\n\tat com.linkedin.restli.internal.server.filter.FilterChainDispatcherImpl.onRequestSuccess(FilterChainDispatcherImpl.java:47)\n\tat com.linkedin.restli.internal.server.filter.RestLiFilterChainIterator.onRequest(RestLiFilterChainIterator.java:86)\n\tat com.linkedin.restli.internal.server.filter.RestLiFilterChainIterator.lambda$onRequest$0(RestLiFilterChainIterator.java:73)\n\tat java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)\n\tat java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683)\n\tat java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010)\n\tat com.linkedin.restli.internal.server.filter.RestLiFilterChainIterator.onRequest(RestLiFilterChainIterator.java:72)\n\tat com.linkedin.restli.internal.server.filter.RestLiFilterChain.onRequest(RestLiFilterChain.java:55)\n\tat com.linkedin.restli.server.BaseRestLiServer.handleResourceRequest(BaseRestLiServer.java:262)\n\tat com.linkedin.restli.server.RestRestLiServer.handleResourceRequestWithRestLiResponse(RestRestLiServer.java:294)\n\tat com.linkedin.restli.server.RestRestLiServer.handleResourceRequest(RestRestLiServer.java:262)\n\tat com.linkedin.restli.server.RestRestLiServer.handleResourceRequest(RestRestLiServer.java:232)\n\tat com.linkedin.restli.server.RestRestLiServer.doHandleRequest(RestRestLiServer.java:215)\n\tat com.linkedin.restli.server.RestRestLiServer.handleRequest(RestRestLiServer.java:171)\n\tat com.linkedin.restli.server.RestLiServer.handleRequest(RestLiServer.java:130)\n\tat com.linkedin.restli.server.DelegatingTransportDispatcher.handleRestRequest(DelegatingTransportDispatcher.java:70)\n\tat com.linkedin.r2.filter.transport.DispatcherRequestFilter.onRestRequest(DispatcherRequestFilter.java:70)\n\tat com.linkedin.r2.filter.TimedRestFilter.onRestRequest(TimedRestFilter.java:76)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:146)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:132)\n\tat com.linkedin.r2.filter.FilterChainIterator.onRequest(FilterChainIterator.java:62)\n\tat com.linkedin.r2.filter.TimedNextFilter.onRequest(TimedNextFilter.java:55)\n\tat com.linkedin.r2.filter.transport.ServerQueryTunnelFilter.onRestRequest(ServerQueryTunnelFilter.java:58)\n\tat com.linkedin.r2.filter.TimedRestFilter.onRestRequest(TimedRestFilter.java:76)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:146)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:132)\n\tat com.linkedin.r2.filter.FilterChainIterator.onRequest(FilterChainIterator.java:62)\n\tat com.linkedin.r2.filter.TimedNextFilter.onRequest(TimedNextFilter.java:55)\n\tat com.linkedin.r2.filter.message.rest.RestFilter.onRestRequest(RestFilter.java:50)\n\tat com.linkedin.r2.filter.TimedRestFilter.onRestRequest(TimedRestFilter.java:76)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:146)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:132)\n\tat com.linkedin.r2.filter.FilterChainIterator.onRequest(FilterChainIterator.java:62)\n\tat com.linkedin.r2.filter.FilterChainImpl.onRestRequest(FilterChainImpl.java:106)\n\tat com.linkedin.r2.filter.transport.FilterChainDispatcher.handleRestRequest(FilterChainDispatcher.java:75)\n\tat com.linkedin.r2.util.finalizer.RequestFinalizerDispatcher.handleRestRequest(RequestFinalizerDispatcher.java:61)\n\tat com.linkedin.r2.transport.http.server.HttpDispatcher.handleRequest(HttpDispatcher.java:101)\n\tat com.linkedin.r2.transport.http.server.AbstractR2Servlet.service(AbstractR2Servlet.java:105)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat com.linkedin.restli.server.spring.ParallelRestliHttpRequestHandler.handleRequest(ParallelRestliHttpRequestHandler.java:63)\n\tat org.springframework.web.context.support.HttpRequestHandlerServlet.service(HttpRequestHandlerServlet.java:73)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)\n\tat org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)\n\tat com.datahub.authentication.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:88)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\n\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\n\tat java.lang.Thread.run(Thread.java:748)\n',
    Is it impossible to get aspects data by using GMS api?
    s
    b
    9 replies
    Copy to Clipboard