https://datahubproject.io logo
Join Slack
Powered by
# datahub-marathon-qa
  • b

    brainy-tent-14503

    02/08/2024, 4:52 PM
    https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/acryl-datahub-actions/templates/deployment.yaml#L145
  • b

    billions-butcher-90660

    02/08/2024, 4:56 PM
    @bulky-shoe-65107 unfortunately I couldn't stay longer, but I would really like to know if there are plans to develop ready-made implementations on other cloud providers such as DigitalOcean or Cloudflare, and if there is any documentation guidance for this.
    đź‘€ 1
    b
    l
    b
    • 4
    • 5
  • c

    careful-painting-49661

    02/08/2024, 4:57 PM
    General ingestion question - Have anyone from the community successfully connected to IBM DB2 data source? I know it is not a native support data source.
    đź‘€ 1
    d
    • 2
    • 2
  • l

    little-megabyte-1074

    02/08/2024, 5:00 PM
    Adding a reminder/placeholder - @bland-balloon-48379 - issue with browse path v2 emitting dupes
    âś… 1
  • d

    dazzling-judge-80093

    02/08/2024, 5:04 PM
    the db2 pr I was talking about -> https://github.com/datahub-project/datahub/issues/8835
    thank you 1
  • b

    brainy-tent-14503

    02/08/2024, 5:05 PM
    https://datahubproject.io/docs/quickstart/
    l
    • 2
    • 1
  • b

    bland-receptionist-85001

    02/08/2024, 5:18 PM
    Hi, How can I ingest metadata(dashboards, charts) from superset that is configured with oauth? Related question in slack https://datahubspace.slack.com/archives/CUMUWQU66/p1707082906520859
    âś… 1
  • b

    bland-balloon-48379

    02/08/2024, 5:19 PM
    Copy code
    Source (sqlalchemy) report:
    {'events_produced': 697,
     'events_produced_per_sec': 30,
     'entities': {'container': ['<example urns>'],
                  'dataset': ['<example urns>',
                              '... sampled of 114 total elements']},
     'aspects': {'container': {'containerProperties': 2, 'status': 2, 'dataPlatformInstance': 2, 'subTypes': 2, 'browsePathsV2': 4, 'container': 1},
                 'dataset': {'container': 114,
                             'status': 114,
                             'datasetProperties': 114,
                             'schemaMetadata': 114,
                             'dataPlatformInstance': 114,
                             'subTypes': 114,
                             'browsePathsV2': 228}},
     'warnings': {},
     'failures': {},
     'soft_deleted_stale_entities': [],
     'tables_scanned': 134,
     'views_scanned': 0,
     'entities_profiled': 0,
     'filtered': ['<example filtered schemas>',
                  '... sampled of 261 total elements'],
     'num_view_definitions_parsed': 0,
     'num_view_definitions_failed_parsing': 0,
     'num_view_definitions_failed_column_parsing': 0,
     'view_definitions_parsing_failures': [],
     'start_time': '2024-02-08 17:06:44.703672 (22.5 seconds ago)',
     'running_time': '22.5 seconds'}
    âś… 1
  • s

    some-alligator-9844

    02/08/2024, 5:27 PM
    Question 1: When we do UI based ingestion( using executor) it always creates the venv in the path "/tmp/datahub/ingest/venv*" Is there a way to change this path? In my usecase after VM restart the /tmp files are getting cleared and the ingestion doesn't work
    âś… 1
    l
    b
    • 3
    • 4
  • s

    some-alligator-9844

    02/08/2024, 5:28 PM
    Question 2: Getting this error while doing a fresh setup of datahub actions (executor). Tried 4-5 times after cleanup still getting the same error. Using Datahub v12.0 and latest Datahub Actions. Is there anything which I am missing?
    Copy code
    Traceback (most recent call last):
      File "/data/vdc/conda/condapub/svc_am_cicd/envs/dh-actions/bin/datahub-actions", line 5, in <module>
        from datahub_actions.entrypoints import main
      File "/data/vdc/conda/condapub/svc_am_cicd/envs/dh-actions/lib/python3.10/site-packages/datahub_actions/entrypoints.py", line 21, in <module>
        from datahub.cli.cli_utils import get_boolean_env_variable
    ImportError: cannot import name 'get_boolean_env_variable' from 'datahub.cli.cli_utils' (/data/vdc/conda/condapub/svc_am_cicd/envs/dh-actions/lib/python3.10/site-packages/datahub/cli/cli_utils.py)
    âś… 1
  • e

    early-account-74529

    02/08/2024, 5:42 PM
    Hi All, I am trying to deploy in kubernets with 0.12.1 Images , Gms is not coming up and I see continuous messages in pod as below, “2024-02-06 223952,134 [ThreadPoolTaskExecutor-1] WARN o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-mce-consumer-job-client-1, groupId=mce-consumer-job-client] Error while fetching metadata with correlation id 34542 : {MetadataChangeEvent_v4=INVALID_REPLICATION_FACTOR} 2024-02-06 223952,137 [ThreadPoolTaskExecutor-1] WARN o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-datahub-usage-event-consumer-job-client-4, groupId=datahub-usage-event-consumer-job-client] Error while fetching metadata with correlation id 35990 : {DataHubUsageEvent_v1=INVALID_REPLICATION_FACTOR} 2024-02-06 223952,137 [ThreadPoolTaskExecutor-1] WARN o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-datahub-usage-event-consumer-job-client-4, groupId=datahub-usage-event-consumer-job-client] Error while fetching metadata with correlation id 35990 : {DataHubUsageEvent_v1=INVALID_REPLICATION_FACTOR}”
    âś… 1
    b
    • 2
    • 8
  • l

    loud-rose-15723

    02/08/2024, 5:57 PM
    Hello! I have one question that is more related to shift-left strategies for metadata management on large companies. I would like to hear if a push-based approach using github actions is something you have seen in production environments. Like creating a custom parser of an YAML and create a custom action to transform it to receipts and ingest - distribute this to all application owners to have on their projects - schema, application description, outlets, inlets, etc. My biggest concern is how to keep those files in sync if someone changes the metadata in the UI. Any ideas of strategies to make this happen? WDYT about this?
    âś… 1
    l
    • 2
    • 2
  • b

    billions-butcher-90660

    02/08/2024, 6:42 PM
    Hey, I have a question. In the last #datahub-marathon-qa I participated in, I asked about whether it's possible to run the profile on the tables and columns only if the table has at least one row. Does it possible now?
    đź‘€ 1
    o
    d
    • 3
    • 2
  • b

    brainy-tent-14503

    02/08/2024, 6:49 PM
    https://github.com/datahub-project/datahub/blob/master/docs/api/openapi/openapi-structured-properties.md#applying-structured-properties
  • b

    brainy-tent-14503

    02/08/2024, 6:54 PM
    https://github.com/datahub-project/datahub/blob/master/metadata-service/openapi-se[…]a/io/datahubproject/openapi/v2/controller/EntityController.java
  • b

    bulky-shoe-65107

    02/09/2024, 3:02 PM
    GM <!channel> 🌻 thanks for attending the Marathon yest…a few quick things: 1. YOU ROCK! celebrate Thx for coming out and we’ll see you again on Mar 5th, 2024 yes? (hit 👍🏾) 2. Unresolved Items - Things move fast.. so if you’re question is still unresolved pl tag me in the thread so I can follow up! 3. Marathon Recording - Avail on Youtube right after the live! 4. DH Town Hall Case Studies - One of my fav parts of this event is learning what companies are doing with DH. If you are interested in doing a 15 min presentation on your DataHub adoption journey pl hit datahubbbb and I’ll be in touch! Have an amazing weekend! Mico xo PS - Our new calendar Luma sent out a quick survey.. pl take 2 mins to complete it with feedback!
    teamwork 1
    datahubbbb 1
    👍🏿 1
    👍🏻 1
  • l

    little-megabyte-1074

    03/07/2024, 1:21 PM
    I think this is the same thread that @handsome-hydrogen-42166 is speaking to https://datahubspace.slack.com/archives/CV2KB471C/p1707072605366349
  • s

    some-alligator-9844

    03/07/2024, 1:46 PM
    Datahub upgrade fails with error while running restore indices 2024-03-06 090138,278 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - metrics so far RestoreIndicesResult(ignored=0, rowsMigrated=19000, timeSqlQueryMs=3, timeGetRowMs=0, timeUrnMs=62, timeEntityRegistryCheckMs=11, aspectCheckMs=10, createRecordMs=1039, sendMessageMs=6485) 2024-03-06 090138,279 [pool-18-thread-1] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - Reading rows 19000 through 20000 from the aspects table started. 2024-03-06 090138,279 [pool-18-thread-1] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - Reading rows 19000 through 20000 from the aspects table completed. 2024-03-06 090138,279 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - Successfully sent MAEs for 19000/19518 rows (97.35% of total). 0 rows ignored (0.00% of total) 2024-03-06 090138,279 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - 0.36 mins taken. 0.01 est. mins to completion. Total mins est. = 0.37. 2024-03-06 090139,291 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090140,873 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090141,961 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090143,363 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090144,560 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090145,957 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090147,172 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090148,668 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090149,781 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090150,965 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090152,079 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:51 - Successfully fed bulk request. Number of events: 1 Took time ms: -1 2024-03-06 090152,332 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - metrics so far RestoreIndicesResult(ignored=0, rowsMigrated=19518, timeSqlQueryMs=3, timeGetRowMs=0, timeUrnMs=62, timeEntityRegistryCheckMs=11, aspectCheckMs=10, createRecordMs=1188, sendMessageMs=20023) 2024-03-06 090152,333 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - Successfully sent MAEs for 19518/19518 rows (100.00% of total). 0 rows ignored (0.00% of total) 2024-03-06 090152,333 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - 0.60 mins taken. 0.00 est. mins to completion. Total mins est. = 0.60. 2024-03-06 090152,335 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - Completed Step 3/3: SendMAEStep successfully. 2024-03-06 090152,335 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - Success! Completed upgrade with id RestoreIndices successfully. 2024-03-06 090152,336 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:15 - Upgrade RestoreIndices completed with result SUCCEEDED. Exiting... 2024-03-06 090152,341 [EbeanHook] INFO io.ebean.datasource:755 - DataSourcePool [gmsEbeanServiceConfig] shutdown min[2] max[50] free[2] busy[0] waiting[0] highWaterMark[1] waitCount[0] hitCount[43] psc[hit:16 miss:24 put:40 rem:0] 2024-03-06 090152,347 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:249 - Shutdown requested 2024-03-06 090152,347 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:252 - Shutting down 2024-03-06 090152,351 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,353 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,353 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,367 [R2 Nio Event Loop-5-1] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,367 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,367 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,367 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,368 [R2 Nio Event Loop-5-2] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,368 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:249 - Shutdown requested 2024-03-06 090152,368 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:252 - Shutting down 2024-03-06 090152,368 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,368 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,369 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,369 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,369 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,369 [R2 Nio Event Loop-5-3] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,369 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,370 [R2 Nio Event Loop-5-4] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,372 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:249 - Shutdown requested 2024-03-06 090152,372 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:252 - Shutting down 2024-03-06 090152,373 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,373 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,373 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,374 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,374 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,374 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,374 [R2 Nio Event Loop-3-1] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,374 [R2 Nio Event Loop-3-2] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,374 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:249 - Shutdown requested 2024-03-06 090152,374 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:252 - Shutting down 2024-03-06 090152,374 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,375 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,375 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,375 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,375 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,375 [R2 Nio Event Loop-3-3] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,375 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,376 [R2 Nio Event Loop-3-4] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,376 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:249 - Shutdown requested 2024-03-06 090152,376 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:252 - Shutting down 2024-03-06 090152,377 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,377 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,377 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,377 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,377 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,377 [R2 Nio Event Loop-1-1] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,377 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,378 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:249 - Shutdown requested 2024-03-06 090152,378 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.AbstractNettyClient:252 - Shutting down 2024-03-06 090152,378 [R2 Nio Event Loop-1-2] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,378 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,378 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,378 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,378 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools 2024-03-06 090152,378 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown 2024-03-06 090152,378 [SpringApplicationShutdownHook] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels 2024-03-06 090152,378 [R2 Nio Event Loop-1-3] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,379 [R2 Nio Event Loop-1-4] INFO c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete 2024-03-06 090152,387 [I/O dispatcher 1] ERROR c.l.m.s.e.update.BulkListener:63 - Error feeding bulk request. No retries left. Request: Failed to perform bulk request: index [vdc_int_tagindex_v2], optype: [UPDATE], type [UPDATE], id [urn%3Ali%3Atag%3AtestInt] org.apache.http.ConnectionClosedException: Connection closed unexpectedly at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.closed(HttpAsyncRequestExecutor.java:146) at org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:77) at org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:40) at org.apache.http.impl.nio.reactor.AbstractIODispatch.disconnected(AbstractIODispatch.java:100) at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionClosed(BaseIOReactor.java:277) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processClosedSessions(AbstractIOReactor.java:449) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:283) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) at java.base/java.lang.Thread.run(Thread.java:833)
  • i

    incalculable-sundown-8765

    03/07/2024, 2:22 PM
    Hi guys, I have a question on
    datahub delete
    . I want to hard delete everything related to redshift. However, I encounter this issue:
    Copy code
    % datahub delete --platform redshift --dry-run
    [2024-03-06 20:13:35,266] INFO     {datahub.cli.delete_cli:341} - Using DataHubGraph: configured to talk to <http://localhost:8080>
    [2024-03-06 20:13:36,009] ERROR    {datahub.entrypoints:201} - Command failed: ('Unable to get metadata from DataHub', {'message': '401 Client Error: Unauthorized for url: <http://localhost:8080/api/graphql'}>)
    Do I need token to run the command? If so, how can I include the token in the command? Thank you. Datahub version: v0.12.1
    i
    • 2
    • 2
  • c

    careful-painting-49661

    03/07/2024, 2:27 PM
    I have a few questions about ingestion - • Targeted Snowflake ingestion - https://datahubspace.slack.com/archives/CUMUWQU66/p1709237571387579 • Classification - https://datahubspace.slack.com/archives/C029A3M079U/p1709141510977839 • CSV ingestion through UI error - https://datahubspace.slack.com/archives/CUMUWQU66/p1709237181476599
    i
    • 2
    • 1
  • l

    little-megabyte-1074

    03/07/2024, 3:01 PM
    Hey folks! Sorry for the mixup - we accidentally ended the Zoom but we’re back live!
  • e

    early-librarian-13786

    03/07/2024, 3:07 PM
    Can we discuss this question? https://datahubspace.slack.com/archives/C029A3M079U/p1709132091519039?thread_ts=1704799365.187019&amp;cid=C029A3M079U
    b
    • 2
    • 4
  • b

    bulky-shoe-65107

    03/07/2024, 3:18 PM
    WE are live on Youtube again -

    https://www.youtube.com/watch?v=ayhg0UO4rB4â–ľ

  • b

    brainy-tent-14503

    03/07/2024, 3:54 PM
    @lively-noon-41363 - @gray-ocean-32209 pointed out this line https://github.com/datahub-project/datahub/blob/7d105477bc30751a0ee2f3ebadbb110c5b[…]ngestion/src/datahub/ingestion/source/snowflake/snowflake_v2.py and it doesn’t appear to be sorted. This likely means whatever snowflake does as far as returning consistent rows.
    g
    • 2
    • 1
  • m

    miniature-train-5342

    03/07/2024, 4:55 PM
    Getting started with Datahub- https://www.loom.com/share/225b9fb98aec4950beb1507fafdc3633?sid=9bc9bb08-2757-4a08-af6f-12b861c5c2ff Ingestion Basics- https://www.loom.com/share/d3ddf5d2983e4c59b81d1d0c6f5ea0e1?sid=ff7d6ad4-bfc0-465b-ac63-0ab1f3ac7602
  • l

    little-megabyte-1074

    03/07/2024, 6:00 PM
    @gray-gold-85760 adding you here so we can follow up on your search-related questions: • ability to retrieve search similarity scores • examples of GraphQL queries to construct AND/OR search queries • thoughts on using search embeds
    g
    • 2
    • 2
  • i

    icy-airplane-5350

    03/07/2024, 6:00 PM
    Want to bring up an error i had last week: https://datahubspace.slack.com/archives/C029A3M079U/p1709582354636159
    b
    m
    • 3
    • 4
  • d

    dry-raincoat-85182

    03/07/2024, 6:07 PM
    Hi Team, we wanted to discuss the use case of limiting the association of glossary terms to only certain types of entities like for example some terms should be applicable to dataset/table level and some should be applicable to only attribute/column level
  • e

    eager-school-14258

    03/07/2024, 6:38 PM
    Hello we have dbt and snowflake lineage. How can I differentiate between the two lineages? I keep seeing them together if enabled. Is there a way I can filter down dbt lineage versus snowflake lineage?
  • b

    boundless-bear-68728

    03/08/2024, 5:15 PM
    Hi Team, Following up on the issue that is causing in the ingestion of Snowflake records. I could see the following error from the logs:
    Copy code
    'The above exception was the direct cause of the following exception:\n'
               '\n'
               'Traceback (most recent call last):\n'
               '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/default_executor.py", line 122, in execute_task\n'
               '    task_event_loop.run_until_complete(task_future)\n'
               '  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete\n'
               '    return future.result()\n'
               '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 214, in execute\n'
               "    raise RuntimeError(f'Something went wrong in the subprocess executor: {e}') from e\n"
               "RuntimeError: Something went wrong in the subprocess executor: ('Unable to emit metadata to DataHub GMS', {'message': "
               "'HTTPConnectionPool(host=\\'datahub-datahub-gms\\', port=8080): Max retries exceeded with url: /aspects?action=ingestProposal (Caused by "
               'ReadTimeoutError("HTTPConnectionPool(host=\\\'datahub-datahub-gms\\\', port=8080): Read timed out. (read timeout=30)"))