polite-application-51650
05/26/2022, 6:55 AMgorgeous-optician-32034
05/26/2022, 4:37 PMstateful_ingestion
and configure it to delete datasets that don't show up in the latest ingestion. For example, see the stateful_*
configuration options on sources like hive. Loving datahub!lemon-hydrogen-83671
05/26/2022, 6:26 PMsimple_add_dataset_tags
transformer will replace any existing tags you have on a dataset. Keep an eye out for that!nutritious-bird-77396
05/26/2022, 10:25 PMselect * from metadata_aspect_v2 where urn like 'urn:li:dataHubIngestionSource:%' and version = 0 and aspect = 'dataHubIngestionSourceInfo'
Do you know what else could trigger this duplicate execution?clean-piano-28976
05/27/2022, 10:01 AMdatahub delete --env PROD --entity_type dataset --platform dbt --hard
calm-dinner-63735
05/28/2022, 6:52 AMpolite-application-51650
05/30/2022, 5:23 AMcuddly-arm-8412
05/30/2022, 8:50 AMsalmon-angle-92685
05/30/2022, 1:25 PMmodern-laptop-12942
05/30/2022, 2:48 PMwitty-butcher-82399
05/30/2022, 4:55 PMalert-football-80212
05/31/2022, 11:52 AMglamorous-microphone-33484
06/01/2022, 1:27 AMnumerous-diamond-76461
06/01/2022, 3:54 AMlemon-zoo-63387
06/01/2022, 7:51 AMlemon-zoo-63387
06/01/2022, 8:30 AMlemon-zoo-63387
06/01/2022, 9:15 AMsudo python3 -m pip install cx_Oracle --upgrade
sudo python3 -m pip install cx_Oracle --upgrade --user
cuddly-arm-8412
06/01/2022, 12:37 PMworried-painting-70907
06/01/2022, 2:13 PMapiVersion: <http://kafka.strimzi.io/v1beta2|kafka.strimzi.io/v1beta2>
kind: KafkaTopic
metadata:
name: my-topic
labels:
<http://strimzi.io/cluster|strimzi.io/cluster>: my-cluster
<http://infra.mycompany.net/dev_team|infra.mycompany.net/dev_team>: dev_team_a
<http://infra.mycompany.net/ops_team|infra.mycompany.net/ops_team>: ops_team_a
<http://infra.mycompany.net/app_id|infra.mycompany.net/app_id>: app-0001
spec:
partitions: 10
replicas: 3
config:
<http://retention.ms|retention.ms>: 604800000
segment.bytes: 1073741824
What would be the best way that I can either get this data into the datahub system? I dont know if it would be best to go and make commits to the kafka integration that supports connecting to kubernetes and getting the metadata, or if there's a better way? Kind of a huge edge case considering the metadata is in the kubernetes layer, but that's the only way we found to manage these kinds of metadatadry-zoo-35797
06/01/2022, 3:16 PMgorgeous-telephone-63628
06/01/2022, 7:19 PMworried-painting-70907
06/01/2022, 7:53 PMrich-policeman-92383
06/02/2022, 5:12 AMused by: java.lang.RuntimeException: Failed to validate entity URN '
'urn:li:dataset:(urn:li:dataPlatform:oracle,b0225565.PORTIN_,MAR22,PROD)\n'
'\tat com.linkedin.metadata.utils.EntityKeyUtils.getUrnFromProposal(EntityKeyUtils.java:33)\n'
'\tat com.linkedin.restli.internal.server.RestLiMethodInvoker.doInvoke(RestLiMethodInvoker.java:177)\n'
'\t... 78 more\n'
'Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields '
'do not have same length\n'
'\tat com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:97)\n'
'\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)\n'
'\tat java.lang.Thread.run(Thread.java:748)\n'
'Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException\n'
'\tat com.datahub.util.RecordUtils.invokeProtectedMethod(RecordUtils.java:355)\n'
'\tat com.datahub.util.RecordUtils.getRecordTemplateField(RecordUtils.java:258)\n'
'\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:30)\n'
'\t... 84 more\n'
'Caused by: java.lang.reflect.InvocationTargetException\n'
'\tat sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)\n'
'\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n'
'\tat com.datahub.util.RecordUtils.invokeProtectedMethod(RecordUtils.java:353)\n'
'\t... 90 more\n'
'Caused by: com.linkedin.data.template.TemplateOutputCastException: Invalid URN syntax: Invalid number of '
'keys.: urn:li:dataset:(urn:li:dataPlatform:oracle,b0225565.PORTIN_,MAR22,PROD)\n'
'\tat com.linkedin.common.urn.DatasetUrn$1.coerceOutput(DatasetUrn.java:78)\n'
'\tat com.linkedin.data.template.RecordTemplate.obtainCustomType(RecordTemplate.java:366)\n'
'\t... 94 more\n'
'Caused by: java.net.URISyntaxException: Invalid number of keys.: '
'urn:li:dataset:(urn:li:dataPlatform:oracle,b0225565.PORTIN_,MAR22,PROD)\n'
'\tat com.linkedin.common.urn.DatasetUrn.createFromUrn(DatasetUrn.java:49)\n'
'\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)\n'
'\tat java.lang.Thread.run(Thread.java:748)\n'
'Caused by: java.lang.RuntimeException: Failed to validate entity URN '
'urn:li:dataset:(urn:li:dataPlatform:oracle,b0225565.PORTIN_,MAR22,PROD)\n'
'\tat com.linkedin.metadata.utils.EntityKeyUtils.getUrnFromProposal(EntityKeyUtils.java:33)\n'
'\tat com.linkedin.restli.internal.server.RestLiMethodInvoker.doInvoke(RestLiMethodInvoker.java:177)\n'
'\t... 78 more\n'
'Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields '
'do not have same length\n'
'\tat com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:97)\n'
lemon-zoo-63387
06/02/2022, 7:37 AMsudo python3 -m pip install 'acryl-datahub[mssql]'
dry-zoo-35797
06/02/2022, 1:33 PMlemon-alarm-94169
06/02/2022, 2:36 PMinclude_table_lineage: True
on Column Stats tab. Is it a feature exclusive to Acryl data?lemon-alarm-94169
06/02/2022, 2:41 PMflat-window-44654
06/02/2022, 8:28 PMview_count
, updated_at
, last_updater_name
, etc.). How could we go about ingesting that additional metadata? Would we have to write a transformer?nutritious-bird-77396
06/02/2022, 9:57 PMuser.props
to be more secure, Either pulled in via ENV var or otherwise...Could you help on this?sparse-raincoat-42898
06/03/2022, 11:09 AM