incalculable-ocean-74010
02/16/2021, 12:53 PM@RestLiCollection(name = "streams", namespace = "com.linkedin.stream", keyName = "stream")
public final class Streams extends BaseBrowsableEntityResource<
// @formatter:off
ComplexResourceKey<StreamKey, EmptyRecord>,
Stream,
StreamUrn,
StreamSnapshot,
StreamAspect,
StreamDocument> {
@Inject
@Named("streamBrowseDao")
private BaseBrowseDAO _browseDAO;
@Inject
@Named("streamSearchDao")
private BaseSearchDAO _searchDAO;
@Inject
@Named("streamDao")
private BaseLocalDAO _localDAO;
...
orange-night-91387
02/16/2021, 9:58 PMline 531-533:
case STRING:
result = new Utf8((String) value);
break;
Value in this case is a DataMap representing a DatasetUrn. The Avro schema defines that as as a String type expecting something like "urnlidataset:..." but since it is in the form "{ platform: {...}, origin: ... , name: ...}" this case results in a ClassCastException. Is there a different way I can generate the GenericRecord with the format I have?
NOTE: This is NOT master, this is a separate development branch that I'm working on in a forked repo. Not a bug report, looking for advice 🙂curved-magazine-23582
02/17/2021, 3:26 AMmammoth-bear-12532
powerful-egg-69769
02/22/2021, 4:11 PMacoustic-printer-83045
02/25/2021, 9:45 PM{'upstreams': [
{'auditStamp': {'time': 0, 'actor': '', 'impersonator': None
}, 'dataset': 'urn:li:dataset:(urn:li:dataPlatform:redshift,events.analytics_dev_garylucas.carr_quarterly,PROD)', 'type': 'TRANSFORMED'
}
]
}
I don’t see an error from that but when I go to load lineage I get the following error in the back end (+ a UI error on the front end)
datahub-frontend | 21:36:25 [application-akka.actor.default-dispatcher-313] ERROR application - Fetch Dataset upstreams error
datahub-frontend | com.linkedin.data.template.TemplateOutputCastException: Invalid URN syntax: Urn doesn't start with 'urn:'. Urn: at index 0:
datahub-frontend | at com.linkedin.common.urn.UrnCoercer.coerceOutput(UrnCoercer.java:25)
datahub-frontend | at com.linkedin.common.urn.UrnCoercer.coerceOutput(UrnCoercer.java:11)
datahub-frontend | at com.linkedin.data.template.DataTemplateUtil.coerceOutput(DataTemplateUtil.java:954)
datahub-frontend | at com.linkedin.data.template.RecordTemplate.obtainCustomType(RecordTemplate.java:365)
datahub-frontend | at com.linkedin.common.AuditStamp.getActor(AuditStamp.java:159)
datahub-frontend | at com.linkedin.datahub.util.DatasetUtil.toLineageView(DatasetUtil.java:97)
datahub-frontend | at com.linkedin.datahub.dao.table.LineageDao.lambda$getUpstreamLineage$1(LineageDao.java:39)
datahub-frontend | at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
datahub-frontend | at java.util.Iterator.forEachRemaining(Iterator.java:116)
datahub-frontend | at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
datahub-frontend | at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
datahub-frontend | at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
datahub-frontend | at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
datahub-frontend | at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
datahub-frontend | at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
datahub-frontend | at com.linkedin.datahub.dao.table.LineageDao.getUpstreamLineage(LineageDao.java:40)
datahub-frontend | at controllers.api.v2.Dataset.getDatasetUpstreams(Dataset.java:250)
datahub-frontend | at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$28$$anonfun$apply$28.apply(Routes.scala:910)
datahub-frontend | at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$28$$anonfun$apply$28.apply(Routes.scala:910)
datahub-frontend | at play.core.routing.HandlerInvokerFactory$$anon$3.resultCall(HandlerInvoker.scala:134)
datahub-frontend | at play.core.routing.HandlerInvokerFactory$$anon$3.resultCall(HandlerInvoker.scala:133)
datahub-frontend | at play.core.routing.HandlerInvokerFactory$JavaActionInvokerFactory$$anon$8$$anon$2$$anon$1.invocation(HandlerInvoker.scala:108)
I’m pretty sure that I’ve misconfigured my upstream lineage object, however it passes validation on the way in. Any suggestions on how to troubleshoot this further?
Thanks in advance and I appreciate any insightincalculable-ocean-74010
03/01/2021, 10:25 AMpip install thrift
) in my python environment to get this farincalculable-ocean-74010
03/01/2021, 5:34 PMwhite-chef-85966
03/02/2021, 8:38 AMincalculable-ocean-74010
03/02/2021, 12:11 PMunable to map type DATE to metadata schema
• unable to map type TIMESTAMP to metadata schema
• unable to map type DECIMAL to metadata schema
calm-sunset-28996
03/02/2021, 2:28 PMincalculable-ocean-74010
03/02/2021, 5:43 PMcalm-sunset-28996
03/04/2021, 3:23 PMbrief-toothbrush-55766
03/05/2021, 12:37 PMpip install -e .
Obtaining file:///home/gama/SDAP/datahub/metadata-ingestion
ERROR: Command errored out with exit status 1:
command: /home/gama/SDAP/datahub/metadata-ingestion/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/gama/SDAP/datahub/metadata-ingestion/setup.py'"'"'; __file__='"'"'/home/gama/SDAP/datahub/metadata-ingestion/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-fs5vr9kr
cwd: /home/gama/SDAP/datahub/metadata-ingestion/
Complete output (5 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/gama/SDAP/datahub/metadata-ingestion/setup.py", line 57, in <module>
packages=setuptools.find_namespace_packages(where="./src"),
AttributeError: module 'setuptools' has no attribute 'find_namespace_packages'
----------------------------------------
WARNING: Discarding file:///home/gama/SDAP/datahub/metadata-ingestion. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
incalculable-ocean-74010
03/05/2021, 6:02 PMincalculable-ocean-74010
03/05/2021, 6:02 PMbrief-toothbrush-55766
03/06/2021, 12:31 PMbrief-toothbrush-55766
03/08/2021, 8:22 PMImportError: cannot import name 'TagSnapshotClass'
while trying to ingest a dataset with source:postgres-> sink:datahub-rest. Again, this worked before, then I did a git pull, started the venv(also installed GeoAlchemy2) and tried to ingest as before. Anything am missingßbreezy-glass-7892
03/09/2021, 9:20 AMdatahub ingest -c ./examples/recipes/example_to_datahub_rest.yml
; I don’t see the dataset in <http://localhost:9001>
. I also loaded the data from bigQuery
source:
type: bigquery
config:
project_id: data-sandbox-123
# options:
# credentials_path: "/service_account_key.json"
sink:
type: "datahub-rest"
config:
server: '<http://localhost:8080>'
Something I might be missing here?calm-sunset-28996
03/09/2021, 7:32 PMgentle-exabyte-43102
03/11/2021, 7:52 PMthrift.transport.TTransport.TTransportException: Bad status: 78 (b'5.7.22-log')
incalculable-ocean-74010
03/12/2021, 3:10 PMcurved-crayon-1929
03/16/2021, 5:16 AMcalm-lawyer-777
03/17/2021, 11:19 AMincalculable-ocean-74010
03/26/2021, 4:51 PMwonderful-quill-11255
03/28/2021, 1:45 PMcalm-lawyer-777
03/30/2021, 10:30 AMable-jelly-81126
03/30/2021, 2:36 PMhigh-hospital-85984
03/31/2021, 10:00 AMbrave-appointment-76997
03/31/2021, 11:21 AM