https://datahubproject.io logo
Join Slack
Powered by
Hello, I wanted to report an issue with running t...
# troubleshoot
b
Hello, I wanted to report an issue with running the 
datahub get --urn
command and retrieving urn name in general. I was able to ingest data to datahub but after clicking the icon on a website to get urn name, nothing is copied and the button does not seem to take any effect. I had tested this on three different web browsers. After trying to get the urn name by running the 
datahub get --urn
command I receive following error:
Copy code
$ datahub get --urn "urn urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/some_data/results/data_info.csv,DEV)"
/home/mluser/.local/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/x509.py:14: CryptographyDeprecationWarning: This version of cryptography contains a temporary pyOpenSSL fallback path. Upgrade pyOpenSSL now.
  warnings.warn(
[2022-05-09 10:24:11,560] ERROR    {datahub.entrypoints:152} - File "/home/mluser/.local/lib/python3.8/site-packages/datahub/entrypoints.py", line 138, in main
    135  def main(**kwargs):
    136      # This wrapper prevents click from suppressing errors.
    137      try:
--> 138          sys.exit(datahub(standalone_mode=False, **kwargs))
    139      except click.exceptions.Abort:
    ..................................................
     kwargs = {}
     datahub = <Group datahub>
     click.exceptions.Abort = <class 'click.exceptions.Abort'>
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    1135  def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
 (...)
--> 1137      return self.main(*args, **kwargs)
    ..................................................
     self = <Group datahub>
     args = ()
     t.Any = typing.Any
     kwargs = {'standalone_mode': False}
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 304, in wrapper
    247  def wrapper(*args: Any, **kwargs: Any) -> Any:
 (...)
    300                  "status": "error",
    301                  "error": get_full_class_name(e),
    302              },
    303          )
--> 304          raise e
    ..................................................
     args = (<click.core.Context object at 0x7f82fa591940>, )
     Any = typing.Any
     kwargs = {'urn': 'urn urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/s
               ome_data/results/data_info.csv,DEV)',
               'aspect': ()}
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 256, in wrapper
    247  def wrapper(*args: Any, **kwargs: Any) -> Any:
 (...)
    252      telemetry_instance.ping(
    253          "function-call", {"function": function, "status": "start"}
    254      )
    255      try:
--> 256          res = func(*args, **kwargs)
    257          telemetry_instance.ping(
    ..................................................
     args = (<click.core.Context object at 0x7f82fa591940>, )
     Any = typing.Any
     kwargs = {'urn': 'urn urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/s
               ome_data/results/data_info.csv,DEV)',
               'aspect': ()}
     telemetry_instance.ping = <method 'Telemetry.ping' of <datahub.telemetry.telemetry.Telemetry object at 0x7f82eca5eb20> telemetry.py:201>
     function = 'datahub.cli.get_cli.get'
     func = <function 'get' get_cli.py:14>
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/datahub/cli/get_cli.py", line 38, in get
    25   def get(ctx: Any, urn: Optional[str], aspect: List[str]) -> None:
 (...)
    34           urn = ctx.args[0]
    35           logger.debug(f"Using urn from args {urn}")
    36       click.echo(
    37           json.dumps(
--> 38               get_aspects_for_entity(entity_urn=urn, aspects=aspect, typed=False),
    39               sort_keys=True,
    ..................................................
     get = <Command get>
     ctx = <click.core.Context object at 0x7f82fa591940>
     Any = typing.Any
     urn = 'urn urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/s
               ome_data/results/data_info.csv,DEV)'
     Optional = typing.Optional
     aspect = ()
     List = typing.List
     ctx.args = []
     logger.debug = <method 'Logger.debug' of <Logger datahub.cli.get_cli (INFO)> __init__.py:1424>
     json.dumps = <function 'dumps' __init__.py:183>
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/datahub/cli/cli_utils.py", line 658, in get_aspects_for_entity
    648  def get_aspects_for_entity(
    649      entity_urn: str,
    650      aspects: List[str],
    651      typed: bool = False,
    652      cached_session_host: Optional[Tuple[Session, str]] = None,
    653  ) -> Dict[str, Union[dict, DictWrapper]]:
    654      # Process non-timeseries aspects
    655      non_timeseries_aspects: List[str] = [
    656          a for a in aspects if a not in timeseries_class_to_aspect_name_map.values()
    657      ]
--> 658      entity_response = get_entity(
    659          entity_urn, non_timeseries_aspects, cached_session_host
    ..................................................
     entity_urn = 'urn urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/s
               ome_data/results/data_info.csv,DEV)'
     aspects = ()
     List = typing.List
     typed = False
     cached_session_host = None
     Optional = typing.Optional
     Tuple = typing.Tuple
     Session = <class 'requests.sessions.Session'>
     Dict = typing.Dict
     Union = typing.Union
     DictWrapper = <class 'avrogen.dict_wrapper.DictWrapper'>
     non_timeseries_aspects = []
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/datahub/cli/cli_utils.py", line 508, in get_entity
    492  def get_entity(
    493      urn: str,
    494      aspect: Optional[List] = None,
    495      cached_session_host: Optional[Tuple[Session, str]] = None,
    496  ) -> Dict:
 (...)
    504          encoded_urn: str = urn
    505      elif urn.startswith("urn:"):
    506          encoded_urn = urllib.parse.quote(urn)
    507      else:
--> 508          raise Exception(
    509              f"urn {urn} does not seem to be a valid raw (starts with urn:) or encoded urn (starts with urn%3A)"
    ..................................................
     urn = 'urn urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/s
               ome_data/results/data_info.csv,DEV)'
     aspect = []
     Optional = typing.Optional
     List = typing.List
     cached_session_host = None
     Tuple = typing.Tuple
     Session = <class 'requests.sessions.Session'>
     Dict = typing.Dict
     urllib.parse.quote = <function 'quote' parse.py:799>
    ..................................................

---- (full traceback above) ----
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/entrypoints.py", line 138, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 304, in wrapper
    raise e
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 256, in wrapper
    res = func(*args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/cli/get_cli.py", line 38, in get
    get_aspects_for_entity(entity_urn=urn, aspects=aspect, typed=False),
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/cli/cli_utils.py", line 658, in get_aspects_for_entity
    entity_response = get_entity(
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/cli/cli_utils.py", line 508, in get_entity
    raise Exception(

Exception: urn urn urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/some_data/results/data_info.csv,DEV) does not seem to be a valid raw (starts with urn:) or encoded urn (starts with urn%3A)
[2022-05-09 10:24:11,561] INFO     {datahub.entrypoints:161} - DataHub CLI version: 0.8.31.6 at /home/mluser/.local/lib/python3.8/site-packages/datahub/__init__.py
[2022-05-09 10:24:11,561] INFO     {datahub.entrypoints:164} - Python version: 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] at /usr/bin/python3 on Linux-5.4.0-96-generic-x86_64-with-glibc2.29
[2022-05-09 10:24:11,561] INFO     {datahub.entrypoints:167} - GMS config {}
The gms logs after running 
docker logs -f datahub-gms
does not update when I am either clicking the get urn running the website button or when running the command mentioned above. There are no logs pointing to any error. This issue was mentioned earlier in my older threads and @hundreds-photographer-13496 was investigating the server logs that I had provided (by the way huge thanks for help!). Could you take a look and help resolve this issue or give some steps on how to get more info about the problem? Thanks in advance!
h
Hi @breezy-portugal-43538, please remove the 
urn
prefix part pf the value of the urn param 
datahub get --urn "urn urn:li:dataset:..."
. It should look like 
datahub get --urn "urn:li:dataset:..."
.
o
Not sure why the copy-to-clipboard feature in the UI isn't working for you though! What browsers have you tried it on? Does it work on https://demo.datahubproject.io for you?
b
Hi, so the error after removing urn, changed to:
Copy code
$ datahub get --urn "urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/some_data/results/data_info.csv,DEV)"
/home/mluser/.local/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/x509.py:14: CryptographyDeprecationWarning: This version of cryptography contains a temporary pyOpenSSL fallback path. Upgrade pyOpenSSL now.
  warnings.warn(
[2022-05-10 08:02:15,169] ERROR    {datahub.entrypoints:152} - File "/home/mluser/.local/lib/python3.8/site-packages/datahub/entrypoints.py", line 138, in main
    135  def main(**kwargs):
    136      # This wrapper prevents click from suppressing errors.
    137      try:
--> 138          sys.exit(datahub(standalone_mode=False, **kwargs))
    139      except click.exceptions.Abort:
    ..................................................
     kwargs = {}
     datahub = <Group datahub>
     click.exceptions.Abort = <class 'click.exceptions.Abort'>
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    1135  def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
 (...)
--> 1137      return self.main(*args, **kwargs)
    ..................................................
     self = <Group datahub>
     args = ()
     t.Any = typing.Any
     kwargs = {'standalone_mode': False}
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 304, in wrapper
    247  def wrapper(*args: Any, **kwargs: Any) -> Any:
 (...)
    300                  "status": "error",
    301                  "error": get_full_class_name(e),
    302              },
    303          )
--> 304          raise e
    ..................................................
     args = (<click.core.Context object at 0x7f6604aaf940>, )
     Any = typing.Any
     kwargs = {'urn': 'urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/some_
               data/results/data_info.csv,DEV)',
               'aspect': ()}
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 256, in wrapper
    247  def wrapper(*args: Any, **kwargs: Any) -> Any:
 (...)
    252      telemetry_instance.ping(
    253          "function-call", {"function": function, "status": "start"}
    254      )
    255      try:
--> 256          res = func(*args, **kwargs)
    257          telemetry_instance.ping(
    ..................................................
     args = (<click.core.Context object at 0x7f6604aaf940>, )
     Any = typing.Any
     kwargs = {'urn': 'urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/some_
               data/results/data_info.csv,DEV)',
               'aspect': ()}
     telemetry_instance.ping = <method 'Telemetry.ping' of <datahub.telemetry.telemetry.Telemetry object at 0x7f65f6f7cb20> telemetry.py:201>
     function = 'datahub.cli.get_cli.get'
     func = <function 'get' get_cli.py:14>
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/datahub/cli/get_cli.py", line 38, in get
    25   def get(ctx: Any, urn: Optional[str], aspect: List[str]) -> None:
 (...)
    34           urn = ctx.args[0]
    35           logger.debug(f"Using urn from args {urn}")
    36       click.echo(
    37           json.dumps(
--> 38               get_aspects_for_entity(entity_urn=urn, aspects=aspect, typed=False),
    39               sort_keys=True,
    ..................................................
     get = <Command get>
     ctx = <click.core.Context object at 0x7f6604aaf940>
     Any = typing.Any
     urn = 'urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/some_
               data/results/data_info.csv,DEV)'
     Optional = typing.Optional
     aspect = ()
     List = typing.List
     ctx.args = []
     logger.debug = <method 'Logger.debug' of <Logger datahub.cli.get_cli (INFO)> __init__.py:1424>
     json.dumps = <function 'dumps' __init__.py:183>
    ..................................................

File "/home/mluser/.local/lib/python3.8/site-packages/datahub/cli/cli_utils.py", line 661, in get_aspects_for_entity
    648  def get_aspects_for_entity(
    649      entity_urn: str,
    650      aspects: List[str],
    651      typed: bool = False,
    652      cached_session_host: Optional[Tuple[Session, str]] = None,
    653  ) -> Dict[str, Union[dict, DictWrapper]]:
 (...)
    657      ]
    658      entity_response = get_entity(
    659          entity_urn, non_timeseries_aspects, cached_session_host
    660      )
--> 661      aspect_list: Dict[str, dict] = entity_response["aspects"]
    662
    ..................................................
     entity_urn = 'urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/some_
               data/results/data_info.csv,DEV)'
     aspects = ()
     List = typing.List
     typed = False
     cached_session_host = None
     Optional = typing.Optional
     Tuple = typing.Tuple
     Session = <class 'requests.sessions.Session'>
     Dict = typing.Dict
     Union = typing.Union
     DictWrapper = <class 'avrogen.dict_wrapper.DictWrapper'>
     entity_response = {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException',
                        'stackTrace': 'com.linkedin.restli.server.RestLiServiceException [HTTP Status:404]\n\tat com.linkedin.restli.server.Res
                        tLiServiceException.fromThrowable(RestLiServiceException.java:315)\n\tat com.linkedin.restli.server.BaseRestLiServer.bui
                        ldPreRoutingError(BaseRestLiServer.java:202)\n\tat com.linkedin.restli.server.RestRestLiServer.buildPreRoutingRestExcept
                        ion(RestRestLiServer.java:254)\n\tat com.linkedin.restli.server.RestRestLiServer.handleResourceRequest(RestRestLiServer.
                        java:228)\n\tat com.linkedin.restli.server.RestRestLiServer.doHandleRequest(RestRestLiServer.java:215)\n\tat com.linkedi
                        n.restli.server.RestRestLiServer.handleRequest(RestRestLiServer.java:171)\n\tat com.linkedin.restli.server.RestLiServer.
                        handleRequest(RestLiServer.java:130)\n\tat com.linkedin.restli.server.DelegatingTransportDispatcher.handleRestRequest(De
                        legatingTransportDispatcher.java:70)\n\tat com.linkedin.r2.filter.transport.DispatcherRequestFilter.onRestRequest(Dispat
                        cherRequestFilter.java:70)\n\tat com.linkedin.r2.filter.TimedRestFilter.onRestRequest(TimedRestFilter.java:76)\n\tat com
                        .linkedin.r2.filter.FilterChainIterator$FilterCh...
     non_timeseries_aspects = []
    ..................................................

---- (full traceback above) ----
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/entrypoints.py", line 138, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/home/mluser/.local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 304, in wrapper
    raise e
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 256, in wrapper
    res = func(*args, **kwargs)
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/cli/get_cli.py", line 38, in get
    get_aspects_for_entity(entity_urn=urn, aspects=aspect, typed=False),
File "/home/mluser/.local/lib/python3.8/site-packages/datahub/cli/cli_utils.py", line 661, in get_aspects_for_entity
    aspect_list: Dict[str, dict] = entity_response["aspects"]

KeyError: 'aspects'
[2022-05-10 08:02:15,170] INFO     {datahub.entrypoints:161} - DataHub CLI version: 0.8.31.6 at /home/mluser/.local/lib/python3.8/site-packages/datahub/__init__.py
[2022-05-10 08:02:15,170] INFO     {datahub.entrypoints:164} - Python version: 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] at /usr/bin/python3 on Linux-5.4.0-96-generic-x86_64-with-glibc2.29
[2022-05-10 08:02:15,170] INFO     {datahub.entrypoints:167} - GMS config {}
And regarding the demo question - yes demo works for me fine. When this error is happening no new logs in 
docker logs -f datahub-gms
are visible.
o
It looks like your Python client is a few versions back, are you running the same version (0.8.31.6) for your server?
b
Hi, @orange-night-91387 I use the same version for the server and i can try to update python to see if it will take any effect - will let you know soon
o
If they're on the same version it should be fine. Does it have the same issue with all urns? Or just this one?
b
@orange-night-91387 It is happening all the time for all the runs. It's like the urn name cannot be retrieved at all. Are there any urn naming conventions that urn should not contain some special characters or something? I'm kind of wondering if this can be connected with it... what does it mean that [aspects] return keyError, do you know what this chunk of code that returns error tries to do? It looks like it tries to retrieve aspects value from some map, is there a way to see what this 
entity_response
contains and why aspects are missing?
h
@breezy-portugal-43538 is the dataset (and its details like schema, properties, etc) showing up alright in datahub UI ?
I believe the issue is due to non-Ok response from datahub server when fetching entity details by urn, when using cli. (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/cli/cli_utils.py#L539) Can you double check datahub cli setup ? Have you tried any other commands with cli ? Try 
datahub ingest list-runs
and see if it works.
b
@hundreds-photographer-13496 dataset is showing correctly. After executing 
datahub ingest list-runs
I receive this output:
Copy code
$ datahub ingest list-runs
/home/mluser/.local/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/x509.py:14: CryptographyDeprecationWarning: This version of cryptography contains a temporary pyOpenSSL fallback path. Upgrade pyOpenSSL now.
  warnings.warn(
+--------------------+--------+----------------------------+
| runId              |   rows | created at                 |
+====================+========+============================+
| no-run-id-provided |     60 | 2022-05-12 14:28:53 (EEST) |
+--------------------+--------+----------------------------+
I can double check the setup, but could you pinpoint the bullets that could have impact on this? Also, I had checked if the deletion of the urn works properly - it does. After using datahub delete --urn <urn> soft I receive correct response and urn is deleted.
h
Oh, I had a doubt if this is due to metadata service auth enabled, and token not setup correctly in datahub init. But doesn't seem like that's the cause, since other datahub cli commands are working fine.
b
Hello @hundreds-photographer-13496 I think I know the origin of the problem for the urn getting. So in my case when i trigger the command 
datahub get --urn "urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data/case_1/test/2022-05-06T14-30-26Z/data_2022-05-06T14-30-26Z/some_data/results/data_info.csv,DEV)"
I receive an error mentioned before, but after changing the '/' character to %2f as it's defined in the actual http url then everything works properly: 
datahub get --urn "urn:li:dataset:(urn:li:dataPlatform:s3,incoming_data%2Fcase_1%2Ftest%2F2022-05-06T14-30-26Z%2Fdata_2022-05-06T14-30-26Z%2Fsome_data%2Fresults%2Fdata_info.csv,DEV)"
for some reason when I upload the data from S3 as a source it takes the web url and transfers the '/' to %2f, but datahub requires the '.' character as a folder separator. I think this can be easily solved by replacing the / to . in url when S3 is a source. What do you think, could it be root cause? Thanks again for all the help : )
hi @hundreds-photographer-13496 Would it be possible to create from this thread github issue? I'd create a separate ticket for this problem, also - do you know if anyone else using S3 encountered this issue?
h
I haven't come across anyone else who has encountered this issue. Let's try to debug further today. Have you tried using file sink to dump resulting mces into file and checking what s3 urns are being generated ? https://datahubproject.io/docs/metadata-ingestion/sink_docs/file Because from my understanding, the slash / in urn should not cause the issue. Also, can you share output of this command you mentioned earlier? does it print only datasetKey aspect or other aspects too ? datahub get --urn "urnlidataset:(urnlidataPlatform:s3,incoming_data%2Fcase_1%2Ftest%2F2022-05-06T14-30-26Z%2Fdata_2022-05-06T14-30-26Z%2Fsome_data%2Fresults%2Fdata_info.csv,DEV)"
b
hi @hundreds-photographer-13496 Sorry for late reply, we are focusing right now on some other more important error, for some reason after running upload to datahub from docker container we receive 504 error with message of "too many bad requests" we are not sure where is it coming from. It happens after some longer period of time and it's quite sporadic, I had already created new thread for this one. As soon as we will resolve this issue, I will get back to the urn problem : )
👍 1
3 Views
Open in Slack
PreviousNext
Powered by