https://datahubproject.io logo
Join SlackCommunities
Powered by
Hello. How I can connect recipe trough ssl to supe...
# getting-started
f
Hello. How I can connect recipe trough ssl to superset? and setting frontend to navegate to Superset? Thanks
m
Hey @freezing-teacher-87574 sorry we weren’t able to support you well on this one. Were you able to make any further progress on this?
@square-activity-64562: when you have some time, could you check how to make the ingestion system work with ssl-enabled superset?
s
@freezing-teacher-87574 Is the certificate valid for the hosted superset that you are trying to connect to? I was able to connect to 
<https://superset.demo.datahubproject.io/>
and get the data in a file. You can use https://www.ssllabs.com/ssltest/ to test your hosted version for valid certificate. If the certificate is present on local machine you can try to use 
REQUESTS_CA_BUNDLE
to verify the certificate https://stackoverflow.com/questions/31448854/how-to-force-requests-use-the-certificates-on-my-ubuntu-system
plus1 1
f
Thanks @square-activity-64562 @mammoth-bear-12532 I going to check again today, I'll tell you the avance. Thanks
Hi @mammoth-bear-12532 @square-activity-64562 This option 
REQUESTS_CA_BUNDLE
fixed the problem to tls connection. Now I've the following error when I'm trying to execute the recipe: 
datahub ingest -c /root/recipes/from_superset.yaml
Recipe:
Copy code
source:
  type: "superset"
  config:
    username: user
    password: password
    provider: db
    connect_uri: <https://superset.XXXXX.com>
sink:
  type: "datahub-rest"
  config:
    server: "<http://XXX.XXX.XXX.XX:8080>"
I've got the next error:
Copy code
[2021-11-03 07:31:59,949] INFO {datahub.cli.ingest_cli:57} - Starting metadata ingestion
 [2021-11-03 07:32:00,751] ERROR {datahub.entrypoints:99} - File "/usr/local/lib/python3.8/site-packages/datahub/entrypoints.py", line 91, in main
 88 def main(**kwargs):
 89 # This wrapper prevents click from suppressing errors.
 90 try:
 --> 91 sys.exit(datahub(standalone_mode=False, **kwargs))
 92 except click.exceptions.Abort:
 ..................................................
 kwargs = {}
 datahub = <Group datahub>
 click.exceptions.Abort = <class 'click.exceptions.Abort'>
 ..................................................

File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
 1126 def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
 (...)
 --> 1128 return self.main(*args, **kwargs)
 ..................................................
 self = <Group datahub>
 args = ()
 t.Any = typing.Any
 kwargs = {'standalone_mode': False}
 ..................................................

File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
 rv = self.invoke(ctx)
 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
 return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
 return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
 return ctx.invoke(self.callback, **ctx.params)
 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
 return __callback(*args, **kwargs)
 File "/usr/local/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 58, in run
 43 def run(config: str) -> None:
 (...)
 54 click.echo(e, err=True)
 55 sys.exit(1)
 56
 57 <http://logger.info|logger.info>("Starting metadata ingestion")
 --> 58 pipeline.run()
 59 <http://logger.info|logger.info>("Finished metadata ingestion")
 ..................................................
 run = <Command run>
 config = '/root/recipes/from_superset.yaml'
 <http://logger.info|logger.info> = <method '<http://Logger.info|Logger.info>' of <Logger datahub.cli.ingest_cli (INFO)> __init__.py:1436>
 pipeline.run = <method 'Pipeline.run' of <datahub.ingestion.run.pipeline.Pipeline object at 0x7fbdd84dca60> pipeline.py:122>
 ..................................................

File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 125, in run
 122 def run(self) -> None:
 123 callback = LoggingCallback()
 124 extractor: Extractor = self.extractor_class()
 --> 125 for wu in self.source.get_workunits():
 126 # TODO: change extractor interface
 ..................................................
 self = <datahub.ingestion.run.pipeline.Pipeline object at 0x7fbdd84dca60>
 callback = <datahub.ingestion.run.pipeline.LoggingCallback object at 0x7fbdd84dcca0>
 LoggingCallback = <class 'datahub.ingestion.run.pipeline.LoggingCallback'>
 extractor = <datahub.ingestion.extractor.mce_extractor.WorkUnitRecordExtractor object at 0x7fbd7637df40>
 Extractor = <class 'datahub.ingestion.api.source.Extractor'>
 self.extractor_class = <class 'datahub.ingestion.extractor.mce_extractor.WorkUnitRecordExtractor'>
 self.source.get_workunits = <method 'SupersetSource.get_workunits' of SupersetSource(ctx=PipelineContext(run_id='superset-2021_11_03-07_31_46')) sup
 erset.py:316>
 ..................................................

File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/superset.py", line 317, in get_workunits
 316 def get_workunits(self) -> Iterable[MetadataWorkUnit]:
 --> 317 yield from self.emit_dashboard_mces()
 318 yield from self.emit_chart_mces()
 ..................................................
 self = SupersetSource(ctx=PipelineContext(run_id='superset-2021_11_03-07_31_46'))
 Iterable = typing.Iterable
 MetadataWorkUnit = <class 'datahub.ingestion.api.workunit.MetadataWorkUnit'>
 self.emit_dashboard_mces = <method 'SupersetSource.emit_dashboard_mces' of SupersetSource(ctx=PipelineContext(run_id='superset-2021_11_03-07_31_46'
 )) superset.py:210>
 self.emit_chart_mces = <method 'SupersetSource.emit_chart_mces' of SupersetSource(ctx=PipelineContext(run_id='superset-2021_11_03-07_31_46')) s
 uperset.py:293>
 ..................................................

File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/superset.py", line 226, in emit_dashboard_mces
 210 def emit_dashboard_mces(self) -> Iterable[MetadataWorkUnit]:
 (...)
 222
 223 current_dashboard_page += 1
 224
 225 payload = dashboard_response.json()
 --> 226 for dashboard_data in payload["result"]:
 227 dashboard_snapshot = self.construct_dashboard_from_api_data(
 ..................................................
 self = SupersetSource(ctx=PipelineContext(run_id='superset-2021_11_03-07_31_46'))
 Iterable = typing.Iterable
 MetadataWorkUnit = <class 'datahub.ingestion.api.workunit.MetadataWorkUnit'>
 current_dashboard_page = 1
 payload = {'msg': 'Missing Authorization Header'}
 dashboard_response.json = <method 'Response.json' of <Response [401]> models.py:881>
 self.construct_dashboard_from_api_data = <method 'SupersetSource.construct_dashboard_from_api_data' of SupersetSource(ctx=PipelineContext(run_id='superset-2021_1
 1_03-07_31_46')) superset.py:168>
 ..................................................

---- (full traceback above) ----
 File "/usr/local/lib/python3.8/site-packages/datahub/entrypoints.py", line 91, in main
 sys.exit(datahub(standalone_mode=False, **kwargs))
 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
 return self.main(*args, **kwargs)
 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
 rv = self.invoke(ctx)
 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
 return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
 return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
 return ctx.invoke(self.callback, **ctx.params)
 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
 return __callback(*args, **kwargs)
 File "/usr/local/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 58, in run
 pipeline.run()
 File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 125, in run
 for wu in self.source.get_workunits():
 File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/superset.py", line 317, in get_workunits
 yield from self.emit_dashboard_mces()
 File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/superset.py", line 226, in emit_dashboard_mces
 for dashboard_data in payload["result"]:

KeyError: 'result'
 command terminated with exit code 1
Please, ¿Can you help me or suggest an idea? . I'm new with Feast and trying to configure a proof of concept. Thanks.
s
@freezing-teacher-87574 Are there dashboards in your superset instance? Are you perhaps able to use superset's REST API docs to try to see if REST API is working for your superset instance? https://superset.apache.org/docs/rest-api
k
@freezing-teacher-87574 I have the same problem with superset.. initially I felt or was the middleware stripping off the authorization headers.. apparently that's not the case and I have not a chance to look at it in anymore details.. I will share if I find something
thank you 1
f
Hi @square-activity-64562 I tried this command:
Copy code
curl -X 'GET' \
  '<https://xxxxxxx/api/v1/dashboard/?q=%7B%0A%22columns%22%3A%20%5B%0A%22dashboard_title%22%0A%5D%2C%0A%22filters%22%3A%20%5B%0A%7B%0A%22col%22%3A%20%22dashboard_title%22%2C%0A%22opr%22%3A%20%22sw%22%2C%0A%22value%22%3A%20%22Video%20Game%20Sales%22%0A%7D%0A%5D%2C%0A%22keys%22%3A%20%5B%0A%22none%22%0A%5D%2C%0A%22order_column%22%3A%20%22dashboard_title%22%2C%0A%22order_direction%22%3A%20%22asc%22%0A%7D>' \
  -H 'accept: application/json'
I got this result:
Copy code
{
  "count": 1,
  "ids": [
    10
  ],
  "result": [
    {
      "dashboard_title": "Video Game Sales"
    }
  ]
}
I think is ok the configuration than you suggest. If you see the trace, there is an error with message "payload = {'msg': 'Missing Authorization Header'}" : [2021-11-03 073159,949] INFO {datahub.cli.ingest_cli:57} - Starting metadata ingestion [2021-11-03 073200,751] ERROR {datahub.entrypoints:99} - File "/usr/local/lib/python3.8/site-packages/datahub/entrypoints.py", line 91, in main 88 def main(**kwargs): 89 # This wrapper prevents click from suppressing errors. 90 try: --> 91 sys.exit(datahub(standalone_mode=False, **kwargs)) 92 except click.exceptions.Abort: .................................................. kwargs = {} datahub = <Group datahub> click.exceptions.Abort = <class 'click.exceptions.Abort'> .................................................. File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in call 1126 def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any: (...) --> 1128 return self.main(*args, **kwargs) .................................................. self = <Group datahub> args = () t.Any = typing.Any kwargs = {'standalone_mode': False} .................................................. File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 58, in run 43 def run(config: str) -> None: (...) 54 click.echo(e, err=True) 55 sys.exit(1) 56 57 logger.info("Starting metadata ingestion") --> 58 pipeline.run() 59 logger.info("Finished metadata ingestion") .................................................. run = <Command run> config = '/root/recipes/from_superset.yaml' logger.info = <method 'Logger.info' of <Logger datahub.cli.ingest_cli (INFO)> init.py:1436> pipeline.run = <method 'Pipeline.run' of <datahub.ingestion.run.pipeline.Pipeline object at 0x7fbdd84dca60> pipeline.py:122> .................................................. File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 125, in run 122 def run(self) -> None: 123 callback = LoggingCallback() 124 extractor: Extractor = self.extractor_class() --> 125 for wu in self.source.get_workunits(): 126 # TODO: change extractor interface .................................................. self = <datahub.ingestion.run.pipeline.Pipeline object at 0x7fbdd84dca60> callback = <datahub.ingestion.run.pipeline.LoggingCallback object at 0x7fbdd84dcca0> LoggingCallback = <class 'datahub.ingestion.run.pipeline.LoggingCallback'> extractor = <datahub.ingestion.extractor.mce_extractor.WorkUnitRecordExtractor object at 0x7fbd7637df40> Extractor = <class 'datahub.ingestion.api.source.Extractor'> self.extractor_class = <class 'datahub.ingestion.extractor.mce_extractor.WorkUnitRecordExtractor'> self.source.get_workunits = <method 'SupersetSource.get_workunits' of SupersetSource(ctx=PipelineContext(run_id='superset-2021_11_03-07_31_46')) sup erset.py:316> .................................................. File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/superset.py", line 317, in get_workunits 316 def get_workunits(self) -> Iterable[MetadataWorkUnit]: --> 317 yield from self.emit_dashboard_mces() 318 yield from self.emit_chart_mces() .................................................. self = SupersetSource(ctx=PipelineContext(run_id='superset-2021_11_03-07_31_46')) Iterable = typing.Iterable MetadataWorkUnit = <class 'datahub.ingestion.api.workunit.MetadataWorkUnit'> self.emit_dashboard_mces = <method 'SupersetSource.emit_dashboard_mces' of SupersetSource(ctx=PipelineContext(run_id='superset-2021_11_03-07_31_46' )) superset.py:210> self.emit_chart_mces = <method 'SupersetSource.emit_chart_mces' of SupersetSource(ctx=PipelineContext(run_id='superset-2021_11_03-07_31_46')) s uperset.py:293> .................................................. File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/superset.py", line 226, in emit_dashboard_mces 210 def emit_dashboard_mces(self) -> Iterable[MetadataWorkUnit]: (...) 222 223 current_dashboard_page += 1 224 225 payload = dashboard_response.json() --> 226 for dashboard_data in payload["result"]: 227 dashboard_snapshot = self.construct_dashboard_from_api_data( .................................................. self = SupersetSource(ctx=PipelineContext(run_id='superset-2021_11_03-07_31_46')) Iterable = typing.Iterable MetadataWorkUnit = <class 'datahub.ingestion.api.workunit.MetadataWorkUnit'> current_dashboard_page = 1 payload = {'msg': 'Missing Authorization Header'} dashboard_response.json = <method 'Response.json' of <Response [401]> models.py:881> self.construct_dashboard_from_api_data = <method 'SupersetSource.construct_dashboard_from_api_data' of SupersetSource(ctx=PipelineContext(run_id='superset-2021_1 1_03-07_31_46')) superset.py:168> .................................................. ---- (full traceback above) ---- File "/usr/local/lib/python3.8/site-packages/datahub/entrypoints.py", line 91, in main sys.exit(datahub(standalone_mode=False, **kwargs)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in call return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 58, in run pipeline.run() File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 125, in run for wu in self.source.get_workunits(): File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/superset.py", line 317, in get_workunits yield from self.emit_dashboard_mces() File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/superset.py", line 226, in emit_dashboard_mces for dashboard_data in payload["result"]: KeyError: 'result' command terminated with exit code 1 Have you any suggests? Thanks!
Hi @kind-dawn-17532 We fixed this error after install the Datahub's version 0.8.17
2 Views
Open in Slack
PreviousNext
Powered by