https://datahubproject.io logo
Join Slack
Powered by
Trying to use the profiling feature for mssql but ...
# ingestion
c
Trying to use the profiling feature for mssql but running into an error.
Copy code
[2021-08-10 19:33:35,758] INFO     {great_expectations.data_context.data_context:2932} -        Profiled 9 columns using 0 rows from None (2.204 sec)
[2021-08-10 19:33:35,758] INFO     {great_expectations.data_context.data_context:2944} -
Profiled the data asset, with 0 total rows and 9 columns in 2.20 seconds.
Generated, evaluated, and stored 51 Expectations during profiling. Please review results using data-docs.
[2021-08-10 19:33:35,759] INFO     {root:1140} - Sending ROLLBACK TRAN
Traceback (most recent call last):
  File "/usr/local/bin/datahub", line 8, in <module>
    sys.exit(datahub())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/datahub/cli/ingest_cli.py", line 58, in run
    pipeline.run()
  File "/home/ec2-user/.local/lib/python3.7/site-packages/datahub/ingestion/run/pipeline.py", line 108, in run
    for wu in self.source.get_workunits():
  File "/home/ec2-user/.local/lib/python3.7/site-packages/datahub/ingestion/source/sql/sql_common.py", line 319, in get_workunits
    inspector, profiler, schema, sql_config
  File "/home/ec2-user/.local/lib/python3.7/site-packages/datahub/ingestion/source/sql/sql_common.py", line 515, in loop_profiler
    **self.prepare_profiler_args(schema=schema, table=table),
  File "/home/ec2-user/.local/lib/python3.7/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 118, in generate_profile
    profile = self._convert_evrs_to_profile(evrs, pretty_name=pretty_name)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 166, in _convert_evrs_to_profile
    profile, col, evrs_for_col, pretty_name=pretty_name
  File "/home/ec2-user/.local/lib/python3.7/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 222, in _handle_convert_column_evrs
    column_profile.nullProportion = res["unexpected_percent"] / 100
TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'
m
Thanks for letting us know @careful-insurance-60247
It seems like this is a table with 0 rows that we are stumbling over
For now you can skip over this table by adding a profile_pattern.deny rule.
We will fix the underlying issue meanwhile.
c
@mammoth-bear-12532 Thank you for the update. Adding the deny pattern worked.
How are the mappings determined?
Copy code
'TEST.dbo.testtable': ['unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema',
                                                  'unable to map type BIT() to metadata schema']
m
we maintain a set of SQL -> datahub-metadata-type mappings that we keep updating as we see issues like the one you are reporting 🙂
This is a somewhat benign error though, the schema will be ingested, you just won't see a great looking icon on this specific field
c
Is that something we could get added to the documentation? It would be nice to see what certain types get mapped to if its not a one to one.
5 Views
Open in Slack
PreviousNext
Powered by