v0.8.7 profiling feature We have some tables with ...
# troubleshoot
s
v0.8.7 profiling feature We have some tables with 0 rows. With all table's profiling enabled we get errors
Copy code
12:36:51  File "/usr/local/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 58, in run
12:36:51      pipeline.run()
12:36:51  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 108, in run
12:36:51      for wu in self.source.get_workunits():
12:36:51  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/sql/sql_common.py", line 318, in get_workunits
12:36:51      yield from self.loop_profiler(
12:36:51  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/sql/sql_common.py", line 513, in loop_profiler
12:36:51      profile = profiler.generate_profile(
12:36:51  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 118, in generate_profile
12:36:51      profile = self._convert_evrs_to_profile(evrs, pretty_name=pretty_name)
12:36:51  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 165, in _convert_evrs_to_profile
12:36:51      self._handle_convert_column_evrs(
12:36:51  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 222, in _handle_convert_column_evrs
12:36:51      column_profile.nullProportion = res["unexpected_percent"] / 100
12:36:51  
12:36:51  TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'
I have this section for profiling. I was trying to remove some tables to avoid above error
Copy code
profiling:
      enabled: true
    profile_pattern:
      allow:
        - "superset.public.*"
      deny:
        - "*.ab_register_user"
        - "*.access_request"
        - "*.alert*"
        - "*.annotation*"
        - "*.clusters"
        - "*.columns"
        - "*emails*"
        - "*log*"
and got errors following errors. Not sure what I am doing wrong.
Copy code
12:44:50  File "/usr/local/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 58, in run
12:44:50      pipeline.run()
12:44:50  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 108, in run
12:44:50      for wu in self.source.get_workunits():
12:44:50  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/sql/sql_common.py", line 318, in get_workunits
12:44:50      yield from self.loop_profiler(
12:44:50  File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/sql/sql_common.py", line 508, in loop_profiler
12:44:50      if not sql_config.profile_pattern.allowed(dataset_name):
12:44:50  File "/usr/local/lib/python3.8/site-packages/datahub/configuration/common.py", line 69, in allowed
12:44:50      if re.match(deny_pattern, string):
12:44:50  File "/usr/local/lib/python3.8/re.py", line 191, in match
12:44:50      return _compile(pattern, flags).match(string)
12:44:50  File "/usr/local/lib/python3.8/re.py", line 304, in _compile
12:44:50      p = sre_compile.compile(pattern, flags)
12:44:50  File "/usr/local/lib/python3.8/sre_compile.py", line 764, in compile
12:44:50      p = sre_parse.parse(p, flags)
12:44:50  File "/usr/local/lib/python3.8/sre_parse.py", line 948, in parse
12:44:50      p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
12:44:50  File "/usr/local/lib/python3.8/sre_parse.py", line 443, in _parse_sub
12:44:50      itemsappend(_parse(source, state, verbose, nested + 1,
12:44:50  File "/usr/local/lib/python3.8/sre_parse.py", line 668, in _parse
12:44:50      raise source.error("nothing to repeat",
12:44:50  
12:44:50  error: nothing to repeat at position 0
I used this and it worked. So the feature is working. But there are edge cases where it is failing
Copy code
profiling:
      enabled: true
    profile_pattern:
      allow:
        - "superset.public.ab_role"
        - "superset.public.ab_user_role"
        - "superset.public.dashboards"
        - "superset.public.slices"
        - "superset.public.slice_user"
m
Thanks for letting us know! We’ll fix the corner cases.