https://datahubproject.io logo
Join Slack
Powered by
Hello Guys, may you help me with a little question...
# ingestion
g
Hello Guys, may you help me with a little question about profiling, please? I'm doing the demo yml with my local SQL DB and colunms (min, max, mean, median) didn't work - Do you now what I doing wrong, please? thank you
b
What is the datatype of the fields in your DB? I ran into this issue and I believe it was some fields being strings. I might be wrong though.
d
as @brainy-park-34475 said min/max is only calculated for numeric fields. Please, check the types of those fields to make sure those are numeric fields.
g
Yes, and I have 3 colums witn numeric datatype - INT(11) and not working. Do you what is it wrong, please?
@brainy-park-34475 and @dazzling-judge-80093, any idea where I can fix this please?
b
Wondering if it has anything to do with them being set at primary keys. I honestly don't know enough to be helpful beyond my first comment. I am still learning datahub sorry.
g
@brainy-park-34475 ok, np - tks for any help 🙂
@early-lamp-41924, man, please do you have any idea to try fix it for me please? I'm doing the demo yml with my local SQL DB and colunms (min, max, mean, median) didn't work (images above)
@mammoth-bear-12532 please do you know who can help me with a little point? I'm doing the demo yml with my local SQL DB and columns (min, max, mean, median) didn't work
h
Hi @gentle-florist-49869, did you see any errors/warnings reported in the logs when you ran 
datahub ingest -c profiling_demo.yml
?
g
Hi @helpful-optician-78938, yes , I received something : [2022-01-26 082629,988] ERROR {datahub.utilities.sqlalchemy_query_combiner:385} - Failed to execute queue using combiner: (pymysql.err.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'anwhwlrchxngeoth AS \n(SELECT count(*) AS count_1 \nFROM source.compras)\n SELECT a' at line 1") Follow the all log attached
profiling.txt
h
@gray-shoe-75895, any idea why this error is happening?
👍 1
g
@gentle-florist-49869 could you try running with 
profiling.query_combiner_enabled
set to false?
g
@gray-shoe-75895 yes, let`s change my yml file , just a second please
@gray-shoe-75895, show the initial logs : fabiocastro@pop-os:~$ datahub ingest -c /home/fabiocastro/datahub/metadata-ingestion/examples/recipes/profilling_demo.yml [2022-01-26 221929,329] ERROR {datahub.entrypoints:110} - File "/home/fabiocastro/.local/lib/python3.9/site-packages/datahub/entrypoints.py", line 102, in main Follow attached all logs
profiling_v2.txt
g
Seems like it’s having trouble reading the yaml config file - could you post your (redacted) config?
Also I realize I should’ve been more clear - it should be nested like this: source -> config -> profiling -> 
query_combiner_enabled: false
g
@gray-shoe-75895 yes, I think I did this in yaml file before, follow this one:
profilling_demo.yml
d
@gentle-florist-49869 here is your recipe fixed:
Copy code
source:
  type: "mysql"
  config:
    #Credentials
    username: root
    password: root
    # Coordinates
    host_port: localhost:3307
    database: source
    schema_pattern:
        deny:
            - "information_schema"
            - "sys"
            - "mysql"
            - "performance_schema" 
    table_pattern:
        allow:
            - "source.compras"
    profiling:
        enabled: true
        #max_number_table_level_only: true
        max_number_of_fields_to_profile: 10
        include_field_min_value: true
        include_field_max_value: true
        include_field_mean_value: true
        include_field_median_value: true
        include_field_stddev_value: true
        query_combiner_enabled: false    
sink:
  type: "datahub-kafka"
  config:
    connection:
      bootstrap: "localhost:9092"
please, can you try this one?
g
@dazzling-judge-80093 tks a lot man...and I saw now that I forgot the letted "d" on word (enabled) - OMG 🙂
teamwork 1
@dazzling-judge-80093 @gray-shoe-75895, profiling run but datahub continue not showing the min, max, mean 😞 fabiocastro@pop-os:~$ datahub ingest -c /home/fabiocastro/datahub/metadata-ingestion/examples/recipes/profilling_demo.yml [2022-01-27 100939,769] INFO {datahub.cli.ingest_cli:81} - Starting metadata ingestion [2022-01-27 100940,771] INFO {datahub.ingestion.run.pipeline:79} - sink wrote workunit source.compras [2022-01-27 100940,772] INFO {datahub.ingestion.source.ge_data_profiler:744} - Will profile 1 table(s) with 1 worker(s) - this may take a while [2022-01-27 100940,792] INFO {datahub.ingestion.source.ge_data_profiler:797} - Profiling source.compras [2022-01-27 100940,812] INFO {datahub.ingestion.source.ge_data_profiler:814} - Finished profiling source.compras; took 0.020 seconds [2022-01-27 100941,772] INFO {datahub.ingestion.run.pipeline:79} - sink wrote workunit profile-source.compras [2022-01-27 100941,772] INFO {datahub.ingestion.source.ge_data_profiler:770} - Profiling 1 table(s) finished in 1.000 seconds [2022-01-27 100941,773] INFO {datahub.cli.ingest_cli:83} - Finished metadata ingestion Source (mysql) report: {'workunits_produced': 2, 'workunit_ids': ['source.compras', 'profile-source.compras'], 'warnings': {}, 'failures': {}, 'tables_scanned': 1, 'views_scanned': 0, 'entities_profiled': 1, 'filtered': ['information_schema.*', 'mysql.*', 'performance_schema.*', 'sys.*'], 'soft_deleted_stale_entities': [], 'query_combiner': {'total_queries': 22, 'uncombined_queries_issued': 22, 'combined_queries_issued': 0, 'queries_combined': 0, 'query_exceptions': 0}} Sink (datahub-kafka) report: {'records_written': 2, 'warnings': [], 'failures': [], 'downstream_start_time': None, 'downstream_end_time': None, 'downstream_total_latency_in_seconds': None} Pipeline finished successfully fabiocastro@pop-os:~$
d
That one is a known bug (we don’t calculate min/max if the distinct count is less than 60) and we are going to fix it.
👍 1
g
ok, very good and the others is very strange too
@gray-shoe-75895 , looks like that's works but the numbers min, max, median are NULL unknow . May you try to help me again, please? any idea
@faint-wolf-61232 hello, may you please try to help me in this case with profiling test where numbers (min, max, median) showing "unknow" ?
profiling_v2.txt
d
@gentle-florist-49869 I think for your issue we just merged a fix -> https://github.com/linkedin/datahub/commit/646dd6d5f0d248a5e6e743dd438c951a8a4b4c58 You can expect to have this when the next python package release happens
g
@dazzling-judge-80093 thank you and I will follow this point and normally when happen this new release, please? just to set a time here the advance this profiling test
d
I think this will happen in the following days. I will ping you here when it is available.
g
ok .. It will be great , tks a lot to help me
Open in Slack
PreviousNext
Powered by