Hi! I'm trying to profile a MSSQL database at inge...
# ingestions
stale-jewelry-2440
09/03/2021, 9:38 AMHi! I'm trying to profile a MSSQL database at ingestion. The ingestion starts good, then many warning are raised like:
[2021-09-03 093406,970] WARNINGÂ {great_expectations.dataset.sqlalchemy_dataset:2023} - Regex is not supported for dialect <sqlalchemy_pytds.dialect.MSDialect_pytds object at 0x7f5b33225df0>
And at the end I just see a bare 'Killed' word, without seeing any recap about the ingestion.
Maybe there are too many warnings, and that makes the procedure explode?
And how to solve that warnings?
Thank you 🙂
c
careful-insurance-60247
09/03/2021, 12:59 PMCan you share your recipe?
s
stale-jewelry-2440
09/03/2021, 3:05 PM Copy code
source:
type: mssql
config:
username: ---
password: ---
host_port: ---
database: ---
include_views: True
include_tables: True
table_pattern:
deny:
- "^.*\\.sys_.*" # deny all tables that start with sys_
allow:
- "AAAA.*"
profiling:
enabled: true
sink:
type: "datahub-rest"
config:
server: '<http://datahub-gms:8080>'l
loud-island-88694
09/03/2021, 3:40 PM
@helpful-optician-78938 ^ can you please take a look?
h
helpful-optician-78938
09/03/2021, 5:25 PMI'll take a look.
s
stale-jewelry-2440
11/10/2021, 9:53 AMbump: this issue is still present!
stale-jewelry-2440
11/10/2021, 12:38 PMUpdate: even solving the warnings, sudden kill happens
stale-jewelry-2440
11/10/2021, 1:09 PMit seems it’s the OS that kills the program : https://unix.stackexchange.com/questions/614950/python-programs-suddenly-get-killed
I soled by reducing the memory cost of profiling:
Copy code
include_field_mean_value: false
include_field_median_value: false
include_field_stddev_value: false
include_field_quantiles: false
include_field_distinct_value_frequencies: false
include_field_histogram: false2 Views