Hi all, i'm ingesting data from postgresql, the fi...
# troubleshoot
b
Hi all, i'm ingesting data from postgresql, the first time running sucessfull, but it turn out to fail since the second running with this can't map type error. Any clue on this? Thank you
Copy code
"'container-urn:li:container:f4f87664cea5ae66d80ae56c7893eef5-to-urn:li:dataset:(urn:li:dataPlatform:postgres,tech.tech_sch.ga2_metrics_hour,PROD)',\n"
           "               'tech.tech_sch.landers_category',\n"
           "               'tech.tech_sch.search_query_category_rnk1-subtypes',\n"
           "               '... sampled of 2271 total elements'],\n"
           ' \'warnings\': {\'tech.location_tree._locations_vn_geo\': ["unable to map type Geometry(from_text=\'ST_GeomFromEWKT\', '
           'name=\'geometry\') to metadata schema"],\n'
           '              \'tech.location_tree.locations_vn\': ["unable to map type Geometry(from_text=\'ST_GeomFromEWKT\', name=\'geometry\') to '
           'metadata schema"],\n'
           '              \'tech.location_tree.locations_ph\': ["unable to map type Geometry(from_text=\'ST_GeomFromEWKT\', name=\'geometry\') to '
           'metadata schema"]},\n'
           " 'failures': {'Stateful Ingestion': ['Fail safe mode triggered, entity difference percent:97.61904761904762 > "
           "fail_safe_threshold:{self.stateful_ingestion_config.fail_safe_threshold}']},\n"
           " 'soft_deleted_stale_entities': [],\n"
           " 'tables_scanned': '656',\n"
plus1 1
m
Hi, so that must be the Stateful ingestion. I get sometimes errors in ingestions when the stateful option is enabled, but they usually dissapear when I disable Stateful ingestion
h
Hey , you need not disable stateful ingestion. You can bypass this behavior for updating
stateful_ingestion.fail_safe_threshold
config for that single run. As mentioned in the doc the failure is due to a benign safety check that -> Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.