Looking for a little guidance on upgrading to 0.11.0 using my Helm Kubernetes deployment. I have made the needed changes to the values.yaml file, and run the helm upgrade as I have in the past (in QA). I see the datahub-datahub-system-update-job come up and the logs looked like they were re-indexing. It took around 10 hours. This morning the job had succeeded but I still see version 0.10.5 in the UI. I did notice the Helm command had timed out because the update job took so long. Wondering if someone could explain what jobs need to run as a part of this upgrade. Also I didnt notice any actual downtime in the UI? **In QA I ran the HELM upgrade again and the job is re-indexing again... back at 1% looking like itll take another 10 hours. Not sure how Im going to make this work in PROD, or if ill be able to upgrade to the newest version.

Hello Drew, how many entities do you have in your catalog? Do you count the rows in the database or entities/size of indexes in ElasticSearch?

not sure about the actual best way to pull that number but referencing the Analytics tab we have...

@brainy-tent-14503 - could you look into this?

When the restore-indices job starts it says there are 19,597,094 rows it needs to send MAE's for. Which seems like way too many. Im thinking we may have alot of DataPrcocessInstances coming from the airlfow package that are really clogging things up.

btw - you can set

capture_executions

to false if you want to stop getting dataProcessInstances in the future

Nice, I didn’t know about that setting. A lot of instances accumulate these entities which because of their ids are probably better modeled as timeseries which allows for bypassing sql storage later and easier truncation in ES.

the delete process is very slow, I am running a daily job to clean them up as we do actually want a days worth of runs in Datahub for users to check

To be clear the retention policy is for retaining historical versions of entities. Only the latest version is indexed in ES and thus the restore indices is only operating on version 0 or the latest version. The retention applies to the historical versions which are only stored in SQL. Setting a retention policy doesn’t change the number of entities, merely the number of previous versions of them. Does that make sense?

@brainy-tent-14503 Yes that does make sense, thank you. We just pulled the MYSQL query to break down our aspect count and see this at the top. The Restore-indices job says there are 17million aspects in the table that it needs to re-hydrate from, which makes sense looking at this. Based on your point about the versions only being stored in SQL, are these 17million pure (version 0) aspects? **edit: we confirmed this is filtered to version = 0*** Or are many of them old versions that wont be rehydrated. The root issue is the restore-indices and re-index jobs are taking far too long.

+----------------------------------+--------------+
| aspect                           | aspect_count |
+----------------------------------+--------------+
| dataProcessInstanceProperties    |      3534964 |
| dataProcessInstanceRelationships |      3526886 |
| dataProcessInstanceKey           |      3526866 |
| dataProcessInstanceOutput        |      3244680 |
| dataProcessInstanceInput         |      2780390 |
| datasetProperties                |        96470 |
| dataJobInfo                      |        56675 |
| dataJobInputOutput               |        54784 |
| browsePaths                      |        37825 |
| browsePathsV2                    |        36718 |
| dataPlatformInstance             |        36430 |
| status                           |        35665 |
| upstreamLineage                  |        29418 |
| schemaMetadata                   |        28851 |
| datasetKey                       |        24744 |
| subTypes                         |        21116 |
| domains                          |        16942 |
| container                        |        16941 |
| viewProperties                   |        12675 |
| ownership                        |        12511 |
| globalTags                       |        11960 |
| dataJobKey                       |         8158 |
| dataHubExecutionRequestInput     |         4332 |
| dataHubExecutionRequestResult    |         4293 |
| dataHubExecutionRequestKey       |         4000 |
| siblings                         |         2222 |
| dataHubStepStateProperties       |          187 |
| dataHubStepStateKey              |          177 |
| dataFlowInfo                     |          148 |
| dataFlowKey                      |          102 |
| containerProperties              |           66 |

I have taken a crack at a few different threaded python scripts to delete dataProcessInstances (hard) efficiently, but considering that the

graph.get_urns_by_filter

returns a scrollAcrossEntities error, as well as it takes hours and hours to cleanup this hasnt been super effective.

Check to see if you’re executing a

soft

delete vs

hard

delete. A soft delete will set a

status

aspect instead of actually removing them. In that case however I’d expect to see a larger number of

status

aspects though, but it depends on how many possible

soft

deletes were executed so far.

They are all hard deletes...

graph.delete_entity(data_process_instance_urn, hard=True)

@numerous-address-22061 @brainy-tent-14503 how did you get around the

"Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [internal:cluster/shard/started] would be [514275828/490.4mb], which is larger than the limit of [510027366/486.3mb], real usage: [514275536/490.4mb], new bytes reserved: [292/292b], usages [request=0/0b, fielddata=2570/2.5kb, in_flight_requests=292/292b, model_inference=0/0b, eql_sequence=0/0b, accounting=4408152/4.2mb]"

error you were seeing a few months back? This is currently effecting our production cluster and have increased memory limits and still has not helped

Increase memory to your elasticsearch instance, looks like it only has maybe 512mb

I think 512 was the jvm heap limit