Hi Elementary community. I've got an issue, we are...
# community-support
c
Hi Elementary community. I've got an issue, we are currently hosting our elementary report with AppEngine in GCP. However, 2 days ago the static file generated by the report grew to large which made AppEngine fail in hosting it. Output: Step #4 - "deploy-appengine-elementary": "description": "File https://storage.googleapis.com/staging/blabla" is too big (35055554 bytes). Max file size is 33554432 bytes.", So we look for other options with AppEngine or at all for serving our report. One way could be to use GCS but then we will need a load balancer for backend and ssl cert to have it private and only accessable by us, which is a mess. Another possible option could be to set "edr report --days-back 5" to lower it from the standard of 7 days, will that shrink the static html file generated? And will that be a sufficient solution long term? Have you guys any plan on what I should do. All help is appriciated!
šŸ‘€ 1
o
HI @Christopher Ekfeldt! As a quick workaround - yes reducing the days back could help as you will have less test results and samples of the failed tests. You can maybe reduce it further by leveraging the following flags - •
--disable-passed-test-metrics
- this disables showing the details on anomaly detection tests that ran successfully. •
--executions-limit
- you can set the number of invocations of tests showing under the
Test Runs
screen (current default is up to 720 invocations). •
--disable-samples
- disables the failed tests sampling Obviously you don't have to use all of them but you can play with these based on your needs as a start including the days-back as you mentioned. A proper solution could be to leverage compression but this depends on how AppEngine supports serving compressed files. Do you use
edr send-report
or
edr report
and then upload it with custom code? Also, shameless plug here - for really big envs sometimes static reports are less optimal and this is something that our cloud solution solves really well . If you'd be interested in this direction as well, the migration itself is seamless as it's built on top of the OSS.
c
Hi @Or Avidov thanks for the quick and very descriptive response! The --disable-passed-test-metrics, can you specify more details here? Is the graph fully removed and you just see "PASS" here or? On --executions-limit, what do you mean with invocations here? --disable-samples, is this the result list shown in the report for rows that failed? 32MB is crazy large even for something holding 7 days of data. I hope lowering the amount of days and disable some of these may fix the issue. But unsure if this will pop up soon as an issue again then when the project grows.. Im also looking into using other tools for hosting it.. We will see šŸ™‚
This is my updated command now: args: [ 'monitor', 'report', '--profiles-dir', '.dbt-profiles', '--profile-target', '${var.dbt_config.environment}', '--file-path', 'appengine/elementary/static/report.html', '--env', '${var.elementary_env[var.dbt_config.environment]}', '--days-back', '5', '--disable-passed-test-metrics', 'true', '--executions-limit', '360' ] The size of our production report is now 29.9 MB. So still very close to reaching 32MB..