Hi all, we've been running DataHub for 14 days using docker quickstart, but today our DataHub front end home page started showing a big red error message:

Oops, an error occurred. This exception has been logged with id xxxxxxxx

(no login page shown, only the error message) Have restarted the quickstart container, have also rebooted the VM. Have followed the advice in https://datahubproject.io/docs/debugging/#how-can-i-confirm-if-all-docker-containers-are-running-as-expected-after-a-quickstart •

datahub docker check

returned everything was OK •

docker logs datahub-frontend-react

returned the following errors:

play.api.UnexpectedException: Unexpected exception[ServerResultException: HTTP 1.0 client does not support chunked response]
        at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:358)
        at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:264)
        at play.core.server.common.ServerResultUtils.validateResult(ServerResultUtils.scala:69)
        at play.core.server.akkahttp.AkkaModelConversion.$anonfun$convertResult$1(AkkaModelConversion.scala:193)
        at play.core.server.common.ServerResultUtils.resultConversionWithErrorHandling(ServerResultUtils.scala:195)
        at play.core.server.akkahttp.AkkaModelConversion.convertResult(AkkaModelConversion.scala:215)
        at play.core.server.AkkaHttpServer.$anonfun$runAction$5(AkkaHttpServer.scala:440)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: play.core.server.common.ServerResultException: HTTP 1.0 client does not support chunked response
        at play.core.server.common.ServerResultUtils.validateResult(ServerResultUtils.scala:68)
        ... 19 common frames omitted
2023-01-26 13:44:29,799 [application-akka.actor.default-dispatcher-19] ERROR p.api.http.DefaultHttpErrorHandler -
! @80d92mm8g - Internal server error, for (GET) [/] ->

play.api.UnexpectedException: Unexpected exception[ServerResultException: HTTP 1.0 client does not support chunked response]
        at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:358)
        at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:264)
        at play.core.server.common.ServerResultUtils.validateResult(ServerResultUtils.scala:69)
        at play.core.server.akkahttp.AkkaModelConversion.$anonfun$convertResult$1(AkkaModelConversion.scala:193)
        at play.core.server.common.ServerResultUtils.resultConversionWithErrorHandling(ServerResultUtils.scala:195)
        at play.core.server.akkahttp.AkkaModelConversion.convertResult(AkkaModelConversion.scala:215)
        at play.core.server.AkkaHttpServer.$anonfun$runAction$5(AkkaHttpServer.scala:440)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: play.core.server.common.ServerResultException: HTTP 1.0 client does not support chunked response
        at play.core.server.common.ServerResultUtils.validateResult(ServerResultUtils.scala:68)
        ... 19 common frames omitted
2023-01-26 13:44:29,863 [application-akka.actor.default-dispatcher-19] ERROR p.api.http.DefaultHttpErrorHandler -
! @80d92mmb1 - Internal server error, for (GET) [/favicon.ico] ->

Would greatly appreciate any suggestions. Thanks!

If you’re having trouble with quickstart, please make sure you’re using the most up-to-date version of DataHub by following the steps in the quickstart deployment guide: https://datahubproject.io/docs/quickstart/#deploying-datahub. Specifically, ensure you’re up to date with the DataHub CLI:

python3 -m pip install --upgrade pip wheel setuptools
python3 -m pip install --upgrade acryl-datahub
datahub version

Hi! Are you running a pure quickstart deployment? (i.e. everything is running locally on a single machine) The chunked response error seems like it can be related to proxies so wondering if you're routing through an LB/proxy/ingress.

If you’re having trouble with quickstart, please make sure you’re using the most up-to-date version of DataHub by following the steps in the quickstart deployment guide: https://datahubproject.io/docs/quickstart/#deploying-datahub. Specifically, ensure you’re up to date with the DataHub CLI:

python3 -m pip install --upgrade pip wheel setuptools
python3 -m pip install --upgrade acryl-datahub
datahub version

bad bot 😛

Hi @orange-night-91387! Yes we're running quickstart on a single machine. The strange thing is that everything was running well for 14 days, we haven't made any changes and suddenly yesterday it started crashing. Now I checked again - and everything works fine! 😂 I haven't touched anything, yet it somehow self-recovered. Bit puzzling... That makes me wonder: is the quickstart container somehow auto-reloading by itself on regular basis? Is there any way to control this? I love the idea of auto-updating to the latest versions of everything, but also wonder if there's a way to "roll back" to the last working version (apart from just rolling back the whole server from backups)?

If you’re having trouble with quickstart, please make sure you’re using the most up-to-date version of DataHub by following the steps in the quickstart deployment guide: https://datahubproject.io/docs/quickstart/#deploying-datahub. Specifically, ensure you’re up to date with the DataHub CLI:

python3 -m pip install --upgrade pip wheel setuptools
python3 -m pip install --upgrade acryl-datahub
datahub version

It shouldn't be auto-reloading, or anything fancy like that. If you restarted the containers it's possible that a component got in a bad state on the machine after running continuously for two weeks in Docker Desktop and restarting fixed it. I know at least for me Docker Desktop makes my machine sad if I leave it running for long periods of time

I see. We're running it on a cloud VM, it's meant to be for the whole company's use. That should be normally always running. Is there anything special that we should set up to make it stable for "production" use?

Deploying all components on a single VM is not really the recommended approach for a production instance. We generally recommend using a Kubernetes cluster with managed backend services. Local quickstart deploys are mainly for POCs where reliability is not a concern

If you’re having trouble with quickstart, please make sure you’re using the most up-to-date version of DataHub by following the steps in the quickstart deployment guide: https://datahubproject.io/docs/quickstart/#deploying-datahub. Specifically, ensure you’re up to date with the DataHub CLI:

python3 -m pip install --upgrade pip wheel setuptools
python3 -m pip install --upgrade acryl-datahub
datahub version

Thanks for the suggestion Ryan!