Title
b
boennemann
03/06/2019, 1:59 PMIs there anyone else experiencing problems with running prisma-server recently? We've done extremely well running it on zeit.co for quite some time, until it started constantly throwing OutOfMemory errors across many versions. We tried switching to heroku and adding a ton more memory, but that didn't seem to fix anything, because it's probably just leaking the memory anway.
03/06 02:44 PM (13m)
Uncaught error from thread [single-server-akka.actor.default-dispatcher-2]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[single-server]
03/06 02:44 PM (13m)
java.lang.OutOfMemoryError: Java heap spaceInitially the server would go down once a week like this, but we've now arrived at pretty much every single request.
It's super frustrating because it's in a production environment and prisma-server is just a black box for us, that is very hard to debug in any way.
Any pointers would be highly appreciated.
Running 1.28.0, but happens for the past couple of versions.
f
Fran Dios
03/06/2019, 2:02 PMhey
I think this is the same issue I had last week
it was related to a wrong deploy
Connect directly to your DB and check Prisma’s management table and see if all the rows there are success
b
boennemann
03/06/2019, 2:10 PM@Fran Dios Hey 👋 Thanks for the answer! That would management/Migrations?
All successful
I even tried running
prisma exportprisma importf
Fran Dios
03/06/2019, 2:14 PMSeems really similar to my issue
ok, next thing
make a backup copy of management table
especially the last row
when you have it copied, delete the last row, and before testing your app or anything, try deploying a fresh new service
delete only the last row in the management table, nothing else
b
boennemann
03/06/2019, 2:15 PMf
Fran Dios
03/06/2019, 2:16 PMin the migrations
b
boennemann
03/06/2019, 2:16 PMI don't have table called management per se, only this group.
K alright
f
Fran Dios
03/06/2019, 2:17 PMit should have something like projectId, revision, schema, functions, status, etc
make a backup and delete the last row
b
boennemann
03/06/2019, 2:17 PMThere are entries for 3 different projects in there? That last one, overall, or the last one for prod?
f
Fran Dios
03/06/2019, 2:18 PMare all of them using the same Prisma server?
b
boennemann
03/06/2019, 2:18 PMyep
f
Fran Dios
03/06/2019, 2:18 PMand none of the apps work, right?
b
boennemann
03/06/2019, 2:18 PMcorrect
f
Fran Dios
03/06/2019, 2:19 PMthen remove the very last one, I think
I’m not an expert, I was told by a Prisma team member to do this few days ago and it worked for me
b
boennemann
03/06/2019, 2:19 PMI mean maybe that's the issue right there. Maybe I should separate my environments completely, and just deploy it entirely separately.
f
Fran Dios
03/06/2019, 2:20 PMnah, I only had prod apps and failed in the same way
3 apps, though. But it’s kind of pointless having only 1 app for 1 DB…
Right now I have 1 DB for prod and 1 for staging in GCP. All my apps are in Zeit Now as well
I also tried deploying to Heroku when I saw this issue but still got the same error
b
boennemann
03/06/2019, 2:21 PMOkay, cool, so exactly the same situation.
By deploying a completely fresh service you mean, deploying a separate project altogether
f
Fran Dios
03/06/2019, 2:21 PMwas about to deploy to 2GB instance in DO when I was told to check the management.migrations table
b
boennemann
03/06/2019, 2:21 PMor just prod again.
f
Fran Dios
03/06/2019, 2:22 PMyou can remove the broken Prisma server from Now and deploy it again
I had 1 instance aliased, so I just removed that one and deployed
but make sure it deploys a new one. Some times in Now v1, if you didn’t change the Dockerfile, it will reuse an existing instance
b
boennemann
03/06/2019, 2:23 PMDeploy running 🙂
f
Fran Dios
03/06/2019, 2:24 PMit’s going to work! 🤞
after that you can test your app or try connecting to the API directly with the playground
b
boennemann
03/06/2019, 2:26 PM03/06 03:25 PM (40s)
Uncaught error from thread [single-server-akka.actor.default-dispatcher-5]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[single-server]
03/06 03:25 PM (40s)
java.lang.OutOfMemoryError: Java heap space
f
Fran Dios
03/06/2019, 2:26 PM🤦
same thing?
b
boennemann
03/06/2019, 2:27 PMyea 😕
f
Fran Dios
03/06/2019, 2:28 PMperhaps there are more wrong rows, not sure
I guess at this point we should ping @Harshit
h
Harshit
03/06/2019, 2:30 PMHi @boennemann
This doesn't look like a migration issue.
You will require to pass some Java options
f
Fran Dios
03/06/2019, 2:31 PMwoops, my error was also “akka-xxx java heap out of memory”, I thought it would be similar 🤔
h
Harshit
03/06/2019, 2:31 PMcan you please share your DockerFile
@Fran Dios
You error was this
com.prisma.deploy.migration.migrator.DeploymentProtocolhis error is clearly related to the memory heap
b
boennemann
03/06/2019, 2:32 PMFROM prismagraphql/prisma:1.28.0
ARG PRISMA_MANAGEMENT_API_SECRET
ARG DATABASE_HOST
ARG DATABASE_PORT
ARG DATABASE_NAME
ARG DATABASE_USER
ARG DATABASE_PASSWORD
ARG PRISMA_CONFIG_PATH
ENV PRISMA_CONFIG_PATH prisma.yml
COPY config.yml prisma.yml
RUN sed -i "s/PRISMA_MANAGEMENT_API_SECRET/$PRISMA_MANAGEMENT_API_SECRET/g" prisma.yml
RUN sed -i "s/DATABASE_HOST/$DATABASE_HOST/g" prisma.yml
RUN sed -i "s/DATABASE_PORT/$DATABASE_PORT/g" prisma.yml
RUN sed -i "s/DATABASE_NAME/$DATABASE_NAME/g" prisma.yml
RUN sed -i "s/DATABASE_USER/$DATABASE_USER/g" prisma.yml
RUN sed -i "s/DATABASE_PASSWORD/$DATABASE_PASSWORD/g" prisma.yml
EXPOSE 4466f
Fran Dios
03/06/2019, 2:32 PMI actually had more than 1 error, I’m pretty sure I saw the Java heap as well 😅
anyway, I’ll let you guys talk about it
b
boennemann
03/06/2019, 2:33 PMThanks @Fran Dios 🙏
h
Harshit
03/06/2019, 2:33 PMFROM prismagraphql/prisma:1.28.0
ARG PRISMA_MANAGEMENT_API_SECRET
ARG DATABASE_HOST
ARG DATABASE_PORT
ARG DATABASE_NAME
ARG DATABASE_USER
ARG DATABASE_PASSWORD
ARG PRISMA_CONFIG_PATH
ENV PRISMA_CONFIG_PATH prisma.yml
ENV JAVA_OPTS -Xmx358m
COPY config.yml prisma.yml
RUN sed -i "s/PRISMA_MANAGEMENT_API_SECRET/$PRISMA_MANAGEMENT_API_SECRET/g" prisma.yml
RUN sed -i "s/DATABASE_HOST/$DATABASE_HOST/g" prisma.yml
RUN sed -i "s/DATABASE_PORT/$DATABASE_PORT/g" prisma.yml
RUN sed -i "s/DATABASE_NAME/$DATABASE_NAME/g" prisma.yml
RUN sed -i "s/DATABASE_USER/$DATABASE_USER/g" prisma.yml
RUN sed -i "s/DATABASE_PASSWORD/$DATABASE_PASSWORD/g" prisma.yml
EXPOSE 4466can you please try this docker file?
Also can you please tell me whether this happened after an update to 1.28 otherwise we might have caught a bug and it will require more investigation
b
boennemann
03/06/2019, 2:34 PMIt's happened before 1.28, too.
h
Harshit
03/06/2019, 2:35 PMok thanks 🙏
b
boennemann
03/06/2019, 2:35 PMJust upgraded today because I hoped that would fix it.
h
Harshit
03/06/2019, 2:35 PMoh sorry I forgot the arg option there
paste this
b
boennemann
03/06/2019, 2:35 PMI saw the same JAVA_OPTS variable passed in the heroku default config, right?
Sure
h
Harshit
03/06/2019, 2:36 PMFROM prismagraphql/prisma:1.28.0
ARG PRISMA_MANAGEMENT_API_SECRET
ARG DATABASE_HOST
ARG DATABASE_PORT
ARG DATABASE_NAME
ARG DATABASE_USER
ARG DATABASE_PASSWORD
ARG PRISMA_CONFIG_PATH
ARG JAVA_OPTS
ENV PRISMA_CONFIG_PATH prisma.yml
ENV JAVA_OPTS -Xmx358m
COPY config.yml prisma.yml
RUN sed -i "s/PRISMA_MANAGEMENT_API_SECRET/$PRISMA_MANAGEMENT_API_SECRET/g" prisma.yml
RUN sed -i "s/DATABASE_HOST/$DATABASE_HOST/g" prisma.yml
RUN sed -i "s/DATABASE_PORT/$DATABASE_PORT/g" prisma.yml
RUN sed -i "s/DATABASE_NAME/$DATABASE_NAME/g" prisma.yml
RUN sed -i "s/DATABASE_USER/$DATABASE_USER/g" prisma.yml
RUN sed -i "s/DATABASE_PASSWORD/$DATABASE_PASSWORD/g" prisma.yml
EXPOSE 4466I am off to dinner now, let me know if this worked. If not we can debug this in a DM
b
boennemann
03/06/2019, 2:36 PMSure, thank you 🙏 Enjoy your dinner!
h
Harshit
03/06/2019, 3:19 PMThis has been resolved in a DM. Solution was to increase the memory heap size even more in the JVM options
🙌 2