Title
b

boennemann

03/06/2019, 1:59 PM
Is there anyone else experiencing problems with running prisma-server recently? We've done extremely well running it on zeit.co for quite some time, until it started constantly throwing OutOfMemory errors across many versions. We tried switching to heroku and adding a ton more memory, but that didn't seem to fix anything, because it's probably just leaking the memory anway.
03/06 02:44 PM (13m)
Uncaught error from thread [single-server-akka.actor.default-dispatcher-2]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[single-server]
03/06 02:44 PM (13m)
java.lang.OutOfMemoryError: Java heap space
Initially the server would go down once a week like this, but we've now arrived at pretty much every single request.
It's super frustrating because it's in a production environment and prisma-server is just a black box for us, that is very hard to debug in any way.
Any pointers would be highly appreciated.
Running 1.28.0, but happens for the past couple of versions.
f

Fran Dios

03/06/2019, 2:02 PM
hey
I think this is the same issue I had last week
it was related to a wrong deploy
Connect directly to your DB and check Prisma’s management table and see if all the rows there are success
b

boennemann

03/06/2019, 2:10 PM
@Fran Dios Hey 👋 Thanks for the answer! That would management/Migrations?
All successful
I even tried running
prisma export
, killing the whole thing, deploying everything fresh, creating the service again, and then running
prisma import
. Wanted to ensure there is no migration weirdness or anything going on, but it doesn't help 😕
f

Fran Dios

03/06/2019, 2:14 PM
Seems really similar to my issue
ok, next thing
make a backup copy of management table
especially the last row
when you have it copied, delete the last row, and before testing your app or anything, try deploying a fresh new service
delete only the last row in the management table, nothing else
b

boennemann

03/06/2019, 2:15 PM
f

Fran Dios

03/06/2019, 2:16 PM
in the migrations
b

boennemann

03/06/2019, 2:16 PM
I don't have table called management per se, only this group.
K alright
f

Fran Dios

03/06/2019, 2:17 PM
it should have something like projectId, revision, schema, functions, status, etc
make a backup and delete the last row
b

boennemann

03/06/2019, 2:17 PM
There are entries for 3 different projects in there? That last one, overall, or the last one for prod?
f

Fran Dios

03/06/2019, 2:18 PM
are all of them using the same Prisma server?
b

boennemann

03/06/2019, 2:18 PM
yep
f

Fran Dios

03/06/2019, 2:18 PM
and none of the apps work, right?
b

boennemann

03/06/2019, 2:18 PM
correct
f

Fran Dios

03/06/2019, 2:19 PM
then remove the very last one, I think
I’m not an expert, I was told by a Prisma team member to do this few days ago and it worked for me
b

boennemann

03/06/2019, 2:19 PM
I mean maybe that's the issue right there. Maybe I should separate my environments completely, and just deploy it entirely separately.
f

Fran Dios

03/06/2019, 2:20 PM
nah, I only had prod apps and failed in the same way
3 apps, though. But it’s kind of pointless having only 1 app for 1 DB…
Right now I have 1 DB for prod and 1 for staging in GCP. All my apps are in Zeit Now as well
I also tried deploying to Heroku when I saw this issue but still got the same error
b

boennemann

03/06/2019, 2:21 PM
Okay, cool, so exactly the same situation.
By deploying a completely fresh service you mean, deploying a separate project altogether
f

Fran Dios

03/06/2019, 2:21 PM
was about to deploy to 2GB instance in DO when I was told to check the management.migrations table
b

boennemann

03/06/2019, 2:21 PM
or just prod again.
f

Fran Dios

03/06/2019, 2:22 PM
you can remove the broken Prisma server from Now and deploy it again
I had 1 instance aliased, so I just removed that one and deployed
but make sure it deploys a new one. Some times in Now v1, if you didn’t change the Dockerfile, it will reuse an existing instance
b

boennemann

03/06/2019, 2:23 PM
Deploy running 🙂
f

Fran Dios

03/06/2019, 2:24 PM
it’s going to work! 🤞
after that you can test your app or try connecting to the API directly with the playground
b

boennemann

03/06/2019, 2:26 PM
03/06 03:25 PM (40s) Uncaught error from thread [single-server-akka.actor.default-dispatcher-5]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[single-server] 03/06 03:25 PM (40s) java.lang.OutOfMemoryError: Java heap space
f

Fran Dios

03/06/2019, 2:26 PM
🤦
same thing?
b

boennemann

03/06/2019, 2:27 PM
yea 😕
f

Fran Dios

03/06/2019, 2:28 PM
perhaps there are more wrong rows, not sure
I guess at this point we should ping @Harshit
h

Harshit

03/06/2019, 2:30 PM
Hi @boennemann This doesn't look like a migration issue. You will require to pass some Java options
f

Fran Dios

03/06/2019, 2:31 PM
woops, my error was also “akka-xxx java heap out of memory”, I thought it would be similar 🤔
h

Harshit

03/06/2019, 2:31 PM
can you please share your DockerFile
@Fran Dios You error was this
com.prisma.deploy.migration.migrator.DeploymentProtocol
read deploy protocol error
his error is clearly related to the memory heap
b

boennemann

03/06/2019, 2:32 PM
FROM prismagraphql/prisma:1.28.0

ARG PRISMA_MANAGEMENT_API_SECRET
ARG DATABASE_HOST
ARG DATABASE_PORT
ARG DATABASE_NAME
ARG DATABASE_USER
ARG DATABASE_PASSWORD
ARG PRISMA_CONFIG_PATH

ENV PRISMA_CONFIG_PATH prisma.yml

COPY config.yml prisma.yml

RUN sed -i "s/PRISMA_MANAGEMENT_API_SECRET/$PRISMA_MANAGEMENT_API_SECRET/g" prisma.yml
RUN sed -i "s/DATABASE_HOST/$DATABASE_HOST/g" prisma.yml
RUN sed -i "s/DATABASE_PORT/$DATABASE_PORT/g" prisma.yml
RUN sed -i "s/DATABASE_NAME/$DATABASE_NAME/g" prisma.yml
RUN sed -i "s/DATABASE_USER/$DATABASE_USER/g" prisma.yml
RUN sed -i "s/DATABASE_PASSWORD/$DATABASE_PASSWORD/g" prisma.yml

EXPOSE 4466
f

Fran Dios

03/06/2019, 2:32 PM
I actually had more than 1 error, I’m pretty sure I saw the Java heap as well 😅
anyway, I’ll let you guys talk about it
b

boennemann

03/06/2019, 2:33 PM
Thanks @Fran Dios 🙏
h

Harshit

03/06/2019, 2:33 PM
FROM prismagraphql/prisma:1.28.0

ARG PRISMA_MANAGEMENT_API_SECRET
ARG DATABASE_HOST
ARG DATABASE_PORT
ARG DATABASE_NAME
ARG DATABASE_USER
ARG DATABASE_PASSWORD
ARG PRISMA_CONFIG_PATH

ENV PRISMA_CONFIG_PATH prisma.yml
ENV JAVA_OPTS -Xmx358m

COPY config.yml prisma.yml

RUN sed -i "s/PRISMA_MANAGEMENT_API_SECRET/$PRISMA_MANAGEMENT_API_SECRET/g" prisma.yml
RUN sed -i "s/DATABASE_HOST/$DATABASE_HOST/g" prisma.yml
RUN sed -i "s/DATABASE_PORT/$DATABASE_PORT/g" prisma.yml
RUN sed -i "s/DATABASE_NAME/$DATABASE_NAME/g" prisma.yml
RUN sed -i "s/DATABASE_USER/$DATABASE_USER/g" prisma.yml
RUN sed -i "s/DATABASE_PASSWORD/$DATABASE_PASSWORD/g" prisma.yml

EXPOSE 4466
can you please try this docker file?
Also can you please tell me whether this happened after an update to 1.28 otherwise we might have caught a bug and it will require more investigation
b

boennemann

03/06/2019, 2:34 PM
It's happened before 1.28, too.
h

Harshit

03/06/2019, 2:35 PM
ok thanks 🙏
b

boennemann

03/06/2019, 2:35 PM
Just upgraded today because I hoped that would fix it.
h

Harshit

03/06/2019, 2:35 PM
oh sorry I forgot the arg option there
paste this
b

boennemann

03/06/2019, 2:35 PM
I saw the same JAVA_OPTS variable passed in the heroku default config, right?
Sure
h

Harshit

03/06/2019, 2:36 PM
FROM prismagraphql/prisma:1.28.0

ARG PRISMA_MANAGEMENT_API_SECRET
ARG DATABASE_HOST
ARG DATABASE_PORT
ARG DATABASE_NAME
ARG DATABASE_USER
ARG DATABASE_PASSWORD
ARG PRISMA_CONFIG_PATH
ARG JAVA_OPTS

ENV PRISMA_CONFIG_PATH prisma.yml
ENV JAVA_OPTS -Xmx358m

COPY config.yml prisma.yml

RUN sed -i "s/PRISMA_MANAGEMENT_API_SECRET/$PRISMA_MANAGEMENT_API_SECRET/g" prisma.yml
RUN sed -i "s/DATABASE_HOST/$DATABASE_HOST/g" prisma.yml
RUN sed -i "s/DATABASE_PORT/$DATABASE_PORT/g" prisma.yml
RUN sed -i "s/DATABASE_NAME/$DATABASE_NAME/g" prisma.yml
RUN sed -i "s/DATABASE_USER/$DATABASE_USER/g" prisma.yml
RUN sed -i "s/DATABASE_PASSWORD/$DATABASE_PASSWORD/g" prisma.yml

EXPOSE 4466
I am off to dinner now, let me know if this worked. If not we can debug this in a DM
b

boennemann

03/06/2019, 2:36 PM
Sure, thank you 🙏 Enjoy your dinner!
h

Harshit

03/06/2019, 3:19 PM
This has been resolved in a DM. Solution was to increase the memory heap size even more in the JVM options
🙌 2