Is there anyone else experiencing problems with ru...
# orm-helpb
boennemann
03/06/2019, 1:59 PMIs there anyone else experiencing problems with running prisma-server recently? We've done extremely well running it on zeit.co for quite some time, until it started constantly throwing OutOfMemory errors across many versions. We tried switching to heroku and adding a ton more memory, but that didn't seem to fix anything, because it's probably just leaking the memory anway.
Copy code
03/06 02:44 PM (13m)
Uncaught error from thread [single-server-akka.actor.default-dispatcher-2]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[single-server]
03/06 02:44 PM (13m)
java.lang.OutOfMemoryError: Java heap spaceboennemann
03/06/2019, 2:00 PMInitially the server would go down once a week like this, but we've now arrived at pretty much every single request.
boennemann
03/06/2019, 2:01 PMIt's super frustrating because it's in a production environment and prisma-server is just a black box for us, that is very hard to debug in any way.
boennemann
03/06/2019, 2:01 PMAny pointers would be highly appreciated.
boennemann
03/06/2019, 2:02 PMRunning 1.28.0, but happens for the past couple of versions.
f
Fran Dios
03/06/2019, 2:02 PMhey
Fran Dios
03/06/2019, 2:03 PMI think this is the same issue I had last week
Fran Dios
03/06/2019, 2:03 PMit was related to a wrong deploy
Fran Dios
03/06/2019, 2:04 PMConnect directly to your DB and check Prisma’s management table and see if all the rows there are success
b
boennemann
03/06/2019, 2:10 PM@Fran Dios Hey 👋 Thanks for the answer! That would management/Migrations?
boennemann
03/06/2019, 2:10 PMAll successful
boennemann
03/06/2019, 2:13 PMI even tried running
prisma exportprisma importf
Fran Dios
03/06/2019, 2:14 PMSeems really similar to my issue
Fran Dios
03/06/2019, 2:14 PMok, next thing
Fran Dios
03/06/2019, 2:14 PMmake a backup copy of management table
Fran Dios
03/06/2019, 2:14 PMespecially the last row
Fran Dios
03/06/2019, 2:14 PMwhen you have it copied, delete the last row, and before testing your app or anything, try deploying a fresh new service
Fran Dios
03/06/2019, 2:15 PMdelete only the last row in the management table, nothing else
b
boennemann
03/06/2019, 2:15 PMf
Fran Dios
03/06/2019, 2:16 PMin the migrations
b
boennemann
03/06/2019, 2:16 PMI don't have table called management per se, only this group.
boennemann
03/06/2019, 2:16 PMK alright
f
Fran Dios
03/06/2019, 2:17 PMit should have something like projectId, revision, schema, functions, status, etc
Fran Dios
03/06/2019, 2:17 PMmake a backup and delete the last row
b
boennemann
03/06/2019, 2:17 PMThere are entries for 3 different projects in there? That last one, overall, or the last one for prod?
f
Fran Dios
03/06/2019, 2:18 PMare all of them using the same Prisma server?
b
boennemann
03/06/2019, 2:18 PMyep
f
Fran Dios
03/06/2019, 2:18 PMand none of the apps work, right?
b
boennemann
03/06/2019, 2:18 PMcorrect
f
Fran Dios
03/06/2019, 2:19 PMthen remove the very last one, I think
Fran Dios
03/06/2019, 2:19 PMI’m not an expert, I was told by a Prisma team member to do this few days ago and it worked for me
b
boennemann
03/06/2019, 2:19 PMI mean maybe that's the issue right there. Maybe I should separate my environments completely, and just deploy it entirely separately.
f
Fran Dios
03/06/2019, 2:20 PMnah, I only had prod apps and failed in the same way
Fran Dios
03/06/2019, 2:20 PM3 apps, though. But it’s kind of pointless having only 1 app for 1 DB…
Fran Dios
03/06/2019, 2:21 PMRight now I have 1 DB for prod and 1 for staging in GCP. All my apps are in Zeit Now as well
Fran Dios
03/06/2019, 2:21 PMI also tried deploying to Heroku when I saw this issue but still got the same error
b
boennemann
03/06/2019, 2:21 PMOkay, cool, so exactly the same situation.
boennemann
03/06/2019, 2:21 PMBy deploying a completely fresh service you mean, deploying a separate project altogether
f
Fran Dios
03/06/2019, 2:21 PMwas about to deploy to 2GB instance in DO when I was told to check the management.migrations table
b
boennemann
03/06/2019, 2:21 PMor just prod again.
f
Fran Dios
03/06/2019, 2:22 PMyou can remove the broken Prisma server from Now and deploy it again
Fran Dios
03/06/2019, 2:22 PMI had 1 instance aliased, so I just removed that one and deployed
Fran Dios
03/06/2019, 2:23 PMbut make sure it deploys a new one. Some times in Now v1, if you didn’t change the Dockerfile, it will reuse an existing instance
b
boennemann
03/06/2019, 2:23 PMDeploy running 🙂
f
Fran Dios
03/06/2019, 2:24 PMit’s going to work! 🤞
Fran Dios
03/06/2019, 2:25 PMafter that you can test your app or try connecting to the API directly with the playground
b
boennemann
03/06/2019, 2:26 PM03/06 03:25 PM (40s)
Uncaught error from thread [single-server-akka.actor.default-dispatcher-5]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[single-server]
03/06 03:25 PM (40s)
java.lang.OutOfMemoryError: Java heap space
f
Fran Dios
03/06/2019, 2:26 PM🤦
Fran Dios
03/06/2019, 2:26 PMsame thing?
b
boennemann
03/06/2019, 2:27 PMyea 😕
f
Fran Dios
03/06/2019, 2:28 PMperhaps there are more wrong rows, not sure
Fran Dios
03/06/2019, 2:28 PMI guess at this point we should ping @Harshit
h
Harshit
03/06/2019, 2:30 PMHi @boennemann
This doesn't look like a migration issue.
You will require to pass some Java options
f
Fran Dios
03/06/2019, 2:31 PMwoops, my error was also “akka-xxx java heap out of memory”, I thought it would be similar 🤔
h
Harshit
03/06/2019, 2:31 PMcan you please share your DockerFile
Harshit
03/06/2019, 2:31 PM@Fran Dios
You error was this
com.prisma.deploy.migration.migrator.DeploymentProtocolHarshit
03/06/2019, 2:32 PMhis error is clearly related to the memory heap
b
boennemann
03/06/2019, 2:32 PM Copy code
FROM prismagraphql/prisma:1.28.0
ARG PRISMA_MANAGEMENT_API_SECRET
ARG DATABASE_HOST
ARG DATABASE_PORT
ARG DATABASE_NAME
ARG DATABASE_USER
ARG DATABASE_PASSWORD
ARG PRISMA_CONFIG_PATH
ENV PRISMA_CONFIG_PATH prisma.yml
COPY config.yml prisma.yml
RUN sed -i "s/PRISMA_MANAGEMENT_API_SECRET/$PRISMA_MANAGEMENT_API_SECRET/g" prisma.yml
RUN sed -i "s/DATABASE_HOST/$DATABASE_HOST/g" prisma.yml
RUN sed -i "s/DATABASE_PORT/$DATABASE_PORT/g" prisma.yml
RUN sed -i "s/DATABASE_NAME/$DATABASE_NAME/g" prisma.yml
RUN sed -i "s/DATABASE_USER/$DATABASE_USER/g" prisma.yml
RUN sed -i "s/DATABASE_PASSWORD/$DATABASE_PASSWORD/g" prisma.yml
EXPOSE 4466f
Fran Dios
03/06/2019, 2:32 PMI actually had more than 1 error, I’m pretty sure I saw the Java heap as well 😅
Fran Dios
03/06/2019, 2:32 PManyway, I’ll let you guys talk about it
b
boennemann
03/06/2019, 2:33 PMThanks @Fran Dios 🙏
h
Harshit
03/06/2019, 2:33 PM Copy code
FROM prismagraphql/prisma:1.28.0
ARG PRISMA_MANAGEMENT_API_SECRET
ARG DATABASE_HOST
ARG DATABASE_PORT
ARG DATABASE_NAME
ARG DATABASE_USER
ARG DATABASE_PASSWORD
ARG PRISMA_CONFIG_PATH
ENV PRISMA_CONFIG_PATH prisma.yml
ENV JAVA_OPTS -Xmx358m
COPY config.yml prisma.yml
RUN sed -i "s/PRISMA_MANAGEMENT_API_SECRET/$PRISMA_MANAGEMENT_API_SECRET/g" prisma.yml
RUN sed -i "s/DATABASE_HOST/$DATABASE_HOST/g" prisma.yml
RUN sed -i "s/DATABASE_PORT/$DATABASE_PORT/g" prisma.yml
RUN sed -i "s/DATABASE_NAME/$DATABASE_NAME/g" prisma.yml
RUN sed -i "s/DATABASE_USER/$DATABASE_USER/g" prisma.yml
RUN sed -i "s/DATABASE_PASSWORD/$DATABASE_PASSWORD/g" prisma.yml
EXPOSE 4466Harshit
03/06/2019, 2:33 PMcan you please try this docker file?
Harshit
03/06/2019, 2:34 PMAlso can you please tell me whether this happened after an update to 1.28 otherwise we might have caught a bug and it will require more investigation
b
boennemann
03/06/2019, 2:34 PMIt's happened before 1.28, too.
h
Harshit
03/06/2019, 2:35 PMok thanks 🙏
b
boennemann
03/06/2019, 2:35 PMJust upgraded today because I hoped that would fix it.
h
Harshit
03/06/2019, 2:35 PMoh sorry I forgot the arg option there
Harshit
03/06/2019, 2:35 PMpaste this
b
boennemann
03/06/2019, 2:35 PMI saw the same JAVA_OPTS variable passed in the heroku default config, right?
boennemann
03/06/2019, 2:35 PMSure
h
Harshit
03/06/2019, 2:36 PM Copy code
FROM prismagraphql/prisma:1.28.0
ARG PRISMA_MANAGEMENT_API_SECRET
ARG DATABASE_HOST
ARG DATABASE_PORT
ARG DATABASE_NAME
ARG DATABASE_USER
ARG DATABASE_PASSWORD
ARG PRISMA_CONFIG_PATH
ARG JAVA_OPTS
ENV PRISMA_CONFIG_PATH prisma.yml
ENV JAVA_OPTS -Xmx358m
COPY config.yml prisma.yml
RUN sed -i "s/PRISMA_MANAGEMENT_API_SECRET/$PRISMA_MANAGEMENT_API_SECRET/g" prisma.yml
RUN sed -i "s/DATABASE_HOST/$DATABASE_HOST/g" prisma.yml
RUN sed -i "s/DATABASE_PORT/$DATABASE_PORT/g" prisma.yml
RUN sed -i "s/DATABASE_NAME/$DATABASE_NAME/g" prisma.yml
RUN sed -i "s/DATABASE_USER/$DATABASE_USER/g" prisma.yml
RUN sed -i "s/DATABASE_PASSWORD/$DATABASE_PASSWORD/g" prisma.yml
EXPOSE 4466Harshit
03/06/2019, 2:36 PMI am off to dinner now, let me know if this worked. If not we can debug this in a DM
b
boennemann
03/06/2019, 2:36 PMSure, thank you 🙏 Enjoy your dinner!
h
Harshit
03/06/2019, 3:19 PMThis has been resolved in a DM. Solution was to increase the memory heap size even more in the JVM options
🙌 2
6 Views