Is there a channel that still does discussion on i...
# orm-helpr
Rashmi Bidanta
04/13/2020, 9:50 PMIs there a channel that still does discussion on issues specifically productions issues related to scalability and stuff with regards to Prisma1
r
Ryan
04/14/2020, 6:48 AMHey @Rashmi Bidanta we don't have a separate channel for that but you could raise any concerns you have here 🙂
r
Rashmi Bidanta
04/14/2020, 1:24 PM@Ryan thank you. In our production set up we are running Prisma1 on k8s cluster but the pods keeps restarting every so often. We have debug logging and this is what the last few lines of the log says before the pod gets restarted
Copy code
[DEBUG] #11: Commit
[DEBUG] #1: SimpleJdbcAction
[DEBUG] #1: SimpleJdbcAction
[DEBUG] #1: SimpleJdbcAction
[DEBUG] #1: SimpleJdbcAction
[DEBUG] #1: SimpleJdbcAction
[DEBUG] #1: SimpleJdbcAction
[DEBUG] #1: SimpleJdbcAction
[DEBUG] #1: SimpleJdbcAction
[DEBUG] #1: SimpleJdbcAction
[DEBUG] #1: SimpleJdbcAction
[DEBUG] #1: SimpleJdbcAction
/app/start.sh: line 4: 8 Killed /app/bin/prisma-prod👍 1
Rashmi Bidanta
04/14/2020, 1:30 PMSome more details: we are on Prisma
1.28👍 1
e
Ervin
04/14/2020, 4:42 PMI'll contribute over here, since Rashmi and I are looking at the same issue. The pods are being killed due to the probe (which is simply accessing
/👍 1
Ervin
04/14/2020, 4:43 PMWe had previously ran into issue with the task queue being full and requests being rejected, but we've increased the resources to our database, then increased the database connection limit, and scaled out the number of pods and haven't seen the exceptions for the queue being full since.
👍 1
r
Ryan
04/15/2020, 6:37 AMThanks a lot for jumping in @Ervin
Rashmi could you try this method and see if it works?
r
Rashmi Bidanta
04/15/2020, 1:37 PM@Ryan Ervin and me are in the same team supporting the same graphql platform with apollo and prisma. What we are seeing is that there are instances when the probes, we have, receive connection timeout from Prisma server the K8s cluster thinks that the prisma server is in a bad state and then it restarts the server. earlier the default connection pool was 9 I believe cause we had 4 core cpu. With that setting we were getting
Copy code
"exception":"java.util.concurrent.RejectedExecutionException: Task slick.basic.BasicBackend$DatabaseDef$$anon$2@6747976d rejected from slick.util.AsyncExecutor$$anon$2$$anon$1@736c10d2[Running, pool size = 9, active threads = 9, queued tasks = 1000, completed tasks = 37987088]"SLICKRashmi Bidanta
04/15/2020, 1:39 PMNow we are seeing Prisma probably gets into a state where it responds with connection timeouts and the requests are getting stalled until the pod is restarted. So we were hoping to get some direction with this issue
r
Ryan
04/15/2020, 1:55 PMSure @Rashmi Bidanta I'll get back to you on this!
🙏🏽 1
h
Harshit
04/16/2020, 8:10 AMSlick library has a bug which was causing connection pool saturation. It is fixed in later versions of Prisma.
Harshit
04/16/2020, 8:11 AMAlso, in later version you can increase the queue size of Prisma
r
Rashmi Bidanta
04/16/2020, 1:40 PM@Harshit thanks for this information
Rashmi Bidanta
04/16/2020, 1:40 PMdo you know any specific version we should be upgrading to?
Rashmi Bidanta
04/16/2020, 1:40 PMwe are currently on Prisma version 1.28
h
Harshit
04/16/2020, 1:42 PMhttps://github.com/prisma/prisma1/releases/tag/1.30.5 if you are using datamodel 1.0
👍🏽 1
r
Rashmi Bidanta
04/16/2020, 1:42 PMIt would also be helpful if you could share the information on how to increase the queue size of prisma
h
Harshit
04/16/2020, 1:42 PMhttps://github.com/prisma/prisma1/releases/tag/1.34.3 if you are using datamodel 1.1
Harshit
04/16/2020, 1:42 PMYou are on 1.28 so that means you should get 1.30.5
r
Rashmi Bidanta
04/16/2020, 1:43 PMyeah that makes sense
Rashmi Bidanta
04/16/2020, 1:43 PMdoes 1.30.5 has the ability to increase prisma queue size?
h
Harshit
04/16/2020, 1:44 PMYep, 1.30.3 introduced the change so 1.30.5 has it
r
Rashmi Bidanta
04/16/2020, 1:44 PMcool cool
Rashmi Bidanta
04/16/2020, 1:44 PMis this a configuration in prisma-config.yaml?
h
Harshit
04/16/2020, 1:46 PMhttps://github.com/prisma/prisma1/blob/9fb51828fa41629d99878f4b64fc87dad982cba4/server/libs/prisma-config/src/test/scala/com/prisma/auth/ConfigLoaderSpec.scala#L26
No, it goes inside PRISMA_CONFIG inside of docker-compose.yml. See this test and add it(without | margin)
Harshit
04/16/2020, 1:46 PMDefault is 1000, you can increase it to like 8000
r
Rashmi Bidanta
04/16/2020, 1:46 PMawesome let me check that out
Rashmi Bidanta
04/16/2020, 1:49 PMWe will try these recommendation in our lower environments and see how things are going
Rashmi Bidanta
04/16/2020, 1:50 PMthank you @Harshit for this information
Rashmi Bidanta
04/17/2020, 4:55 PM@Harshit is there anything we need to do for the rabbitmq when we upgrade prisma setup
h
Harshit
04/18/2020, 4:49 AMNo, use the same rabbit url. No changes required there 🙂
👍🏽 1
r
Rashmi Bidanta
04/21/2020, 11:27 PM@Harshit We did upgrade our prod instance to 1.30.5 but our prisma deploy scripts are not going through. This is preventing us from doing any new upload. The response to prisma deploy --force --no-seed is
Rashmi Bidanta
04/21/2020, 11:28 PM Copy code
Creating stage prod for service app-name... !
error 21-Apr-2020 17:19:43
error 21-Apr-2020 17:19:43 ERROR: Whoops. Looks like an internal server error. Search your server logs for request ID: local:ck9aj51yf01220852j6lcdd33
error 21-Apr-2020 17:19:43
error 21-Apr-2020 17:19:43 {
error 21-Apr-2020 17:19:43 "data": {
error 21-Apr-2020 17:19:43 "addProject": null
error 21-Apr-2020 17:19:43 },
error 21-Apr-2020 17:19:43 "errors": [
error 21-Apr-2020 17:19:43 {
error 21-Apr-2020 17:19:43 "message": "Whoops. Looks like an internal server error. Search your server logs for request ID: local:ck9aj51yf01220852j6lcdd33",
error 21-Apr-2020 17:19:43 "path": [
error 21-Apr-2020 17:19:43 "addProject"
error 21-Apr-2020 17:19:43 ],
error 21-Apr-2020 17:19:43 "locations": [
error 21-Apr-2020 17:19:43 {
error 21-Apr-2020 17:19:43 "line": 2,
error 21-Apr-2020 17:19:43 "column": 9
error 21-Apr-2020 17:19:43 }
error 21-Apr-2020 17:19:43 ],
error 21-Apr-2020 17:19:43 "requestId": "local:ck9aj51yf01220852j6lcdd33"
error 21-Apr-2020 17:19:43 }
error 21-Apr-2020 17:19:43 ],
error 21-Apr-2020 17:19:43 "status": 200
error 21-Apr-2020 17:19:43 }
error 21-Apr-2020 17:19:43
error 21-Apr-2020 17:19:43 Get in touch if you need help: <https://www.prisma.io/forum/>
error 21-Apr-2020 17:19:43 To get more detailed output, run $ export DEBUG="*"
error 21-Apr-2020 17:19:43 npm ERR! code ELIFECYCLE
error 21-Apr-2020 17:19:43 npm ERR! errno 1
error 21-Apr-2020 17:19:43 npm ERR! app-name@1.0.0 prisma-deploy-force: `prisma deploy --force --no-seed`
error 21-Apr-2020 17:19:43 npm ERR! Exit status 1
error 21-Apr-2020 17:19:43 npm ERR!
error 21-Apr-2020 17:19:43 npm ERR! Failed at the app-name@1.0.0 prisma-deploy-force script.
error 21-Apr-2020 17:19:43 npm ERR! This is probably not a problem with npm. There is likely additional logging output above.Rashmi Bidanta
04/21/2020, 11:28 PMThe deploy seems to be timing out. We have upgraded the bamboo node as well to use prisma 1.30.5 cli, but we are still getting that error
h
Harshit
04/22/2020, 5:51 AMWhat does the error log say?
r
Rashmi Bidanta
04/22/2020, 1:10 PMHi @Harshit
Rashmi Bidanta
04/22/2020, 1:10 PMLet me get you the error log
Rashmi Bidanta
04/22/2020, 1:11 PMdo you mean the prisma log?
Rashmi Bidanta
04/22/2020, 1:13 PM Copy code
java.util.concurrent.TimeoutException: Futures timed out after [50 seconds]Rashmi Bidanta
04/22/2020, 1:13 PMthis is what the prisma logs are saying
Rashmi Bidanta
04/22/2020, 3:35 PM@Harshit we realized that there were a lot of select queries that were blocked on the database for some reason. Which resulted in DDL query for adding schema changes to timeout. We will try to deploy the changes again today. However we are still not clear what might have caused the select queries to have been blocked.
3 Views