anyone know what this error means? it’s a first-ti...
# help
s
anyone know what this error means? it’s a first-time deploy:
Copy code
prod-microservices-api | CREATE_FAILED | AWS::AppSync::GraphQLApi | AppSyncApiEBDAEC17 Failed to assume service-linked role arn:aws:iam::xx:role/aws-service-role/appsync.amazonaws.com/AWSServiceRoleForAppSync, please retry. (Service: AWSAppSync; Status Code: 403; Error Code: AccessDeniedException; Request ID: ba7b1b89-9add-4d89-bbbb-0def2024ff5b; Proxy: null)
I’m in an account with admin access
I think this is the same issue I had before when my dev tried to deploy in a fresh account. @Frank this time I captured a debug log
f
Hey @Sam Hulick, AWS IAM has been unstable the past couple of days https://status.aws.amazon.com
Is it working for you now?
s
oh, I didn’t realize that. I have no idea.. haven’t set up a fresh deployment since then
f
Yup, let me know if u run into it again.
s
will do. thanks!
f
I didn’t spot anything weird in ur log, except for not able to assume the IAM role. So it’s likely related to the service interruption.
s
things have been pretty smooth the past day or so. the only thing haunting me at this point is the 2 - 2.5 minute pauses when working in live dev mode
f
Yup.. I’m going to take a look at the GH issue. Do you consistently get the 2 minute hang every time you add a route?
s
yeah, ~2 minutes
happening when I deploy code to production too. just a big long polling loop saying the update was complete, and the process not moving onto the next stack
Copy code
[2021-08-26T16:49:54.655] [DEBUG] core - deploy stack: poll stack status: unknown
[2021-08-26T16:49:57.846] [DEBUG] core - deploy stack: poll stack status: {
  StackStatus: 'UPDATE_COMPLETE',
  LastUpdatedTime: 2021-08-26T20:13:32.975Z
}
[2021-08-26T16:49:57.847] [DEBUG] core - deploy stack: poll stack status: unknown
[2021-08-26T16:50:01.059] [DEBUG] core - deploy stack: poll stack status: {
  StackStatus: 'UPDATE_COMPLETE',
  LastUpdatedTime: 2021-08-26T20:13:32.975Z
}
[2021-08-26T16:50:01.059] [DEBUG] core - deploy stack: poll stack status: unknown
[2021-08-26T16:50:04.540] [DEBUG] core - deploy stack: poll stack status: {
  StackStatus: 'UPDATE_COMPLETE',
  LastUpdatedTime: 2021-08-26T20:13:32.975Z
}
and CloudFormation is all green.. nothing happening on that end, so it seems like a CLI-side issue
wow.. this time I think it’s exceeding 4 minutes 😞
gonna kill & restart the deploy
f
Looking into it now
s
so this was a
yarn deploy api --stage prod
BTW.. not even deploying the Core stack, or deploy all
I’ll deploy all stacks and see if that makes a difference
f
I see
s
nope, still slow for full deploy. I’m trying to find a pattern. I changed a function that most API routes depend on. next, I’ll try just changing the code for one function and compare
f
Figured it out.
s
oh yeah??
f
So instead of directly deploying a CFN template, CDK always takes the template and generates a CFN changeset against the stack, and then execute the changeset.
And CFN is taking 2-4min to generate the changeset, most likely b/c you have a large stack.
s
ahhh
so everyone using CDK with large applications is having this issue?
f
s
well, shoot
f
(cc’ing @thdxr to keep him in the loop)
t
let's rewrite SST with pulumi
f
Let me take a look at CDK’s code. I think we can bypass CDK’s deploy command, and submit the templates to CFN directly.
What do u guys think?
s
sounds good, if that’s feasible and still works. one of the CDK authors made it sound like they use changesets for a specific reason.. not sure what
t
yeah that would be cool if there's no tradeoff, didn't know we could do that
s
the funny thing is, I look at the changeset JSON changes in CFN, and it’s hardly anything. like, why does it take so long?
(this is where Dax busts out the CloudFormation meme…) 😄
t
lol