Has anybody seen issues with components of a stack just randomly getting updated or even deleted without making changes to the stack? I just had my ec2.BastionHostLinux instance terminated randomly with zero changes to the stack. Made a small update to a lambda function and before I know it my Bastion Host is gone. I’m not even sure how to get back into a good state as I didn’t change the stack code. Any advice for debugging? For context, this is probably the 20th deploy I’ve done since creating that bastion host.

Hm that's bizarre

Yes, it was removed

I'm not sure if this is from SST as we just rely on CDK and CFN for things like this

this is in the root of the stack:

const bastionHostLinux = new ec2.BastionHostLinux(this, 'BastionHostLinux', {
      vpc: vpc,
      securityGroup: bastionSecurityGroup,
      subnetSelection: {
        subnetType: ec2.SubnetType.PUBLIC,
      },
    });

Hey @Patrick Gold, just following up on this. Did you manage to find the culprit?

Nope. Never found the root cause, but updating cdk and SST brought back the bastion host without changing the stack

hmm.. if this ever happens again, here’s how i’d debug: • Open up

.build/cdk.out

and see if the EC2 instance is still in the template. If it is not, the issue is likely within SST/CDK. • Then, go to CloudFormation console and find the stack, open up the Resources tab and see if the instance is still in there, and what state it is in. If it’s in

CREATE_COMPLETE

or

UPDATE_COMPLETE

it means CFN didn’t remove it. • Then, go to CloudTrail and look through recent logs and you should see the IAM user/role that turned off the instance.

Thanks Frank. Some extra info is that I had my Production bastion host termination protection enabled and when it deployed, it failed to delete. After I updated cdk/sst, it created a brand new instance, but my old bastion host is still running and seemingly not connected to the stack

I see. Appreciate the details!

@Frank of course this happens when I don’t necessarily have time to debug, but a similar thing happened today where most of my resources seemingly required and update but I’m not sure why. (No changes to stack). This also triggered the deletion and recreation of the bastion host again.

UPDATE_IN_PROGRESS | AWS::EC2::Instance | BastionHostLinuxD318DEDE | Requested update requires the creation of a new physical resource; hence creating one.

UPDATE_IN_PROGRESS | AWS::EC2::Instance | BastionHostLinuxD318DEDE | Resource creation Initiated

UPDATE_COMPLETE | AWS::EC2::Instance | BastionHostLinuxD318DEDE

DELETE_IN_PROGRESS | AWS::EC2::Instance | BastionHostLinuxD318DEDE

@Patrick Gold sorry for the late followup. I’m curious what needed to update in the

AWS::EC2::Instance

that required to replace the resource.

Yeah that's the issue. this happens on CI before I even can debug. I can add sst diff to run every deployment

yeah.. let’s do that. always run sst diff before deploy.

Is that an expensive operation? Just thinking about CI timing

should be fairly quick

I have termination protection enabled in prod to prevent deletion. I think the stack has created 3-4 ec2 extra instances so far that I've had to clear out.

I see

Wouldn't be such a big deal if I had some of the configuration scripts as part of the bastion host setup

So when CDK creates EC2 instances, it will look up the latest AMI image to use. It will then cache the image that’s used in

cdk.context.json

, so when u deploy again in the future, it doesn’t use the newer AMI image.

Ah, perhaps that's it

I have a feeling this might be causing it.. running a

git status

after deploy in CI would answer that.

I'm not at my terminal at the moment but can check later

Sounds good. If it was updated, you can commit the file in CI. Or print out the file, change it on ur local and commit it.

Appreciate the follow up on this!

ah yeah I’ve noticed that too. Most likely some metadata CDK added.

it seems fairly harmless but makes me nervous. 😅

Yeah lemme create an issue to track this https://github.com/serverless-stack/serverless-stack/issues/1417

@Patrick Gold do you know if it's related to updating sst? We do have an issue right now where when you update sst all your functions redeploy

I actually had to update sst to even get my bastion host back last time.