Hey there, do you guys think it would be valuable (or even archivable) to create a construct that simplifies adding long running tasks, something similar to a function but using fargate and docker or something like that? I have not looked into this but it does seem like something very useful.

https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ecs-patterns.ScheduledFargateTask.html definately possible

Ah nice… so this would be for scheduling, my thought is if we could make it simpler to set up the task and the cluster. Something like

my_task = new Task(this, "MyTask", {
  handler: "src/tasks/index.main",
});

And it does all the magic behind the scene. Now we could either schedule it using

new Cron(this, "Cron", {
  schedule: "rate(1 minute)",
  job: my_task,
});

Or we could use an api call (maybe directly to ECS or create a small helper package) to run a task from our code (use case for this is for e.x I delete a record and it has thousands of children that you also need to delete ).

maybe you can make a lambda to delete a few children and parallelize it as async lambda calls?

Yeah that is what I will do now, but I can think of other long running tasks that this would be very beneficial. One other example is for report generation

maybe check out EventBridge?

How would EventBridge help with the long running tasks ?

You can set it to run jobs on a schedule, that's what I use for running periodic tasks

Yeah or just tasks that can run for hours, if we could have something that wraps a JS function (similar to lambda) but without time limitation that would be gold

https://laconiajs.io/docs/api/batch this could solve some of your problems without mucking around at an infrastructure layer

While I do like this idea, it would be a pretty involved construct. The solution you are looking for is ECS, which can run fargate, runs docker containers, and shuts down on exit similar to a lambda. What is NOT like lambda is anything else. We could potentially create something lambda-like by creating a base docker container all tasks use, and a standardized entry point, but this would limit people in some respects. We would also need to make decisions on things like logging, and I have no idea if we could do the SST websocket dev environment magic.

Yeah that was my thought also…

If you are looking for something that makes this easier have you tried https://github.com/aws/copilot-cli

This looks pretty interesting…

@Kujtim Hoxha Curious about the long running task - is it something ML related? Or some existing code?

So the best I can think of now is create ECS and task definitions using pure CDK and than just use the aws sdk to start tasks in my functions

For me as well.

Curious about the long running task - is it something ML related? Or some existing code?

I have used this for backup/DR related items and as a backup to reporting in the past (will try lambda first).

Yeah agreed it’s frustrating copilot push the cli tool over all else