We have been using EC2 and we have also been using serverless. We are very familiar with the container and the managed environment for containers provided by AWS.
Serverless(Lambda) is a great way to tap out of the server-based compute approach to an approach where compute is available on demand. They are perfect for applications with heavy compute or variable traffic. Lambda scale on demand and they let you deploy hundreds of operations concurrently. But then there are applications where the limitations of lambda(serverless) hold you off from doing awesome things you could do, knowing it's computed on go.
We were looking for an alternative to an AWS media convert. AWS media convert works great. But they are not great when you are cost-conscious. The service makes pay heavy and the same was for us.
We needed a solution that is elastically scalable, cost-effective but also quick to respond.
Our approach was to jump on lambda. Initially, it worked great. Use the FFmpeg layer, attached it to your lambda, and do your stuff. They are a great solution for low-cost media conversion. But soon we realize the rendering or transcoding of lengthly videos is not possible on serverless.
Mainly for two reasons
- The temp storage size of lambda is 512 MB. Hence a video larger than that won't fit here. Some might argue to stream the video instead of upload and transcode on the fly. That's great for containers like mkv or mov but not a great fit for mp4.
- The max compute time available with each invocation can't exceed more than 15 mins. That's a bigger bottleneck.
We prefer a slightly different route. We preferred to use ECS fargate. ECS fargate is a great way to create on-demand compute whenever you need it.
To do so you need following
- An image of your transcoding code pushed in ECR
- An ECS cluster
- A service definition for a cluster
- A task definition associated with the service
- A lambda function that invokes an ECS on the fly.
Once you set up your ECS cluster, you need to invoke your lambda using the API gateway.
The lambda will let you control what you can pass to the container(Task). You can pass requests params, dynamic environment vars, and any extra information you think your container will need. This container is a one-off container, which means it will die as soon as your job is over.
To improve this a bit, use FARGATE_SPOT as and container capacityProviderStrategy. FARGATE_SPOT behaves the same as the ec2 spot. They come with a 70% reduction in cost compare to the default FARGATE capacityProviderStrategy.
Everything else remains the same. Let me know if you want an article on the implementation or you want to collaborate on an application where such a use-case fits. Also, drop your questions if you have any.
Hope you enjoyed this. Peace.