ECR Image Blunder and Recovery

Well, we do make mistakes from time to time, be it small as keeping a console.log in production or accidentally deleting data from the production database. This is the same kind of story, but with AWS ECR.

I was assigned to fix a docker image that was working fine in past but had some issues in recent times. The docker file was not updated since last August. The fix was made and deployed via Codedeploy.

I was unaware of the effect the Codedeploy would have on ECR image as I am new to this product. It came to my sense when one of my colleagues reported the failure in Jenkins jobs which were running fine until now.

Turns out the Codedeploy replace the working image (build-in last august) with the new one which I thought I just fixed. So, no it was not fixed through the code deploy went successfully but there were some dependence missing in docker and it was causing the failure.

The dreadfulness started when I realize the previous docker images were untagged.

I considered of creating a new task definition manually. Luckily you point out a specific docker image using docker digest in ECR.

So I created a new task definition with the previous image digest but it did;t help as every Jenkins job was creating a new task definition automatically with the image pointed to the latest tag which is now broken.

I went another way to fix this and it worked. I figured that you can retag the ECR images, but only via CLI.

First download the metadata of a specific image which you want for retagging.

MANIFEST=$(aws ecr batch-get-image --repository-name jenkins/python-mysql-agent --image-ids imageTag=latest --output json | jq --raw-output --join-output '.images[0].imageManifest')

But since the earlier image had no tag, so I need to alter the command a little.

MANIFEST=$(aws ecr batch-get-image --repository-name jenkins/python-mysql-agent --image-ids imageDigest=sha:2736472372 --output json | jq --raw-output --join-output '.images[0].imageManifest')

The following step is to assign a new tag to the image. Use the --image-tag option of the put-image command to put the image manifest to Amazon ECR with a new tag. In this example, the image is tagged as lastest.

aws ecr put-image --repository-name jenkins/python-mysql-agent --image-tag latest --image-manifest "$MANIFEST"

And lastly, verify that your new image tag is attached to your image.

aws ecr describe-images --repository-name jenkins/python-mysql-agent

Now make as many blunders required to nail the docker image as there are commands to revert the changes .

Cheers!!!