Upload to AWS S3 from a Bash Script

The Big River’s cloud storage is very good and cheap too, so it is an ideal place to store backups of various sorts. I wanted to do this upload from a Bash script in as simple a way as possible.

I have previous with the API having used it from PHP, both using the AWS SDK and also rolling my own simplified upload function. That wasn’t exactly easy to do1, so I didn’t want to go there again.

A bit of Googling came up with this Github gist by Chris Parsons which is almost exactly what I needed. I just had to parameterise it a bit more and add the facility to specify storage class and AWS region 2.

Here is the finished result:

#S3 parameters
S3KEY="my-key"
S3SECRET="my-secret"
S3BUCKET="my-bucket"
S3STORAGETYPE="STANDARD" #REDUCED_REDUNDANCY or STANDARD etc.
AWSREGION="s3-xxxxxx"

function putS3
{
  path=$1
  file=$2
  aws_path=$3
  bucket="${S3BUCKET}"
  date=$(date +"%a, %d %b %Y %T %z")
  acl="x-amz-acl:private"
  content_type="application/octet-stream"
  storage_type="x-amz-storage-class:${S3STORAGETYPE}"
  string="PUT\n\n$content_type\n$date\n$acl\n$storage_type\n/$bucket$aws_path$file"
  signature=$(echo -en "${string}" | openssl sha1 -hmac "${S3SECRET}" -binary | base64)
  curl -s -X PUT -T "$path/$file" \
    -H "Host: $bucket.${AWSREGION}.amazonaws.com" \
    -H "Date: $date" \
    -H "Content-Type: $content_type" \
    -H "$storage_type" \
    -H "$acl" \
    -H "Authorization: AWS ${S3KEY}:$signature" \
    "https://$bucket.${AWSREGION}.amazonaws.com$aws_path$file"
}

This works a treat on the small-to-medium sized compressed files that I use it for.

The only real snag with this function is that it doesn’t set an exit code, so it’s not easy to exit a script if the upload fails for some reason. I got round it by grepping the output of the curl and looking for the text “error”. This finds all the cases I tried apart from incorrect region.

The last line is changed to the following:

"https://$bucket.${AWSREGION}.amazonaws.com$aws_path$file" --stderr - | grep -i "error"
 #the --stderr bit onward is so that a return code is set if curl returns an 
 #error message from AWS
 #note that the value of the code is inverted from usual error interpretation: 
 #0=found "error", <>0=not found "error"

I wasn’t entirely happy with this rather crude workround, but it did achieve the desired result.

Postscript:

I later found and tested the –fail option for Curl that sets the exit code properly. That means the grep hack isn’t needed to allow error detection and the normal testing of the $? bash variable will work.

(I also tidied up and made the function variables proper locals – see the Google cloud version for example Google cloud version)

Notes:

  1. The HTTP headers have to be sorted into alphabetical order before hashing them. A fact easily missed and a problem that is quite hard to debug…
  2. Rembering to add the extra headers in the correct order – LOL