CodeSolid

A Go Programming Notebook

Fast S3 Updates With Golang and Goroutines

I recently wrote a go program to optimize my S3 uploads for this blog. The idea was to write a program that would only upload the files that have changed (rather like RSync), rather than bulk uploading the whole site each time. Although there are third party tools to do this, I wanted to learn more about the golang AWS API, while at the same time getting a chance to run some goroutine benchmarks. To see the code and the benchmarks I did, read on!

Some Preliminaries

The AWS API for golang exposes an S3 service that we’ll use both for querying the files already in the Amazon cloud and for uploading the files that are new or have changed. Let’s see that and quickly look at all the imports we’ll be using:

// main.go

import (
    "bytes"
    "crypto/md5"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "os"
    "path/filepath"
    "strings"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)

/* getS3Session returns the AWS session with our configuration.
   For our purposes this could have gone in getS3Service,
   but in case we end up relying on other AWS clients
   it's handy to have the session available separately.
*/ 
func getS3Session() *session.Session {
    // Use shared credentials ($HOME/.aws/credentials),
    // but set the region
    return session.Must(session.NewSession(
        &aws.Config{
            Region:      aws.String(getConfiguration().region),
            Credentials: credentials.NewSharedCredentials("", "default"),
        }))
}

/*
    getS3Service returns a service we use to query our bucket 
    and to put new / changed files.
*/
func getS3Service() *s3.S3 {
    return s3.New(getS3Session())
}

The function getS3Session relies on a configuration function that we use to set up our program. This is simply a struct to keep whatever settings we need, and which you’d need to change up to match your local path and S3 bucket name. Since this was a personal project we chose a configuration object over a set of command line parameters. Here’s the code for our configuration type / function.

type config struct {
    region         string /* AWS region to use for session */
    bucket         string /* S3 bucket where are files are stored */
    localDirectory string /* Local files to sync with S3 bucket */
}

func getConfiguration() *config {
    configuration := config{
        region:         "us-east-2",
        bucket:         "codesolid.com",
        localDirectory: "/home/john/source/codesolid/public/"}
    return &configuration
}

Program Design

Our strategy will be simple. If we can build a map of remote S3 object (file) names to file checksums, any local files that either aren’t in the map or who are in the map but have a different checksum need to be uploaded. We’re concerned with new or updated files – any deletes are ignored and we can handle separately.

So our first function builds the map of remote files to checksums.

/**
getAwsS3ItemMap constructs and returns a map of keys (relative filenames)
to checksums for the given bucket-configured s3 service.  
It is assumed  that the objects have not been multipart-uploaded, 
which will change the checksum.
*/
func getAwsS3ItemMap(s3Service *s3.S3) (map[string]string, error) {

    var loi s3.ListObjectsInput
    loi.SetBucket(getConfiguration().bucket)

    obj, err := s3Service.ListObjects(&loi)

    // Uncomment this to see what AWS returns for ListObjects
    // fmt.Printf("%s\n", obj)

    var items = make(map[string]string)

    // Based on response from aws API, map relative filenames to checksums
    // The "Key" in S3 is the filename, while the ETag contains the 
    // checksum
    if err == nil {
        for _, s3obj := range obj.Contents {
            eTag := strings.Trim(*(s3obj.ETag), "\"")
            items[*(s3obj.Key)] = eTag
        }
        return items, nil
    }

    return nil, err
}

Next, we need to walk our local directory that contains the corresponding files we’ll want to check for updates. Since we’ll be walking the directory tree and opening each file in turn to run a checksum on it, there’s a lot of file input going on that makes the task a good fit for optimizing with GoRoutines.

In an early draft of this approach, I confess, I did it wrong. By using shared memory which needed to be synchronized to add the files to a list, I actually made the program slower than without the goroutines (as my benchmarks showed)! Once I used channels and moved the building of the list to the main goroutine, the benchmarks improved significantly for the goroutine version as compared to the same code without goroutines and channels.

I paid the price for ignorning the Go Blog’s advice:

“Do not communicate by sharing memory; instead, share memory by communicating.”

Before we get to the benchmarks let’s look at the final version of the code. Let’s start our discussion with the traditional, synchronous version, with no go-routines or channels.

/*
fileWalkWithoutGoroutine walks the local directory using golang's
filepath.Walk.  Ignoring directories, for each file we get 
the checksum of the relative file path from the awsItems map.
We then pass the full filename and checksum to testFileNoGoroutine,
which returns either an empty string if the file doesn't need to be 
uploaded, or a non-empty string if there's no checksum (file not on the remote) 
or if the checksum doesn't match the local file (file has changed).

Return the list of files to be uploaded as a list.
*/
func fileWalkWithoutGoroutine(awsItems map[string]string) []string {
    var files int
    var filesToUpload []string
    blogDir := getConfiguration().localDirectory
    filepath.Walk(blogDir, func(path string, info os.FileInfo, err error)   error {
        if err != nil {
            return err
        }
        if !info.IsDir() {
            files++
            relativeFile := strings.Replace(path, blogDir, "", 1)
            checksumRemote, _ := awsItems[relativeFile]

            filename := testFileNoGoroutine(path, checksumRemote)
            if len(filename) > 0 {
                filesToUpload = append(filesToUpload, filename)
            }
        }
        return nil
    })

    return filesToUpload
}

/**
    testFileNoGoroutine returns a blank string if the given 
    filename does not need to be updated (checksum blank or does 
    not match local file), or a non-blank filename if the 
    
*/
func testFileNoGoroutine(filename string, checksumRemote string) string {
    if checksumRemote == "" {
        return filename
    }

    contents, err := ioutil.ReadFile(filename)
    if err != nil {
        return ""
    }
    sum := md5.Sum(contents)
    sumString := fmt.Sprintf("%x", sum)
    // checksums don't match, mark for upload
    if sumString != checksumRemote {
        return filename
    }
    return ""
}

Now, let’s look at the same code using goroutines and channels. This time, as we walk the local directory, we call testFile repeatedly as a goroutine. In testFile, nstead of returning a string as in our earlier program, we pass the filename or a blank string back on the channel that we was passed from fileWalk.

In fileWalk, we keep a count of the files we walked in numfiles,and when we’re done walking the files, we read from the channel we passed to testFiles numfiles times to build our list of files to return.

func fileWalk(awsItems map[string]string) []string {
    blogDir := getConfiguration().localDirectory
    ch := make(chan string)
    var numfiles int
    filepath.Walk(blogDir, func(path string, info os.FileInfo, err error) error {
        if err != nil {
            return err
        }
        if !info.IsDir() {
            numfiles++
            relativeFile := strings.Replace(path, blogDir, "", 1)
            checksumRemote, _ := awsItems[relativeFile]

            go testFile(path, checksumRemote, ch)
            //fmt.Println(path, info.Size())
        }
        return nil
    })

    var filesToUpload []string

    var f string
    for i := 0; i < numfiles; i++ {
        f = <-ch
        if len(f) > 0 {
            filesToUpload = append(filesToUpload, f)
        }
    }

    return filesToUpload
}

func testFile(filename string, checksumRemote string, ch chan<- string) {

    if checksumRemote == "" {
        ch <- filename
        return
    }

    contents, err := ioutil.ReadFile(filename)
    if err == nil {
        sum := md5.Sum(contents)
        sumString := fmt.Sprintf("%x", sum)
        // checksums don't match, mark for upload
        if sumString != checksumRemote {
            ch <- filename
            return
        } else {
            // Files matched
            ch <- ""
            return
        }
    }
}

Benchmarking the Two Versions

We’ll leave aside the main method and the code to upload the files for the time being, but at this point we have code that tells us just what files need to be uploaded.

To see how much of a performance boost we get from the goroutine version, we write some simple benchmarks in main_test.go. Here’s our main_test.go file, with some non-benchmark tests we wrote excluded.

// main_test.go
package main

import (
	"log"
	"os"
	"testing"
)

var awsItems map[string]string

/* Set up the awsItems map that we'll need */
func TestMain(m *testing.M) {

	s3Service := getS3Service()
	var err error
	awsItems, err = getAwsS3ItemMap(s3Service)
	if err != nil {
		log.Fatal(err)
	}

	// call flag.Parse() here if TestMain uses flags
	os.Exit(m.Run())
}

func BenchmarkFilewalkGoroutine(b *testing.B) {
	for n := 0; n < b.N; n++ {
		fileWalk(awsItems)
	}
}

func BenchmarkFilewalkWithoutGoroutine(b *testing.B) {
	for n := 0; n < b.N; n++ {
		fileWalkWithoutGoroutine(awsItems)
	}
}

I stumbled on the prettybench utility one time, which I like to use to improve the output of Go benchmark tests. Using that tool to format the output gives:

$ go test -v -bench=. | prettybench
goos: linux
goarch: amd64
pkg: aws
PASS
benchmark                             iter     time/iter
---------                             ----     ---------
BenchmarkFilewalkGoroutine-4            50   27.26 ms/op
BenchmarkFilewalkWithoutGoroutine-4     50   32.82 ms/op
ok  	aws	3.543s

Of course, we’re running an SSD drive here and Go is hecka fast to begin with, but even with that the goroutine version improves the performance by about 20%. I’ll publish an update to this article or a link to the github version with the upload code and the rest of the program soon, but for now it’s late so I think I’ll use my AWS go file upload utility one last time to push this article to the cloud! Cheers.