Custom S3 requests with AWS Go SDK

November 22, 2022
manav@ente.io

Maybe there is an easier way to do this.

Apologies for starting this post on an indecisive note, but I'm still not sure if this is indeed the "correct" way. It works though 🤷

What I wanted was to make a request to a custom endpoint provided by an S3 compatible service. This endpoint looks and behaves very much like standard S3 APIs, but since it is not part of the suite that AWS provides the AWS SDKs don't have a way to directly use it.

Of course, I could make an HTTP request on my own. I'd even be fine with parsing the XML (yuck!). But what I didn't want to deal with were signatures.

So I thought to myself that there would be standard recipes to use the AWS SDK (I'm using the golang one) to make requests to such custom endpoints. But to my surprise, I didn't find any documentation or posts about doing this. So, yes, I wrote one 🙂

The end result is quite simple - we tell the AWS Go SDK about our custom input and output payloads, and the HTTP path, and that's about it, it does all the heavy lifting (signing the requests, the HTTP request itself, and XML parsing) for us.

You can scroll to the bottom if you just want the code.

Otherwise, I'll walk through this in two parts:

  • First we'll see how requests to standard endpoints are made under the hood.

  • Then we'll use that knowledge to get the SDK to make our custom payloads and request go through the same code paths.

Enough background, let's get started. Let's use the Go SDK to fetch the versioning status of an S3 bucket.

Why versioning? It is a simple GET request, conceptually and in code – we're just fetching a couple of attributes attached to a bucket. This way, we can focus on the mechanics of the request without getting lost in the details of the specific operation we're performing.

Here is the code. Create a session. Use the session to create a client. Use the client to make the request.

package main

import (
	"fmt"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {
	// Example credentials to connect to local MinIO. Don't hardcode your
	// credentials if you're doing this for real!
	creds := credentials.NewStaticCredentials("minioadmin", "minioadmin", "")
	sess, err := session.NewSession(&aws.Config{
		Credentials: creds,
		Region:      aws.String("us-east-1"),
		// Makes it easy to connect to localhost MinIO instances.
		S3ForcePathStyle: aws.Bool(true),
		Endpoint:         aws.String("http://localhost:9000"),
		LogLevel:         aws.LogLevel(aws.LogDebugWithHTTPBody),
	})
	if err != nil {
		fmt.Printf("NewSession error: %s\n", err)
		return
	}

	s3Client := s3.New(sess)

	bucket := "my-bucket"

	// Create a bucket, ignoring any errors if it already exists.
	s3Client.CreateBucket(&s3.CreateBucketInput{
		Bucket: aws.String(bucket),
	})

	// Everything above was just preparation, let's make the actual request.

	output, err := s3Client.GetBucketVersioning(&s3.GetBucketVersioningInput{
		Bucket: aws.String(bucket),
	})
	if err != nil {
		fmt.Printf("GetBucketVersioning error: %s\n", err)
		return
	}

	fmt.Printf("GetBucketVersioning output: %v\n", output)
}

To run this code, add it to a new go project.

$ mkdir s3-playground
$ cd s3-playground
$ go mod init example.org/s3-playground
# Paste the above code into main.go
$ pbpaste > main.go
$ go get

In a separate terminal, start a local MinIO instance.

$ docker run -it --rm -p 9000:9000 minio/minio server /data

Back in our original terminal, run the program.

$ go run main.go

If everything went according to plan, you'll now see debug logs of the HTTP requests made by our program, and finally it will print the response for the bucket versioning request:

GetBucketVersioning output: {

}

Great. So now that we have our playground, let's dissect the request. The heart of the program is this bit of code:

output, err := s3Client.GetBucketVersioning(&s3.GetBucketVersioningInput{
	Bucket: aws.String(bucket),
})

It is all quite straightforward.

  1. Create an input payload (GetBucketVersioningInput)
  2. Make an API call with that input (GetBucketVersioning)
  3. Get back the output (of type GetBucketVersioningOutput)

So if we were to make a custom API request, we already know that we'll need a MyCustomOperationInput and MyCustomOperationOutput. But that's not enough, we'll also need to change the HTTP request path, possibly even the HTTP method.

So we need to unravel one layer.

If you were to click through to the source of GetBucketVersioning in your editor, you'll see the following code (in aws-sdk-go/service/s3/api.go)

func (c *S3) GetBucketVersioning(input *GetBucketVersioningInput)
        (*GetBucketVersioningOutput, error) {
	req, out := c.GetBucketVersioningRequest(input)
	return out, req.Send()
}

It is creating a Request and an Output. Sending the request. And returning the Output.

Let's keep digging. Looking at the source of GetBucketVersioningRequest:

func (c *S3) GetBucketVersioningRequest(input *GetBucketVersioningInput)
        (req *request.Request, output *GetBucketVersioningOutput) {
	op := &request.Operation{
		Name:       opGetBucketVersioning,
		HTTPMethod: "GET",
		HTTPPath:   "/{Bucket}?versioning",
	}

	if input == nil {
		input = &GetBucketVersioningInput{}
	}

	output = &GetBucketVersioningOutput{}
	req = c.newRequest(op, input, output)
	return
}

Nice, we can see where HTTP request path and HTTP method are being set. Let us also look into the source of newRequest:

func (c *S3) newRequest(op *request.Operation, params, data interface{})
        *request.Request {
	req := c.NewRequest(op, params, data)

	// Run custom request initialization if present
	if initRequest != nil {
		initRequest(req)
	}

	return req
}

This is essentially doing c.NewRequest (the rest of the code is for allowing us to intercept the request before it is used).

So now we have all the actors on the stage. Let us inline the code that we saw above in the AWS Go SDK source into our own program. We will replace its original heart:

output, err := s3Client.GetBucketVersioning(&s3.GetBucketVersioningInput{
	Bucket: aws.String(bucket),
})

with this inlined / rearranged version:

input := &s3.GetBucketVersioningInput{
	Bucket: aws.String(bucket),
}

// this'll need to import "github.com/aws/aws-sdk-go/aws/request"
op := &request.Operation{
	Name:       "GetBucketVersioning",
	HTTPMethod: "GET",
	HTTPPath:   "/{Bucket}?versioning",
}

output := &s3.GetBucketVersioningOutput{}
req := s3Client.NewRequest(op, input, output)

err = req.Send()

Is that it? The code is simple enough, and we can see the input payload, the request itself, and the parsed output. If we substitute each of these with our custom versions, will it just work?

There's only one way to find out.

For this second part of this post, I tried finding an example MinIO-only API for pedagogical purposes, but couldn't find one off hand. So let's continue by using an example related to the actual custom endpoint that I'd needed to use.

We use Wasabi as one of the replicas for storing encrypted user data. We'll put out an article soon with details about ente's replication, but for our purposes here it suffices to mention that we use the Wasabi's "Compliance" feature to ensure that user data cannot be deleted even if some attacker were to get hold of Wasabi API keys.

To fit this into our replication strategy, we need to make an API call from our servers to tell Wasabi to "unlock" the file and remove it from the compliance protection when the user herself deletes it, so that it can then be scheduled for deletion after the compliance period is over.

To keep the post simple, let us consider a related but simpler custom Wasabi API : getting the current compliance settings for a bucket.

The compliance settings for a bucket can be retrieved by getting the bucket with the "?compliance" query string. For example:

GET http://s3.wasabisys.com/my-buck?complianceHTTP/1.1

Response body:

<BucketComplianceConfiguration xml ns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Status>enabled</Status>
   <LockTime>2016-11-07T15:08:05Z</LockTime>
   <IsLocked>false</IsLocked>
   <RetentionDays>0</RetentionDays>
   <ConditionalHold>false</ConditionalHold>
   <DeleteAfterRetention>false</DeleteAfterRetention>
</BucketComplianceConfiguration>

As you can see, it looks and behaves just like other S3 REST APIs. Except that the AWS Go SDK does not have a pre-existing method to perform this operation.

So how do we call this API using the AWS Go SDK?

Let us modify code that we ended up with in first section, but retrofit the input / request / output objects to match the documentation of this custom API.

Let's start with definition of GetBucketVersioningInput from the AWS Go SDK:

type GetBucketVersioningInput struct {
	_ struct{} `locationName:"GetBucketVersioningRequest" type:"structure"`

	Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"`

	ExpectedBucketOwner *string `location:"header" locationName:"x-amz-expected-bucket-owner" type:"string"`
}

Hmm. We don't seem to need the ExpectedBucketOwner field, so let's remove that. We do need the Bucket field, and it is passed in the as location:"uri", so let's keep the rest as it is, and arrive at our retrofitted GetBucketComplianceInput:

type GetBucketComplianceInput struct {
	_ struct{} `locationName:"GetBucketComplianceRequest" type:"structure"`

	Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"`
}

Let us repeat the process for the Output. Here's the original:

type GetBucketVersioningOutput struct {
	_ struct{} `type:"structure"`

	MFADelete *string `locationName:"MfaDelete" type:"string" enum:"MFADeleteStatus"`

	Status *string `type:"string" enum:"BucketVersioningStatus"`
}

And here it is, modified to match the documentation of the custom API:

type GetBucketComplianceOutput struct {
	_ struct{} `type:"structure"`

	Status               *string `type:"string"`
	LockTime             *string `type:"string"`
	RetentionDays        *int64  `type:"integer"`
	ConditionalHold      *bool   `type:"boolean"`
	DeleteAfterRetention *bool   `type:"boolean"`
}

Finally, let us change the middle part – the request.

op := &request.Operation{
	Name:       "GetBucketCompliance",
	HTTPMethod: "GET",
	HTTPPath:   "/{Bucket}?compliance",
}

Adding the new definitions of GetBucketComplianceInput and GetBucketComplianceOutput and making the other changes, we'll end up with this final version of the heart of our program:

input := &GetBucketComplianceInput{
	Bucket: aws.String(bucket),
}

op := &request.Operation{
	Name:       "GetBucketCompliance",
	HTTPMethod: "GET",
	HTTPPath:   "/{Bucket}?compliance",
}

output := &GetBucketComplianceOutput{}
req := s3Client.NewRequest(op, input, output)

err = req.Send()
if err != nil {
	fmt.Printf("GetBucketCompliance error: %s\n", err)
	return
}

fmt.Printf("\nGetBucketCompliance status: %v\n", *output.Status)

Will this work 😅? Now we'll find out.

You'll have to take my word (or get a Wasabi account), but if I take the program with these changes, add proper credentials, and run it, it does indeed work.

$ go run main.go
...
2022/11/22 21:29:38 DEBUG: Response s3/GetBucketCompliance Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 136
Content-Type: text/xml
Date: Tue, 22 Nov 2022 15:59:38 GMT
Server: WasabiS3/7.9.1306-2022-11-09-489242991d (head3)
X-Amz-Id-2: ...
X-Amz-Request-Id: ...


-----------------------------------------------------
2022/11/22 21:29:38 <BucketComplianceConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Status>disabled</Status></BucketComplianceConfiguration>

GetBucketCompliance status: disabled

Sweet!

Also, hats off to the AWS engineers for a well designed SDK - we needed to just modify the payloads & change the HTTP path, everything else just worked.

In the end, the solution is simple and is just the obvious set of changes one can expect, but when I was doing this I hadn't been sure until the moment I actually made the final request if it'll work or not. So I hope this post might be useful to someone who finds themselves on the same road - Carry on, it'll work!

Till next time, happy coding 🧑‍💻