HTTPS Proxying by Example
How to create a HTTPS Proxy server for AWS Services
Admittedly, there is quite a bit of hype around Kubernetes. For a new software engineer, it is impossible to design a production-grade Kubernetes cluster without having a decent understanding of certificates. I have fashioned some code below to illustrate what happens when there is a Agent between you and the https service you are trying to get to. Not only will this showcase how https works at an application level, it will give you ideas on how to use such an Agent for your own applications.
A Short HTTPS Primer
Focus on step “(2)” above. The SSL Server will present a certificate to the client. Let us assume that the SSL Server is Amazon. This certificate should be trusted because the certificate came from a Certificate Authority the client trusts. If there was an Agent between the Client and Amazon, it too must return a certificate. If it returned the Amazon’s certificate, messages form the Client could “never” be decrypted. If we return our own Certificate, it won’t be verified by the Certificate Authority because we aren’t Amazon.
If we control both the client and the Agent, we can certainly establish a trusted, secure connection for requests the Agent will proxy.
The reason this is interesting is when we want to do things like the following:
- Record which AWS APIs are called
- Do things instead of calling an AWS API.
Building the HTTPS Server
We will use the goproxy library. There is an existing example for https hijacking, but it is a bit too complicated for our use case. All we wish to do is something when a particular request is made or response comes back.
Above is the starter code for building an https proxy for Amazon. If you take a look at these lines and do bit a digging, you’ll notice that we decide to send back a default CA Certificate that is bundled with the goproxy package. If we were building a real proxy server, we would likely include our own self-signed certificates.
Running the Server
Try building and running the server above. After it is up, try and perform and AWS API request using the command line
$ HTTP_PROXY=http://localhost:8080 HTTPS_PROXY=https://localhost:8080 aws cloudformation describe-stacks[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
This is happening because the AWS command line tool does not trust the Certificate. It does this by using libraries (ssl.c) that looks for trusted certificates on your computer. If you interested, check out the manual entry for ssl: $ man ssl. I believe “installing” the goproxy CA certificate would remedy this error.
Fortunately, the AWS command line tool has a option to skip SSL verification. This is dangerous but we think we know what we are doing.
HTTP_PROXY=http://localhost:8080 HTTPS_PROXY=https://localhost:8080 aws cloudformation describe-stacks --no-verify-sslDoing Something Interesting
What I will illustrate below is how to modify a returning request from AWS using the AWS SDK.
AWS uses XML for API calls
It took a bit of time to figure out how the Go API works. By adding the debug option to our AWS command line tool, we can inspect requests that are being made by the tool.
HTTP_PROXY=http://localhost:8080 HTTPS_PROXY=https://localhost:8080 aws cloudformation describe-stacks --no-verify-ssl --debugYou should see some XML as a the response to this request. This tells us that we need to unmarshal XML on our proxy server to interact with the data.
AWS Go SDK deep dive
Because we are building out server in Go, we have to see how a client would interact with the aws-sdk-go to interact with AWS. We can find their examples for cloudformation here.
The issue here is that we don’t actually want to make an API request. We will follow their code to where it makes the request and take the code that runs after the response comes back.
We start in /service/cloudformation/api.go. This file is so large that Github won’t render it.
func (c *CloudFormation) DescribeStacks(input *DescribeStacksInput) (*DescribeStacksOutput, error) {
req, out := c.DescribeStacksRequest(input)
err := req.Send()
return out, err
}Somehow, out is being in the Send() call. If we look at c.DescribeStacksRequest we see the following:
func (c *CloudFormation) DescribeStacksRequest(input *DescribeStacksInput) (req *request.Request, output *DescribeStacksOutput) {
op := &request.Operation{
Name: opDescribeStacks,
HTTPMethod: "POST",
HTTPPath: "/",
Paginator: &request.Paginator{
InputTokens: []string{"NextToken"},
OutputTokens: []string{"NextToken"},
LimitToken: "",
TruncationToken: "",
},
}if input == nil {
input = &DescribeStacksInput{}
}output = &DescribeStacksOutput{}
req = c.newRequest(op, input, output)
return
}
This is important because we need the req object in the Send() call.
If we look at c.newRequst, we see this calls c.NewRequst. This is important because we set the handlers for use to unmarshal the XML response.
If we look at Send(), we finally come to the part that sends the data and the part that decodes the data. Because we don’t want to send a request, we will take out the part that decodes the data.
When any of the HandlerLists Run, they need to operate on an http request. This means we have to place our resp *http.Response into our Request before doing an unmarshalling.
Almost There
Unfortunately, this reads the data out of the response’s Body. We are faced with the following error the AWS command line tool:
IncompleteRead(0 bytes read)To finish up, we have to put a new set of the data back into our response object. To finish up the example. I will simply make a copy of the HTTP Response. We can take the unexposed copyHTTPRequest function in AWS’s library. With some modification from this article, we should be good to go!
