Building a proxy server in Golang

Maurício Linhares / @mauriciojr / Technical Lead at DigitalOcean

What is a proxy server?

I mean, seriously

Proxies act as intermediaries between clients and servers, they can perform processing or just forward requests downstream.

They come in many flavours

Database proxies
TCP proxies
HTTP proxies (the ones we'll focus today)

Why bother?

SSL termination
Connection management
Protocol upgrade/downgrade
Security, auditing, load balancing, caching, compression and so many more

HTTP proxies!

They're much more complicated than you think

Correctly handle hop-by-hop headers

Proxy-*
Upgrade
Keep-Alive
Transfer-Encoding

Forward original client headers

X-Forwarded-Proto
X-Forwarded-Host

Streaming?

Transfer-Encoding: chuncked
Large Content-Length values

But be careful with impossible situations

You can't both have a Content-Length and Transfer-Encoding: chuncked at the same time.

Assume the worst

CloudFlare and CloudBleed? Yup, been there, done that.

Respect `Cache-Control` headers

Or try, at least.

Careful with buffers

Do not read data without bounds from request bodies, make sure you're buffering and have clear limits on how much memory or connections you can use.

Be specific on your errors

No chuncked support? Return 411 Length Required
Request does not contain authentication details? 401 Unauthorized
Request contains authentication but creds are invalid? 403 Forbidden

Use a `Via`/`Server` header to define the source of responses

Log everything, log nothing

Some headers are sensitive
Some request bodies are sensitive
Carefully select the information you want to log

Why are the fine folks at DigitalOcean building a proxy?

Microservices!

Our lovely monolith needs to die

In a huge fire

Edge Gateway comes to the rescue

Routing
Authentication
Logging
Rate limiting
Health checking

No batteries included

All communication is over HTTP and headers
Downstream services do not depend on the proxy
Proxy receives service registration requests and directs traffic to them

Why?

No libraries to depend on
No lockstep deployments or circular dependencies

Problems?

HTTP is open ended, parsing bodies is still under service implementors
Services still have to manage their own HTTP servers

What did we look at?

Vulcan
FastHTTP
Go's std HTTP server
Gorilla HTTP stack

go-kit

Gorilla won

Fast enough
Nice integration between router and mux
Websockets implementation was a plus

Why not use a packaged solution?

Most other solutions are hard to customize (Luascript or Nodejs scripting)
Go is the main language being used internally now, so JVM based solutions wouldn't make much sense
Easier to integrate with existing internal infrastructure services

How does it look like?

Filter based design

Request arrives
Is matched against a specific route
Goes through the configured collection of before filters
Is sent to the backend service
Goes through the configured after filters
Delivers response to client

Dead simple

                
type Filter interface {
	Name() string
}

type BeforeFilter interface {
	Filter
	BlacklistedHeaders() []string
	DoBefore(context context.Context) BeforeFilterError
}

type AfterFilter interface {
	Filter
	DoAfter(context context.Context) AfterFilterError
}

Disable URL cleaning

On Gorilla Route.SkipClean(true)
Don't use http.ServeMux

What about the proxy client side?

Limit everything

Go's HTTP client defaults are pretty awful

                client: &http.Client{
    Timeout: clientTimeout,
    Transport: trace.HTTPTransport(&http.Transport{
        Dial:                  dialer.Dial(),
        DialContext:           dialer.DialContext(),
        TLSHandshakeTimeout:   tlsHandshakeTimeout,
        ResponseHeaderTimeout: responseHeaderTimeout,
        ExpectContinueTimeout: time.Second,
        MaxIdleConns:          int(maxIdleConns),
        DisableCompression:    true,
    }),
    CheckRedirect: func(req *http.Request, via []*http.Request) error {
        return http.ErrUseLastResponse
    },
},

Got a response? Close it!

                
response, err := h.client().Do(request)

if response != nil && response.Body != nil {
    defer func() {
        io.Copy(ioutil.Discard, response.Body)
        response.Body.Close()
    }()
}

Health checking?

Don't reuse connections
Set the user-agent header
Timeouts, don't forget the timeouts
Rails app that requires HTTPS? Remember to allow for header overrides
Log the actual error if it fails

Have metrics

How granular? Service level? Route level?
Be careful with URL based metrics in Restful services (/droplets/1)
How do we account for timeouts?

The proxy is always to blame

The proxy is broken!
The service is broken!
I don't know what is broken!

Make sure troubleshooting is easy

Limit connections

Go's standard dialer and HTTP client does not limit connections, you can run out of file handles if you don't limit them.