A Go HTTP server deployed behind a load balancer will eventually leak file descriptors or drop inflight requests during rolling deployments. The standard library makes it trivial to spin up an HTTP listener, but http.ListenAndServe ships with zero timeouts by default, leaving your process vulnerable to stale connections and ungraceful termination.

The Default Pitfall

Most tutorials start with a minimal server implementation.

go
package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "ok")
	})

	// This blocks indefinitely and uses DefaultServer with no timeouts.
	http.ListenAndServe(":8080", nil)
}

When you deploy this to production, two distinct failures happen over time. First, clients that open TCP connections and send partial headers or stall midway through a transfer tie up goroutines and file descriptors permanently. Second, when your orchestrator sends SIGTERM to rotate the container, the process exits instantly, dropping active connections and generating 502 Bad Gateway responses at the load balancer.

Checking open file descriptors on Linux reveals hundreds of idle sockets stuck in ESTABLISHED or CLOSE_WAIT states long after clients vanish:

text
# lsof -p <pid> -a -i TCP
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
server  14892  app    3u  IPv6  48291      0t0  TCP *:8080 (LISTEN)
server  14892  app    6u  IPv6  49102      0t0  TCP app.internal:8080->10.0.1.45:51234 (ESTABLISHED)
server  14892  app    7u  IPv6  49103      0t0  TCP app.internal:8080->10.0.1.48:51236 (ESTABLISHED)
server  14892  app    8u  IPv6  50211      0t0  TCP app.internal:8080->10.0.2.12:49812 (CLOSE_WAIT)

How the Timeouts Actually Work

net/http.Server exposes several timeout fields. Confusing their boundaries is common.

ReadHeaderTimeout covers the time from when the connection is accepted until the server completely reads the request headers. If a slow client sends one byte every 10 seconds, ReadHeaderTimeout resets or aborts depending on whether headers finish within that duration. This is your primary defense against Slowloris attacks.

ReadTimeout covers the time from connection acceptance until the entire request body is completely read. Setting this too low breaks large payload uploads.

WriteTimeout covers the time from the end of the request header read to the end of the response write. It operates on an absolute wall-clock deadline. If your handler streams data or processes a long database query, WriteTimeout will fire and reset the connection before the handler finishes writing, often logging http: Handler wrote to read-only body or client connection reset errors.

IdleTimeout governs keep-alive connections. It sets the maximum time a connection can sit idle waiting for the next request. If IdleTimeout is zero, Go falls back to the value of ReadTimeout. If both are zero, keep-alive connections live until the client or OS closes them.

Here is the correct baseline configuration for an API server handling standard JSON payloads:

go
package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/api/work", func(w http.ResponseWriter, r *http.Request) {
		// Simulate processing
		time.Sleep(200 * time.Millisecond)
		w.Header().Set("Content-Type", "application/json")
		w.WriteHeader(http.StatusOK)
		w.Write([]byte(`{"status":"completed"}`))
	})

	srv := &http.Server{
		Addr:              ":8080",
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
		ReadTimeout:       15 * time.Second,
		WriteTimeout:      30 * time.Second,
		IdleTimeout:       60 * time.Second,
	}

	// Channel to listen for errors coming from the listener.
	serverErrors := make(chan error, 1)

	go func() {
		log.Printf("server starting on %s", srv.Addr)
		serverErrors <- srv.ListenAndServe()
	}()

	// Channel to listen for OS termination signals.
	shutdown := make(chan os.Signal, 1)
	signal.Notify(shutdown, os.Interrupt, syscall.SIGTERM)

	select {
	case err := <-serverErrors:
		if !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("server failed to start: %v", err)
		}
	case sig := <-shutdown:
		log.Printf("signal received: %v, starting graceful shutdown", sig)

		// Give active requests 25 seconds to complete.
		ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
		defer cancel()

		if err := srv.Shutdown(ctx); err != nil {
			log.Printf("graceful shutdown failed: %v, forcing close", err)
			if closeErr := srv.Close(); closeErr != nil {
				log.Printf("error closing server: %v", closeErr)
			}
		}
	}
	log.Println("server stopped cleanly")
}

How Shutdown Works Under the Hood

When you call srv.Shutdown(ctx), the server does four things in order:

  1. It immediately closes all underlying listeners to stop accepting new incoming TCP connections.
  2. It closes all idle keep-alive connections that are waiting between requests.
  3. It signals all active requests to continue executing until their handlers return.
  4. It waits until all active requests have finished or until the passed context.Context expires.

If the context expires before the handlers finish, srv.Shutdown returns context.DeadlineExceeded. Falling back to srv.Close() terminates any lingering sockets immediately.

Gotcha 1: WriteTimeout Breaks Large File Uploads and Streaming

If you set WriteTimeout: 10 * time.Second, that deadline is calculated starting immediately after headers are parsed, not when your handler starts writing data.

If a client sends a 200MB file over a slow connection, reading r.Body inside your handler counts against WriteTimeout. If reading the body takes 12 seconds, the connection closes before your handler can even call w.WriteHeader().

The client sees a sudden reset:

text
curl: (56) Recv failure: Connection reset by peer

And your Go logs show:

text
http: Handler timeout: response write took longer than WriteTimeout

To avoid this while still defending against dead clients, leave WriteTimeout unset or sufficiently large, and wrap specific routes with http.TimeoutHandler, or manage deadlines per-request via http.ResponseController:

go
mux.HandleFunc("/api/upload", func(w http.ResponseWriter, r *http.Request) {
	rc := http.NewResponseController(w)

	// Disable write deadline for this specific route if streaming
	if err := rc.SetWriteDeadline(time.Time{}); err != nil {
		log.Printf("could not clear write deadline: %v", err)
	}

	// Read large body
	// ...
})

http.ResponseController was added in Go 1.20 and replaces the older, brittle interface assertion tricks against w.(http.Flusher) or hidden deadline setters.

Gotcha 2: The Reverse Proxy Keep-Alive Race

When your Go service sits behind an upstream proxy like Nginx, AWS ALB, or Envoy, mismatched idle timeouts create transient 502 Bad Gateway errors.

Here is the exact sequence:

  1. The client sends a request. Go handles it and sends the response.
  2. The proxy keeps the TCP connection to Go open in its connection pool.
  3. Go reaches its IdleTimeout (say, 60 seconds) and sends a FIN packet to close the socket.
  4. At that exact millisecond, the proxy receives a new incoming client request and selects that same connection from its pool, sending an HTTP request down the wire.
  5. Go receives the request on a half-closed socket and responds with an RST.
  6. The proxy considers the backend dead and returns a 502 to the user.

To prevent this race condition, your Go application’s IdleTimeout must always be slightly higher than the upstream load balancer’s keep-alive timeout.

If an AWS Application Load Balancer has an idle timeout of 60 seconds:

  • ALB idle timeout: 60s
  • Go IdleTimeout: 65s

This guarantees the proxy initiates connection tear-downs, not your Go application.

Verifying Timeouts and Shutdown

To test that ReadHeaderTimeout works, run the server and use nc to simulate a Slowloris client that connects, sends a partial line, and sleeps:

bash
nc localhost 8080
POST /api/work HTTP/1.1
Host: localhost:8080
User-Agent: slow-test

Do not hit enter twice. Wait 5 seconds. The server will close the connection from its side without leaking the goroutine.

To test graceful shutdown, trigger a long-running request and immediately send SIGTERM to the server:

In terminal 1:

bash
go run main.go

In terminal 2:

bash
curl -i http://localhost:8080/api/work &
sleep 0.05
kill -TERM $(pgrep -f "go run main.go" | tail -n 1)

Terminal 2 will complete cleanly and output:

text
HTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 10 Mar 2025 21:04:12 GMT
Content-Length: 22

{"status":"completed"}

And terminal 1 prints:

text
2025/03/10 21:04:12 signal received: terminated, starting graceful shutdown
2025/03/10 21:04:12 server stopped cleanly

If you start a third terminal and run curl http://localhost:8080/api/work the instant kill is invoked, the connection will be refused immediately instead of hanging, proving the listener closed while the existing flight finished.