A Go HTTP server deployed behind a load balancer will eventually leak file descriptors or drop inflight requests during rolling deployments. The standard library makes it trivial to spin up an HTTP listener, but http.ListenAndServe ships with zero timeouts by default, leaving your process vulnerable to stale connections and ungraceful termination.
The Default Pitfall
Most tutorials start with a minimal server implementation.
gopackage main
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "ok")
})
// This blocks indefinitely and uses DefaultServer with no timeouts.
http.ListenAndServe(":8080", nil)
}
When you deploy this to production, two distinct failures happen over time. First, clients that open TCP connections and send partial headers or stall midway through a transfer tie up goroutines and file descriptors permanently. Second, when your orchestrator sends SIGTERM to rotate the container, the process exits instantly, dropping active connections and generating 502 Bad Gateway responses at the load balancer.
Checking open file descriptors on Linux reveals hundreds of idle sockets stuck in ESTABLISHED or CLOSE_WAIT states long after clients vanish:
text# lsof -p <pid> -a -i TCP
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
server 14892 app 3u IPv6 48291 0t0 TCP *:8080 (LISTEN)
server 14892 app 6u IPv6 49102 0t0 TCP app.internal:8080->10.0.1.45:51234 (ESTABLISHED)
server 14892 app 7u IPv6 49103 0t0 TCP app.internal:8080->10.0.1.48:51236 (ESTABLISHED)
server 14892 app 8u IPv6 50211 0t0 TCP app.internal:8080->10.0.2.12:49812 (CLOSE_WAIT)
How the Timeouts Actually Work
net/http.Server exposes several timeout fields. Confusing their boundaries is common.
ReadHeaderTimeout covers the time from when the connection is accepted until the server completely reads the request headers. If a slow client sends one byte every 10 seconds, ReadHeaderTimeout resets or aborts depending on whether headers finish within that duration. This is your primary defense against Slowloris attacks.
ReadTimeout covers the time from connection acceptance until the entire request body is completely read. Setting this too low breaks large payload uploads.
WriteTimeout covers the time from the end of the request header read to the end of the response write. It operates on an absolute wall-clock deadline. If your handler streams data or processes a long database query, WriteTimeout will fire and reset the connection before the handler finishes writing, often logging http: Handler wrote to read-only body or client connection reset errors.
IdleTimeout governs keep-alive connections. It sets the maximum time a connection can sit idle waiting for the next request. If IdleTimeout is zero, Go falls back to the value of ReadTimeout. If both are zero, keep-alive connections live until the client or OS closes them.
Here is the correct baseline configuration for an API server handling standard JSON payloads:
gopackage main
import (
"context"
"errors"
"log"
"net/http"
"os"
"os/signal"
"syscall"
"time"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/api/work", func(w http.ResponseWriter, r *http.Request) {
// Simulate processing
time.Sleep(200 * time.Millisecond)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{"status":"completed"}`))
})
srv := &http.Server{
Addr: ":8080",
Handler: mux,
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 15 * time.Second,
WriteTimeout: 30 * time.Second,
IdleTimeout: 60 * time.Second,
}
// Channel to listen for errors coming from the listener.
serverErrors := make(chan error, 1)
go func() {
log.Printf("server starting on %s", srv.Addr)
serverErrors <- srv.ListenAndServe()
}()
// Channel to listen for OS termination signals.
shutdown := make(chan os.Signal, 1)
signal.Notify(shutdown, os.Interrupt, syscall.SIGTERM)
select {
case err := <-serverErrors:
if !errors.Is(err, http.ErrServerClosed) {
log.Fatalf("server failed to start: %v", err)
}
case sig := <-shutdown:
log.Printf("signal received: %v, starting graceful shutdown", sig)
// Give active requests 25 seconds to complete.
ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Printf("graceful shutdown failed: %v, forcing close", err)
if closeErr := srv.Close(); closeErr != nil {
log.Printf("error closing server: %v", closeErr)
}
}
}
log.Println("server stopped cleanly")
}
How Shutdown Works Under the Hood
When you call srv.Shutdown(ctx), the server does four things in order:
- It immediately closes all underlying listeners to stop accepting new incoming TCP connections.
- It closes all idle keep-alive connections that are waiting between requests.
- It signals all active requests to continue executing until their handlers return.
- It waits until all active requests have finished or until the passed
context.Contextexpires.
If the context expires before the handlers finish, srv.Shutdown returns context.DeadlineExceeded. Falling back to srv.Close() terminates any lingering sockets immediately.
Gotcha 1: WriteTimeout Breaks Large File Uploads and Streaming
If you set WriteTimeout: 10 * time.Second, that deadline is calculated starting immediately after headers are parsed, not when your handler starts writing data.
If a client sends a 200MB file over a slow connection, reading r.Body inside your handler counts against WriteTimeout. If reading the body takes 12 seconds, the connection closes before your handler can even call w.WriteHeader().
The client sees a sudden reset:
textcurl: (56) Recv failure: Connection reset by peer
And your Go logs show:
texthttp: Handler timeout: response write took longer than WriteTimeout
To avoid this while still defending against dead clients, leave WriteTimeout unset or sufficiently large, and wrap specific routes with http.TimeoutHandler, or manage deadlines per-request via http.ResponseController:
gomux.HandleFunc("/api/upload", func(w http.ResponseWriter, r *http.Request) {
rc := http.NewResponseController(w)
// Disable write deadline for this specific route if streaming
if err := rc.SetWriteDeadline(time.Time{}); err != nil {
log.Printf("could not clear write deadline: %v", err)
}
// Read large body
// ...
})
http.ResponseController was added in Go 1.20 and replaces the older, brittle interface assertion tricks against w.(http.Flusher) or hidden deadline setters.
Gotcha 2: The Reverse Proxy Keep-Alive Race
When your Go service sits behind an upstream proxy like Nginx, AWS ALB, or Envoy, mismatched idle timeouts create transient 502 Bad Gateway errors.
Here is the exact sequence:
- The client sends a request. Go handles it and sends the response.
- The proxy keeps the TCP connection to Go open in its connection pool.
- Go reaches its
IdleTimeout(say, 60 seconds) and sends aFINpacket to close the socket. - At that exact millisecond, the proxy receives a new incoming client request and selects that same connection from its pool, sending an HTTP request down the wire.
- Go receives the request on a half-closed socket and responds with an
RST. - The proxy considers the backend dead and returns a 502 to the user.
To prevent this race condition, your Go application’s IdleTimeout must always be slightly higher than the upstream load balancer’s keep-alive timeout.
If an AWS Application Load Balancer has an idle timeout of 60 seconds:
- ALB idle timeout: 60s
- Go
IdleTimeout: 65s
This guarantees the proxy initiates connection tear-downs, not your Go application.
Verifying Timeouts and Shutdown
To test that ReadHeaderTimeout works, run the server and use nc to simulate a Slowloris client that connects, sends a partial line, and sleeps:
bashnc localhost 8080
POST /api/work HTTP/1.1
Host: localhost:8080
User-Agent: slow-test
Do not hit enter twice. Wait 5 seconds. The server will close the connection from its side without leaking the goroutine.
To test graceful shutdown, trigger a long-running request and immediately send SIGTERM to the server:
In terminal 1:
bashgo run main.go
In terminal 2:
bashcurl -i http://localhost:8080/api/work &
sleep 0.05
kill -TERM $(pgrep -f "go run main.go" | tail -n 1)
Terminal 2 will complete cleanly and output:
textHTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 10 Mar 2025 21:04:12 GMT
Content-Length: 22
{"status":"completed"}
And terminal 1 prints:
text2025/03/10 21:04:12 signal received: terminated, starting graceful shutdown
2025/03/10 21:04:12 server stopped cleanly
If you start a third terminal and run curl http://localhost:8080/api/work the instant kill is invoked, the connection will be refused immediately instead of hanging, proving the listener closed while the existing flight finished.