Token bucket
Token bucket is a rate limiting algorithm where a virtual bucket accumulates tokens at a fixed refill rate (e.g., 100 tokens/second) up to a maximum capacity (the burst limit). Each request consumes one or more tokens. If the bucket has enough tokens the request proceeds immediately; if not, it is either queued or rejected with a 429. The bucket capacity determines the maximum burst size above the steady-state rate.
Formula
refill_rate = steady-state requests/second. bucket_capacity = maximum burst requests. Burst above refill_rate is allowed until the bucket empties; then requests are limited to refill_rate.Why it matters in practice
Token bucket wins in production API gateways because it splits two controls that fixed-window conflates: the refill rate (steady-state throughput) and the bucket capacity (burst tolerance). Real clients are bursty - quiet, then a flurry of calls - and the bucket absorbs that without raising the long-term average. AWS API Gateway usage plans and Kong's rate-limiting plugin implement it directly; Nginx's `limit_req` is leaky bucket, but `nodelay` makes it behave like token bucket. When clients spike at window open, raise `burst` before `rate` - bumping `rate` also raises sustained load on the backend.
Common mistakes
- •Setting burst capacity equal to rate limit - this effectively disables burst support and forces all clients to spread requests exactly evenly over time, which is unrealistic.
- •Not considering that bucket capacity accumulates during idle periods - a client that is quiet for 60 seconds and then sends 60 seconds worth of burst traffic is within the rules, but may overwhelm a backend not designed for burst.
- •Confusing limit_req burst=10 nodelay in Nginx with a strict rate - nodelay allows the burst to be consumed instantly without queuing.
Related Terms
Leaky bucket
Processes requests at constant output rate, queueing excess.
Burst limit
Maximum requests above steady-state rate in short period.
API throttling vs rate limiting
Rate limiting rejects excess; throttling delays to preserve availability.
Sliding window rate limiting
Maintains rolling count of requests over recent N seconds.