Background Work, Caching & Rate Limiting¶
Work that runs after the response, the cache interface, rate-limit strategies, and runtime instrumentation.
Result-cache backend interface.
A backend stores opaque bytes under a string key with a per-entry TTL in
seconds. The methods are async so a network-backed store does not block the
event loop. cached serialises and deserialises the values, so a backend
only moves bytes.
Bases: Cache
A process-local, size-bounded result cache with per-entry TTL.
Fine for a single process and tests. It does not share state across workers,
so a multi-worker deployment needs a shared backend such as
veloce.contrib.redis.RedisCache. TTLs use a monotonic clock, so a wall-clock
change cannot prematurely expire or extend an entry.
Usage::
from veloce import InMemoryCache
cache = InMemoryCache(max_entries=2048)
cached(cache: Cache, *, ttl: int, key: Callable[..., str] | None = None) -> Callable[[Callable[..., Any]], Callable[..., Any]]
Memoise an async function's JSON-serialisable return in cache.
The key defaults to the function's qualified name plus a digest of its
arguments (arguments that are not JSON-serialisable, such as an injected
Request, are ignored), so a handler is cached by its scalar inputs. Pass
key= a callable taking the same arguments for full control.
The result must be JSON-serialisable (a Pydantic model is dumped in JSON
mode); a non-serialisable result raises TypeError. A cache hit returns
the JSON-decoded value, so cache results you will re-serialise (handler
returns, API payloads) rather than rich objects you need back by type. Only
async functions are supported.
rate_limit(strategy: RateLimitStrategy) -> Callable[[T_handler], T_handler]
Attach a per-route rate-limit strategy to a handler.
A decorated handler is limited by strategy with its own per-client counter,
overriding the RateLimitMiddleware default; undecorated handlers keep the
default. Because the limit lives on the handler, there is no route string to
mistype. Place it below the route decorator so the route registers the tagged
handler.
Usage::
from veloce import TokenBucket, rate_limit
@app.post("/login")
@rate_limit(TokenBucket(rate=5, per=60))
async def login(request): ...
Where per-client rate-limit state lives, and the atomic read-modify-write.
evaluate loads the state for key, runs strategy.evaluate, persists the
next state with its TTL, and returns the decision - all atomically, so two
concurrent requests for the same client cannot both read a stale count.
Subclasses must declare __slots__ (even __slots__ = ()).
Bases: RateLimitBackend
Process-local rate-limit state - the default backend.
Not shared across workers, so a multi-worker deployment enforces roughly
N x the limit; use veloce.contrib.redis.RedisRateLimitBackend for one
shared limit. State is size-bounded to cap memory across many client keys.
dataclass
¶
The outcome of evaluating one request against a strategy.
A rate-limit algorithm as a pure state transition.
evaluate takes the client's prior state (None on the first request or
after expiry) and the current wall-clock time, and returns the decision, the
next state to persist, and a TTL in seconds after which that state can be
dropped. It performs no I/O, so a backend can run it under its own atomicity
(a dict mutation in-process, a watched transaction on Redis).
Subclasses must declare __slots__ (even __slots__ = ()).
Bases: RateLimitStrategy
Allow limit requests per fixed window seconds.
Simple and cheap, but a burst straddling a window boundary can admit up to
2 x limit briefly - use SlidingWindow or TokenBucket when that
matters.
Bases: RateLimitStrategy
Allow limit requests per rolling window seconds.
Weights the previous window's count by how far the current window has
progressed, so the boundary burst FixedWindow allows is smoothed away while
keeping only two counters of state.
Bases: RateLimitStrategy
Refill rate tokens per per seconds, allowing bursts up to burst.
Each request spends one token; an empty bucket rejects until it refills. The
bucket tolerates a short burst (up to burst, default rate) while holding
the long-run average to rate/per. A leaky-bucket-style strict limiter is
TokenBucket(rate, per, burst=1).
Detects event-loop stalls and reports the blocked stack.
A heartbeat callback re-arms itself on the loop every interval
seconds; a separate daemon thread measures how long it has been since
the last heartbeat while the loop is running. When that gap exceeds
stall_threshold something is blocking the loop, and the watchdog
logs a warning with the loop thread's current stack plus a
prescriptive hint (blocking-I/O versus CPU-bound).
Each distinct stall is reported once - the heartbeat counter is frozen for the stall's whole duration, and the watch thread reports a given counter value at most once.
dataclass
¶
A finished HTTP request, as seen by an instrumentation hook.
route is the matched route's path template (/items/{id}), which is
safe to use as a metric label; it is None whenever no route+method
pair matched - both a 404 (no such path) and a 405 (the path
exists but the method is not allowed). Group by (route, status_code)
to keep those apart. path is the concrete request path and is
high-cardinality - prefer route for aggregation.
streamed is True when the response body is a streaming iterator
(StreamingResponse, EventSourceResponse, a large FileResponse).
For those, the hook fires before the body is emitted on the ASGI send
path, so duration_ms and status_code reflect only the time to
produce the response object - not the time to drain the stream, and not
a failure that happens mid-stream. A tracing bridge that needs accurate
end-of-request timing should skip records with streamed set.
end_time_ns is the wall-clock (time.time_ns()) instant the request
finished, captured the moment dispatch returned - before any
instrumentation hook or request_finished receiver runs. A tracing
bridge should anchor its span window to this value (and duration_ms)
rather than reading the clock when its own hook executes, so a slow
earlier hook cannot shift the span past the real request boundary.
error_type is the low-cardinality class name (type(exc).__qualname__)
of the exception that produced a 5xx, set only when an unhandled
raised exception turned into a server error (the debug traceback page,
the generic 500 response, or a propagated exception). It is None for
every other outcome - a 2xx/3xx/4xx, or a 5xx deliberately
returned by a handler/exception handler without a raised exception. A
tracing bridge can record it as the OpenTelemetry error.type attribute
without capturing the full traceback or exception instance, keeping the
record allocation-light. The class name only is carried, never the
message (which may hold attacker-controlled or sensitive text).
instrument_access_log(app: Veloce, *, logger: Logger | None = None, json: bool = False, include_streamed: bool = True) -> Callable[[RequestMetrics], None]
Register the unified access-log hook (text or JSON), route-keyed.
log_requests_as_json(app: Veloce, *, logger: Logger | None = None, level: int = INFO, include_path: bool = False) -> Callable[[RequestMetrics], None]
Register a hook emitting one JSON access-log record per request.