Python SDK · BufferedRecorder

The default recorder buffers events in memory and flushes in batches on a background thread. Because record() only enqueues, it's safe to call from async def handlers without blocking the event loop:

rec = es.new_recorder(buffer_size=2000, flush_interval=2.0)
# or
rec = recorder.new(project_id, api_key, buffer_size=2000)

Options

Option	Default	Purpose
`buffer_size`	`1000`	Capacity of the in-memory event buffer.
`flush_size`	`100`	Pending-event count that triggers an immediate flush, and the most events per send.
`flush_interval`	`5.0`	Max seconds between flushes when the size threshold isn't reached.
`flush_timeout`	`30.0`	Socket timeout (seconds) for each delivery attempt. It bounds each connect and read, not the whole attempt, and doesn't cover the DNS lookup.
`overflow`	`"drop-newest"`	Behavior when the buffer is full: `"drop-newest"`, `"block"`, or `"error"`.
`drain_timeout`	`30.0`	Max seconds `close()` waits for pending events, plus up to 1 s after it stops retrying. `0` or less means it doesn't wait.
`initial_backoff`	`0.5`	Base wait (seconds) before the first retry of a failed batch.
`max_backoff`	`30.0`	Cap (seconds) on a single wait between retries.
`max_retry_duration`	`300.0`	How long (seconds) one batch is retried before its events are dropped as `RETRIES_EXHAUSTED`.
`on_drop`	`None`	Callback invoked with the events, a `DropReason` and an error whenever events are discarded. See What happens when delivery fails.
`request_timeout`	`10.0`	Per-request HTTP timeout (seconds) when the HTTP recorder is used directly.
`logger`	module logger	A `logging.Logger` for diagnostics (overflow warnings, flush errors).
`base_url`	production	Override the ingestion endpoint (tests, staging).
`transport`	`urllib`	Custom transport callable (primarily for tests).
`auto_idempotency_key`	off	Copy `event.id` into `event.idempotency_key` when the latter is empty.

Times are in seconds (Python convention), where the Node SDK uses milliseconds. overflow accepts either the OverflowPolicy enum or the equivalent string. flush_size, flush_interval and the backoff and retry options treat None, <= 0 or NaN as the default.

Overflow policies

"drop-newest"  # drop the incoming event, log a warning (default)
"block"        # wait for space (until the buffer drains or the recorder closes)
"error"        # raise BufferFullError

Under an async server keep the default "drop-newest". The ASGI middleware records the request event on the event loop, so a "block" wait there stalls every request for up to 15 s.

Close & flush

close() drains pending events, with retries, for up to drain_timeout. It raises DrainTimeoutError if the drain timed out, so log what it raises. Call it once, on the path that actually runs at shutdown.
flush(timeout=None) returns once the events queued at call time are delivered or reported to on_drop. Useful in tests and for graceful shutdown sequences. Always pass a timeout: retries during an outage can take minutes. It raises TimeoutError if the timeout elapses first, and never cancels a send.
stats() returns a BufferedStats with dropped (only BUFFER_FULL drops), dropped_by_reason, flushed, flush_errs, retries, pending and buffer_size for observability.

What happens when delivery fails

record() only puts the event on an in-memory buffer, so your request path never waits on the network. A background thread sends batches and handles failures:

Failure	What happens
`NetworkError`, `TimeoutError`, `429` or `5xx`	Retried with jittered exponential backoff: 0.5 s first wait, 30 s max wait, server `Retry-After` honored up to that cap. After 5 minutes the batch is dropped as `RETRIES_EXHAUSTED`. Tune with `initial_backoff`, `max_backoff` and `max_retry_duration`.
`400`, `409`, `413` or `EncodeError`	The batch is split in half and each half retried, so one bad event does not sink the rest. An event that still fails is dropped as `REJECTED`.
Any other error	The whole batch is dropped as `REJECTED`.
Buffer full	Depends on the overflow policy. `"drop-newest"` (default) drops the new event as `BUFFER_FULL`, `"block"` makes `record()` wait for space, `"error"` raises `BufferFullError`.
`close()` drain timeout	Retries stop and what is left is dropped as `SHUTDOWN`.
`record()` after `close()`	The event is dropped as `CLOSED`, as is a `"block"` wait that `close()` ends.

Pass on_drop to hear about every dropped event:

import logging

log = logging.getLogger("audit")

def on_drop(events, reason, err):
    log.error("everscribe: dropped %d events (%s): %s", len(events), reason.value, err)

rec = es.new_recorder(on_drop=on_drop)

reason is a DropReason: BUFFER_FULL, REJECTED, RETRIES_EXHAUSTED, SHUTDOWN or CLOSED. reason.value gives the lowercase name for metric labels.
err is the last delivery error for REJECTED and RETRIES_EXHAUSTED. For a BUFFER_FULL drop that happens when the middleware records the request event, it is a BufferFullError under "error" and a TimeoutError (the 15 s wait) under "block". Otherwise it is None.
The callback must not block and must not call record, flush or close on the same recorder. It may run on several threads at once. An Exception it raises is caught and logged; a BaseException raised on the background thread stops it, can leave the rest of that batch unreported, and later records are dropped as CLOSED.
A dropped event may have been delivered anyway, for example when the last request timed out. Resending it with its original id is safe: the server deduplicates it.
Delivered events are immutable. Dropped events cannot be recovered, so route on_drop to your own logs or queue.

Always call close() before the process exits. It drains the buffer, with retries, for up to drain_timeout (30 s by default). At the timeout it stops retrying, waits up to 1 s more, and raises DrainTimeoutError. The background thread is a daemon thread, so a process that exits without close() loses what is buffered without an on_drop report.

Give flush a deadline. flush(timeout=...) returns once the events queued at the call are delivered or reported to on_drop, or raises TimeoutError if the timeout elapses first. It does not raise delivery errors and never cancels a send.

Watch stats(). dropped_by_reason counts drops per reason, retries counts retries, flush_errs counts failed attempts, and dropped counts only BUFFER_FULL drops.

For the split budget, custom recorders and middleware edge cases, see Delivery details.

Delivery details

Splitting is limited to 32 sends per batch. Events left when that budget runs out are dropped as REJECTED.
An event json.dumps cannot encode (a set or a datetime in metadata, for example), or one with a wrongly typed field (a string occurred_at, for example), fails with EncodeError and is handled like a 400: that event is dropped as REJECTED and the rest of the batch is delivered. NaN and infinity are written as-is and the server rejects them with a 400, so the outcome is the same.
record() stores a shallow copy of the event, and a retry can keep it queued for minutes. Don't mutate an event's actor, target or metadata after recording it.
BUFFER_FULL and CLOSED reach on_drop on the thread that called record(). The other reasons arrive on the background thread or the thread calling close().
A custom inner recorder decides what is retried through its errors: NetworkError, TimeoutError, and anything with a truthy transient attribute or transient() method are retried. Anything else, after any split, drops the batch as REJECTED. record_batch gets timeout=flush_timeout if it accepts a timeout keyword. An inner recorder without record_batch gets one record() call per event, and a failure that isn't retried drops only that event. A BaseException such as SystemExit from the inner recorder drops that send as REJECTED and the background thread keeps running.
A running inner call can't be interrupted. If one is still running after drain_timeout plus the 1 s grace, close() reports what is still queued as SHUTDOWN itself and raises DrainTimeoutError. The in-flight batch is reported as SHUTDOWN when the call returns, unless it succeeded, which can be after close() has returned.
Calling close() again, or concurrently, waits for the first call and gets the same outcome. A close() interrupted by KeyboardInterrupt can be called again. flush() after close() returns at once, as does a flush() once the background thread has stopped, without delivering anything.

Drops of the middleware's event

The middleware records the request event when the handler finishes, and has no caller to return an error to. If the buffer is full then, the event is reported to on_drop as BUFFER_FULL. Under "error" the report carries the BufferFullError. Under "block" the middleware waits up to 15 s for space, then gives up and the report carries a TimeoutError.

A custom Recorder that reports its own drops can call everscribe.event.is_auto_recording(e) to tell this case apart.

Where `close` goes

The recorder is constructed once at import time and lives as long as the process, so nothing in request handling ever closes it. Hook the framework's shutdown. For FastAPI or Starlette that is the lifespan context manager:

import logging
from contextlib import asynccontextmanager

from everscribe.recorder import DrainTimeoutError
from fastapi import FastAPI

@asynccontextmanager
async def lifespan(app: FastAPI):
    yield  # the app serves for the duration of the yield
    try:
        rec.close()  # runs, and drains
    except DrainTimeoutError as err:
        logging.error("everscribe: drain incomplete: %s", err)

app = FastAPI(lifespan=lifespan)

close() is synchronous: it joins the background thread and blocks for up to drain_timeout plus 1 s. That is fine at shutdown, when nothing else is being served. Use await asyncio.to_thread(rec.close) if you would rather keep the event loop free while it drains.

Flask, Django, and any other WSGI app use their process manager's equivalent hook (Gunicorn's worker_exit, for instance). What matters is that the hook runs when the worker is sent SIGTERM, not only on a clean interpreter exit.