ferryman-edge
An mTLS-terminating L7 reverse proxy in Rust. Every request passes a TLS
client-certificate check, an RS256 JWT check, and a per-tenant rate limit
before it is routed to an upstream behind a per-upstream circuit breaker.
TLS material and routes hot-reload on SIGUSR1 without dropping
connections.
cargo install ferryman-edge # installs the `ferryman-edge-server` binary
- Overview — the problem, architecture, and stack.
- Request pipeline & design — gates, status codes, header handling, breaker semantics, design tradeoffs, benchmarks.
- Operations — configuration reference, reload, shutdown, metrics, troubleshooting, container.
- API reference — the two crates on docs.rs.
- Releasing — how versions are cut and published.
- Changelog
Source: github.com/Bunty9/ferryman-edge. Licensed under MIT or Apache-2.0.
Overview
The problem
Real edge proxies do mTLS termination, JWT validation in-line,
per-route rate limiting, and cert hot-reload — and they make
defensible decisions on body buffering, HTTP/2 stream control, and
connection pooling. ferryman-edge is the project that gets the
Cloudflare Pingora team to reply.
Architecture (delta on top of P2)
Client (with mTLS cert)
|
v
+--------+---------+
| rustls TLS+mTLS | cert reload via SIGUSR1
| + JWT validate | (no connection drops)
+--------+---------+
|
v
+--------+---------+
| per-tenant rate | governor crate, keyed by JWT.sub
| limit (per route)|
+--------+---------+
|
v
+--------+---------+
| RouteService |
| (P2's table) |
+--------+---------+
|
v
+--------+---------+
| hyper client | pool: 1 conn per upstream * N
| HTTP/2 multiplex | feature flag: boxed_body vs collected
+--------+---------+
|
v
upstream svc
Stack
| Layer | Crate / Tool |
|---|---|
| Async runtime | tokio 1.47 (full) |
| HTTP server | hyper 1.5 + hyper-util + tower-http |
| TLS / mTLS | rustls 0.23 (aws-lc-rs provider) + tokio-rustls 0.26 + rustls-pemfile 2 + rustls-pki-types |
| AuthN | jsonwebtoken 9 + moka 0.12 (future cache, 10k × 5min) |
| Rate limit | governor 0.7 (keyed GCRA) |
| Config / hot-swap | serde + toml 0.8 + arc-swap; reload via SIGUSR1 |
| Observability | tracing + metrics-exporter-prometheus 0.16 |
| CLI | clap 4 |
| Container build | cargo-chef multi-stage; distroless final (NOT scratch — aws-lc-rs needs libc) |
| Deploy | Fly.io 2-region (sin + iad) |
| CI | GHA (stable + beta) + cargo-deny + cargo-nextest + criterion (non-blocking) + mTLS smoke |
Pinned versions live in Cargo.toml.
Request pipeline & design
Request pipeline
Every request passes the same gates, in order:
| Gate | Reject with | Metric |
|---|---|---|
TLS handshake, client cert must chain to client_ca_path (10 s timeout) | connection closed | ferryman_tls_handshake_failures_total, ferryman_tls_handshake_seconds |
Authorization: Bearer <RS256 JWT>: exp (also on cache hits), nbf, and iss/aud when configured | 401 + www-authenticate: Bearer | ferryman_auth_failures_total{reason} |
Per-tenant GCRA limit keyed by sub (tenant_rps, 0 disables) | 429 + retry-after: 1 | ferryman_ratelimited_total |
No . / .. path segments (incl. %2e) | 400 | ferryman_requests_total{status} |
| Longest-prefix route on a path-segment boundary; no fall-through to a shorter prefix | 404 no route, 503 breaker open | |
| Body ≤ 8 MiB | 413 | |
| Client body read within 30 s (collected mode; read before route lookup) | 408 slow client, 400 body error | |
| Upstream round trip within 30 s, counted from when the body is ready (plus the response body in collected mode) | 502 transport/response-body error, 504 timeout | ferryman_request_duration_seconds{upstream} (success path) |
On the way through, the proxy strips hop-by-hop headers (both directions,
including any named in Connection), then stamps x-ferryman-tenant: <sub>
(any client-supplied value is dropped first). It rewrites Host to the
upstream, replaces x-forwarded-for with the peer IP (dropping client-sent
Forwarded / X-Real-IP), sets x-forwarded-proto: https, and downgrades
the outbound request to HTTP/1.1. Inbound protocol is pinned from ALPN
(h2 or http/1.1). A connection with no request within 10 s of the
handshake is closed (this also covers a stalled h2 preface); h2 connections
then get keep-alive pings and a 64-stream cap.
Each upstream has a Closed / Open / HalfOpen circuit breaker
(ferryman_circuit_state{upstream}: 0/1/2; cooldown_secs must be ≥ 1).
A transport error, a 502–504, or a timeout opens it — under boxed_body a
timeout only counts if the client had finished uploading; after cooldown_secs exactly one request is let through
as the probe. A plain 500 does not trip it, and neither does a failure
caused by the client’s own body (size cap, disconnect). The active health
checker (GET <upstream>/health every health_interval_secs) opens and
closes it too. A route reload keeps breaker state for rules whose prefix,
upstream, and cooldown are unchanged.
SIGTERM / SIGINT stop accepting and drain in-flight connections for up to
25 s. SIGUSR1 reloads TLS material and the routing table; the JWT settings
and tenant_rps are read once at boot.
Set [jwt] issuer and audience for anything beyond local dev — without
them, any token signed by the issuer key is accepted, whichever service it
was minted for.
Design tradeoffs
P4 makes three decisions worth defending in a hiring loop.
(a) boxed_body is off by default
Cargo feature boxed_body swaps the upstream client to
http_body_util::BoxBody and streams request and response bodies. With it
off, the proxy collects each body once into a Full<Bytes> before
forwarding. Both builds enforce the 8 MiB request cap; under streaming, a
chunked upload with no Content-Length that exceeds it is cut mid-stream
and answered 413, without counting against the upstream’s breaker.
Streaming mode has no separate body-read deadline: a slow upload runs
inside the upstream’s 30 s budget and ends as 504.
Estimates from the design spec (not yet measured here): boxed adds ~200 µs per request at 10 MB; collected
adds ~80 µs at 1 KB but allocates ~req_size. For an internal proxy
fronting JSON APIs under ~256 KB the collected path wins on code
complexity, allocator pressure (because the JSON allocator already paid
the cost), and steady-state latency. Pingora picks streaming for
general-purpose CDN traffic where payloads skew big and bimodal; that
calculus inverts for an internal API edge. Flip the feature on (cargo build --features boxed_body) when your p99 latency tells you the
collect-first cost dominates.
(b) SIGUSR1 reload over filesystem-watch
P2 (ferryman) reloads its routing table via notify filesystem events.
P4 deliberately swaps to SIGUSR1 because:
- k8s mounts ConfigMaps via a symlink-swap dance.
notifyreports this as a chain of remove + create events on the symlink target, not the watched path. Without per-platform special-casing the watcher silently misses the reload — a worst-case failure mode for a security-sensitive hot-swap of TLS material. - Editors emit a parade of
Modifyevents for in-place writes that have no business triggering a reload (cursor moves, autosave drafts). SIGUSR1is one POSIX call with predictable semantics across every deploy target. The operator runskill -USR1 $(pidof ferryman-edge-server)afterkubectl rollout restartof the ConfigMap, or wires it into cert-manager’s renewal hook.
Both cert reload (tls::ReloadingTls) and route reload
(server/src/reload.rs) share the same signal — one trigger swaps both
surfaces atomically from the operator’s perspective.
(c) rustls + aws-lc-rs over OpenSSL
- Pure-Rust audit story. rustls is the only TLS stack with a clean
memory-safety argument all the way to the cipher implementations
(
aws-lc-rsis the AWS-libcrypto Rust binding; ring is the historical alternative). For an edge proxy that terminates customer-data TLS, that argument matters more than the C/Go ecosystem’s parity. - FIPS path.
aws-lc-rshas a FIPS-mode build via the same crate. No swap-out at deploy time, no separate provider — flip a feature flag and recompile. OpenSSL FIPS 3.0 modules ship, but the build process is brittle and OS-distribution-specific. - Cost. Distroless final image instead of scratch (
aws-lc-rsneeds libc + dynamic loader). ~12 MB extra over a musl/scratch build. Worth it for the audit + FIPS leverage.
Benchmarks
# JWT verify: cache hit vs miss (criterion).
cargo bench -p ferryman-edge-core --bench jwt_verify
# Zero-loss reload check: curl workers (fresh mTLS handshake per request)
# while SIGUSR1 fires at 50% and 75% of the run. Needs the server up and
# an upstream answering /svc-a/echo.
./benches/reload.sh 60 8
wrk/wrk2 cannot present a TLS client certificate, so benches/wrk2.lua
only works against a listener without mTLS; the 50k rps target below needs
an mTLS-capable load generator and is not measured yet.
| Metric | Target | Measured |
|---|---|---|
| JWT verify, cache hit vs miss | ≥ 10× | 0.68 µs vs 150 µs (~220×), criterion, dev laptop |
| Hot reload under load | zero failed reqs | 3725 / 3725 OK across 2× SIGUSR1 (60 s, 8 workers, release) |
| Throughput @ mTLS + JWT | 50,000 rps | — |
| p99 latency | < 8 ms | — |
| TLS handshake p99 (full chain validation) | < 50 ms | 119 ms, but client and server shared one box (contended) |
Operating ferryman-edge
How to configure, run, reload, observe and debug the proxy. For the request pipeline and design rationale see the README.
Configuration reference
config.toml (path via --config / FERRYMAN_EDGE_CONFIG). Relative paths
are resolved against the process working directory.
| Key | Default | Reloaded on SIGUSR1 | Meaning |
|---|---|---|---|
health_interval_secs | 5 | no | Active health probe interval (GET <upstream>/health). |
default_cooldown_secs | 30 | yes (routes) | Breaker cooldown for routes that don’t set one. |
tenant_rps | 1000 | no | Per-tenant GCRA limit keyed by JWT sub. 0 disables rate limiting. |
[tls] cert_path | — | yes | Server certificate chain, PEM (leaf first, then intermediates). |
[tls] key_path | — | yes | Server private key, PEM (PKCS#8, PKCS#1 or SEC1). |
[tls] client_ca_path | — | yes | Client CA bundle, PEM. Every client cert must chain to one of these. |
[jwt] jwks_path | — | no | RSA public key, PEM, used for RS256 verification. |
[jwt] issuer | unset | no | Required iss. Unset = not checked. |
[jwt] audience | unset | no | Required aud. Unset = not checked; tokens carrying any aud are then rejected. |
[[routes]] prefix | — | yes | Path prefix, matched on a segment boundary. Longest prefix wins. |
[[routes]] upstream | — | yes | http://host:port of the backend. Must have an authority. |
[[routes]] cooldown_secs | default_cooldown_secs | yes | Per-route breaker cooldown, must be ≥ 1. |
Set issuer and audience in every non-local deployment. Without them the
proxy accepts any token signed by the issuer key, whichever service it was
minted for.
CLI / environment:
| Flag | Env | Default |
|---|---|---|
--config | FERRYMAN_EDGE_CONFIG | config.toml |
--bind | FERRYMAN_EDGE_BIND | 0.0.0.0:8443 |
--metrics-bind | FERRYMAN_EDGE_METRICS_BIND | 0.0.0.0:9090 |
| — | RUST_LOG | info (JSON logs to stdout) |
Fixed limits (constants in crates/server/src): 8 MiB request body, 30 s
client-body read, 30 s upstream round trip, 10 s TLS handshake, 10 s to the
first request on a new connection, 64 concurrent h2 streams per connection,
25 s shutdown drain.
Local run
./scripts/gen-test-certs.sh # certs/ (gitignored)
cargo run -p ferryman-edge -- --config config.toml
curl --cacert certs/ca.crt --cert certs/client.crt --key certs/client.key \
-H "Authorization: Bearer $(scripts/mint-jwt.sh tenant-a 3600)" \
https://localhost:8443/svc-a/hello
scripts/mint-jwt.sh [sub] [ttl_secs] [scope] signs with
certs/jwt-priv.pem; set JWT_ISS / JWT_AUD to add iss / aud.
The upstreams in config.toml (localhost:8001, localhost:8002) must
serve GET /health with a 2xx, or the health checker keeps their breaker
open and requests get 503. The proxy forwards the full path, prefix
included (/svc-a/hello reaches the upstream as /svc-a/hello).
Hot reload
kill -USR1 "$(pidof ferryman-edge-server)"
One signal reloads both the TLS material and the routing table. A reload
that fails to parse or load keeps the old config and logs
mTLS reload failed; keeping old or route reload failed; keeping old table with the cause. Live connections keep the TLS config they
handshook with; new connections get the new one. Breaker state carries
over for routes whose prefix, upstream and cooldown did not change.
Not reloaded: the JWT key and claims settings, tenant_rps,
health_interval_secs, bind addresses. Restart for those.
Use pidof, not pgrep -x: the binary name is longer than the 15-char
kernel comm, so pgrep -x ferryman-edge-server never matches.
Shutdown
SIGTERM or SIGINT stops accepting, lets in-flight connections finish for up
to 25 s, then exits. fly.toml sends SIGINT with a 30 s kill timeout.
Metrics
Prometheus text on --metrics-bind at /metrics. Keep it off the public
network (fly.toml uses Fly’s internal [metrics] scrape).
| Metric | Type | Labels |
|---|---|---|
ferryman_requests_total | counter | status, upstream (when one was chosen) |
ferryman_request_duration_seconds | summary | upstream (successful round trips) |
ferryman_auth_failures_total | counter | reason = missing | invalid |
ferryman_ratelimited_total | counter | — (never labelled by tenant) |
ferryman_tls_handshake_seconds | summary | — |
ferryman_tls_handshake_failures_total | counter | — |
ferryman_circuit_state | gauge | upstream; 0 closed, 1 open, 2 half-open |
ferryman_upstream_alive | gauge | upstream; last health probe result |
upstream is host:port.
Troubleshooting
| Symptom | Likely cause |
|---|---|
503 upstream unavailable right after boot | First health probe ran before the upstream was up; the breaker closes on the next successful probe (≤ health_interval_secs). |
503 persists | Upstream has no 2xx /health, or keeps failing. Check ferryman_upstream_alive. |
404 no route | No prefix matches on a segment boundary (/svc-a does not match /svc-abc). |
400 bad path | Path has a . or .. segment (also %2e). |
401 with a token you believe is valid | Expired (60 s leeway), nbf in the future, wrong key, or iss/aud mismatch. ferryman_auth_failures_total{reason="invalid"} counts these. |
| curl exits 56 / handshake failure | No client cert, or it doesn’t chain to client_ca_path. ferryman_tls_handshake_failures_total counts these. |
Python client: CA cert does not include key usage extension | Root CA generated without extensions; regenerate with the current gen-test-certs.sh. |
Container exits with GLIBC_2.38 not found | Builder and runtime images on different Debian releases; the Dockerfile pins both to bookworm. |
Container
docker build -t ferryman-edge .
docker run -p 8443:8443 -v "$PWD/certs:/app/certs:ro" ferryman-edge
The image ships no key material and no certs; mount them at /app/certs
(the paths in the baked-in /app/config.toml) or the server exits at boot.
Examples
| Example | Read it if you want to… |
|---|---|
edge-demo | run the ferryman-edge proxy in front of your services: PKI, tokens, config, upstreams, and every feature exercised end to end |
embed-core | add mTLS, JWT auth and per-tenant rate limiting to your own axum service with ferryman-edge-core |
Both are workspace members (publish = false), so CI builds, lints and runs them.
edge-demo: the ferryman-edge proxy in action
A runnable reference for the operator path: the real ferryman-edge-server
binary in front of sample upstream services, with every feature exercised and
checked. Unix only (the proxy is controlled with SIGUSR1 / SIGTERM).
The example crate is ferryman-edge-demo, with two binaries:
backend: a sample upstream. It shows how a service consumes thex-ferryman-tenantheader the proxy stamps.edge-demo: a driver with three subcommands:setup(PKI + config),token(mint a JWT) andrun(the narrated end-to-end walkthrough).
What you will see
edge-demo run starts three backends and the proxy on free ports, then walks
through eleven scenarios, printing ✓ / ✗ per check and exiting non-zero if
any fails (about 5 seconds):
- mTLS: valid cert over HTTP/2 and HTTP/1.1 works; no cert and a cert from another CA are refused.
- JWT: missing, garbage, expired, wrong
aud, wrongiss, futurenbfand wrongly-signed tokens are all 401; a valid one is 200. - Identity propagation: the backend sees
x-ferryman-tenantequal to the JWTsub(client-supplied values are discarded), the peer IP inx-forwarded-for, andx-forwarded-proto: https. - Routing: longest prefix on a path-segment boundary;
..is rejected. - Bodies: 6 MiB passes intact, 9 MiB is 413.
- Rate limiting: per tenant, 6th immediate request is 429 with
retry-after. - Circuit breaker + health checks: kill a backend, watch 503 and the
ferryman_circuit_stategauge, restart it, watch recovery. - Hot reload (routes): add a route,
SIGUSR1, it is live. - Hot reload (certificate): rotate the server cert,
SIGUSR1, new handshakes present it; existing clients keep working. - Metrics: Prometheus text on a separate port.
- Graceful shutdown:
SIGTERMlets an in-flight request finish.
Run it
From the repository root:
examples/edge-demo/run.sh
which is cargo build -p ferryman-edge -p ferryman-edge-demo followed by
target/debug/edge-demo run. Note that both packages must be built:
cargo run -p ferryman-edge-demo alone builds neither the proxy nor backend
(and the demo says so if it cannot find them). run.sh passes its arguments to
cargo build, so ./run.sh --release works.
run keeps its keys, config and logs in examples/edge-demo/.demo/run/
(gitignored, never commit it), separate from the setup material in
.demo/. Look at .demo/run/proxy.log for the proxy’s JSON logs. edge-demo run --keep leaves the stack running until Ctrl-C.
Do it by hand
Build, then generate the PKI and config for fixed local ports (proxy 8443, metrics 9090, backends 9101 to 9103):
cargo build -p ferryman-edge -p ferryman-edge-demo
target/debug/edge-demo setup
Start the backends and the proxy (setup prints equivalent commands with your
absolute paths). Backends first; they must only be reachable from the proxy.
The commands keep the process IDs so nothing else on your machine is touched:
target/debug/backend --name orders --bind 127.0.0.1:9101 & ORDERS=$!
target/debug/backend --name inventory --bind 127.0.0.1:9102 & INVENTORY=$!
target/debug/backend --name payments --bind 127.0.0.1:9103 & PAYMENTS=$!
target/debug/ferryman-edge-server \
--config examples/edge-demo/.demo/ferryman.toml \
--bind 127.0.0.1:8443 --metrics-bind 127.0.0.1:9090 & PROXY=$!
Set up a shorthand for a mutual-TLS request, then one request per feature. The rate limit is 5 requests per second per tenant, so run these at human speed:
D=examples/edge-demo/.demo
P=https://127.0.0.1:8443
TOKEN=$(target/debug/edge-demo token --sub acme)
AUTH="Authorization: Bearer $TOKEN"
edge() { curl -sS --cacert $D/ca.crt --cert $D/client.crt --key $D/client.key "$@"; }
# 1. mTLS: a client cert works (prints the backend's JSON echo) ...
edge -H "$AUTH" $P/orders/42
# ... no client cert fails in the TLS handshake
curl -sS --cacert $D/ca.crt -H "$AUTH" $P/orders/42
# 2. JWT: no token -> 401 with `www-authenticate: Bearer`; expired token -> 401
edge -i $P/orders/42 | head -3
edge -o /dev/null -w '%{http_code}\n' \
-H "Authorization: Bearer $(target/debug/edge-demo token --sub acme --ttl -120)" $P/orders/42
# 3. identity: tenant is "acme" although we claim to be admin, and
# forwarded_for is our real address although we claim 6.6.6.6
edge -H "$AUTH" -H 'x-ferryman-tenant: admin' -H 'x-forwarded-for: 6.6.6.6' $P/orders/42
# 4. routing: 404 off a segment boundary; 400 for a `..` segment
edge -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/ordersX
edge --path-as-is -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/orders/../inventory
# 5. bodies: 9 MiB is refused with 413
head -c 9437184 /dev/zero | edge -o /dev/null -w '%{http_code}\n' -X POST --data-binary @- \
-H "$AUTH" $P/orders/upload
# 6. rate limit: 8 concurrent requests from one tenant; the burst is 5
BURST="Authorization: Bearer $(target/debug/edge-demo token --sub burst)"
seq 8 | xargs -P8 -I{} curl -sS --cacert $D/ca.crt --cert $D/client.crt --key $D/client.key \
-o /dev/null -w '%{http_code}\n' -H "$BURST" $P/orders/1 | sort | uniq -c
# 5 200
# 3 429 (typically; a token refills every 200 ms, so a slow run can show 6 x 200)
# 7. circuit breaker: kill inventory, wait two health intervals -> 503
kill $INVENTORY; sleep 2
edge -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/inventory/x
curl -s 127.0.0.1:9090/metrics | grep '^ferryman_circuit_state' # ...} 1 = open
target/debug/backend --name inventory --bind 127.0.0.1:9102 & INVENTORY=$!
sleep 2
edge -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/inventory/x # recovered
# 8. hot reload: rename the /payments route to /billing and signal the proxy
edge -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/payments/p1 # 200
sed -i 's|"/payments"|"/billing"|' $D/ferryman.toml
kill -USR1 $PROXY; sleep 1
edge -o /dev/null -w '%{http_code} ' -H "$AUTH" $P/payments/p1 # 404
edge -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/billing/p1 # 200
# 9. certificate hot reload is easiest to see in `edge-demo run`; by hand you
# replace server.crt/server.key and send SIGUSR1 as above.
# 10. metrics
curl -s 127.0.0.1:9090/metrics | grep -E '^ferryman_(requests_total|ratelimited_total)'
# 11. graceful shutdown: SIGTERM lets in-flight requests finish, then stop the rest
kill -TERM $PROXY; kill $ORDERS $INVENTORY $PAYMENTS
Run it with Docker Compose
compose/
runs the same topology in containers: the proxy, three backends, and
Prometheus. From the repository root:
# PKI + tokens into examples/edge-demo/.demo/ (mounted read-only into the proxy)
cargo run -q -p ferryman-edge-demo --bin edge-demo -- setup
# First build compiles both images in release mode: expect several minutes.
docker compose -f examples/edge-demo/compose/docker-compose.yml up -d --build
TOKEN=$(cargo run -q -p ferryman-edge-demo --bin edge-demo -- token --sub acme)
MTLS="--cacert examples/edge-demo/.demo/ca.crt --cert examples/edge-demo/.demo/client.crt --key examples/edge-demo/.demo/client.key"
curl -s $MTLS -H "Authorization: Bearer $TOKEN" https://localhost:8443/orders/42
# {"service":"orders","tenant":"acme","path":"/orders/42",...}
# (503 for the first second or two: the proxy's health checker hasn't seen the backend yet)
curl -s -o /dev/null -w '%{http_code}\n' $MTLS https://localhost:8443/orders/42
# 401
docker compose -f examples/edge-demo/compose/docker-compose.yml kill -s SIGUSR1 edge
docker compose -f examples/edge-demo/compose/docker-compose.yml logs edge | grep reloaded
# "mTLS config reloaded" and "routing table reloaded"
curl -s localhost:9091/api/v1/targets | grep -o '"health":"[a-z]*"'
# "health":"up" (Prometheus UI: http://localhost:9091)
docker compose -f examples/edge-demo/compose/docker-compose.yml down
Only edge publishes a port (8443). Metrics (9090) and the backends
stay on the compose network, so from the host the proxy is the only way in.
Inside that network any container can still reach a backend directly. In
production, enforce “only the proxy talks to backends” with network
policy, because the backends trust x-ferryman-tenant.
compose/ferryman.toml is the container version of the config: container
paths under /app/certs, service names as upstreams, and the same issuer
and audience as the demo tokens. Re-running setup regenerates the PKI, so
restart edge afterwards: the JWT key is read at boot, and SIGUSR1 only
reloads TLS material and routes.
Adapting this to your project
- PKI: replace the generated files with your own CA.
[tls] client_ca_pathis the CA that signs client certificates;cert_path/key_pathis the server leaf your clients will verify (its SANs must match how they connect). Renew by replacing the files and sendingSIGUSR1. - JWT: point
[jwt] jwks_pathat your identity provider’s RSA public key (PEM, RS256) and setissuerandaudience; without them any token signed by that key is accepted for any service. The key is read at boot; a rotation needs a restart (SIGUSR1 reloads TLS and routes only). - Backends: read
x-ferryman-tenantas the caller identity and do not re-authenticate. That is only safe when the proxy is the only thing that can reach them (private network, no published ports). If clients can reach a backend directly they can forge the header. - Tuning:
tenant_rps,health_interval_secs,default_cooldown_secsand per-routecooldown_secsare in the config; the full reference is in docs/operations.md.
Streaming mode
By default the proxy buffers request bodies (fast for typical JSON). Build it
with --features ferryman-edge/boxed_body for streaming forwarding; the demo
passes unchanged:
examples/edge-demo/run.sh --features ferryman-edge/boxed_body
The trade-offs are in the main README.
embed-core: mTLS + JWT + rate limiting inside your own axum service
ferryman-edge-core is the set of primitives behind the
ferryman-edge proxy. This example
uses them directly in an axum 0.8 app, with no proxy in front:
src/lib.rs:router, therequire_jwtmiddleware, andserve_tls, an accept loop over aReloadingTls.src/main.rs: env-configured binary.tests/embed.rs: real mTLS handshakes against the server, in-process.
Embed the core, or run the proxy?
Embed when you have one service and want mTLS + JWT + per-tenant limits without an extra network hop or a second deployable. Run the proxy when you front several upstreams.
What you give up by embedding:
- Routing: no prefix table; your router does that.
- Circuit breaker and active health checks: they protect a proxy’s upstreams, and you have none.
- A separate trust boundary: a bug in your handlers now runs in the process that holds the TLS private key.
- Hot route reload: only certificates reload (
SIGUSR1, orReloadingTls::reload()).
The middleware, to copy
#![allow(unused)]
fn main() {
pub async fn require_jwt(State(st): State<AppState>, mut req: Request, next: Next) -> Response {
let claims = match bearer(&req) {
Some(token) => st.jwt.verify(token).await,
None => None,
};
let Some(claims) = claims else {
return (StatusCode::UNAUTHORIZED,
[(header::WWW_AUTHENTICATE, HeaderValue::from_static("Bearer"))],
"unauthorized").into_response();
};
if let Some(l) = &st.limiter {
if !ferryman_edge_core::check(l, &claims.sub) {
return (StatusCode::TOO_MANY_REQUESTS,
[(header::RETRY_AFTER, HeaderValue::from_static("1"))],
"rate limited").into_response();
}
}
req.extensions_mut().insert(claims); // handlers: Extension<Claims>
next.run(req).await
}
fn bearer(req: &Request) -> Option<&str> {
let raw = req.headers().get(header::AUTHORIZATION)?.to_str().ok()?;
let (scheme, token) = raw.split_once(' ')?;
(scheme.eq_ignore_ascii_case("bearer") && !token.is_empty()).then_some(token)
}
}
Wire it with middleware::from_fn_with_state(state, require_jwt) on the
routes that need auth, and leave /health outside it.
Run it
Configuration is by environment:
| Variable | Default | Meaning |
|---|---|---|
EMBED_BIND | 127.0.0.1:8444 | listen address |
EMBED_CERT, EMBED_KEY | required | server certificate chain and key (PEM) |
EMBED_CLIENT_CA | required | CA bundle that client certificates must chain to |
EMBED_JWT_PUB | required | RSA public key (PEM) for RS256 tokens |
EMBED_ISSUER, EMBED_AUDIENCE | required | the iss / aud every token must carry. Required so a token signed by the same key for another service is not accepted |
EMBED_TENANT_RPS | 100 | per-sub rate limit, 0 disables |
EMBED_LOG_JSON | unset | 1 for JSON logs |
The sibling edge-demo
example generates a matching set of certs and tokens (this depends on that
example being present). Build both first so curl doesn’t race a compile:
cargo build -p ferryman-edge-embed-example -p ferryman-edge-demo
target/debug/edge-demo setup # writes examples/edge-demo/.demo/
D=examples/edge-demo/.demo
EMBED_CERT=$D/server.crt EMBED_KEY=$D/server.key EMBED_CLIENT_CA=$D/ca.crt \
EMBED_JWT_PUB=$D/jwt-signing.pub \
EMBED_ISSUER=https://issuer.demo.local EMBED_AUDIENCE=ferryman-edge \
target/debug/ferryman-edge-embed-example >embed.log 2>&1 &
PID=$!
TOKEN=$(target/debug/edge-demo token --sub acme)
curl --cacert $D/ca.crt --cert $D/client.crt --key $D/client.key \
-H "authorization: Bearer $TOKEN" https://localhost:8444/whoami
# {"scope":"","sub":"acme"} (the demo tokens carry no scope)
kill $PID
Rotate the certificate files and kill -USR1 $(pidof ferryman-edge-embed-example);
new connections get the new certificate, existing ones are untouched.
Before production
This is a reference, not a hardened service. At minimum:
- Timeouts: add a request/handler timeout, e.g.
tower_http::timeout::TimeoutLayer.serve_tlsbounds the handshake and the first request, not slow handlers or slow bodies. - Body limits: axum’s built-in extractors cap bodies at 2 MiB; anything reading the raw body (or with
DefaultBodyLimit::disable) is unbounded. SetDefaultBodyLimitdeliberately. - Metrics: none are exported here; count auth failures, 429s and handshake failures (never label by tenant or path).
- Key rotation: the JWT key is read once at startup; there is no reload for it, only for the TLS files. Plan a restart or extend it.
- A real CA: the demo CA is for demos. Use your own CA and rotate server and client certificates.
- Issuer and audience are required for a reason; keep them.
Test
cargo test -p ferryman-edge-embed-example
API reference
| Crate | Use it for | Docs |
|---|---|---|
ferryman-edge-core | The primitives: ReloadingTls, JwtVerifier, the per-tenant limiter, RouteTable with its circuit breaker, the health loop, the config schema. | docs.rs/ferryman-edge-core |
ferryman-edge | The ferryman-edge-server binary. Its library API (serve, AppState, proxy, reload) exists for the binary and its tests and is not yet semver-stable. | docs.rs/ferryman-edge |
[dependencies]
ferryman-edge-core = "0.1"
Publishing to crates.io
Two crates, one shared version ([workspace.package] version):
| Crate | What users get | Depends on |
|---|---|---|
ferryman-edge-core | library: TLS reload, JWT verifier, rate limiter, routing + breaker | — |
ferryman-edge | cargo install ferryman-edge → ferryman-edge-server binary, plus the (unstable) serve library | ferryman-edge-core (same version) |
Publish order is always core first, then server. cargo publish --workspace
does this automatically.
What’s already set up
- Metadata inherited from
[workspace.package]:license = "MIT OR Apache-2.0",authors,repository,homepage,keywords,categories,rust-version = "1.88". - License texts:
LICENSE-APACHE/LICENSE-MITat the root, symlinked into each crate so they land in each package. - READMEs: server uses the root
README.md; core hascrates/core/README.md. crates.io rewrites relative links against the GitHub repository. exclude = ["tests/", "benches/"]in both crates. The tests and benches need repo-level fixtures, including test-only private keys that secret scanners would flag in a published crate; they run in CI from the repo. The packages contain onlysrc/, the manifest, README and licenses.- The server’s path dependency on core carries
version = "0.1.0", which is what crates.io uses. cargo publish --workspace --dry-runpasses: both crates package and build in isolation, with server resolved against core through a temporary registry, which is how crates.io will resolve it.
Status
v0.1.0 of both crates is published (2026-09-28). The first publish hit crates.io’s new-crate rate limit (429, “published too many new crates in a short period”); new crates refill about one per 10 minutes. Publishing core and then the proxy needed two waits. Later version bumps of existing crates use a separate, looser limit.
v0.1.1 (2026-09-30) was the first release published by the release workflow through Trusted Publishing. Nothing was published from a laptop.
Decisions taken
- Names:
ferryman-edge-core(library) andferryman-edge(proxy). The proxy’s binary staysferryman-edge-server, so Docker, CI,pidofand the docs keep working. - The proxy crate’s library API (
serve,AppState,proxy,reload) is documented as not semver-stable; it exists for the binary and its tests.
How releases are published: Trusted Publishing
Starting with 0.1.1, releases are published by CI, not from a laptop. Pushing a
v* tag runs
.github/workflows/release.yml:
- verify: the tag must equal the workspace version, and
CHANGELOG.mdmust have that version’s section. Tests run in both body modes, thencargo publish --workspace --dry-run. - publish (GitHub environment
release, which only acceptsv*tags):rust-lang/crates-io-auth-actionswaps the job’s GitHub OIDC token for a crates.io token that lives 30 minutes and is revoked when the job ends.cargo publish --workspacethen publishes core, then the proxy. The job finishes by creating the GitHub release from the CHANGELOG section.
The trust is configured on both sides:
| Where | Setting |
|---|---|
| crates.io, each crate → Settings → Trusted Publishing | owner Bunty9, repository ferryman-edge, workflow release.yml, environment release |
GitHub → Settings → Environments → release | deployment tags: v* only |
Renaming release.yml, or the environment, breaks publishing until the
crates.io config is updated to match. No CARGO_REGISTRY_TOKEN secret
exists or is needed.
Release checklist
export PATH=$HOME/.cargo/bin:$PATH
# 1. Clean tree on main, CI green for HEAD.
git status --short && gh run list --limit 1
# 2. Bump [workspace.package] version AND the version on the
# ferryman-edge-core dependency in crates/server/Cargo.toml.
# Move CHANGELOG "Unreleased" entries under a "## [X.Y.Z] — date" heading
# and add the compare link at the bottom.
# 3. Local gate (CI repeats it; this just saves a round trip).
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
cargo publish --workspace --dry-run
# 4. Commit and push; wait for CI on main to go green.
git commit -am "release: vX.Y.Z" && git push origin main
# 5. Tag and push the tag. This publishes: irreversible, since a version
# can be yanked but never deleted or reused.
git tag -a vX.Y.Z -m "vX.Y.Z" && git push origin vX.Y.Z
gh run watch "$(gh run list --workflow release.yml --limit 1 --json databaseId -q '.[0].databaseId')"
After publishing:
- Check https://docs.rs/ferryman-edge-core and
https://docs.rs/ferryman-edge build. docs.rs builds
aws-lc-sys; if it fails there, add[package.metadata.docs.rs]settings rather than changing the TLS provider. cargo install ferryman-edgeon a clean machine and run the README quick start.
If something goes wrong
- The verify job fails (e.g. tag/version mismatch): nothing was
published. Delete the tag (
git push origin :refs/tags/vX.Y.Z && git tag -d vX.Y.Z), fix, and tag again. - The publish job fails after core went out: re-running it would fail
on the already-published core, so publish only the proxy:
cargo publish -p ferryman-edgelocally, with an API token scoped topublish-updatefor that crate. Don’t re-bump core. - Bad release:
cargo yank --version X.Y.Z ferryman-edge(and core if needed), fix, and release X.Y.Z+1. Yanking stops new lockfiles from picking it up; it does not delete it. - Leaked secret in a package: yank it, rotate the secret, and contact crates.io support — yanked crates stay downloadable.
Changelog
All notable changes to ferryman-edge-core and ferryman-edge (the proxy).
Both crates share one version. Format follows
Keep a Changelog; versions follow
SemVer (pre-1.0: minor bumps may break).
Unreleased
0.1.1 — 2026-09-30
No code changes in either crate.
Added
- Reference examples, built and run in CI:
examples/edge-demo(the proxy in front of sample services, 55 end-to-end checks, and a docker-compose topology with Prometheus) andexamples/embed-core(ferryman-edge-core inside an axum service). - Project guide at https://bunty9.github.io/ferryman-edge/ (now the crates’ homepage).
Changed
- Releases are published from CI through crates.io Trusted Publishing (OIDC); no long-lived API token is used.
- READMEs: install section, crates.io / docs.rs badges.
Fixed
- Docker image: builder pinned to bookworm to match the distroless runtime’s
glibc;
.dockerignorekeepstarget/and keys out of the build context.
0.1.0 — 2026-09-28
Added
- mTLS termination on rustls 0.23 + aws-lc-rs; client certs required and chained to a configured CA bundle; ALPN h2 / http/1.1.
- RS256 JWT auth with a moka cache;
expre-checked on cache hits,nbfenforced, optionaliss/aud. - Per-tenant GCRA rate limiting keyed by JWT
sub(tenant_rps = 0disables it). - Segment-boundary longest-prefix routing with a lock-free Closed / Open / HalfOpen circuit breaker per upstream and an active health checker.
- SIGUSR1 hot reload of TLS material and routes without dropping connections; breaker state kept for unchanged routes.
- Graceful drain on SIGTERM / SIGINT.
boxed_bodyfeature: stream request and response bodies instead of collecting them.- Prometheus metrics (requests, auth failures, rate limiting, TLS handshakes, breaker state, upstream health).