Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ferryman-edge

An mTLS-terminating L7 reverse proxy in Rust. Every request passes a TLS client-certificate check, an RS256 JWT check, and a per-tenant rate limit before it is routed to an upstream behind a per-upstream circuit breaker. TLS material and routes hot-reload on SIGUSR1 without dropping connections.

cargo install ferryman-edge      # installs the `ferryman-edge-server` binary

Source: github.com/Bunty9/ferryman-edge. Licensed under MIT or Apache-2.0.

Overview

The problem

Real edge proxies do mTLS termination, JWT validation in-line, per-route rate limiting, and cert hot-reload — and they make defensible decisions on body buffering, HTTP/2 stream control, and connection pooling. ferryman-edge is the project that gets the Cloudflare Pingora team to reply.

Architecture (delta on top of P2)

                Client (with mTLS cert)
                       |
                       v
              +--------+---------+
              | rustls TLS+mTLS  |  cert reload via SIGUSR1
              | + JWT validate   |  (no connection drops)
              +--------+---------+
                       |
                       v
              +--------+---------+
              | per-tenant rate  |  governor crate, keyed by JWT.sub
              | limit (per route)|
              +--------+---------+
                       |
                       v
              +--------+---------+
              | RouteService     |
              | (P2's table)     |
              +--------+---------+
                       |
                       v
              +--------+---------+
              | hyper client     |  pool: 1 conn per upstream * N
              | HTTP/2 multiplex |  feature flag: boxed_body vs collected
              +--------+---------+
                       |
                       v
                  upstream svc

Stack

LayerCrate / Tool
Async runtimetokio 1.47 (full)
HTTP serverhyper 1.5 + hyper-util + tower-http
TLS / mTLSrustls 0.23 (aws-lc-rs provider) + tokio-rustls 0.26 + rustls-pemfile 2 + rustls-pki-types
AuthNjsonwebtoken 9 + moka 0.12 (future cache, 10k × 5min)
Rate limitgovernor 0.7 (keyed GCRA)
Config / hot-swapserde + toml 0.8 + arc-swap; reload via SIGUSR1
Observabilitytracing + metrics-exporter-prometheus 0.16
CLIclap 4
Container buildcargo-chef multi-stage; distroless final (NOT scratch — aws-lc-rs needs libc)
DeployFly.io 2-region (sin + iad)
CIGHA (stable + beta) + cargo-deny + cargo-nextest + criterion (non-blocking) + mTLS smoke

Pinned versions live in Cargo.toml.

Request pipeline & design

Request pipeline

Every request passes the same gates, in order:

GateReject withMetric
TLS handshake, client cert must chain to client_ca_path (10 s timeout)connection closedferryman_tls_handshake_failures_total, ferryman_tls_handshake_seconds
Authorization: Bearer <RS256 JWT>: exp (also on cache hits), nbf, and iss/aud when configured401 + www-authenticate: Bearerferryman_auth_failures_total{reason}
Per-tenant GCRA limit keyed by sub (tenant_rps, 0 disables)429 + retry-after: 1ferryman_ratelimited_total
No . / .. path segments (incl. %2e)400ferryman_requests_total{status}
Longest-prefix route on a path-segment boundary; no fall-through to a shorter prefix404 no route, 503 breaker open
Body ≤ 8 MiB413
Client body read within 30 s (collected mode; read before route lookup)408 slow client, 400 body error
Upstream round trip within 30 s, counted from when the body is ready (plus the response body in collected mode)502 transport/response-body error, 504 timeoutferryman_request_duration_seconds{upstream} (success path)

On the way through, the proxy strips hop-by-hop headers (both directions, including any named in Connection), then stamps x-ferryman-tenant: <sub> (any client-supplied value is dropped first). It rewrites Host to the upstream, replaces x-forwarded-for with the peer IP (dropping client-sent Forwarded / X-Real-IP), sets x-forwarded-proto: https, and downgrades the outbound request to HTTP/1.1. Inbound protocol is pinned from ALPN (h2 or http/1.1). A connection with no request within 10 s of the handshake is closed (this also covers a stalled h2 preface); h2 connections then get keep-alive pings and a 64-stream cap.

Each upstream has a Closed / Open / HalfOpen circuit breaker (ferryman_circuit_state{upstream}: 0/1/2; cooldown_secs must be ≥ 1). A transport error, a 502–504, or a timeout opens it — under boxed_body a timeout only counts if the client had finished uploading; after cooldown_secs exactly one request is let through as the probe. A plain 500 does not trip it, and neither does a failure caused by the client’s own body (size cap, disconnect). The active health checker (GET <upstream>/health every health_interval_secs) opens and closes it too. A route reload keeps breaker state for rules whose prefix, upstream, and cooldown are unchanged.

SIGTERM / SIGINT stop accepting and drain in-flight connections for up to 25 s. SIGUSR1 reloads TLS material and the routing table; the JWT settings and tenant_rps are read once at boot.

Set [jwt] issuer and audience for anything beyond local dev — without them, any token signed by the issuer key is accepted, whichever service it was minted for.

Design tradeoffs

P4 makes three decisions worth defending in a hiring loop.

(a) boxed_body is off by default

Cargo feature boxed_body swaps the upstream client to http_body_util::BoxBody and streams request and response bodies. With it off, the proxy collects each body once into a Full<Bytes> before forwarding. Both builds enforce the 8 MiB request cap; under streaming, a chunked upload with no Content-Length that exceeds it is cut mid-stream and answered 413, without counting against the upstream’s breaker. Streaming mode has no separate body-read deadline: a slow upload runs inside the upstream’s 30 s budget and ends as 504.

Estimates from the design spec (not yet measured here): boxed adds ~200 µs per request at 10 MB; collected adds ~80 µs at 1 KB but allocates ~req_size. For an internal proxy fronting JSON APIs under ~256 KB the collected path wins on code complexity, allocator pressure (because the JSON allocator already paid the cost), and steady-state latency. Pingora picks streaming for general-purpose CDN traffic where payloads skew big and bimodal; that calculus inverts for an internal API edge. Flip the feature on (cargo build --features boxed_body) when your p99 latency tells you the collect-first cost dominates.

(b) SIGUSR1 reload over filesystem-watch

P2 (ferryman) reloads its routing table via notify filesystem events. P4 deliberately swaps to SIGUSR1 because:

  • k8s mounts ConfigMaps via a symlink-swap dance. notify reports this as a chain of remove + create events on the symlink target, not the watched path. Without per-platform special-casing the watcher silently misses the reload — a worst-case failure mode for a security-sensitive hot-swap of TLS material.
  • Editors emit a parade of Modify events for in-place writes that have no business triggering a reload (cursor moves, autosave drafts).
  • SIGUSR1 is one POSIX call with predictable semantics across every deploy target. The operator runs kill -USR1 $(pidof ferryman-edge-server) after kubectl rollout restart of the ConfigMap, or wires it into cert-manager’s renewal hook.

Both cert reload (tls::ReloadingTls) and route reload (server/src/reload.rs) share the same signal — one trigger swaps both surfaces atomically from the operator’s perspective.

(c) rustls + aws-lc-rs over OpenSSL

  • Pure-Rust audit story. rustls is the only TLS stack with a clean memory-safety argument all the way to the cipher implementations (aws-lc-rs is the AWS-libcrypto Rust binding; ring is the historical alternative). For an edge proxy that terminates customer-data TLS, that argument matters more than the C/Go ecosystem’s parity.
  • FIPS path. aws-lc-rs has a FIPS-mode build via the same crate. No swap-out at deploy time, no separate provider — flip a feature flag and recompile. OpenSSL FIPS 3.0 modules ship, but the build process is brittle and OS-distribution-specific.
  • Cost. Distroless final image instead of scratch (aws-lc-rs needs libc + dynamic loader). ~12 MB extra over a musl/scratch build. Worth it for the audit + FIPS leverage.

Benchmarks

# JWT verify: cache hit vs miss (criterion).
cargo bench -p ferryman-edge-core --bench jwt_verify

# Zero-loss reload check: curl workers (fresh mTLS handshake per request)
# while SIGUSR1 fires at 50% and 75% of the run. Needs the server up and
# an upstream answering /svc-a/echo.
./benches/reload.sh 60 8

wrk/wrk2 cannot present a TLS client certificate, so benches/wrk2.lua only works against a listener without mTLS; the 50k rps target below needs an mTLS-capable load generator and is not measured yet.

MetricTargetMeasured
JWT verify, cache hit vs miss≥ 10×0.68 µs vs 150 µs (~220×), criterion, dev laptop
Hot reload under loadzero failed reqs3725 / 3725 OK across 2× SIGUSR1 (60 s, 8 workers, release)
Throughput @ mTLS + JWT50,000 rps—
p99 latency< 8 ms—
TLS handshake p99 (full chain validation)< 50 ms119 ms, but client and server shared one box (contended)

Operating ferryman-edge

How to configure, run, reload, observe and debug the proxy. For the request pipeline and design rationale see the README.

Configuration reference

config.toml (path via --config / FERRYMAN_EDGE_CONFIG). Relative paths are resolved against the process working directory.

KeyDefaultReloaded on SIGUSR1Meaning
health_interval_secs5noActive health probe interval (GET <upstream>/health).
default_cooldown_secs30yes (routes)Breaker cooldown for routes that don’t set one.
tenant_rps1000noPer-tenant GCRA limit keyed by JWT sub. 0 disables rate limiting.
[tls] cert_path—yesServer certificate chain, PEM (leaf first, then intermediates).
[tls] key_path—yesServer private key, PEM (PKCS#8, PKCS#1 or SEC1).
[tls] client_ca_path—yesClient CA bundle, PEM. Every client cert must chain to one of these.
[jwt] jwks_path—noRSA public key, PEM, used for RS256 verification.
[jwt] issuerunsetnoRequired iss. Unset = not checked.
[jwt] audienceunsetnoRequired aud. Unset = not checked; tokens carrying any aud are then rejected.
[[routes]] prefix—yesPath prefix, matched on a segment boundary. Longest prefix wins.
[[routes]] upstream—yeshttp://host:port of the backend. Must have an authority.
[[routes]] cooldown_secsdefault_cooldown_secsyesPer-route breaker cooldown, must be ≥ 1.

Set issuer and audience in every non-local deployment. Without them the proxy accepts any token signed by the issuer key, whichever service it was minted for.

CLI / environment:

FlagEnvDefault
--configFERRYMAN_EDGE_CONFIGconfig.toml
--bindFERRYMAN_EDGE_BIND0.0.0.0:8443
--metrics-bindFERRYMAN_EDGE_METRICS_BIND0.0.0.0:9090
—RUST_LOGinfo (JSON logs to stdout)

Fixed limits (constants in crates/server/src): 8 MiB request body, 30 s client-body read, 30 s upstream round trip, 10 s TLS handshake, 10 s to the first request on a new connection, 64 concurrent h2 streams per connection, 25 s shutdown drain.

Local run

./scripts/gen-test-certs.sh                       # certs/ (gitignored)
cargo run -p ferryman-edge -- --config config.toml

curl --cacert certs/ca.crt --cert certs/client.crt --key certs/client.key \
     -H "Authorization: Bearer $(scripts/mint-jwt.sh tenant-a 3600)" \
     https://localhost:8443/svc-a/hello

scripts/mint-jwt.sh [sub] [ttl_secs] [scope] signs with certs/jwt-priv.pem; set JWT_ISS / JWT_AUD to add iss / aud.

The upstreams in config.toml (localhost:8001, localhost:8002) must serve GET /health with a 2xx, or the health checker keeps their breaker open and requests get 503. The proxy forwards the full path, prefix included (/svc-a/hello reaches the upstream as /svc-a/hello).

Hot reload

kill -USR1 "$(pidof ferryman-edge-server)"

One signal reloads both the TLS material and the routing table. A reload that fails to parse or load keeps the old config and logs mTLS reload failed; keeping old or route reload failed; keeping old table with the cause. Live connections keep the TLS config they handshook with; new connections get the new one. Breaker state carries over for routes whose prefix, upstream and cooldown did not change.

Not reloaded: the JWT key and claims settings, tenant_rps, health_interval_secs, bind addresses. Restart for those.

Use pidof, not pgrep -x: the binary name is longer than the 15-char kernel comm, so pgrep -x ferryman-edge-server never matches.

Shutdown

SIGTERM or SIGINT stops accepting, lets in-flight connections finish for up to 25 s, then exits. fly.toml sends SIGINT with a 30 s kill timeout.

Metrics

Prometheus text on --metrics-bind at /metrics. Keep it off the public network (fly.toml uses Fly’s internal [metrics] scrape).

MetricTypeLabels
ferryman_requests_totalcounterstatus, upstream (when one was chosen)
ferryman_request_duration_secondssummaryupstream (successful round trips)
ferryman_auth_failures_totalcounterreason = missing | invalid
ferryman_ratelimited_totalcounter— (never labelled by tenant)
ferryman_tls_handshake_secondssummary—
ferryman_tls_handshake_failures_totalcounter—
ferryman_circuit_stategaugeupstream; 0 closed, 1 open, 2 half-open
ferryman_upstream_alivegaugeupstream; last health probe result

upstream is host:port.

Troubleshooting

SymptomLikely cause
503 upstream unavailable right after bootFirst health probe ran before the upstream was up; the breaker closes on the next successful probe (≤ health_interval_secs).
503 persistsUpstream has no 2xx /health, or keeps failing. Check ferryman_upstream_alive.
404 no routeNo prefix matches on a segment boundary (/svc-a does not match /svc-abc).
400 bad pathPath has a . or .. segment (also %2e).
401 with a token you believe is validExpired (60 s leeway), nbf in the future, wrong key, or iss/aud mismatch. ferryman_auth_failures_total{reason="invalid"} counts these.
curl exits 56 / handshake failureNo client cert, or it doesn’t chain to client_ca_path. ferryman_tls_handshake_failures_total counts these.
Python client: CA cert does not include key usage extensionRoot CA generated without extensions; regenerate with the current gen-test-certs.sh.
Container exits with GLIBC_2.38 not foundBuilder and runtime images on different Debian releases; the Dockerfile pins both to bookworm.

Container

docker build -t ferryman-edge .
docker run -p 8443:8443 -v "$PWD/certs:/app/certs:ro" ferryman-edge

The image ships no key material and no certs; mount them at /app/certs (the paths in the baked-in /app/config.toml) or the server exits at boot.

Examples

ExampleRead it if you want to…
edge-demorun the ferryman-edge proxy in front of your services: PKI, tokens, config, upstreams, and every feature exercised end to end
embed-coreadd mTLS, JWT auth and per-tenant rate limiting to your own axum service with ferryman-edge-core

Both are workspace members (publish = false), so CI builds, lints and runs them.

edge-demo: the ferryman-edge proxy in action

A runnable reference for the operator path: the real ferryman-edge-server binary in front of sample upstream services, with every feature exercised and checked. Unix only (the proxy is controlled with SIGUSR1 / SIGTERM).

The example crate is ferryman-edge-demo, with two binaries:

  • backend: a sample upstream. It shows how a service consumes the x-ferryman-tenant header the proxy stamps.
  • edge-demo: a driver with three subcommands: setup (PKI + config), token (mint a JWT) and run (the narrated end-to-end walkthrough).

What you will see

edge-demo run starts three backends and the proxy on free ports, then walks through eleven scenarios, printing ✓ / ✗ per check and exiting non-zero if any fails (about 5 seconds):

  1. mTLS: valid cert over HTTP/2 and HTTP/1.1 works; no cert and a cert from another CA are refused.
  2. JWT: missing, garbage, expired, wrong aud, wrong iss, future nbf and wrongly-signed tokens are all 401; a valid one is 200.
  3. Identity propagation: the backend sees x-ferryman-tenant equal to the JWT sub (client-supplied values are discarded), the peer IP in x-forwarded-for, and x-forwarded-proto: https.
  4. Routing: longest prefix on a path-segment boundary; .. is rejected.
  5. Bodies: 6 MiB passes intact, 9 MiB is 413.
  6. Rate limiting: per tenant, 6th immediate request is 429 with retry-after.
  7. Circuit breaker + health checks: kill a backend, watch 503 and the ferryman_circuit_state gauge, restart it, watch recovery.
  8. Hot reload (routes): add a route, SIGUSR1, it is live.
  9. Hot reload (certificate): rotate the server cert, SIGUSR1, new handshakes present it; existing clients keep working.
  10. Metrics: Prometheus text on a separate port.
  11. Graceful shutdown: SIGTERM lets an in-flight request finish.

Run it

From the repository root:

examples/edge-demo/run.sh

which is cargo build -p ferryman-edge -p ferryman-edge-demo followed by target/debug/edge-demo run. Note that both packages must be built: cargo run -p ferryman-edge-demo alone builds neither the proxy nor backend (and the demo says so if it cannot find them). run.sh passes its arguments to cargo build, so ./run.sh --release works.

run keeps its keys, config and logs in examples/edge-demo/.demo/run/ (gitignored, never commit it), separate from the setup material in .demo/. Look at .demo/run/proxy.log for the proxy’s JSON logs. edge-demo run --keep leaves the stack running until Ctrl-C.

Do it by hand

Build, then generate the PKI and config for fixed local ports (proxy 8443, metrics 9090, backends 9101 to 9103):

cargo build -p ferryman-edge -p ferryman-edge-demo
target/debug/edge-demo setup

Start the backends and the proxy (setup prints equivalent commands with your absolute paths). Backends first; they must only be reachable from the proxy. The commands keep the process IDs so nothing else on your machine is touched:

target/debug/backend --name orders    --bind 127.0.0.1:9101 & ORDERS=$!
target/debug/backend --name inventory --bind 127.0.0.1:9102 & INVENTORY=$!
target/debug/backend --name payments  --bind 127.0.0.1:9103 & PAYMENTS=$!
target/debug/ferryman-edge-server \
  --config examples/edge-demo/.demo/ferryman.toml \
  --bind 127.0.0.1:8443 --metrics-bind 127.0.0.1:9090 & PROXY=$!

Set up a shorthand for a mutual-TLS request, then one request per feature. The rate limit is 5 requests per second per tenant, so run these at human speed:

D=examples/edge-demo/.demo
P=https://127.0.0.1:8443
TOKEN=$(target/debug/edge-demo token --sub acme)
AUTH="Authorization: Bearer $TOKEN"
edge() { curl -sS --cacert $D/ca.crt --cert $D/client.crt --key $D/client.key "$@"; }

# 1. mTLS: a client cert works (prints the backend's JSON echo) ...
edge -H "$AUTH" $P/orders/42
# ... no client cert fails in the TLS handshake
curl -sS --cacert $D/ca.crt -H "$AUTH" $P/orders/42

# 2. JWT: no token -> 401 with `www-authenticate: Bearer`; expired token -> 401
edge -i $P/orders/42 | head -3
edge -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $(target/debug/edge-demo token --sub acme --ttl -120)" $P/orders/42

# 3. identity: tenant is "acme" although we claim to be admin, and
#    forwarded_for is our real address although we claim 6.6.6.6
edge -H "$AUTH" -H 'x-ferryman-tenant: admin' -H 'x-forwarded-for: 6.6.6.6' $P/orders/42

# 4. routing: 404 off a segment boundary; 400 for a `..` segment
edge -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/ordersX
edge --path-as-is -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/orders/../inventory

# 5. bodies: 9 MiB is refused with 413
head -c 9437184 /dev/zero | edge -o /dev/null -w '%{http_code}\n' -X POST --data-binary @- \
  -H "$AUTH" $P/orders/upload

# 6. rate limit: 8 concurrent requests from one tenant; the burst is 5
BURST="Authorization: Bearer $(target/debug/edge-demo token --sub burst)"
seq 8 | xargs -P8 -I{} curl -sS --cacert $D/ca.crt --cert $D/client.crt --key $D/client.key \
  -o /dev/null -w '%{http_code}\n' -H "$BURST" $P/orders/1 | sort | uniq -c
#   5 200
#   3 429      (typically; a token refills every 200 ms, so a slow run can show 6 x 200)

# 7. circuit breaker: kill inventory, wait two health intervals -> 503
kill $INVENTORY; sleep 2
edge -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/inventory/x
curl -s 127.0.0.1:9090/metrics | grep '^ferryman_circuit_state'   # ...} 1 = open
target/debug/backend --name inventory --bind 127.0.0.1:9102 & INVENTORY=$!
sleep 2
edge -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/inventory/x   # recovered

# 8. hot reload: rename the /payments route to /billing and signal the proxy
edge -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/payments/p1   # 200
sed -i 's|"/payments"|"/billing"|' $D/ferryman.toml
kill -USR1 $PROXY; sleep 1
edge -o /dev/null -w '%{http_code} ' -H "$AUTH" $P/payments/p1     # 404
edge -o /dev/null -w '%{http_code}\n' -H "$AUTH" $P/billing/p1     # 200

# 9. certificate hot reload is easiest to see in `edge-demo run`; by hand you
#    replace server.crt/server.key and send SIGUSR1 as above.

# 10. metrics
curl -s 127.0.0.1:9090/metrics | grep -E '^ferryman_(requests_total|ratelimited_total)'

# 11. graceful shutdown: SIGTERM lets in-flight requests finish, then stop the rest
kill -TERM $PROXY; kill $ORDERS $INVENTORY $PAYMENTS

Run it with Docker Compose

compose/ runs the same topology in containers: the proxy, three backends, and Prometheus. From the repository root:

# PKI + tokens into examples/edge-demo/.demo/ (mounted read-only into the proxy)
cargo run -q -p ferryman-edge-demo --bin edge-demo -- setup

# First build compiles both images in release mode: expect several minutes.
docker compose -f examples/edge-demo/compose/docker-compose.yml up -d --build

TOKEN=$(cargo run -q -p ferryman-edge-demo --bin edge-demo -- token --sub acme)
MTLS="--cacert examples/edge-demo/.demo/ca.crt --cert examples/edge-demo/.demo/client.crt --key examples/edge-demo/.demo/client.key"

curl -s $MTLS -H "Authorization: Bearer $TOKEN" https://localhost:8443/orders/42
# {"service":"orders","tenant":"acme","path":"/orders/42",...}
# (503 for the first second or two: the proxy's health checker hasn't seen the backend yet)

curl -s -o /dev/null -w '%{http_code}\n' $MTLS https://localhost:8443/orders/42
# 401

docker compose -f examples/edge-demo/compose/docker-compose.yml kill -s SIGUSR1 edge
docker compose -f examples/edge-demo/compose/docker-compose.yml logs edge | grep reloaded
# "mTLS config reloaded" and "routing table reloaded"

curl -s localhost:9091/api/v1/targets | grep -o '"health":"[a-z]*"'
# "health":"up"   (Prometheus UI: http://localhost:9091)

docker compose -f examples/edge-demo/compose/docker-compose.yml down

Only edge publishes a port (8443). Metrics (9090) and the backends stay on the compose network, so from the host the proxy is the only way in. Inside that network any container can still reach a backend directly. In production, enforce “only the proxy talks to backends” with network policy, because the backends trust x-ferryman-tenant.

compose/ferryman.toml is the container version of the config: container paths under /app/certs, service names as upstreams, and the same issuer and audience as the demo tokens. Re-running setup regenerates the PKI, so restart edge afterwards: the JWT key is read at boot, and SIGUSR1 only reloads TLS material and routes.

Adapting this to your project

  • PKI: replace the generated files with your own CA. [tls] client_ca_path is the CA that signs client certificates; cert_path/key_path is the server leaf your clients will verify (its SANs must match how they connect). Renew by replacing the files and sending SIGUSR1.
  • JWT: point [jwt] jwks_path at your identity provider’s RSA public key (PEM, RS256) and set issuer and audience; without them any token signed by that key is accepted for any service. The key is read at boot; a rotation needs a restart (SIGUSR1 reloads TLS and routes only).
  • Backends: read x-ferryman-tenant as the caller identity and do not re-authenticate. That is only safe when the proxy is the only thing that can reach them (private network, no published ports). If clients can reach a backend directly they can forge the header.
  • Tuning: tenant_rps, health_interval_secs, default_cooldown_secs and per-route cooldown_secs are in the config; the full reference is in docs/operations.md.

Streaming mode

By default the proxy buffers request bodies (fast for typical JSON). Build it with --features ferryman-edge/boxed_body for streaming forwarding; the demo passes unchanged:

examples/edge-demo/run.sh --features ferryman-edge/boxed_body

The trade-offs are in the main README.

embed-core: mTLS + JWT + rate limiting inside your own axum service

ferryman-edge-core is the set of primitives behind the ferryman-edge proxy. This example uses them directly in an axum 0.8 app, with no proxy in front:

  • src/lib.rs: router, the require_jwt middleware, and serve_tls, an accept loop over a ReloadingTls.
  • src/main.rs: env-configured binary.
  • tests/embed.rs: real mTLS handshakes against the server, in-process.

Embed the core, or run the proxy?

Embed when you have one service and want mTLS + JWT + per-tenant limits without an extra network hop or a second deployable. Run the proxy when you front several upstreams.

What you give up by embedding:

  • Routing: no prefix table; your router does that.
  • Circuit breaker and active health checks: they protect a proxy’s upstreams, and you have none.
  • A separate trust boundary: a bug in your handlers now runs in the process that holds the TLS private key.
  • Hot route reload: only certificates reload (SIGUSR1, or ReloadingTls::reload()).

The middleware, to copy

#![allow(unused)]
fn main() {
pub async fn require_jwt(State(st): State<AppState>, mut req: Request, next: Next) -> Response {
    let claims = match bearer(&req) {
        Some(token) => st.jwt.verify(token).await,
        None => None,
    };
    let Some(claims) = claims else {
        return (StatusCode::UNAUTHORIZED,
                [(header::WWW_AUTHENTICATE, HeaderValue::from_static("Bearer"))],
                "unauthorized").into_response();
    };
    if let Some(l) = &st.limiter {
        if !ferryman_edge_core::check(l, &claims.sub) {
            return (StatusCode::TOO_MANY_REQUESTS,
                    [(header::RETRY_AFTER, HeaderValue::from_static("1"))],
                    "rate limited").into_response();
        }
    }
    req.extensions_mut().insert(claims); // handlers: Extension<Claims>
    next.run(req).await
}

fn bearer(req: &Request) -> Option<&str> {
    let raw = req.headers().get(header::AUTHORIZATION)?.to_str().ok()?;
    let (scheme, token) = raw.split_once(' ')?;
    (scheme.eq_ignore_ascii_case("bearer") && !token.is_empty()).then_some(token)
}
}

Wire it with middleware::from_fn_with_state(state, require_jwt) on the routes that need auth, and leave /health outside it.

Run it

Configuration is by environment:

VariableDefaultMeaning
EMBED_BIND127.0.0.1:8444listen address
EMBED_CERT, EMBED_KEYrequiredserver certificate chain and key (PEM)
EMBED_CLIENT_CArequiredCA bundle that client certificates must chain to
EMBED_JWT_PUBrequiredRSA public key (PEM) for RS256 tokens
EMBED_ISSUER, EMBED_AUDIENCErequiredthe iss / aud every token must carry. Required so a token signed by the same key for another service is not accepted
EMBED_TENANT_RPS100per-sub rate limit, 0 disables
EMBED_LOG_JSONunset1 for JSON logs

The sibling edge-demo example generates a matching set of certs and tokens (this depends on that example being present). Build both first so curl doesn’t race a compile:

cargo build -p ferryman-edge-embed-example -p ferryman-edge-demo
target/debug/edge-demo setup            # writes examples/edge-demo/.demo/
D=examples/edge-demo/.demo
EMBED_CERT=$D/server.crt EMBED_KEY=$D/server.key EMBED_CLIENT_CA=$D/ca.crt \
EMBED_JWT_PUB=$D/jwt-signing.pub \
EMBED_ISSUER=https://issuer.demo.local EMBED_AUDIENCE=ferryman-edge \
  target/debug/ferryman-edge-embed-example >embed.log 2>&1 &
PID=$!

TOKEN=$(target/debug/edge-demo token --sub acme)
curl --cacert $D/ca.crt --cert $D/client.crt --key $D/client.key \
     -H "authorization: Bearer $TOKEN" https://localhost:8444/whoami
# {"scope":"","sub":"acme"}    (the demo tokens carry no scope)
kill $PID

Rotate the certificate files and kill -USR1 $(pidof ferryman-edge-embed-example); new connections get the new certificate, existing ones are untouched.

Before production

This is a reference, not a hardened service. At minimum:

  • Timeouts: add a request/handler timeout, e.g. tower_http::timeout::TimeoutLayer. serve_tls bounds the handshake and the first request, not slow handlers or slow bodies.
  • Body limits: axum’s built-in extractors cap bodies at 2 MiB; anything reading the raw body (or with DefaultBodyLimit::disable) is unbounded. Set DefaultBodyLimit deliberately.
  • Metrics: none are exported here; count auth failures, 429s and handshake failures (never label by tenant or path).
  • Key rotation: the JWT key is read once at startup; there is no reload for it, only for the TLS files. Plan a restart or extend it.
  • A real CA: the demo CA is for demos. Use your own CA and rotate server and client certificates.
  • Issuer and audience are required for a reason; keep them.

Test

cargo test -p ferryman-edge-embed-example

API reference

CrateUse it forDocs
ferryman-edge-coreThe primitives: ReloadingTls, JwtVerifier, the per-tenant limiter, RouteTable with its circuit breaker, the health loop, the config schema.docs.rs/ferryman-edge-core
ferryman-edgeThe ferryman-edge-server binary. Its library API (serve, AppState, proxy, reload) exists for the binary and its tests and is not yet semver-stable.docs.rs/ferryman-edge
[dependencies]
ferryman-edge-core = "0.1"

Publishing to crates.io

Two crates, one shared version ([workspace.package] version):

CrateWhat users getDepends on
ferryman-edge-corelibrary: TLS reload, JWT verifier, rate limiter, routing + breaker—
ferryman-edgecargo install ferryman-edge → ferryman-edge-server binary, plus the (unstable) serve libraryferryman-edge-core (same version)

Publish order is always core first, then server. cargo publish --workspace does this automatically.

What’s already set up

  • Metadata inherited from [workspace.package]: license = "MIT OR Apache-2.0", authors, repository, homepage, keywords, categories, rust-version = "1.88".
  • License texts: LICENSE-APACHE / LICENSE-MIT at the root, symlinked into each crate so they land in each package.
  • READMEs: server uses the root README.md; core has crates/core/README.md. crates.io rewrites relative links against the GitHub repository.
  • exclude = ["tests/", "benches/"] in both crates. The tests and benches need repo-level fixtures, including test-only private keys that secret scanners would flag in a published crate; they run in CI from the repo. The packages contain only src/, the manifest, README and licenses.
  • The server’s path dependency on core carries version = "0.1.0", which is what crates.io uses.
  • cargo publish --workspace --dry-run passes: both crates package and build in isolation, with server resolved against core through a temporary registry, which is how crates.io will resolve it.

Status

v0.1.0 of both crates is published (2026-09-28). The first publish hit crates.io’s new-crate rate limit (429, “published too many new crates in a short period”); new crates refill about one per 10 minutes. Publishing core and then the proxy needed two waits. Later version bumps of existing crates use a separate, looser limit.

v0.1.1 (2026-09-30) was the first release published by the release workflow through Trusted Publishing. Nothing was published from a laptop.

Decisions taken

  • Names: ferryman-edge-core (library) and ferryman-edge (proxy). The proxy’s binary stays ferryman-edge-server, so Docker, CI, pidof and the docs keep working.
  • The proxy crate’s library API (serve, AppState, proxy, reload) is documented as not semver-stable; it exists for the binary and its tests.

How releases are published: Trusted Publishing

Starting with 0.1.1, releases are published by CI, not from a laptop. Pushing a v* tag runs .github/workflows/release.yml:

  1. verify: the tag must equal the workspace version, and CHANGELOG.md must have that version’s section. Tests run in both body modes, then cargo publish --workspace --dry-run.
  2. publish (GitHub environment release, which only accepts v* tags): rust-lang/crates-io-auth-action swaps the job’s GitHub OIDC token for a crates.io token that lives 30 minutes and is revoked when the job ends. cargo publish --workspace then publishes core, then the proxy. The job finishes by creating the GitHub release from the CHANGELOG section.

The trust is configured on both sides:

WhereSetting
crates.io, each crate → Settings → Trusted Publishingowner Bunty9, repository ferryman-edge, workflow release.yml, environment release
GitHub → Settings → Environments → releasedeployment tags: v* only

Renaming release.yml, or the environment, breaks publishing until the crates.io config is updated to match. No CARGO_REGISTRY_TOKEN secret exists or is needed.

Release checklist

export PATH=$HOME/.cargo/bin:$PATH

# 1. Clean tree on main, CI green for HEAD.
git status --short && gh run list --limit 1

# 2. Bump [workspace.package] version AND the version on the
#    ferryman-edge-core dependency in crates/server/Cargo.toml.
#    Move CHANGELOG "Unreleased" entries under a "## [X.Y.Z] — date" heading
#    and add the compare link at the bottom.

# 3. Local gate (CI repeats it; this just saves a round trip).
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
cargo publish --workspace --dry-run

# 4. Commit and push; wait for CI on main to go green.
git commit -am "release: vX.Y.Z" && git push origin main

# 5. Tag and push the tag. This publishes: irreversible, since a version
#    can be yanked but never deleted or reused.
git tag -a vX.Y.Z -m "vX.Y.Z" && git push origin vX.Y.Z
gh run watch "$(gh run list --workflow release.yml --limit 1 --json databaseId -q '.[0].databaseId')"

After publishing:

If something goes wrong

  • The verify job fails (e.g. tag/version mismatch): nothing was published. Delete the tag (git push origin :refs/tags/vX.Y.Z && git tag -d vX.Y.Z), fix, and tag again.
  • The publish job fails after core went out: re-running it would fail on the already-published core, so publish only the proxy: cargo publish -p ferryman-edge locally, with an API token scoped to publish-update for that crate. Don’t re-bump core.
  • Bad release: cargo yank --version X.Y.Z ferryman-edge (and core if needed), fix, and release X.Y.Z+1. Yanking stops new lockfiles from picking it up; it does not delete it.
  • Leaked secret in a package: yank it, rotate the secret, and contact crates.io support — yanked crates stay downloadable.

Changelog

All notable changes to ferryman-edge-core and ferryman-edge (the proxy). Both crates share one version. Format follows Keep a Changelog; versions follow SemVer (pre-1.0: minor bumps may break).

Unreleased

0.1.1 — 2026-09-30

No code changes in either crate.

Added

  • Reference examples, built and run in CI: examples/edge-demo (the proxy in front of sample services, 55 end-to-end checks, and a docker-compose topology with Prometheus) and examples/embed-core (ferryman-edge-core inside an axum service).
  • Project guide at https://bunty9.github.io/ferryman-edge/ (now the crates’ homepage).

Changed

  • Releases are published from CI through crates.io Trusted Publishing (OIDC); no long-lived API token is used.
  • READMEs: install section, crates.io / docs.rs badges.

Fixed

  • Docker image: builder pinned to bookworm to match the distroless runtime’s glibc; .dockerignore keeps target/ and keys out of the build context.

0.1.0 — 2026-09-28

Added

  • mTLS termination on rustls 0.23 + aws-lc-rs; client certs required and chained to a configured CA bundle; ALPN h2 / http/1.1.
  • RS256 JWT auth with a moka cache; exp re-checked on cache hits, nbf enforced, optional iss / aud.
  • Per-tenant GCRA rate limiting keyed by JWT sub (tenant_rps = 0 disables it).
  • Segment-boundary longest-prefix routing with a lock-free Closed / Open / HalfOpen circuit breaker per upstream and an active health checker.
  • SIGUSR1 hot reload of TLS material and routes without dropping connections; breaker state kept for unchanged routes.
  • Graceful drain on SIGTERM / SIGINT.
  • boxed_body feature: stream request and response bodies instead of collecting them.
  • Prometheus metrics (requests, auth failures, rate limiting, TLS handshakes, breaker state, upstream health).