Introduction

Repath is a transparent proxy that sits between your application and any LLM provider. It enables progressive delivery for AI — the same canary deployment patterns used for traditional software, but adapted for the non-deterministic world of LLMs.

When you change a prompt or switch model versions, Repath routes a small percentage of traffic to the new version, evaluates response quality with an AI judge, and automatically promotes or rolls back based on your thresholds — before users notice anything is wrong.

How it works

1. Point your app at Repath — change one line (your base URL)
2. Create a rollout — define baseline vs candidate version, traffic split, and quality thresholds
3. Repath splits traffic — routes X% to candidate, rest to baseline
4. LLM Judge scores responses — evaluates every response async (~120ms), zero latency added
5. Controller decides — every 30 seconds, checks rolling scores. Quality holds → advance. Quality drops → rollback

Quick Start

Get Repath running in under 5 minutes. You need an account and an OpenAI API key.

1. Sign up and get your gateway URL

After signing up at tryrepath.com/signup, your dashboard shows your unique gateway URL:

https://api.tryrepath.com/v1

2. Change one line in your app

from openai import OpenAI

# Before
client = OpenAI(api_key="sk-...")

# After — that's the entire integration
client = OpenAI(
    api_key="sk-...",
    base_url="https://api.tryrepath.com/v1",
    default_headers={"X-Repath-Key": "rp_live_YOUR_KEY"}
)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: "https://api.tryrepath.com/v1",
  defaultHeaders: { "X-Repath-Key": "rp_live_YOUR_KEY" },
});

Vercel AI SDK

The AI SDK talks the OpenAI wire protocol, so it needs the same two changes. Streaming works — Repath forwards every SSE chunk to your app as it arrives and evaluates the assembled response afterwards, so streamText stays token-by-token.

import { createOpenAI } from "@ai-sdk/openai";
import { streamText } from "ai";

const repath = createOpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: "https://api.tryrepath.com/v1",
  headers: { "X-Repath-Key": process.env.REPATH_KEY! },
});

const result = streamText({
  model: repath("gpt-4o-mini"),
  prompt: "Explain progressive delivery in one paragraph.",
});

OpenRouter, Anthropic and Gemini

Point the same client at Repath and name the provider’s model. For Anthropic, Repath translates the request and response between the OpenAI and Anthropic formats, so your OpenAI client keeps working unchanged. Store that provider’s key under Settings → Provider API Keys first, and Repath uses it instead of forwarding yours.

// OpenRouter — model ids are namespaced
const result = streamText({
  model: repath("anthropic/claude-sonnet-5-5"),
  prompt: "…",
});

3. Create your first rollout

Go to your dashboard → Rollouts → New Rollout. The form maps one-to-one onto this rollout definition:

# rollout.yaml
apiVersion: repath/v1
kind: Rollout
metadata:
  name: my-first-canary

spec:
  baseline:
    provider: openai
    model: gpt-4o-mini
    prompt:
      system: "You are a helpful customer support agent."

  candidate:
    provider: openai
    model: gpt-4o-mini
    prompt:
      system: "You are a precise, empathetic support agent. Always confirm you understood the issue before answering."

  strategy:
    type: canary
    steps:
      - weight: 5        # percent of traffic on the candidate
        duration: 5m     # minimum time at this step
        gate: { quality_score: ">= 0.80" }
      - weight: 25
        duration: 10m
        gate: { quality_score: ">= 0.80" }
      - weight: 100      # promoted

    rollback:
      trigger: { quality_score: "< 0.70" }
      action: rollback

  # Optional. Decisions wait for this many judged responses on the
  # candidate (default 100) — lower it while your traffic is light.
  policy:
    min_samples: 30

Integration

Supported providers

Specifying which model to use

Pass the model in the request body as normal. Repath sends it to the model's own provider:claude-* to Anthropic (with the request format translated), gemini-* to Google, a vendor-prefixed id like x-ai/grok-4.7 to OpenRouter, and everything else to OpenAI. Your provider key goes in Authorization as usual.

# OpenAI model
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)

# Anthropic model — same syntax, Repath translates
response = client.chat.completions.create(
    model="claude-sonnet-5-5",
    messages=[{"role": "user", "content": "Hello"}]
)

Sending traffic to a rollout

Name the rollout in the X-Repath-Rollout header, by name or id. Repath then chooses the baseline or the candidate for each request, and the rollout's own model and prompt apply. Without the header, a request uses your most recently started rollout, if any.

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,          // your provider key
  baseURL: "https://api.tryrepath.com/v1",
  defaultHeaders: {
    "X-Repath-Key": process.env.REPATH_KEY!,   // your Repath key
    "X-Repath-Rollout": "checkout-assistant",  // which experiment
  },
});

Core Concepts

Creating a Rollout

A rollout has three required fields: baseline, candidate, and strategy.

spec:
  baseline:
    provider: openai          # openai · anthropic · gemini · openrouter · vercel
    model: gpt-4o-mini
    prompt:
      system: "Current system prompt"
    parameters:
      temperature: 0.7
      max_tokens: 512

  candidate:
    provider: openai
    model: gpt-4o             # Testing a bigger model
    prompt:
      system: "Improved system prompt"
    parameters:
      temperature: 0.5
      max_tokens: 512

Traffic Splitting

Weights

Traffic weight is a float from 0.0 to 1.0. A weight of 0.10 routes 10% of requests to the candidate.

Sticky sessions

Pass X-User-Id or X-Session-Id in your requests. Repath uses consistent hashing so the same user always sees the same version during a rollout — no jarring switches mid-conversation.

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    extra_headers={"X-User-Id": "user_abc123"}  # Sticky routing
)

Quality Gates

The controller checks every rollout every 30 seconds against the last 10 minutes of judged traffic. It advances to the next step when the candidate's judged quality meets the gate, its error rate is under the ceiling and the step's minimum duration has passed. It rolls back the moment quality falls below the rollback trigger.

strategy:
  type: canary
  steps:
    - weight: 5                    # percent of traffic on the candidate
      duration: 5m                 # minimum time at this step
      gate:
        quality_score: ">= 0.80"   # judged quality needed to advance

    - weight: 25
      duration: 10m
      gate:
        quality_score: ">= 0.80"

    - weight: 100                  # final step: promoted

  rollback:
    trigger:
      quality_score: "< 0.70"      # roll back below this
      error_rate: "> 0.05"         # or above this error rate (default 5%)
    action: rollback

Shadow Mode

LLM-as-Judge: How It Works

After every proxied request, Repath enqueues the request/response pair to a Redis Stream. Python evaluation workers consume from this stream and score the response using an LLM judge. This is fully asynchronous — evaluation never blocks or slows your actual requests.

Evaluation latency: ~120ms. User-visible latency added: 0ms.

Evaluation Criteria

The judge scores every sampled response against three criteria, each rated 1–5 and normalized to 0–1. Each score comes with a one-sentence reason, which you can read for any request in the request log.

The Judge Model

Every judged request is scored by gpt-6-luna, the same judge for every account, so a score of 0.82 means the same thing in every rollout you run. The judge is independent of the models you are comparing: your baseline and candidate can be any provider and any model. It runs through the Vercel AI Gateway, so one vendor's outage or billing issue cannot stop judging.

Composite Scoring

The overall quality score is a weighted average of all criteria scores:

composite_score = Σ (criterion_score × weight)

Example:
  helpfulness: 0.85 × 0.5 = 0.425
  accuracy:    0.92 × 0.3 = 0.276
  clarity:     0.78 × 0.2 = 0.156
  ─────────────────────────────────
  composite:               0.857 → meets a 0.80 gate

Auto-Rollback: How It Works

The Repath controller runs on a 30-second tick. On each tick, it queries the rolling 10-minute average quality score for the candidate version. If the score falls below the rollback threshold, it immediately sets candidate weight to 0%, writes a rollback decision to the audit log, and updates the routing config.

Timeline after a quality drop:

1. Bad response evaluated by judge (~120ms after response)
2. Score written to PostgreSQL
3. Controller tick runs (max 30s wait)
4. Rolling average computed, threshold exceeded
5. Rollback decision written, config updated
6. Gateway reads new config (within 5s via cache refresh)
7. 100% traffic back to baseline

Total time from quality drop to full rollback: typically 35–60 seconds.

Rollback Configuration

rollback:
  trigger:
    quality_score: "< 0.70"    # roll back when judged quality drops below 0.70
    error_rate: "> 0.10"       # or when more than 10% of requests fail
  action: rollback

Rollback is immediate: within one controller cycle the candidate gets 0% of traffic, and every rollback is recorded with the metrics that caused it. To be told when it happens, add a Slack webhook or email address in Settings → Notifications, or a signed webhook in Settings → Webhooks. Those apply to all of your rollouts.

Audit Trail

Every advance and rollback decision is stored in the decisions table with the exact scores that triggered it. View via the dashboard or API:

# Get decision history for a rollout
curl -H "Authorization: Bearer $REPATH_API_TOKEN" \
  https://api.tryrepath.com/api/v1/rollouts/MY_ROLLOUT_ID/decisions

{
  "decisions": [
    {
      "id": "dec_01abc",
      "action": "rollback",
      "reason": "Quality score 0.61 dropped below threshold 0.70",
      "previous_weight": 0.25,
      "new_weight": 0.0,
      "triggered_by": "controller",
      "metrics_snapshot": {
        "avg_quality_candidate": 0.61,
        "avg_quality_baseline": 0.91,
        "sample_count": 847
      },
      "created_at": "2026-06-20T03:47:22Z"
    }
  ]
}

Provider Failover

When a provider returns 5xx errors or times out, Repath automatically retries once (after 300ms) then switches to the next provider in your fallback chain. Your app never sees the outage.

Failover Configuration

In Settings → Providers, save a key for each provider you want to fail over to, then order them in the failover chain (up to four). When your primary returns a 5xx or 429, Repath retries once, then walks the chain in order.

A fallback provider cannot serve the model you asked the primary for, so Repath picks the current model of the same size there. A nano request never starts billing at flagship rates, and a flagship request is never answered by a nano model:

OpenRouter and the Vercel AI Gateway need no mapping: Repath namespaces the model for them (gpt-5.4 becomes openai/gpt-5.4, and claude-sonnet-5-5 becomes anthropic/claude-sonnet-5.5, their spelling), which makes either a good last entry in the chain.

Circuit Breaker

The circuit breaker tracks error rates per provider per tenant. After 3 consecutive failures, it opens the circuit — all requests for that tenant bypass to the next provider in the chain. After a 5-second cooldown, one probe request is allowed through. If it succeeds, the circuit closes.

API Authentication

All management API endpoints require a Bearer token. Get yours from the dashboard under Settings → API Token.

curl -H "Authorization: Bearer $REPATH_API_TOKEN" \
  https://api.tryrepath.com/api/v1/system/health

Rollouts API

Decisions API

# Get audit log
GET /api/v1/rollouts/:id/decisions

# Response
{
  "decisions": [{
    "id": "dec_01abc",
    "action": "advance | rollback | promote | pause",
    "reason": "Human-readable explanation",
    "previous_weight": 0.10,
    "new_weight": 0.25,
    "triggered_by": "controller | manual",
    "metrics_snapshot": { ... },
    "created_at": "2026-06-20T03:47:22Z"
  }]
}

Webhooks

Repath can POST to your endpoint on rollout events. Configure in your rollout YAML or dashboard:

notify:
  webhook: "https://your-app.com/webhooks/repath"
  events:
    - rollback        # Quality dropped, rolled back
    - promote         # Successfully promoted to 100%
    - advance         # Advanced to next step

Webhook payload:

{
  "event": "rollback",
  "rollout_id": "rol_abc123",
  "rollout_name": "my-canary",
  "action": "rollback",
  "reason": "Quality score 0.61 < threshold 0.70",
  "timestamp": "2026-06-20T03:47:22Z",
  "metrics": {
    "avg_quality_candidate": 0.61,
    "avg_quality_baseline": 0.91
  }
}