KServe

Use KServe with agentgateway.

KServe is a Kubernetes-native platform for serving machine learning models. With agentgateway in front of KServe, you can enforce traffic management policies, such as token-based rate limiting, for inference requests without modifying your inference services.

Before you begin

Follow the Get started guide to install agentgateway.
Follow the Sample app guide to create a gateway proxy with an HTTP listener and deploy the httpbin sample app.
Get the external address of the gateway and save it in an environment variable.
Tip
Kind cluster? Kind does not support LoadBalancer services by default. To use this option with a Kind cluster, install and run cloud-provider-kind.
export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") echo $INGRESS_GW_ADDRESS
kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80

Step 1: Install cert-manager

Install cert-manager, which KServe requires for webhook certificates.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml

Wait for cert-manager to be ready before you continue.

kubectl wait --for=condition=available deployment --all -n cert-manager --timeout=120s

Step 2: Create the KServe namespace and gateway

Create the kserve namespace.
```
kubectl create namespace kserve
```

Create a Gateway resource that agentgateway manages. KServe attaches HTTPRoute resources to this gateway automatically for each LLMInferenceService you deploy.

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kserve-ingress-gateway
  namespace: kserve
spec:
  gatewayClassName: agentgateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
  infrastructure:
    labels:
      serving.kserve.io/gateway: kserve-ingress-gateway
EOF

Verify that the Gateway is programmed.

kubectl get gateway kserve-ingress-gateway -n kserve

Example output:

NAME                     CLASS          ADDRESS   PROGRAMMED   AGE
kserve-ingress-gateway   agentgateway             True         11s

Step 3: Install KServe

Install the KServe LLMInferenceService CRDs.

helm install kserve-llmisvc-crd \
  oci://ghcr.io/kserve/charts/kserve-llmisvc-crd \
  --version v0.19.0 \
  --namespace kserve

Install the KServe LLMInferenceService resources by using Helm.

helm install kserve-llmisvc-resources \
  oci://ghcr.io/kserve/charts/kserve-llmisvc-resources \
  --version v0.19.0 \
  --namespace kserve \
  --set kserve.controller.deploymentMode=Standard \
  --set kserve.controller.gateway.ingressGateway.enableGatewayApi=true \
  --set kserve.controller.gateway.ingressGateway.createGateway=false \
  --set kserve.controller.gateway.ingressGateway.kserveGateway=kserve/kserve-ingress-gateway \
  --set kserve.controller.gateway.ingressGateway.className=agentgateway \
  --set kserve.controller.gateway.disableIstioVirtualHost=true \
  --set kserve.controller.gateway.disableIngressCreation=false \
  --set kserve.controller.knativeAddressableResolver.enabled=false \
  --set kserve.controller.gateway.localGateway.gateway="" \
  --set kserve.controller.gateway.localGateway.gatewayService=""

Install the default LLMInferenceServiceConfig resources. KServe merges these defaults with the settings in each LLMInferenceService.

helm install kserve-runtime-configs \
  oci://ghcr.io/kserve/charts/kserve-runtime-configs \
  --version v0.19.0 \
  --namespace kserve \
  --set kserve.llmisvcConfigs.enabled=true

Verify that the KServe LLMInferenceService controller is available.

kubectl wait --for=condition=available \
  deployment/llmisvc-controller-manager \
  -n kserve \
  --timeout=180s
kubectl get deployment llmisvc-controller-manager -n kserve

Example output:

deployment.apps/llmisvc-controller-manager condition met
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
llmisvc-controller-manager     1/1     1            1           45s

Step 4: Deploy a mocked LLM with llm-d-inference-sim

Instead of a real model, this guide uses llm-d-inference-sim to serve a mock OpenAI compatible endpoint. llm-d-inference-sim’s /v1/chat/completions path returns a properly structured OpenAI chat completion response, including usage.total_tokens in the response body, which agentgateway reads to enforce token-based rate limits.

Create the test namespace.
```
kubectl create namespace kserve-test
```

Create an AgentgatewayBackend that points to the workload service that KServe creates for the LLMInferenceService. The backend identifies the endpoint as an OpenAI-compatible LLM so that agentgateway can apply LLM-aware features.

kubectl apply -f - <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: mock-llm-backend
  namespace: kserve-test
spec:
  ai:
    provider:
      openai:
        model: mock-llm
      host: mock-llm-kserve-workload-svc.kserve-test.svc.cluster.local
      port: 8000
      path: "/v1/chat/completions"
EOF

Deploy an LLMInferenceService that runs llm-d-inference-sim. The spec.router.route.http settings instruct KServe to generate an HTTPRoute that references the AgentgatewayBackend directly. Because the simulator does not need model files, the example disables KServe’s storage initializer.

kubectl apply -f - <<EOF
apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: mock-llm
  namespace: kserve-test
spec:
  model:
    name: mock-llm
    uri: hf://mock/mock-llm
  replicas: 1
  storageInitializer:
    enabled: false
  router:
    route:
      http:
        spec:
          parentRefs:
            - group: gateway.networking.k8s.io
              kind: Gateway
              name: kserve-ingress-gateway
              namespace: kserve
          hostnames:
            - mock-llm-kserve-test.example.com
          rules:
            - backendRefs:
                - group: agentgateway.dev
                  kind: AgentgatewayBackend
                  name: mock-llm-backend
              matches:
                - path:
                    type: PathPrefix
                    value: /v1/chat/completions
              timeouts:
                backendRequest: 0s
                request: 0s
  template:
    containers:
      - name: main
        image: ghcr.io/llm-d/llm-d-inference-sim:v0.9.0-rc3
        command:
          - /app/llm-d-inference-sim
        args:
          - --model
          - mock-llm
          - --port
          - "8000"
          - --mode
          - echo
        ports:
          - name: http
            containerPort: 8000
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
EOF

Wait for the LLMInferenceService to become ready.

kubectl wait --for=condition=Ready \
  llminferenceservice/mock-llm \
  -n kserve-test \
  --timeout=300s

Verify that KServe created one HTTPRoute whose backend is the AgentgatewayBackend.

kubectl get httproute mock-llm-kserve-route -n kserve-test \
  -o jsonpath='{.spec.rules[0].backendRefs[0]}'

Example output:

{"group":"agentgateway.dev","kind":"AgentgatewayBackend","name":"mock-llm-backend","weight":1}

Optional Step 4b: Apply a transformation policy to the KServe-generated HTTPRoute

Without a policy, agentgateway forwards requests and responses as-is. This step shows how a transformation policy can enrich responses with additional headers — without touching the inference service itself.

Verify that KServe created an HTTPRoute after the LLMInferenceService becomes Ready. The route attaches to kserve/kserve-ingress-gateway with hostname mock-llm-kserve-test.example.com.
```
kubectl get httproute mock-llm-kserve-route -n kserve-test -o yaml
```

Get the external address of the gateway and save it in an environment variable.

export INGRESS_GW_ADDRESS=$(kubectl get svc -n kserve kserve-ingress-gateway \
  -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS

Confirm that the response contains no custom headers.

curl -s -X POST http://$INGRESS_GW_ADDRESS/v1/chat/completions \
  -H "Host: mock-llm-kserve-test.example.com" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mock-llm",
    "messages": [{"role": "user", "content": "Hello"}]
  }' -v 2>&1 | grep "^<"

Example Output:

< HTTP/1.1 200 OK
< server: fasthttp
< date: Mon, 18 May 2026 21:55:33 GMT
< content-type: application/json
< content-length: 353

Apply a transformation policy that reads the model name from the request and response body and injects them as response headers.

kubectl apply -f - <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: model-echo-headers
  namespace: kserve-test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: mock-llm-kserve-route
  traffic:
    transformation:
      response:
        set:
          - name: x-requested-model
            value: 'string(json(request.body).model)'
          - name: x-actual-model
            value: 'string(json(response.body).model)'
EOF

Send the same request again and check the headers.

curl -s -X POST http://$INGRESS_GW_ADDRESS/v1/chat/completions \
  -H "Host: mock-llm-kserve-test.example.com" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mock-llm",
    "messages": [{"role": "user", "content": "Hello"}]
  }' -v 2>&1 | grep "^<"

Example output:

< HTTP/1.1 200 OK
< server: fasthttp
< date: Mon, 18 May 2026 21:56:12 GMT
< content-type: application/json
< content-length: 353
< x-requested-model: mock-llm
< x-actual-model: mock-llm

Port-forward the gateway to your local machine.

kubectl port-forward -n kserve svc/kserve-ingress-gateway 8080:80

Confirm that the response contains no custom headers.

curl -s -X POST http://localhost:8080/v1/chat/completions \
  -H "Host: mock-llm-kserve-test.example.com" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mock-llm",
    "messages": [{"role": "user", "content": "Hello"}]
  }' -v 2>&1 | grep "^<"

Example Output:

< HTTP/1.1 200 OK
< server: fasthttp
< date: Mon, 18 May 2026 21:55:33 GMT
< content-type: application/json
< content-length: 353

Apply a transformation policy that reads the model name from the request and response body and injects them as response headers.

kubectl apply -f - <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: model-echo-headers
  namespace: kserve-test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: mock-llm-kserve-route
  traffic:
    transformation:
      response:
        set:
          - name: x-requested-model
            value: 'string(json(request.body).model)'
          - name: x-actual-model
            value: 'string(json(response.body).model)'
EOF

Send the same request again and check the headers.

curl -s -X POST http://localhost:8080/v1/chat/completions \
  -H "Host: mock-llm-kserve-test.example.com" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mock-llm",
    "messages": [{"role": "user", "content": "Hello"}]
  }' -v 2>&1 | grep "^<"

Example output:

< HTTP/1.1 200 OK
< server: fasthttp
< date: Mon, 18 May 2026 21:56:12 GMT
< content-type: application/json
< content-length: 353
< x-requested-model: mock-llm
< x-actual-model: mock-llm

Step 5: Test the endpoint

Get the external address of the gateway and save it in an environment variable.

export INGRESS_GW_ADDRESS=$(kubectl get svc -n kserve kserve-ingress-gateway \
  -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS

Send a request to verify the setup works end-to-end.

curl -s http://$INGRESS_GW_ADDRESS/v1/chat/completions \
  -H "Host: mock-llm-kserve-test.example.com" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mock-llm",
    "messages": [
      {"role": "user", "content": "Hello"}
    ]
  }' | jq

Example output:

{
  "model": "mock-llm",
  "usage": {
    "prompt_tokens": 6,
    "completion_tokens": 1,
    "total_tokens": 7,
    "prompt_tokens_detail": {
      "cached_tokens": 0
    }
  },
  "choices": [
    {
      "message": {
        "content": "Hello",
        "role": "assistant"
      },
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "id": "chatcmpl-98473698-57bc-5d69-b91e-af0aace83ac9",
  "object": "chat.completion",
  "kv_transfer_params": null,
  "created": 1779134384
}

Port-forward the gateway to your local machine.

kubectl port-forward -n kserve svc/kserve-ingress-gateway 8080:80

Send a single request to confirm the setup works end-to-end.

curl -s -X POST http://localhost:8080/v1/chat/completions \
  -H "Host: mock-llm-kserve-test.example.com" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mock-llm",
    "messages": [{"role": "user", "content": "Hello"}]
  }' | jq

Example output:

{
  "model": "mock-llm",
  "usage": {
    "prompt_tokens": 6,
    "completion_tokens": 1,
    "total_tokens": 7,
    "prompt_tokens_detail": {
      "cached_tokens": 0
    }
  },
  "choices": [
    {
      "message": {
        "content": "Hello",
        "role": "assistant"
      },
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "id": "chatcmpl-98473698-57bc-5d69-b91e-af0aace83ac9",
  "object": "chat.completion",
  "kv_transfer_params": null,
  "created": 1779134384
}

Optional Step 6: Apply token-based rate limiting

How token counting works: Agentgateway reads usage.total_tokens from the JSON response body returned by the inference service. Each request deducts that many tokens from the bucket. When the bucket empties, subsequent requests receive 429 Too Many Requests until the next fill interval.

Apply an AgentgatewayPolicy that caps requests at 70 tokens per minute. The policy targets the KServe-generated mock-llm-kserve-route, which selects the AgentgatewayBackend.

kubectl apply -f - <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: llm-token-budget
  namespace: kserve-test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: mock-llm-kserve-route
  traffic:
    rateLimit:
      local:
        - tokens: 70
          unit: Minutes
EOF

Verify the policy is accepted and attached. Both Accepted and Attached conditions must be True.

kubectl get agentgatewaypolicy llm-token-budget -n kserve-test \
  -o jsonpath='{.status.ancestors[0].conditions}'

Get the external address of the gateway and save it in an environment variable.

export INGRESS_GW_ADDRESS=$(kubectl get svc -n kserve kserve-ingress-gateway \
  -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS

Run a burst of requests to trigger the token rate limit. With tokens: 70 and each response consuming 7 tokens, the budget exhausts after roughly 10 requests.

for i in $(seq 1 30); do
  curl -s -o /dev/null -w "%{http_code}\n" \
    -X POST http://$INGRESS_GW_ADDRESS/v1/chat/completions \
    -H "Host: mock-llm-kserve-test.example.com" \
    -H "Content-Type: application/json" \
    -d '{"model": "mock-llm", "messages": [{"role": "user", "content": "Hello"}]}'
done

Example output:

Port-forward the gateway to your local machine.

kubectl port-forward -n kserve svc/kserve-ingress-gateway 8080:80

Run a burst of requests to trigger the token rate limit. With tokens: 70 and each response consuming 7 tokens, the budget exhausts after roughly 10 requests.

for i in $(seq 1 30); do
  curl -s -o /dev/null -w "%{http_code}\n" \
    -X POST http://localhost:8080/v1/chat/completions \
    -H "Host: mock-llm-kserve-test.example.com" \
    -H "Content-Type: application/json" \
    -d '{"model": "mock-llm", "messages": [{"role": "user", "content": "Hello"}]}'
done

Example output:

Cleanup

Remove the resources created in this guide.

kubectl delete agentgatewaypolicy llm-token-budget -n kserve-test
kubectl delete AgentgatewayPolicy -n kserve-test model-echo-headers
kubectl delete llminferenceservice mock-llm -n kserve-test
kubectl delete agentgatewaybackend mock-llm-backend -n kserve-test
kubectl delete namespace kserve-test
helm uninstall kserve-runtime-configs -n kserve
helm uninstall kserve-llmisvc-resources -n kserve
helm uninstall kserve-llmisvc-crd -n kserve
kubectl delete gateway kserve-ingress-gateway -n kserve
kubectl delete namespace kserve

Argo Rollouts LLM client configuration

Was this page helpful?

KServe

Before you begin

Step 1: Install cert-manager

Step 2: Create the KServe namespace and gateway

Step 3: Install KServe

Step 4: Deploy a mocked LLM with llm-d-inference-sim

Optional Step 4b: Apply a transformation policy to the KServe-generated HTTPRoute

Step 5: Test the endpoint

Optional Step 6: Apply token-based rate limiting

Cleanup

What could be improved?