Knative

Example Versions: Serving/Eventing/Kourier 1.23.0; Operator 1.23.1. Verify Kubernetes/EKS compatibility before installation. Last Updated: September 11, 2026

Overview and Learning Objectives
Knative Architecture
EKS Installation and Configuration
Knative Serving Deep Dive
Knative Eventing Deep Dive
KEDA vs Knative Comparison
Production Operations
Best Practices
References

Overview and Learning Objectives

What Is Knative?

Knative is a CNCF Graduated project that extends Kubernetes to provide a set of middleware components for building, deploying, and managing modern serverless workloads. Rather than replacing Kubernetes primitives, Knative builds on top of them, offering higher-level abstractions that simplify common patterns such as request-driven autoscaling, event delivery, and traffic management.

CNCF accepted Knative at the Incubating level on March 2, 2022, and graduated it on September 11, 2025. Project maturity does not establish production readiness of an individual deployment.

Knative consists of two independently installable components:

Knative Serving -- Manages the lifecycle of serverless workloads. It automates deployment, scaling (including scale-to-zero), revision tracking, and traffic routing.
Knative Eventing -- Provides infrastructure for producing, routing, and consuming events following the CloudEvents specification. It decouples event producers from consumers, enabling loosely coupled, event-driven architectures.

Serverless on Kubernetes

Traditional Kubernetes Deployments require operators to pre-configure replica counts, HPA thresholds, and resource budgets. Knative shifts this burden:

Workloads automatically scale from zero to many replicas based on incoming request concurrency or RPS.
Revision specs snapshot the workload template; traffic can be shifted to retained ready Revisions. Referenced Secrets/ConfigMaps, registry access and external state are not frozen, and routing/rollback takes reconciliation time.
Sources/adapters and Triggers separate event ingestion from delivery. Some adapters poll external systems, and application processing, authentication and failure handling still need implementation.

The result is a platform that retains the full power of Kubernetes (scheduling, RBAC, networking, storage) while providing a developer experience closer to that of a fully managed serverless platform.

Knative Serving vs Eventing

Aspect	Knative Serving	Knative Eventing
Primary purpose	Request-driven workload lifecycle	Event routing and delivery
Scaling trigger	HTTP request concurrency / RPS	Consumer/adapter-specific; a Broker/Trigger does not generically scale every subscriber
Scale-to-zero	Yes (built-in)	Depends on consumer (Serving-backed consumers can)
Core resources	Service, Configuration, Revision, Route	Broker, Trigger, Channel, Subscription, Source
Typical use case	APIs, web apps, microservices	Async pipelines, webhooks, CDC streams

Knative vs AWS Lambda and AWS Fargate

Feature	Knative Serving on EKS	AWS Lambda (standard compute)	AWS Fargate
Runtime	Containers meeting the Serving runtime contract	Supported Lambda runtimes or images implementing its runtime API	Supported ECS tasks/EKS Pods; platform restrictions apply
Request/execution lifetime	Configurable request timeout; defaults300s, allowed maximum600s unless reconfigured	Standard function timeout up to900s	Governed by task/Pod lifecycle, not a Lambda-style invocation limit
Scale-to-zero	KPA supports idle Revision scale-to-zero	On-demand execution	ECS desired tasks or Kubernetes replicas can be0; activation needs an appropriate controller/metric
Cold starts	Baseline replicas, image/startup/readiness tuning	Runtime-specific optimization and provisioned concurrency	Task/Pod startup and image/network setup; no fixed timing comparison
Memory	Node allocatable capacity, container requests and sidecars	128–10,240MB configurable	EKS Fargate slots up to120GB; platform overhead and supported CPU/memory combinations apply
GPU	Requires suitable nodes, device plugins, resource requests and enabled PodSpec features	No GPU in standard Lambda compute	GPUs are not available on Fargate
Networking	Kubernetes/CNI and gateway configuration	Customer VPC attachment is optional	VPC networking; EKS Fargate has platform constraints
Portability	Kubernetes/Knative APIs; cloud identity/storage integrations still matter	AWS-specific runtime/service APIs	ECS or EKS integration and supported platform APIs
Event input	HTTP/CloudEvents and configured adapters	Supported AWS event integrations	Application/controller integrations; not a generic event-routing service
Local testing	Local Kubernetes helps, but cloud behavior still needs validation	Local tools/emulators are optional and have fidelity limits	Container logic can be tested locally; platform behavior differs
Observability	Configure metrics, logs and tracing exporters	CloudWatch and supported tracing integrations	Configure supported AWS/OpenTelemetry collection
Cost basis	Allocated cluster, storage, load-balancer and supporting resources	Requests/duration plus selected features	Allocated task/Pod CPU and memory, plus associated resources

This table scopes Lambda to standard compute. Eligible asynchronous/event-source invocations on Lambda Managed Instances can allow up to90minutes (with service-specific exceptions); they have different operating constraints. Pod scale-to-zero does not itself remove EC2 nodes, persistent storage or load-balancer charges.

Learning Objectives

By the end of this document you will be able to:

Explain Knative's architecture and how Serving and Eventing complement each other.
Install and configure Knative on Amazon EKS with Kourier, DNS, and TLS.
Deploy serverless workloads with fine-grained concurrency-based autoscaling.
Implement traffic splitting strategies (canary, blue-green) using Revisions and Routes.
Build event-driven pipelines with Brokers, Triggers, and CloudEvents.
Compare KEDA and Knative and decide when to use each (or both).
Operate Knative in production with monitoring, high availability, and garbage collection policies.

Knative Architecture

Serving Architecture

Serving has control-plane components in knative-serving and a Queue Proxy in each Revision Pod’s workload namespace. Their responsibilities include reconciliation, routing, admission and scaling; they are not five Deployments all in the same namespace.

Serving control-plane components run in the Serving namespace, while each Revision Pod has Queue Proxy in its workload namespace. Routing can include Activator at zero or for burst-capacity handling; autoscaling reconciles the target scale.

Component responsibilities:

Component	Role
Activator	Receives requests when a Revision is scaled to zero. Buffers requests, triggers scale-up, then proxies the buffered requests once pods are ready. Also acts as a load balancer when the system is in "burst capacity" mode.
Autoscaler	Collects concurrency and RPS metrics from Queue Proxy sidecars. KPA and its reconciler drive the target scale; the optional HPA integration manages Kubernetes HPA resources. The Serving Controller reconciles workload resources, not a universal intermediate for every scale decision.
Queue Proxy	Injected as a sidecar into every Knative pod. Enforces `containerConcurrency` limits, reports real-time concurrency to the Autoscaler, performs health checking, and handles graceful shutdown during scale-down.
Controller	Reconciles Knative CRDs (Service, Configuration, Revision, Route) into underlying Kubernetes resources (Deployments, Services, Ingress objects). Manages revision creation and garbage collection.
Webhook	Validates and defaults Knative resource specifications on admission. Ensures that invalid configurations are rejected before they reach the Controller.

Eventing Architecture

Knative Eventing provides a declarative way to bind event sources to consumers. It supports two delivery patterns: Broker/Trigger (content-based routing) and Channel/Subscription (direct pub-sub).

Knative's Broker/Trigger pattern, where a Broker routes events by Trigger filter to consumer services and sends failed deliveries to a Dead Letter Sink, beside the Channel/Subscription pattern, where a Channel fans events out to Subscriptions.

Eventing core concepts:

Concept	Description
Event Source	A resource that generates or imports events. Knative provides built-in sources (ApiServerSource, PingSource) and the community maintains sources for Kafka, AWS SQS, GitHub, and more.
Broker	Receives events and delivers to matching Triggers through its selected implementation. Buffering, persistence and failure behavior depend on the class/backing transport.
Trigger	A filter attached to a Broker. Each Trigger selects events by CloudEvent attributes (type, source, extensions) and routes matches to a subscriber.
Channel	A durable or in-memory event transport. Unlike Brokers, Channels do not filter -- every Subscription receives every event.
Subscription	Connects a Channel to a subscriber and optionally a reply destination.
Dead Letter Sink	A fallback destination for events that cannot be delivered after exhausting retry policies.
CloudEvents	The CNCF standard envelope format (v1.0) used by all Knative Eventing components. Provides interoperability across sources and consumers.

EKS Installation and Configuration

The installation snippets target an isolated test environment and contain placeholders. Applying Serving with a LoadBalancer may provision AWS resources through your installed controller; no such action was executed in this audit. Operator-owned ConfigMaps, Services and workloads are configured through the KnativeServing/KnativeEventing resources. Preserve existing configuration when preparing merge patches, especially list fields. Create referenced application namespaces, ServiceAccounts, Secrets, images, Kafka/SQS resources and IAM roles before using later examples. Schema checks do not establish production readiness.

Prerequisites

An existing, provider-supported EKS version validated with the selected Knative components. The examples use Serving/Eventing/Kourier1.23.0 and Operator1.23.1; this is not an open-ended Kubernetes1.28+ compatibility claim.
kubectl configured with cluster admin access.
(Optional) helm v3.12+ for Helm-based installations.

Step 1: Install Knative Operator

The Knative Operator manages the installation and lifecycle of Knative Serving and Eventing components. Using the Operator simplifies version upgrades and configuration management.

bash

# Fresh installation on the intended test cluster; review versioned upgrade guidance for existing installs.
kubectl config current-context
kubectl apply --server-side -f https://github.com/knative/operator/releases/download/knative-v1.23.1/operator.yaml
kubectl wait --for=condition=Established crd/knativeservings.operator.knative.dev crd/knativeeventings.operator.knative.dev --timeout=120s
kubectl wait --for=condition=Available deployment/knative-operator deployment/operator-webhook -n knative-operator --timeout=300s

Step 2: Install Knative Serving via the Operator

Create a KnativeServing custom resource to deploy Serving components:

yaml

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  version: 1.23.0
  ingress:
    kourier:
      enabled: true
  config:
    network:
      ingress-class: kourier.ingress.networking.knative.dev
    autoscaler:
      pod-autoscaler-class: kpa.autoscaling.knative.dev
      container-concurrency-target-percentage: '70'
      enable-scale-to-zero: 'true'
    defaults:
      revision-timeout-seconds: '300'
      max-revision-timeout-seconds: '600'
    deployment:
      queue-sidecar-cpu-request: 25m
      queue-sidecar-memory-request: 400Mi
      queue-sidecar-memory-limit: 800Mi
  services:
  - name: kourier
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: external
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

bash

kubectl create namespace knative-serving --dry-run=client -o yaml | kubectl apply -f -
kubectl create namespace knative-demo --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f knative-serving.yaml
kubectl wait --for=condition=Ready knativeserving/knative-serving -n knative-serving --timeout=600s
kubectl get deployments,pods,services -n knative-serving

Step 3: Install Kourier (Lightweight Ingress)

Kourier is an Envoy-based networking implementation for Knative. It is a suitable option when its supported features meet the workload; resource or latency advantages require measurement.

The Operator installs Kourier from the selected Serving bundle. In this workflow the gateway Service is in knative-serving. A standalone manual Kourier manifest normally uses kourier-system; do not apply it on top of the Operator-managed installation. Inspect the managed resources:

bash

# Operator-managed Kourier uses the KnativeServing namespace.
kubectl get deployment net-kourier-controller 3scale-kourier-gateway -n knative-serving
kubectl get service kourier -n knative-serving -o yaml

The base KnativeServing resource above configures Kourier Service annotations for an NLB managed by the AWS Load Balancer Controller. The controller, IAM and selected subnets must already be prepared. A Service annotation does not create an ALB; an ALB requires an appropriate L7 controller/routing design. EKS Auto Mode uses a different load-balancer management path. Verify the actual class/annotations rather than assuming every EKS LoadBalancer Service defaults to NLB:

bash

kubectl get service kourier -n knative-serving -o jsonpath='{.spec.loadBalancerClass}{"\n"}{.metadata.annotations}{"\n"}'

Step 4: DNS Configuration

Knative generates URLs for each Service in the form <service>.<namespace>.<domain>. You must configure DNS so that these URLs resolve to the Ingress gateway.

Option A: Magic DNS (sslip.io) -- Development Only

sslip.io resolves supported IP-encoded names and is useful in suitable development environments. An AWS load-balancer hostname and changing backend IPs are not a durable production DNS mapping. This Operator workflow uses an explicit domain; the standalone default-domain helper must not compete with Operator-owned configuration.

bash

# Inspect the address type; this Operator workflow uses an explicitly configured domain.
kubectl get service kourier -n knative-serving -o jsonpath='{.status.loadBalancer.ingress}'
kubectl get ksvc -n knative-demo

Option B: Real DNS with Amazon Route 53 -- Production

Prepare a domain/hosted zone you control and inspect the generated change before applying it. The commands below update that zone when run; the audit did not execute them. Use one DNS owner (manual/IaC or ExternalDNS) per record. ExternalDNS requires its knative-serving source, permissions and a hostname matching the actual Route/DomainMapping; a hostname annotation alone does not create a new Knative route.

bash

# Review the intended zone/domain and wait for the NLB hostname before preparing a DNS change.
set -euo pipefail
export KOURIER_HOST
KOURIER_HOST=$(kubectl get service kourier -n knative-serving -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
: "${KOURIER_HOST:?Wait for the load-balancer hostname}"
export KNATIVE_DOMAIN="knative.example.com"
export HOSTED_ZONE_ID="REPLACE_WITH_HOSTED_ZONE_ID"
python3 - <<'PYDNS'
import json, os
from pathlib import Path
host = os.environ["KOURIER_HOST"].strip()
if not host or any(c.isspace() for c in host):
    raise SystemExit("Invalid load-balancer hostname")
change = {"Changes": [{"Action": "UPSERT", "ResourceRecordSet": {
    "Name": "*." + os.environ["KNATIVE_DOMAIN"], "Type": "CNAME", "TTL": 300,
    "ResourceRecords": [{"Value": host}]
}}]}
Path("knative-dns-change.json").write_text(json.dumps(change, indent=2))
patch = {"spec": {"config": {"domain": {os.environ["KNATIVE_DOMAIN"]: ""}}}}
Path("knative-domain.patch.json").write_text(json.dumps(patch))
PYDNS
# These commands change the selected DNS zone and Operator configuration when run.
aws route53 change-resource-record-sets --hosted-zone-id "$HOSTED_ZONE_ID" --change-batch file://knative-dns-change.json
kubectl patch knativeserving knative-serving -n knative-serving --type merge --patch-file knative-domain.patch.json

Step 5: TLS with cert-manager

Serving1.23 includes the cert-manager integration in its controller. The old separate net-certmanager repository is archived; do not install a nonexistent1.16/1.23 release from it. Install and configure a compatible cert-manager separately before enabling the integration, including its own Route53 DNS01 identity and least-privilege zone access.

bash

kubectl get deployments -n cert-manager
kubectl get crd certificates.cert-manager.io clusterissuers.cert-manager.io

Save this merge patch for the existing KnativeServing resource as serving-tls.patch.yaml. It is not a standalone ConfigMap to apply over Operator-owned state:

yaml

spec:
  config:
    network:
      certificate-class: cert-manager.certificate.networking.knative.dev
      external-domain-tls: Enabled
      http-protocol: Redirected
    certmanager:
      issuerRef: |
        group: cert-manager.io
        kind: ClusterIssuer
        name: letsencrypt-staging

For the first DNS/IAM validation use a staging ClusterIssuer. Replace the contact, hosted zone and Region with reviewed values. Staging certificates are not publicly trusted; switch to a production issuer only after validating the challenge flow and certificate lifecycle:

yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: REPLACE_WITH_CERTIFICATE_CONTACT_EMAIL
    privateKeySecretRef:
      name: letsencrypt-staging-key
    solvers:
    - dns01:
        route53:
          region: us-west-2
          hostedZoneID: REPLACE_WITH_HOSTED_ZONE_ID

Save the issuer as cluster-issuer.yaml and the preceding Operator patch as serving-tls.patch.yaml, then apply in this order:

bash

# cert-manager and its Route 53 identity must already be configured.
kubectl apply -f cluster-issuer.yaml
kubectl wait --for=condition=Ready clusterissuer/letsencrypt-staging --timeout=180s
kubectl patch knativeserving knative-serving -n knative-serving --type merge --patch-file serving-tls.patch.yaml
# The integration starts in the Serving controller after the setting is effective.
kubectl get configmap config-network -n knative-serving -o yaml
kubectl rollout restart deployment/controller -n knative-serving
kubectl rollout status deployment/controller -n knative-serving --timeout=300s

Step 6: HPA vs KPA Autoscaler Selection

Knative supports two autoscaler implementations. The choice affects scaling behavior significantly.

Feature	KPA (Knative Pod Autoscaler)	HPA (Kubernetes HPA)
Scale-to-zero	Yes	No
Metrics	Concurrency, RPS	CPU, Memory, Custom metrics
Scaling speed	Fast (panic/stable windows)	Standard HPA intervals
Configuration	Knative annotations	Standard HPA spec
Best for	HTTP workloads, latency-sensitive	CPU/memory-bound workloads

Configure the default autoscaler class through the Operator. The optional HPA extension and a working Kubernetes resource metrics API are prerequisites for HPA-class Revisions; this patch includes a pinned HPA manifest. Avoid duplicate management if your distribution already supplies it:

yaml

spec:
  additionalManifests:
  - URL: https://github.com/knative/serving/releases/download/knative-v1.23.0/serving-hpa.yaml
  config:
    autoscaler:
      pod-autoscaler-class: kpa.autoscaling.knative.dev
      stable-window: 60s
      panic-window-percentage: '10'
      panic-threshold-percentage: '200'
      scale-to-zero-grace-period: 30s
      scale-to-zero-pod-retention-period: 0s
      target-burst-capacity: '211'
      requests-per-second-target-default: '200'
      container-concurrency-target-default: '100'

Save as serving-autoscaler.patch.yaml and merge it into the existing Operator resource. Preserve any other additionalManifests entries when preparing the patch:

bash

kubectl patch knativeserving knative-serving -n knative-serving --type merge --patch-file serving-autoscaler.patch.yaml
kubectl get deployment autoscaler-hpa -n knative-serving

Override per Revision under spec.template.metadata.annotations on the Knative Service (the following is a template fragment):

yaml

metadata:
  annotations:
    autoscaling.knative.dev/class: "hpa.autoscaling.knative.dev"
    autoscaling.knative.dev/metric: "cpu"
    autoscaling.knative.dev/target: "70"

Step 7: Install Knative Eventing

yaml

apiVersion: operator.knative.dev/v1beta1
kind: KnativeEventing
metadata:
  name: knative-eventing
  namespace: knative-eventing
spec:
  version: 1.23.0
  defaultBrokerClass: MTChannelBasedBroker
  config:
    default-ch-webhook:
      default-ch-config: "clusterDefault:\n  apiVersion: messaging.knative.dev/v1\n\
        \  kind: InMemoryChannel\n"
  sinkBindingSelectionMode: inclusion

bash

kubectl create namespace knative-eventing --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f knative-eventing.yaml
kubectl wait --for=condition=Ready knativeeventing/knative-eventing -n knative-eventing --timeout=600s
kubectl get deployments,pods -n knative-eventing
kubectl get crd inmemorychannels.messaging.knative.dev integrationsources.sources.knative.dev

Knative Serving Deep Dive

Resource Model

Knative Serving introduces four primary custom resources that work together to manage the complete lifecycle of a serverless workload.

A Service owns a Configuration and Route; workload-template changes create immutable Revision specs, while external references can change. The Route sends configured traffic percentages to retained Revisions, not always100% to the newest.

Resource	Description
Service (`ksvc`)	The top-level resource. Manages the entire lifecycle by owning a Configuration and a Route. Most users interact only with Services.
Configuration	Describes the workload template; relevant template changes create a Revision. Metadata-only or traffic changes do not necessarily create one.
Revision	Immutable workload spec, with external references that can still change. Retention/garbage collection determines whether an old Revision remains available.
Route	Maps network traffic to one or more Revisions. Enables canary deployments, blue-green releases, and percentage-based traffic splitting.

Complete Knative Service YAML

Provide the referenced image, ServiceAccount and Secret first. The Serving controller must resolve registry tags/digests, while the node must separately be able to pull the image; EKS node credentials alone do not establish controller registry access. Readiness/liveness paths must match the real application and should not cause restart storms for downstream outages.

The following is a configuration example, not a tested production deployment. It defines a Knative Service with explicit autoscaling, resource limits, health checks, and scaling boundaries:

yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: order-api
  namespace: knative-demo
  labels:
    app.kubernetes.io/name: order-api
    app.kubernetes.io/part-of: ecommerce
    app.kubernetes.io/managed-by: knative
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
        autoscaling.knative.dev/metric: concurrency
        autoscaling.knative.dev/target: '100'
        autoscaling.knative.dev/target-utilization-percentage: '70'
        autoscaling.knative.dev/min-scale: '2'
        autoscaling.knative.dev/max-scale: '50'
        autoscaling.knative.dev/initial-scale: '3'
        autoscaling.knative.dev/scale-down-delay: 15m
        autoscaling.knative.dev/window: 60s
      name: order-api-v1
    spec:
      containerConcurrency: 0
      timeoutSeconds: 300
      containers:
      - image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/order-api:v1.2.3
        ports:
        - containerPort: 8080
          protocol: TCP
        env:
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: host
        - name: LOG_LEVEL
          value: info
        resources:
          requests:
            cpu: 250m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 1Gi
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20
      serviceAccountName: order-api-sa

Traffic Splitting: Canary Deployments

Traffic splitting allows you to gradually shift traffic between Revisions. This is the foundation for canary and blue-green deployment strategies.

Canary Deployment

These are alternative exercises starting from the complete base Service above and its ready order-api-v1 Revision. Replace images and names with verified artifacts; every template change needs a new unused Revision name. The JSON patch preserves environment, ServiceAccount, probes and resource settings instead of replacing the whole containers list. It assumes the base example’s single ingress container is index0. A0% tag makes a route available for testing, not a guarantee of full warm capacity.

Save as canary-template.patch.json:

json

[
  {
    "op": "add",
    "path": "/spec/traffic",
    "value": [
      {
        "revisionName": "order-api-v1",
        "percent": 100
      },
      {
        "revisionName": "order-api-v2",
        "percent": 0,
        "tag": "canary"
      }
    ]
  },
  {
    "op": "add",
    "path": "/spec/template/metadata/name",
    "value": "order-api-v2"
  },
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/image",
    "value": "123456789012.dkr.ecr.us-west-2.amazonaws.com/order-api:v1.3.0"
  }
]

bash

kubectl wait --for=condition=Ready revision/order-api-v1 -n knative-demo --timeout=300s
kubectl patch ksvc order-api -n knative-demo --type json --patch-file canary-template.patch.json
kubectl wait --for=jsonpath='{.status.latestCreatedRevisionName}'=order-api-v2 ksvc/order-api -n knative-demo --timeout=180s
kubectl wait --for=condition=Ready revision/order-api-v2 -n knative-demo --timeout=300s
kubectl get ksvc order-api -n knative-demo -o jsonpath='{.status.traffic}'

# Route10%, then50%, then100% only after validating each stage.
kubectl patch ksvc order-api -n knative-demo --type merge --patch '{"spec":{"traffic":[{"revisionName":"order-api-v1","percent":90},{"revisionName":"order-api-v2","percent":10,"tag":"canary"}]}}'
kubectl patch ksvc order-api -n knative-demo --type merge --patch '{"spec":{"traffic":[{"revisionName":"order-api-v1","percent":50},{"revisionName":"order-api-v2","percent":50,"tag":"canary"}]}}'
kubectl patch ksvc order-api -n knative-demo --type merge --patch '{"spec":{"traffic":[{"revisionName":"order-api-v1","percent":0},{"revisionName":"order-api-v2","percent":100,"tag":"canary"}]}}'

Read the actual tag URL from status.traffic; it depends on the domain/tag templates and TLS settings. Check Route readiness, observed configuration and application signals between changes. Percentages apply to routing policy, not an exact count in a small request sample.

Blue-Green Deployment

Use this as a separate exercise from the base order-api-v1, or update the stable name to the Revision currently serving your verified baseline. Save the following as green-template.patch.json:

json

[
  {
    "op": "add",
    "path": "/spec/traffic",
    "value": [
      {
        "revisionName": "order-api-v1",
        "percent": 100
      },
      {
        "revisionName": "order-api-green",
        "percent": 0,
        "tag": "green"
      }
    ]
  },
  {
    "op": "add",
    "path": "/spec/template/metadata/name",
    "value": "order-api-green"
  },
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/image",
    "value": "123456789012.dkr.ecr.us-west-2.amazonaws.com/order-api:v2.0.0"
  }
]

bash

kubectl patch ksvc order-api -n knative-demo --type json --patch-file green-template.patch.json
kubectl wait --for=jsonpath='{.status.latestCreatedRevisionName}'=order-api-green ksvc/order-api -n knative-demo --timeout=180s
kubectl wait --for=condition=Ready revision/order-api-green -n knative-demo --timeout=300s
kubectl get ksvc order-api -n knative-demo -o jsonpath='{.status.traffic}'
# Validate the green tag URL and capacity before requesting this switch.
kubectl patch ksvc order-api -n knative-demo --type merge --patch '{"spec":{"traffic":[{"revisionName":"order-api-v1","percent":0},{"revisionName":"order-api-green","percent":100,"tag":"green"}]}}'

The traffic change is reconciled asynchronously; connections and in-flight requests can still use the previous Revision. Retain the old Revision and its external dependencies for rollback, and measure actual propagation/readiness. These commands were not executed against a cluster.

Scale-to-Zero Behavior

Scale-to-zero is a defining feature of Knative Serving. When a Revision is eligible for scale-to-zero, its idle-window, scale-down delay, retention and internal routing readiness affect when the last Pod is removed. When a new request arrives, the Activator buffers it, triggers a scale-up, and proxies the request once a pod is ready.

Scale-to-zero prepares the Activator route before removing the last Pod. A new request can wait for ready capacity within buffer/timeouts; Activator may remain in the path according to burst-capacity settings.

Key parameters controlling scale-to-zero:

Annotation / Config	Default	Description
`scale-to-zero-grace-period` (global)	30s	Upper bound for internal scale-from-zero routing preparation before removing the last replica; not a guaranteed idle-retention timer.
`scale-to-zero-pod-retention-period` (global)	0s	Minimum retention of the last Pod after the autoscaler decides to scale to zero.
`autoscaling.knative.dev/scale-to-zero-pod-retention-period` (per-Revision)	inherited	Per-Revision override of the global retention period.
`enable-scale-to-zero` (global)	true	Master toggle. Set to false to disable scale-to-zero cluster-wide.

Concurrency-Based Scaling

Knative's KPA scales based on observed concurrency (in-flight requests) or requests per second (RPS). The algorithm maintains two windows:

Stable window (default 60s): Average concurrency over this period drives the steady-state scale decision.
Panic window (default 6s, i.e., 10% of stable): If average concurrency in this window exceeds the panic threshold (default 200% of target), the system scales up aggressively.

Key annotations:

Annotation	Example	Description
`autoscaling.knative.dev/metric`	`"concurrency"` or `"rps"`	Which metric to scale on.
`autoscaling.knative.dev/target`	`"100"`	Target value for the metric (e.g., 100 concurrent requests per pod).
`autoscaling.knative.dev/target-utilization-percentage`	`"70"`	The Autoscaler aims to keep average utilization at this percentage of the target. Effective target = target * utilization / 100.
`spec.template.spec.containerConcurrency`	`0`	Per-Pod forwarded-request concurrency limit;0 disables this hard cap. Bounded waiting/rejection and autoscaling still apply.1 does not make application threads or all replicas globally serial.

Scaling formula:

desiredReplicas = ceil( observedConcurrency / (target * targetUtilization / 100) )

For example, with target=100, targetUtilization=70%, and 350 observed concurrent requests:

desiredReplicas = ceil(350 / (100 * 0.70)) = ceil(350 / 70) = ceil(5.0) = 5

Cold Start Optimization

Cold starts -- the latency penalty when scaling from zero -- are a common concern. Knative provides several mechanisms to mitigate them:

Strategy	Configuration	Trade-off
minScale	`autoscaling.knative.dev/min-scale: "2"`	Keeps baseline capacity during normal idle operation; new Revisions, restarts and scale-out can still initialize cold.
initialScale	`autoscaling.knative.dev/initial-scale: "3"`	Number of pods created when a new Revision is first deployed. Does not prevent scale-to-zero later.
scale-down-delay	`autoscaling.knative.dev/scale-down-delay: "15m"`	Delays scale-down decisions. Useful for bursty workloads to avoid frequent cold starts.
Container image caching	Use EKS node-level image caching or pre-pull DaemonSets	Reduces container pull time during cold start.
Lightweight base images	Use distroless or Alpine-based images	Reduces image size and pull time.
Application warmup	Implement readiness probes for the actual readiness contract	Avoids advertising readiness prematurely; full-load performance must still be measured.

yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: latency-critical-api
  namespace: knative-demo
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: '3'
        autoscaling.knative.dev/initial-scale: '5'
        autoscaling.knative.dev/scale-down-delay: 10m
        autoscaling.knative.dev/target: '50'
        autoscaling.knative.dev/window: 30s
    spec:
      containerConcurrency: 100
      containers:
      - image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/api:v1.0.0
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 5

Private and Public Services

By default, Knative Services are exposed externally through the ingress gateway. You can make a Service cluster-internal only:

yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: internal-processor
  namespace: knative-demo
  labels:
    networking.knative.dev/visibility: cluster-local
spec:
  template:
    spec:
      containers:
      - image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/processor:v1.0.0

The cluster-local label changes Knative routing visibility and generated internal URLs. It is not an authentication policy or a complete network-isolation boundary; private network paths, NetworkPolicy and application authorization still matter.

There is no config-network.visibility setting in the selected release. Apply the visibility label on each Service or enforce it with an appropriate admission policy. default-external-scheme only affects generated external URL schemes; it does not make a Service private or configure TLS by itself. The following is a Service metadata fragment:

yaml

metadata:
  labels:
    networking.knative.dev/visibility: cluster-local

Knative Eventing Deep Dive

Event Sources

Event Sources are Knative resources that connect external systems to the eventing mesh. Each Source emits CloudEvents to a configured sink (a Broker, Channel, or directly to a Knative Service).

ApiServerSource

Watches the Kubernetes API server for resource events and forwards them as CloudEvents:

yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-events-sa
  namespace: knative-demo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: k8s-events-reader
  namespace: knative-demo
rules:
- apiGroups:
  - ''
  resources:
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: k8s-events-reader
  namespace: knative-demo
subjects:
- kind: ServiceAccount
  name: k8s-events-sa
  namespace: knative-demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: k8s-events-reader
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: knative-demo-namespace-discovery
rules:
- apiGroups:
  - ''
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: knative-demo-namespace-discovery
subjects:
- kind: ServiceAccount
  name: k8s-events-sa
  namespace: knative-demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: knative-demo-namespace-discovery
---
apiVersion: sources.knative.dev/v1
kind: ApiServerSource
metadata:
  name: k8s-events
  namespace: knative-demo
spec:
  serviceAccountName: k8s-events-sa
  mode: Reference
  resources:
  - apiVersion: v1
    kind: Pod
  - apiVersion: apps/v1
    kind: Deployment
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: default
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: knative-demo

SinkBinding

The selected Eventing configuration uses inclusion mode. Opt only the demo namespace into binding injection, and ensure the subject Deployment exists in that namespace with matching metadata labels/name. Changes to an already-running workload may require its normal rollout before new injected environment is used:

bash

kubectl label namespace knative-demo bindings.knative.dev/include=true --overwrite

Injects environment variables (specifically K_SINK) into any Kubernetes workload so it can send events to a sink without hardcoding the destination:

yaml

apiVersion: sources.knative.dev/v1
kind: SinkBinding
metadata:
  name: order-producer-binding
  namespace: knative-demo
spec:
  subject:
    apiVersion: apps/v1
    kind: Deployment
    name: order-producer
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: default
  ceOverrides:
    extensions:
      producer: order-system

The application must read K_SINK and apply K_CE_OVERRIDES; SinkBinding does not rewrite arbitrary outgoing Python requests. The producer below uses structured HTTP encoding (the complete CloudEvent body with the matching content type), a bounded timeout and status checks. Keep one event_id across retries of the same logical event:

python

import json
import os
import re

import requests
from cloudevents.http import CloudEvent
from cloudevents.conversion import to_structured

CORE_ATTRIBUTES = {"specversion", "id", "source", "type", "time", "subject", "datacontenttype", "dataschema"}


def emit_order_event(order_id, event_type, data, event_id):
    """Use one stable event_id for retries of the same logical event."""
    attributes = {
        "specversion": "1.0", "id": event_id,
        "type": f"com.example.order.{event_type}",
        "source": "/orders/api", "subject": f"order/{order_id}",
        "datacontenttype": "application/json",
    }
    overrides = json.loads(os.environ.get("K_CE_OVERRIDES", "{}"))
    for key, value in overrides.get("extensions", {}).items():
        if key in CORE_ATTRIBUTES or not re.fullmatch(r"[a-z0-9]+", key):
            raise ValueError("Only valid extension attributes may be overridden")
        attributes[key] = value
    event = CloudEvent(attributes, data)
    headers, body = to_structured(event)
    response = requests.post(
        os.environ["K_SINK"], data=body, headers=headers, timeout=(3, 10)
    )
    response.raise_for_status()
    return response.status_code

KafkaSource

KafkaSource/KafkaChannel require the matching Kafka extension and an existing configured Kafka cluster (including bootstrap/TLS/SASL and sufficient brokers for the requested replication factor). Merge these1.23.1 addon URLs with any existing additionalManifests in the KnativeEventing resource; do not simultaneously enable another management path for the same addon. Save as eventing-kafka.patch.yaml:

yaml

spec:
  additionalManifests:
  - URL: https://github.com/knative-extensions/eventing-kafka-broker/releases/download/knative-v1.23.1/eventing-kafka-controller.yaml
  - URL: https://github.com/knative-extensions/eventing-kafka-broker/releases/download/knative-v1.23.1/eventing-kafka-source.yaml
  - URL: https://github.com/knative-extensions/eventing-kafka-broker/releases/download/knative-v1.23.1/eventing-kafka-channel.yaml

bash

kubectl patch knativeeventing knative-eventing -n knative-eventing --type merge --patch-file eventing-kafka.patch.yaml
GENERATION=$(kubectl get knativeeventing knative-eventing -n knative-eventing -o jsonpath='{.metadata.generation}')
kubectl wait --for=jsonpath='{.status.observedGeneration}'="$GENERATION" knativeeventing/knative-eventing -n knative-eventing --timeout=600s
kubectl wait --for=condition=Ready knativeeventing/knative-eventing -n knative-eventing --timeout=600s
kubectl get crd kafkasources.sources.knative.dev kafkachannels.messaging.knative.dev

Consumes messages from Apache Kafka topics and delivers them as CloudEvents:

yaml

apiVersion: sources.knative.dev/v1
kind: KafkaSource
metadata:
  name: payment-events
  namespace: knative-demo
spec:
  consumerGroup: knative-payment-consumer
  bootstrapServers:
  - kafka-bootstrap.kafka.svc.cluster.local:9092
  topics:
  - payment-events
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: default
  net:
    sasl:
      enable: true
      type:
        secretKeyRef:
          name: kafka-credentials
          key: sasl-type
      user:
        secretKeyRef:
          name: kafka-credentials
          key: username
      password:
        secretKeyRef:
          name: kafka-credentials
          key: password
    tls:
      enable: true

SQSSource (AWS)

The1.23 core IntegrationSource supports SQS and replaces the archived TriggerMesh example here. It is an alpha API; verify the installed CRD and adapter images. The dedicated queue and its IAM role must already exist. This source consumes/deletes messages when run; no source was deployed during the audit:

bash

# Verify the core IntegrationSource CRD
kubectl get crd integrationsources.sources.knative.dev

yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sqs-event-source
  namespace: knative-demo
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/KnativeSqsSourceRole
---
apiVersion: sources.knative.dev/v1alpha1
kind: IntegrationSource
metadata:
  name: sqs-order-events
  namespace: knative-demo
spec:
  aws:
    sqs:
      arn: arn:aws:sqs:us-west-2:123456789012:knative-demo-orders
      region: us-west-2
      autoCreateQueue: false
      deleteAfterRead: true
      visibilityTimeout: 120
    auth:
      serviceAccountName: sqs-event-source
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: default

The example uses a dedicated ServiceAccount with an existing IRSA role. Scope ReceiveMessage/DeleteMessage/GetQueueAttributes/GetQueueUrl permissions to the test queue and validate acknowledgements, visibility timeouts and failure behavior. autoCreateQueue is false. The emitted CloudEvent type/source comes from the adapter; inspect it before copying the manual order-event Trigger filters. EKS Pod Identity is another credential option where the selected EKS compute, agent and SDK support it; Fargate does not support EKS Pod Identity.

Broker/Trigger Pattern

The Broker/Trigger pattern provides content-based event routing. A Broker acts as an event hub; Triggers filter events by CloudEvent attributes and route them to subscribers.

Complete Broker/Trigger Example

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-broker-channel
  namespace: knative-demo
data:
  channel-template-spec: |
    apiVersion: messaging.knative.dev/v1
    kind: InMemoryChannel
---
apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
  name: default
  namespace: knative-demo
  annotations:
    eventing.knative.dev/broker.class: MTChannelBasedBroker
spec:
  config:
    apiVersion: v1
    kind: ConfigMap
    name: demo-broker-channel
    namespace: knative-demo
  delivery:
    retry: 5
    backoffPolicy: exponential
    backoffDelay: PT2S
    deadLetterSink:
      ref:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: dead-letter-handler
---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: order-created-trigger
  namespace: knative-demo
spec:
  broker: default
  filter:
    attributes:
      type: com.example.order.created
      source: /orders/api
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: order-processor
---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: payment-processed-trigger
  namespace: knative-demo
spec:
  broker: default
  filter:
    attributes:
      type: com.example.payment.processed
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: payment-reconciler
  delivery:
    retry: 10
    backoffPolicy: exponential
    backoffDelay: PT5S
    deadLetterSink:
      ref:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: payment-dead-letter
---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: analytics-trigger
  namespace: knative-demo
spec:
  broker: default
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: analytics-collector

CloudEvents Standard

Knative Eventing event delivery uses the CloudEvents specification (v1.0); Kubernetes control-plane operations retain their own APIs. CloudEvents defines a common envelope with required and optional attributes:

Attribute	Required	Example	Description
`specversion`	Yes	`"1.0"`	CloudEvents specification version.
`type`	Yes	`"com.example.order.created"`	Event type. Used for routing by Triggers.
`source`	Yes	`"/orders/api"`	Event origin. Combined with type for filtering.
`id`	Yes	`"evt-abc123"`	Identifier unique within its source; deduplicate with `(source, id)`.
`time`	No	`"2025-06-15T10:30:00Z"`	Timestamp of event occurrence.
`datacontenttype`	No	`"application/json"`	Content type of the `data` attribute.
`subject`	No	`"order-12345"`	Subject of the event in context of the source.
`data`	No	`{"orderId": "12345"}`	Event payload.

Receipt-only consumer example

Save this separate Flask application as app.py and run it with a WSGI server. It validates CloudEvents and acknowledges receipt; it does not implement order/payment transactions or durable processing. Application consumers must complete their intended transaction/idempotency step before acknowledging.

python

import json

from flask import Flask, request
from cloudevents.http import from_http


def create_app():
    app = Flask(__name__)
    app.config["MAX_CONTENT_LENGTH"] = 1024 * 1024

    @app.post("/")
    def receive_event():
        try:
            event = from_http(request.headers, request.get_data())
            metadata = {key: event[key] for key in ("source", "id", "type")}
        except Exception:
            # This boundary converts malformed input into a client error.
            return "invalid CloudEvent", 400
        app.logger.info("Received CloudEvent metadata: %s", json.dumps(metadata))
        # Receipt-only demo. Real consumers must commit processing before acknowledging.
        return "", 204

    return app

Channel/Subscription Pattern

Create all referenced subscribers/reply Channels first. A reply destination receives a valid CloudEvent returned by the subscriber; an empty204 receipt does not produce a reply event. Kafka durability also depends on broker replication/retention/acknowledgements and failure handling, not merely the KafkaChannel kind.

The Channel/Subscription pattern provides direct pub-sub without content-based filtering. Every Subscription on a Channel receives every event.

yaml

apiVersion: messaging.knative.dev/v1
kind: KafkaChannel
metadata:
  name: audit-events
  namespace: knative-demo
spec:
  numPartitions: 6
  replicationFactor: 3
  retentionDuration: PT168H
---
apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
  name: audit-log-subscription
  namespace: knative-demo
spec:
  channel:
    apiVersion: messaging.knative.dev/v1
    kind: KafkaChannel
    name: audit-events
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: audit-logger
  reply:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: audit-response-handler
---
apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
  name: compliance-subscription
  namespace: knative-demo
spec:
  channel:
    apiVersion: messaging.knative.dev/v1
    kind: KafkaChannel
    name: audit-events
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: compliance-checker
  delivery:
    deadLetterSink:
      ref:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: dead-letter-handler
    retry: 3
    backoffPolicy: linear
    backoffDelay: PT10S

Dead Letter Sink

When the configured delivery policy cannot deliver an event to its subscriber, the implementation can attempt the configured DLS. The handler must persist/process before acknowledging; a DLS Service name or S3_BUCKET variable alone does not create storage behavior, permissions or a no-loss guarantee.

yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: dead-letter-writer
  namespace: knative-demo
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/KnativeDeadLetterWriterRole
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: dead-letter-handler
  namespace: knative-demo
  labels:
    networking.knative.dev/visibility: cluster-local
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: '1'
    spec:
      containers:
      - image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/dead-letter:v1.0.0
        env:
        - name: S3_BUCKET
          value: REPLACE_WITH_EXISTING_BUCKET
        - name: AWS_REGION
          value: us-west-2
        ports:
        - containerPort: 8080
        command:
        - gunicorn
        args:
        - --bind
        - 0.0.0.0:8080
        - --workers
        - '1'
        - --threads
        - '4'
        - --timeout
        - '90'
        - --graceful-timeout
        - '60'
        - app:create_app()
      serviceAccountName: dead-letter-writer
      timeoutSeconds: 60

Delivery settings can exist at Broker, Trigger and Subscription levels, with implementation-specific support and inheritance. Verify the selected class and status, including the resolved DLS. A configured DLS can also fail; it is not a no-loss guarantee.

Example S3-backed dead-letter handler

Build this separate app.py with Flask, CloudEvents, boto3 and Gunicorn. Use an existing protected bucket, a dedicated workload ServiceAccount with scoped s3:PutObject permission, required KMS permissions and a verified network path. Configure S3_BUCKET and AWS_REGION; do not embed static AWS credentials. The container must run gunicorn --bind 0.0.0.0:8080 --workers 1 --threads 4 --timeout 90 --graceful-timeout 60 app:create_app(). Align these example timeouts with Knative/proxy/Pod termination settings.

python

import base64
import hashlib
import json
import os

import boto3
from botocore.config import Config
from botocore.exceptions import BotoCoreError, ClientError
from cloudevents.http import from_http
from flask import Flask, request


def create_app(s3_client=None):
    app = Flask(__name__)
    app.config["MAX_CONTENT_LENGTH"] = 1024 * 1024
    bucket = os.environ["S3_BUCKET"]
    if s3_client is None:
        # Create once per application worker; IRSA/Pod Identity uses the credential chain.
        s3_client = boto3.client(
            "s3", region_name=os.environ["AWS_REGION"],
            config=Config(connect_timeout=3, read_timeout=10,
                          retries={"mode": "standard", "total_max_attempts": 2}),
        )

    @app.post("/")
    def store_dead_letter():
        raw_body = request.get_data()
        try:
            event = from_http(request.headers, raw_body)
            source, event_id = str(event["source"]), str(event["id"])
        except Exception:
            return "invalid CloudEvent", 400
        identity = json.dumps([source, event_id], ensure_ascii=False,
                              separators=(",", ":")).encode("utf-8")
        key = "dead-letters/" + hashlib.sha256(identity).hexdigest() + ".json"
        record = {
            "source": source, "id": event_id,
            "content_type": request.headers.get("Content-Type", "application/octet-stream"),
            # Preserve CloudEvents transport attributes, never Authorization/Cookie headers.
            "ce_headers": {k.lower(): v for k, v in request.headers.items()
                           if k.lower().startswith("ce-")},
            "body_base64": base64.b64encode(raw_body).decode("ascii"),
        }
        try:
            s3_client.put_object(
                Bucket=bucket, Key=key,
                Body=json.dumps(record, ensure_ascii=False).encode("utf-8"),
                ContentType="application/json", IfNoneMatch="*",
            )
        except ClientError as exc:
            # HTTP boundary: acknowledge a stored duplicate; retry other storage failures.
            status = exc.response.get("ResponseMetadata", {}).get("HTTPStatusCode")
            if status == 412:
                return "", 204
            app.logger.error("DLS storage failed: %s", exc.response.get("Error", {}).get("Code"))
            return "storage unavailable", 503
        except BotoCoreError as exc:
            app.logger.error("DLS storage unavailable: %s", type(exc).__name__)
            return "storage unavailable", 503
        return "", 204

    return app

The handler stores the original body as Base64 and only CloudEvents transport headers, so binary events are preserved and Authorization/Cookie headers are excluded. A deterministic (source, id) key plus conditional put avoids overwriting an existing event;412 acknowledges an existing key, while409 and other storage failures return503 for delivery-policy retry. Retention/deletion and producer ID reuse affect deduplication; this is not an exactly-once business-processing guarantee. Requests over1MiB are rejected. The code was tested with local mocks, not a live bucket or Eventing deployment.

Event Filtering

Triggers support filtering on CloudEvent attributes and extensions.

Attribute Filtering

yaml

spec:
  filter:
    attributes:
      type: com.example.order.created
      source: /orders/api

This Trigger fires only when both type AND source match (logical AND).

Extension Filtering

You can filter on custom CloudEvent extensions set by producers:

yaml

spec:
  filter:
    attributes:
      type: com.example.order.created
      myextension: priority-high

Multiple Triggers for OR Logic

Legacy spec.filter.attributes combines attributes with AND. Current spec.filters supports richer expressions such as any, all and not. Multiple Triggers are another option, but overlapping matches may deliver duplicates. Use one filter form at a time and verify broker support:

yaml

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: order-created
  namespace: knative-demo
spec:
  broker: default
  filter:
    attributes:
      type: com.example.order.created
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: notification-service
---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: order-cancelled
  namespace: knative-demo
spec:
  broker: default
  filter:
    attributes:
      type: com.example.order.cancelled
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: notification-service

KEDA vs Knative Comparison

Both KEDA and Knative enable event-driven scaling on Kubernetes, but they operate at different levels of abstraction and serve complementary roles.

Scaling Model Differences

Aspect	KEDA	Knative
Abstraction level	Extends HPA with custom metric sources	Full serverless platform (deployment, routing, scaling)
Scaling mechanism	ScaledObject: operator activation plus HPA; ScaledJob: direct Job creation	KPA or the optional HPA extension
Primary metric	External metrics (queue depth, DB rows, custom)	HTTP concurrency / RPS
Workload type	Any Deployment, StatefulSet, Job	Knative Service (manages its own Deployment)
CRDs	ScaledObject, ScaledJob, TriggerAuthentication	Service, Configuration, Revision, Route
Built-in routing	No	Yes (traffic splitting, revisions, canary)
Built-in eventing	No (focuses on scaling only)	Yes (Broker/Trigger, Channel/Subscription)

Scale-to-Zero Behavior Differences

Behavior	KEDA	Knative (KPA)
Scale-to-zero trigger	Configured activation triggers inactive with zero enabled; cooldown/error behavior also matters	Idle/scale-down eligibility, retention and routing readiness
Activation mechanism	Operator activates nonzero replicas according to trigger/minimum/idle settings	Activator buffers within bounds and drives activation
Request buffering	Core KEDA does not buffer HTTP; a separate HTTP add-on exists	Activator buffers within capacity and timeout limits
Cool-down period	`cooldownPeriod` on ScaledObject	`scale-to-zero-grace-period` + `stable-window`
Jobs	ScaledJob stops creating new Jobs when inactive; existing Jobs may continue	Serving manages HTTP workload Revisions, not batch Job lifecycles

Roles in Event-Driven Architecture

KEDA ScaledObject uses operator activation and HPA for nonzero worker replicas; ScaledJob creates Jobs separately. Knative Eventing routes CloudEvents to configured consumers, whose scaling and buffering depend on their implementation.

When to Use KEDA vs Knative

Use Case	Recommended	Reason
Scale workers based on SQS queue depth	KEDA	KEDA has a native SQS scaler; no HTTP routing needed.
Deploy HTTP APIs with auto-scaling and traffic splitting	Knative	Serving provides revision management, traffic splitting, and HTTP-aware autoscaling.
Scale based on Prometheus metrics	KEDA	KEDA's Prometheus scaler is mature and well-tested.
Event-driven microservices with CloudEvents	Knative	Eventing provides Broker/Trigger, dead letter handling, and CloudEvents support.
Scheduled Jobs / event-driven batch	CronJob / KEDA	Use Kubernetes CronJob for a Job on a schedule; ScaledJob creates Jobs for event demand. A KEDA Cron trigger supplies a window-based replica target rather than scaling a CronJob.
CPU/memory plus zero activation	Requires an external activation signal	CPU/memory-only metrics cannot measure an absent worker; combine an appropriate event trigger rather than promising zero activation from those metrics alone.
Serverless platform for developers	Knative	Higher-level abstraction; developers deploy with `kn service create`.

Using KEDA and Knative Together

KEDA and Knative are not mutually exclusive. A common architecture uses:

Knative Serving for HTTP-facing services (APIs, web applications) with concurrency-based autoscaling.
KEDA for background workers (queue consumers, batch processors) with external-metric-based autoscaling.
Knative Eventing for event routing. SinkBinding configures producers’ destination/overrides; it does not make an arbitrary zero-replica HTTP worker activatable. Use a suitable queue/adapter path and one scaling owner per workload.

yaml

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-aws-credentials
  namespace: knative-demo
spec:
  podIdentity:
    provider: aws
    identityOwner: keda
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: event-enricher
  namespace: knative-demo
spec:
  template:
    spec:
      containers:
      - image: event-enricher:v1
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-worker-scaler
  namespace: knative-demo
spec:
  scaleTargetRef:
    name: sqs-worker
  minReplicaCount: 0
  maxReplicaCount: 100
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: https://sqs.us-west-2.amazonaws.com/123456789012/enriched-events
      queueLength: '5'
      awsRegion: us-west-2
    authenticationRef:
      name: keda-aws-credentials

Production Operations

Resource Limits and QoS

Plan requests and limits for application containers and Queue Proxy from measured workload needs. QoS depends on all relevant containers: Guaranteed requires matching nonzero CPU/memory requests and limits; other specified budgets are often Burstable. Neither class prevents OOM, eviction or interference, and a memory limit can itself trigger an OOM kill.

yaml

spec:
  config:
    deployment:
      queue-sidecar-cpu-request: 50m
      queue-sidecar-cpu-limit: 500m
      queue-sidecar-memory-request: 400Mi
      queue-sidecar-memory-limit: 800Mi

Save as serving-deployment.patch.yaml and merge into the Operator resource; it is not a standalone ConfigMap.

bash

kubectl patch knativeserving knative-serving -n knative-serving --type merge --patch-file serving-deployment.patch.yaml

Revision Garbage Collection

Over time, old Revisions accumulate. Configure garbage collection to limit the number of retained Revisions:

yaml

spec:
  config:
    gc:
      min-non-active-revisions: '2'
      max-non-active-revisions: '10'
      retain-since-create-time: 48h
      retain-since-last-active-time: 24h

Save as serving-gc.patch.yaml and merge into the Operator resource; it is not a standalone ConfigMap.

bash

kubectl patch knativeserving knative-serving -n knative-serving --type merge --patch-file serving-gc.patch.yaml

High Availability Configuration

For production workloads, configure Knative Serving components for high availability:

yaml

spec:
  version: 1.23.0
  high-availability:
    replicas: 3
  ingress:
    kourier:
      enabled: true
  workloads:
  - name: activator
    replicas: 3
    resources:
    - container: activator
      requests:
        cpu: 300m
        memory: 256Mi
      limits:
        cpu: 1000m
        memory: 512Mi
  - name: controller
    replicas: 2
  - name: webhook
    replicas: 2

Save this Operator merge patch as serving-ha.patch.yaml; preserve existing workloads/list entries. Verify placement, resources and failure behavior rather than assuming replicas alone establish HA.

bash

kubectl patch knativeserving knative-serving -n knative-serving --type merge --patch-file serving-ha.patch.yaml

Additionally, configure Pod Disruption Budgets for Knative system components:

yaml

spec:
  podDisruptionBudgets:
  - name: activator-pdb
    minAvailable: 2
  - name: webhook-pdb
    minAvailable: 1

Save as serving-pdb.patch.yaml and patch the KnativeServing resource. These override existing Operator-managed PDBs; they limit voluntary eviction and do not guarantee service availability.

bash

kubectl patch knativeserving knative-serving -n knative-serving --type merge --patch-file serving-pdb.patch.yaml

Spread system pods across Availability Zones using topology constraints:

yaml

spec:
  workloads:
  - name: activator
    topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: activator
      minDomains: 3

This Operator fragment requires three eligible zones and enough capacity; DoNotSchedule can leave Pods pending. Merge it with existing workloads overrides before applying.

Monitoring with Prometheus

Knative1.23 uses OpenTelemetry and metric export is off by default. This example assumes an existing Prometheus/Prometheus Operator deployment. Enable its OTLP receiver (--web.enable-otlp-receiver, or the equivalent supported Operator setting), restrict access appropriately, and configure resource-attribute promotion. The following is a native Prometheus configuration fragment, not Helm values; check support in your installed Prometheus version:

yaml

otlp:
  translation_strategy: UnderscoreEscapingWithSuffixes
  convert_histograms_to_nhcb: false
  promote_resource_attributes:
  - k8s.namespace.name
  - k8s.pod.name
  - kn.service.name
  - kn.configuration.name
  - kn.revision.name

Configure the Operator-managed Serving resource with this merge patch, saved as serving-metrics.patch.yaml. Replace the endpoint with the actual OTLP receiver; it is not a pre-existing guaranteed service name. This setup scrapes control-plane metrics and pushes request metrics, so it does not scrape Queue Proxy’s admin port:

yaml

spec:
  config:
    observability:
      metrics-protocol: prometheus
      request-metrics-protocol: http/protobuf
      request-metrics-endpoint: http://prometheus-operated.monitoring.svc.cluster.local:9090/api/v1/otlp/v1/metrics
      request-metrics-export-interval: 10s

bash

kubectl patch knativeserving knative-serving -n knative-serving --type merge --patch-file serving-metrics.patch.yaml
kubectl patch knativeeventing knative-eventing -n knative-eventing --type merge --patch '{"spec":{"config":{"observability":{"metrics-protocol":"prometheus"}}}}'
kubectl get service prometheus-operated -n monitoring

The ServiceMonitor selector below matches the released control-plane Services and their http-metrics Service port. Its namespace/labels must also match the Prometheus resource’s monitor selectors. Verify the installed endpoints after configuration or any required rollout:

yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: knative-serving-control-plane
  namespace: monitoring
  labels:
    release: prometheus
spec:
  namespaceSelector:
    matchNames:
    - knative-serving
  selector:
    matchExpressions:
    - key: app
      operator: In
      values:
      - controller
      - webhook
      - autoscaler
      - activator
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
    honorLabels: true

Current metric vocabulary (OTel instrument names; the selected Prometheus translation replaces dots and adds real unit suffixes):

Instrument	Meaning
`kn.serving.invocation.duration`	Request completion duration histogram in seconds; its count can derive completed request rate.
`kn.serving.queue.depth`	Queue Proxy queue/in-flight sample; not an exact always-current global concurrency count.
`kn.revision.pods.desired` / `kn.revision.pods.requested` / `kn.revision.pods.count`	Autoscaler desired, requested and currently allocated Pod gauges.
`kn.revision.pods.not_ready.count` / `kn.revision.pods.pending.count`	Unready/pending Pod gauges.
`kn.revision.concurrency.stable` / `kn.revision.concurrency.panic`	Per-observed-Pod concurrency averages over the two windows; do not divide by Pod count again.
`kn.revision.request.concurrency`	Requests routed through Activator; Activator traffic is not equivalent to a cold-start counter.
`kn.workqueue.depth` / `kn.workqueue.process.duration`	Controller queue depth and processing duration.

Eventing broker/source/backend metrics vary by implementation. Its official metric page warns that parts of the old OpenCensus table have not completed migration; do not assume broker_event_count or trigger_filter_event_count exists. Inspect the selected implementation’s exported metrics and delivery status. No live scrape or trace export was performed in this audit.

Grafana Dashboard

This is a dashboard JSON definition, not the HTTP API dashboard wrapper. Replace the datasource UID before import. Queries assume the translation strategy, classic histogram buckets and promoted attributes above; endpoint/scrape labels and aggregation must be checked in your deployment. The panic-window series is not a Boolean panic-mode indicator, and Activator activity can occur while replicas are nonzero:

json

{
  "id": null,
  "uid": "knative-demo-overview",
  "title": "Knative Demo Overview",
  "schemaVersion": 39,
  "version": 1,
  "refresh": "30s",
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "panels": [
    {
      "id": 1,
      "title": "Completed Request Rate",
      "type": "timeseries",
      "gridPos": {
        "x": 0,
        "y": 0,
        "w": 12,
        "h": 8
      },
      "datasource": {
        "type": "prometheus",
        "uid": "REPLACE_WITH_PROMETHEUS_DATASOURCE_UID"
      },
      "fieldConfig": {
        "defaults": {
          "unit": "reqps"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "expr": "sum by (k8s_namespace_name, kn_revision_name) (rate(kn_serving_invocation_duration_seconds_count{k8s_namespace_name=\"knative-demo\"}[5m]))",
          "legendFormat": "{{kn_revision_name}}"
        }
      ]
    },
    {
      "id": 2,
      "title": "Request Duration P99",
      "type": "timeseries",
      "gridPos": {
        "x": 12,
        "y": 0,
        "w": 12,
        "h": 8
      },
      "datasource": {
        "type": "prometheus",
        "uid": "REPLACE_WITH_PROMETHEUS_DATASOURCE_UID"
      },
      "fieldConfig": {
        "defaults": {
          "unit": "s"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "expr": "histogram_quantile(0.99, sum by (le, k8s_namespace_name, kn_revision_name) (rate(kn_serving_invocation_duration_seconds_bucket{k8s_namespace_name=\"knative-demo\"}[5m])))",
          "legendFormat": "{{kn_revision_name}}"
        }
      ]
    },
    {
      "id": 3,
      "title": "Queue Depth Sample",
      "type": "timeseries",
      "gridPos": {
        "x": 0,
        "y": 8,
        "w": 12,
        "h": 8
      },
      "datasource": {
        "type": "prometheus",
        "uid": "REPLACE_WITH_PROMETHEUS_DATASOURCE_UID"
      },
      "fieldConfig": {
        "defaults": {
          "unit": "short"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "expr": "max by (k8s_namespace_name, kn_revision_name) (kn_serving_queue_depth{k8s_namespace_name=\"knative-demo\"})",
          "legendFormat": "{{kn_revision_name}}"
        }
      ]
    },
    {
      "id": 4,
      "title": "Desired and Actual Pods",
      "type": "timeseries",
      "gridPos": {
        "x": 12,
        "y": 8,
        "w": 12,
        "h": 8
      },
      "datasource": {
        "type": "prometheus",
        "uid": "REPLACE_WITH_PROMETHEUS_DATASOURCE_UID"
      },
      "fieldConfig": {
        "defaults": {
          "unit": "short"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "expr": "max by (k8s_namespace_name, kn_revision_name) (kn_revision_pods_desired{k8s_namespace_name=\"knative-demo\"})",
          "legendFormat": "{{kn_revision_name}}"
        },
        {
          "refId": "B",
          "expr": "max by (k8s_namespace_name, kn_revision_name) (kn_revision_pods_count{k8s_namespace_name=\"knative-demo\"})",
          "legendFormat": "actual {{kn_revision_name}}"
        }
      ]
    },
    {
      "id": 5,
      "title": "Stable and Panic Window Concurrency",
      "type": "timeseries",
      "gridPos": {
        "x": 0,
        "y": 16,
        "w": 12,
        "h": 8
      },
      "datasource": {
        "type": "prometheus",
        "uid": "REPLACE_WITH_PROMETHEUS_DATASOURCE_UID"
      },
      "fieldConfig": {
        "defaults": {
          "unit": "short"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "expr": "max by (k8s_namespace_name, kn_revision_name) (kn_revision_concurrency_stable{k8s_namespace_name=\"knative-demo\"})",
          "legendFormat": "{{kn_revision_name}}"
        },
        {
          "refId": "B",
          "expr": "max by (k8s_namespace_name, kn_revision_name) (kn_revision_concurrency_panic{k8s_namespace_name=\"knative-demo\"})",
          "legendFormat": "panic window {{kn_revision_name}}"
        }
      ]
    },
    {
      "id": 6,
      "title": "Requests Through Activator",
      "type": "timeseries",
      "gridPos": {
        "x": 12,
        "y": 16,
        "w": 12,
        "h": 8
      },
      "datasource": {
        "type": "prometheus",
        "uid": "REPLACE_WITH_PROMETHEUS_DATASOURCE_UID"
      },
      "fieldConfig": {
        "defaults": {
          "unit": "short"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "expr": "max by (k8s_namespace_name, kn_revision_name) (kn_revision_request_concurrency{k8s_namespace_name=\"knative-demo\"})",
          "legendFormat": "{{kn_revision_name}}"
        }
      ]
    }
  ]
}

Save as knative-overview.json. A Grafana sidecar can load it from a ConfigMap only if its label/namespace selection is configured; do not embed an extra dashboard wrapper. Grafana rendering and actual metric ingestion remain unverified.

bash

kubectl create configmap knative-serving-dashboard -n monitoring --from-file=knative-serving.json=knative-overview.json --dry-run=client -o yaml | kubectl apply -f -
kubectl label configmap knative-serving-dashboard -n monitoring grafana_dashboard=1 --overwrite

Troubleshooting

Revision-scoped annotation changes must clear the existing spec.template.metadata.name (null in a JSON merge patch) or use a new, unused Revision name. To apply the new configuration to the entire demo Service, set spec.traffic to [{"latestRevision": true, "percent": 100}]. This replaces any canary split or pinned Revision routing; if those routes must stay in place, validate the new Revision on a separate test route before switching traffic.

Cold Start Latency Is Too High

Symptoms: First request after idle period takes several seconds.

Diagnosis:

bash

# Check if the Revision is scaled to zero
kubectl get ksvc order-api -n knative-demo -o jsonpath='{.status.conditions}' | jq .

# Check Activator logs for buffering duration
kubectl logs -l app=activator -n knative-serving --tail=50

# Check pod startup time
kubectl get pods -l serving.knative.dev/service=order-api -n knative-demo \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions}{"\n"}{end}'

Solutions:

Set autoscaling.knative.dev/min-scale: "1" to keep at least one pod warm.
Reduce container image size.
Use readiness probes with short intervals.
Pre-pull images using a DaemonSet.

Scaling Is Too Slow or Oscillating

Symptoms: Pod count does not keep up with load, or scales up and down repeatedly.

Diagnosis:

bash

# Check Autoscaler metrics
kubectl logs -l app=autoscaler -n knative-serving --tail=100

# View current scale decisions
kubectl get podautoscaler -n knative-demo
kubectl describe podautoscaler order-api-00001 -n knative-demo

Solutions:

Reduce the stable-window for faster reactions (e.g., 30s).
Lower target utilization when more spare capacity is required; raising it generally asks each Pod to handle more load before scaling.
Adjust panic-window-percentage and panic-threshold-percentage for burst handling.
HPA synchronization is a separate control-plane setting; increasing its period slows sampling. Do not assume EKS exposes that flag. Inspect supported HPA behavior, metric availability and workload limits.

Events Not Being Delivered

Symptoms: Events are produced but Triggers do not fire.

Diagnosis:

bash

# Verify Broker is ready
kubectl get broker default -n knative-demo -o yaml

# Check Trigger status
kubectl get triggers -n knative-demo
kubectl describe trigger order-created-trigger -n knative-demo

# Inspect Eventing controller logs
kubectl logs -l app=eventing-controller -n knative-eventing --tail=100

# Check dead letter sink for failed events
kubectl logs -l serving.knative.dev/service=dead-letter-handler -n knative-demo --tail=50

Solutions:

Verify the Trigger filter attributes match the CloudEvent attributes exactly (case-sensitive).
Check that the subscriber Service is ready and reachable.
Ensure the Broker's backing channel is healthy.
Distinguish Kubernetes API permissions for reading source resources from HTTP/Eventing authentication and authorization at the Broker. Inspect the actual HTTP error and configured EventPolicies/identity mechanism.

DNS Resolution Failures

Symptoms: Knative Service URLs return NXDOMAIN or connection timeouts.

Diagnosis:

bash

# Verify Kourier service has an external address
kubectl get svc kourier -n kourier-system

# Check config-domain
kubectl get cm config-domain -n knative-serving -o yaml

# Test DNS resolution
nslookup order-api.knative-demo.knative.example.com

# Check the Knative Service URL
kubectl get ksvc order-api -n knative-demo -o jsonpath='{.status.url}'

Solutions:

For sslip.io: ensure the external IP is reachable and port 80/443 is not blocked by security groups.
For Route 53: verify the wildcard CNAME record resolves to the Kourier load balancer.
Check that config-domain has the correct domain entry.

Best Practices

Service Design Patterns

One container per Knative Service. Knative Services are designed for a single application container plus the Queue Proxy sidecar. Avoid multi-container pods unless absolutely necessary (Knative does support them, but the scaling model assumes a single primary container).
Use containerConcurrency deliberately. Set it to 0 (unlimited) for thread-safe applications that handle many concurrent requests. Set it to 1 for single-threaded processors (e.g., ML inference on a single GPU) where concurrent requests would degrade performance.
Separate read and write paths. Deploy read-heavy APIs and write-heavy processors as separate Knative Services with different scaling profiles. Read services may have a high target (100+ concurrency), while write services may need a low target (10-20) to avoid overwhelming the database.
Retain tested rollback targets. Keep known-good Revisions referenced/retained as needed and verify their dependencies. Changing the route takes reconciliation time:

bash

kn service update order-api --namespace knative-demo --tag order-api-v1=stable --tag @latest=canary

Control internal routing visibility. Apply the cluster-local label where appropriate, then separately enforce network/application authorization. A shared Knative gateway/load balancer may still exist, so the label does not guarantee a separate load-balancer cost reduction.

Event-Driven Microservices Patterns

For CQRS/event sourcing, the command path must commit the authoritative event or a transactional outbox before acknowledging the command. A Broker is a delivery layer, not the authoritative event store. An asynchronous archive consumer alone does not make command handling transactional; validate ordering, idempotency, replay and failure windows.

Use Brokers for multi-consumer routing. When multiple services need to react to the same event type, use a single Broker with multiple Triggers rather than duplicating the event source.
Always configure Dead Letter Sinks. Undeliverable events should never be silently dropped. Configure a DLS at the Broker level as a safety net and at individual Trigger levels for critical paths.
Adopt a CloudEvents naming convention. Use reverse-DNS notation for event types: com.<company>.<domain>.<action> (e.g., com.example.order.created). This prevents naming collisions and makes Trigger filters clear.
Idempotent consumers. Retries can deliver duplicates. Use the CloudEvent (source, id) pair and a durable application transaction/deduplication design; id is not necessarily globally unique and Eventing is not an exactly-once business-processing guarantee.
Plan a durable transport before ingestion. An InMemoryChannel is not durable. A correctly configured Kafka-backed transport can persist events, but changing a live Broker’s backing configuration is not a data migration. Create and validate a separate configuration/Broker, and plan replay/draining for any transition. The following is a channel-template fragment for that new configuration:

yaml

# config-br-default-channel ConfigMap
data:
  channel-template-spec: |
    apiVersion: messaging.knative.dev/v1
    kind: KafkaChannel
    spec:
      numPartitions: 6
      replicationFactor: 3

Cost Optimization with Scale-to-Zero

Enable scale-to-zero for non-critical services. Development, staging, and low-traffic production services should scale to zero when idle. The original60–80% figure is retained only as an unsourced historical estimate, not a measured result or current savings promise. Allocated node/storage/load-balancer costs and commitments must be evaluated separately.
Use scale-down-delay for bursty workloads. If traffic comes in bursts separated by short idle periods, setting a scale-down delay (e.g., 5-15 minutes) avoids repeated cold starts without keeping pods running indefinitely.
Combine with Karpenter for node-level efficiency. When Knative scales pods to zero, the freed capacity allows Karpenter to consolidate or terminate underutilized nodes:

Layer	Tool	Action
Application (Pods)	Knative Serving	Scale pods to zero on idle
Infrastructure (Nodes)	Karpenter	Consolidate and terminate empty nodes
Cost visibility	AWS Cost Explorer / Kubecost	Track savings from scale-to-zero

Set minScale only where needed. Reserve minScale > 0 for latency-critical paths. For everything else, let pods scale to zero.

Original Cost Illustrations (Unverified)

The original illustrative figures below are preserved without treating them as measurements or current pricing. Pod-hours are not the same as billed node-hours, and these figures omit control-plane, storage, load-balancer and commitment costs.

Environment	Services	Always-on assumption	Scale-to-zero assumption	Original stated saving
Development	30	30 Pods ×24h	Average5 Pods ×8h	~83%
Staging	20	20 Pods ×24h	Average3 Pods ×12h	~92%
Production at night	10	10 Pods ×24h	2 Pods ×8h at night	~33%

The stated percentages are not all implied by the displayed assumptions. With zero Pods during the remaining hours, development is720→40 Pod-hours (94.4%) and staging480→36 (92.5%). If production means10 Pods for16 daytime hours plus2 for8 nighttime hours, it is240→176 (26.7%). These are conditional arithmetic checks, not rerun benchmarks.

The original GPU illustration assumed p3.2xlarge at$3.06/hour:24hours=$73.44/day,4hours=$12.24/day, approximately83% less. The rate has no verified Region/date provenance here and is not a current quote. A zero-replica Service does not guarantee immediate GPU-node termination; include retention delays, baseline replicas, model storage and node/commitment costs.

Knative with GPU Workloads

For a model PVC, populate it first and use CSI/access modes compatible with concurrent Pods/nodes/AZs (such as suitable ReadOnlyMany/ReadWriteMany storage), or per-Pod copies. RWO does not mean one Pod, but can block cross-node attachment. The example mounts models read-only, so persistent-volume-write is not enabled. min-scale0 permits cold activation; choose a positive baseline only when the latency/cost policy calls for it. The averaging window is not the request timeout.

The PodSpec extensions below must be enabled for node selectors, tolerations and the optional read-only PVC example. Save as serving-gpu-features.patch.yaml and merge into the existing Operator resource. The referenced gpu NodePool must expose sufficient allocatable CPU/memory/GPU capacity after system and Queue Proxy overhead, with the required driver/device plugin. These are unverified model-serving examples, not measured GPU sizing.

yaml

spec:
  config:
    features:
      kubernetes.podspec-nodeselector: enabled
      kubernetes.podspec-tolerations: enabled
      kubernetes.podspec-persistent-volume-claim: enabled

bash

kubectl patch knativeserving knative-serving -n knative-serving --type merge --patch-file serving-gpu-features.patch.yaml

Knative can serve GPU-accelerated workloads (e.g., ML inference) by scheduling pods on GPU nodes. Key considerations:

yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: llm-inference
  namespace: knative-demo
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
        autoscaling.knative.dev/metric: concurrency
        autoscaling.knative.dev/target: '1'
        autoscaling.knative.dev/min-scale: '1'
        autoscaling.knative.dev/max-scale: '4'
    spec:
      containerConcurrency: 1
      timeoutSeconds: 600
      containers:
      - image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/llm-server:v1
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: '4'
            memory: 16Gi
            nvidia.com/gpu: '1'
          limits:
            cpu: '6'
            memory: 24Gi
            nvidia.com/gpu: '1'
      nodeSelector:
        karpenter.sh/nodepool: gpu
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule

GPU-specific tips:

Set containerConcurrency: 1 if the model cannot batch concurrent requests. Increase it if the serving framework supports dynamic batching (e.g., vLLM, Triton Inference Server).
Consider a positive min-scale for a warm baseline. New Revisions, restarts and additional GPU replicas can still load cold; measure image/model startup.
Use Karpenter with GPU NodePools to dynamically provision GPU nodes as Knative scales up.
Monitor GPU utilization with DCGM Exporter and NVIDIA GPU Operator metrics.

References

Official Documentation

AWS and EKS Resources

Previous: Karpenter | Next: None

Primary references verified for this revision: Serving1.23, Operator1.23.1, CNCF milestone, Operator configuration, Scale-to-zero semantics, HPA implementation, SQS IntegrationSource, CloudEvents HTTP binding, S3 conditional put, Serving metrics, Prometheus OTLP configuration.

Traffic Management

Security

Observability

Resilience

Advanced

Comparison Guide

ArgoCD

Metrics

Logging

Tracing

Alerting

Observability End-to-End Labs

Knative ​

Table of Contents ​

Overview and Learning Objectives ​

What Is Knative? ​

Serverless on Kubernetes ​

Knative Serving vs Eventing ​

Knative vs AWS Lambda and AWS Fargate ​

Learning Objectives ​

Knative Architecture ​

Serving Architecture ​

Eventing Architecture ​

EKS Installation and Configuration ​

Prerequisites ​

Step 1: Install Knative Operator ​

Step 2: Install Knative Serving via the Operator ​

Step 3: Install Kourier (Lightweight Ingress) ​

Step 4: DNS Configuration ​

Option A: Magic DNS (sslip.io) -- Development Only ​

Option B: Real DNS with Amazon Route 53 -- Production ​

Step 5: TLS with cert-manager ​

Step 6: HPA vs KPA Autoscaler Selection ​

Step 7: Install Knative Eventing ​

Knative Serving Deep Dive ​

Resource Model ​

Complete Knative Service YAML ​

Traffic Splitting: Canary Deployments ​

Canary Deployment ​

Blue-Green Deployment ​

Scale-to-Zero Behavior ​

Concurrency-Based Scaling ​

Cold Start Optimization ​

Private and Public Services ​

Knative Eventing Deep Dive ​

Event Sources ​

ApiServerSource ​

SinkBinding ​

KafkaSource ​

SQSSource (AWS) ​

Broker/Trigger Pattern ​

Complete Broker/Trigger Example ​

CloudEvents Standard ​

Receipt-only consumer example ​

Channel/Subscription Pattern ​

Dead Letter Sink ​

Example S3-backed dead-letter handler ​

Event Filtering ​

Attribute Filtering ​

Extension Filtering ​

Multiple Triggers for OR Logic ​

KEDA vs Knative Comparison ​

Knative

Table of Contents

Overview and Learning Objectives

What Is Knative?

Serverless on Kubernetes

Knative Serving vs Eventing

Knative vs AWS Lambda and AWS Fargate

Learning Objectives

Knative Architecture

Serving Architecture

Eventing Architecture

EKS Installation and Configuration

Prerequisites

Step 1: Install Knative Operator

Step 2: Install Knative Serving via the Operator

Step 3: Install Kourier (Lightweight Ingress)

Step 4: DNS Configuration

Option A: Magic DNS (sslip.io) -- Development Only

Option B: Real DNS with Amazon Route 53 -- Production

Step 5: TLS with cert-manager

Step 6: HPA vs KPA Autoscaler Selection

Step 7: Install Knative Eventing

Knative Serving Deep Dive

Resource Model

Complete Knative Service YAML

Traffic Splitting: Canary Deployments

Canary Deployment

Blue-Green Deployment

Scale-to-Zero Behavior

Concurrency-Based Scaling

Cold Start Optimization

Private and Public Services

Knative Eventing Deep Dive

Event Sources

ApiServerSource

SinkBinding

KafkaSource

SQSSource (AWS)

Broker/Trigger Pattern

Complete Broker/Trigger Example

CloudEvents Standard

Receipt-only consumer example

Channel/Subscription Pattern

Dead Letter Sink

Example S3-backed dead-letter handler

Event Filtering

Attribute Filtering

Extension Filtering

Multiple Triggers for OR Logic

KEDA vs Knative Comparison