Tekton Pipelines
Supported Versions: Tekton Pipelines v0.62+, Tekton Triggers v0.28+ Last Updated: June 2025
< Previous: FinOps Cost Visibility Platform | Table of Contents | [Next: None] >
Overview
Tekton is a Kubernetes-native, open-source framework for building CI/CD systems. Unlike traditional CI platforms that bolt onto Kubernetes as an afterthought, Tekton runs pipelines as first-class Kubernetes resources: every Task is a Pod, every Step is a container, and every Pipeline is orchestrated by custom controllers watching Custom Resource Definitions (CRDs). This makes Tekton inherently portable, scalable, and deeply integrated with the Kubernetes ecosystem.
Tekton originated at Google as part of the Knative project (specifically knative/build) before being donated to the Continuous Delivery Foundation (CDF) in 2019, where it sits alongside Jenkins, Jenkins X, and Spinnaker. It graduated to a mature CDF project and is also a CNCF ecosystem participant through its integration with projects like Sigstore and in-toto.
Why Tekton Over Traditional CI/CD?
Traditional CI/CD systems (Jenkins, GitHub Actions, GitLab CI) operate as centralized services that dispatch work to runners. This model introduces several challenges in Kubernetes environments:
- Infrastructure duplication: A Jenkins controller runs alongside the Kubernetes cluster, requiring its own availability, storage, and scaling concerns.
- Secret sprawl: CI platforms need their own credential stores, duplicating what Kubernetes already manages through Secrets, IRSA, and Pod Identity.
- Limited Kubernetes awareness: External CI systems lack native access to Kubernetes APIs, requiring kubectl configuration and kubeconfig distribution.
- Vendor lock-in: Pipeline definitions are tightly coupled to the CI platform's proprietary YAML or Groovy DSL.
Tekton eliminates these issues by defining pipelines as Kubernetes CRDs. The cluster is both the execution environment and the orchestration layer.
Tekton vs. Other CI/CD Platforms
| Feature | Tekton | Jenkins | GitHub Actions | GitLab CI |
|---|---|---|---|---|
| Runtime | Kubernetes-native (CRDs) | JVM-based controller | GitHub-hosted / self-hosted runners | GitLab-hosted / self-hosted runners |
| Pipeline Definition | Kubernetes YAML | Groovy (Jenkinsfile) | GitHub YAML | GitLab YAML |
| Scalability | Scales with Kubernetes (Pod per TaskRun) | Requires agent management | Runner autoscaling | Runner autoscaling |
| Portability | Any Kubernetes cluster | Any server with JVM | GitHub ecosystem | GitLab ecosystem |
| Supply Chain Security | Tekton Chains (SLSA, Sigstore) | Plugins required | Artifact attestations | Dependency scanning |
| Event Handling | Tekton Triggers (webhooks, CEL) | Webhook + plugins | Native GitHub events | Native GitLab events |
| UI/Dashboard | Tekton Dashboard (optional) | Jenkins UI (built-in) | GitHub UI (built-in) | GitLab UI (built-in) |
| Secret Management | Kubernetes Secrets, IRSA, CSI driver | Jenkins Credentials | GitHub Secrets | GitLab CI Variables |
| Learning Curve | Moderate (requires Kubernetes knowledge) | High (Groovy, plugin ecosystem) | Low (familiar YAML) | Low (familiar YAML) |
| Cost | Free (compute only) | Free (infra maintenance) | Per-minute billing (hosted) | Per-minute billing (hosted) |
Learning Objectives
After completing this guide, you will be able to:
- Understand Tekton's architecture and how its CRDs map to CI/CD concepts
- Install and configure Tekton Pipelines, Triggers, Dashboard, and Chains on Amazon EKS
- Author reusable Tasks with step composition, script execution, results, and sidecars
- Construct multi-stage Pipelines with parallel execution, conditional logic, and finally blocks
- Configure webhook-based automatic pipeline triggers using Tekton Triggers
- Implement supply chain security with Tekton Chains, including OCI image signing and SLSA provenance
- Integrate Tekton (CI) with ArgoCD (CD) for a complete GitOps-based CI/CD architecture
- Operate Tekton in production with cleanup policies, monitoring, and troubleshooting strategies
1. Tekton Architecture
Tekton extends the Kubernetes API with a set of Custom Resource Definitions (CRDs) that model CI/CD concepts. The Tekton controller watches these CRDs and reconciles desired state into Pods and containers.
1.1 Core CRDs
Task: A reusable, parameterized unit of work. Each Task defines one or more Steps that execute sequentially in containers within a single Pod. Tasks also declare Workspaces (for shared data) and Results (for output values).
TaskRun: An instantiation of a Task with concrete parameter values and workspace bindings. Creating a TaskRun causes the controller to spin up a Pod that executes the Task's Steps.
Pipeline: An ordered collection of Tasks. Pipelines define execution order (sequential, parallel, or DAG-based), parameter passing between Tasks, conditional execution (when expressions), and finally blocks for cleanup.
PipelineRun: An instantiation of a Pipeline. Creating a PipelineRun causes the controller to create TaskRuns for each Task in the Pipeline.
Workspace: A mechanism for sharing data between Steps (within a Task) or between Tasks (within a Pipeline). Workspaces can be backed by PersistentVolumeClaims, ConfigMaps, Secrets, or emptyDir volumes.
Result: A string value produced by a Task that can be consumed by downstream Tasks in a Pipeline. Results are written to /tekton/results/<name> and stored in the TaskRun status.
1.2 Controller Architecture
tekton-pipelines-controller: The core reconciliation controller. It watches for new PipelineRun and TaskRun resources, creates Pods for execution, monitors container completion, propagates results, and updates status.
tekton-pipelines-webhook: An admission webhook that validates and mutates Tekton CRDs before they are persisted. It enforces schema correctness and applies defaults.
tekton-chains-controller: A separate controller that watches TaskRun completions and automatically signs OCI images, generates SLSA provenance attestations, and stores them in OCI registries or transparency logs.
tekton-triggers-controller: Processes incoming webhook events (from GitHub, GitLab, etc.), evaluates interceptors and CEL expressions, and creates PipelineRun resources based on TriggerTemplate definitions.
tekton-dashboard: An optional web UI for viewing PipelineRuns, TaskRuns, logs, and cluster resources.
1.3 Tekton Chains and Supply Chain Security
Tekton Chains is a dedicated component for software supply chain security. When a TaskRun completes, Chains automatically:
- Signs OCI images produced by the TaskRun using Cosign (Sigstore)
- Generates SLSA provenance attestations recording what was built, from what source, using which builder
- Stores attestations in the OCI registry as image signatures or in transparency logs (Rekor)
- Produces in-toto attestation metadata linking source materials to build outputs
This provides tamper-evident records that downstream consumers (admission controllers, policy engines) can verify before deploying images.
2. EKS Installation and Configuration
2.1 Prerequisites
Before installing Tekton, ensure your EKS cluster meets the following requirements:
# Verify cluster version (1.28+)
kubectl version --short
# Verify cluster has sufficient capacity
# Tekton controllers need ~500m CPU and ~512Mi memory total
kubectl top nodes
# Ensure the cluster has a default StorageClass for Workspaces
kubectl get storageclass2.2 Tekton Pipelines Installation
# Install Tekton Pipelines (core)
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.62.2/release.yaml
# Verify installation
kubectl get pods -n tekton-pipelines --watchExpected output:
NAME READY STATUS RESTARTS AGE
tekton-pipelines-controller-7f6d5b8bc4-xxxxx 1/1 Running 0 30s
tekton-pipelines-webhook-6c9b5d7d8f-xxxxx 1/1 Running 0 30sConfigure Tekton Pipelines feature flags:
# tekton-pipelines-feature-flags.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: feature-flags
namespace: tekton-pipelines
data:
# Enable Step-level resource requirements
disable-affinity-assistant: "true"
# Enable alpha API features (required for some Chains features)
enable-api-fields: "beta"
# Set result extraction method
results-from: "termination-message"
# Enable Workspace isolation
running-in-environment-with-injected-sidecars: "true"
# Set default timeout (1 hour)
default-timeout-minutes: "60"
# Enable larger results (4096 bytes max by default)
max-result-size: "4096"
# Enable Kubernetes events for PipelineRuns
send-cloudevents-for-runs: "false"kubectl apply -f tekton-pipelines-feature-flags.yaml2.3 Tekton Triggers Installation
# Install Tekton Triggers
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/previous/v0.28.0/release.yaml
# Install Tekton Triggers Interceptors (required for GitHub/GitLab webhooks)
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/previous/v0.28.0/interceptors.yaml
# Verify installation
kubectl get pods -n tekton-pipelines -l app.kubernetes.io/part-of=tekton-triggers2.4 Tekton Dashboard Installation
# Install Tekton Dashboard (read-only mode for production)
kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/previous/v0.46.0/release-full.yaml
# Verify installation
kubectl get pods -n tekton-pipelines -l app.kubernetes.io/part-of=tekton-dashboardExpose the Dashboard via an internal ALB Ingress:
# tekton-dashboard-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tekton-dashboard
namespace: tekton-pipelines
annotations:
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/abc-123
alb.ingress.kubernetes.io/group.name: internal-services
spec:
ingressClassName: alb
rules:
- host: tekton.internal.mycompany.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: tekton-dashboard
port:
number: 90972.5 Tekton Chains Installation
# Install Tekton Chains
kubectl apply --filename https://storage.googleapis.com/tekton-releases/chains/previous/v0.22.1/release.yaml
# Verify installation
kubectl get pods -n tekton-chains2.6 IRSA Configuration for ECR and S3
Tekton Tasks that push images to ECR or upload artifacts to S3 require IAM permissions. Use IAM Roles for Service Accounts (IRSA) to grant least-privilege access.
# Create OIDC provider (skip if already configured)
eksctl utils associate-iam-oidc-provider \
--cluster my-eks-cluster \
--region us-east-1 \
--approveECR Push IAM Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:PutImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:DescribeRepositories",
"ecr:CreateRepository",
"ecr:TagResource"
],
"Resource": "arn:aws:ecr:us-east-1:123456789012:repository/*"
}
]
}S3 Artifact Upload IAM Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-tekton-artifacts",
"arn:aws:s3:::my-tekton-artifacts/*"
]
}
]
}Create the IRSA-annotated ServiceAccount:
# Create IAM role and ServiceAccount for Tekton builds
eksctl create iamserviceaccount \
--cluster my-eks-cluster \
--namespace tekton-builds \
--name tekton-build-sa \
--attach-policy-arn arn:aws:iam::123456789012:policy/TektonECRPush \
--attach-policy-arn arn:aws:iam::123456789012:policy/TektonS3Artifacts \
--approve \
--override-existing-serviceaccountsVerify the annotation:
kubectl get sa tekton-build-sa -n tekton-builds -o yaml
# Should show: eks.amazonaws.com/role-arn annotation3. Task Authoring
Tasks are the fundamental building blocks in Tekton. A well-designed Task is reusable, parameterized, and produces meaningful results.
3.1 Step Composition
Each Step in a Task runs as a container within the same Pod. Steps execute sequentially, sharing the same network namespace and Workspace volumes.
# task-go-build.yaml
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: go-build
labels:
app.kubernetes.io/version: "1.0"
annotations:
tekton.dev/pipelines.minVersion: "0.62.0"
tekton.dev/categories: Build
tekton.dev/tags: go, build
tekton.dev/displayName: "Go Build and Test"
spec:
description: >
Builds and tests a Go application, producing a binary artifact.
params:
- name: package
type: string
description: "Go package path to build"
default: "./cmd/server"
- name: go-version
type: string
description: "Go version to use"
default: "1.22"
- name: goos
type: string
description: "Target operating system"
default: "linux"
- name: goarch
type: string
description: "Target architecture"
default: "amd64"
- name: flags
type: string
description: "Go build flags"
default: "-v"
- name: ldflags
type: string
description: "Go linker flags"
default: ""
workspaces:
- name: source
description: "Workspace containing Go source code"
- name: go-cache
description: "Go module and build cache"
optional: true
results:
- name: binary-path
description: "Path to the built binary"
- name: test-passed
description: "Whether tests passed (true/false)"
- name: binary-sha256
description: "SHA256 hash of the built binary"
steps:
- name: unit-test
image: golang:$(params.go-version)
workingDir: $(workspaces.source.path)
env:
- name: GOMODCACHE
value: "$(workspaces.go-cache.path)/mod"
- name: GOCACHE
value: "$(workspaces.go-cache.path)/build"
script: |
#!/usr/bin/env bash
set -euo pipefail
echo "Running unit tests..."
go test -race -coverprofile=coverage.out ./...
COVERAGE=$(go tool cover -func=coverage.out | grep total | awk '{print $3}')
echo "Test coverage: ${COVERAGE}"
printf "true" > $(results.test-passed.path)
- name: build
image: golang:$(params.go-version)
workingDir: $(workspaces.source.path)
env:
- name: GOOS
value: "$(params.goos)"
- name: GOARCH
value: "$(params.goarch)"
- name: CGO_ENABLED
value: "0"
- name: GOMODCACHE
value: "$(workspaces.go-cache.path)/mod"
- name: GOCACHE
value: "$(workspaces.go-cache.path)/build"
script: |
#!/usr/bin/env bash
set -euo pipefail
OUTPUT_PATH="$(workspaces.source.path)/output/app"
mkdir -p "$(dirname ${OUTPUT_PATH})"
echo "Building $(params.package)..."
go build $(params.flags) \
-ldflags "$(params.ldflags)" \
-o "${OUTPUT_PATH}" \
"$(params.package)"
HASH=$(sha256sum "${OUTPUT_PATH}" | awk '{print $1}')
echo "Binary built: ${OUTPUT_PATH} (SHA256: ${HASH})"
printf "%s" "${OUTPUT_PATH}" > $(results.binary-path.path)
printf "%s" "${HASH}" > $(results.binary-sha256.path)3.2 Script Execution
Steps can run inline scripts for complex logic. The script field accepts any interpreter available in the container image.
# task-security-scan.yaml
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: trivy-image-scan
spec:
description: "Scans a container image for vulnerabilities using Trivy"
params:
- name: image-url
type: string
description: "Full image URL including tag or digest"
- name: severity-threshold
type: string
description: "Minimum severity to report (CRITICAL, HIGH, MEDIUM, LOW)"
default: "HIGH,CRITICAL"
- name: exit-code
type: string
description: "Exit code when vulnerabilities found (0=warn, 1=fail)"
default: "1"
workspaces:
- name: trivy-cache
description: "Cache for Trivy vulnerability database"
optional: true
results:
- name: vulnerability-count
description: "Number of vulnerabilities found"
- name: scan-result
description: "PASS or FAIL"
steps:
- name: scan
image: aquasec/trivy:0.53.0
env:
- name: TRIVY_CACHE_DIR
value: "$(workspaces.trivy-cache.path)"
script: |
#!/usr/bin/env sh
set -uo pipefail
echo "Scanning image: $(params.image-url)"
echo "Severity threshold: $(params.severity-threshold)"
trivy image \
--severity "$(params.severity-threshold)" \
--format json \
--output /tmp/trivy-report.json \
--exit-code 0 \
"$(params.image-url)"
VULN_COUNT=$(cat /tmp/trivy-report.json | \
grep -o '"VulnerabilityID"' | wc -l || echo "0")
printf "%s" "${VULN_COUNT}" > $(results.vulnerability-count.path)
if [ "${VULN_COUNT}" -gt "0" ]; then
echo "Found ${VULN_COUNT} vulnerabilities"
trivy image \
--severity "$(params.severity-threshold)" \
--format table \
"$(params.image-url)"
printf "FAIL" > $(results.scan-result.path)
exit $(params.exit-code)
else
echo "No vulnerabilities found"
printf "PASS" > $(results.scan-result.path)
fi3.3 Results Passing Between Tasks
Results from one Task can be referenced by downstream Tasks in a Pipeline using the $(tasks.<taskName>.results.<resultName>) syntax. Results are limited to 4096 bytes by default.
# task-generate-tag.yaml
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: generate-image-tag
spec:
description: "Generates a deterministic image tag from git commit and timestamp"
params:
- name: image-base
type: string
description: "Base image URL (registry/repository)"
workspaces:
- name: source
description: "Workspace containing git repository"
results:
- name: image-tag
description: "Generated image tag"
- name: image-url
description: "Full image URL with tag"
- name: short-sha
description: "Short git commit SHA"
steps:
- name: generate
image: alpine/git:2.43.0
workingDir: $(workspaces.source.path)
script: |
#!/usr/bin/env sh
set -euo pipefail
SHORT_SHA=$(git rev-parse --short HEAD)
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
TAG="${SHORT_SHA}-${TIMESTAMP}"
printf "%s" "${TAG}" > $(results.image-tag.path)
printf "%s" "$(params.image-base):${TAG}" > $(results.image-url.path)
printf "%s" "${SHORT_SHA}" > $(results.short-sha.path)
echo "Generated tag: ${TAG}"
echo "Full image URL: $(params.image-base):${TAG}"3.4 Sidecars
Sidecars are long-running containers that run alongside Steps, useful for services like Docker daemon, databases for integration tests, or caches.
# task-integration-test.yaml
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: integration-test-with-postgres
spec:
description: "Runs integration tests against a PostgreSQL sidecar"
params:
- name: go-version
type: string
default: "1.22"
- name: postgres-version
type: string
default: "16"
- name: test-packages
type: string
default: "./internal/integration/..."
workspaces:
- name: source
sidecars:
- name: postgres
image: postgres:$(params.postgres-version)
env:
- name: POSTGRES_DB
value: "testdb"
- name: POSTGRES_USER
value: "testuser"
- name: POSTGRES_PASSWORD
value: "testpassword"
ports:
- containerPort: 5432
readinessProbe:
exec:
command: ["pg_isready", "-U", "testuser", "-d", "testdb"]
initialDelaySeconds: 5
periodSeconds: 2
steps:
- name: wait-for-postgres
image: postgres:$(params.postgres-version)
script: |
#!/usr/bin/env bash
set -euo pipefail
echo "Waiting for PostgreSQL to become ready..."
for i in $(seq 1 30); do
if pg_isready -h localhost -U testuser -d testdb; then
echo "PostgreSQL is ready"
exit 0
fi
sleep 2
done
echo "PostgreSQL failed to start"
exit 1
- name: run-tests
image: golang:$(params.go-version)
workingDir: $(workspaces.source.path)
env:
- name: DATABASE_URL
value: "postgres://testuser:testpassword@localhost:5432/testdb?sslmode=disable"
script: |
#!/usr/bin/env bash
set -euo pipefail
echo "Running integration tests..."
go test -v -count=1 -timeout 10m $(params.test-packages)
echo "Integration tests passed"3.5 Reusable Tasks from Tekton Hub
Tekton Hub provides a catalog of community-maintained, reusable Tasks. Install them directly into your cluster:
# Install the git-clone Task (official catalog)
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-clone/0.9/git-clone.yaml
# Install the kaniko Task (for building images without Docker daemon)
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/kaniko/0.7/kaniko.yaml
# Install the kubernetes-actions Task (for kubectl operations)
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/kubernetes-actions/0.2/kubernetes-actions.yamlVerify installed Tasks:
kubectl get tasks -n tekton-builds3.6 Complete Build-Push Task (Kaniko + ECR)
This Task builds a container image using Kaniko (no Docker daemon required) and pushes it to Amazon ECR:
# task-kaniko-ecr-build.yaml
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: kaniko-ecr-build
spec:
description: >
Builds a container image with Kaniko and pushes to Amazon ECR.
Requires an IRSA-annotated ServiceAccount with ECR push permissions.
params:
- name: image
type: string
description: "Full ECR image URL (without tag)"
- name: tag
type: string
description: "Image tag"
default: "latest"
- name: dockerfile
type: string
description: "Path to Dockerfile relative to workspace root"
default: "Dockerfile"
- name: context
type: string
description: "Build context directory relative to workspace root"
default: "."
- name: build-args
type: array
description: "List of build arguments"
default: []
- name: cache-repo
type: string
description: "ECR repository for layer caching (empty to disable)"
default: ""
workspaces:
- name: source
description: "Workspace containing Dockerfile and source code"
- name: dockerconfig
description: "Docker config.json for registry authentication"
optional: true
results:
- name: image-url
description: "Full image URL with tag"
- name: image-digest
description: "Image digest (sha256)"
steps:
- name: ecr-login
image: amazon/aws-cli:2.17.0
script: |
#!/usr/bin/env bash
set -euo pipefail
REGION=$(echo "$(params.image)" | cut -d'.' -f4)
ACCOUNT=$(echo "$(params.image)" | cut -d'.' -f1)
echo "Logging into ECR: ${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com"
aws ecr get-login-password --region "${REGION}" | \
docker-credential-ecr-login login \
--username AWS \
--password-stdin \
"${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com"
mkdir -p /kaniko/.docker
aws ecr get-login-password --region "${REGION}" | \
python3 -c "
import sys, json, base64
password = sys.stdin.read().strip()
auth = base64.b64encode(f'AWS:{password}'.encode()).decode()
registry = '${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com'
config = {'auths': {registry: {'auth': auth}}}
json.dump(config, open('/kaniko/.docker/config.json', 'w'))
"
volumeMounts:
- name: kaniko-config
mountPath: /kaniko/.docker
- name: build-and-push
image: gcr.io/kaniko-project/executor:v1.23.2
args:
- --dockerfile=$(workspaces.source.path)/$(params.dockerfile)
- --context=$(workspaces.source.path)/$(params.context)
- --destination=$(params.image):$(params.tag)
- --digest-file=$(results.image-digest.path)
- --snapshotMode=redo
- --compressed-caching=false
- --use-new-run
volumeMounts:
- name: kaniko-config
mountPath: /kaniko/.docker
volumes:
- name: kaniko-config
emptyDir: {}
stepTemplate:
securityContext:
runAsUser: 04. Pipeline Construction
Pipelines orchestrate multiple Tasks into a directed acyclic graph (DAG). Tasks can run in parallel, conditionally, or sequentially based on explicit dependencies.
4.1 Task Ordering and Parallel Execution
In the diagram above, lint and unit-test run in parallel after git-clone completes. build-image waits for both to succeed. This is achieved by using runAfter in the Pipeline spec.
4.2 Conditional Execution with When Expressions
When expressions allow Tasks to be skipped based on parameter values or results from previous Tasks:
# When expression examples (used within Pipeline tasks)
when:
- input: "$(params.run-security-scan)"
operator: in
values: ["true"]
- input: "$(tasks.unit-test.results.test-passed)"
operator: in
values: ["true"]4.3 Finally Tasks
Finally Tasks always execute regardless of Pipeline success or failure — similar to try/finally in code. They are ideal for cleanup, notifications, and status reporting.
4.4 Complete CI/CD Pipeline
The following Pipeline implements a full CI/CD workflow: clone, test, build, push, scan, and deploy.
# pipeline-ci-cd.yaml
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: application-ci-cd
labels:
app.kubernetes.io/version: "1.0"
spec:
description: >
Complete CI/CD pipeline: clones source, runs tests, builds and pushes
a container image to ECR, scans for vulnerabilities, and deploys to
the target environment.
params:
- name: git-url
type: string
description: "Git repository URL"
- name: git-revision
type: string
description: "Git revision (branch, tag, or commit SHA)"
default: "main"
- name: image-registry
type: string
description: "ECR registry URL (e.g., 123456789012.dkr.ecr.us-east-1.amazonaws.com)"
- name: image-name
type: string
description: "Image repository name"
- name: dockerfile
type: string
description: "Path to Dockerfile"
default: "Dockerfile"
- name: deploy-environment
type: string
description: "Target deployment environment"
default: "staging"
- name: deploy-namespace
type: string
description: "Kubernetes namespace to deploy into"
default: "staging"
- name: run-security-scan
type: string
description: "Whether to run security scan (true/false)"
default: "true"
workspaces:
- name: shared-workspace
description: "Shared workspace for source code and build artifacts"
- name: git-credentials
description: "Git credentials (SSH key or token)"
optional: true
- name: docker-credentials
description: "Docker registry credentials"
optional: true
tasks:
# ---- Stage 1: Clone ----
- name: clone
taskRef:
name: git-clone
params:
- name: url
value: "$(params.git-url)"
- name: revision
value: "$(params.git-revision)"
- name: deleteExisting
value: "true"
workspaces:
- name: output
workspace: shared-workspace
- name: ssh-directory
workspace: git-credentials
# ---- Stage 2: Test + Lint (parallel) ----
- name: unit-test
taskRef:
name: go-build
runAfter:
- clone
params:
- name: package
value: "./..."
workspaces:
- name: source
workspace: shared-workspace
- name: lint
taskRef:
name: golangci-lint
runAfter:
- clone
params:
- name: flags
value: "--timeout=5m"
workspaces:
- name: source
workspace: shared-workspace
# ---- Stage 3: Generate Tag ----
- name: generate-tag
taskRef:
name: generate-image-tag
runAfter:
- clone
params:
- name: image-base
value: "$(params.image-registry)/$(params.image-name)"
workspaces:
- name: source
workspace: shared-workspace
# ---- Stage 4: Build and Push ----
- name: build-push
taskRef:
name: kaniko-ecr-build
runAfter:
- unit-test
- lint
- generate-tag
params:
- name: image
value: "$(params.image-registry)/$(params.image-name)"
- name: tag
value: "$(tasks.generate-tag.results.image-tag)"
- name: dockerfile
value: "$(params.dockerfile)"
workspaces:
- name: source
workspace: shared-workspace
# ---- Stage 5: Security Scan (conditional) ----
- name: security-scan
taskRef:
name: trivy-image-scan
runAfter:
- build-push
when:
- input: "$(params.run-security-scan)"
operator: in
values: ["true"]
params:
- name: image-url
value: "$(tasks.build-push.results.image-url)"
- name: severity-threshold
value: "HIGH,CRITICAL"
- name: exit-code
value: "1"
# ---- Stage 6: Deploy ----
- name: deploy
taskRef:
name: kubernetes-deploy
runAfter:
- security-scan
params:
- name: image-url
value: "$(tasks.build-push.results.image-url)"
- name: namespace
value: "$(params.deploy-namespace)"
- name: environment
value: "$(params.deploy-environment)"
workspaces:
- name: source
workspace: shared-workspace
finally:
# ---- Cleanup and Notification ----
- name: notify-slack
taskRef:
name: send-slack-notification
params:
- name: webhook-url-secret
value: "slack-webhook"
- name: webhook-url-secret-key
value: "url"
- name: message
value: >
Pipeline $(context.pipelineRun.name) for
$(params.image-name):$(tasks.generate-tag.results.image-tag)
completed with status: $(tasks.status)
- name: cleanup-workspace
taskRef:
name: cleanup
workspaces:
- name: source
workspace: shared-workspace4.5 Running the Pipeline
Create a PipelineRun to execute the Pipeline:
# pipelinerun-example.yaml
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
generateName: app-ci-cd-run-
namespace: tekton-builds
labels:
tekton.dev/pipeline: application-ci-cd
app: my-service
spec:
pipelineRef:
name: application-ci-cd
params:
- name: git-url
value: "git@github.com:myorg/my-service.git"
- name: git-revision
value: "main"
- name: image-registry
value: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
- name: image-name
value: "my-service"
- name: deploy-environment
value: "staging"
- name: deploy-namespace
value: "staging"
workspaces:
- name: shared-workspace
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: gp3
- name: git-credentials
secret:
secretName: git-ssh-key
taskRunTemplate:
serviceAccountName: tekton-build-sa
timeouts:
pipeline: "1h0m0s"
tasks: "45m0s"
finally: "15m0s"# Create the PipelineRun
kubectl create -f pipelinerun-example.yaml
# Watch the execution
kubectl get pipelineruns -n tekton-builds --watch
# View logs for a specific TaskRun
kubectl logs -n tekton-builds -l tekton.dev/pipelineRun=app-ci-cd-run-xxxxx --all-containers -f5. Tekton Triggers
Tekton Triggers enable automatic Pipeline execution in response to external events such as Git pushes, pull requests, or any webhook-based event.
5.1 Trigger Architecture
EventListener: A Kubernetes Service that receives incoming webhook HTTP requests and routes them through Triggers.
TriggerBinding: Extracts parameter values from the incoming event payload using JSONPath expressions.
TriggerTemplate: A template for creating Tekton resources (typically PipelineRuns) with parameters populated from the TriggerBinding.
Interceptor: A processing step that validates, filters, and transforms incoming events. Built-in interceptors include GitHub (webhook signature validation), GitLab, Bitbucket, and CEL (Common Expression Language for filtering).
5.2 TriggerBinding
# triggerbinding-github-push.yaml
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
name: github-push-binding
namespace: tekton-builds
spec:
params:
- name: git-url
value: "$(body.repository.ssh_url)"
- name: git-revision
value: "$(body.after)"
- name: git-branch
value: "$(body.ref)"
- name: repository-name
value: "$(body.repository.name)"
- name: repository-full-name
value: "$(body.repository.full_name)"
- name: pusher-name
value: "$(body.pusher.name)"
- name: commit-message
value: "$(body.head_commit.message)"
---
# triggerbinding-github-pr.yaml
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
name: github-pr-binding
namespace: tekton-builds
spec:
params:
- name: git-url
value: "$(body.pull_request.head.repo.ssh_url)"
- name: git-revision
value: "$(body.pull_request.head.sha)"
- name: git-branch
value: "$(body.pull_request.head.ref)"
- name: pr-number
value: "$(body.pull_request.number)"
- name: pr-title
value: "$(body.pull_request.title)"
- name: repository-name
value: "$(body.repository.name)"
- name: action
value: "$(body.action)"5.3 TriggerTemplate
# triggertemplate-ci-cd.yaml
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
name: ci-cd-trigger-template
namespace: tekton-builds
spec:
params:
- name: git-url
description: "Git repository URL"
- name: git-revision
description: "Git commit SHA"
- name: git-branch
description: "Git branch reference"
- name: repository-name
description: "Repository name"
resourcetemplates:
- apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
generateName: "$(tt.params.repository-name)-run-"
namespace: tekton-builds
labels:
tekton.dev/pipeline: application-ci-cd
triggers.tekton.dev/trigger: github-push
app: "$(tt.params.repository-name)"
annotations:
tekton.dev/git-branch: "$(tt.params.git-branch)"
spec:
pipelineRef:
name: application-ci-cd
params:
- name: git-url
value: "$(tt.params.git-url)"
- name: git-revision
value: "$(tt.params.git-revision)"
- name: image-registry
value: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
- name: image-name
value: "$(tt.params.repository-name)"
- name: deploy-environment
value: "staging"
- name: deploy-namespace
value: "staging"
workspaces:
- name: shared-workspace
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: gp3
- name: git-credentials
secret:
secretName: git-ssh-key
taskRunTemplate:
serviceAccountName: tekton-build-sa
timeouts:
pipeline: "1h0m0s"5.4 EventListener with Interceptors
# eventlistener-github.yaml
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
name: github-listener
namespace: tekton-builds
spec:
serviceAccountName: tekton-triggers-sa
triggers:
# Trigger 1: Push to main branch
- name: github-push-main
interceptors:
- ref:
name: "github"
params:
- name: "secretRef"
value:
secretName: github-webhook-secret
secretKey: token
- name: "eventTypes"
value: ["push"]
- ref:
name: "cel"
params:
- name: "filter"
value: >
body.ref == 'refs/heads/main' &&
!body.head_commit.message.contains('[skip ci]')
- name: "overlays"
value:
- key: truncated-sha
expression: "body.after.truncate(7)"
bindings:
- ref: github-push-binding
template:
ref: ci-cd-trigger-template
# Trigger 2: Pull request opened or synchronized
- name: github-pr
interceptors:
- ref:
name: "github"
params:
- name: "secretRef"
value:
secretName: github-webhook-secret
secretKey: token
- name: "eventTypes"
value: ["pull_request"]
- ref:
name: "cel"
params:
- name: "filter"
value: >
body.action in ['opened', 'synchronize'] &&
!body.pull_request.draft
bindings:
- ref: github-pr-binding
template:
ref: ci-cd-trigger-template
resources:
kubernetesResource:
spec:
template:
spec:
serviceAccountName: tekton-triggers-sa
containers: []
metadata:
labels:
app: tekton-triggers-eventlistener
---
# ServiceAccount for Tekton Triggers
apiVersion: v1
kind: ServiceAccount
metadata:
name: tekton-triggers-sa
namespace: tekton-builds
---
# RBAC: Allow Triggers to create PipelineRuns
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tekton-triggers-role
namespace: tekton-builds
rules:
- apiGroups: ["triggers.tekton.dev"]
resources: ["eventlisteners", "triggerbindings", "triggertemplates", "triggers"]
verbs: ["get", "list", "watch"]
- apiGroups: ["tekton.dev"]
resources: ["pipelineresources", "pipelineruns", "taskruns"]
verbs: ["create"]
- apiGroups: [""]
resources: ["configmaps", "secrets"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tekton-triggers-rolebinding
namespace: tekton-builds
subjects:
- kind: ServiceAccount
name: tekton-triggers-sa
roleRef:
kind: Role
name: tekton-triggers-role
apiGroup: rbac.authorization.k8s.io5.5 Expose EventListener
# eventlistener-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tekton-triggers-webhook
namespace: tekton-builds
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/abc-123
alb.ingress.kubernetes.io/healthcheck-path: /live
spec:
ingressClassName: alb
rules:
- host: tekton-hooks.mycompany.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: el-github-listener
port:
number: 8080Configure the webhook in GitHub:
Payload URL: https://tekton-hooks.mycompany.com
Content type: application/json
Secret: <same value as github-webhook-secret>
Events: Pushes, Pull requests5.6 GitLab Interceptor Example
For GitLab webhooks, replace the GitHub interceptor:
interceptors:
- ref:
name: "gitlab"
params:
- name: "secretRef"
value:
secretName: gitlab-webhook-secret
secretKey: token
- name: "eventTypes"
value: ["Push Hook", "Merge Request Hook"]
- ref:
name: "cel"
params:
- name: "filter"
value: >
(header.match('X-Gitlab-Event', 'Push Hook') &&
body.ref == 'refs/heads/main') ||
(header.match('X-Gitlab-Event', 'Merge Request Hook') &&
body.object_attributes.action in ['open', 'update'])6. Tekton Chains (Supply Chain Security)
Tekton Chains provides automated supply chain security for your CI/CD pipelines. It observes TaskRun completions and automatically generates cryptographic attestations about what was built, how, and from what source.
6.1 How Chains Works
6.2 Chains Configuration
# tekton-chains-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: chains-config
namespace: tekton-chains
data:
# Artifact storage (oci = store in OCI registry alongside image)
artifacts.taskrun.format: "in-toto"
artifacts.taskrun.storage: "oci"
artifacts.taskrun.signer: "x509"
artifacts.oci.format: "simplesigning"
artifacts.oci.storage: "oci"
artifacts.oci.signer: "x509"
# SLSA provenance configuration
artifacts.pipelinerun.format: "in-toto"
artifacts.pipelinerun.storage: "oci"
artifacts.pipelinerun.signer: "x509"
# Sigstore configuration
sigstore.fulcio.enabled: "false"
# For keyless signing with Fulcio (recommended for production):
# sigstore.fulcio.enabled: "true"
# sigstore.fulcio.address: "https://fulcio.sigstore.dev"
# sigstore.rekor.url: "https://rekor.sigstore.dev"
# Transparency log
transparency.enabled: "true"
transparency.url: "https://rekor.sigstore.dev"6.3 Signing Key Setup
For key-based signing (non-keyless), generate a Cosign key pair and store it as a Kubernetes Secret:
# Generate a Cosign signing key pair
cosign generate-key-pair k8s://tekton-chains/signing-secretsThis creates a Secret named signing-secrets in the tekton-chains namespace containing the private key. The public key is output to cosign.pub.
For production environments using AWS KMS:
# Create a KMS key for signing
aws kms create-key \
--description "Tekton Chains image signing key" \
--key-usage SIGN_VERIFY \
--key-spec ECC_NIST_P256 \
--tags TagKey=Purpose,TagValue=tekton-chains
# Use KMS key with Cosign
cosign generate-key-pair --kms awskms:///arn:aws:kms:us-east-1:123456789012:key/abc-123Update Chains config for KMS:
# chains-config-kms.yaml (merge with chains-config)
data:
sigstore.kms: "awskms:///arn:aws:kms:us-east-1:123456789012:key/abc-123"6.4 SLSA Provenance
Tekton Chains generates SLSA (Supply-chain Levels for Software Artifacts) provenance attestations automatically. The provenance records:
- Builder: Which Tekton Task/Pipeline produced the artifact
- Source: The git repository, branch, and commit
- Build configuration: The Pipeline parameters and Task steps
- Materials: Input artifacts (source code, base images)
- Output: The built artifact (image digest)
Verify provenance after a build:
# Verify image signature
cosign verify \
--key cosign.pub \
123456789012.dkr.ecr.us-east-1.amazonaws.com/my-service:abc1234-20250622-120000
# Verify and view SLSA provenance attestation
cosign verify-attestation \
--key cosign.pub \
--type slsaprovenance \
123456789012.dkr.ecr.us-east-1.amazonaws.com/my-service:abc1234-20250622-120000 | \
jq -r '.payload' | base64 -d | jq .Example SLSA provenance output (abbreviated):
{
"_type": "https://in-toto.io/Statement/v0.1",
"predicateType": "https://slsa.dev/provenance/v0.2",
"subject": [
{
"name": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-service",
"digest": {
"sha256": "a1b2c3d4e5f6..."
}
}
],
"predicate": {
"builder": {
"id": "https://tekton.dev/chains/v2"
},
"buildType": "tekton.dev/v1beta1/TaskRun",
"invocation": {
"configSource": {},
"parameters": {
"image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-service",
"tag": "abc1234-20250622-120000"
}
},
"buildConfig": {
"steps": [
{ "entryPoint": "...", "image": "gcr.io/kaniko-project/executor:v1.23.2" }
]
},
"materials": [
{
"uri": "git+https://github.com/myorg/my-service.git",
"digest": { "sha1": "abc1234..." }
}
]
}
}6.5 Policy Enforcement with Kyverno
Use Kyverno to enforce that only signed images with valid provenance can be deployed:
# kyverno-verify-image-signature.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-tekton-chains-signature
annotations:
policies.kyverno.io/title: Verify Tekton Chains Image Signature
policies.kyverno.io/category: Supply Chain Security
policies.kyverno.io/severity: critical
spec:
validationFailureAction: Enforce
webhookTimeoutSeconds: 30
rules:
- name: verify-image-signature
match:
any:
- resources:
kinds:
- Pod
exclude:
any:
- resources:
namespaces:
- kube-system
- tekton-pipelines
- tekton-chains
verifyImages:
- imageReferences:
- "123456789012.dkr.ecr.us-east-1.amazonaws.com/*"
attestors:
- entries:
- keys:
publicKeys: |-
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
-----END PUBLIC KEY-----
attestations:
- type: https://slsa.dev/provenance/v0.2
conditions:
- all:
- key: "{{ builder.id }}"
operator: Equals
value: "https://tekton.dev/chains/v2"7. ArgoCD + Tekton Integration
The most effective production architecture separates CI (build, test, push) from CD (deploy). Tekton handles CI while ArgoCD handles CD via GitOps. This separation provides clear ownership, independent scaling, and a reliable audit trail.
7.1 CI/CD Separation Architecture
7.2 GitOps Repository Update Task
After a successful build, Tekton updates the image tag in the GitOps repository. ArgoCD detects the change and reconciles the deployment.
# task-update-gitops-repo.yaml
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: update-gitops-repo
spec:
description: >
Updates the image tag in a GitOps repository, triggering ArgoCD
to reconcile and deploy the new version.
params:
- name: gitops-repo-url
type: string
description: "GitOps repository SSH URL"
- name: gitops-branch
type: string
description: "Branch to update"
default: "main"
- name: image-name
type: string
description: "Application name (matches directory in gitops repo)"
- name: new-image-url
type: string
description: "Full image URL with new tag"
- name: environment
type: string
description: "Target environment (staging, production)"
default: "staging"
- name: author-name
type: string
default: "Tekton CI"
- name: author-email
type: string
default: "tekton@mycompany.com"
workspaces:
- name: ssh-directory
description: "SSH credentials for git push"
steps:
- name: update-and-push
image: alpine/git:2.43.0
script: |
#!/usr/bin/env sh
set -euo pipefail
# Configure SSH
mkdir -p ~/.ssh
cp $(workspaces.ssh-directory.path)/id_rsa ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan github.com >> ~/.ssh/known_hosts
# Clone the GitOps repository
WORK_DIR="/tmp/gitops"
git clone --branch "$(params.gitops-branch)" \
--single-branch --depth 1 \
"$(params.gitops-repo-url)" "${WORK_DIR}"
cd "${WORK_DIR}"
# Update the image tag in the Kustomize overlay
OVERLAY_DIR="overlays/$(params.environment)/$(params.image-name)"
if [ -f "${OVERLAY_DIR}/kustomization.yaml" ]; then
echo "Updating Kustomize overlay..."
cd "${OVERLAY_DIR}"
# Extract registry/repo and tag from full URL
IMAGE_REF=$(echo "$(params.new-image-url)" | cut -d':' -f1)
IMAGE_TAG=$(echo "$(params.new-image-url)" | cut -d':' -f2)
# Use kustomize edit to update the image
kustomize edit set image "${IMAGE_REF}:${IMAGE_TAG}"
cd "${WORK_DIR}"
else
echo "ERROR: Overlay directory not found: ${OVERLAY_DIR}"
exit 1
fi
# Commit and push
git config user.name "$(params.author-name)"
git config user.email "$(params.author-email)"
git add -A
git diff --cached --quiet && {
echo "No changes to commit"
exit 0
}
git commit -m "chore($(params.environment)): update $(params.image-name) to $(params.new-image-url)
Triggered by Tekton PipelineRun
Image: $(params.new-image-url)"
git push origin "$(params.gitops-branch)"
echo "GitOps repository updated successfully"7.3 Complete CI Pipeline with GitOps Handoff
Add the GitOps update as the final step in the CI Pipeline:
# Add to the pipeline-ci-cd.yaml tasks list (after security-scan)
- name: update-gitops
taskRef:
name: update-gitops-repo
runAfter:
- security-scan
params:
- name: gitops-repo-url
value: "git@github.com:myorg/k8s-gitops.git"
- name: gitops-branch
value: "main"
- name: image-name
value: "$(params.image-name)"
- name: new-image-url
value: "$(tasks.build-push.results.image-url)"
- name: environment
value: "$(params.deploy-environment)"
workspaces:
- name: ssh-directory
workspace: git-credentials7.4 ArgoCD Application Configuration
# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-service-staging
namespace: argocd
annotations:
notifications.argoproj.io/subscribe.on-sync-succeeded.slack: ci-cd-notifications
notifications.argoproj.io/subscribe.on-sync-failed.slack: ci-cd-alerts
spec:
project: default
source:
repoURL: git@github.com:myorg/k8s-gitops.git
targetRevision: main
path: overlays/staging/my-service
destination:
server: https://kubernetes.default.svc
namespace: staging
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- PruneLast=true
retry:
limit: 3
backoff:
duration: 5s
factor: 2
maxDuration: 3m7.5 End-to-End Workflow Summary
- Developer pushes code to the application repository
- GitHub webhook fires to Tekton Triggers
- EventListener receives the event, validates the signature, filters with CEL
- PipelineRun is created automatically
- Tekton Pipeline runs: clone, test, lint, build, push to ECR, security scan
- Tekton Chains signs the image and generates SLSA provenance
- GitOps update Task commits the new image tag to the GitOps repository
- ArgoCD detects the change and syncs the deployment
- Kyverno verifies the image signature before allowing the Pod to run
- New version is deployed to the cluster
8. Production Operations
8.1 PipelineRun Cleanup Policies
PipelineRuns and TaskRuns accumulate in the cluster and consume etcd storage. Configure automatic cleanup:
# tekton-pruner-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: config-leader-election
namespace: tekton-pipelines
data: {}
---
# CronJob-based cleanup (recommended for fine-grained control)
apiVersion: batch/v1
kind: CronJob
metadata:
name: tekton-cleanup
namespace: tekton-pipelines
spec:
schedule: "0 2 * * *" # Daily at 2:00 AM UTC
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 1
activeDeadlineSeconds: 600
template:
spec:
serviceAccountName: tekton-cleanup-sa
restartPolicy: OnFailure
containers:
- name: cleanup
image: bitnami/kubectl:1.30
command:
- /bin/bash
- -c
- |
set -euo pipefail
# Delete completed PipelineRuns older than 7 days
echo "Cleaning up PipelineRuns older than 7 days..."
kubectl get pipelineruns -A \
-o jsonpath='{range .items[?(@.status.completionTime)]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.completionTime}{"\n"}{end}' | \
while IFS=$'\t' read -r ns name completed; do
if [ "$(date -d "${completed}" +%s)" -lt "$(date -d '7 days ago' +%s)" ]; then
echo "Deleting PipelineRun ${ns}/${name} (completed: ${completed})"
kubectl delete pipelinerun "${name}" -n "${ns}" --wait=false
fi
done
# Delete orphaned TaskRuns older than 7 days
echo "Cleaning up orphaned TaskRuns..."
kubectl get taskruns -A \
-o jsonpath='{range .items[?(@.status.completionTime)]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.completionTime}{"\n"}{end}' | \
while IFS=$'\t' read -r ns name completed; do
if [ "$(date -d "${completed}" +%s)" -lt "$(date -d '7 days ago' +%s)" ]; then
echo "Deleting TaskRun ${ns}/${name} (completed: ${completed})"
kubectl delete taskrun "${name}" -n "${ns}" --wait=false
fi
done
# Clean up completed Pods from Tekton builds
echo "Cleaning up completed build Pods..."
kubectl delete pods -A \
-l app.kubernetes.io/managed-by=tekton-pipelines \
--field-selector=status.phase=Succeeded \
--wait=false || true
echo "Cleanup complete"
resources:
requests: { cpu: "100m", memory: "128Mi" }
limits: { cpu: "500m", memory: "256Mi" }
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: tekton-cleanup-sa
namespace: tekton-pipelines
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tekton-cleanup
rules:
- apiGroups: ["tekton.dev"]
resources: ["pipelineruns", "taskruns"]
verbs: ["get", "list", "delete"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tekton-cleanup
subjects:
- kind: ServiceAccount
name: tekton-cleanup-sa
namespace: tekton-pipelines
roleRef:
kind: ClusterRole
name: tekton-cleanup
apiGroup: rbac.authorization.k8s.io8.2 Resource Management
Configure resource requests and limits for Tekton controllers and build Pods:
# tekton-controller-resources.yaml
# Patch the Tekton controller deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: tekton-pipelines-controller
namespace: tekton-pipelines
spec:
template:
spec:
containers:
- name: tekton-pipelines-controller
resources:
requests:
cpu: "200m"
memory: "256Mi"
limits:
cpu: "1000m"
memory: "1Gi"Set default resource limits for TaskRun Pods using LimitRange:
# tekton-builds-limitrange.yaml
apiVersion: v1
kind: LimitRange
metadata:
name: tekton-build-limits
namespace: tekton-builds
spec:
limits:
- type: Container
default:
cpu: "500m"
memory: "512Mi"
defaultRequest:
cpu: "100m"
memory: "128Mi"
max:
cpu: "4"
memory: "8Gi"
- type: PersistentVolumeClaim
max:
storage: "20Gi"Set a ResourceQuota to cap total build resource consumption:
# tekton-builds-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: tekton-build-quota
namespace: tekton-builds
spec:
hard:
requests.cpu: "16"
requests.memory: "32Gi"
limits.cpu: "32"
limits.memory: "64Gi"
persistentvolumeclaims: "20"
pods: "50"8.3 Monitoring with Prometheus
Tekton exposes Prometheus metrics on the controller's :9090/metrics endpoint. Configure a ServiceMonitor for scraping:
# tekton-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: tekton-pipelines
namespace: monitoring
labels:
release: prometheus
spec:
namespaceSelector:
matchNames:
- tekton-pipelines
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/part-of: tekton-pipelines
endpoints:
- port: metrics
interval: 30s
path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: tekton-triggers
namespace: monitoring
labels:
release: prometheus
spec:
namespaceSelector:
matchNames:
- tekton-pipelines
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/part-of: tekton-triggers
endpoints:
- port: metrics
interval: 30sKey Tekton metrics to monitor:
| Metric | Description | Alert Threshold |
|---|---|---|
tekton_pipelines_controller_pipelinerun_duration_seconds | PipelineRun execution duration | > 30 minutes |
tekton_pipelines_controller_pipelinerun_count | Total PipelineRun count by status | High failure rate |
tekton_pipelines_controller_taskrun_duration_seconds | TaskRun execution duration | > 15 minutes |
tekton_pipelines_controller_taskrun_count | Total TaskRun count by status | High failure rate |
tekton_pipelines_controller_running_pipelineruns_count | Currently running PipelineRuns | > 20 concurrent |
tekton_pipelines_controller_client_latency | API server request latency | > 5 seconds |
tekton_triggers_eventlistener_event_count | Incoming webhook events | Drop to 0 (indicates webhook failure) |
PrometheusRule for alerting:
# tekton-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: tekton-pipelines-alerts
namespace: monitoring
labels:
release: prometheus
spec:
groups:
- name: tekton-pipelines
interval: 60s
rules:
- alert: TektonPipelineRunHighFailureRate
expr: |
(
sum(rate(tekton_pipelines_controller_pipelinerun_count{status="failed"}[1h]))
/
sum(rate(tekton_pipelines_controller_pipelinerun_count[1h]))
) > 0.3
for: 15m
labels:
severity: warning
annotations:
summary: "High PipelineRun failure rate (> 30%)"
description: "More than 30% of PipelineRuns have failed in the last hour."
- alert: TektonPipelineRunStuck
expr: |
tekton_pipelines_controller_running_pipelineruns_count > 0
and
tekton_pipelines_controller_pipelinerun_duration_seconds{status="running"} > 3600
for: 10m
labels:
severity: critical
annotations:
summary: "PipelineRun stuck for over 1 hour"
description: "A PipelineRun has been running for more than 1 hour and may be stuck."
- alert: TektonControllerDown
expr: |
absent(up{job="tekton-pipelines-controller"} == 1)
for: 5m
labels:
severity: critical
annotations:
summary: "Tekton Pipelines controller is down"
description: "The Tekton Pipelines controller has been unreachable for 5 minutes."
- alert: TektonWebhookEventsDropped
expr: |
rate(tekton_triggers_eventlistener_event_count[5m]) == 0
and
tekton_triggers_eventlistener_event_count > 0
for: 30m
labels:
severity: warning
annotations:
summary: "No webhook events received for 30 minutes"
description: "The Tekton EventListener has not received any events. Verify webhook configuration."8.4 Log Management
Forward Tekton build logs to a centralized logging system:
# fluentbit-tekton-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-tekton
namespace: logging
data:
tekton-pipelines.conf: |
[INPUT]
Name tail
Tag tekton.*
Path /var/log/containers/*tekton-pipelines*.log
Parser docker
DB /var/log/flb_tekton.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[FILTER]
Name kubernetes
Match tekton.*
Kube_URL https://kubernetes.default.svc:443
Kube_Tag_Prefix tekton.var.log.containers.
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
[FILTER]
Name modify
Match tekton.*
Add log_source tekton-pipelines
[OUTPUT]
Name opensearch
Match tekton.*
Host opensearch.logging.svc
Port 9200
Index tekton-logs
Type _doc
Logstash_Format On
Logstash_Prefix tekton
Retry_Limit 58.5 Troubleshooting Guide
| Symptom | Possible Cause | Resolution |
|---|---|---|
PipelineRun stuck in Running | Pod scheduling failure or Step hanging | Check kubectl describe pod for the TaskRun Pod; look for resource quota exhaustion or node affinity issues |
TaskRun fails with ImagePullBackOff | Incorrect image reference or missing registry credentials | Verify image URL and that the ServiceAccount has imagePullSecrets or IRSA annotation |
| Workspace volume not mounted | PVC provisioning failure or StorageClass missing | Check kubectl get pvc in the build namespace; verify the StorageClass exists and has available capacity |
| EventListener not receiving events | Webhook URL misconfiguration or Ingress issue | Verify the Ingress is healthy; check kubectl logs for the EventListener Pod; test with curl -X POST |
| Chains not signing images | Chains controller not configured or signing key missing | Check kubectl logs -n tekton-chains; verify signing-secrets Secret exists; check chains-config ConfigMap |
| Results too large | Result exceeds 4096-byte limit | Use Workspaces to pass large data between Tasks instead of Results |
| Build runs out of disk | Workspace PVC too small for build cache | Increase the PVC size in the PipelineRun workspace volumeClaimTemplate |
| Parallel Tasks bottleneck | Too many concurrent PipelineRuns exceeding node capacity | Set ResourceQuota limits; use PriorityClasses to ensure critical builds are scheduled first |
Common debugging commands:
# View PipelineRun status and conditions
kubectl get pipelinerun <name> -n tekton-builds -o yaml | \
yq '.status.conditions'
# View TaskRun logs (all steps)
kubectl logs -n tekton-builds \
-l tekton.dev/pipelineRun=<pipelinerun-name> \
--all-containers --timestamps
# View specific Step logs
kubectl logs -n tekton-builds <taskrun-pod-name> -c step-<step-name>
# List all running PipelineRuns
kubectl get pipelineruns -A --field-selector=status.conditions[0].reason=Running
# Describe a failed TaskRun for error details
kubectl describe taskrun <name> -n tekton-builds
# Check Tekton controller logs for reconciliation errors
kubectl logs -n tekton-pipelines deployment/tekton-pipelines-controller \
--tail=100 --since=10m
# Check Chains controller logs
kubectl logs -n tekton-chains deployment/tekton-chains-controller \
--tail=50 --since=10m
# Verify EventListener is healthy
kubectl get eventlisteners -n tekton-builds
kubectl logs -n tekton-builds -l eventlistener=github-listener --tail=509. Best Practices
9.1 Task Reuse Patterns
Keep Tasks small and focused. A Task should do one thing well (clone, build, scan, deploy). Avoid monolithic Tasks that combine build, test, and deploy into a single 500-line script. Small Tasks are reusable across Pipelines and easier to debug.
Parameterize everything. Use params for image versions, flags, thresholds, and paths. This makes Tasks reusable across projects without modification.
Publish internal Tasks to a shared namespace. Create a
tekton-catalognamespace for organization-wide reusable Tasks. Teams reference them by name rather than copying YAML.Pin image tags in Steps. Never use
latesttags in Step images. Pin to specific versions (e.g.,golang:1.22.4,kaniko:v1.23.2) to ensure reproducible builds.Use Results for lightweight data passing. Pass commit SHAs, image tags, and status flags via Results. Use Workspaces for large data like source code and build artifacts.
9.2 Security
Use IRSA instead of long-lived credentials. Never store AWS access keys as Kubernetes Secrets. Use IRSA-annotated ServiceAccounts for ECR, S3, and KMS access.
Run build containers as non-root when possible. Set
securityContext.runAsNonRoot: truein the Task stepTemplate. Kaniko requires root, but most other Steps do not.Isolate build namespaces. Run Tekton builds in a dedicated namespace with NetworkPolicies that restrict egress to only necessary registries and APIs.
Rotate webhook secrets regularly. Use a CronJob or external secret manager to rotate GitHub/GitLab webhook secrets and update the corresponding Kubernetes Secrets.
Enable Tekton Chains for all production builds. Supply chain security is not optional. Every image deployed to production should have a verified signature and SLSA provenance.
# network-policy-tekton-builds.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: tekton-builds-egress
namespace: tekton-builds
spec:
podSelector: {}
policyTypes:
- Egress
egress:
# Allow DNS
- to: []
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Allow HTTPS to ECR, GitHub, and package registries
- to: []
ports:
- protocol: TCP
port: 443
# Allow access to Kubernetes API server
- to:
- ipBlock:
cidr: 172.20.0.1/32 # Adjust to your cluster API server IP
ports:
- protocol: TCP
port: 4439.3 Performance Optimization
- Use Workspace caching for Go modules, npm packages, and Maven repositories. Persistent Workspaces backed by PVCs avoid re-downloading dependencies on every build.
# Reusable PVC for Go module cache
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: go-module-cache
namespace: tekton-builds
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: gp3- Enable Kaniko layer caching. Use an ECR repository as a layer cache to speed up multi-stage Docker builds:
# In the kaniko build step args
- --cache=true
- --cache-repo=123456789012.dkr.ecr.us-east-1.amazonaws.com/kaniko-cache
- --cache-ttl=168h # 7 days- Disable the affinity assistant. The affinity assistant ensures Workspace PVCs are co-located with Pods, but it limits parallelism. When using
ReadWriteManyPVCs oremptyDirworkspaces, disable it:
# In feature-flags ConfigMap
disable-affinity-assistant: "true"- Use node selectors for build workloads. Schedule build Pods on dedicated node groups with fast local storage and high CPU:
# In PipelineRun taskRunTemplate
taskRunTemplate:
podTemplate:
nodeSelector:
node-role.kubernetes.io/build: "true"
tolerations:
- key: "build-workload"
operator: "Equal"
value: "true"
effect: "NoSchedule"- Set aggressive timeouts. Prevent runaway builds from consuming resources indefinitely. Set Pipeline-level, Task-level, and finally-block timeouts.
9.4 Cost Optimization
- Use Spot instances for build nodes. Build workloads are inherently interruptible. Use Karpenter with a Spot-only NodePool for build Pods:
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: tekton-builds
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
- key: node.kubernetes.io/instance-type
operator: In
values: ["m6i.xlarge", "m6a.xlarge", "m5.xlarge"]
taints:
- key: build-workload
value: "true"
effect: NoSchedule
limits:
cpu: "64"
memory: "128Gi"
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 30sScale build infrastructure to zero. When no PipelineRuns are active, Karpenter drains and terminates build nodes. There is zero compute cost during idle periods.
Clean up PVCs aggressively. Use
volumeClaimTemplatein PipelineRuns instead of pre-provisioned PVCs. The PVC is automatically deleted when the PipelineRun is garbage-collected.Right-size Step resource requests. Profile your builds and set accurate CPU/memory requests. Over-provisioned build containers waste node resources.
10. References
External References
- Tekton Documentation - Official Tekton project documentation
- Tekton Hub - Catalog of reusable Tasks and Pipelines
- Tekton Chains Documentation - Supply chain security with Tekton
- SLSA Framework - Supply-chain Levels for Software Artifacts
- Sigstore / Cosign - Container image signing and verification
- CD Foundation - Continuous Delivery Foundation (Tekton's home)
- Kaniko - Container image builds without Docker daemon
- in-toto Attestation Framework - Software supply chain attestation
Internal References
- CI Pipelines (GitLab Runner, GitHub Actions on EKS) - EKS-based CI runner infrastructure
- ArgoCD Installation - ArgoCD setup and configuration
- Image Security - Container image scanning, signing, and admission policies
- GitOps Multi-Cluster - Multi-cluster deployment with ArgoCD
- Secrets Management - Kubernetes secret handling best practices
- Kyverno Policy Management - Policy enforcement for Kubernetes