Cloud Native Operations

English

한국어

English

한국어

Appearance

Extending Kubernetes Quiz ​

This quiz tests your conceptual and practical knowledge about Kubernetes extension mechanisms. It covers topics such as Custom Resource Definitions (CRD), custom controllers, API extensions, webhooks, and the operator pattern.

Multiple Choice Questions ​

  1. What is the most common way to define custom resources in Kubernetes?
    • A) Define resource schema using ConfigMap
    • B) Create a CustomResourceDefinition (CRD)
    • C) Directly modify API server code
    • D) Use the Aggregation Layer
Show Answer

Answer: B) Create a CustomResourceDefinition (CRD)

Explanation: The most common way to define custom resources in Kubernetes is to create a CustomResourceDefinition (CRD). CRD is a mechanism that allows you to extend the Kubernetes API by defining new resource types.

Using CRDs provides the following benefits:

  • You can add new resource types without modifying the existing Kubernetes API server.
  • You can manage custom resources using standard Kubernetes tools like kubectl.
  • You can leverage features such as resource validation, versioning, and status subresources.

CRD example:

yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

Issues with other options:

  • ConfigMaps are used to store configuration data and are not suitable for API extensions.
  • Directly modifying API server code is complex, difficult to maintain, and can cause issues during upgrades.
  • The Aggregation Layer is another way to define custom resources, but it's more complex than CRDs since it requires implementing a separate API server.
  1. What is the main purpose of a Kubernetes Operator?
    • A) Optimize resource usage of cluster nodes
    • B) Encode operational knowledge of complex applications into automated software
    • C) Improve Kubernetes API server performance
    • D) Extend cluster networking functionality
Show Answer

Answer: B) Encode operational knowledge of complex applications into automated software

Explanation: The main purpose of a Kubernetes Operator is to encode operational knowledge of complex applications into automated software. The operator pattern is a software implementation of how a human operator manages complex applications.

Key characteristics of operators:

  • Combines custom resources with custom controllers to implement application-specific logic.
  • Automates operational tasks such as application deployment, upgrades, backup, recovery, and scaling.
  • Continuously monitors application state and adjusts to the desired state.
  • Codifies domain knowledge to enable declarative management of complex applications.

Common use cases for operators:

  • Automated management of databases (e.g., PostgreSQL, MySQL, MongoDB)
  • Deployment and configuration of messaging systems (e.g., Kafka, RabbitMQ)
  • Setup and maintenance of monitoring systems (e.g., Prometheus)
  • Management of service meshes (e.g., Istio)

Operator example - Prometheus Operator:

yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: prometheus
  replicas: 2
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi

With this simple declarative configuration, the Prometheus Operator automatically handles complex tasks such as:

  • Deploying Prometheus servers
  • Creating and managing configuration files
  • Automatic discovery of service monitoring targets
  • High availability setup
  • Storage management when a storage/volumeClaimTemplate is configured
  • Upgrade coordination

Other options are not the main purpose of operators:

  • Resource usage optimization is the role of HPA (Horizontal Pod Autoscaler), VPA (Vertical Pod Autoscaler), etc.
  • Improving API server performance is not the main purpose of operators.
  • Extending networking functionality is the role of CNI plugins or service meshes.
  1. What is the main function of Admission Webhooks in Kubernetes?
    • A) Handle authentication for API server
    • B) Intercept resource creation or modification requests to validate or modify them
    • C) Monitor cluster events and send alerts
    • D) Provide APIs for integration with external systems
Show Answer

Answer: B) Intercept resource creation or modification requests to validate or modify them

Explanation: The main function of Admission Webhooks in Kubernetes is to intercept resource creation or modification requests to validate or modify them. Admission webhooks provide a mechanism to intercept requests before the Kubernetes API server stores them in persistent storage (etcd).

Two types of admission webhooks:

  1. Validating Webhook:

    • Validates resource create, update, and delete requests.
    • Can allow or deny requests but cannot modify them.
    • Used for policy enforcement, security checks, configuration validation, etc.

  2. Mutating Webhook:

    • Can validate and modify resource create and update requests.
    • Used for setting defaults, injecting sidecar containers, adding labels, etc.
    • Runs before validating webhooks.

Admission webhook configuration example:

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-policy-validator
webhooks:
- name: pod-policy.example.com
  clientConfig:
    service:
      namespace: webhook-namespace
      name: webhook-service
      path: "/validate-pods"
    caBundle: <base64-encoded-ca-cert>
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE", "UPDATE"]
    resources: ["pods"]
    scope: "Namespaced"
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5

Common use cases for admission webhooks:

  • Enforcing security policies (e.g., prohibiting privileged containers)
  • Enforcing resource requests and limits
  • Automatic injection of sidecar containers (e.g., Istio)
  • Automatic addition of labels and annotations
  • Restricting image registries
  • Applying namespace-based policies

Issues with other options:

  • Handling authentication for API server is the role of Authentication Plugins.
  • Monitoring cluster events and sending alerts is the role of event listeners or monitoring tools.
  • Providing APIs for integration with external systems is a general purpose of API extensions, but not the main function of admission webhooks.
  1. What is the main purpose of the Aggregation Layer in the Kubernetes API server?
    • A) Optimize API server performance
    • B) Consolidate multiple API servers into a single API server
    • C) Integrate custom API servers into the main API server to extend the API
    • D) Provide service discovery within the cluster
Show Answer

Answer: C) Integrate custom API servers into the main API server to extend the API

Explanation: The main purpose of the Aggregation Layer in the Kubernetes API server is to integrate custom API servers into the main API server to extend the API. This provides another way to extend the Kubernetes API, offering more powerful features than CRDs, though more complex.

Key characteristics of the Aggregation Layer:

  • Integrates custom API servers into the URL space of the Kubernetes API server.
  • Custom API servers can have their own storage, business logic, API versions, etc.
  • The main API server proxies requests to the appropriate custom API server.
  • The main API server authenticates the user; the extension must verify the authenticating proxy and perform delegated authorization (SubjectAccessReview).

Cases where the Aggregation Layer is used:

  • When complex validation logic is needed
  • When a custom storage backend is required
  • When different behavior from existing APIs is needed
  • When resource transformation or special state calculation is required

APIService resource example:

yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  caBundle: <base64-encoded-serving-ca>
  groupPriorityMinimum: 100
  versionPriority: 100

This configuration routes requests for the metrics.k8s.io/v1beta1 API group to the metrics-server service in the kube-system namespace.

Real examples using the Aggregation Layer:

  • metrics-server: Provides node and pod resource usage metrics
  • service-catalog (archived): historical service-broker integration
  • custom-metrics-apiserver: Provides custom metrics for HPA

Issues with other options:

  • The Aggregation Layer is not for optimizing API server performance.
  • Consolidating multiple API servers into one is not an accurate description. The Aggregation Layer integrates multiple API servers into a single URL space, but the servers run separately.
  • The Aggregation Layer does not provide service discovery. That is the role of Kubernetes Services.
  1. What is the main role of a Custom Controller in Kubernetes?
    • A) Monitor resource usage of cluster nodes
    • B) Observe the state of custom resources and reconcile to the desired state
    • C) Handle authentication and authorization for API server requests
    • D) Manage cluster networking
Show Answer

Answer: B) Observe the state of custom resources and reconcile to the desired state

Explanation: The main role of a Custom Controller in Kubernetes is to observe the state of custom resources and reconcile to the desired state. Controllers implement the "reconciliation loop," a core operational pattern in Kubernetes, to continuously adjust the system's actual state to the desired state.

Key characteristics of custom controllers:

  • Watch specific resource types (typically custom resources defined via CRDs).
  • React to resource change events and execute business logic.
  • Perform tasks to adjust the actual state of resources to the desired state.
  • Update the status field of resources to reflect the current state.

Common components of custom controllers:

  1. Informer: Watches the Kubernetes API server and receives resource change events.
  2. Work Queue: Stores and manages events to be processed.
  3. Reconciler: Compares the desired state and actual state of resources and performs necessary tasks.
  4. Client: Interacts with the Kubernetes API to create, update, and delete resources.

Custom controller example - simple reconcile function:

go
func (c *Controller) reconcile(key string) error {
    // Split key into namespace and name
    namespace, name, err := cache.SplitMetaNamespaceKey(key)
    if err != nil {
        return err
    }

    // Get custom resource
    instance, err := c.customResourceLister.CustomResources(namespace).Get(name)
    if errors.IsNotFound(err) {
        // Already deleted. External cleanup must run earlier using a finalizer.
        return nil
    }
    if err != nil {
        return err
    }

    // Check resource state and perform necessary tasks
    // e.g., create sub-resources, integrate with external systems, update status, etc.

    // Update status
    instanceCopy := instance.DeepCopy()
    instanceCopy.Status.Phase = "Reconciled"
    _, err = c.customResourceClient.CustomResources(namespace).UpdateStatus(context.TODO(), instanceCopy, metav1.UpdateOptions{})
    return err
}

Common use cases for custom controllers:

  • Automating complex application deployment and management
  • Integration with external systems (e.g., cloud resource provisioning)
  • Automating backup and recovery processes
  • Implementing advanced deployment strategies (e.g., canary deployments, blue-green deployments)

Issues with other options:

  • Monitoring resource usage of cluster nodes is the role of metrics-server or monitoring tools like Prometheus.
  • Handling authentication and authorization for API server requests is the role of authentication and authorization plugins.
  • Managing cluster networking is the role of CNI plugins or network controllers.
  1. What format is used to define validation schemas for CRDs (CustomResourceDefinitions) in Kubernetes?
    • A) JSON Schema
    • B) XML Schema
    • C) OpenAPI v3 Schema
    • D) GraphQL Schema
Show Answer

Answer: C) OpenAPI v3 Schema

Explanation: The format used to define validation schemas for CRDs (CustomResourceDefinitions) in Kubernetes is OpenAPI v3 Schema. This schema defines the structure and field types of custom resources and is used by the API server to validate resource creation and update requests.

Example of CRD validation schema:

yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          required: ["spec"]
          properties:
            spec:
              type: object
              required: ["cronSpec", "image"]
              properties:
                cronSpec:
                  type: string
                  pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

In this example, the openAPIV3Schema field defines the following validation rules:

  • The spec field is required.
  • Within spec, the cronSpec and image fields are required.
  • cronSpec is a string and must follow the cron expression pattern.
  • image is a string.
  • replicas is an integer and must be between 1 and 10.

OpenAPI v3 Schema provides various validation features:

  • Specifying required fields
  • Data type validation (string, number, boolean, object, array, etc.)
  • String pattern validation (regular expressions)
  • Number range validation (minimum, maximum)
  • Array length validation
  • Enum value validation
  • Nested object structure definition

Issues with other options:

  • JSON Schema is the basis for OpenAPI, but Kubernetes specifically uses OpenAPI v3 Schema.
  • XML Schema is not used in the Kubernetes API.
  • GraphQL Schema is used for GraphQL APIs but not for the Kubernetes API.
  1. What is the main benefit of enabling the status subresource for custom resources in Kubernetes?
    • A) Faster resource creation
    • B) Separation of spec and status updates and RBAC control
    • C) Automatic backup and recovery features
    • D) Simplified resource version management
Show Answer

Answer: B) Separation of spec and status updates and RBAC control

Explanation: The main benefit of enabling the status subresource for custom resources in Kubernetes is the separation of spec and status updates and the ability to control access through RBAC (Role-Based Access Control).

Benefits of enabling the status subresource:

  1. Separation of Concerns:

    • The spec field defines the desired state specified by users.
    • The status field reports the actual state observed by controllers.
    • This separation clarifies the responsibilities of users and controllers.

  2. RBAC Control:

    • Status update permissions can be granted only to controllers while giving regular users read-only permissions.
    • This prevents unauthorized modification of status information.

  3. Conflict Prevention:

    • Spec and status are separated, but both share resourceVersion, so concurrent writes can still return 409 Conflict.
    • Read the latest object and retry/reconcile on conflicts.

  4. Scale Subresource Support:

    • The scale subresource is configured independently, with specReplicasPath, statusReplicasPath and an optional labelSelectorPath.
    • This enables the use of standard Kubernetes scaling tools like HPA (Horizontal Pod Autoscaler).

Example of enabling status subresource in CRD:

yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
  - name: v1
    served: true
    storage: true
    subresources:
      status: {}
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              cronSpec:
                type: string
              image:
                type: string
          status:
            type: object
            properties:
              phase:
                type: string
              message:
                type: string
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab

Example of status update from controller:

go
// Update status only
statusUpdate := instance.DeepCopy() // Preserve resourceVersion from a current read.
statusUpdate.Status.Phase = "Running"
statusUpdate.Status.Message = "Resource is running"

_, err = c.clientset.MyGroup().MyResources(namespace).UpdateStatus(ctx, statusUpdate, metav1.UpdateOptions{})

Issues with other options:

  • The status subresource does not improve resource creation speed.
  • It does not provide automatic backup and recovery features.
  • Resource version management is handled through the CRD's versions field and is not directly related to the status subresource.
  1. What is the main purpose of Webhook Conversion in Kubernetes?
    • A) Route API requests to external services
    • B) Handle conversion between different versions of custom resources
    • C) Convert authentication tokens to different formats
    • D) Convert log data to structured formats
Show Answer

Answer: B) Handle conversion between different versions of custom resources

Explanation: The main purpose of Webhook Conversion in Kubernetes is to handle conversion between different versions of custom resources. This feature supports multiple versions of CRDs and enables smooth migration between versions.

Key characteristics of webhook conversion:

  1. Multi-Version Support:

    • CRDs can support multiple API versions simultaneously (e.g., v1alpha1, v1beta1, v1).
    • Each version can have different schemas and fields.

  2. Automatic Conversion:

    • The API server automatically handles conversion between the version requested by the client and the storage version.
    • The conversion webhook provides custom conversion logic in this process.

  3. Storage Version Independence:

    • Even if the storage version changes, clients using previous versions can continue to work.
    • The webhook handles bidirectional conversion between served versions; changing storage:true does not rewrite existing data. Migrate stored objects before removing old status.storedVersions entries.

Example of conversion webhook configuration in CRD:

yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        required:
        - spec
        properties:
          spec:
            type: object
            required:
            - cronSpec
            - image
            properties:
              cronSpec:
                type: string
              image:
                type: string
              replicas:
                type: integer
          status:
            type: object
            properties:
              phase:
                type: string
  - name: v1beta1
    served: true
    storage: false
    schema:
      openAPIV3Schema:
        type: object
        required:
        - spec
        properties:
          spec:
            type: object
            required:
            - cron
            - image
            properties:
              cron:
                type: string
              image:
                type: string
              replicas:
                type: integer
          status:
            type: object
            properties:
              phase:
                type: string
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        service:
          namespace: webhook-system
          name: crd-conversion-webhook
          path: /convert
        caBundle: <base64-encoded-ca-cert>
      conversionReviewVersions:
      - v1
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab

Example of conversion webhook server implementation:

go
// Include encoding/json, net/http, fmt and the Kubernetes types below in the server.
// apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
// metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
// "k8s.io/apimachinery/pkg/runtime"
func convertCron(raw []byte, desired string) ([]byte, error) {
    var obj map[string]interface{}
    if err := json.Unmarshal(raw, &obj); err != nil { return nil, err }
    source, _ := obj["apiVersion"].(string)
    valid := func(v string) bool { return v == "stable.example.com/v1" || v == "stable.example.com/v1beta1" }
    if !valid(source) || !valid(desired) || obj["kind"] != "CronTab" {
        return nil, fmt.Errorf("unsupported CronTab conversion %q -> %q", source, desired)
    }
    spec, ok := obj["spec"].(map[string]interface{})
    if !ok { return nil, fmt.Errorf("spec object required") }
    if source != desired {
        oldField, newField := "cron", "cronSpec"
        if desired == "stable.example.com/v1beta1" { oldField, newField = newField, oldField }
        value, ok := spec[oldField]
        if !ok { return nil, fmt.Errorf("missing %s", oldField) }
        if _, collision := spec[newField]; collision { return nil, fmt.Errorf("ambiguous cron fields") }
        spec[newField] = value
        delete(spec, oldField)
    }
    obj["apiVersion"] = desired
    // Preserve metadata (including UID/resourceVersion), status and all other fields.
    return json.Marshal(obj)
}

func serveConvert(w http.ResponseWriter, r *http.Request) {
    var review apiextensionsv1.ConversionReview
    if r.Method != http.MethodPost || r.Body == nil {
        http.Error(w, "POST body required", http.StatusBadRequest); return
    }
    if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 8<<20)).Decode(&review); err != nil ||
        review.APIVersion != "apiextensions.k8s.io/v1" || review.Kind != "ConversionReview" ||
        review.Request == nil || review.Request.UID == "" {
        http.Error(w, "Invalid ConversionReview v1", http.StatusBadRequest); return
    }
    response := &apiextensionsv1.ConversionResponse{
        UID: review.Request.UID, Result: metav1.Status{Status: "Success"},
    }
    for _, obj := range review.Request.Objects {
        raw, err := convertCron(obj.Raw, review.Request.DesiredAPIVersion)
        if err != nil {
            response.Result = metav1.Status{Status: "Failure", Message: err.Error()}
            response.ConvertedObjects = nil
            break
        }
        response.ConvertedObjects = append(response.ConvertedObjects, runtime.RawExtension{Raw: raw})
    }
    out := apiextensionsv1.ConversionReview{
        TypeMeta: metav1.TypeMeta{APIVersion: "apiextensions.k8s.io/v1", Kind: "ConversionReview"},
        Response: response,
    }
    w.Header().Set("Content-Type", "application/json")
    _ = json.NewEncoder(w).Encode(out)
}

Issues with other options:

  • Routing API requests to external services is the role of API aggregation.
  • Converting authentication tokens is the role of authentication plugins.
  • Converting log data is the role of logging systems.
  1. Which of the following is NOT a framework that helps develop Kubernetes operators?
    • A) Operator Framework
    • B) Kubebuilder
    • C) Metacontroller
    • D) Kubespray
Show Answer

Answer: D) Kubespray

Explanation: Kubespray is not a framework that helps develop Kubernetes operators. Kubespray is a tool for deploying and managing Kubernetes clusters using Ansible playbooks. It focuses on cluster installation and configuration and is not related to operator development.

Actual frameworks for Kubernetes operator development are:

  1. Operator Framework:

    • Operator development toolkit developed by Red Hat
    • Key components:
      • Operator SDK: Operator scaffolding and development tools
      • Operator Lifecycle Manager (OLM): Operator installation and upgrade management
      • Operator Metering: Operator usage reporting
    • Supports Go, Ansible, and Helm-based operator development

  2. Kubebuilder:

    • Framework developed by Kubernetes SIG (Special Interest Group)
    • SDK for developing controllers in Go
    • Provides code generation, CRD management, and testing tools
    • Based on controller-runtime library

  3. Metacontroller:

    • Lightweight webhook-based operator framework
    • Can implement controller logic in various languages
    • Declarative controller definition
    • Suitable for quickly developing simple controllers

Feature comparison of each framework:

FrameworkPrimary LanguageComplexityFeatures
Operator FrameworkGo, Ansible, HelmMedium-HighComprehensive toolkit, various development options
KubebuilderGoMediumStandardized patterns, code generation tools
MetacontrollerLanguage-independentLowWebhook-based, simple implementation

Operator development example (using Kubebuilder):

bash
# Initialize project
kubebuilder init --domain example.com --repo github.com/example/my-operator

# Create API
kubebuilder create api --group apps --version v1alpha1 --kind MyApp

# Implement controller (edit controller.go file)

# Create CRD and deploy controller
make install
make deploy

Explanations of other options:

  • Operator Framework is Red Hat's comprehensive operator development toolkit.
  • Kubebuilder is a Go-based controller development framework developed by Kubernetes SIG.
  • Metacontroller is a lightweight webhook-based operator framework.
  1. What is the main difference between Custom Resource Definitions (CRD) and Aggregated APIs in Kubernetes?
    • A) CRDs do not support versioning, but Aggregated APIs do
    • B) CRDs do not support validation schemas, but Aggregated APIs do
    • C) CRDs are simple to implement but have limited flexibility, while Aggregated APIs are complex to implement but provide more flexibility
    • D) CRDs only support cluster-scoped resources, and Aggregated APIs only support namespace-scoped resources
Show Answer

Answer: C) CRDs are simple to implement but have limited flexibility, while Aggregated APIs are complex to implement but provide more flexibility

Explanation: The main difference between Custom Resource Definitions (CRD) and Aggregated APIs in Kubernetes is the balance between implementation complexity and flexibility. CRDs are simple to implement but have limited flexibility, while Aggregated APIs are complex to implement but provide more flexibility.

Characteristics of CRDs (CustomResourceDefinitions):

  • Simple implementation: You can define new API resources with a single YAML file.
  • Uses existing API server: No need to implement a separate API server.
  • Limited flexibility:
    • Storage is limited to etcd.
    • Inherits the behavior of the default API server.
    • Supports CEL validation and admission/conversion webhooks; custom storage and nonstandard API behavior remain differentiators.
  • Extension through webhooks: Some functionality can be extended through validating webhooks, conversion webhooks, etc.

Characteristics of Aggregated APIs:

  • Complex implementation: Must develop and deploy a separate API server.
  • High flexibility:
    • Can use custom storage backends
    • Can implement complex business logic
    • Can implement custom authentication and authorization logic
    • Can implement special state calculation and conversion logic
  • Full API server functionality: Can leverage all features of a standard Kubernetes API server.

Selection Criteria:

CriteriaChoose CRDChoose Aggregated API
Implementation complexityLow - Simple YAML definitionHigh - Requires separate API server development
Development timeShort - Can implement in minutesLong - Requires complete API server development
MaintenanceEasy - Managed by existing API serverDifficult - Requires maintaining separate service
Storage optionsetcd onlyCustom storage backends possible
Business logicLimited - Implement via controllersFlexible - Can implement directly in API server
PerformanceGenerally goodCustom optimization possible
Use casesSimple CRUD operations, standard patternsComplex API behavior, special validation/conversion

Example Scenarios:

Suitable cases for CRDs:

  • Simple application configuration management
  • Basic CRUD operations as main requirements
  • Rapid prototyping and development

Suitable cases for Aggregated APIs:

  • Integration with external databases
  • Complex data transformation and validation
  • Special authentication mechanisms needed
  • High-performance or special-purpose APIs

Issues with other options:

  • CRDs do support versioning (A is incorrect).
  • CRDs do support validation schemas through OpenAPI v3 Schema (B is incorrect).
  • Both CRDs and Aggregated APIs support cluster-scoped and namespace-scoped resources (D is incorrect).

The cronSpec regex only checks a restricted five-field shape using numbers, stars and steps. It does not validate ranges, lists, names or all legal time values. Add a real cron parser in the controller/webhook. A CRD alone does not execute scheduled work.

Short Answer Questions ​

  1. Explain how to define custom resources using CustomResourceDefinition (CRD) and how to set validation rules for those resources.
Show Answer

Answer:

How to Define CustomResourceDefinition (CRD):

CRD is a mechanism that extends the Kubernetes API to define new resource types. When you create a CRD, a new RESTful API endpoint is created, and you can manage that resource using standard tools like kubectl.

1. Basic CRD Structure:

yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: <plural>.<group>
spec:
  group: <api-group>
  names:
    kind: <kind-name>
    plural: <plural-name>
    singular: <singular-name>
    shortNames:
    - <short-name>
  scope: Namespaced
  versions:
  - name: <version>
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        required:
        - spec
        properties:
          spec:
            type: object
            required:
            - cronSpec
            - image
            properties:
              cronSpec:
                type: string
              image:
                type: string
              replicas:
                type: integer
          status:
            type: object
            properties:
              phase:
                type: string

2. Setting Validation Rules:

Validation rules for CRDs are set through the openAPIV3Schema field. This schema follows OpenAPI v3 format and defines the structure and field types of resources.

Basic Validation Rules Example:

yaml
schema:
  openAPIV3Schema:
    type: object
    required: ["spec"]
    properties:
      spec:
        type: object
        required: ["cronSpec", "image"]
        properties:
          cronSpec:
            type: string
            pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
          image:
            type: string
          replicas:
            type: integer
            minimum: 1
            maximum: 10
            default: 1

3. Advanced Validation Features:

OpenAPI v3 Schema provides various validation features:

  • Required fields: Add field names to the required array

    yaml
    required: ["fieldName1", "fieldName2"]

  • Data types: Specify data types using the type field

    yaml
    type: string  # Choose one supported type, not a pipe-separated union.

  • String constraints: Validate string length and patterns

    yaml
    minLength: 3
maxLength: 64
pattern: '^[a-z0-9]([-a-z0-9]*[a-z0-9])?$'

  • Number constraints: Validate number ranges

    yaml
    minimum: 0
maximum: 100
multipleOf: 5

  • Array constraints: Validate array length and items

    yaml
    minItems: 1
maxItems: 10
x-kubernetes-list-type: set
items:
  type: string

  • Enum values: Specify allowed value list

    yaml
    enum: ["value1", "value2", "value3"]

  • Default values: Specify default values for fields

    yaml
    default: "default-value"

  • Map values: Define typed map entries; additionalProperties: false is forbidden in CRDs. Unknown fields are normally pruned.

    yaml
    additionalProperties:
  type: string

4. Complete CRD Example:

yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          required: ["spec"]
          properties:
            spec:
              type: object
              required: ["cronSpec", "image"]
              properties:
                cronSpec:
                  type: string
                  pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
                  default: 1
            status:
              type: object
              properties:
                active:
                  type: boolean
                lastScheduleTime:
                  type: string
                  format: date-time
      subresources:
        status: {}  # Enable status subresource
      additionalPrinterColumns:
        - name: Schedule
          type: string
          description: The cron schedule
          jsonPath: .spec.cronSpec
        - name: Image
          type: string
          description: The image to use
          jsonPath: .spec.image
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

5. Applying and Using CRD:

bash
# Apply CRD
kubectl apply -f crontab-crd.yaml

# Create custom resource
cat <<EOF | kubectl apply -f -
apiVersion: stable.example.com/v1
kind: CronTab
metadata:
  name: my-crontab
spec:
  cronSpec: "* * * * */5"
  image: my-cron-image
  replicas: 3
EOF

# View custom resources
kubectl get crontabs
kubectl get ct  # Using short name

6. Testing Validation Rules:

Creating a resource with invalid values will result in validation errors:

bash
cat <<EOF | kubectl apply -f -
apiVersion: stable.example.com/v1
kind: CronTab
metadata:
  name: invalid-crontab
spec:
  cronSpec: "invalid-cron-spec"  # Pattern mismatch
  image: my-cron-image
  replicas: 20  # Exceeds maximum
EOF

This command will return errors like: The API server rejects invalid cronSpec/replicas through CRD schema validation. No separate admission webhook is installed in this example; exact error wording varies by version.

7. Best Practices:

  • Clear and detailed schema definitions
  • Explicitly specify required fields
  • Provide appropriate default values
  • Limit string patterns and number ranges
  • Enable status subresource
  • Configure additional printer columns
  • Establish versioning strategy
  1. Explain the core concepts of the Kubernetes Operator Pattern and the common methods for implementing operators.
Show Answer

Answer:

Core Concepts of the Kubernetes Operator Pattern:

The operator pattern is a Kubernetes extension mechanism that encodes application-specific operational knowledge into software to automatically manage complex applications. This pattern mimics how a human operator manages complex systems.

1. Core Concepts:

  • Declarative Management: Users declare the desired state, and the operator adjusts the current state to the desired state.
  • Encoding Domain Knowledge: Codifies operational knowledge and best practices for specific applications.
  • Reconciliation Loop: Continuously observes actual state and adjusts to desired state.
  • Custom Resources: Used to store application-specific configuration and state.
  • Controllers: Watch changes to custom resources and perform necessary tasks.

2. Common Functions of Operators:

  • Installation and Upgrades: Deploy application components and upgrade versions
  • Automatic Recovery: Detect failures and perform recovery tasks
  • Backup and Restore: Automate data backup and restore processes
  • Scaling: Automatic expansion and contraction based on workload requirements
  • Configuration Management: Handle application-specific configuration changes
  • Operations Automation: Automate routine operational tasks (e.g., database compaction, index rebuilding)

Methods for Implementing Operators:

1. Using Operator SDK:

Operator SDK is part of Red Hat's Operator Framework and is a tool that simplifies operator development.

bash
: "${OPERATOR_IMAGE:?Set a registry image tag or digest you control}"
# Install a supported released SDK/CLI and verify its checksum before scaffolding.

# Create Go-based operator project
operator-sdk init --domain example.com --repo github.com/example/my-operator

# Create API
operator-sdk create api --group apps --version v1alpha1 --kind MyApp --resource --controller

# Create CRD
make manifests

# Build and deploy operator
make docker-build docker-push IMG="$OPERATOR_IMAGE"
make deploy IMG="$OPERATOR_IMAGE"

2. Using Kubebuilder:

Kubebuilder is a framework developed by Kubernetes SIG that provides tools for controller development.

bash
: "${OPERATOR_IMAGE:?Set a registry image tag or digest you control}"
# Install a supported released SDK/CLI and verify its checksum before scaffolding.

# Initialize project
kubebuilder init --domain example.com --repo github.com/example/my-operator

# Create API
kubebuilder create api --group apps --version v1alpha1 --kind MyApp

# Create CRD and deploy controller
make install
make deploy IMG="$OPERATOR_IMAGE"

3. Controller Implementation:

The core of an operator is the reconciliation function. This function observes the current state of custom resources and performs necessary tasks.

go
// Uses the scaffold's client.Client and Scheme. Import controllerutil and sort.
func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var myApp appsv1alpha1.MyApp
    if err := r.Get(ctx, req.NamespacedName, &myApp); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    if !myApp.DeletionTimestamp.IsZero() { return ctrl.Result{}, nil }
    deployment := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: myApp.Name, Namespace: myApp.Namespace}}
    _, err := controllerutil.CreateOrUpdate(ctx, r.Client, deployment, func() error {
        if deployment.UID != "" && !metav1.IsControlledBy(deployment, &myApp) {
            return fmt.Errorf("refusing to modify Deployment owned by another actor")
        }
        labels := map[string]string{"myapp.example.com/owner-uid": string(myApp.UID)}
        if deployment.Spec.Selector == nil {
            deployment.Spec.Selector = &metav1.LabelSelector{MatchLabels: labels}
        }
        if deployment.Spec.Template.Labels == nil { deployment.Spec.Template.Labels = map[string]string{} }
        deployment.Spec.Template.Labels["myapp.example.com/owner-uid"] = string(myApp.UID)
        size := myApp.Spec.Size
        deployment.Spec.Replicas = &size
        keys := []string{}
        for key := range myApp.Spec.Config { keys = append(keys, key) }
        sort.Strings(keys) // Stable output avoids unnecessary updates.
        env := []corev1.EnvVar{}
        for _, key := range keys { env = append(env, corev1.EnvVar{Name: key, Value: myApp.Spec.Config[key]}) }
        // This operator owns its application container; update image as well as replicas.
        deployment.Spec.Template.Spec.Containers = []corev1.Container{{
            Name: "myapp", Image: myApp.Spec.Image, Env: env,
            Ports: []corev1.ContainerPort{{ContainerPort: 8080, Name: "http"}},
        }}
        return ctrl.SetControllerReference(&myApp, deployment, r.Scheme)
    })
    if err != nil { return ctrl.Result{}, err }
    if myApp.Status.AvailableReplicas != deployment.Status.AvailableReplicas {
        myApp.Status.AvailableReplicas = deployment.Status.AvailableReplicas
        if err := r.Status().Update(ctx, &myApp); err != nil { return ctrl.Result{}, err }
    }
    return ctrl.Result{}, nil
}

func (r *MyAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&appsv1alpha1.MyApp{}).
        Owns(&appsv1.Deployment{}).
        Complete(r)
}

4. Custom Resource Definition:

Define the API for custom resources managed by the operator.

go
// MyApp API
type MyAppSpec struct {
    // Application image
    Image string `json:"image"`

    // Number of replicas
    Size int32 `json:"size"`

    // Configuration options
    Config map[string]string `json:"config,omitempty"`
}

type MyAppStatus struct {
    // Number of available replicas
    AvailableReplicas int32 `json:"availableReplicas"`

    // Last update time
    LastUpdateTime metav1.Time `json:"lastUpdateTime,omitempty"`

    // Status message
    Message string `json:"message,omitempty"`
}

// MyApp resource
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=`.spec.size`
// +kubebuilder:printcolumn:name="Available",type=integer,JSONPath=`.status.availableReplicas`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type MyApp struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MyAppSpec   `json:"spec,omitempty"`
    Status MyAppStatus `json:"status,omitempty"`
}

5. Ansible or Helm-Based Operators:

Operator SDK also supports operator development using Ansible or Helm in addition to Go.

Ansible-based operator:

bash
# Create Ansible-based operator
operator-sdk init --plugins=ansible --domain example.com
operator-sdk create api --group apps --version v1alpha1 --kind MyApp --generate-role

# Edit roles/myapp/tasks/main.yml

Helm-based operator:

bash
# Create Helm-based operator
operator-sdk init --plugins=helm --domain example.com --helm-chart=<chart-name>

6. Deploying and Testing Operators:

bash
# Deploy operator
make deploy

# Create custom resource
kubectl apply -f config/samples/

# Check operator logs
kubectl logs -f deployment/my-operator-controller-manager -n my-operator-system

# Check resource status
kubectl get myapps
kubectl describe myapp my-app

7. Operator Maturity Levels:

The operator maturity model defines the capability levels of operators:

  1. Basic Install: Application installation and configuration
  2. Seamless Upgrades: Automatic upgrades between versions
  3. Full Lifecycle: Backup, restore, failure recovery, etc.
  4. Deep Insights: Auto-scaling, tuning, etc.
  5. Auto Pilot: Monitoring-based automatic optimization

8. Best Practices:

  • Implement features incrementally (start with simple things)
  • Thorough error handling and logging
  • Ensure idempotency (same result for same input)
  • Set owner references (resource hierarchy and garbage collection)
  • Report progress through status updates
  • Write unit tests and integration tests
  • Clear documentation
  1. Explain the types of Admission Webhooks in Kubernetes and the use cases for each.
Show Answer

Answer:

Types and Use Cases of Kubernetes Admission Webhooks:

Admission webhooks are mechanisms that can intercept and modify or validate requests before the Kubernetes API server stores them in persistent storage (etcd). Admission webhooks are broadly divided into two types: Mutating webhooks and Validating webhooks.

1. Mutating Webhook:

Mutating webhooks can modify request objects coming into the API server. These webhooks run before validating webhooks.

Key Characteristics:

  • Can modify request objects
  • Multiple mutating webhooks run in a chain
  • Each webhook receives the object modified by the previous webhook

Configuration Example:

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
- name: sidecar-injector.example.com
  clientConfig:
    service:
      namespace: webhook-system
      name: webhook-server
      path: "/mutate"
    caBundle: <base64-encoded-ca-cert>
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE"]
    resources: ["pods"]
    scope: "Namespaced"
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5

Common Use Cases:

  1. Sidecar Container Injection:

    • Automatic proxy sidecar injection in service meshes like Istio and Linkerd
    • Adding logging and monitoring sidecars
    • Example: Istio's sidecar injector automatically adds Envoy proxy containers when pods are created

  2. Setting Defaults:

    • Automatic resource requests and limits setting
    • Applying security context defaults
    • Automatic addition of labels and annotations
    • Example: Setting default CPU and memory requests for all pods

  3. Applying Image Policies:

    • Modifying image registry URLs
    • Converting image tags to digests
    • Example: Changing nginx:latest to internal-registry.example.com/nginx:1.30.4

  4. Volume Modifications:

    • Adding default volume mounts
    • Automatic ConfigMap or Secret mounting
    • Example: ServiceAccount admission already injects projected tokens unless automountServiceAccountToken is false; do not override that opt-out

  5. Pod Network Configuration:

    • Adding default network settings
    • Modifying DNS configuration
    • Example: Applying specific DNS settings to all pods in a specific namespace

2. Validating Webhook:

Validating webhooks can validate requests coming into the API server and allow or deny them. These webhooks run after mutating webhooks.

Key Characteristics:

  • Cannot modify request objects
  • Can only allow or deny requests
  • Multiple validating webhooks run in parallel
  • Every matching webhook decision must allow; call errors/timeouts follow failurePolicy, so Ignore can allow a request without a response

Configuration Example:

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-policy-validator
webhooks:
- name: pod-policy.example.com
  clientConfig:
    service:
      namespace: webhook-system
      name: webhook-server
      path: "/validate"
    caBundle: <base64-encoded-ca-cert>
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE", "UPDATE"]
    resources: ["pods"]
    scope: "Namespaced"
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5

Common Use Cases:

  1. Enforcing Security Policies:

    • Prohibiting privileged containers
    • Prohibiting root user execution
    • Restricting host network/PID/IPC usage
    • Example: Denying pods running in privileged mode

  2. Validating Resource Constraints:

    • Making resource requests and limits mandatory
    • Applying resource caps
    • Enforcing QoS classes
    • Example: Denying pods without memory limits

  3. Validating Image Policies:

    • Allowing only approved registries
    • Prohibiting latest tag usage
    • Verifying vulnerability scan results
    • Example: Allowing only images from official registries

  4. Validating Labels and Annotations:

    • Checking required labels
    • Validating label formats
    • Example: Making app and environment labels mandatory for all pods

  5. Namespace-Based Policies:

    • Per-namespace resource limits
    • Per-namespace feature restrictions
    • Example: Applying stricter policies in production namespaces

3. Webhook Implementation Methods:

Admission webhooks are implemented as services that provide HTTPS endpoints. These services receive AdmissionReview requests, process them, and return AdmissionResponse.

Basic Webhook Server Implementation Example (Go):

go
package main

import (
    "encoding/json"
    "log"
    "net/http"
    "os"
    "time"
    admissionv1 "k8s.io/api/admission/v1"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/api/resource"
)

func readPodReview(w http.ResponseWriter, r *http.Request) (*admissionv1.AdmissionRequest, *corev1.Pod, bool) {
    if r.Method != http.MethodPost || r.Body == nil {
        http.Error(w, "POST body required", http.StatusBadRequest)
        return nil, nil, false
    }
    var review admissionv1.AdmissionReview
    if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 2<<20)).Decode(&review); err != nil {
        http.Error(w, "Invalid AdmissionReview JSON", http.StatusBadRequest)
        return nil, nil, false
    }
    req := review.Request
    if review.APIVersion != "admission.k8s.io/v1" || review.Kind != "AdmissionReview" || req == nil || req.UID == "" {
        http.Error(w, "AdmissionReview v1 request and UID required", http.StatusBadRequest)
        return nil, nil, false
    }
    if req.Kind.Group != "" || req.Kind.Version != "v1" || req.Kind.Kind != "Pod" ||
        (req.Operation != admissionv1.Create && req.Operation != admissionv1.Update) {
        http.Error(w, "Only Pod CREATE/UPDATE is supported", http.StatusBadRequest)
        return nil, nil, false
    }
    var pod corev1.Pod
    if err := json.Unmarshal(req.Object.Raw, &pod); err != nil {
        http.Error(w, "Invalid Pod JSON", http.StatusBadRequest)
        return nil, nil, false
    }
    return req, &pod, true
}

func writeReview(w http.ResponseWriter, response admissionv1.AdmissionResponse) {
    review := admissionv1.AdmissionReview{
        TypeMeta: metav1.TypeMeta{APIVersion: "admission.k8s.io/v1", Kind: "AdmissionReview"},
        Response: &response,
    }
    data, err := json.Marshal(review)
    if err != nil {
        http.Error(w, "Response encoding failed", http.StatusInternalServerError)
        return
    }
    w.Header().Set("Content-Type", "application/json")
    _, _ = w.Write(data)
}

func writePatch(w http.ResponseWriter, req *admissionv1.AdmissionRequest, patches []map[string]interface{}) {
    response := admissionv1.AdmissionResponse{UID: req.UID, Allowed: true}
    if len(patches) > 0 {
        data, err := json.Marshal(patches)
        if err != nil {
            http.Error(w, "Patch encoding failed", http.StatusInternalServerError)
            return
        }
        patchType := admissionv1.PatchTypeJSONPatch
        response.PatchType, response.Patch = &patchType, data
    }
    writeReview(w, response)
}

func deny(w http.ResponseWriter, req *admissionv1.AdmissionRequest, message string) {
    writeReview(w, admissionv1.AdmissionResponse{
        UID: req.UID, Allowed: false,
        Result: &metav1.Status{Status: "Failure", Reason: metav1.StatusReasonForbidden, Code: 403, Message: message},
    })
}
var sidecarImage string

func mutateHandler(w http.ResponseWriter, r *http.Request) {
    req, pod, ok := readPodReview(w, r)
    if !ok { return }
    // This example targets explicitly declared Linux Pods on CREATE only.
    if req.Operation != admissionv1.Create || pod.Spec.OS == nil || pod.Spec.OS.Name != corev1.Linux {
        writePatch(w, req, nil)
        return
    }
    if sidecarImage == "" {
        deny(w, req, "Configure a tested SIDE_CAR_IMAGE first")
        return
    }
    changed, foundVolume, foundSidecar := false, false, false
    for _, volume := range pod.Spec.Volumes {
        if volume.Name == "shared-data" {
            if volume.EmptyDir == nil {
                deny(w, req, "shared-data is reserved for an emptyDir volume")
                return
            }
            foundVolume = true
        }
    }
    if !foundVolume {
        pod.Spec.Volumes = append(pod.Spec.Volumes, corev1.Volume{
            Name: "shared-data", VolumeSource: corev1.VolumeSource{EmptyDir: &corev1.EmptyDirVolumeSource{}},
        })
        changed = true
    }
    for i := range pod.Spec.Containers {
        c := &pod.Spec.Containers[i]
        if c.Name == "monitoring-sidecar" {
            if c.Image != sidecarImage {
                deny(w, req, "monitoring-sidecar name is reserved")
                return
            }
            foundSidecar = true
        }
        mounted := false
        for _, mount := range c.VolumeMounts {
            if mount.MountPath == "/var/monitoring" {
                if mount.Name != "shared-data" || mount.SubPath != "" || mount.SubPathExpr != "" {
                    deny(w, req, "/var/monitoring is reserved for shared-data")
                    return
                }
                mounted = true
            }
        }
        if !mounted {
            c.VolumeMounts = append(c.VolumeMounts, corev1.VolumeMount{Name: "shared-data", MountPath: "/var/monitoring"})
            changed = true
        }
    }
    if !foundSidecar {
        pod.Spec.Containers = append(pod.Spec.Containers, corev1.Container{
            Name: "monitoring-sidecar", Image: sidecarImage,
            VolumeMounts: []corev1.VolumeMount{{Name: "shared-data", MountPath: "/var/monitoring", ReadOnly: true}},
            Resources: corev1.ResourceRequirements{
                Requests: corev1.ResourceList{corev1.ResourceCPU: resource.MustParse("50m"), corev1.ResourceMemory: resource.MustParse("50Mi")},
                Limits: corev1.ResourceList{corev1.ResourceCPU: resource.MustParse("100m"), corev1.ResourceMemory: resource.MustParse("100Mi")},
            },
        })
        changed = true
    }
    if !changed { writePatch(w, req, nil); return }
    // Whole-array add also works when volumes was absent; preserve all existing entries.
    writePatch(w, req, []map[string]interface{}{
        {"op": "add", "path": "/spec/volumes", "value": pod.Spec.Volumes},
        {"op": "add", "path": "/spec/containers", "value": pod.Spec.Containers},
    })
}

func validateHandler(w http.ResponseWriter, r *http.Request) {
    req, pod, ok := readPodReview(w, r)
    if !ok { return }
    contexts := []*corev1.SecurityContext{}
    for _, c := range pod.Spec.Containers { contexts = append(contexts, c.SecurityContext) }
    for _, c := range pod.Spec.InitContainers { contexts = append(contexts, c.SecurityContext) }
    for _, c := range pod.Spec.EphemeralContainers { contexts = append(contexts, c.SecurityContext) }
    for _, sc := range contexts {
        if sc != nil && sc.Privileged != nil && *sc.Privileged {
            deny(w, req, "Privileged containers are not allowed")
            return
        }
    }
    writeReview(w, admissionv1.AdmissionResponse{UID: req.UID, Allowed: true})
}

func main() {
    sidecarImage = os.Getenv("SIDE_CAR_IMAGE")
    if sidecarImage == "" { log.Fatal("Set SIDE_CAR_IMAGE to a tested Linux image tag/digest") }
    mux := http.NewServeMux()
    mux.HandleFunc("/mutate", mutateHandler)
    mux.HandleFunc("/validate", validateHandler)
    server := &http.Server{Addr: ":8443", Handler: mux, ReadHeaderTimeout: 5*time.Second}
    log.Fatal(server.ListenAndServeTLS("/etc/webhook/certs/tls.crt", "/etc/webhook/certs/tls.key"))
}

4. Webhook Deployment and Configuration:

Build the complete server above, replace the server/sidecar image placeholders, and create webhook-system plus webhook-server-tls. The serving certificate must cover webhook-server.webhook-system.svc (and any alternate Service name used), and caBundle must contain its signing CA. The sidecar image must support the application’s filesystem permissions. This illustrates a regular sidecar; configure shutdown/native-sidecar behavior separately for Jobs.

yaml
# Webhook server deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webhook-server
  namespace: webhook-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webhook-server
  template:
    metadata:
      labels:
        app: webhook-server
    spec:
      containers:
      - name: server
        image: example/webhook-server:REPLACE_WITH_TESTED_RELEASE
        env:
        - name: SIDE_CAR_IMAGE
          value: example/monitoring-agent:REPLACE_WITH_TESTED_RELEASE
        ports:
        - containerPort: 8443
        volumeMounts:
        - name: tls
          mountPath: "/etc/webhook/certs"
          readOnly: true
      volumes:
      - name: tls
        secret:
          secretName: webhook-server-tls

---
# Webhook server service
apiVersion: v1
kind: Service
metadata:
  name: webhook-server
  namespace: webhook-system
spec:
  selector:
    app: webhook-server
  ports:
  - port: 443
    targetPort: 8443

5. Best Practices:

  • Performance Optimization: Webhooks are in the API server request path, so they must respond quickly.
  • Error Handling: Consider behavior when webhook server fails and set failurePolicy appropriately.
  • Scope Limitation: Apply webhooks only to necessary resources and operations.
  • Testing: Thoroughly test webhook behavior in various scenarios.
  • Monitoring: Monitor webhook server performance and errors.
  • Version Management: Advertise only AdmissionReview versions actually implemented and return the same version as the request.

Hands-on Questions ​

  1. Write a CustomResourceDefinition (CRD) that meets the following requirements:
    • API Group: webapp.example.com
    • Version: v1
    • Kind: WebApp
    • Scope: Namespaced
    • Required fields: spec.image, spec.replicas
    • Validation rules: replicas must be an integer between 1 and 10
    • Enable status subresource
Show Answer

Answer:

yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: webapps.webapp.example.com
spec:
  group: webapp.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          required: ["spec"]
          properties:
            spec:
              type: object
              required: ["image", "replicas"]
              properties:
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
                port:
                  type: integer
                  default: 80
                env:
                  type: array
                  items:
                    type: object
                    required: ["name"]
                    properties:
                      name:
                        type: string
                      value:
                        type: string
            status:
              type: object
              properties:
                availableReplicas:
                  type: integer
                phase:
                  type: string
      subresources:
        status: {}
      additionalPrinterColumns:
      - name: Replicas
        type: integer
        jsonPath: .spec.replicas
      - name: Image
        type: string
        jsonPath: .spec.image
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
  scope: Namespaced
  names:
    plural: webapps
    singular: webapp
    kind: WebApp
    shortNames:
    - wa

This CRD has the following characteristics:

  1. Basic Information:

    • API Group: webapp.example.com
    • Version: v1
    • Kind: WebApp
    • Scope: Namespaced

  2. Schema Validation:

    • The spec field is required.
    • spec.image and spec.replicas are required fields.
    • spec.replicas must be an integer between 1 and 10.
    • spec.port is an optional field with a default value of 80.
    • spec.env is an optional field defining an array of environment variables.

  3. Status Subresource:

    • The status subresource is enabled so controllers can update status.
    • status.availableReplicas and status.phase fields are defined.

  4. Additional Printer Columns:

    • When running kubectl get webapps, Replicas, Image, and Age columns are displayed.

  5. Name Configuration:

    • Plural: webapps
    • Singular: webapp
    • Kind: WebApp
    • Short name: wa

You can create custom resources like this using this CRD:

yaml
apiVersion: webapp.example.com/v1
kind: WebApp
metadata:
  name: my-webapp
  namespace: default
spec:
  image: nginx:1.30.4
  replicas: 3
  port: 80
  env:
    - name: ENV_VAR1
      value: "value1"
    - name: ENV_VAR2
      value: "value2"

Controllers can watch this resource and update status:

yaml
status:
  availableReplicas: 3
  phase: Running
  1. Write a Mutating Admission Webhook configuration that meets the following requirements:
    • Add a sidecar to CREATE requests for Pods explicitly declaring spec.os.name: linux
    • Apply only to a specific namespace (monitoring)
    • Webhook service: webhook-service.webhook-system.svc
    • Path: /mutate
Show Answer

Answer:

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
- name: sidecar-injector.example.com
  clientConfig:
    service:
      namespace: webhook-system
      name: webhook-service
      path: "/mutate"
    caBundle: ${CA_BUNDLE}  # Replace with base64-encoded CA certificate in actual environment
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE"]
    resources: ["pods"]
    scope: "Namespaced"
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: monitoring
  matchConditions:
  - name: explicit-linux-pod
    expression: "has(object.spec.os) && object.spec.os.name == 'linux'"
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5
  failurePolicy: Fail
---
# Add label to namespace
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  labels:
    kubernetes.io/metadata.name: monitoring

This configuration has the following characteristics:

  1. Webhook Configuration:

    • Name: sidecar-injector
    • Webhook service: webhook-service.webhook-system.svc
    • Path: /mutate

  2. Scope:

    • API Group: "" (core API group)
    • API Version: v1
    • Operation: CREATE (apply only on pod creation)
    • Resource: pods
    • Scope: Namespaced

  3. Namespace Selector:

    • Apply only to namespaces with kubernetes.io/metadata.name: monitoring label
    • The API server supplies this immutable name label on the monitoring namespace

  4. Additional Settings:

    • admissionReviewVersions: Supported AdmissionReview API versions
    • sideEffects: No side effects from webhook
    • timeoutSeconds: Webhook response wait time
    • failurePolicy: Deny requests on webhook failure

The webhook server should implement logic like this:

Use mutateHandler and the common request/response helpers from the complete server above. It creates a missing volumes array, preserves existing entries, prevents duplicate injection, and mounts shared-data in both application and sidecar containers. Reserved name/path conflicts are denied.

This webhook automatically adds a monitoring sidecar container to Pods declaring spec.os.name: linux created in the monitoring namespace. The sidecar container uses a monitoring agent image and mounts a shared volume to share data with the main container.

Controller snippets require the generated project types, client and RBAC. They check Deployment ownership, reconcile image/replicas/environment configuration and use an Owns watch for status changes. Clean up external resources with finalizers before deletion. No Kubernetes/AWS deployment was executed during this review.

Verification References ​