A Practical Guide Implementing Custom Kubernetes Operator

Learn how Kubernetes custom operators work, from API server watch events and informer caches to work queues, reconciliation, and update handling.

Kubernetes makes more sense when you start with its basic contract: you declare the state you want, and the system keeps trying to make the real world match it. A Deployment says how many Pods should exist. A Service says how traffic should reach them. A ConfigMap says what configuration data should be available. You are not usually issuing imperative commands like “create exactly this Pod right now and never look again.” You are storing desired state in the Kubernetes API and letting control loops converge toward it.

The API server sits at the center of that contract. It exposes the REST API, accepts reads and writes, runs API-level machinery such as authentication, authorization, admission, defaulting, conversion, validation, optimistic concurrency checks, subresource handling, and built-in resource strategy behavior, and then persists accepted state to etcd. For built-in resources, Kubernetes also ships controllers that watch this stored state and act on it. The Deployment controller watches Deployments and ReplicaSets. The ReplicaSet controller watches ReplicaSets and Pods. The node controller, endpoint controllers, job controller, and others all follow the same broad pattern: observe desired and actual state, then make safe progress toward convergence.

A custom operator extends that same model beyond Kubernetes’ built-in objects. A CustomResourceDefinition defines a new API type, such as an EC2Instance. Once the CRD exists, the API server can store, validate, serve, and watch EC2Instance objects, but it still does not know how to call AWS, wait for a VM to boot, tag it, clean it up, or record its external ID. That business logic lives in a controller that you run. The CRD is the shared data contract; the controller is the convergence loop attached to that contract.

That is the central idea of an operator: it is not an event handler that blindly reacts to commands. It is a level-driven convergence loop built around API-server state, watch delivery, cache maintenance, queue deduplication, and idempotent reconciliation. The rest of the machinery exists to make that loop reliable and scalable.

The diagram is easiest to read from left to right. A client writes desired state to the API server. The API server persists the accepted object and broadcasts resource-versioned changes to watchers. Controller-side informer machinery receives those changes, maintains a local cache, and invokes handlers that enqueue object keys. Workers pop those keys and run reconciliation. The reconciler reads the latest visible state, compares it with the external world, and writes back any required changes, usually to status or to related Kubernetes objects.

The API Server Stores State, Not Your Workflow

When a client creates or updates an EC2Instance, that successful write goes through the API server first. Kubernetes clients do not write to etcd directly. The API server accepts the request only after its normal API path has completed, including validation and resource-version checks where they apply. Then it persists the object to etcd and makes the new version visible through the Kubernetes API.

After persistence, the API server reports the change to authorized watchers over long-lived HTTP watch connections. Watches are ordered by resourceVersion for a given resource stream, and in many clusters they are served through the API server’s watch cache rather than by having each watcher read directly from etcd. A watch event is not a field-level diff. It is a wrapper with a type such as ADDED, MODIFIED, DELETED, or ERROR, plus the object as it exists at that moment through the Kubernetes runtime object machinery.

This matters because the API server is deliberately not trying to understand each controller’s private intent. If you change one field on an EC2Instance, the API server does not decide whether that field is meaningful for provisioning, billing, policy, or audit. It stores the accepted state transition and reports it to watchers. That decoupling is what lets multiple controllers observe the same CRD without forcing all workflow semantics into the API server.

For example, one controller may watch EC2Instance objects and create the actual VM from spec. Another controller may watch the same objects for governance and react after status.instanceID appears by applying cost-center tags or security controls. A third controller might update inventory or compliance records. They share the object, but each attaches different behavior around it.

Watches Feed the Cache Before They Feed Reconciliation

A controller does not usually call the API server for every read inside every reconciliation. That would be slow and would put unnecessary read pressure on the control plane. Instead, controller libraries build a watch-backed local view of the objects the controller cares about.

At the lower level, a reflector maintains the list/watch connection to the API server. As list and watch results arrive, client-go machinery converts them into deltas and places them into a DeltaFIFO. The DeltaFIFO is not an audit log that preserves every intermediate version forever. Multiple changes to the same key may be accumulated or coalesced, so a controller should not depend on it as a complete change history. Its role is to preserve enough ordered change information for informer processing.

The informer drains the DeltaFIFO, updates a thread-safe local cache called the indexer, and then invokes registered event handlers. The indexer is not an exact replica of etcd. It is a scoped, eventually consistent in-memory cache for the resource types and namespaces the informer is configured to watch. That cache is still extremely important: it gives the controller a recent local view of Kubernetes state and avoids turning every reconciliation into a fresh API-server read.

Only after the cache is updated do handlers decide what to enqueue. This ordering is subtle but important. The worker that later processes the item should be able to fetch the latest visible object from the cache instead of being handed an object payload that may already be stale.

The Queue Stores Identity, Not Objects

Event handlers normally do not run business logic. They translate informer events into work-queue entries, usually by extracting the object’s identity. For namespaced objects, that identity is namespace/name. For cluster-scoped objects, it is just the name carried in a NamespacedName with an empty namespace. In controller-runtime’s usual Reconcile(ctx, req) API, the reconciler receives this request key, not the raw watch event, and not the full object that arrived on the watch stream.

That key-only design is one of the main reasons Kubernetes controllers are level-driven. If the queue stored whole objects, a worker could pop an object that was already out of date by the time reconciliation began. By storing only the key, the worker is forced to read the latest visible state from the informer-backed cache when it actually runs. The event wakes the controller up, but the current state drives the decision.

The rate-limited work queue also deduplicates keys. If five updates to the same EC2Instance arrive while the key is waiting in the queue, the queue does not need five independent copies of the same key. If the key is re-added while a worker is already processing it, the standard client-go queue tracks that too. Another worker from the same controller queue should not concurrently process the same key; the key is typically marked dirty and processed again after the current attempt finishes.

This should not be stretched into a global concurrency guarantee. Workers are not spawned per object. A controller owns a pool of workers, and MaxConcurrentReconciles limits concurrent reconciles for that controller. The standard queue prevents the same key from being handed to two workers within that one controller queue at the same time, but different controllers, different controller-manager processes, or misconfigured leader election can still act on the same object concurrently. Your reconciler must still be idempotent and conflict-aware.

Reconciliation Reads the Current Level

Once a worker pops a key, the reconciler starts from a simple pattern: fetch the object, handle not-found cases, inspect desired state, inspect actual state, and make one safe move toward convergence. It should not assume that it is handling a specific command like “the user changed exactly this field.” In the normal controller-runtime flow, event type can influence enqueueing through handlers or predicates before reconciliation, but the reconcile function usually receives only the key.

func (r *EC2InstanceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var instance ec2v1.EC2Instance
    if err := r.Get(ctx, req.NamespacedName, &instance); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    return ctrl.Result{}, nil
}

For an EC2Instance, the real reconciler would read the current custom resource, check whether the external cloud instance exists, create it if missing, update status with fields such as the provider instance ID or IP address, and requeue if the external operation is still progressing. The same loop handles creation, updates, retries, process restarts, and delayed external completion because it always recomputes from current state.

The return value tells the queue what should happen next. A clean ctrl.Result{}, nil means the key is done for now. Returning an error requeues with exponential backoff, which is appropriate for unexpected failures such as network timeouts or permission errors. ctrl.Result{RequeueAfter: duration}, nil is useful when the controller successfully started or observed an asynchronous external operation and wants to poll later. ctrl.Result{Requeue: true}, nil requests another pass through controller-runtime’s rate-limited behavior rather than guaranteeing an immediate retry, and newer guidance generally prefers returning errors for retryable failures or using RequeueAfter for polling.

The important part is not memorizing every return value; it is understanding the shape of the loop. Every pass should be safe to repeat. If an AWS create call already succeeded but the controller crashed before updating status, the next pass should discover the existing instance instead of creating a duplicate. If status is already correct, the reconciler should avoid writing it again. Idempotence is the controller’s survival trait.

Create, Delete, and Update Events Wake the Same Loop Differently

Creation is the easiest case to reason about. A new object is accepted, persisted, and watchers observe an ADDED event with the initial state. The controller enqueues the key, reads the object, and begins moving the external world toward the declared spec.

Deletion is less absolute than it first looks because finalizers can split deletion into phases. If no finalizers block deletion, the object can be removed and watchers eventually observe a DELETED event. If finalizers are present, the delete request first updates the object by setting metadata.deletionTimestamp. That is a modification, not final removal. The object remains in storage while controllers perform cleanup and remove their finalizers. Only after finalizers are gone does the object actually disappear and the final DELETED event appear. Lower-level informer handlers may also receive tombstones when they learn about a deletion for an object that was not present in the local cache, so delete handling should tolerate that case.

Updates are where the watch model becomes most visible. Every accepted mutation creates a new stored version and a watch notification. That includes writes performed by your own controller. If a reconciler updates status, the API server persists that status update and broadcasts it like any other change. Status updates normally do not increment metadata.generation, so a GenerationChangedPredicate can be useful when a controller wants to ignore status-only changes and react mainly to spec changes. But predicates are optional filters, not a mandatory part of update handling. A controller can reconcile every update, drop some events through predicates, or use custom handlers.

This is the transition point where write behavior starts to matter. Since every accepted write becomes watch traffic and cache churn, a reconciler that writes repeatedly inside one pass is feeding its own event stream.

Repeated Writes Create Churn

Consider a reconciler that performs two separate status updates for one logical transition:

func (r *EC2InstanceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var instance ec2v1.EC2Instance
    if err := r.Get(ctx, req.NamespacedName, &instance); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    instance.Status.Phase = "Provisioning"
    if err := r.Status().Update(ctx, &instance); err != nil {
        return ctrl.Result{}, err
    }

    instance.Status.IPAddress = "10.0.0.1"
    if err := r.Status().Update(ctx, &instance); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

The problem is not that the queue will literally store two permanent duplicate objects. The queue deduplicates identical keys while they are queued. The real cost is that each Status().Update() sends an HTTP request to the API server, passes through validation and admission behavior, persists a new object version if accepted, emits a watch event, updates informer state, invokes handlers, and may cause another reconciliation after the current one finishes. Two writes for one logical state transition means two resource-version boundaries and two rounds of control-plane churn.

The better pattern is to compute the complete mutation in memory and then perform one write:

func (r *EC2InstanceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var instance ec2v1.EC2Instance
    if err := r.Get(ctx, req.NamespacedName, &instance); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    patch := client.MergeFrom(instance.DeepCopy())
    instance.Status.Phase = "Provisioning"
    instance.Status.IPAddress = "10.0.0.1"
    if err := r.Status().Patch(ctx, &instance, patch); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

Here, “one write” is not magic. It is atomic only in the limited Kubernetes sense: one accepted write to one object at one resourceVersion boundary. It is not a multi-object transaction, and it can still fail with a conflict if the object version is stale. But it matches the watch/reconcile model better. Build the desired object change locally, write once, and let the next watch event represent one coherent state transition.

Choosing Update, MergeFrom, and StrategicMergeFrom

Once you care about minimizing unnecessary writes, the difference between update and patch operations becomes practical rather than academic. Both update and patch go through the API server, both are subject to validation and admission, and both can encounter conflict behavior. They differ in what you send and how much of the object you risk overwriting.

Update is a full-object replacement of the stored resource version. You usually fetch the object, mutate fields, and send the entire object back. That is fine when you truly own the whole object shape you are writing, but it requires care when users or other controllers may also modify fields. You must preserve everything you do not intend to change, and you generally need the current resourceVersion.

Patch lets you send a smaller change. In controller-runtime, MergeFrom creates a JSON merge patch by comparing a base object with a mutated object. The base object is the “before” snapshot; the mutated object is the “after” state. The base must remain unchanged, because it is the reference used to compute the diff.

func updateReplicas(ctx context.Context, c ctrlclient.Client, deploy *appsv1.Deployment) error {
    base := deploy.DeepCopy()

    replicas := int32(3)
    deploy.Spec.Replicas = &replicas

    return c.Patch(ctx, deploy, ctrlclient.MergeFrom(base))
}

MergeFrom is often a good fit when you want to update a small part of an object and can accept JSON merge patch behavior. One important consequence is that lists are generally replaced rather than merged item by item. If you are patching list fields and expect identity-aware merging, plain merge patch may not be the tool you want.

StrategicMergeFrom uses Kubernetes strategic merge patch semantics. On built-in Kubernetes types such as Deployment, strategic merge can use schema metadata to merge list entries by identity. Containers, for example, can be matched by name, so changing one container does not have to replace the whole containers list.

func updateSidecarImage(ctx context.Context, c ctrlclient.Client, deploy *appsv1.Deployment) error {
    base := deploy.DeepCopy()

    for i := range deploy.Spec.Template.Spec.Containers {
        if deploy.Spec.Template.Spec.Containers[i].Name == "sidecar" {
            deploy.Spec.Template.Spec.Containers[i].Image = "busybox:1.36"
            break
        }
    }

    return c.Patch(ctx, deploy, ctrlclient.StrategicMergeFrom(base))
}

That strategic behavior is mainly a built-in-type story. Strategic merge patch is not generally supported for custom resources. CRD schemas use structural OpenAPI extensions such as x-kubernetes-list-type and x-kubernetes-list-map-keys primarily for server-side apply merge semantics, pruning, and validation behavior, not to make strategic merge patch behave for CRDs the same way it does for built-in resources. For CRDs, prefer JSON merge patch, JSON patch, or server-side apply depending on ownership needs.

A compact rule is enough for most controller code. Use Update when you own the full object state you are writing. Use MergeFrom when you want a partial update and JSON merge patch semantics are acceptable. Use StrategicMergeFrom when you are patching built-in Kubernetes types and specifically want Kubernetes-aware list merging based on schema metadata.

The Operator Mental Model

A custom operator is a feedback loop around a shared API object. The API server owns API semantics, persistence, and watch delivery. Informers maintain a scoped, eventually consistent cache. Handlers and optional predicates decide which events become queue entries. The queue deduplicates keys and manages retry behavior. Workers run reconciliation. The reconciler reads the latest visible state and makes idempotent progress toward the declared desired state.

That model explains the details that otherwise feel odd. The queue stores keys instead of objects because current state matters more than the event payload. The reconciler usually does not care which exact watch event woke it up because it is level-driven, not command-driven. Multiple writes inside one reconcile loop are expensive because every accepted write becomes another stored version, another watch event, another cache update, and possibly another reconciliation. The local cache is useful because it reduces API-server reads, but it is not etcd and not a source of durable truth.

Once you see an operator as a level-driven convergence machine, the pieces line up cleanly: declare intent in the API server, observe state through watches, maintain a local cache, enqueue identity, and reconcile repeatedly until the outside world matches the declaration.