How to Use Mixed Version Proxy for Safer Kubernetes Upgrades

By ✦ min read

Introduction

Upgrading a Kubernetes cluster with multiple control plane nodes can be risky. When API servers run different versions, some may not recognize newer resources, causing requests to fail with a 404 Not Found error—even though those resources exist on other nodes. This can lead to unintended garbage collection or blocking namespace deletions. The Mixed Version Proxy (MVP), now beta in Kubernetes 1.36, solves this by routing requests to the correct API server. This guide walks you through leveraging MVP to make cluster upgrades safer and more reliable.

What You Need

• A Kubernetes cluster with multiple control plane nodes (e.g., a high-availability setup).
• At least two API servers running different versions (e.g., during an upgrade).
• kubectl installed and configured to access the cluster.
• Kubernetes 1.36 or later (MVP is enabled by default).
• Basic understanding of API server discovery and resource versioning.

Step-by-Step Guide

Step 1: Verify That Mixed Version Proxy Is Active

In Kubernetes 1.36, MVP is enabled by default via the UnknownVersionInteroperabilityProxy feature gate. To confirm it’s running, check the API server logs or use the following command on any control plane node:

kubectl get --raw /readyz?verbose

Look for a line indicating the proxy is active. You can also inspect the kube-apiserver pod logs for messages like "MixedVersionProxy: proxying request to peer". If MVP is not found, ensure your cluster is on version 1.36 or later.

Step 2: Understand How MVP Routes Requests

MVP works by using Aggregated Discovery (replacing the older StorageVersion API). Each API server maintains a cache of what resources its peers serve. When a request arrives for a resource the local server doesn’t recognize, it:

1. Checks its discovery cache for a peer that can serve the resource.
2. Proxies the request to that peer, adding an x-kubernetes-peer-proxied header.
3. Returns the response to the client as if it came from the original server.

This works for resource requests (e.g., GET /apis/batch/v1/jobs) but not for discovery requests (that limitation is addressed in future releases).

Step 3: Simulate an Upgrade Scenario

To test MVP, perform a rolling upgrade of your control plane nodes:

• Upgrade one API server to a newer version that introduces a new API version (e.g., example.com/v2).
• Leave at least one other API server on the old version.
• Create a resource using the new API version and attempt to read or list it via the old API server.

Without MVP, you would see a 404. With MVP, the request should succeed. Use kubectl —server=https://old-server:6443 get ... to force a request to the old server and verify.

Step 4: Monitor Proxy Activity and Logs

Enable verbose logging on the API server to track proxied requests. Add the flag —v=4 or higher to the kube-apiserver startup parameters. Look for log entries containing "proxying" or "peer-proxied". For example:

I0415 12:34:56.789012   12345 proxy.go:200] Proxying request for resource v2 to peer at 10.0.0.2:6443

You can also check metrics (if enabled) for proxy counts using the kubernetes_build_info metric.

Step 5: Troubleshoot Common Issues

If MVP is not working as expected, check these areas:

• Discovery cache: The aggregated discovery between peers may be stale. Force a refresh by restarting the API server or waiting for the default 60-second cache TTL.
• Network connectivity: Ensure all API servers can reach each other on the secure port (default 6443). Firewalls must allow inter-node traffic.
• Feature gate: On clusters upgraded from earlier versions, the feature gate might be disabled. Verify with kubectl get nodes -o yaml | grep featureGates (though this is not directly visible; check API server pod manifest).
• CRDs and aggregated APIs: MVP now supports these via aggregated discovery, but ensure your CRD or aggregated API server registers correctly with the main API server. Use kubectl get apiservices to confirm.

Step 6: Plan for Future Upgrades

With MVP enabled by default in 1.36, you can safely perform rolling upgrades without worrying about broken resource requests. However, be aware of the following:

• Discovery requests (e.g., kubectl api-resources) are still served by the local API server. In 1.36, a peer-aggregated discovery feature is being added to close this gap. Check the feature gate list for updates.
• If you rely on the old StorageVersion API (alpha), note that MVP no longer uses it. The new aggregated discovery approach is more reliable and works with CRDs.
• Always test upgrades in a non-production environment first.

Tips and Best Practices

Here are final pointers to get the most out of Mixed Version Proxy:

• Test with real workloads: Before upgrading production, simulate an upgrade on a staging cluster with similar API versions to ensure MVP handles proxying correctly.
• Monitor API server metrics: Use Prometheus to track request errors and proxy counts. A sudden spike in proxied requests may indicate an imbalance in version distribution.
• Keep nodes evenly staggered: During upgrades, avoid having too many old API servers left. MVP works best when capable peers are available to handle proxied requests.
• Stay updated: The feature may evolve in future releases. Review release notes for changes to MVP or related discovery components.
• Back up etcd: Always backup etcd before upgrading. While MVP prevents resource request failures, it doesn’t protect against etcd corruption.

By following these steps, you can leverage Kubernetes’ Mixed Version Proxy to make cluster upgrades smoother and avoid the dreaded 404 errors that can disrupt operations.