Mitigating IP Blocks When Accessing registry.k8s.io From Hetzner Cloud

In this short post, I will show you an easy to implement mitigation for blocked Hetzner Cloud IPs when pulling from the Kubernetes registry registry.k8s.io.

Problem statement

When running Kubernetes clusters on Hetzner Cloud you might have noticed this nasty error when a new Node is coming up:

failed to resolve reference "registry.k8s.io/pause:3.9": unexpected status from HEAD request to https://registry.k8s.io/v2/pause/manifests/3.9: 403 Forbidden

See the related Issue kubernetes/registry.k8s.io #138 for a discussion about this problem.

The root cause, according to the Hetzner Support: some GeoIP databases wrongly locate some of their IPs in Iran.

So there isn't much we can do about it—or can we? As this issue is causing constant problems when upgrading / scaling Kubernetes clusters (and as I have noticed, it happens mostly in the Hetzner Cloud Falkenstein region fsn1), we need some mitigiation for this.

And it's not only the pause image (which for example containerd requires), there are other essential container images coming from registry.k8s.io to name only a few:

• coredns/coredns
• external-dns/external-dns
• kube-state-metrics/kube-state-metrics
• metrics-server/metrics-server
• sig-storage/livenessprobe
• sig-storage/csi-resizer

Possible solutions

How can we tackle this issue? We need a way to circumvent pulling the image directly from upstream every time a new Node / Pods comes up.

In this blog post I will go with a mix of the second and third option, so feel free to jump to the Mitigation section.

1. Your own registry

One obvious one: using your own registry as a pull-through cache for registry.k8s.io.

If you already have a registry (like Harbor or CNCF distribution) then great, you should very likely use it to help circumventing this issue.

You then just need to tell your CRI (like containerd) to use your registry as a mirror and you should be good to go.

2. Preloading images

Another way to tackle this would be to preload all images when building the HCloud snapshot which is used to start your instances from. When using image-builder you can configure images which should be loaded at build time (see the "Customization" section in the image-builder docs).

So for preloading the pause image you would create a custom JSON file for image-builder - I've highlighted the important lines below:

{
  "additional_registry_images": "true",
  "additional_registry_images_list": "registry.k8s.io/pause:3.9",
  "containerd_version": "1.7.27",
  "crictl_version": "1.32.0",
  "kubernetes_cni_semver": "v1.6.2",
  "kubernetes_deb_version": "1.32.4-1.1",
  "kubernetes_semver": "v1.32.4",
  "kubernetes_series": "v1.32",
  "load_additional_components": "true"
}

This works perfectly fine, but whats with all the other images coming from registry.k8s.io? You would need to update your node image every time you update or add a new component which retrieves container images from this registry.

Another downside: the image gets bigger and bigger for every new image you add to preload (which increases node start up time and requires more storage).

3. Spegel

"Spegel" is the swedish word for "mirror" (or in german: "Spiegel") and was initially developed by Xenit AB. So what is spegel? It describes itself as a "stateless cluster local OCI registry mirror" and yes, this is what it does: no more, no less.

You can configure which image registries to mirror (or if not configured, mirror everything used in your cluster) and then runs as DaemonSet. But the cool thing is: it does not need any persistent storage at all.

Mitigation

For my setup, I went with a combination of 2 and 3 because it's easy implement and does not require a separate registry deployment:

1. Preload the pause image into your node image
2. Use spegel to mirror all registry.k8s.io images running in my cluster

We need the pause image to be preloaded or otherwise containerd will not get up and running. For all other images, we then use spegel to make sure they are available anytime.

Before deploying spegel in your cluster, you need to update your containerd config, which can also be done very easily with image-builder. Spegel has two requirements for containerd:

• config_path: already set when using image-bulder
• discard_unpacked_layers: needs to be added (configures garbage collection to not remove layers after successfully unpacking those)

  [plugins."io.containerd.grpc.v1.cri".containerd]
     discard_unpacked_layers = false

So we are adding the line above as a base64 encoded string to our image-builder JSON:

{
  "additional_registry_images": "true",
  "additional_registry_images_list": "registry.k8s.io/pause:3.9",
  "containerd_additional_settings": "ICBbcGx1Z2lucy4iaW8uY29udGFpbmVyZC5ncnBjLnYxLmNyaSIuY29udGFpbmVyZF0KICAgICBkaXNjYXJkX3VucGFja2VkX2xheWVycyA9IGZhbHNlCg==",
  "containerd_version": "1.7.27",
  "crictl_version": "1.32.0",
  "kubernetes_cni_semver": "v1.6.2",
  "kubernetes_deb_version": "1.32.4-1.1",
  "kubernetes_semver": "v1.32.4",
  "kubernetes_series": "v1.32",
  "load_additional_components": "true"
}

And then build a new image with it:

PACKER_VAR_FILES=hetzer-flatcar-amd64-1.32.4.json make build-hcloud-flatcar

Then deploy spegel with Flux like this:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: spegel
spec:
  type: "oci"
  interval: 5m0s
  url: oci://ghcr.io/spegel-org/helm-charts

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: spegel
spec:
  interval: 5m
  chart:
    spec:
      chart: spegel
      sourceRef:
        kind: HelmRepository
        name: spegel
  driftDetection:
    mode: enabled
  values:
    spegel:
      mirroredRegistries:
        - "https://registry.k8s.io"

And that's it 👏 The pause image is now baked in the HCloud snapshot we use for our Kubernetes Nodes and spegel will make sure the images from registry.k8s.io which are used in our cluster are always available.