Skip to content

About

Containerd backed registry for Skaffold builds

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Registryless

Ever wanted to push images directly to your Kubernetes cluster?

Registryless is designed for Skaffold workflows. Push to a standard registry endpoint and your images land directly in the nodes' containerd stores. Multicast distributes new content to hosts with the same platform while the push is in progress, reducing the image downloads needed during deployment.

The nodes' runtimes provide the image storage, with no separate registry volume. Receivers verify digests, repair missing data over HTTP, and register and unpack images in containerd. This first iteration is experimental and does not pin images.

First iteration

  • A host-network DaemonSet, with no PVC or separate registry storage.
  • IPv4 NIC selected automatically from the lowest-metric usable default route. Override it on multihomed hosts with multicast.interface.
  • SHA-256 of the normalized OS/architecture/variant selects an address in 239.192.0.0/14. Messages also carry the full platform to reject collisions.
  • OIDC browser login with PKCE and personal Docker credentials; existing namespace push/pull credentials remain available for build robots.
  • Resumable blob uploads; requests landing on another Pod are forwarded to the upload owner. Monolithic blob uploads work too.
  • Manifest commits wait up to nine minutes for layers uploaded through other agents to arrive, allowing Ingress to select any Pod throughout a push.
  • Multicast chunks with bounded reordering and HTTP range repair. HTTP repair uses expiring tickets authenticated by the release distribution key, without mTLS.
  • Expiring containerd leases protect uploads and received content for one hour. Published manifests have GC links and image records; ordinary containerd and kubelet garbage collection handle retention. No pin labels are set.

Managed image records and blobs carry codemowers.io/registryless=true; image records also carry codemowers.io/registryless.repository. The daemon image has the OCI label codemowers.io/registryless.component=daemon.

The first version accepts single-platform runnable OCI/Docker schema 2 images matching the receiving host. Multi-platform indexes, historical image discovery through nodestatd, remote pull redirects, repository listing, and deletion are not implemented yet. By default push success confirms local storage and unpacking; redistribution is asynchronous and best effort. Use a homogeneous node pool for an Ingress push endpoint in this iteration.

Personal Docker credentials

Enable oidc.enabled=true to register an OIDCClient with Passmower. The chart reads its generated oidc-client-<release>-owner-secrets Secret. Visit the registry hostname, sign in, and select Reset Docker password to get a copyable docker login command. Each reset replaces the previous Argon2id password hash; the plaintext password is shown only in that response. Choose an expiry when generating it: 1 hour, 24 hours, 72 hours, 1 week, 2 weeks, 1 month, 3 months, or 6 months. Months use calendar dates. oidc.passwordTTL selects the default (normally 24h). Resetting does not change your OIDC password.

Hashes, signed-session keys, and the distribution key live in <release>-auth in the Helm release namespace. Runtime RBAC grants access to that exact Secret, without permission to list or create Secrets. Helm supplies the namespace; upgrades preserve runtime-owned Secret data.

One personal Docker login works across namespaces. The first repository path component selects an existing target namespace. Registryless checks Kubernetes RBAC on each operation: create deployments.apps permits pushes and get pods permits reads. OIDC username/group mapping must match your API server; the default username is <issuer>#<sub>, with unprefixed groups. Group claims remain those from the login used to reset the credential until its next reset or expiry.

Namespace robot credentials

The first repository path component is its target Kubernetes namespace:

registry.example.com/myproject/api:dev
                                   ^ namespace

Create registryless-push and/or registryless-pull in that existing namespace. Their presence opts the namespace into the corresponding operation. Each Secret has a password key containing a distinct random credential of at least 32 bytes; the Docker username is the namespace name. Secrets are not generated by the chart.

# Each file contains a different random password, without a trailing newline.
kubectl -n myproject create secret generic registryless-push \
  --from-file=password=/secure/path/push-password
kubectl -n myproject create secret generic registryless-pull \
  --from-file=password=/secure/path/pull-password

docker login registry.example.com \
  --username myproject --password-stdin < /secure/path/push-password
skaffold run --default-repo registry.example.com/myproject

Push credentials also allow reads and existence checks needed by the push protocol. Pull credentials cannot upload. All public reads require a credential. Use the pull credential in workload imagePullSecrets when needed. Docker stores one credential per registry hostname in each user's Docker config. For local work across namespaces, use a separate DOCKER_CONFIG directory per project, outside the build context. Kaniko mounts its namespace's config.json Secret directly. Valid robot credentials are cached for 30 seconds, including after rotation or deletion. Registryless has get permission only on those two Secret names, with no Secret list/watch access. Disable these legacy credentials with auth.legacyNamespaceCredentials=false after migrating build clients.

Set auth.mode=none and oidc.enabled=false for anonymous push and pull. This disables public registry authentication and namespace RBAC checks; internal announcements, forwarding and repair still use the release's signed tickets. The chart omits the cross-namespace credential and authorization-review permissions in this mode. The named local Secret remains necessary for internal keys.

Synchronous pushes

Set push.sync=true to delay the final manifest's successful response until all discovered agents on the publisher's full platform report the image registered and, when enabled, unpacked. push.syncTimeout defaults to 5m; discovery or timeout failures return HTTP 503. Leave it false for asynchronous redistribution.

Targets are a snapshot of current agents, excluding terminating and completed Pods. This does not cover future nodes or prevent subsequent garbage collection. Keep ordinary pull policies and pull credentials for now; this switch alone does not make imagePullPolicy: Never reliable.

Install

Build/publish the container using the Dockerfile, then install into an existing namespace that allows host networking, root, and hostPath mounts:

helm upgrade --install registryless ./chart --namespace registry \
  --set image=your-registry/registryless:your-tag \
  --set registryHost=registry.example.com \
  --set ingress.enabled=true

On k3s add --set containerd.socket=/run/k3s/containerd/containerd.sock. Ingress TLS, class, annotations, and upload limits are deployment-specific values. Use HTTPS for the external registry endpoint. skaffold.yaml can build and install the chart when REGISTRYLESS_HOST, the publishing repository, kube context, and existing target namespace are supplied by your environment. Bootstrap Registryless using an already available registry.

The image includes no CA bundle. Platform admission supplies workload trust. The root URL serves a homepage even when OIDC is disabled. HTTP-to-HTTPS redirects belong to the ingress platform. Environment-specific overrides belong in gitignored chart/values.*.yaml files. Skaffold reads chart/values.local.yaml. Copy chart/values.yaml to chart/values.local.yaml, then configure it and skaffold.env. The local, gitignored scripts/promote-images.sh copies the source image's pull credential and deploys the same immutable image with Helm. Set ARTIFACTS_FILE to a fresh Skaffold build artifact file to promote that image instead. It never copies personal credentials or internal distribution/session keys between clusters.

Allow host-to-host UDP 5283 for multicast, TCP 5282 for repair and upload forwarding, and the configured multicast range on the selected NIC. Multicast TTL defaults to 1. HTTP and multicast ports must be unique if running multiple instances per host. The network must deliver multicast; interface discovery does not configure switches, IGMP snooping, or routing.

Internal transfers are plaintext and assume a trusted host network. Digests protect content integrity; authenticated announcements protect publication. Every participating host can access participating namespaces' image contents. The containerd socket grants control over the host runtime.

Configuration

Environment variable Default Purpose
REGISTRY_HOST required Canonical image registry hostname, without scheme
REGISTRY_NAMESPACE registry Auth Secret and agent discovery namespace; Helm supplies its release namespace
AUTH_SECRET_NAME registryless-auth Precreated named Secret for hashes and internal keys
AUTH_MODE rbac rbac requires credentials; none allows anonymous push/pull
PUSH_SYNC false Wait for matching-platform agents before confirming the final manifest
PUSH_SYNC_TIMEOUT 5m Sync deadline, at most 30m
LEGACY_NAMESPACE_AUTH true Accept existing namespace robot credentials
OIDC_ENABLED false Serve browser sign-in and credential reset
OIDC_ISSUER, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET required when enabled Provider settings from the generated OIDC Secret
OIDC_USERNAME_CLAIM sub Kubernetes username claim
OIDC_USERNAME_PREFIX issuer + # for sub Match API server username mapping
OIDC_GROUPS_PREFIX empty Match API server groups mapping
DOCKER_PASSWORD_TTL 24h Generated personal credential lifetime
CONTAINERD_ADDRESS /run/containerd/containerd.sock Host runtime socket
CONTAINERD_NAMESPACE k8s.io Runtime namespace
SNAPSHOTTER containerd default Snapshotter used for unpacking
MULTICAST_INTERFACE auto Host NIC override
ADVERTISE_IP auto IPv4 address on the selected NIC
HTTP_PORT 5282 Registry and authenticated internal HTTP
MULTICAST_PORT 5283 Platform multicast UDP port
MULTICAST_TTL 1 Multicast TTL
MULTICAST_BYTES_PER_SECOND 10Mi Per-publisher payload rate; Kubernetes quantities such as 10Mi, 10M, or raw bytes
MAX_BLOB_BYTES 2147483648 Maximum blob size (up to 16 GiB)
UNPACK_IMAGES true Unpack received/pushed images
METRICS_ADDRESS :5284 /metrics and /healthz listener
KUBECONFIG in-cluster Optional local Kubernetes credentials

Receive metrics

The chart creates a PodMonitor when monitoring.coreos.com/v1/PodMonitor is available. It selects this release's agents and scrapes their named metrics port over internal HTTP. Prometheus must select the PodMonitor and its namespace.

Scrape each Pod at port 5284, /metrics. All receive counters have the project_namespace label; transfer outcomes additionally have outcome.

Metric Meaning
registryless_receive_multicast_bytes_total Unique payload bytes observed, including reordered chunks
registryless_receive_duplicate_bytes_total Duplicate payload bytes
registryless_receive_missing_bytes_total Expected bytes not observed before completion's two-second reorder window (or five-second inactivity timeout)
registryless_receive_expected_bytes_total Expected bytes of transfers whose receive window closed
registryless_receive_http_repair_bytes_total Bytes downloaded to repair transfers and missing manifest dependencies
registryless_receive_dropped_packets_total Locally dropped packets due to bounded queues/buffers
registryless_receive_transfers_total complete, repaired, failed, or already_present

Missing bytes include network/socket loss and receiver queue drops. The duplicate and unique-byte counters observe only packets that reach a transfer worker; reorder-buffer drops are counted separately. A fully missed announcement cannot be attributed to a project and does not appear in these counters. Already-present blobs are skipped and excluded from expected/missing byte totals. Thus this is observed transfer loss, not an exact physical network loss measurement.

Example missing payload ratio per namespace:

sum by (project_namespace) (rate(registryless_receive_missing_bytes_total[5m]))
/
sum by (project_namespace) (rate(registryless_receive_expected_bytes_total[5m]))

Development

go test ./...
go vet ./...
go build ./cmd/registryless
helm lint chart --set registryHost=less.example.test

For integration tests, run a separate containerd with its own root, state, and socket. Point REGISTRYLESS_TEST_CONTAINERD at that socket, then run:

REGISTRYLESS_TEST_CONTAINERD=/path/to/test/containerd.sock \
  go test ./internal/multicast -v -count=1

These tests use real TCP/UDP sockets and create isolated containerd namespaces. They cover push/pull authorization, cross-node upload forwarding, actual multicast, dropped/reordered/duplicated chunks, HTTP repair, and loss metrics. They disable unpacking so they can run without mount privileges. Never point them at a production daemon. REGISTRYLESS_TEST_INTERFACE overrides the test NIC.

About

Containerd backed registry for Skaffold builds

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages