Replace y-kustomize with buckety-controller - #80
Open
solsson wants to merge 15 commits into
Open
Conversation
Collaborator
Author
|
Review from the buckety-controller side. The direction is right; the way the controller is consumed can shrink a lot, and the webhook can stay on. Four points: 1. Replace
|
Both acceptance scripts tore the cluster down on every EXIT, so the only way to inspect a converged ystack was to re-run the phases by hand. --keep leaves it up regardless of outcome; the exit code still propagates, so a kept cluster does not mask a failure. --keep-on-failure previously existed only in the linux script; both scripts now parse both flags with identical semantics. Split the unconditional teardown out of the EXIT trap into teardown_cluster(). The pre-run wipe calls that directly, so a cluster kept by a previous --keep run is still removed before this run provisions -- routing the pre-run wipe through the flag-aware trap handler would have skipped it. osx-amd64 is left as is; it is the stripped variant that stops after 20-gateway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ller
First downstream consumer of Yolean/buckety-controller (initial-impl
on main). Swaps the y-kustomize HTTP-fetched setup-topic Job pattern
for the Buckety CRD on the kafka path; blobs path stays on the
existing y-kustomize Job until the s3 buckety driver lands.
What's in:
- buckety-controller/upstream/ vendors deploy/kustomize/{crd,controller}/
from the buckety-controller repo. The webhook.yaml is intentionally
not vendored (and the corresponding `- webhook.yaml` line is
stripped from controller/kustomization.yaml): ystack does not run
cert-manager, and the upstream webhook requires it for CA-bundle
injection. Documented in upstream/README.md.
- buckety-controller/ is the local overlay: namespace `buckety`, a
secretGenerator-driven config Secret with one kadm backend pointing
at y-bootstrap.kafka.svc.cluster.local:9092, a clusterrolebinding
subject patch onto the namespaced ServiceAccount, and a
deployment-patch.yaml that passes --enable-webhook=false so the
controller starts without a TLS cert (matches the cert-less
deployment recipe documented in buckety-controller's
docs/SCAFFOLDING.md after issue #4).
- k3s/15-buckety/ is the new yconverge step. It depends only on
40-kafka (controller talks to the broker on first reconcile, so
staging it after kafka avoids early-reconcile noise).
- registry/builds-topic/ replaces its HTTP fetch with a Buckety:
backend=cluster-kafka, spec.name=ystack.builds-registry.stream.json
(the friendly operator-chosen name COSI could not give us),
defaultAccess mints Secret topic-builds-registry with flat keys
bootstrap+topic that the registry consumes via plain
secretKeyRef. No init-container shim.
- kafka/validate-topic/ does the same swap with
spec.name=y-cluster-validate-ystack so the existing
y-cluster-validate-ystack produce/consume script can keep
using its hard-coded topic name.
- registry/builds-prep/ drops the setup-topic-prep URL fetch
(no per-namespace ServiceAccount needed when the controller
does the topic create). The setup-bucket-prep URL stays until
the s3 driver lands.
- k3s/60-builds-registry/yconverge.cue replaces the
41-kafka-y-kustomize dep with 15-buckety. The blobs path
keeps 31-blobs-y-kustomize.
- bin/y-cluster-validate-ystack swaps the kafka topic create
check from `kubectl wait Job complete` to
`kubectl wait Buckety Ready`, and drops the now-orphan
y-kustomize kafka base check.
- e2e/agents-clusterautomation-acceptance-linux-amd64.sh adds
one sideload step after provision:
y-cluster images load $BUCKETY_CONTROLLER_OCI --context=local
Default path: ~/Yolean/buckety-controller/oci, overridable via
env. Fails fast with a build hint if the OCI layout is missing.
Verified locally against PR #3 head (commit cb45545, image digest
sha256:11ae9b338310e953b48b26fe31cbde3247fdd09928a1d0c3efd89964d387ccfd):
- 8 yconverge phases green from a clean k3s+docker provision
- y-cluster-validate-ystack: 35 passed / 0 failed (including
kafka topic create + produce + consume against the
buckety-provisioned topic)
- Buckety/builds-registry reaches Ready=True in ~5s
- `rpk topic list` shows the operator-chosen names
(ystack.builds-registry.stream.json,
y-cluster-validate-ystack) end-to-end
- Issued Secrets carry the flat keys (bootstrap, topic) per
the kadm driver schema
Not in this commit (deliberate scope):
- s3 driver swap. The buckety-controller s3 driver is still a
stub returning ErrNotImplemented; blobs-versitygw remains on
the existing y-kustomize Job pattern. Lands in a follow-up
once the upstream s3 driver fills in.
- Image pin. The acceptance script sideloads a locally-built
OCI from ~/Yolean/buckety-controller/oci; once
buckety-controller cuts a tagged release with the new
per-commit timestamp tag + pinned digest workflow
(Yolean/buckety-controller@172e1ca), this overlay should
switch to vendoring deploy/kustomize/release/ and pinning
the image by digest. Follow-up.
- Driver-version e2e (SPEC scenario #9 in
buckety-controller). Not exercised; needs three
differently-stamped binaries that the upstream
driver-version harness doesn't yet ship.
Builds on 7a8e4cc (kafka topic path) with the s3 bucket path now live against the buckety-controller s3 driver (Yolean/buckety-controller@8ea9e13, image digest sha256:0c56d6f9205b243766e777fa2a4b0f028fa62c2487c4c24ccc54e587943ef495). What's new: - buckety-controller/buckety-controller.yaml adds a second backend `cluster-objects` for the s3 driver: implementation versitygw, endpoint y-s3-api.blobs.svc.cluster.local:9000, region us-east-1, forcePathStyle. accessKeyID/secretAccessKey are envsubst:"true" fields populated from a local-copy versitygw-creds Secret via the controller Deployment's env. - buckety-controller/kustomization.yaml adds a second secretGenerator (versitygw-creds) carrying the same dev literal credentials as blobs-versitygw/defaultsecret/. The upstream Secret in the blobs namespace cannot cross into buckety; envsubst is startup-time so the controller Pod needs these as env vars at first boot. Rotating credentials means rotating both copies. - buckety-controller/deployment-patch.yaml adds VERSITYGW_ROOT_ACCESSKEY and VERSITYGW_ROOT_SECRETKEY env vars sourced from versitygw-creds via secretKeyRef. Kustomize rewrites the Secret name-suffix hash automatically. - registry/builds-bucket/ swaps from the y-kustomize URL fetch for the setup-bucket Job to a Buckety: metadata.name builds-registry-bucket (distinct from the kafka-side builds-registry-topic), backend cluster-objects, spec.name ystack-builds-registry (friendly bucket name on versitygw), defaultAccess ReadWrite with credentialsSecretName builds-registry-bucket. - registry/builds-topic/ rename: metadata.name builds-registry -> builds-registry-topic so both Bucketies coexist in the ystack namespace. - k3s/60-builds-registry/deployment-s3.yaml secret key names updated to match the s3 driver's output keys: accesskey -> accessKeyID secretkey -> secretAccessKey REGISTRY_STORAGE_S3_BUCKET and REGISTRY_STORAGE_S3_REGION switch from hardcoded literals to secretKeyRef so the bucket name and region come from the issued Secret. - registry/builds-prep/ removed. Its only contents were two y-kustomize URL fetches for per-namespace setup-Job RBAC; with both Buckety swaps in place the per-namespace SA is redundant (the controller does the backend work). - k3s/60-builds-registry/kustomization.yaml drops the builds-prep resource entry. yconverge.cue drops the 31-blobs-y-kustomize and 29-y-kustomize imports; the dep chain is now 60-builds-registry -> 15-buckety -> {30-blobs, 40-kafka}. y-kustomize is entirely retired from the converge graph for this stack. - k3s/15-buckety/yconverge.cue adds 30-blobs to its deps: the s3 driver needs versitygw reachable on first reconcile. Verified locally: - 8 yconverge phases green, no y-kustomize-related fetches in the converge path. - y-cluster-validate-ystack: 33 passed / 0 failed. Drop from the 35 in 7a8e4cc reflects the now-retired y-kustomize blobs-base serving checks. - y-build builds an image and pushes it to the buckety-provisioned S3-backed registry; node-pull via the registries.yaml mirror succeeds. - Both Bucketies (builds-registry-topic, builds-registry-bucket) reach Ready=True; issued Secrets carry the documented flat keys consumed by the registry Deployment via plain secretKeyRef. - Backend resource names on the broker / object store match the operator-chosen identities (ystack.builds-registry.stream.json, ystack-builds-registry, y-cluster-validate-ystack). Not in this commit (deliberate scope): - Image pin via buckety-controller tagged-release artefacts. Once the per-commit timestamp tag + pinned-digest workflow (Yolean/buckety-controller@172e1ca) cuts a release, this overlay should vendor deploy/kustomize/release/ instead of base/ and pin the image by digest. - Driver-version e2e. Needs three differently-stamped binaries the upstream harness doesn't yet ship. - Per-consumer credential scoping (SASL/SCRAM, IAM users). v1alpha2 work upstream; for now all BucketyAccess instances for the same Buckety receive identical root credentials.
Builds on 946ce24 (s3 path swap). With both Bucketies in place nothing in the converge graph references y-kustomize anymore. This commit deletes the now-unreachable source tree, the provision-side port forward, and the comment scaffolding around the old serving model. Deleted: - y-kustomize/ (Deployment + Service + HTTPRoute + serve config for the in-cluster y-cluster-serve instance) - k3s/29-y-kustomize/, k3s/31-blobs-y-kustomize/, k3s/41-kafka-y-kustomize/ (the yconverge steps that installed the Deployment and loaded per-domain source Secrets) - blobs-versitygw/setup-bucket-{,prep-}y-kustomize/ and kafka/setup-topic-{,prep-}y-kustomize/ (the source bases that were rendered into y-kustomize.{group}.{name} Secrets for the in-cluster service to serve) - bin/acceptance-y-kustomize-local (the standalone test for the host-local serve mode; no consumer in ystack any more) Updated: - cluster-configs/local-{docker,qemu}/y-cluster-provision.yaml drop the host:8944 -> guest:8944 PortForward and the surrounding comment block. ystack now only forwards 6443 / 80 / 443. - bin/y-cluster-validate-ystack drops the y-kustomize-bases section; rewords one example comment that referenced y-kustomize:8944 to a generic placeholder. - e2e/agents-clusterautomation-acceptance-linux-amd64.sh drops the cleanup-trap probe for stale host-local y-cluster serve on :8944, the Phase 2 comment block about the in-cluster Deployment, and the y-kustomize-flavoured phase labels. - k3s/README.md rewritten to describe the buckety-controller flow instead of the y-kustomize-serving flow that no longer exists. Retained (intentional): - bin/y-kustomize and bin/y-kustomize-traverse wrappers for the upstream sigs.k8s.io/kustomize CLI and y-cluster's kustomize-traverse subcommand respectively. Unrelated to the in-cluster serve. - One comment in k3s/60-builds-registry/yconverge.cue noting the historical migration ("y-kustomize is retired from this dep chain ..."). Kept for archaeology. - runner.Dockerfile lines copying the wrapper binaries. Verified: - y-cluster-validate-ystack: 33 passed / 0 failed against a clean provision with no y-kustomize Deployment, no host port 8944 forward, no host-local serve. - Acceptance script runs to teardown without --keep-on-failure tripping. - git grep "y-kustomize" returns only the intentional references above. Companion spec filed: Yolean/specs@9281c13 documents which y-cluster surfaces ystack has stopped depending on, so the y-cluster maintainer can plan accordingly. No phase-out strategy proposed there or here.
Parallel converge bases for a minio-backed objects path: buckety-controller-minio/, k3s/15-buckety-minio-disabled/, k3s/60-builds-registry-minio-disabled/, k3s/62-buildkit-minio-disabled/. Versitygw stays the default; the -disabled suffix keeps the minio bases out of any global converge sweep. Consumers select minio by pointing yconverge at the -minio-disabled bases. The e2e acceptance and y-cluster-validate-ystack gain an internal toggle for exercising the minio path; the rollout/restart checks target deploy/minio when set. 30-blobs-minio-disabled/yconverge.cue is no longer a stub. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The existing call in k3s/20-gateway/yconverge.cue ran with no
HTTPRoutes/GRPCRoutes yet present, so /etc/hosts never picked up
route hostnames like builds-registry.ystack.svc.cluster.local.
Acceptance silently relied on whatever stale entry happened to be
in /etc/hosts from prior testing; the y-build registry probe
("Skaffold need local access ...") surfaced as a sporadic late
failure whenever that stale entry got cleared.
Adds the same idempotent exec check to every base that creates a
route -- 50-monitoring, 60-builds-registry{,-minio-disabled},
62-buildkit{,-minio-disabled} -- so y-k8s-ingress-hosts runs after
the new routes exist and the write picks them up.
Also drops the `|| echo 'WARNING ...'` fallback from all six call
sites. Swallowing the failure let the converge step succeed with
/etc/hosts inconsistent, then deferred the user-visible failure to
a much later step with no traceable cause. Hard fail at the source.
This deepens the y-cluster <-> y-k8s-ingress-hosts coupling that
Yolean/y-cluster#11 flags as a
distribution concern: the host-side /etc/hosts contract now hard
requires the helper to run inside the converge, not as an optional
post-step. The dns-hint-ip mechanism the helper consumes was
adopted in commit 2058e5c ("gateway: retire OVERRIDE_IP env var,
adopt yolean.se/dns-hint-ip"); this commit extends that contract
to every base that adds routes after the Gateway is up.
Verified with BLOBSTORE=minio acceptance: 33/0 (including y-build,
which previously failed at host-side curl for both blobstores once
/etc/hosts had been cleared between runs).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopts the CI-tagged buckety-controller image from commit ccfb662 on the YoleanAgents/buckety-controller secret-safety-and-events branch. Vendored CRD + controller manifests are unchanged (they've been identical since edc8eb0 "initial impl"); only the image ref in upstream/controller/deployment.yaml pins to the per-commit timestamp tag published by the fork's e2e workflow. Consumer setup: crane pull the image into $HOME/Yolean/buckety-controller/oci (or override BUCKETY_CONTROLLER_OCI) so the acceptance script's sideload picks up the tagged image; kubelet then finds it locally at IfNotPresent. Verified: 33/0 on default (versitygw) and BLOBSTORE=minio acceptance runs. Buckety reconcile of both the kafka topic and the s3 bucket succeeds; registry pod picks up the resulting credential Secret and serves /v2/ as before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
solsson
force-pushed
the
buckety-instead-of-y-kustomize
branch
from
August 31, 2026 14:56
eb68d83 to
4c3cabf
Compare
Replaces the vendored copy of deploy/kustomize/{crd,controller} with a
git ref at 9afdccb, the same commit cluster-g2/buckety pins in checkit.
Vendoring is what checkit reserves for bases that y-site-generate
converges unattended; a backend base like this one is applied by
yconverge from a checkout that already needs the network for images,
so a transient fetch failure is a retry rather than an outage. The ref
carries the image pin, so a version bump is now a one-line ref change
with no digest to keep in sync.
9afdccb over the v0.1.1 tag for the two defects the checkit #7555
rollout hit in production: a condition message no longer drives a
status patch (a persistently failing reconcile fed its own watch at
~18/s, because the provider error went verbatim into Ready and the
provider mints a fresh error id per rejection, so every message was a
real change and the exponential limiter - which governs requeues, not
watch events - never applied), and the first bind of a freshly created
service account counts as provisioning rather than failure. Neither is
gcs-specific in nature; both are reconciler behaviour that any driver
able to fail repeatedly would meet.
The webhook is now ENABLED, via release-tls-selfsigned. Passing
--enable-webhook=false was this branch's answer to ystack having no
cert-manager, and buckety-controller's webhook-certgen was written to
remove that excuse - it cites this branch by name. Running without
admission means bad parameters, immutability violations, unresolvable
name templates and refused adoptions surface as Ready=False on an
object that was accepted, instead of failing the apply where yconverge
reports it. The base mounts the certgen-minted Secret optional:true so
the Pod starts before the Jobs finish and picks the cert up when it
lands, and args need no patch because the base already passes
-c /etc/buckety.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cluster-kafka and cluster-objects named the local implementation, so
every CR's spec.backend said where it runs rather than what it wants.
checkit's cluster-g2 backend is site-userdata-blobs for the opposite
reason: the name is the portable contract, and the same CR is meant to
resolve to gcs on g2 and to versitygw here (buckety-controller#18,
whose CI asserts a byte-identical CR provisions on minio, versitygw and
the GCS emulator). Renamed to builds-registry-stream,
acceptance-validation-stream and builds-registry-blobs.
The two kadm entries are the same broker on purpose. Duplicating one
physical backend into several named instances is the pattern that makes
each CR state its use case, and it means moving the acceptance probe
topic to another broker is a config change rather than a CR rewrite.
Deliberately NOT site-userdata-blobs for the s3 entry: that name is
per-site end-user blobs and this bucket holds build artifacts. When
cluster-local starts hosting sites, the move is to ADD a
site-userdata-blobs entry pointing at the same versitygw, at which
point checkit's per-site CRs become byte-identical here - which is the
point of the contract.
Drops `defaults: zone: local` from both entries. defaults resolve
${backend.X} substitutions in spec.name templates and never reach the
driver; every ystack CR spells its name as a literal, so the entry was
inert config that read as if it did something. Same check checkit ran
before dropping defaults.location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MinIO discontinued its community edition, so carrying a parallel minio backend, a parallel controller overlay and three -minio-disabled converge steps buys an escape hatch we would not take. The BLOBSTORE switch and its export go with them, from the acceptance script and from y-cluster-validate-ystack's rollout checks. k3s/30-blobs-minio-disabled is deliberately left alone: it predates this branch and is the minio blob store itself rather than a buckety backend. Retiring it is a separate decision about what ystack offers as a store, not about how buckety provisions into one. Acceptance also stops sideloading an OCI layout from a sibling buckety-controller checkout. That existed because the image was built locally with contain and unpublished; the release ref this branch now installs from pins a published image, and ghcr.io/yolean/buckety-controller:20260729T151707Z resolves to exactly the digest that base carries. One fewer host precondition, and the linux acceptance run no longer fails on a machine that has no buckety-controller checkout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Now that versitygw is the only blobstore, it should not be three minor versions behind. The range carries security fixes that matter for a store the s3 buckety driver authenticates against: SigV4 SignedHeaders validation is now enforced, a copy-source parsing mismatch that could bypass path validation is fixed, and an LDAP access-key filter injection is closed. Several nil-pointer panics in error and proxy paths go with them. Nothing in the deployment contract moves: the image entrypoint is unchanged, the posix /data backend and the ROOT_ACCESS_KEY_ID / ROOT_SECRET_ACCESS_KEY env pair are the same, and neither release exposes ports in image config, so the 7070 containerPort stands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
v0.2.0 is the first tag containing the two reconciler fixes we pinned an
untagged main commit for after the checkit #7555 rollout, so the pin is a
release again rather than a pick.
The rendered manifests move in exactly two ways:
- image 20260729T151707Z/0b083f26 -> 20260904T064507Z/19e3172a, carried by
the release base, still linux/amd64 only
- the controller ClusterRole narrows: bucketies drop to
get/list/watch/update/patch, create/delete stay on bucketyaccesses.
Every ystack Buckety is applied by yconverge, so nothing here wants a
verb it lost.
Nothing in the overlay needed to follow: the CRDs are byte-identical across
the range, pkg/config only gained a method, and buckety-controller.yaml's
fields still match the upstream reference configs. Upstream now sets
`namespace: buckety` in its own bases, which our `namespace:` and
clusterrolebinding-patch.yaml agree with rather than contradict.
Verified with kustomize build of both refs, diffed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
v0.5.2 is v0.5.0 plus the preflight-privileged-ports PR (#32), which is what the macOS acceptance path was blocked on: d5df972 fix(preflight): probe the address the provider actually binds 4961329 feat(inventory): record provisioned clusters so teardown can list 9c23971 fix(preflight): probe tcp4 so the wildcard probe sees docker's IPv4 listener checkHostPort used to bind 127.0.0.1:<port> and read any error as "in use". Darwin enforces the reserved-port rule on a specific-address bind but not on the wildcard one, so a non-root probe of 80/443 came back EACCES and every acceptance run on a Mac failed preflight with "host port 80 in use (likely another cluster)" while nothing was listening. v0.5.2 probes the wildcard first and only calls a port taken on EADDRINUSE. Checksums generated with bin/y-bin-download-next; the wrapper's download verifies and the binary reports v0.5.2 (3e57231). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The portable half of that contract is that every cluster defines the names, which is what site-userdata-blobs was already meant to get for buckets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces y-kustomize from #70 with https://github.com/Yolean/buckety-controller/.
Inspired by https://github.com/GoogleCloudPlatform/k8s-config-connector but vendor neutral and with the assumption that we can make in-tree scope creeps if/when our need grows.