The build shared by the componentized repositories: make files, scripts and GitHub workflows, synced into each repository at a released version.
Each repository keeps a copy of the build and records the version it came from, so a release tag of a repository contains the exact build that published it. A copy is never edited in place. Changes are made here, released, and each repository updates when it's ready.
Makefile repository-owned: settings, `include build/build.mk`, its own targets
build/ synced, never edited in the repository
VERSION the release synced, e.g. v0.2.0
MANIFEST a checksum of each synced file, checked by `make check-build`
build.mk the targets every repository has, includes the other make files
tools.mk wit.mk components.mk publish.mk sync.mk pinned.jq local.jq completions.jq
tools/Cargo.toml the tools the build provides, see Tools
scripts/ publish.sh, bump-version.sh, update-build.sh, ...
.github/workflows/ ci.yaml, bump-version.yaml, release.yaml, update-build.yaml: synced
tools/Cargo.toml repository-owned and optional, the repository's own tools and overrides, bumped by dependabot
SYNC lists the paths synced into each repository.
make update-build updates to the latest release, make update-build BUILD_VERSION=v0.2.0 to a given one. It removes the files synced from the previous version, copies in the release's files, and writes build/VERSION and build/MANIFEST.
The Update build workflow does the same each week, and opens a pull request when there is a newer release, like dependabot. The pull request is created with the custodian GitHub App, the GITHUB_TOKEN can't change workflow files. The app needs the contents, workflows and pull-requests write permissions, and the CUSTODIAN_CLIENT_ID and CUSTODIAN_PRIVATE_KEY secrets must be available to the repository.
The versions of the tools the build provides come with it, see Tools. The update removes a tool the repository overrides in its tools/Cargo.toml once the build pins the same version or a newer one.
The CI workflow runs build/scripts/check-build.sh first, which fails when a synced file was changed or removed, or a file was added to build/. Fix the build here rather than in the repository, so the repositories don't drift apart again.
The repository's Makefile sets these before include build/build.mk. Each has a default that fits most repositories, detected where it can be.
| Setting | Default | |
|---|---|---|
COMPONENT_TARGET |
wasm32-unknown-unknown |
the rust target of the components cargo builds. wasm32-wasip3 installs wasi-sdk with the tools, or uses WASI_SDK_PATH. wasm32-wasip2 copies the component rust links, without COMPONENT_OPTIMIZE. A component names its own in its Cargo.toml, [package.metadata.componentized] target = "wasm32-wasip2", a binary crate builds a command |
COMPONENT_PROFILE |
component when Cargo.toml has a [profile.component], otherwise release |
the cargo profile of a component's release build |
COMPONENT_DEBUG_PROFILE |
component-debug when Cargo.toml has a [profile.component-debug], otherwise dev |
the cargo profile of a component's debug build |
COMPONENT_OPTIMIZE |
optimize the release builds again for size with wasm-opt when set, e.g. COMPONENT_OPTIMIZE := true |
|
CHECK_COMPONENTS_SKIP |
the components whose release and debug builds may have different WIT, skipped by make check-components, e.g. a wasm32-wasip3 component whose release build wasm-opt optimizes, dropping imports of the standard library its debug build keeps |
|
CARGO_COMPONENT_DEPS |
files the components cargo builds need besides their sources, e.g. a component the library embeds | |
LOCAL_TOOLS |
tools built from the workspace rather than pinned, each a package, e.g. componentized-constants-cli in componentized/constants. Installed with the pinned tools, with a variable for each binary like a pinned tool, e.g. $(CONSTANTS), and installed again when the sources of the package, or of the workspace packages it depends on, change. The components built with a local tool are built again when it changes. A local tool replaces a pinned tool of the same name, one the build provides too, and a file it embeds is added as a prerequisite, e.g. $(call tool,componentized-static-config-cli): $(CONFIG_ADAPTER_WASM) |
|
CLI_BIN |
the cli built from the repository, e.g. constants. The CI workflow builds it for each platform, smoke tests it with scripts/smoke-test-cli.sh <cli> <dir> when the repository has one, and attaches the archives to the release |
|
PUBLISH_CRATES |
the crates the release workflow publishes to crates.io, e.g. componentized-constants componentized-constants-cli, checked with cargo publish --dry-run by the CI workflow |
|
PUBLISH_INTERFACE |
true |
publish the interface, interface.wasm, false for a repository whose interface isn't one to depend on, e.g. componentized/build, whose interface is there to test the build. The components published are those that aren't dep-, internal- or test- |
Publishing reads DRY_RUN: make publish DRY_RUN=true shows each image it would push, with its annotations, without pushing or signing anything. In the workflows the repository variable DRY_RUN=true does the same, also skipping the login to the registry and the GitHub release the CI workflow drafts for a tag, and the release workflow checks the crates with cargo publish --dry-run rather than publishing them.
make components also generates an SBOM of the interface and of each component, CycloneDX JSON beside the file it describes, e.g. target/components/gate/gate.cdx.json for gate.wasm, see build/scripts/sbom.sh. It lists what the file is made from: a component built with cargo, the crates it depends on built for its target, from cargo-cyclonedx, those only proc-macros use as it's compiled, e.g. syn, scoped excluded as cargo tree -e normal,no-proc-macro doesn't list them, the Rust standard library by the commit of rustc, the wasi-libc it links, for wasm32-wasip3 the one of the wasi-sdk sysroot or built from WASI_LIBC_VERSION, and each WASI adapter it embeds, e.g. wasi_snapshot_preview1, by its SHA-256; a composition, .wac or .plug, the components it's composed from with what each is made from; a component generated with the constants or static-config cli, the cli's crate; a component pulled with wkg, the image it's pulled from, by its OCI coordinates, on top of the SBOM attested to the image when cosign verifies one signed by a GitHub Actions workflow of the image's org, so a component composed from another repository's lists what that component is made from; the interface, the WIT packages in wkg.lock. The SBOMs of a commit are the same each time they're generated, dated by the commit. Each SBOM names the component it describes by the directory's id of the package it's published as, e.g. componentized:build-greeter, and publishing adds its version and image, and attests it to the image with cosign, cosign verify-attestation --type cyclonedx <image> verifies it, and writes the SBOMs published to target/sbom/published, laid out like target/components. The CI workflow packs the components into the components.tar of the release it drafts with an SBOM for each, see build/scripts/pack-components.sh: the one it's published with; for a debug build not published as it's identical to the release build, the release build's; and for a file that isn't published, e.g. a dep- component, the build's, which for a component pulled with wkg is based on the SBOM it was published with.
The variable COMPONENTIZED_DIRECTORY_SCAN=true asks componentized/directory to scan the components the CI workflow publishes, there is no scan for a dry run. It is set for every repository in the componentized organization, so a repository outside it, without the custodian's access to the directory, doesn't request a scan that would fail. The scan is requested with the custodian GitHub App, which needs the contents write permission on componentized/directory. With it set, the release notes the CI workflow drafts lead each published interface and component with the directory's icon for its kind, a package, component or composition, linking to the page of its version in the directory, e.g. componentized:logging-levels@0.3.1 for ghcr.io/componentized/logging/levels:0.3.1, which exists once the scan has indexed it. The icons are the directory's own, from componentized.dev/_icons, in the colors of the reader's GitHub theme.
The build provides the tools its rules run, pinned in build/tools/Cargo.toml and bumped by dependabot here: cargo-cyclonedx, wac, wkg, wasm-opt, wasm-tools, the constants and static-config clis, and componentized-build, see Updating a dependency, run as $(COMPONENTIZED_BUILD). It also provides cosign, a release binary pinned in build/tools.mk and checked against the release's checksums, which signs the published components, attests their SBOMs and verifies the SBOMs of the components pulled with wkg. make tools installs them in target/tools, with cargo binstall when it's installed, and a recipe runs each by path from a variable named after its binary, e.g. $(WASM_TOOLS).
A repository changes a tool the build provides, or adds its own, without changing the build:
- A newer version: pin it in the repository's
tools/Cargo.toml, which overrides the build's, with the binaries it installs in[package.metadata.bins]. Dependabot bumps it there, andmake update-buildremoves it once the build catches up, pinning the same version or a newer one. - Built from the workspace: name its package in
LOCAL_TOOLS, e.g.LOCAL_TOOLS := componentized-constants-cliin componentized/constants, built from the repository's sources in place of the version the build provides. - A binary built elsewhere, to try a change: set its variable on the command line, e.g.
make WKG=../wkg/target/release/wkg. The scripts the recipes run find the installed tools on thePATH. - Its own tools, e.g.
wasmtime-clifor its tests: pin them intools/Cargo.tomllike an override. A repository with no tools of its own needs notools/Cargo.toml, nor its dependabot entry.
The tools complete in bash and zsh once their completions are registered at the shell's startup, after the tools are on the PATH:
PATH="$(make -s tools-path):${PATH}"
source /path/to/repository/build/scripts/completions.shA binary that prints its completions is listed in [package.metadata.completions] with the arguments that print them, the shell's name appended, e.g. wasm-tools = ["completion"] for wasm-tools completion bash. The build lists those of the tools it provides, cosign too, and a repository lists its own tools in its tools/Cargo.toml, or overrides one of the build's, [] for none. make tools generates target/tools/completions from them, read as each shell starts, a tool not installed yet is skipped.
make runs tools, generate, components and test. Other targets: wit, publish, tools-path, clean, clean-all, update-build, check-build, print-<variable>.
A repository extends a target by adding prerequisites after the include, make merges them with the target's own:
| Target | Run by | Extended with, e.g. |
|---|---|---|
generate |
the CI workflow, which fails when the generated files differ from those committed | generate: config |
test |
make, the CI workflow |
test: test-grammar |
test-integration |
the CI workflow, after make test |
test-integration: test-valkey, tests that need a server in a container |
tools |
make, the workflows |
tools: $(WASI_SDK), a tool installed other than with cargo |
- Remove
make/, andscripts/publish.sh,scripts/bump-version.shandscripts/wac-deps.sh. - Sync the build, from the repository's root:
mkdir -p build/scripts curl -fsSL https://raw.githubusercontent.com/componentized/build/v0.1.0/build/scripts/update-build.sh -o build/scripts/update-build.sh bash build/scripts/update-build.sh v0.1.0
- Replace the shared part of the
Makefilewithinclude build/build.mk: the settings before it, the repository's own targets after it. A repository-specific CI step becomes a prerequisite of a target, e.g.test-integration: test-valkey. - Remove the
github-actionsecosystem from.github/dependabot.yml, the workflows are synced, their actions are bumped here. - Add
wkg.lock.origto.gitignore. - Check
makeandmake check-buildpass.
The default version of the bump-version workflow, the repository's current version, is the one line of a synced file the repository keeps: make update-build carries it forward, from the workflow of the previous build too, or starts it at the interface version, and make check-build leaves it out. build/scripts/bump-version.sh updates it with each bump.
A repository depends on others for their interfaces, WIT packages it names at a version, e.g. componentized:secrets/store@0.1.1-dev, pinned in wkg.lock by digest, and for their components, pulled with wkg from the images a components/dep-*/*.wkg file pins by digest. The componentized-build cli, the componentized-build-cli crate, crates/cli, updates them all at once, looking up each version and digest in the registry before anything changes:
# or `make tools`, the build provides it, run with `$(make -s tools-path)/componentized-build`
cargo install componentized-build-cli
componentized-build deps # what the repository depends on, by version
componentized-build dep componentized:secrets 0.1.1 # the interface and components of componentized/secrets
componentized-build dep componentized:secrets # the latest build of the version it's at, e.g. 0.1.1-dev
componentized-build dep componentized:secrets-gate 0.1.1 # one component alone
componentized-build dep componentized:secrets 0.1.1 -n # what would change, without changing itA dev version is published again with each build of its repository, at the same tag, so a repository testing the latest dev build updates to it again, pinning its new digests. make then fetches the WIT and pulls the components. componentized-build completions registers completions with the shell, which complete the dependencies of the repository and the versions published.
This repository is built by its own build, with the synced workflows, like any other repository. Its components and cli are just enough to smoke test each kind of thing the build makes. The components are tests, test-, and its interface is too, PUBLISH_INTERFACE := false, so nothing of it is published but its cli:
wit/ |
componentized:build, the interface, built into interface.wasm, not published: a greeter, and its defaults |
test-greeter |
a component built with cargo, implementing greeter, importing defaults and wasi:logging |
test-greeter-wasip2 |
a component built with cargo for wasm32-wasip2, implementing greeter with the default name from the environment, linking the wasi-libc the Rust toolchain bundles, run by make test-integration with wasmtime |
test-greeter-wasip3 |
a component built with cargo for wasm32-wasip3, like test-greeter-wasip2 greeting on a cooperative thread, linking the wasi-libc of the wasi-sdk sysroot, run by make test-integration with wasmtime |
test-defaults |
a component built with the constants cli, implementing defaults from the @value of each function |
dep-componentized-logging-to-stderr |
a component pulled from a registry, wasi:logging written to stderr |
test-hello |
a composition of the three, run by make test-integration with wasmtime |
test-hello-plug |
the three plugged together with wac plug, run by make test-integration with wasmtime |
crates/cli |
componentized-build, the cli that updates a repository's dependencies, see Updating a dependency, built for each platform, smoke tested with scripts/smoke-test-cli.sh, checked with cargo publish --dry-run, and installed as a local tool, $(COMPONENTIZED_BUILD), run by make test-integration |
make test also runs test/sync-test.sh, which syncs the build into a new repository and checks the drift check. The Lint workflow runs shellcheck. As the source of the build rather than a copy of it, this repository has no build/VERSION or build/MANIFEST: make check-build passes, and the Update build workflow skips it.
Pushes to main and tags go through publishing with nothing to publish, and the release of a tag is drafted with components.tar and the cli archives. The release workflow publishes the componentized-build-cli crate, the cli every repository gets as a tool, at the version build/tools/Cargo.toml pins. crates.io's trusted publishing only applies to a crate that exists, so its first version was published by hand.
Change the files here, and check them with make. Then try the change in a repository before releasing it:
BUILD_SOURCE=../build make update-buildRelease with the Bump version workflow like any other repository, the version of the build is the version of its componentized:build interface. It tags the release, e.g. v0.1.0, and moves on to the next pre-release.
The CI workflow drafts the GitHub release of the tag, publish it once it's reviewed: publishing it publishes the cli crate, and the Update build workflow of each repository looks for the latest release. Say in the release notes when a repository has to change, e.g. a new setting or a tool it must pin.