The gitops stack is the deployment infrastructure of the FAForever project. It describes everything that runs on our Kubernetes clusters, and ArgoCD keeps the clusters in sync with the state of this repository.
Every directory below apps/, infra/, ops/ and cluster/ is a Helm chart that ArgoCD deploys as its own application.
| Directory | Contents | Namespace |
|---|---|---|
apps/ |
FAF services and the third-party apps they rely on (API, lobby server, website, forums, message broker, ...) | faf-apps |
infra/ |
Databases and other stateful infrastructure (MariaDB, PostgreSQL, MongoDB) | faf-infra |
ops/ |
Operational tooling (Infisical, Keel, monitoring) | faf-ops |
cluster/ |
Cluster-wide services that are not FAF-specific (ArgoCD, Traefik, storage). See cluster/README.md | named after the directory |
common/ |
Shared library charts used as dependencies by other charts (e.g. infisical-secret for pulling secrets from Infisical) |
- |
config/ |
Environment-wide Helm values (test.yaml, prod.yaml, local.yaml) such as base domain and Infisical environment |
- |
tilt/, Tiltfile, tilt_config.json |
Local development setup, see Local Development Using Tilt | - |
scripts/, utils/, testing/ |
Helper scripts and ad-hoc manifests for debugging and testing | - |
The app-set-*.yaml files define ArgoCD ApplicationSets that generate one application per chart directory:
app-set-test.yamldeploys thedevelopbranch to the test clusterapp-set-prod.yamldeploys themainbranch to the production clusterapp-set-local.yamldeploys thedevelopbranch to a local cluster usingconfig/local.yaml
Each application is rendered with two layers of values on top of the chart's own values.yaml:
- The environment-wide file from
config/(e.g./config/test.yaml) - The chart's environment-specific override (e.g.
apps/faf-api/values-test.yaml), if one exists
Secrets are not stored in this repository. They are synced from Infisical into the cluster via the infisical-secret common chart.
| Branch | Cluster | Domain |
|---|---|---|
develop |
Test | faforever.xyz |
main |
Production | faforever.com |
ArgoCD syncs both branches automatically, so a push to either branch deploys right away. Changes move through the clusters like this:
- Open a pull request against
develop. The CI checks in.github/workflows/checks.ymlrun on every pull request todevelop. Renovate also opens its dependency update PRs againstdevelop. - Merge into
develop. ArgoCD rolls the change out to the test cluster. - Verify on the test cluster. Check that the affected services start and work as expected. If something breaks, fix it on
developbefore promoting. - Promote to production. Cherry-pick the verified commits from
developontomain. ArgoCD then rolls them out to the production cluster. - Realign
develop. Rebasedevelopontomainand force-push it, so that the two branches share the same history again.
Where possible, avoid committing directly to main and send changes through develop first, so that what reaches production has been tested.
This repository and the cluster are setup to do the following:
- Keel is used to update mutable auto-update images (e.g.
nodebb:4.4updating to 4.4.x releases) - Renovate is used to create PRs when image tags are outdated (for pinned versions as well as mutable versions)
Especially for 3rd party apps we try to use release-line tags. For our own apps we use pinned versions for apps that require combined deployments, for other apps we might use a release branch like main.
In some cases only the test cluster points to rolling-release tags, while prod uses pinned release.
Attention: Renovate does not detect version changes if prod/dev use helm value based release tags.
First, we need to make the cluster gitops-ready by installing ArgoCD:
kubectl create namespace argocd
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocdThen you can connect to argocd using the credentials generated:
PW=$(kubectl get secret -n argocd argocd-initial-admin-secret -o "jsonpath={.data.password}" | base64 -d)
echo "Access argocd via http://localhost:8080"
echo "User: admin, password: $PW"
kubectl port-forward -n argocd svc/argocd-server 8080:80Now, we need to deploy the appsets via
kubectl apply -f app-set-test.yamlMany apps will fail to start, because the lack the secrets that will be generated by infisical. But infiscal needs be setup too. We use the cloud edition, but there is also a self hosted one we do not cover here. For our stack you need to create a machine identity in the web ui and add its credentials as a secret in all affected namespaces:
for namespace in faf-apps faf-ops argocd replay-mounter; do
kubectl create secret generic infisical-machine-identity \
-n "$namespace" \
--from-literal=clientId=<your-client-id-here> \
--from-literal=clientSecret=<your-client-secret-here>
doneThis repository provides a ready-to-go Tilt setup for running ephemeral FAF infrastructure on a local Kubernetes cluster.
- Tilt
- A local Kubernetes cluster. Docker Desktop or Minikube are recommended if you are new to Kubernetes. See also Tilt's guide on choosing a cluster.
- Helm, used to render the Kubernetes manifests from the charts in this repository.
- Windows only: a bash executable. Git Bash at
C:\Program Files\Git\bin\bash.exeis used by default; setwindows-bash-pathintilt_config.jsonto use a different one. - Resources: expect roughly 4 GB of memory for the containers alone. With Docker Desktop on Windows, the full Kubernetes setup used about 11 GB.
From the repository root, run:
tilt upThis starts the FAF services in dependency order. By default every service runs; use the run setting (see Configuration) to start only a subset. Open the Tilt UI at http://localhost:10350 to see the status of each service, restart services, or disable them.
Settings can be put in tilt_config.json in the repository root, or passed on the command line after -- (for example tilt up -- --local faf-api --host-ip 192.168.1.10). List settings such as run and local can be repeated on the command line.
| Key | Description | Default |
|---|---|---|
run |
Services (or groups of services) to start. The client group starts everything needed to connect the FAF client. |
None (everything runs) |
local |
Services you are running yourself from source. Traffic for these is proxied from the cluster to your machine. | None |
host-ip |
IP address of your machine, used when proxying local services. Only needed when hostname is not set. |
None |
hostname |
Hostname of your machine, used when proxying local services. |
host.docker.internal on Windows with Docker Desktop, otherwise none |
base-domain |
Base domain for all services. | faforever.localhost |
faf-data-dir |
FAF data directory. | C:/ProgramData/FAForever on Windows, ~/.faforever elsewhere |
windows-bash-path |
Path to bash.exe on Windows. |
C:\Program Files\Git\bin\bash.exe |
default-pull-policy |
Image pull policy for containers. | IfNotPresent |
Example:
{
"run": ["client"],
"local": ["faf-api"]
}run accepts individual service names, group names, or a mix of both. Groups are defined in run_groups at the top of the Tiltfile. The available groups are:
| Group | Description |
|---|---|
client |
The minimal set of services needed to connect with the FAF client |
Whenever run is set, the populate-db, populate-featured-mod-files, create-hosts-file-content and reloader resources and the OAuth client setup jobs are also enabled. Tilt starts the dependencies of every enabled resource (databases, Traefik, Hydra and so on) automatically.
For example, to run only what the client needs:
tilt up -- --run clientOr to run the client services plus the website:
tilt up -- --run client --run wordpressTo work on a service, add its name to local (in tilt_config.json or with --local on the command line) and run it from source on your machine. Requests inside the cluster for that service are then forwarded to your local instance.
You may need to adjust the service's default configuration to match the cluster. The values the cluster uses can be found in that service's ConfigMap under apps/. If the proxied connections do not reach your machine, set hostname, or host-ip if your machine has no hostname the cluster can resolve.
The default test data comes from faf-db. It is not loaded automatically: trigger the populate-db resource (under the database label in the Tilt UI) once after first startup, and again any time you want to reset the data.
Traefik uses the same ingress routes locally as in production, with a base domain of faforever.localhost. Services are reachable at api.faforever.localhost, hydra.faforever.localhost, and so on.
These names should resolve to 127.0.0.1 automatically. If they don't, add them to your hosts file. Once all services are up, trigger the create-hosts-file-content resource to print the lines you need.