Skip to content

feat: private networking #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mweibel wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: private networking #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
06d1bd9
feat: private networking
mweibel Mar 27, 2026
f8a9589
review fixes
mweibel May 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/e2e-weekly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,11 @@ jobs:
- name: Run weekly e2e tests
env:
CLOUDSCALE_API_TOKEN: ${{ secrets.CLOUDSCALE_API_TOKEN }}
CLOUDSCALE_NETWORK_UUID: ${{ secrets.CLOUDSCALE_NETWORK_UUID }}
TAG: e2e-weekly-${{ github.sha }}
run: |
make test-e2e \
GINKGO_LABEL_FILTER="ha || upgrade || self-hosted || kcp-remediation || conformance" \
GINKGO_LABEL_FILTER="ha || upgrade || self-hosted || kcp-remediation || conformance || byo-networking" \
KUBETEST_CONFIGURATION=./data/kubetest/conformance-fast.yaml

- name: Install regctl
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,9 @@ jobs:
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
ls dist/cluster-template*.yaml >/dev/null || { echo "No cluster templates found in dist/"; exit 1; }
gh release create "$TAG" \
--generate-notes \
dist/infrastructure-components.yaml \
dist/metadata.yaml \
dist/cluster-template.yaml
dist/cluster-template*.yaml
2 changes: 2 additions & 0 deletions .github/workflows/test-e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ on:
- test-e2e-upgrade
- test-e2e-self-hosted
- test-e2e-md-remediation
- test-e2e-byo-networking
- test-e2e-conformance
- test-e2e-conformance-fast

Expand Down Expand Up @@ -53,6 +54,7 @@ jobs:
- name: Run e2e tests
env:
CLOUDSCALE_API_TOKEN: ${{ secrets.CLOUDSCALE_API_TOKEN }}
CLOUDSCALE_NETWORK_UUID: ${{ secrets.CLOUDSCALE_NETWORK_UUID }}
TAG: e2e-manual-${{ github.sha }}
TEST_TARGET: ${{ github.event.inputs.test_target }}
run: make $TEST_TARGET
Expand Down
36 changes: 26 additions & 10 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,17 +125,20 @@ generate-e2e-cni: ## Regenerate Cilium CNI manifest from Helm chart
generate-e2e-ccm: ## Regenerate cloudscale CCM manifest
@CCM_VERSION=$(CCM_VERSION) hack/generate-e2e-ccm.sh

E2E_CLUSTER_TEMPLATES := cluster-template \
cluster-template-ha \
cluster-template-upgrades \
cluster-template-md-remediation \
cluster-template-byo-network \
cluster-template-public-lb-private-nodes \
cluster-template-fip

.PHONY: generate-e2e-templates
generate-e2e-templates: $(KUSTOMIZE) generate-e2e-cni generate-e2e-ccm ## Generate e2e cluster templates using kustomize overlays
@mkdir -p $(E2E_TEMPLATES)/main
@echo "Generating cluster-template.yaml..."
@"$(KUSTOMIZE)" build --load-restrictor LoadRestrictionsNone $(E2E_TEMPLATES)/cluster-template > $(E2E_TEMPLATES)/main/cluster-template.yaml
@echo "Generating cluster-template-ha.yaml..."
@"$(KUSTOMIZE)" build --load-restrictor LoadRestrictionsNone $(E2E_TEMPLATES)/cluster-template-ha > $(E2E_TEMPLATES)/main/cluster-template-ha.yaml
@echo "Generating cluster-template-upgrades.yaml..."
@"$(KUSTOMIZE)" build --load-restrictor LoadRestrictionsNone $(E2E_TEMPLATES)/cluster-template-upgrades > $(E2E_TEMPLATES)/main/cluster-template-upgrades.yaml
@echo "Generating cluster-template-md-remediation.yaml..."
@"$(KUSTOMIZE)" build --load-restrictor LoadRestrictionsNone $(E2E_TEMPLATES)/cluster-template-md-remediation > $(E2E_TEMPLATES)/main/cluster-template-md-remediation.yaml
@$(foreach tmpl,$(E2E_CLUSTER_TEMPLATES),\
echo "Generating $(tmpl).yaml..." && \
"$(KUSTOMIZE)" build --load-restrictor LoadRestrictionsNone $(E2E_TEMPLATES)/$(tmpl) > $(E2E_TEMPLATES)/main/$(tmpl).yaml &&) true
@echo "Templates generated successfully."

.PHONY: generate-e2e-config
Expand Down Expand Up @@ -222,6 +225,19 @@ test-e2e-md-remediation: $(GINKGO) generate-e2e-templates generate-e2e-config do
-e2e.skip-resource-cleanup=$(SKIP_RESOURCE_CLEANUP) \
-e2e.use-existing-cluster=$(USE_EXISTING_CLUSTER)

.PHONY: test-e2e-byo-networking
test-e2e-byo-networking: $(GINKGO) generate-e2e-templates generate-e2e-config docker-build ## Run BYO networking e2e tests
$(GINKGO) -v --trace --tags=e2e \
--nodes=$(GINKGO_NODES) \
--label-filter="byo-networking" \
--timeout=90m \
--output-dir="$(E2E_ARTIFACTS_FOLDER)" --junit-report="junit.e2e_byo_networking.xml" \
./test/e2e -- \
-e2e.config=$(E2E_CONF_FILE) \
-e2e.artifacts-folder=$(E2E_ARTIFACTS_FOLDER) \
-e2e.skip-resource-cleanup=$(SKIP_RESOURCE_CLEANUP) \
-e2e.use-existing-cluster=$(USE_EXISTING_CLUSTER)

.PHONY: test-e2e-conformance
test-e2e-conformance: $(GINKGO) generate-e2e-templates generate-e2e-config docker-build ## Run K8s conformance e2e tests
$(GINKGO) -v --trace --tags=e2e \
Expand Down Expand Up @@ -298,9 +314,9 @@ build-installer: manifests generate kustomize ## Generate a consolidated YAML wi
"$(KUSTOMIZE)" build config/default > dist/infrastructure-components.yaml

.PHONY: release-manifests
release-manifests: build-installer ## Build all release artifacts into dist/ (infrastructure-components.yaml, metadata.yaml, cluster-template.yaml).
release-manifests: build-installer ## Build all release artifacts into dist/ (infrastructure-components.yaml, metadata.yaml, cluster templates).
cp metadata.yaml dist/metadata.yaml
cp templates/cluster-template.yaml dist/cluster-template.yaml
cp templates/cluster-template*.yaml dist/

##@ Deployment

Expand Down
57 changes: 45 additions & 12 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@ for [cloudscale.ch](https://www.cloudscale.ch).

## Features

- **CloudscaleCluster**: Network, Subnet, Load Balancer management
- **CloudscaleMachine**: Server provisioning with cloud-init
- **CloudscaleCluster**: Multi-network management (managed or BYO), Load Balancer (public or private VIP), Floating IP
support
- **CloudscaleMachine**: Server provisioning with cloud-init and configurable network interfaces
- **CloudscaleMachineTemplate**: Immutable machine templates for KubeadmControlPlane/MachineDeployment

## Prerequisites
Expand Down Expand Up @@ -42,6 +43,9 @@ clusterctl generate cluster my-cluster \
| kubectl apply -f -
```

This uses the default template (public nodes, managed network). See [Cluster Templates](#cluster-templates) for other
network topologies.

Watch the cluster come up:

```bash
Expand All @@ -50,15 +54,40 @@ clusterctl describe cluster my-cluster

## Environment Variables

| Variable | Description | Example |
|-------------------------------------------|--------------------------------|-----------------------------------|
| `CLOUDSCALE_API_TOKEN` | cloudscale.ch API token | `abc123...` |
| `CLOUDSCALE_SSH_PUBLIC_KEY` | SSH public key added to nodes | `ssh-ed25519 AAAA...` |
| `CLOUDSCALE_REGION` | cloudscale.ch region | `lpg` or `rma` |
| `CLOUDSCALE_MACHINE_IMAGE` | Server image for nodes | `custom:ubuntu-2404-kube-v1.xx.x` |
| `CLOUDSCALE_CONTROL_PLANE_MACHINE_FLAVOR` | Flavor for control plane nodes | `flex-4-2` |
| `CLOUDSCALE_WORKER_MACHINE_FLAVOR` | Flavor for worker nodes | `flex-4-2` |
| `CLOUDSCALE_ROOT_VOLUME_SIZE` | Root volume size in GB | `50` |
| Variable | Description | Example |
|-------------------------------------------|-------------------------------------------|-----------------------------------|
| `CLOUDSCALE_API_TOKEN` | cloudscale.ch API token | `abc123...` |
| `CLOUDSCALE_SSH_PUBLIC_KEY` | SSH public key added to nodes | `ssh-ed25519 AAAA...` |
| `CLOUDSCALE_REGION` | cloudscale.ch region | `lpg` or `rma` |
| `CLOUDSCALE_MACHINE_IMAGE` | Server image for nodes | `custom:ubuntu-2404-kube-v1.xx.x` |
| `CLOUDSCALE_CONTROL_PLANE_MACHINE_FLAVOR` | Flavor for control plane nodes | `flex-4-2` |
| `CLOUDSCALE_WORKER_MACHINE_FLAVOR` | Flavor for worker nodes | `flex-4-2` |
| `CLOUDSCALE_ROOT_VOLUME_SIZE` | Root volume size in GB | `50` |
| `CLOUDSCALE_NETWORK_UUID` | Existing cloudscale.ch network UUID (BYO) | `2db69ba3-...` |

> **Note:** `CLOUDSCALE_NETWORK_UUID` is required by the `fip`, `public-lb-private-nodes`, and `byo-network`
> template flavors. It is not needed for the default template.

## Cluster Templates

CAPCS ships several cluster templates for different network topologies. Use `clusterctl generate cluster` with the
`--flavor` flag to select one:

```bash
clusterctl generate cluster my-cluster \
--kubernetes-version v1.32.0 \
--control-plane-machine-count 1 \
--worker-machine-count 2 \
--flavor <flavor-name> \
| kubectl apply -f -
```

| Flavor | Network | CP Endpoint | Node Connectivity | Extra Env Vars | Notes |
|---------------------------|---------------------------|-----------------------|-------------------|---------------------------|----------------------|
| *(default)* | Managed (`10.100.0.0/24`) | Public LB (DualStack) | Public + cluster | — | |
| `fip` | BYO | Floating IP (IPv4) | Public + cluster | `CLOUDSCALE_NETWORK_UUID` | |
| `public-lb-private-nodes` | BYO + NAT | Public LB | Private only | `CLOUDSCALE_NETWORK_UUID` | Requires NAT gateway |
| `byo-network` | BYO | Public LB (DualStack) | Public + cluster | `CLOUDSCALE_NETWORK_UUID` | |

## Development

Expand Down Expand Up @@ -92,14 +121,16 @@ filtering and are split into suites of increasing cost, scheduled accordingly:
| Cluster upgrade | `upgrade` | Rolling K8s version upgrade (v1.34 → v1.35) | < 10 min | Weekly | `test-e2e-upgrade` |
| Self-hosted | `self-hosted` | clusterctl move (pivot) to workload cluster. Requires container image in public registry | < 15 min | Weekly | `test-e2e-self-hosted` |
| MD remediation | `md-remediation` | MachineHealthCheck auto-replacement of unhealthy workers | < 10 min | Weekly | `test-e2e-md-remediation` |
| BYO networking | `byo-networking` | BYO network: public-LB + private-nodes and floating-IP variants | < 10 min | Weekly | `test-e2e-byo-networking` |
| Conformance (fast) | `conformance` | K8s conformance, skip Serial tests | < 60 min | Weekly | `test-e2e-conformance-fast` |
| Conformance (full) | `conformance` | Full K8s conformance including Serial tests | < 120 min | Biweekly | `test-e2e-conformance` |

Durations are approximate from a real CI run; conformance varies with cluster size.

**Why this split?** The single-CP lifecycle test is the cheapest smoke test and runs
nightly to catch regressions early. HA, upgrade, self-hosted, and remediation tests are more
resource-intensive and run weekly. Full K8s conformance is the most expensive and runs biweekly
resource-intensive and run weekly. Private networking tests require `CLOUDSCALE_NETWORK_UUID` to be set and are
skipped otherwise. Full K8s conformance is the most expensive and runs biweekly
(1st + 15th of month). All suites can be triggered manually via the `test-e2e.yml` workflow
dispatch. E2E tests share a concurrency group so only one suite runs at a time.

Expand Down Expand Up @@ -143,6 +174,8 @@ kustomize_substitutions:
CLOUDSCALE_WORKER_MACHINE_FLAVOR: "flex-4-2"
CLOUDSCALE_MACHINE_IMAGE: "IMAGE_NAME"
CLOUDSCALE_ROOT_VOLUME_SIZE: "50"
# Required for BYO network flavors (fip, public-lb-private-nodes, byo-network):
# CLOUDSCALE_NETWORK_UUID: "UUID_HERE"
extra_args:
cloudscale:
- "--zap-log-level=5"
Expand Down
Loading
Loading