monit fails on 1.1183: monit wrapper confused with cgroupv2 originated from bosh release workload

[gberche-orange]

Observed symtom

When running bosh-stemcell-1.1183-vsphere-esxi-ubuntu-jammy-go_agent.tgz along with bosh release which manipulates cgroupsv2 entries (https://github.com/orange-cloudfoundry/k3s-wrapper-boshrelease ) we see unresponsive agent issue

Interactive use of monit fails

server/11aa3763-7a95-47ad-8089-7399a97a2d0d:~# monit summary
mkdir: cannot create directory ‘/sys/fs/cgroup/unified\n’: Read-only file system

It seems that the monit wrapper in 1.1183 with #563 is yet not robust to additional cgroupv2 fs mounts
https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blame/ubuntu-jammy/v1.1183/stemcell_builder/stages/bosh_monit/assets/monit-access-helper.sh#L17-L20

Analysis

Analysis from @ogrand : permit_monit_access() fails in face of additional cgroupv2 fsmounts entries present in the host as a result of the bosh release (cilium in our case)

https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/v1.1183/stemcell_builder/stages/bosh_monit/assets/monit-access-helper.sh#L17-L20

        # cgroupv2 (unified hierarchy)
        # Create a sub-cgroup under the current process's cgroup and move into it.
        # The iptables rules match on this cgroup path.
        cgroup_mount="$(awk '$3 == "cgroup2" { print $2 }' /proc/self/mounts)"

Comparing the output of this command across bosh deployments

On a deployment not manipulating cgroupv2 fs type mounts

    dns-recursor/1cf11aec-d00e-4121-b406-8fca86240e7e:~# cgroup_mount="$(awk '$3 == "cgroup2" { print $2 }' /proc/self/mounts)" ; echo "${cgroup_mount}"
    /sys/fs/cgroup/unified

On a k3s deployment with cilium workload, two lines (mount points) are returned instead, which breaks the remaining of the script logic

server/11aa3763-7a95-47ad-8089-7399a97a2d0d:~# cgroup_mount="$(awk '$3 == "cgroup2" { print $2 }' /proc/self/mounts)" ; echo "${cgroup_mount}"
    /sys/fs/cgroup/unified
    /run/cilium/cgroupv2

Annotated version of the fs mounts matching the grep command

    server/11aa3763-7a95-47ad-8089-7399a97a2d0d:~# cat /proc/self/mounts | grep cgroup2
     # device  mount_point         fs_type dummy
    cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
    none    /run/cilium/cgroupv2   cgroup2 rw,relatime 0 0

Details on cilium cgroupv2 fs mount

https://github.com/cilium/cilium/blob/e53e1341f483c722bdaaf56a72cc01334d8aaac2/Documentation/network/kubernetes/kubeproxy-free.rst#L114

Cilium will automatically mount cgroup v2 filesystem required to attach BPF cgroup programs by default at the path /run/cilium/cgroupv2. To do that, it needs to mount the host /proc inside an init container launched by the DaemonSet temporarily.

background on /proc/self/mounts and its format

https://manpages.ubuntu.com/manpages/noble/man5/proc_pid_mounts.5.html
/proc/mounts

Before Linux 2.4.19, this file was a list of all the filesystems currently mounted on the system. With the introduction of per-process mount namespaces in Linux 2.4.19 (see [mount_namespaces(7)](https://manpages.ubuntu.com/manpages/noble/man7/mount_namespaces.7.html)), this file became a link to /proc/self/mounts, which lists the mounts of the process's own mount namespace. The format of this file is documented in [fstab(5)](https://manpages.ubuntu.com/manpages/noble/man5/fstab.5.html).

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/4/html/reference_guide/s2-proc-mounts

the first column specifies the device that is mounted, the second column reveals the mount point, and the third column tells the file system type, and the fourth column tells you if it is mounted read-only (ro) or read-write (rw). The fifth and sixth columns are dummy values designed to match the format used in /etc/mtab.

https://man7.org/linux/man-pages/man7/cgroups.7.html

Note that on many modern systems, systemd(1) automatically mounts
the cgroup2 filesystem at /sys/fs/cgroup/unified during the boot
process.

Workaround

We're currently testing a (ugly) workaround to patch the monit script (using the https://github.com/orange-cloudfoundry/generic-scripting-release/)
from
cgroup_mount="$(awk '$3 == "cgroup2" { print $2 }' /proc/self/mounts)"
to
cgroup_mount="$(awk '$1 == "cgroup2" && $3 == "cgroup2" { print $2 }' /proc/self/mounts)"

using

sed -i 's:cgroup_mount=.*:cgroup_mount="$(awk '\''\$1 == "cgroup2" \&\& $3 == "cgroup2" { print $2 }'\'' \/proc\/self\/mounts)":g' /var/vcap/bosh/etc/monit-access-helper.sh

[gberche-orange]

@mkocher when testing stemcell 1.193 in our system running cilium (which creates cgroupv2 fs mounts with no device), I see that the branch monit_using_unified_cgroup_v2 is returning false, and hence following the cgroup v1 branch

bosh-linux-stemcell-builder/stemcell_builder/stages/bosh_monit/assets/monit-access-helper.sh

Lines 22 to 51 in 16ec166

	monit_using_unified_cgroup_v2() {
	[ -f /sys/fs/cgroup/cgroup.controllers ] && return 0
	[ "$(stat -fc %T /sys/fs/cgroup 2>/dev/null)" = "cgroup2fs" ]
	}
	permit_monit_access() {
	if monit_using_unified_cgroup_v2; then
	# cgroupv2 (unified hierarchy)
	# Create a sub-cgroup under the current process's cgroup and move into it.
	# The iptables rules match on this cgroup path.
	cgroup_mount="$(awk '$3 == "cgroup2" { print $2 }' /proc/self/mounts)"
	current_cgroup="$(grep '^0::' /proc/self/cgroup | cut -d: -f3)"
	if [ -z "${cgroup_mount}" ] || [ -z "${current_cgroup}" ]; then
	echo "permit_monit_access: unable to resolve cgroup v2 mount or path" >&2
	return 1
	fi
	monit_access_cgroup="${cgroup_mount}${current_cgroup}/monit-api-access"
	mkdir -p "${monit_access_cgroup}"
	echo $$ > "${monit_access_cgroup}/cgroup.procs"
	else
	# this seems to work in docker but net_cls_location is empty in garden
	net_cls_location="$(cat /proc/self/mounts | grep ^cgroup | grep net_cls | awk '{ print $2 }' )"
	net_cls_subproc="$(grep net_cls /proc/self/cgroup | awk -F ":" '{ print $3 }' )"
	monit_access_cgroup="${net_cls_location}/${net_cls_subproc}/monit-api-access"
	mkdir -p "${monit_access_cgroup}"
	echo "${monit_isolation_classid}" > "${monit_access_cgroup}/net_cls.classid"
	echo $$ > "${monit_access_cgroup}/tasks"
	fi
	}

Details

While preparing PR on jammy head using https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/stemcell_builder/stages/bosh_monit/assets/monit-access-helper.sh

bosh-linux-stemcell-builder/stemcell_builder/stages/bosh_monit/assets/monit-access-helper.sh

Lines 1 to 51 in 2d3933d

	# This is the integer value of the argument "0xb0540001", which is
	# b054:0001 . The major number (the left-hand side) is "BOSH", leet-ified.
	# The minor number (the right-hand side) is 1, indicating that this is the
	# first thing in our "BOSH" classid namespace.
	#
	# _Hopefully_ noone uses a major number of "b054", and we avoid collisions _forever_!
	# If you need to select new classids for firewall rules or traffic control rules, keep
	# the major number "b054" for bosh stuff, unless there's a good reason to not.
	#
	# The net_cls.classid structure is described in more detail here:
	# https://www.kernel.org/doc/Documentation/cgroup-v1/net_cls.txt
	monit_isolation_classid=2958295041
	# True when /sys/fs/cgroup is the root of a cgroup2 mount (unified hierarchy).
	# Do not use /proc/self/cgroup's "0::" entry alone: under systemd hybrid mode a
	# 0:: line can refer to the small cgroup2 tracking hierarchy while resource
	# controllers (including net_cls) remain on cgroup v1.
	#
	# Prefer cgroup.controllers; also accept stat(2) filesystem type for hosts where
	# the file is missing from the mount view but the root is still cgroup2fs.
	monit_using_unified_cgroup_v2() {
	[ -f /sys/fs/cgroup/cgroup.controllers ] && return 0
	[ "$(stat -fc %T /sys/fs/cgroup 2>/dev/null)" = "cgroup2fs" ]
	}
	permit_monit_access() {
	if monit_using_unified_cgroup_v2; then
	# cgroupv2 (unified hierarchy)
	# Create a sub-cgroup under the current process's cgroup and move into it.
	# The iptables rules match on this cgroup path.
	cgroup_mount="$(awk '$3 == "cgroup2" { print $2 }' /proc/self/mounts)"
	current_cgroup="$(grep '^0::' /proc/self/cgroup | cut -d: -f3)"
	if [ -z "${cgroup_mount}" ] || [ -z "${current_cgroup}" ]; then
	echo "permit_monit_access: unable to resolve cgroup v2 mount or path" >&2
	return 1
	fi
	monit_access_cgroup="${cgroup_mount}${current_cgroup}/monit-api-access"
	mkdir -p "${monit_access_cgroup}"
	echo $$ > "${monit_access_cgroup}/cgroup.procs"
	else
	# this seems to work in docker but net_cls_location is empty in garden
	net_cls_location="$(cat /proc/self/mounts | grep ^cgroup | grep net_cls | awk '{ print $2 }' )"
	net_cls_subproc="$(grep net_cls /proc/self/cgroup | awk -F ":" '{ print $3 }' )"
	monit_access_cgroup="${net_cls_location}/${net_cls_subproc}/monit-api-access"
	mkdir -p "${monit_access_cgroup}"
	echo "${monit_isolation_classid}" > "${monit_access_cgroup}/net_cls.classid"
	echo $$ > "${monit_access_cgroup}/tasks"
	fi
	}

monit summary problem does not appear anymore following fix 5c6f3a5

instrumenting /var/vcap/bosh/bin/monit with tracing

#!/bin/bash

set -e
set -x
source /var/vcap/bosh/etc/monit-access-helper.sh
permit_monit_access
set +x
exec /var/vcap/bosh/bin/monit-actual "$@"

and reproducing on core-connectivity server/0 instance group on which the problem was detected

server/11aa3763-7a95-47ad-8089-7399a97a2d0d:~# monit summary                                
+ source /var/vcap/bosh/etc/monit-access-helper.sh
++ monit_isolation_classid=2958295041
+ permit_monit_access
+ monit_using_unified_cgroup_v2          
+ '[' -f /sys/fs/cgroup/cgroup.controllers ']' 
++ stat -fc %T /sys/fs/cgroup
+ '[' tmpfs = cgroup2fs ']'
++ cat /proc/self/mounts
++ grep '^cgroup'
++ grep net_cls
++ awk '{ print $2 }'
+ net_cls_location=/sys/fs/cgroup/net_cls,net_prio
++ grep net_cls /proc/self/cgroup
++ awk -F : '{ print $3 }'
+ net_cls_subproc=/
+ monit_access_cgroup=/sys/fs/cgroup/net_cls,net_prio///monit-api-access
+ mkdir -p /sys/fs/cgroup/net_cls,net_prio///monit-api-access
+ echo 2958295041
+ echo 146676
+ set +x
The Monit daemon 5.2.5 uptime: 7d 20h 27m 

The monit_using_unified_cgroup_v2 is returning false, and the cgroupv1 path is used instead.

monit_using_unified_cgroup_v2() {
    [ -f /sys/fs/cgroup/cgroup.controllers ] && return 0
    [ "$(stat -fc %T /sys/fs/cgroup 2>/dev/null)" = "cgroup2fs" ]
}

permit_monit_access() {
    if monit_using_unified_cgroup_v2; then
        # cgroupv2 (unified hierarchy)
        # Create a sub-cgroup under the current process's cgroup and move into it.
        # The iptables rules match on this cgroup path.
        cgroup_mount="$(awk '$3 == "cgroup2" { print $2 }' /proc/self/mounts)"
        current_cgroup="$(grep '^0::' /proc/self/cgroup | cut -d: -f3)"
...
else
        # this seems to work in docker but net_cls_location is empty in garden
        net_cls_location="$(cat /proc/self/mounts | grep ^cgroup | grep net_cls | awk '{ print $2 }' )"

Can you confirm that the function monit_using_unified_cgroup_v2 intends to detect whether the system is configured to use cgroupsv2, and is unrelated to configuration done by bosh agent ?

I've prepared #599 to harden then vcgroupv2 branch to avoid failing on unrelated additional cgroupsv2. This might require further refinements