test_import_loop_detection_self() fails since cpython 3.13.8 RecursionError change

[marc-hb]

(Since Python 3.13.8 commit python/cpython@ebccd1de88d,) test_import_loop_detection_self() fails when trying to run it with pytest-xdist 3.8 - even with a single process (-n 1)

complaint = "unsupported operand type(s) for +: 'int' and 'Reader'"

Reproduced with today's commit ed868f7.

This is worth noting because this is currently the only test failing with pytest-xdist. EDIT: xdist was only the messenger, merely offsetting the stack. See #919 for reproduction without xdist

I checked out the commit 16e2250 where this test was introduced + various others and that test was always incompatible with xdist 3.8.0 (warning: don't forget export WEST_SKIP_SLOW_TESTS=0)

To reproduce:

--- a/pyproject.toml
+++ b/pyproject.toml
@@ -45,6 +45,7 @@ test = [
     "coverage~=7.10",
     "pytest>=8.4,<10.0",
     "pytest-cov~=7.0",
+    "pytest-xdist", # not needed after all, see PR #919 instead
 ]
 types = [
     "mypy~=1.18",

EDIT: newer and much better stack trace in later comment below #908 (comment)

Details

``` uv run poe test -k import_loop_detect -n 1 # test passes without "-n 1"

<very long stack, most likely due to the import recursion being tested>

src/west/manifest.py:2300: in _import_path_from_self
self._import_pathobj_from_self(pathobj_abs, pathobj)
src/west/manifest.py:2327: in _import_pathobj_from_self
Manifest(topdir=self.topdir, internal_import_ctx=child_ctx)
src/west/manifest.py:1620: in init
self._load_validated()
src/west/manifest.py:2089: in _load_validated
self._load_self(manifest_data)
src/west/manifest.py:2212: in _load_self
self._import_from_self(imp)
src/west/manifest.py:2281: in _import_from_self
self._import_path_from_self(imp)
src/west/manifest.py:2300: in _import_path_from_self
self._import_pathobj_from_self(pathobj_abs, pathobj)
src/west/manifest.py:2327: in _import_pathobj_from_self
Manifest(topdir=self.topdir, internal_import_ctx=child_ctx)
src/west/manifest.py:1620: in init
self._load_validated()
src/west/manifest.py:2089: in _load_validated
self._load_self(manifest_data)
src/west/manifest.py:2212: in _load_self
self._import_from_self(imp)
src/west/manifest.py:2281: in _import_from_self
self._import_path_from_self(imp)
src/west/manifest.py:2300: in _import_path_from_self
self._import_pathobj_from_self(pathobj_abs, pathobj)
src/west/manifest.py:2327: in _import_pathobj_from_self
Manifest(topdir=self.topdir, internal_import_ctx=child_ctx)
src/west/manifest.py:1616: in init
self._malformed(te.args[0], parent=te)

---

self = <west.manifest.Manifest object at 0x7f8d1df93850>, complaint = "unsupported operand type(s) for +: 'int' and 'Reader'"
parent = TypeError("unsupported operand type(s) for +: 'int' and 'Reader'")

def _malformed(self, complaint: str, parent: Exception | None = None) -> NoReturn:
    context = f'file: {self._ctx.current_abspath} ' if self._ctx.current_abspath else 'data'
    args = [f'Malformed manifest {context}', f'Schema file: {_SCHEMA_PATH}']
    if complaint:
        args.append('Hint: ' + complaint)
    exc = MalformedManifest(*args)
    if parent:

      raise exc from parent

E west.manifest.MalformedManifest: ('Malformed manifest file: /tmp/pytest-/pytest-34/popen-gw0/test_import_loop_detection_sel0/topdir/mp/foo.yml ', 'Schema file: src/west/manifest-schema.yml',
"Hint: unsupported operand type(s) for +: 'int' and 'Reader'")

src/west/manifest.py:1975: MalformedManifest

</details>

[marc-hb]

This could be caused by some recent pytest-xdist changes? EDIT: this was a cpython 3.13.8 commit, see below. Because I remember all tests passing with xdist not long ago.

To reproduce with older west versions:

--- a/tox.ini
+++ b/tox.ini
@@ -19,6 +19,7 @@ ignore_missing_imports=True
 
 [testenv]
 deps =
+    pytest-xdist
     setuptools-scm
     pytest
     pytest-cov
@@ -27,6 +28,4 @@ deps =
 setenv =
     TOXTEMPDIR={envtmpdir}
 commands =
-  python -m flake8 --config={toxinidir}/tox.ini {toxinidir}
-  python -m mypy --config-file={toxinidir}/tox.ini --package=west
   python -m pytest --cov=west {posargs:tests}

tox -- -k import_loop_detect -n 1

[marc-hb]

There is some subtle dependency issue triggering this.

I can reproduce with both pytest -k loop -n 1 and uv run poe test -k loop -n 1 on Fedora 42 right now and on Arch Linux a few weeks ago but not with either command on Debian 12 Bookworm: both commands are desperately passing on Debian 12.

uv usually protects us from subtle dependency variations like this but not this time.

[marc-hb]

This could be caused by some recent pytest-xdist changes? Because I remember all tests passing with xdist not long ago.

The "regression" did not come xdist but from cpython. Reproduction requires Python version 3.13.8 or above. 3.13.7 and below work fine. Confirmed on both Debian 12 and Fedora 42.

# pass
uv run --python 3.13.7 poe test -k loop -n 1
# fail
uv run --python 3.13.8 poe test -k loop -n 1

Note this west loop detection relies on https://docs.python.org/3/library/exceptions.html#RecursionError

uv usually protects us from subtle dependency variations like this but not this time.

On the other hand, uv run --python ... made bisecting these Python versions incredibly easy!

[marc-hb]

xdist regression bisected to this cpython commit:
python/cpython@ebccd1de88d
python/cpython#132744: Check recursion limit in _PY_FRAME_GENERAL (python/cpython#132746) (python/cpython#138032)

[marc-hb]

I think this is a heisenbug of the worst kind introduced by cpython 3.13.8.

Better reproduction below.

• MERGED NOW get manifest.py: stop catching TypeError in validate() and obfuscating #917 if not merged yet . This is critical to reveal the actual, useful stack trace quoted below.

• OPTIONAL: make these small ruamel changes

pip install ruamel.yaml==0.19.1

--- a/yaml/reader.py
+++ b/yaml/reader.py
@@ -120,6 +120,12 @@ class Reader:
             self.determine_encoding()
 
     def peek(self, index: int = 0) -> Text:
+        if not isinstance(index, int):
+            # Unlike remote_pdb, trying to call this undefined function does expose the
+            # heisenbug. remote_pdb hides the heisenbug and the RecursionError test passes :-(
+            crash_here()
+#           from remote_pdb import RemotePdb
+#           RemotePdb('0.0.0.0', 4444).set_trace()
         try:
             return self.buffer[self.pointer + index]
         except IndexError:
diff --git a/yaml/scanner.py b/yaml/scanner.py
index 58a224e37ba3..f0f4a8bac2a9 100644
--- a/yaml/scanner.py
+++ b/yaml/scanner.py
@@ -1590,7 +1590,7 @@ class Scanner:
             length += 1
         whitespaces = self.reader.prefix(length)
         self.reader.forward(length)
-        ch = srp()
+        ch = srp()  #  no self.reader.peek() argument: use the default 0 int
         if ch in '\r\n\x85\u2028\u2029':
             line_break = self.scan_line_break()
             self.allow_simple_key = True

Incredible ruamel corruption: the default index argument changes type?!?

cd west/
pytest -k loop_detection -vvv -s -n 1 # without -n 1 the test passes

platform linux -- Python 3.13.11, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: ~/west
configfile: pyproject.toml
plugins: anyio-4.8.0, xdist-3.6.1
1 worker [1 item]      
scheduling tests via LoadScheduling

tests/test_manifest.py::test_import_loop_detection_self warning: re-init: ignored --initial-branch=master

.
.
# massive stack trace due to the infinite recursion
.
.
west/src/west/manifest.py:2340: in _import_pathobj_from_self
    Manifest(topdir=self.topdir, internal_import_ctx=child_ctx)
west/src/west/manifest.py:1622: in __init__
    self._load_validated()
west/src/west/manifest.py:2100: in _load_validated
    self._load_self(manifest_data)
west/src/west/manifest.py:2223: in _load_self
    self._import_from_self(imp)
west/src/west/manifest.py:2292: in _import_from_self
    self._import_path_from_self(imp)
west/src/west/manifest.py:2313: in _import_path_from_self
    self._import_pathobj_from_self(pathobj_abs, pathobj)
west/src/west/manifest.py:2340: in _import_pathobj_from_self
    Manifest(topdir=self.topdir, internal_import_ctx=child_ctx)
west/src/west/manifest.py:1611: in __init__
    self._ctx = self._ctx._replace(current_data=validate(self._ctx.current_data))
west/src/west/manifest.py:637: in validate
    pykwalify.core.Core(source_data=data, schema_files=[_SCHEMA_PATH]).validate()
/usr/lib/python3.13/site-packages/pykwalify/core.py:118: in __init__
    data = yml.load(stream)
.local/lib/python3.13/site-packages/ruamel/yaml/main.py:454: in load
    return constructor.get_single_data()
.local/lib/python3.13/site-packages/ruamel/yaml/constructor.py:117: in get_single_data
    node = self.composer.get_single_node()
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:77: in get_single_node
    document = self.compose_document()
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:100: in compose_document
    node = self.compose_node(None, None)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:144: in compose_node
    node = self.compose_mapping_node(anchor)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:227: in compose_mapping_node
    item_value = self.compose_node(node, item_key)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:144: in compose_node
    node = self.compose_mapping_node(anchor)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:227: in compose_mapping_node
    item_value = self.compose_node(node, item_key)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:144: in compose_node
    node = self.compose_mapping_node(anchor)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:227: in compose_mapping_node
    item_value = self.compose_node(node, item_key)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:142: in compose_node
    node = self.compose_sequence_node(anchor)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:188: in compose_sequence_node
    node.value.append(self.compose_node(node, index))
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:144: in compose_node
    node = self.compose_mapping_node(anchor)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:227: in compose_mapping_node
    item_value = self.compose_node(node, item_key)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:144: in compose_node
    node = self.compose_mapping_node(anchor)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:227: in compose_mapping_node
    item_value = self.compose_node(node, item_key)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:144: in compose_node
    node = self.compose_mapping_node(anchor)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:227: in compose_mapping_node
    item_value = self.compose_node(node, item_key)
.local/lib/python3.13/site-packages/ruamel/yaml/composer.py:111: in compose_node
    if self.parser.check_event(AliasEvent):
.local/lib/python3.13/site-packages/ruamel/yaml/parser.py:141: in check_event
    self.current_event = self.state()
.local/lib/python3.13/site-packages/ruamel/yaml/parser.py:631: in parse_block_mapping_value
    if self.scanner.check_token(ValueToken):
.local/lib/python3.13/site-packages/ruamel/yaml/scanner.py:170: in check_token
    self.fetch_more_tokens()
.local/lib/python3.13/site-packages/ruamel/yaml/scanner.py:310: in fetch_more_tokens
    return self.fetch_plain()
.local/lib/python3.13/site-packages/ruamel/yaml/scanner.py:715: in fetch_plain
    self.tokens.append(self.scan_plain())
.local/lib/python3.13/site-packages/ruamel/yaml/scanner.py:1554: in scan_plain
    spaces = self.scan_plain_spaces(indent, start_mark)
.local/lib/python3.13/site-packages/ruamel/yaml/scanner.py:1593: in scan_plain_spaces
    ch = srp()  #  no self.reader.peek() argument: use the default 0 int
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <ruamel.yaml.reader.Reader object at 0x7fae3f1b4d70>, index = <ruamel.yaml.reader.Reader object at 0x7fae3f1b4d70>

    def peek(self, index: int = 0) -> Text:
        if not isinstance(index, int):
            # Unlike remote_pdb, trying to call this undefined function does expose the
            # heisenbug. remote_pdb hides it and the RecursionError test passes :-(
>           crash_here()
E           NameError: name 'crash_here' is not defined

.local/lib/python3.13/site-packages/ruamel/yaml/reader.py:126: NameError
=============================================================================================== short test summary info ===============================================================================================
FAILED tests/test_manifest.py::test_import_loop_detection_self - NameError: name 'crash_here' is not defined
================================================================================================== 1 failed in 8.07s ==================================================================================================

Running with valgrind does not change the behavior and does not seem to find anything either:

valgrind --show-leak-kinds=none -s pytest -k loop -n 1

Replacing pkwalify with jsonschema thanks to #904 removes the dependency on ruamel and makes the problem go away: all tests pass with xdist.