test_ast: test_nodes_have_inline_values() fails when test_ast is run twice #161354

Activity

1. vstinner

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   testsTests in the Lib/test dirTests in the Lib/test dir

   on Nov 4, 2026
2. vstinner commented on Nov 4, 2026

   

   vstinner

   on Nov 4, 2026

   MemberAuthor

   More actions

   The regression was introduced by commit aa0e410 (PR gh-160616).

   commit aa0e4109a8681cbe6fb65e4072b0058e02adaef9
   Author: Lysandros Nikolaou <lisandrosnik@gmail.com>
   Date:   Sun Oct 25 10:08:26 2026 -0700
   
       gh-153568: Use a managed dict for AST nodes (#160616)
       
       AST nodes had an explicit `__dict__` slot, so every node built by
       `ast.parse` allocated its own dict. With `Py_TPFLAGS_MANAGED_DICT`,
       node attributes are stored as inline values with shared keys instead.
       
       `ast.parse` is ~15.8% faster, walking trees is ~15.9% faster, and
       the memory used by the AST trees is reduced by ~51.6%.
   
3. vstinner commented on Nov 5, 2026

   

   vstinner

   on Nov 5, 2026

   MemberAuthor

   More actions

   I can reproduce the failure with only two test methods and only run test_ast once:

   ./python -m test test_ast -m test_nodes_have_inline_values -m test_compare_literals
   
4. vstinner commented on Nov 5, 2026

   

   vstinner

   on Nov 5, 2026

   MemberAuthor

   More actions

   The ast.Constant(value=1, kind=None) object may or may not have inline values depending on _PyInlineValuesSize(type) (shared_keys_usable_size(keys)) of the AST type. Sometimes, the instance is allocated with a capacity of 23 values (when test_compare_literals() is not run), sometimes only with a capacity of 3 values. When the instance only has a capacity of 3 values, inline values can be invalidated (valid set to 0) when the dictionary is resized (when setting an attribute).

   To make the test more deterministic (not depend on other run tests), maybe it should be run in a subprocess, so we control better the internal state of AST types.

   ---

   • When test_compare_literals() is not run, PyDictValues.capacity is 23: in _PyInlineValuesSize(type), shared_keys_usable_size(keys) is 24.
   • When test_compare_literals() is run, PyDictValues.capacity is 3: in _PyInlineValuesSize(type), shared_keys_usable_size(keys) is 4.

   ---

   values->valid is set to 0 by invalidate_and_clear_inline_values().

   Call: ast2obj_expr() => ... => PyObject_SetAttr() => ... => store_instance_attr_lock_held() => ... => setitem_take2_lock_held_known_hash() => ... => dictresize() => invalidate_and_clear_inline_values().

   PyObject_SetAttr() is called with "col_offset" name. When ast2obj_expr() is called, the dictionary is equal to {'value': 1, 'kind': None, 'lineno': 1}.

   store_instance_attr_lock_held():

       if (ix == DKIX_EMPTY) {
           int res;
           if (dict == NULL) {
               // Make the dict but don't publish it in the object
               // so that no one else will see it.
               dict = make_dict_from_instance_attributes(keys, values);
               if (dict == NULL ||
                   _PyDict_SetItem_LockHeld(dict, name, value) < 0) { // <================= HERE
                   Py_XDECREF(dict);
                   return -1;

   dictresize():

   2279	        if (oldvalues->embedded) {
   2280	            assert(oldvalues->embedded == 1);
   2281	            assert(oldvalues->valid == 1);
   2282	            invalidate_and_clear_inline_values(oldvalues);

   gdb:

   #14 0x000000000083e3d5 in ast2obj_expr (state=state@entry=0xa390e0 <_PyRuntime+217856>, _o=0xf88ad8) at Python/Python-ast.c:6295
   6295	    if (PyObject_SetAttr(result, state->col_offset, value) < 0)
   (gdb) p result
   $4 = <Constant(value=1, kind=None, lineno=1) at remote 0x7fffe8fe8de0>
   
5. vstinner commented on Nov 5, 2026

   

   vstinner

   on Nov 5, 2026

   MemberAuthor

   More actions

   When state->Constant_type is created in init_types(), dk_usable is 30 and dk_nentries is 0:

   (gdb) p ((PyHeapTypeObject*)state->Constant_type)->ht_cached_keys->dk_usable
   $6 = 30
   (gdb) p ((PyHeapTypeObject*)state->Constant_type)->ht_cached_keys->dk_nentries
   $7 = 0
   

   When the first Constant instance is created:

   • type_call() calls indirectly _PyObject_InitInlineValues() which decrements dk_usable to 29.
   • Then, type_call() calls ast_type_init() which calls indirectly split_keys_entry_added(): set dk_usable to 28 and set dk_nentries is 1.

   test_compare_literals() creates many Constant() objects which decrements dk_usable to 29.

   When test_nodes_have_inline_values() is called, dk_usable=0 and dk_nentries=3.
6. vstinner

   added a commit that references this issue on Nov 5, 2026

   gh-161354: Fix test_ast.test_nodes_have_inline_values() (#161406)

   ...

   429fd0a
7. lysnikolaou commented on Nov 5, 2026

   

   lysnikolaou

   on Nov 5, 2026

   Member

   More actions

   I think we can close this now, right @vstinner?
8. vstinner commented on Nov 5, 2026

   

   vstinner

   on Nov 5, 2026

   Last edited by vstinner

   MemberAuthor

   More actions

   I always forget to close the issue. It's now closed!
9. vstinner

   closed this as completedon Nov 5, 2026