gh-161170: Lazily materialize coroutine objects for immediate awaits

Doc/c-api/frame.rst

@@ -85,6 +85,10 @@ See also :ref:`Reflection <reflection>`.
 
    .. versionadded:: 3.11
 
+   .. versionchanged:: 3.16
+      May return ``NULL`` for a coroutine frame before its first suspension;
+      see :attr:`frame.f_generator`.
+
 
 .. c:function:: PyObject* PyFrame_GetGlobals(PyFrameObject *frame)
 

Doc/library/dis.rst

@@ -806,7 +806,8 @@ not have to be) the original ``STACK[-2]``.
    Implements ``STACK[-1] = get_awaitable(STACK[-1])``, where ``get_awaitable(o)``
    returns ``o`` if ``o`` is a coroutine object or a generator object with
    the :data:`~inspect.CO_ITERABLE_COROUTINE` flag, or resolves
-   ``o.__await__``.
+   ``o.__await__``.  Then pushes ``NULL`` and ``None``, the initial stack
+   layout for the :opcode:`SEND` loop that follows.
 
     If the ``where`` operand is nonzero, it indicates where the instruction
     occurs:
@@ -819,6 +820,10 @@ not have to be) the original ``STACK[-2]``.
    .. versionchanged:: 3.11
       Previously, this instruction did not have an oparg.
 
+   .. versionchanged:: 3.16
+      Also pushes ``NULL`` and ``None``; previously the compiler emitted
+      :opcode:`PUSH_NULL` and :opcode:`LOAD_COMMON_CONSTANT` after it.
+
 
 .. opcode:: GET_AITER
 

Doc/library/inspect.rst

@@ -151,7 +151,9 @@ attributes (see :ref:`import-mod-attrs` for module attributes):
 |                 |                   | coroutine object that     |
 |                 |                   | owns this frame, or       |
 |                 |                   | ``None`` if the frame is  |
-|                 |                   | of a regular function     |
+|                 |                   | of a regular function or  |
+|                 |                   | the generator is not      |
+|                 |                   | materialized yet          |
 +-----------------+-------------------+---------------------------+
 |                 | f_trace           | tracing function for this |
 |                 |                   | frame, or ``None``        |

Doc/reference/datamodel.rst

@@ -1736,10 +1736,22 @@ Special read-only attributes
 
    * - .. attribute:: frame.f_generator
      - The :term:`generator` or :term:`coroutine` object that owns this frame,
-       or ``None`` if the frame is a normal function.
+       or ``None`` if the frame is a normal function or if the generator
+       is not materialized yet.
+
+       .. impl-detail::
+
+          A coroutine that is awaited as soon as it is called
+          (``await coro_func()``) may run without a coroutine object until
+          it first suspends, so this attribute is ``None`` before that.
+          Use :attr:`~codeobject.co_flags` of :attr:`frame.f_code` to tell
+          coroutine frames apart from normal function frames.
 
        .. versionadded:: 3.14
 
+       .. versionchanged:: 3.16
+          May be ``None`` for a coroutine frame before its first suspension.
+
 .. index::
    single: f_trace (frame attribute)
    single: f_trace_lines (frame attribute)

Doc/whatsnew/3.16.rst

@@ -1015,6 +1015,19 @@ datetime and time
   (Contributed by Paul Ganssle in :gh:`158252`, based on work by
   Stan Ulbrych.)
 
+Interpreter
+-----------
+
+* A coroutine that is awaited as soon as it is called
+  (``await coro_func()``) now runs without creating a coroutine object
+  unless it suspends, making awaits that complete immediately about 40%
+  faster in a micro-benchmark (``await noop()``: 69 ns to 41 ns on x86-64).
+  This does not apply while coroutine origin tracking is enabled or inside
+  code compiled by the experimental JIT.
+  Until it first suspends, :attr:`frame.f_generator` is ``None`` for such a
+  coroutine; see :ref:`whatsnew316-porting`.
+  (Contributed by Andrew Svetlov in :gh:`161170`.)
+
 re
 --
 
@@ -1233,12 +1246,23 @@ New deprecations
 .. include:: ../deprecations/pending-removal-in-future.rst
 
 
+.. _whatsnew316-porting:
+
 Porting to Python 3.16
 ======================
 
 This section lists previously described changes and other bugfixes
 that may require changes to your code.
 
+* :attr:`frame.f_generator` and :c:func:`PyFrame_GetGenerator` may now
+  return ``None``/``NULL`` for a coroutine frame before its first
+  suspension, because a coroutine that is awaited as soon as it is called
+  (``await coro_func()``) runs without a coroutine object until then.
+  Code that used ``frame.f_generator is None`` to detect frames of normal
+  functions should check ``frame.f_code.co_flags`` for
+  :data:`~inspect.CO_COROUTINE` instead.
+  (Contributed by Andrew Svetlov in :gh:`161170`.)
+
 * In :mod:`tkinter`, the *name* parameter of the
   :meth:`~tkinter.Misc.wait_variable`, :meth:`~tkinter.Misc.setvar` and
   :meth:`~tkinter.Misc.getvar` methods and the *value* parameter of
@@ -1294,6 +1318,15 @@ that may require changes to your code.
   (Contributed by An Long in :gh:`86768`.)
 
 
+CPython bytecode changes
+========================
+
+* :opcode:`GET_AWAITABLE` now also pushes the ``NULL`` and ``None`` that
+  start the :opcode:`SEND` loop following it, so the compiler no longer
+  emits :opcode:`PUSH_NULL` and :opcode:`LOAD_COMMON_CONSTANT` after it.
+  (Contributed by Andrew Svetlov in :gh:`161170`.)
+
+
 Build changes
 =============
 

Include/internal/pycore_ceval.h

@@ -267,6 +267,14 @@ extern int _PyEval_EnsureBuiltinsWithModule(
     PyObject **p_builtins);
 
 PyAPI_FUNC(PyObject *)_Py_MakeCoro(PyFunctionObject *func);
+// Lazily created coroutines (see RETURN_GENERATOR in bytecodes.c).
+// Exported for _testinternalcapi and external JIT support.
+PyAPI_FUNC(PyObject *) _PyCoro_MaterializeLazy(_PyInterpreterFrame *frame);
+PyAPI_FUNC(_Py_CODEUNIT *) _PyEval_LazyCoroSendInstr(PyThreadState *tstate,
+                                                     _PyInterpreterFrame *frame);
+PyAPI_FUNC(_PyInterpreterFrame *) _PyEval_MoveLazyCoroFrame(PyThreadState *tstate,
+                                                      PyGenObject *gen,
+                                                      _PyInterpreterFrame *frame);
 
 /* Handle signals, pending calls, GIL drop request
    and asynchronous exception */

Include/internal/pycore_interpframe.h

@@ -152,6 +152,8 @@ static inline void _PyFrame_Copy(_PyInterpreterFrame *src, _PyInterpreterFrame *
     dest->stackpointer = dest->localsplus + stacktop;
     // visited is GC bookkeeping for the current stack walk, not frame state.
     dest->visited = 0;
+    // A lazily started coroutine stops being lazy once its frame is copied.
+    dest->lazy_coro = 0;
 #ifdef Py_DEBUG
     dest->stackpointer_valid =  src->stackpointer_valid;
     dest->lltrace = src->lltrace;
@@ -208,6 +210,7 @@ _PyFrame_Initialize(
     frame->return_offset = 0;
     frame->owner = FRAME_OWNED_BY_THREAD;
     frame->visited = 0;
+    frame->lazy_coro = 0;
 #ifdef Py_DEBUG
     frame->stackpointer_valid = 1;
     frame->lltrace = 0;
@@ -441,6 +444,7 @@ _PyFrame_PushTrampolineUnchecked(PyThreadState *tstate, PyCodeObject *code, int
 #endif
     frame->owner = FRAME_OWNED_BY_THREAD;
     frame->visited = 0;
+    frame->lazy_coro = 0;
 #ifdef Py_DEBUG
     frame->stackpointer_valid = 1;
     frame->lltrace = 0;

Include/internal/pycore_interpframe_structs.h

@@ -44,9 +44,13 @@ struct _PyInterpreterFrame {
     char owner;
     uint8_t visited:1;
     uint8_t external:1; /* Set when the frame was pushed by something other than the VM */
+    /* Set for a thread-owned frame running a coroutine body whose coroutine
+     * object has not been created yet (see RETURN_GENERATOR in bytecodes.c).
+     * Its own _PyErr_StackItem lives on the data stack right after it. */
+    uint8_t lazy_coro:1;
 #ifdef Py_DEBUG
     uint8_t stackpointer_valid:1;
-    uint8_t lltrace:5;
+    uint8_t lltrace:4;
 #endif
     /* Locals and stack */
     _PyStackRef localsplus[1];

Include/internal/pycore_magic_number.h

@@ -304,6 +304,7 @@ Known values:
     Python 3.16a1 3704 (Replace DELETE_ATTR with PUSH_NULL; STORE_ATTR)
     Python 3.16a1 3705 (Add INTRINSIC_ADD_CONDITIONAL_ANNOTATION)
     Python 3.16a1 3706 (Add INTRINSIC_MATCH_CLASS_ISINSTANCE)
+    Python 3.16a1 3707 (GET_AWAITABLE pushes NULL and None for SEND)
 
     Python 3.17 will start with 3750
 
@@ -313,7 +314,7 @@ Known values:
 
 */
 
-#define PYC_MAGIC_NUMBER 3706
+#define PYC_MAGIC_NUMBER 3707
 /* This is equivalent to converting PYC_MAGIC_NUMBER to 2 bytes
    (little-endian) and then appending b'\r\n'. */
 #define PYC_MAGIC_NUMBER_TOKEN \

InternalDocs/generators.md

@@ -45,7 +45,21 @@ The generator's frame is initialized as a copy of the frame in which
 `RETURN_GENERATOR` is executing, but its `owner` field is overwritten to indicate
 that it is owned by a generator. Finally, `RETURN_GENERATOR` pushes the new generator
 object to the stack and returns to the caller of the generator function (at
-which time its frame is destroyed). When the generator is next resumed by
+which time its frame is destroyed).
+
+There is one exception: a coroutine whose caller is about to await it right
+away (`await f()`, i.e. the caller's next instruction is `GET_AWAITABLE`,
+which the compiler only emits directly before a `SEND` loop) keeps running in the frame
+in which `RETURN_GENERATOR` is executing, with the `lazy_coro` flag set and its
+own `_PyErr_StackItem` pushed on the data stack right after the frame.
+The caller's frame is set up as if `SEND_GEN` had pushed the coroutine frame
+(`[None, NULL]` on its stack, instruction pointer at the `SEND`), so that a body
+that returns or raises without suspending never creates a coroutine object.
+If the body reaches `YIELD_VALUE`, the coroutine object is created then and
+the frame is moved into it (see `_PyCoro_MaterializeLazy()` and
+`_PyEval_MoveLazyCoroFrame()`).
+
+When the generator is next resumed by
 [`gen_send_ex2()`](../Objects/genobject.c), `_PyEval_EvalFrame()` is called
 to continue executing the generator function, in the frame that is embedded in
 the generator object.

Lib/asyncio/graph.py

@@ -1,6 +1,7 @@
 """Introspection utils for tasks call graphs."""
 
 import dataclasses
+import inspect
 import io
 import sys
 import types
@@ -37,6 +38,26 @@ class FutureCallGraph:
     awaited_by: tuple["FutureCallGraph", ...]
 
 
+# A coroutine that is awaited right away runs without a coroutine
+# object until it first suspends, so ``f.f_generator`` may be None
+# for a coroutine frame; classify frames by their code instead.
+_CO_ASYNC_FLAGS = inspect.CO_COROUTINE | inspect.CO_ASYNC_GENERATOR
+_CO_GEN_FLAGS = _CO_ASYNC_FLAGS | inspect.CO_GENERATOR
+_FRAME_TAGS = {
+    inspect.CO_COROUTINE: 'async ',
+    inspect.CO_ASYNC_GENERATOR: 'async generator ',
+    inspect.CO_GENERATOR: 'generator ',
+}
+
+
+def _is_async_frame(f):
+    return bool(f.f_code.co_flags & _CO_ASYNC_FLAGS)
+
+
+def _is_sync_frame(f):
+    return not (f.f_code.co_flags & _CO_GEN_FLAGS)
+
+
 def _build_graph_for_future(
     future: futures.Future,
     *,
@@ -161,12 +182,10 @@ def capture_call_graph(
     try:
         while f is not None:
             # gh-156988: sync gen should not clear the call chain
-            is_async = isinstance(
-                f.f_generator, (types.CoroutineType, types.AsyncGeneratorType))
             call_stack.append(FrameCallGraphEntry(f))
 
-            if is_async:
-                if f.f_back is not None and f.f_back.f_generator is None:
+            if _is_async_frame(f):
+                if f.f_back is not None and _is_sync_frame(f.f_back):
                     # We've reached the bottom of the coroutine stack, which
                     # must be the Task that runs it.
                     break
@@ -221,36 +240,12 @@ def add_line(line: str) -> None:
             )
             for ste in st.call_stack:
                 f = ste.frame
-
-                if f.f_generator is None:
-                    f = ste.frame
-                    add_line(
-                        f'  |   File {f.f_code.co_filename!r},'
-                        f' line {f.f_lineno}, in'
-                        f' {f.f_code.co_qualname}()'
-                    )
-                else:
-                    c = f.f_generator
-
-                    try:
-                        f = c.cr_frame
-                        code = c.cr_code
-                        tag = 'async'
-                    except AttributeError:
-                        try:
-                            f = c.ag_frame
-                            code = c.ag_code
-                            tag = 'async generator'
-                        except AttributeError:
-                            f = c.gi_frame
-                            code = c.gi_code
-                            tag = 'generator'
-
-                    add_line(
-                        f'  |   File {f.f_code.co_filename!r},'
-                        f' line {f.f_lineno}, in'
-                        f' {tag} {code.co_qualname}()'
-                    )
+                tag = _FRAME_TAGS.get(f.f_code.co_flags & _CO_GEN_FLAGS, '')
+                add_line(
+                    f'  |   File {f.f_code.co_filename!r},'
+                    f' line {f.f_lineno}, in'
+                    f' {tag}{f.f_code.co_qualname}()'
+                )
 
         if st.awaited_by:
             add_line(