PEP 489: Multi-phase extension module initialization

Known limitations of the current implementation:

- documentation changes are incomplete
- there's a reference leak I haven't tracked down yet

The leak is most visible by running:

  ./python -m test -R3:3 test_importlib

However, you can also see it by running:

  ./python -X showrefcount

Importing the array or _testmultiphase modules, and
then deleting them from both sys.modules and the local
namespace shows significant increases in the total
number of active references each cycle. By contrast,
with _testcapi (which continues to use single-phase
initialisation) the global refcounts stabilise after
a couple of cycles.

Author: ncoghlan

Doc/c-api/module.rst

@@ -7,8 +7,6 @@ Module Objects
 
 .. index:: object: module
 
-There are only a few functions special to module objects.
-
 
 .. c:var:: PyTypeObject PyModule_Type
 
@@ -109,6 +107,14 @@ There are only a few functions special to module objects.
       unencodable filenames, use :c:func:`PyModule_GetFilenameObject` instead.
 
 
+Per-interpreter module state
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Single-phase initialization creates singleton modules that can store additional
+information as part of the interpreter, allow that state to be retrieved later
+with only a reference to the module definition, rather than to the module
+itself.
+
 .. c:function:: void* PyModule_GetState(PyObject *module)
 
    Return the "state" of the module, that is, a pointer to the block of memory
@@ -146,27 +152,6 @@ There are only a few functions special to module objects.
 Initializing C modules
 ^^^^^^^^^^^^^^^^^^^^^^
 
-These functions are usually used in the module initialization function.
-
-.. c:function:: PyObject* PyModule_Create(PyModuleDef *module)
-
-   Create a new module object, given the definition in *module*.  This behaves
-   like :c:func:`PyModule_Create2` with *module_api_version* set to
-   :const:`PYTHON_API_VERSION`.
-
-
-.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version)
-
-   Create a new module object, given the definition in *module*, assuming the
-   API version *module_api_version*.  If that version does not match the version
-   of the running interpreter, a :exc:`RuntimeWarning` is emitted.
-
-   .. note::
-
-      Most uses of this function should be using :c:func:`PyModule_Create`
-      instead; only use this if you are sure you need it.
-
-
 .. c:type:: PyModuleDef
 
    This struct holds all information that is needed to create a module object.
@@ -210,9 +195,10 @@ These functions are usually used in the module initialization function.
       A pointer to a table of module-level functions, described by
       :c:type:`PyMethodDef` values.  Can be *NULL* if no functions are present.
 
-   .. c:member:: inquiry m_reload
+   .. c:member:: PyModuleDef_Slot* m_slots
 
-      Currently unused, should be *NULL*.
+      An array of slot definitions for multi-phase initialization, terminated by
+      a *NULL* entry.
 
    .. c:member:: traverseproc m_traverse
 
@@ -229,14 +215,68 @@ These functions are usually used in the module initialization function.
       A function to call during deallocation of the module object, or *NULL* if
       not needed.
 
+The module initialization function may create and return the module object
+directly. This is referred to as "single-phase initialization", and uses one
+of the following two module creation functions:
+
+.. c:function:: PyObject* PyModule_Create(PyModuleDef *module)
+
+   Create a new module object, given the definition in *module*.  This behaves
+   like :c:func:`PyModule_Create2` with *module_api_version* set to
+   :const:`PYTHON_API_VERSION`.
+
+
+.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version)
+
+   Create a new module object, given the definition in *module*, assuming the
+   API version *module_api_version*.  If that version does not match the version
+   of the running interpreter, a :exc:`RuntimeWarning` is emitted.
+
+   .. note::
+
+      Most uses of this function should be using :c:func:`PyModule_Create`
+      instead; only use this if you are sure you need it.
+
+
+Alternatively, the module initialization function may instead return a
+:c:type:`PyModuleDef` instance with a non-empty ``m_slots`` array. This is
+referred to as "multi-phase initialization", and ``PyModuleDef`` instance
+should be initialized with the following function:
+
+.. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *module)
+
+   Ensures a module definition is a properly initialized Python object that
+   correctly reports its type and reference count.
+
+.. XXX (ncoghlan): It's not clear if it makes sense to document PyModule_ExecDef
+   PyModule_FromDefAndSpec or PyModule_FromDefAndSpec2 here, as end user code
+   generally shouldn't be calling those.
+
+The module initialization function (if using single phase initialization) or
+a function called from a module execution slot (if using multiphase
+initialization), can use the following functions to help initialize the module
+state:
+
+.. c:function:: int PyModule_SetDocString(PyObject *module, const char *docstring)
+
+   Set the docstring for *module* to *docstring*. Return ``-1`` on error, ``0``
+   on success.
+
+.. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions)
+
+   Add the functions from the ``NULL`` terminated *functions* array to *module*.
+   Refer to the :c:type:`PyMethodDef` documentation for details on individual
+   entries (due to the lack of a shared module namespace, module level
+   "functions" implemented in C typically receive the module as their first
+   parameter, making them similar to instance methods on Python classes).
+
 
 .. c:function:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value)
 
    Add an object to *module* as *name*.  This is a convenience function which can
    be used from the module's initialization function.  This steals a reference to
    *value*.  Return ``-1`` on error, ``0`` on success.
 
-
 .. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value)
 
    Add an integer constant to *module* as *name*.  This convenience function can be

Doc/library/importlib.rst

@@ -55,6 +55,12 @@ generically as an :term:`importer`) to participate in the import process.
     :pep:`451`
         A ModuleSpec Type for the Import System
 
+    :pep:`488`
+        Elimination of PYO files
+
+    :pep:`489`
+        Multi-phase extension module initialization
+
     :pep:`3120`
         Using UTF-8 as the Default Source Encoding
 
@@ -756,9 +762,9 @@ find and load modules.
     Only class methods are defined by this class to alleviate the need for
     instantiation.
 
-    .. note::
-       Due to limitations in the extension module C-API, for now
-       BuiltinImporter does not implement :meth:`Loader.exec_module`.
+    .. versionchanged:: 3.5
+       As part of :pep:`489`, the builtin importer now implements
+       :meth:`Loader.create_module` and :meth:`Loader.exec_module`
 
 
 .. class:: FrozenImporter
@@ -973,14 +979,18 @@ find and load modules.
 
       Path to the extension module.
 
-   .. method:: load_module(name=None)
+   .. method:: create_module(spec)
+
+      Creates the module object from the given specification in accordance
+      with :pep:`489`.
+
+      .. versionadded:: 3.5
+
+   .. method:: exec_module(module)
 
-      Loads the extension module if and only if *fullname* is the same as
-      :attr:`name` or is ``None``.
+      Initializes the given module object in accordance with :pep:`489`.
 
-      .. note::
-         Due to limitations in the extension module C-API, for now
-         ExtensionFileLoader does not implement :meth:`Loader.exec_module`.
+      .. versionadded:: 3.5
 
    .. method:: is_package(fullname)
 

Doc/whatsnew/3.5.rst

@@ -93,6 +93,7 @@ Implementation improvements:
   (:issue:`19977`).
 
 * :pep:`488`, the elimination of ``.pyo`` files.
+* :pep:`489`, multi-phase initialization of extension modules.
 
 Significantly Improved Library Modules:
 
@@ -265,6 +266,21 @@ updated API to help with this change.
    :pep:`488` -- Elimination of PYO files
 
 
+PEP 489: Multi-phase extension module initialization
+----------------------------------------------------
+
+:pep:`489` updates extension module initialization to take advantage of the
+two step module loading mechanism introduced by :pep:`451` in Python 3.4.
+
+This change brings the import semantics of extension modules that opt-in to
+using the new mechanism much closer to those of Python source and bytecode
+modules, including the ability to any valid identifier as a module name,
+rather than being restricted to ASCII.
+
+.. seealso::
+
+   :pep:`488` -- Multi-phase extension module initialization
+
 Other Language Changes
 ======================
 
@@ -667,7 +683,7 @@ time
 tkinter
 -------
 
-* The :module:`tkinter._fix` module used for setting up the Tcl/Tk environment
+* The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment
   on Windows has been replaced by a private function in the :module:`_tkinter`
   module which makes no permanent changes to environment variables.
   (Contributed by Zachary Ware in :issue:`20035`.)
@@ -1012,7 +1028,6 @@ Changes in the Python API
   program depends on patching the module level variable to capture the debug
   output, you will need to update it to capture sys.stderr instead.
 
-
 Changes in the C API
 --------------------
 

Include/modsupport.h

@@ -12,13 +12,13 @@ extern "C" {
 /* If PY_SSIZE_T_CLEAN is defined, each functions treats #-specifier
    to mean Py_ssize_t */
 #ifdef PY_SSIZE_T_CLEAN
-#define PyArg_Parse			_PyArg_Parse_SizeT
-#define PyArg_ParseTuple		_PyArg_ParseTuple_SizeT
-#define PyArg_ParseTupleAndKeywords	_PyArg_ParseTupleAndKeywords_SizeT
-#define PyArg_VaParse			_PyArg_VaParse_SizeT
-#define PyArg_VaParseTupleAndKeywords	_PyArg_VaParseTupleAndKeywords_SizeT
-#define Py_BuildValue			_Py_BuildValue_SizeT
-#define Py_VaBuildValue			_Py_VaBuildValue_SizeT
+#define PyArg_Parse                     _PyArg_Parse_SizeT
+#define PyArg_ParseTuple                _PyArg_ParseTuple_SizeT
+#define PyArg_ParseTupleAndKeywords     _PyArg_ParseTupleAndKeywords_SizeT
+#define PyArg_VaParse                   _PyArg_VaParse_SizeT
+#define PyArg_VaParseTupleAndKeywords   _PyArg_VaParseTupleAndKeywords_SizeT
+#define Py_BuildValue                   _Py_BuildValue_SizeT
+#define Py_VaBuildValue                 _Py_VaBuildValue_SizeT
 #else
 PyAPI_FUNC(PyObject *) _Py_VaBuildValue_SizeT(const char *, va_list);
 #endif
@@ -49,6 +49,9 @@ PyAPI_FUNC(int) PyModule_AddIntConstant(PyObject *, const char *, long);
 PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char *);
 #define PyModule_AddIntMacro(m, c) PyModule_AddIntConstant(m, #c, c)
 #define PyModule_AddStringMacro(m, c) PyModule_AddStringConstant(m, #c, c)
+PyAPI_FUNC(int) PyModule_SetDocString(PyObject *, const char *);
+PyAPI_FUNC(int) PyModule_AddFunctions(PyObject *, PyMethodDef *);
+PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def);
 
 #define Py_CLEANUP_SUPPORTED 0x20000
 
@@ -67,35 +70,35 @@ PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char
    Please add a line or two to the top of this log for each API
    version change:
 
-   22-Feb-2006  MvL	1013	PEP 353 - long indices for sequence lengths
+   22-Feb-2006  MvL     1013    PEP 353 - long indices for sequence lengths
 
-   19-Aug-2002  GvR	1012	Changes to string object struct for
-   				interning changes, saving 3 bytes.
+   19-Aug-2002  GvR     1012    Changes to string object struct for
+                                interning changes, saving 3 bytes.
 
-   17-Jul-2001	GvR	1011	Descr-branch, just to be on the safe side
+   17-Jul-2001  GvR     1011    Descr-branch, just to be on the safe side
 
    25-Jan-2001  FLD     1010    Parameters added to PyCode_New() and
                                 PyFrame_New(); Python 2.1a2
 
    14-Mar-2000  GvR     1009    Unicode API added
 
-   3-Jan-1999	GvR	1007	Decided to change back!  (Don't reuse 1008!)
+   3-Jan-1999   GvR     1007    Decided to change back!  (Don't reuse 1008!)
 
-   3-Dec-1998	GvR	1008	Python 1.5.2b1
+   3-Dec-1998   GvR     1008    Python 1.5.2b1
 
-   18-Jan-1997	GvR	1007	string interning and other speedups
+   18-Jan-1997  GvR     1007    string interning and other speedups
 
-   11-Oct-1996	GvR	renamed Py_Ellipses to Py_Ellipsis :-(
+   11-Oct-1996  GvR     renamed Py_Ellipses to Py_Ellipsis :-(
 
-   30-Jul-1996	GvR	Slice and ellipses syntax added
+   30-Jul-1996  GvR     Slice and ellipses syntax added
 
-   23-Jul-1996	GvR	For 1.4 -- better safe than sorry this time :-)
+   23-Jul-1996  GvR     For 1.4 -- better safe than sorry this time :-)
 
-   7-Nov-1995	GvR	Keyword arguments (should've been done at 1.3 :-( )
+   7-Nov-1995   GvR     Keyword arguments (should've been done at 1.3 :-( )
 
-   10-Jan-1995	GvR	Renamed globals to new naming scheme
+   10-Jan-1995  GvR     Renamed globals to new naming scheme
 
-   9-Jan-1995	GvR	Initial version (incompatible with older API)
+   9-Jan-1995   GvR     Initial version (incompatible with older API)
 */
 
 /* The PYTHON_ABI_VERSION is introduced in PEP 384. For the lifetime of
@@ -105,21 +108,34 @@ PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char
 #define PYTHON_ABI_STRING "3"
 
 #ifdef Py_TRACE_REFS
- /* When we are tracing reference counts, rename PyModule_Create2 so
+ /* When we are tracing reference counts, rename module creation functions so
     modules compiled with incompatible settings will generate a
     link-time error. */
  #define PyModule_Create2 PyModule_Create2TraceRefs
+ #define PyModule_FromDefAndSpec2 PyModule_FromDefAndSpec2TraceRefs
 #endif
 
 PyAPI_FUNC(PyObject *) PyModule_Create2(struct PyModuleDef*,
                                      int apiver);
 
 #ifdef Py_LIMITED_API
 #define PyModule_Create(module) \
-	PyModule_Create2(module, PYTHON_ABI_VERSION)
+        PyModule_Create2(module, PYTHON_ABI_VERSION)
 #else
 #define PyModule_Create(module) \
-	PyModule_Create2(module, PYTHON_API_VERSION)
+        PyModule_Create2(module, PYTHON_API_VERSION)
+#endif
+
+PyAPI_FUNC(PyObject *) PyModule_FromDefAndSpec2(PyModuleDef *def,
+                                                PyObject *spec,
+                                                int module_api_version);
+
+#ifdef Py_LIMITED_API
+#define PyModule_FromDefAndSpec(module, spec) \
+    PyModule_FromDefAndSpec2(module, spec, PYTHON_ABI_VERSION)
+#else
+#define PyModule_FromDefAndSpec(module, spec) \
+    PyModule_FromDefAndSpec2(module, spec, PYTHON_API_VERSION)
 #endif
 
 #ifndef Py_LIMITED_API

Include/moduleobject.h

@@ -30,6 +30,9 @@ PyAPI_FUNC(void) _PyModule_ClearDict(PyObject *);
 PyAPI_FUNC(struct PyModuleDef*) PyModule_GetDef(PyObject*);
 PyAPI_FUNC(void*) PyModule_GetState(PyObject*);
 
+PyAPI_FUNC(PyObject *) PyModuleDef_Init(struct PyModuleDef*);
+PyTypeObject PyModuleDef_Type;
+
 typedef struct PyModuleDef_Base {
   PyObject_HEAD
   PyObject* (*m_init)(void);
@@ -44,18 +47,29 @@ typedef struct PyModuleDef_Base {
     NULL, /* m_copy */          \
   }
 
+typedef struct PyModuleDef_Slot{
+    int slot;
+    void *value;
+} PyModuleDef_Slot;
+
 typedef struct PyModuleDef{
   PyModuleDef_Base m_base;
   const char* m_name;
   const char* m_doc;
   Py_ssize_t m_size;
   PyMethodDef *m_methods;
-  inquiry m_reload;
+  PyModuleDef_Slot* m_slots;
   traverseproc m_traverse;
   inquiry m_clear;
   freefunc m_free;
 }PyModuleDef;
 
+#define Py_mod_create 1
+#define Py_mod_exec 2
+
+#ifndef Py_LIMITED_API
+#define _Py_mod_LAST_SLOT 2
+#endif
 
 #ifdef __cplusplus
 }