Easily import a dynamically created module

Python modules are typically Python source files that are used like simple namespaces for variables. Take this simple module:

mod.py:

x = "Hi, I'm a module variable."

Provided mod.py is on the import path (which usually includes the current working directory), you can run Python and see:

>>> import mod
>>> print mod.x
"Hi, I'm a module variable."

Of course, this is Python and Python modules are objects of type 'module'. You can dynamically create an object of this type easily enough:

>>> from types import ModuleType
>>> mymodule = ModuleType('mymodule')
>>> mymodule.x = "Hi, I'm in a dynamically-created module."
>>> print mymodule.x
"Hi, I'm in a dynamically-created module."

Pretty much what you'd expect. ModuleType is the same thing you'd get back from import sys; type(sys). That's actually what the types module uses.

But, you can't import mymodule because it's not a real, source-based module on the import path. If you want other Pythonauts to be able to use standard import syntax to import your dynamic module, a little more trickery is involved.

dynmod.py:

import sys
from types import ModuleType

not_present = "Hi, I was masked by the dynamic module."

dm = ModuleType(__name__)
dm.x = 5

sys.modules[__name__] = dm

If you're doing this you probably already know, but __name__ is just a string containing the name of the current module, based on the filename ('dynmod'). Using __name__ just keeps things a bit more consistent.

Now, look what happens when you import dynmod:

>>> import dynmod
>>> dynmod.x
5
>>> dynmod.not_present
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'module' object has no attribute 'not_present'
>>> dir(dynmod)
['__doc__', '__name__', 'x']

All the normal import syntax should work, including from dynmod import x, even though the module isn't strictly what's in the source file.

The dynamic behavior is only executed once, on initial import. When imported, Python modules create a name->module_object entry in the sys.modules dictionary. From then on, if the module name is found in sys.modules, the associated module object is used. That's how normal modules work, and that's how we want our dynamic module to work, too.

The key here is that the sys.modules entry seems to be created before the module code is run, allowing the module code to modify its own entry. Be careful with this, because Python's import system has some complex locking behavior that can make certain manipulations not work:

dynmod_broken.py:

# Breaks some import locks or something
import sys
from types import ModuleType

not_present = "masked"

sys.modules[__name__]   = ModuleType(__name__)
sys.modules[__name__].x = 5

(In fact, according to my testing, this left every single variable in the namespace set to None. Pretty PDW, if you ask me.)

If that's not interesting enough for you, then stay tuned for the follow-up post on more advanced dynamic modules where we show you how to use modules' simple and familiar nature to do your darkest biddings.

(Note: if you're looking for the right way to simply mask certain variables, consider using __all__. Also, you could probably do more fancy things with Import Hooks, but this is tested to work on Python 2.7 and 3.1, so maybe keep it simple, if simple's all you need.)