Python 3.x and packaging issues

Greetings to ...

We will create a very simple Python 3.4 command line greeting app. It will take two arguments. An user name and an effect (if any) that should be applied to the user name. Then it will print out a greeting message. That's all.

While implementing and testing, we will run into and solve imports and packaging issues that may occur when someone used to Python 2.x is creating, testing and distributing a Python 3.x application containing a sub-package.

We will need

Good Python applications obeys standard layout. The code-base should be divided into modules and packages. For more information regarding structuring your apps please refer to Structuring Your Project section of the The Hitchhiker’s Guide to Python!.

Our app tree looks like this:

greeter/
└── greeter
    ├── effects
    │   ├── dashed.py
    │   └── __init__.py
    ├── app.py
    └── __init__.py

So we have a greeter/ parent directory with greeter package inside it. This package contains the main app.py module and an effects sub-package with a dashed.py module.

Now we will implement the actual code necessary for this app. All __init__.py files remains empty.
Here is the code for effects.dashed.py module:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-


def add_dashes(string):
    """
    Add dashes between string characters

    :argument string: target string
    :type string: str

    :return str

    """
    return '-'.join(string)

And this is the app.py code:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-


import argparse

from effects.dashed import add_dashes


#: constants
DASHED = 'dashed'

#: effects mapping
effects = {DASHED: add_dashes}


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
                        description='A simple CLI greeter',
                        usage=('python3 app.py  [effect]'))

    parser.add_argument('-n', '--name', required=True, help='User name')
    parser.add_argument('-e', '--effect', choices=[DASHED],
                        help='Effect that will be applied to user name')

    args = parser.parse_args()
    name = args.name.strip()

    if args.effect is not None:
        effect = effects[args.effect]
        name = effect(name)

    print('Greetings {name}!'.format(name=name))

Now you can use it from the command line:

dm@Z580:~/workspace/venv/greeter/greeter$ python3 app.py -n John
Greetings John!

Or with dashed effect:

dm@Z580:~/workspace/venv/greeter/greeter$ python3 app.py -n John -e dashed
Greetings J-o-h-n!

We are satisfied with out greeter app and want to share it with others.

Packaging

To be able to distribute a package we need to add at least these two files (for more details please refer to Packaging and Distributing Projects) to the parent directory: setup.py and README.rst. After this our app tree will look like this:

greeter/
├── greeter
│   ├── effects
│   │   ├── dashed.py
│   │   └── __init__.py
│   ├── app.py
│   └── __init__.py
├── README.rst
└── setup.py

README.rst content:

Greeter
=======

A simple CLI greeter.

setup.py content:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-


from setuptools import setup


"""Greeter package installer"""


setup(
    name='greeter',
    version='0.1.0',
    keywords='greet user',
    description='A simple CLI greeter',
    packages=[
        'greeter',
        'greeter.effects'
    ],
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Developers',
        'Programming Language :: Python :: 3.4',
        'Operating System :: POSIX :: Linux',
        'Natural Language :: English',
    ]
)

Testing and fixing

Now we can try to install our package with pip (ideally in a virtual environment) and then use it from command line to test if everything is working as expected:

(venv) dm@Z580:~/workspace/venv/greeter$ python setup.py install
...
Installed /home/dm/workspace/venv/lib/python3.4/site-packages/greeter-0.1.0-py3.4.egg
Processing dependencies for greeter==0.1.0
Finished processing dependencies for greeter==0.1.0

(venv) dm@Z580:~/workspace/venv/greeter$ python
Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from greeter import app
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/dm/workspace/venv/greeter/greeter/app.py", line 7, in 
    from effects.dashed import add_dashes
ImportError: No module named 'effects'

Here we crashed into first problem. We can't import the greeter.effects package. The following line of code in app.py is causing this:

from effects.dashed import add_dashes

It can be fixed by a relative import which Python 3 requires inside packages.

from .effects.dashed import add_dashes  # notice the '.' before effects

After we apply this update and re-install our greeter package all seems well. At least in the Python interactive console. So the relative import saves the day. Or does it? What will happen when we try to run our app as a script ...

dm@Z580:~/workspace/venv/greeter/greeter$ python3 app.py -n John
Traceback (most recent call last):
  File "app.py", line 7, in 
    from .effects.dashed import add_dashes
SystemError: Parent module '' not loaded, cannot perform relative import

Here we have hit an issues that is caused by the relative import introduced by our previous update. For more information please refer to PEP 0366 and this stackoverflow thread. It also proposes a fix. A rather hacky one, but it works.
So we need to change the app.py module again:

import os
import sys

parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(1, parent_dir)

from greeter.effects.dashed import add_dashes

Finally ...

Now everything is working as it should and the main app.py module looks like this after all updates:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-


import os
import sys
import argparse

parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(1, parent_dir)

from greeter.effects.dashed import add_dashes


#: constants
DASHED = 'dashed'

#: effects mapping
effects = {DASHED: add_dashes}


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
                        description='A simple CLI greeter',
                        usage=('python3 app.py  [effect]'))

    parser.add_argument('-n', '--name', required=True, help='User name')
    parser.add_argument('-e', '--effect', choices=[DASHED],
                        help='Effect that will be applied to user name')

    args = parser.parse_args()
    name = args.name.strip()

    if args.effect is not None:
        effect = effects[args.effect]
        name = effect(name)

    print('Greetings {name}!'.format(name=name))

If we rename the app.py module to __main__.py we would be able to run the program like this:

dm@Z580:~/workspace/venv/greeter$ python3 greeter -n John -e dashed
Greetings J-o-h-n!