Standalone executables

Creating standalone executables is a commonly overlooked topic in materials that cover packaging of Python code. This is mainly because Python lacks proper tools in its standard library that could allow programmers to create simple executables that could be run by users without the need to install the Python interpreter.

Compiled languages have a big advantage over Python in that they allow creation of an executable application for the given system architecture that could be run by users in a way that does not require them to have any knowledge of the underlying technology. Python code, when distributed as a package, requires the Python interpreter in order to be run. This creates a big inconvenience for users who do not have enough technical proficiency.

Developer-friendly operating systems such as Mac OS X or most Linux distributions come with Python pre-installed. So, for their users, the Python-based application still could be distributed as a source package that relies on specific interpreter directive in the main script file, which is popularly called shebang. For most Python applications, this takes the following form:

#!/usr/bin/env python

Such directive, when used as a first line of script, will mark it to be interpreted by default by the Python version for the given environment. This can, of course, take a more detailed form, which requires a specific Python version such as python3.4, python3, or python2. Note that this will work in most popular POSIX systems, but isn't portable at all by definition. This solution relies on the existence of specific Python versions and also availability of env executable exactly at /usr/bin/env. Both of these assumptions may fail on some operating systems. Also, shebangs will not work on Windows at all. Additionally, bootstrapping of the Python environment on Windows can be a challenge even for experienced developers, so you cannot expect that nontechnical users will be able to do that by themselves.

The other thing to consider is the simple user experience in the desktop environment. Users usually expect that applications can be run from the desktop by simply clicking on them. Not every desktop environment will support that with Python applications distributed as a source.

Therefore, it would be best if we are able to create a binary distribution that would work as any other compiled executable. Fortunately, it is possible to create an executable that has both the Python interpreter and our project embedded. This allows users to open our application without caring about Python or any other dependency.

When are standalone executables useful?

Standalone executables are useful in situations where simplicity of user experience is more important than the user's ability to interfere with applications' code. Note that the fact that you are distributing application as executable only makes code reading or modification harder—not impossible. It is not a way to secure applications code and should only be used as a way to make interacting with an application simpler.

Standalone executables should be a preferred way of distributing applications for nontechnical end users and also seems to be the only reasonable way of distributing a Python application for Windows.

Standalone executables are usually a good choice for:

  • Applications that depend on specific Python versions that may not be easily available on the target operating systems
  • Applications that rely on modified precompiled CPython sources
  • Applications with graphical interfaces
  • Projects that have many binary extensions written in different languages
  • Games

Popular tools

Python does not have any built-in support for building standalone executables. Fortunately, there are some community projects solving that problem with varied success. The four most notable are:

  • PyInstaller
  • cx_Freeze
  • py2exe
  • py2app

Each one of them is slightly different in use and also each one of them has slightly different limitations. Before choosing your tool, you need to decide which platform you want to target, because every packaging tool can support only a specific set of operating systems.

The best case scenario is if you make such a decision at the very beginning of the project's life. None of these tools, of course, require deep interaction in your code, but if you start building standalone packages early, you can automate the whole process and save future integration time and costs. If you leave this for later, you may find yourself in a situation where the project is built in such a sophisticated way that none of the available tools will work. Providing a standalone executable for such a project will be problematic and will take a lot of your time.

PyInstaller

PyInstaller (http://www.pyinstaller.org/) is by far the most advanced program to freeze Python packages into standalone executables. It provides the most extensive multiplatform compatibility among every available solution at the moment, so it is the most recommended one. Platforms that PyInstaller supports are:

  • Windows (32-bit and 64-bit)
  • Linux (32-bit and 64-bit)
  • Mac OS X (32-bit and 64-bit)
  • FreeBSD, Solaris, and AIX

Supported versions of Python are Python 2.7 and Python 3.3, 3.4, and 3.5. It is available on PyPI, so it can be installed in your working environment using pip. If you have problems installing it this way, you can always download the installer from the project's page.

Unfortunately, cross-platform building (cross-compilation) is not supported so if you want to build your standalone executable for a specific platform, then you need to perform building on that platform. This is not a big trouble today with the advent of many virtualization tools. If you don't have a specific system installed on your computer, you can always use Vagrant that will provide you with the desired operating system as a virtual machine.

Usage for simple applications is easy. Let's assume our application is contained in the script named myscript.py. This is a simple "Hello world!" application. We want to create a standalone executable for Windows users and we had our sources located under D://dev/app in the filesystem. Our application can be bundled with the following short command:

$ pyinstaller myscript.py

2121 INFO: PyInstaller: 3.1
2121 INFO: Python: 2.7.10
2121 INFO: Platform: Windows-7-6.1.7601-SP1
2121 INFO: wrote D:devappmyscript.spec
2137 INFO: UPX is not available.
2138 INFO: Extending PYTHONPATH with paths
['D:\dev\app', 'D:\dev\app']
2138 INFO: checking Analysis
2138 INFO: Building Analysis because out00-Analysis.toc is non existent
2138 INFO: Initializing module dependency graph...
2154 INFO: Initializing module graph hooks...
2325 INFO: running Analysis out00-Analysis.toc
(...)
25884 INFO: Updating resource type 24 name 2 language 1033

PyInstaller's standard output is quite long even for simple applications, so it was truncated in the preceding example for the sake of brevity. If run on Windows, the resulting structure of directories and files will be as follows:

$ tree /0066
│   myscript.py
│   myscript.spec

├───build
│   └───myscript
│           myscript.exe
│           myscript.exe.manifest
│           out00-Analysis.toc
│           out00-COLLECT.toc
│           out00-EXE.toc
│           out00-PKG.pkg
│           out00-PKG.toc
│           out00-PYZ.pyz
│           out00-PYZ.toc
│           warnmyscript.txt

└───dist
    └───myscript
            bz2.pyd
            Microsoft.VC90.CRT.manifest
            msvcm90.dll
            msvcp90.dll
            msvcr90.dll
            myscript.exe
            myscript.exe.manifest
            python27.dll
            select.pyd
            unicodedata.pyd
            _hashlib.pyd

The dist/myscript directory contains the built application that can now be distributed to the users. Note that the whole directory must be distributed. It contains all additional files that are required to run our application (DLLs, compiled extension libraries, and so on). A more compact distribution can be obtained with the --onefile switch of the pyinstaller command:

$ pyinstaller --onefile myscript.py
(...)
$ tree /f
├───build
│   └───myscript
│           myscript.exe.manifest
│           out00-Analysis.toc
│           out00-EXE.toc
│           out00-PKG.pkg
│           out00-PKG.toc
│           out00-PYZ.pyz
│           out00-PYZ.toc
│           warnmyscript.txt

└───dist
        myscript.exe

When built with the --onefile option, the only file you need to distribute to other users is the single executable found in the dist directory (here, myscript.exe). For small applications, this is probably the preferred option.

One of the side effects of running the pyinstaller command is the creation of the *.spec file. This is an autogenerated Python module containing specification on how to create executables from your sources. For example, we already used this in the following code:

# -*- mode: python -*-

block_cipher = None


a = Analysis(['myscript.py'],
             pathex=['D:\dev\app'],
             binaries=None,
             datas=None,
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          name='myscript',
          debug=False,
          strip=False,
          upx=True,
          console=True )

This .spec file contains all pyinstaller arguments specified earlier. This is very useful if you have performed a lot of customizations to your build because this can be used instead of building scripts that would have to store your configuration. Once created, you can use it as an argument to the pyinstaller command instead of your Python script:

$ pyinstaller.exe myscript.spec

Note that this is a real Python module, so you can extend it and perform more complex customizations to the building procedure using a language that you already know. Customizing the .spec file is especially useful when you are targeting many different platforms. Also, not all of the pyinstaller options are available through the command-line arguments and can be used only when modifying .spec file.

PyInstaller is an extensive tool, which by its usage is very simple for the great majority of programs. Anyway, the thorough reading of its documentation is recommended if you are interested in using it as a tool to distribute your applications.

cx_Freeze

cx_Freeze (http://cx-freeze.sourceforge.net/) is another tool for creating standalone executables. It is a simpler solution than PyInstaller, but also supports the three major platforms:

  • Windows
  • Linux
  • Mac OS X

Same as PyInstaller, it does not allow us to perform cross-platform builds, so you need to create your executables on the same operating system you are distributing to. The major disadvantage of cx_Freeze is that it does not allow us to create real single-file executables. Applications built with it need to be distributed with related DLL files and libraries. Assuming that we have the same application as featured in the PyInstaller section, the example usage is very simple as well:

$ cxfreeze myscript.py

copying C:Python27libsite-packagescx_FreezeasesConsole.exe -> D:devappdistmyscript.exe
copying C:Windowssystem32python27.dll -> D:devappdistpython27.dll
writing zip file D:devappdistmyscript.exe
(...)
copying C:Python27DLLsz2.pyd -> D:devappdistz2.pyd
copying C:Python27DLLsunicodedata.pyd -> D:devappdistunicodedata.pyd

Resulting structure of files is as follows:

$ tree /f
│   myscript.py

└───dist
        bz2.pyd
        myscript.exe
        python27.dll
        unicodedata.pyd

Instead of providing the own format for build specification (like PyInstaller does), cx_Freeze extends the distutils package. This means you can configure how your standalone executable is built with the familiar setup.py script. This makes cx_Freeze very convenient if you already distribute your package using setuptools or distutils because additional integration requires only small changes to your setup.py script. Here is an example of such a setup.py script using cx_Freeze.setup() for creating standalone executables on Windows:

import sys
from cx_Freeze import setup, Executable

# Dependencies are automatically detected, but it might need fine tuning.
build_exe_options = {"packages": ["os"], "excludes": ["tkinter"]}

setup(
    name="myscript",
    version="0.0.1",
    description="My Hello World application!",
    options={
        "build_exe": build_exe_options
    },
    executables=[Executable("myscript.py")]
)

With such a file, the new executable can be created using the new build_exe command added to the setup.py script:

$ python setup.py build_exe

The usage of cx_Freeze seems a bit easier than PyInstaller's and distutils integration is a very useful feature. Unfortunately this project may cause some troubles for inexperienced developers:

  • Installation using pip may be problematic under Windows
  • The official documentation is very brief and lacking in some places

py2exe and py2app

py2exe (http://www.py2exe.org/) and py2app (https://pythonhosted.org/py2app/) are two other programs that integrate with Python packaging either via distutils or setuptools in order to create standalone executables. Here they are mentioned together because they are very similar in both usage and their limitations. The major drawback of py2exe and py2app is that they target only a single platform:

  • py2exe allows building Windows executables
  • py2app allows building Mac OS X apps

Because the usage is very similar and requires only modification of the setup.py script, these packages seem to complement each other. The documentation of py2app projects the following example of the setup.py script that allows to build standalone executables with the right tool (either py2exe or py2app), depending on the platform used:

import sys
from setuptools import setup

mainscript = 'MyApplication.py'

if sys.platform == 'darwin':
    extra_options = dict(
        setup_requires=['py2app'],
        app=[mainscript],
        # Cross-platform applications generally expect sys.argv to
        # be used for opening files.
        options=dict(py2app=dict(argv_emulation=True)),
    )
elif sys.platform == 'win32':
    extra_options = dict(
        setup_requires=['py2exe'],
        app=[mainscript],
    )
else:
    extra_options = dict(
        # Normally unix-like platforms will use "setup.py install"
        # and install the main script as such
        scripts=[mainscript],
    )

setup(
    name="MyApplication",
    **extra_options
)

With such a script, you can build your Windows executable using the python setup.py py2exe command and Mac OS X app using python setup.py py2app. Cross compilation is of course not possible.

Despite some limitations and less elasticity than PyInstaller or cx_Freeze, it is good to know that there are always py2exe and py2app projects. In some cases, PyInstaller or cx_Freeze might fail to build executable for the project properly. In such situations, it is always worth checking whether other solutions can handle our code.

Security of Python code in executable packages

It is important to know that standalone executables does not make application code secure by any means. It is not an easy task to decompile the embedded code from such executable files, but it is doable for sure. What is even more important is that the results of such de-compilation (if done with proper tools) might look strikingly similar to original sources.

This fact makes standalone Python executables not a viable solution for closed source projects where leaking of the application code could harm the organization. So, if your whole business can be copied simply by copying the source code of your application, then you should think of other ways to distribute the application. Maybe providing software as a service will be a better choice for you.

Making decompilation harder

As already said, there is no reliable way to secure applications from de-compilation with the tools available at the moment. Still, there are some ways to make this process harder. But harder does not mean less probable. For some of us, the most tempting challenges are the hardest ones. And we all know that the eventual prize in this challenge is very high: the code that you tried to secure.

Usually the process of de-compilation consists of a few steps:

  1. Extracting the project's binary representation of bytecode from standalone executables.
  2. Mapping of a binary representation to bytecode of a specific Python version.
  3. Translation of bytecode to AST.
  4. Recreation of sources directly from AST.

Providing the exact solutions for deterring developers from such reverse-engineering of standalone executables would be pointless for obvious reasons. So here are only some ideas for hampering of the de-compilation process or devaluating its results:

  • Removing any code metadata available at runtime (docstrings), so the eventual results will be a bit less readable
  • Modifying the bytecode values used by the CPython interpreter so that conversion from binary to bytecode and later to AST requires more effort
  • Using a version of CPython sources modified in such a complex way that even if decompiled sources of the application are available they are useless without decompiling the modified CPython binary
  • Using obfuscation scripts on sources before bundling them into executables, which will make sources less valuable after the de-compilation

Such solutions make the development process a lot harder. Some of the above ideas require a very deep understanding of Python runtime but each one of them is riddled with many pitfalls and disadvantages. Mostly, they only defer what is inevitable. Once your trick is broken, it renders all your additional efforts a waste of time and resources.

The only reliable way to not allow your closed code leak outside of your application is to not ship it directly to users in any form. And this is only true if other aspects of your organization's security stay airtight.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset