Embedding Pyrex HOWTO

Introduction

For hordes of Python programmers, the Python/C Wrapper Language Pyrex has proved to be an absolute god-send, as it eliminates all the 'red tape' involved with interfacing Python code with C code.

However, most perception and usage of Pyrex has been geared towards building extension types and modules which can be loaded like regular Python modules.

This mini-tutorial steps out of this usage model, and maps out a recipe which allows you to write standalone programs in Pyrex which embed and run a Python interpreter.

Embedding Python? You IDIOT! Why!??

So often, operating systems, languages and development tools become somewhat of a religion. Python has not escaped from this syndrome.

If you even breathe the dreaded "e-word" (embedding), a lot of Python programmers will lynch you and deride you to no end, bleating the mantra extend, don't embed.

In a lot of cases, they're right. On the whole, it makes more sense to extend Python rather than embed it into C applications.

But there are situations where embedding can deliver greater convenience.

For example:

What This Tutorial Covers

In this HOWTO, we walk you through the process of building a simple Pyrex program which can be compiled to run as a standalone binary, or a binary which looks just like any other binary, except that it just happens to dynamically link to libpython2.x.so.

The Recipe

Create a simple .pyx file

The following code is a full Pyrex source file which can be compiled and run stand-alone:
"""
Experiment with embedding Python via Pyrex
"""

# get stuff we need from C header files

cdef extern from "stdio.h":

    int printf(char *format,...)

cdef extern from "Python.h":

    # embedding funcs
    void Py_Initialize()
    void Py_Finalize()
    void PySys_SetArgv(int argc, char **argv)

    # declare any other Python/C API functions we might need
    void Py_INCREF(object o)
    void Py_DECREF(object o)
    object PyString_FromStringAndSize(char *, int)
    # ... etc etc...

# IMPORTANT - we need to explicitly prototype the function
# 'init<mymodulename>()', where 'mymodulename' is the
# filename this code resides in. I called the file 'testpyx.pyx',
# hence the name below

cdef public void inittestpyx()


# With all this out of the way, we can declare any amount of
# Pyrex python extension types, and regular python classes/functions

cdef class Testclass:
    
    cdef public int someint
    cdef public char *somestring
    
    def __init__(self):
        
        self.someint = 43
        self.somestring = "this is a string"
    
    def hello(self):
        
        print "Hello, this is an instance of %s" % self.__class__.__name__


# Now, need to declare the needed C main() function

cdef public int main(int argc, char **argv):

    # warm up python
    Py_Initialize()
    
    # Setup sys.argv
    PySys_SetArgv(argc, argv)

    printf("initialising testpyx\n")
    inittestpyx() # mandatory

    # Initialisation done - we can now do Python stuff

    printf("testmain: instantiating Testclass\n")
    testobj = Testclass()
    printf("testmain: created testobj\n")

    print "testobj.someint = %s" % testobj.someint
    print "testobj.somestring = %s" % testobj.somestring
    
    print "calling testobj.hello()"
    testobj.hello()

    # clean up before we leave - this probably isn't strictly necessary
    print "cleaning up"
    Py_Finalize()

    

Now, the Makefile

The following Makefile excerpt works fine for me. Hopefully it'll work for you too:
# Makefile for creating our standalone Pyrex program
PYVERSION=2.3
PYPREFIX=/usr
INCLUDES=-I$(PYPREFIX)/include/python$(PYVERSION)

testpyx: testpyx.o
	gcc -o $@ $^ -lpython$(PYVERSION)

testpyx.o: testpyx.c
	gcc -c $^ $(INCLUDES)

testpyx.c: testpyx.pyx
	pyrexc testpyx.pyx

    

Conclusion

To build a working standalone binary program written in Pyrex, we had do:
  1. Import the Python/C API functions Py_Initialize(), Py_Finalize() and PySys_SetArgv(), plus whatever other API functions we might need
  2. Explicitly declare a C prototype for the module-init function which Pyrex generates - in our case, 'inittestpyx' if we are compiling the pyrex source file 'inittestpyx.pyx'.
  3. Create a public C-callable 'main()' function, which explicitly:
    1. Initialises the Python interpreter
    2. Calls the Pyrex-generated module init function
    3. Freely executes any number of Python or statements, as our application requires
    4. Terminates the Python interpreter
  4. Create a Makefile, which:
    1. Explicitly compiles our pyx file to a C file, using the standalone Pyrex 'pyrexc' command
    2. Compiles the generated .c file to a .o or .obj object file, making sure to provide the correct include path
    3. Links the resulting .o or .obj file into a standalone executable binary, making sure to specify the python runtime library.

David McNab
Last modified: Wed Jun 16 12:27:08 NZST 2004