Introduction
Compyler is a native x86 python compiler I worked on in 2006 before
my interest moved on to other things. Instead of targeting C,
compyler targets native x86. It even includes its own assembler which
generates native windows COFF files (but no linker so you still need a
toolchain to generate a complete executable).
Motivation
No I wasn't trying to get a faster program. I was primarily
interested in generating python apps with small footprints.
Secondarily, I considering marketing and selling the program as a
obfusticater since that request seemed to come up once every two weeks
on comp.lang.python. Luckily, that didn't work out so I decided to
release the program under a BSD-style license.
Status
The program is basically abandonware. I don't really have any
interest in doing anything more with this in my spare time, and I
doubt anyone would pay me to work on it. That being said, I made a
reasonable amount of progress. Several test cases of python code will
successfully compile and execute. The most complex piece of code to
run is pystones.py (from the standard python distro) but it runs
excessively slow. It looks like I don't have reference counting
evened out. There is a testcase that can generate a primitive .pyd
file that can be imported by python that works, but generated .exes
currently throw an exception when run.
Right now it is windows only. pyasm (the component that generates
the
assembly) will run on linux, but it doesn't generate either elf files
or something that a GNU assembler can understand.
Approach
The basic approach is to use python to generate normal python
bytecode, and then transliterate the bytecode into x86 assembly. For
example, the RETURN_VALUE bytecode gets translated to the instructions
POP EAX and JMP exit.
Components
There are three basic components that make everything work.
pyasm
pyasm is a dynamic assembler for python. It can generate either coff
files as binary output to create an executable, or dynamically compile
code, load it to memory, and bind it to a python variable for real
time execution.
The files are located at site-packages/pyasm. There is some
more documentation under the 'doc' subdirectory there.
xpython
xpython is a slightly modified version of the python interpreter. It
primarily contains two modifications. There are two new builtin
objects called xcodeobject and xfunctionobject. These run the
compiled code, and are hooked into ceval. The pertinent files are
where you'd expect them to be in the python source tree.
compyler
compyler performs the actual translation of python code to x86 code.
It is located under site-packages/compyler. The bulk of the work is
being done in codeObjectAssembler.py.
Usage
The best places to start or the test and benchmark subdirectories
under site-packages/compyler. test/test_all.py will run all of the
unit tests. test_exe.bat and test_pyd.bat will (attempt) to generate
a stand-alone .exe and .pyd. All python code needs to be run with the
modified python interpreter listed above, and the batch files will
probably require some environment-specific configuration (basically
the path to your vcvars32.bat file is). The benchmark directory has a
batch file that will run pystones.py to compare against a standard
python app.
codeObjectAssembler.py has a variable 'debug' that can be set to
different levels to control the amount of diagnostic information.
Where to
get started
Okay, so if you've actually gotten this far, things seem to be
running, and you're interested in trying to polish things up, here's
where I'd start.
refcounts
and pystone
Try to correct the refcounts. This seems to be the reason pytstone
runs so poorly. If the refcounts get straightened out, it'll
(hopefully) run favorably compared to python. There is also a test
that does a fibonacci sequence in the test dir that blows up memory.
Python 2.5
Update xpython to python 2.5. I don't think there were any bytecode
changes introduced, and since compyler starts with the bytecode you
should be able to get all new language features for free.
.exe stub code
I was unable to get the stub code to bootstrap an exe funtioning
correctly. Start looking at about line 168 in codeObjectAssembler.
front end
interface
Make some files like 'compyle' and 'makepyd' as a front end
interface. Right now we're using the api.
Get it running on Linux/OSX
Right now pyasm generates coff files. You could either generate
binary elf files, or generate actual assembly to hand off to the GNU
assembler.
compyle compyler
The ultimate test for any self respecting compiler. Get compyler to
the point where it can generate a pyd of itself.
License
The code is released under a standard BSD-style license. |
ď Grant Olson, Mar 14, 2010, 5:12 PM ď Grant Olson, Mar 14, 2010, 5:14 PM
|