Introduction
A few weeks ago I proposed a new concept for not-quite-OO programming. Recently I’ve been working on turning this into a reality. I haven’t written a new programming language. Rather, I’ve extended Python using import hooks. I’ve called this extension “Pyj”. In this article I’ll describe what Pyj does (or will someday do) and how I’ve implemented it.
The Pyj Philosophy
Pyj is quite experimental and I’ve invented it based on thoughts I’ve had that usually start with “Wouldn’t it be cool if…”. Only time will tell whether these ideas are useful and worthwhile. Here are the main principles I’ve based Pyj upon:
- small files, declaring all dependencies;
- files shouldn’t need to be organised perfectly;
- behaviour of objects is more important than some kind of class hierarchy; and
- functions should be able to easily access ancestors of the current “context” object—see this article for more details.
It is worth noting that Pyj files are actually valid Python files. The difference lies in how they are treated when imported into Python.
Small Files
I’ve designed Pyj around the idea of keeping each file simple and small, so that programmers don’t have to search through a whole bunch of code to find the part of the file that they want. In practice what this looks like is each Pyj file representing a single function or object. Pyj is implemented so that Pyj modules behave almost identically to the function or object that they represent. So if you write /stoop/heap/heart.pyj and you make it define a function, you can call the function using stoop.heap.heart() rather than having to add the name of the function, as in stoop.heap.heart.myFunction(). Within a Pyj source file, whichever function or object is called “this” will be the function or object represented by that module.
As an example, if you wanted to write a function to add two numbers together, your Pyj file wolud look like:
def this(a, b):
return a + b
You would not put any other functions in the same file. And because you have named the function “this”, you would be able to access it using the module’s name. For example, if the file was called addTwoNumbers.pyj, you would simply type addTwoNumbers(x, y) to invoke the function.
No Need to Organise Perfectly
In going with small files, I’ve assumed that developers have tools (such as grep) available to quickly search and find which file they want. Small files mean lots of files, and without this assumption, developers would be forever creating sub-packages to organise all their pyj modules, and would have much less time to do the actual programming.
Behaviour is More Important than Hierarchy
Because of the idea of functions running in contexts (I assume you’ve already read my previous post), and because of the flexible nature of Python, I’ve decided to throw most of object-oriented programming (OOP) out the window and resort to an object model similar to that used in JavaScript. That is, we don’t care about classes and inheritance, we care about what an object does. So rather than a class with a constructor, we have factory functions—functions which create and return objects. This has allowed me to reinvent a very flexible kind of OOP using the context in place of the current object. I hope to present a complete working example of a useful Pyj program in a few weeks, and you’ll be able to see what I mean.
Functions Can Access Ancestors of the Context
For this one you really should read my previous post. Basically though, each object has a name and can have a parent. Every function is called in a context, and that context may be any object. If a function is running in a particular context and the context has a parent named breathe which has a parent named imposter which has a parent named wisdom, the function can access wisdom by the name wisdom rather than having to type self.breathe.imposter.wisdom. Too complicated? My previous post explains in more depth.
Python Import Hooks
The idea of import hooks in Python is that you can customise the way in which Python imports modules. Pyj makes use of import hooks in order to distinguish Pyj modules from ordinary Python modules. Ordinary Python modules have a .py extension, and the compiled byte-code is saved in a file with a .pyc or .pyo extension (.pyo for optimised bytecode). Pyj modules have a .pyj extension, and compiled byte-code is saved in files with .pyjc or .pyjo extensions. I’ve written an import hook, so that when an import statement is executed:
- if there is a .py, .pyo, or .pyc file, that will be imported as a normal Python module;
- otherwise, if there is a .pyj, .pyjc, or .pyjo file, that will be imported as a Pyj module.
(As an aside, I’m considering changing it so that if there is a .pyc and .pyj but no .py it will load the .pyj rather than the .pyc, but for now the default is always normal Python modules over Pyj modules.)
To figure out how to write the import hook, I exploited these useful resources:
- PEP 302 describes in detail how import hooks work; and
- the source code of the ihooks and pkgutil modules (found in your Python directory—I’m using Python 2.6).
So if you were writing a Python package which includes some Pyj modules, how would you install the import hooks? Simply put the following line in the __init__.py of your package.
import pyj
That’s it! Your __init__.py file will be loaded first before any attempt to import a sub-package or module within the package. When Pyj is imported, it installs the import hook.
How Pyj Works
The main challenge in writing Pyj was figuring out how to change the way a Python function looked up global names. Normally if you try to access a global name from within a function, the globals dictionary of the module is looked up and that’s it. But for Pyj, I needed the lookup to depend on the current context. It took me a few attempts to get this right.
Subclassing Dict
My first attempt was to create a subclass of the dict built-in type, and customise the way that lookups are done. My reasoning was that if the function looks up global variables in a dict, looking them up in my subclass of dict should allow me to change the way that look-ups are done. Unfortunately this plan was thwarted by the fact that Python does its global lookups using the default dict lookup even if your subclass customises dict lookups.
Intelligent Name Substitution
The approach which I had to go with in the end was this: at the time that a function is called, copy any relevant names into the global namespace, then when the function ends, replace them with what they were before. A relevant name is a name which:
- is used in a function in the Pyj module (this is determined by inspecting function.func_code.co_names); and
- is the name of an ancestor of the current context, or a special name like self.
In order to actually achieve this, every function declared in a Pyj module is wrapped in a simple class which intercepts calls. And performs the appropriate substitutions before and after the execution of the function.
Can I try this out?
At the time of writing, I’m still working hard on getting Pyj to do exactly what I want it to. But in a couple of weeks I hope to release a working example of a program written in Pyj, for you to have a look at. For the time being, you can have a look at my code here. I’ve also set up a project page for Pyj here.
In Closing
I’ve explained the general concepts behind Pyj and a few of the specifics about how Pyj works. Pyj is entirely experimental, but I’m hoping that it’s a lot of fun to play (erm… I mean work…) with. I also hope that my explanation of Pyj has inspired you to imagine other ways in which conventional approaches to programming might be challenged. Certainly, some such challenges will be prove unworthy, and be defeated. But the more people dream about what programming could be like, the more chance we’ll have of making real improvements to how we do things.