Something Fishy with Python ASTs

Ok, so far so good. Now comes the interesting bit.

When you parse an AST with a __future__ import, it looks as if it stores some hidden flags that suppress this error. These flags, if they exist, are clearly not pickled. No amount of Python introspection has revealed these flags to me.

Yes, I know, Python’s source code is available, so it shouldn’t be too hard to find out if such flags exist. A 15 minute poke around the CPython source code hasn’t cleared up the mystery for me. At a glance, it looks as if the compile() function just investigates the AST it’s given in order to find the line after which __future__ imports are invalid, but that doesn’t explain the behaviour above.

Another day I will figure this one out (unless a friendly Python code dev wants to post a comment explaining it to me?).

Update: (2012-03-21) with the help of a coworker during a lunch hour, I’ve narrowed the problem down to pickle.loads() returning non-interned strings. The function future_parse() in future.c compares the module of the ImportFrom to the interned string '__future__'. The workaround is to explicitly set node.module = '__future__', which will use the interned string. When I get a chance I might get the latest version of Python and see if this issue still occurs there. The problem definitely occurs in Python 3.2.2 and Python 2.7.2.

Update 2: (2012-04-03) I identified the issue and created a ticket on the Python issue tracker. It has since been fixed in Python 3.2.4 and Python 2.7.4. My favourite comment on the issue was “After this commit the buildbots are dying randomly with segfaults.”

This entry was posted in short and tagged , , . Bookmark the permalink.

12 Responses to Something Fishy with Python ASTs