|
1 2 3 4 5 6 7 8 9 10 |
$ python Python 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import ast >>> m = ast.parse('from __future__ import division') >>> m <_ast.Module object at 0xb764418c> >>> compile(m, '<string>', 'exec') at 0xb7636e78, file "<string>", line 1> |
Ok, so far so good. Now comes the interesting bit.
|
1 2 3 4 5 6 7 8 9 |
>>> import pickle >>> m2 = pickle.loads(pickle.dumps(m)) >>> m2 <_ast.Module object at 0xb7651c0c> >>> compile(m2, '<string>', 'exec') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 1 SyntaxError: from __future__ imports must occur at the beginning of the file |
When you parse an AST with a __future__ import, it looks as if it stores some hidden flags that suppress this error. These flags, if they exist, are clearly not pickled. No amount of Python introspection has revealed these flags to me.
Yes, I know, Python’s source code is available, so it shouldn’t be too hard to find out if such flags exist. A 15 minute poke around the CPython source code hasn’t cleared up the mystery for me. At a glance, it looks as if the compile() function just investigates the AST it’s given in order to find the line after which __future__ imports are invalid, but that doesn’t explain the behaviour above.
Another day I will figure this one out (unless a friendly Python code dev wants to post a comment explaining it to me?).
Update: (2012-03-21) with the help of a coworker during a lunch hour, I’ve narrowed the problem down to pickle.loads() returning non-interned strings. The function future_parse() in future.c compares the module of the ImportFrom to the interned string '__future__'. The workaround is to explicitly set node.module = '__future__', which will use the interned string. When I get a chance I might get the latest version of Python and see if this issue still occurs there. The problem definitely occurs in Python 3.2.2 and Python 2.7.2.
Update 2: (2012-04-03) I identified the issue and created a ticket on the Python issue tracker. It has since been fixed in Python 3.2.4 and Python 2.7.4. My favourite comment on the issue was “After this commit the buildbots are dying randomly with segfaults.”
12 Responses to Something Fishy with Python ASTs