Something Fishy with Python ASTs

Ok, so far so good. Now comes the interesting bit.

When you parse an AST with a __future__ import, it looks as if it stores some hidden flags that suppress this error. These flags, if they exist, are clearly not pickled. No amount of Python introspection has revealed these flags to me.

Yes, I know, Python’s source code is available, so it shouldn’t be too hard to find out if such flags exist. A 15 minute poke around the CPython source code hasn’t cleared up the mystery for me. At a glance, it looks as if the compile() function just investigates the AST it’s given in order to find the line after which __future__ imports are invalid, but that doesn’t explain the behaviour above.

Another day I will figure this one out (unless a friendly Python code dev wants to post a comment explaining it to me?).

Update: (2012-03-21) with the help of a coworker during a lunch hour, I’ve narrowed the problem down to pickle.loads() returning non-interned strings. The function future_parse() in future.c compares the module of the ImportFrom to the interned string '__future__'. The workaround is to explicitly set node.module = '__future__', which will use the interned string. When I get a chance I might get the latest version of Python and see if this issue still occurs there. The problem definitely occurs in Python 3.2.2 and Python 2.7.2.

Update 2: (2012-04-03) I identified the issue and created a ticket on the Python issue tracker. It has since been fixed in Python 3.2.4 and Python 2.7.4. My favourite comment on the issue was “After this commit the buildbots are dying randomly with segfaults.”

This entry was posted in short and tagged , , . Bookmark the permalink.

12 Responses to Something Fishy with Python ASTs

  1. Audete says:

    i can only compliment you for the good job you have been doing in this blog, congratulations and keep it coming.

  2. Kimberly says:

    the materials doesn`t nrgcaehehe is my script:import bpyimport randomclass random_mat_panel(bpy.types.Panel): bl_idname = matPanel bl_label = Random Material Assigner bl_space_type = VIEW_3D’ bl_region_type = Tools’ def draw(self, context): layout = self.layout layout.operator( object.random_material )class mat_assigner(bpy.types.Operator): bl_idname = object.random_material bl_label = Assign Random Material bl_description = Use this to assign a random material bl_options = { REGISTER’, UNDO’} searchString = bpy.props.StringProperty(name= Filter ) seedProp = bpy.props.IntProperty(name= Seed ) availableMaterials = [] def invoke(self, context, event): self.randomize() return { FINISHED’} def check(self, context): self.randomize() def randomize(self): random.seed(self.seedProp) self.availableMaterials = [] for mat in bpy.data.materials: if(self.searchString.lower() in mat.name.lower()): self.availableMaterials.append(mat) for ob in bpy.context.selected_objects: randNum = random.randint(0, len(self.availableMaterials)-1) active_material = self.availableMaterials[randNum] def execute(self, context): return { FINISHED’}bpy.utils.register_class(mat_assigner)bpy.utils.register_class(random_mat_panel)

  3. Erinn Silevinac says:

    Hi guys, excellent webpage, really prosperous in articles and properly carefully assumed out, personally I found right here substantially fascinating and advantageous

  4. Staci Mcjunkins says:

    Quite excellent publish. I just stumbled on your blog page and wanted to state that I’ve seriously really liked examining your blog website posts. Any way I?¯ll be subscribing in your feed and I hope you submit yet again soon.

  5. Kendal Dumlao says:

    Its like you read my thoughts! You appear to know much about this, like you wrote the ebook in it or a thing. I feel you could do with some pics to push the concept house somewhat, but aside from that, this really is amazing blog. An awesome look at. Sick definitely be again.

  6. Canon Camera Reviews says:

    Just wanted to say you have a great site and thanks for posting!…

  7. J. D. Bartlett says:

    Hi Kimberly, unfortunately when you posted your comment it lost all its newlines so I can’t quite tell what you’re saying. Is it related to my post about Python ASTs? In general if you have a Python question, asking in #python on irc.freenode.net or asking on the python-tutor mailing list are good places to start.

  8. Vitamin B says:

    Hello, I just stopped in to visit your website and thought I’d say thank you.

  9. Ajuricaba says:

    i’m learning how to write well for my articles, any tips? i would really appreciate your help. [url withheld]

  10. how to study says:

    Very interesting info!Perfect just what I was looking for!

  11. J. D. Bartlett says:

    Hi Ajuricaba, a lot of writing well comes from practice, and getting feedback so you can improve. I recommend rereading your article a few times before you publish it, to check for any errors or things that might flow better if reworded. It’s even good to get someone else to proofread your articles just to get a feel for things that you as the author might have overlooked.

  12. J. D. Bartlett says:

    @Audete, how to study: Thanks!