Inline Deferreds in Twisted (Without Yield)

Introduction

By buffer of new stuff to blog about is getting low, so today I bring you something I played with a bit last year.

After being introduced to Twisted‘s inlineDeferred decorator, I decided to have some fun with that and byteplay. The result is a decorator that does the same as inlineDeferred, but without you having to use yields.

Warning

I know a Twisted developer. I showed him this code. He pasted it to other Twisted developers. The overwhelming response was one of horror. I think the horror was directed at the fact that the code makes things so magical. Use this code at your own peril.

How to use my code

The example below demonstrates how my code could be used.

from twisted.internet import defer, reactor, task

from async import async

def slowFunction(x=False):
    '''
    This is an example of a normal twisted function which returns a
    deferred.
    '''
    d = defer.Deferred()
    if x:
        reactor.callLater(2, d.errback, ValueError('x was '
                'true'))
    else:
        reactor.callLater(5.5, d.callback, 'twisted is awesome')
    return d

@async
def main():
    '''
    This is an example of an asynchronous function, where any calls which
    return defers appear to behave like normal functions.
    '''
    print slowFunction()
    try:
        print slowFunction(True)
    except ValueError, E:
        print repr(E)

    print 'The end.'
    reactor.callLater(1, reactor.stop)

# Start a timer to show that the demo is asynchronous.
def timer(_x=[0]):
    _x[0] += 1
    print _x[0]
task.LoopingCall(timer).start(1, False)

# Start the asynchronous function.
main()

reactor.run()

How it works

My module essentially goes through the byte-code of any function that you decorate with @async, and translates:

• every function call foo(*args, **kwargs) into yield maybeDeferred(foo, *args, **kwargs)
• every return statement return foo into returnValue(foo)

It then decorates the whole function with inlineDeferred and returns it. Note that this code will not work for functions which are already generators.

The dodgy bits

If you look at my code below, you’ll notice that it loads the functions maybeDeferred and returnValue into the global scope of the decorated function with the names _async_maybeDeferred and _async_returnValue. I couldn’t see any easy way of avoiding this. This means that any module that contains a function decorated with @async will magically have access to these functions. But it would be extremely bad practice to rely on this fact.

Prerequisites

To run my code you will need:

• cPython 2.6
• Twisted (duh!)
• rfk’s promise module (because rfk has updated byteplay to work with cPython 2.6)

The code itself

This code is licensed under the MIT license. You may not use, modify, or distribute it unless you agree to the license. The code for async.py is listed below, or if you would like to download it, click here (zipped Python source including example code).

# Copyright (c) 2010 Joshua D. Bartlett
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

import promise
from promise.byteplay import CO_GENERATOR
from twisted.internet.defer import maybeDeferred, returnValue, inlineCallbacks
import types

def async(x):
    if x.func_code.co_flags & CO_GENERATOR:
        raise TypeError('@async does not work on generators')

    code = promise.Code.from_code(x.func_code)

    for op, arg in code.code:
        if op == promise.CALL_FUNCTION:
            break
    else:
        # Doesn't call any functions, so don't touch it.
        return x

    i = len(code.code) - 1
    insert_positions = []
    stack = 1
    while i >= 0:
        op, arg = code.code[i]
        if op == promise.RETURN_VALUE:
            insert_positions.append((stack-1,
                    (promise.LOAD_GLOBAL, '_async_returnValue')))
            code.code.insert(i,   (promise.CALL_FUNCTION, 1))
            code.code.insert(i+1, (promise.POP_TOP, None))
            code.code.insert(i+2, (promise.LOAD_CONST, None))
        else:
            if op == promise.CALL_FUNCTION:
                insert_positions.append((stack-1,
                        (promise.LOAD_GLOBAL, '_async_maybeDeferred')))
                code.code[i] = (op, arg + 1)
                code.code.insert(i+1, (promise.YIELD_VALUE, None))
            try:
                pop, push = promise.getse(op, arg)
            except ValueError:
                pass
            else:
                stack = stack - push + pop

        while len(insert_positions) > 0 and stack == insert_positions[-1][0]:
            operation = insert_positions.pop()[1]
            code.code.insert(i, operation)
        i -= 1

    fn = types.FunctionType(code.to_code(), x.func_globals, x.func_name,
            x.func_defaults, x.func_closure)

    x.func_globals.setdefault('_async_returnValue', returnValue)
    x.func_globals.setdefault('_async_maybeDeferred', maybeDeferred)
    return inlineCallbacks(fn)