Tips for writing tests for Twisted code

  1. Trial basics
  2. Twisted-specific quirks: reactor, Deferreds, callLater

Trial basics

Trial is Twisted's testing framework. It provides a library for building test cases (similar to the Python standard library's unittest module) and utility functions for working with the Twisted environment in your tests, and a command-line utility for running your tests. For instance, to run all the Twisted tests, do:

$ trial -R twisted

Refer to the Trial man page for other command-line options.

Twisted-specific quirks: reactor, Deferreds, callLater

The standard Python unittest framework, from which Trial is derived, is ideal for testing code with a fairly linear flow of control. Twisted is an asynchronous networking framework which provides a clean, sensible way to establish functions that are run in response to events (like timers and incoming data), which creates a highly non-linear flow of control. Trial has a few extensions which help to test this kind of code. This section provides some hints on how to use these extensions and how to best structure your tests.

Leave the Reactor as you found it

Trial runs the entire test suite (over one thousand tests) in a single process, with a single reactor. Therefore it is important that your test leave the reactor in the same state as it found it. Leftover timers may expire during somebody else's unsuspecting test. Leftover connection attempts may complete (and fail) during a later test. These lead to intermittent failures that wander from test to test and are very time-consuming to track down.

Your test is responsible for cleaning up after itself. The tearDown method is an ideal place for this cleanup code: it is always run regardless of whether your test passes or fails (like a bare except clause in a try-except construct). Exceptions in tearDown are flagged as errors and flunk the test.

reactor.stop is considered very harmful, and should only be used by reactor-specific test cases which know how to restore the state that it kills. If you must use reactor.run, use reactor.crash to stop it instead of reactor.stop.

Calls to reactor.callLater create IDelayedCalls. These need to be run or cancelled during a test, otherwise they will outlive the test. This would be bad, because they could interfere with a later test, causing confusing failures in unrelated tests! For this reason, Trial checks the reactor to make sure there are no leftover IDelayedCalls in the reactor after a test, and will fail the test if there are.

Similarly, sockets created during a test should be closed by the end of the test. This applies to both listening ports and client connections. So, calls to reactor.listenTCP (and listenUNIX, and so on) return IListeningPorts, and these should be cleaned up before a test ends by calling their stopListening method (note that this can return a Deferred, so you should wait on it to make sure it has really stopped before the test ends. Calls to reactor.connectTCP return IConnectors, which should be cleaned up by calling their disconnect method. Trial will warn about unclosed sockets.

deferredResult

If your test creates a Deferred and simply wants to verify something about its result, use deferredResult. It will wait for the Deferred to fire and give you the result. If the Deferred runs the errback handler instead, it will raise an exception so your test can fail. Note that the only thing that will terminate a deferredResult call is if the Deferred fires; in particular, timers which raise exceptions will not cause it to return.

Waiting for Things

The preferred way to run a test that waits for something to happen (always triggered by other things that you have done) is to use a while not self.done loop that does reactor.iterate(0.1) at the beginning of each pass. The 0.1 argument sets a limit on how long the reactor will wait to return if there is nothing to do. 100 milliseconds is long enough to avoid spamming the CPU while your timers wait to expire.

Using Timers to Detect Failing Tests

It is common for tests to establish some kind of fail-safe timeout that will terminate the test in case something unexpected has happened and none of the normal test-failure paths are followed. This timeout puts an upper bound on the time that a test can consume, and prevents the entire test suite from stalling because of a single test. This is especially important for the Twisted test suite, because it is run automatically by the buildbot whenever changes are committed to the Subversion repository.

Trial tests indicate they have failed by raising a FailTest exception (self.fail and friends are just wrappers around this raise statement). Exceptions that are raised inside a callRemote timer are caught and logged but otherwise ignored. Trial uses a logging hook to notice when errors have been logged by the test that just completed (so such errors will flunk the test), but this happens after the fact: they will not be noticed by the main body of your test code. Therefore callRemote timers can not be used directly to establish timeouts which terminate and flunk the test.

The right way to implement this sort of timeout is to have a self.done flag, and a while loop which iterates the reactor until it becomes true. Anything that causes the test to be finished (success or failure) can set self.done to cause the loop to exit.

Most of the code in Twisted is run by the reactor as a result of socket activity. This is almost always started by Protocol.connectionMade or Protocol.dataReceived (because the output side goes through a buffer which queues data for transmission). Exceptions that are raised by code called in this way (by the reactor, through doRead or doWrite) are caught, logged, handed to connectionLost, and otherwise ignored.

This means that your Protocol's connectionLost method, if invoked because of an exception, must also set this self.done flag. Otherwise the test will not terminate.

Exceptions that are raised in a Deferred callback are turned into a Failure and stashed inside the Deferred. When an errback handler is attached, the Failure is given to it. If the Deferred goes out of scope while an error is still pending, the error is logged just like exceptions that happen in timers or protocol handlers. This will cause the current test to flunk (eventually), but it is not checked until after the test fails. So again, it is a good idea to add errbacks to your Deferreds that will terminate your test's main loop.

Here is a brief example that demonstrates a few of these techniques.

class MyTest(unittest.TestCase):
    def setUp(self):
        self.done = False
        self.failure = None

    def tearDown(self):
        self.server.stopListening()
        # TODO: also shut down client
        try:
            self.timeout.cancel()
        except (error.AlreadyCancelled, error.AlreadyCalled):
            pass

    def succeeded(self):
        self.done = True

    def failed(self, why):
        self.done = True
        self.failure = why

    def testServer(self):
        self.server = reactor.listenTCP(port, factory)
        self.client = reactor.connectTCP(port, factory)
        # you should give the factories a way to call our 'succeeded' or
        # 'failed' methods
        self.timeout = reactor.callLater(5, self.failed, "timeout")
        while not self.done:
            reactor.iterate(0.1)

        # we get here if the test is finished, for good or for bad
        if self.failure:
            self.fail(self.failure)
        # otherwise it probably passed. Cleanup will be done in tearDown()

Index

Version: 2.1.0