In a recent article, Testing Concurrent Programs, published on TheServerSide.com, Brian Goetz acknowledges that testing programs with multiple concurrent threads of execution is hard:

The failure modes of concurrent programs are less predictable and repeatable than for sequential programs... Failures in concurrent programs...tend to be rare probabilistic events.

Because of this, reproducing failures in concurrent programs can be maddeningly difficult. Not only might the failure be rare, and therefore not manifest itself frequently, but it might not occur at all in certain platform configurations, so that bug that happens daily at your customer's site might never happen at all in your test lab. Further, attempts to debug or monitor the program can introduce timing or synchronization artifacts that prevents the bug from appearing at all. As in Heisenberg's uncertainty principle, observing the state of the system may in fact change it.

Goetz suggests that the first step toward testing the behavior of concurrent programs is to isolate concurrency to well-defined portions of the code, such as into workflow managers, resource pools, or work queues.

Enforcing thread-safety of objects inside those concurrent constructs is the responsibility of the application. According to Goetz, the best way to achieve thread safety of objects interacting with concurrent program elements is to make those objects immutable. If an immutable object interface is not feasible, Goetz suggests making such objects effectively immutable:

Effectively immutable objects are those [that] are not necessarily immutable by design—they may have mutable state—but [that] the program treats as if they were immutable after they are published where they might be accessed by other threads.

In other words, once you put a mutable object into a shared data structure, where other threads might then have access to it, make sure that it is not modified again by any thread. The judicious use of immutability and effective immutability limit the range of potentially incorrect concurrent actions by limiting mutability to a few core classes that can be strenuously unit-tested.

Goetz also makes a few suggestions for testing classes whose states are modified by concurrent threads. The most interesting suggestion is to take into account the probabilistic nature of concurrent errors in testing:

Because the failures you are searching for are probabilistic ones, your test cases are (at best) probabilistic as well. To give yourself the maximum chance of running across the right conditions to trigger a failure, you should plan to run concurrent tests for much longer.

What techniques have you found useful in testing concurrent code?