The Artima Developer Community
Sponsored Link

Weblogs Forum
Mixins considered harmful/2

10 replies on 1 page. Most recent reply: Jan 6, 2009 12:38 AM by Michele Simionato

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 10 replies on 1 page
Michele Simionato

Posts: 222
Nickname: micheles
Registered: Jun, 2008

Mixins considered harmful/2 (View in Weblogs)
Posted: Jan 6, 2009 12:38 AM
Reply to this message Reply
Summary
In this post I clarify my position with respect to mixins, I show a few common design mistakes and I suggest some alternative solution.
Advertisement

A few conceptual issues with mixins

In the first article of this series I have discussed a very serious problem of the mixin approach, i.e. the namespace overpopulation issue.

http://www.phyast.pitt.edu/~micheles/python/Overpopulation.jpg

The namespace overpopulation issue

The overpopulation issue comes from the idea of growing functionality by adding more and more mixin classes, which is just plain wrong. It is true that you can use the idea in little frameworks with little damage, but that does not make it a good design solution. Small frameworks have a tendency to grow, and you should not start with a weak design.

Some reader argued that this is not a problem of mixins per se, but a problem of bad design. That is true, but I maintain that a technique which is so easy to misuse even by expert programmers, should be regarded with suspicion, especially when there are better solutions available. Moreover, I have a few conceptual issues with mixins - as implemented in most languages - which are independent of the overpopulation problem.

First of all, I think everybody agrees that the best way to solve a complex problem is to split it in smaller subproblems, by following the dividi et impera principle. The disturbing thing about mixins is that the principle is applied at the beginning (the problem is decomposed in smaller independent units) but at the end all the functionalities are added back to the client class as an undifferentiated soup of methods.

Therefore a design based on mixins looks clean to the framework writer - everything is well separated in his mind - but it looks messy to the framework user - she sees methods coming from all directions without a clear separation. It is really the same situation than using the from module import * idiom, which is rightly frowned upon.

I find it most unpythonic that mixins make the life of the framework writer easier, but the life of the framework reader more difficult, since the goal of Python is to make code easy to read, not easy to write. The scenario I have in mind is the usual one: a poor programmer who needs to debug an object coming from a gigantic framework which is terra incognita to her, without any documentation and with a strict deadline (do you see yourself in there?). In such conditions a framework heavily based on mixins makes things harder, since the programmer gets drowned under hundreds of methods which are properly ordered in mixin classes on the paper, but not on the battle field.

There is also another conceptual issue. The idea behind mixins is that they should be used for generic functionality which can be applied to different classes (think of mixins like Persistent, Comparable, Printable, etc.). But this is exactly the same situation where you want to use generic functions.

In this post of mine I actually argue that generic functions (a.k.a. multimethods) are a better solution than mixins. I also provide a very concrete example, which I think generalizes. The advantage of generic functions is that they are clearly defined outside classes, whereas the mixin approach is kind of schizophrenic: the functionality is actually defined externally, but that fact is made invisible to the final user.

I am a big fan of generic functions which are already used in the Python word - print is a generic function, the comparison operators are generic functions, numpy universal functions (ufunctions) are generic functions, etc - but should be used even more. With generic functions, mixins becomes useless. A side effect is that the class namespace becomes much slimmer: for instance, in CLOS classes are used just to contain state, whereas the methods live in a separate namespace. In most languages instead, classes are used as a namespace control mechanism, performing double duty - namespace control should be the job of modules.

A particularly bad usage of mixins

A tipical beginner's tutorial (for instance, I recommend Using Mix-ins with Python , by Chuck Esterbrook which is very well written and very informative, even if the point of view is exactly the opposite of mine) will tell you that mixins are used to add functionality to the classes they mix in. For instance a mixin class WithLog could be used to enhance a pre-existing class C with a logging capability:

class C(object):
  "A base class"
class WithLog(object):
  "A mixin class"
  @property
  def log(self):
    return logging.getLogger(self.__class__.__name__)
class C_WithLog(C, WithLog):
  "A mixin-enhanced class"

An example of usage is the following:

>>> c = C_WithLog()
>>> c.log.warn("hello")

That prints

WARNING:C_WithLog:hello.

The usage of mixins you see here is wrong: why would you use inheritance when you need just one method? You can just import the one method you need! Generally speaking, a mixin class has sense only when you have a set of methods which belong together: if you have a single method, or a set of disconnected methods, you are much better off by defining the methods externally, in an utility module, and then by importing them in the class namespace. Of course, here I am assuming that you really want the external method to end up in the class namespace, possibly because of interface requirements, but I am not saying that this is always a good idea. You can import the method in your class as simply as that:

class CWithLog(C):
  from utility import log # log is the property defined above

This approach is very little used in Python, probably because most people coming from other languages do not know it is possible, but it is in my opinion a much clearer solution than inheritance. The problem with inheritance is that it requires a substantial cognitive load: when I see the line of code class C_WithLog(C, WithLog) I immediately I ask myself many questions: which methods are exported by C_WithLog? is there any method of C which accidentally overrides one of the methods of C_WithLog? if yes, is there any method cooperation mechanism (super) or not? what are the ancestors of C_WithLog? which methods are coming from them? are such methods overridden by some C method? is there a cooperation mechanism on C_WithLog ancestors? What's the method resolution order of the hierarchy? On the other hand, if I see from utility import log I have very little to understand and very little to worry about. The only caution in this specific example is that I will have a single logger shared by all instances of the class since logging.getLogger(self.__class__.__name__) will return always the same object. If I need different loggers with different configurations for different instances I will have to override the .log attribute on a case by case basis, or I will have to use a different strategy, such as the dependency injection pattern, i.e. I will have to pass the logger to the constructor.

Acceptable usages of mixins

There are usages for mixins which are restricted in scope and not dangerous: for instance, you can use mixins for implementing the comparison interface, or the mapping interface. This is actually the approach suggested by the standard library, and by the new ABC's in Python 2.6. This is an acceptable usage: in this case there is no incontrollable growth of methods, since you are actually implementing well know interfaces - typically a few specific special methods. In order to give a practical example, let me discuss a toy application.

Suppose you want to define a PictureContainer class in an application to manage pictures and photos. A PictureContainer object may contain both plain pictures (instances of a Picture class) and PictureContainer objects, recursively. From the point of view of the Python programmer it could make sense to implement such a class by using a dictionary. A Picture object will contain information such as the picture title, the picture date, and a few methods to read and write the picture on the storage (the file system, a relation database, an object database like the ZODB or the AppEngine datastore, or anything else).

The first version of the PictureContainer class could be something like that:

class SimplePictureContainer(object):
    "A wrapper around the .data dictionary, labelled by an id"
    def __init__(self, id, pictures_or_containers):
      self.id = id
      self.data = {} # the inner dictionary
      for poc in pictures_or_containers:
        # both pictures and containers must have an .id
        self.data[poc.id] = poc

At this point, one realized that it is annoying to call the inner dictionary directly and that it would be nicer to expose its methods. A simple solution is to leverage on the standard library class UserDict.DictMixin which is there just for that use case. Since we are at it, we can also add the logging functionality: that means that the low-level interface (calling directly the inner dictionary methods) will not log whereas the high level interface will log:

class BetterPictureContainer(SimplePictureContainer, DictMixin):
  from utility import log

  def __getitem__(self, id):
    return self.data[id]

  def __setitem__(self, id, value):
    self.log.info('Adding or replacing %s into %s', id, self.id)
    self.data[id] = value

  def __delitem__(self, id):
    self.log.warn('Deleting %s', id)
    del self.data[id]

  def keys(self):
    return self.data.keys()

Using DictMixin is acceptable, since

  1. DictMixin provided to its subclasses the standard interface of a dictionary, a conceptually tied set of methods;
  2. the dictionary interface is well know to anybody knowing how to use dictionaries in Python, so that the cognitive load is zero;
  3. DictMixin allows a substantial code reuse: we redefined explicitly just 4 methods, but actually we are indirecty affecting 17 other methods: __cmp__, __contains__, __iter__, __len__, __repr__, clear, get, has_key, items, iteritems, iterkeys, itervalues, pop, popitem, setdefault, update, values: without DictMixin we would need to reimplement all of them.

However, notice that in this example the usage of DictMixin as mixin class is acceptable, but not optimal: the best solution is to use DictMixin as a base class, not as a mixin class.

The core problem is that we started from a wrong desing: we wrote SimplePictureContainer when we did not know of the existence of DictMixin. Now, a posteriori, we are trying to fix the mistake by using multiple inheritance, but that it not the Rigth Thing (TM) to do. The right thing would be to change the source code of SimplePictureContainer and to derive directly from DictMixin.

In the real world usually you do not have complete control of the code: you may leverage on a third party library with a design error, or simply with an old library, written when DictMixin did not exist. In such a situation you may have no way to modify the source code. Then using DictMixin and multiple inheritance is a perfectly acceptable workaround, but it is a workaround still, and it should not be traded for a clever design.

Moreover, even the best examples of mixins could be replaced by generic functions: this is why I would not provide mixins, should I write a new language from scratch. Of course, in an existing language like Python, one has to follow the common idioms, so I use mixins in a few controlled cases, and I have no problems with that. For instance, one could define an EqualityMixin which defines the special methods __eq__ and __ne__, with __ne__ being the logical negation of __eq__ (Python does not do that by default). That would be a fine usage but I don't do that, I prefer to duplicate two lines of code and to write the __ne__ method explicitly, to avoid complicating my inheritance hierarchy. One should should decide when to use a mixin or not on a case by case basis, with a bias for the not.


Carson Gross

Posts: 153
Nickname: cgross
Registered: Oct, 2006

Re: Mixins considered harmful/2 Posted: Jan 7, 2009 3:35 AM
Reply to this message Reply
First, I'd like to clarify something. When you say generic function, you mean it in the lisp sense, not the java sense, correct? (See http://en.wikipedia.org/wiki/Generic_function)

I'm finding this discussion tied fairly tightly to python (probably because I don't know python very well.) I think you are saying that people shouldn't use mixins to drag in "convenience" features, logging being a canonical example, because those methods are exposed publicly as well, polluting the object's name space. Is that correct? If so, I can't see any objections to what you are saying. An object's public interface shouldn't include implementation details, I think we can all agree.

In my simple, java influenced, caveman-like brain, I look at your first example and think "Well, there should be a Logger object stored as a field on the object." No fuss, and only a bit of muss: logger.warn( "hello" ) rather than warn( "hello" ), and all logging functionality stays bottled up inside that little object. Maybe I'm underestimating the muss? What's your opinion as a python guy?

Also, regarding name space pollution, in the java world I find that I have something of an inverse of that problem: libraries tend towards decomposition into many small classes and you end up having to work with a whole host of objects to get anything done, rather than work with one object that knows how to do quite a few things. I've started leaning towards what some people might term "G*d" objects (so long as they are tastefully composed, of course) where a lot of related functionality is packed into a smaller number of classes. Maybe I'm just getting batty in my old age...

Cheers,
Carson

Achilleas Margaritis

Posts: 674
Nickname: achilleas
Registered: Feb, 2005

Re: Mixins considered harmful/2 Posted: Jan 7, 2009 5:32 AM
Reply to this message Reply
I agree that when we simply want to use some functionality within a class, we shouldn't pollute the namespace of the class with the functionality we want to use. I don't see why classes should inherit the logger instead of simply using it, in your example.

Simplicity should be the number 1 target for library developers. Complexity is what kills software.

Michele Simionato

Posts: 222
Nickname: micheles
Registered: Jun, 2008

Re: Mixins considered harmful/2 Posted: Jan 7, 2009 5:44 AM
Reply to this message Reply
> I agree that when we simply want to use some functionality
> within a class, we shouldn't pollute the namespace of the
> class with the functionality we want to use. I don't see
> why classes should inherit the logger instead of simply
> using it, in your example.

You are right, a functionality as generic as a logger should be left outside a specific class (this is exactly my point when I talk about generic functions). However, here I assumed there was some reason to have a .log method: typically it could be an interface requirement coming from an external framework.

> Simplicity should be the number 1 target for library
> developers. Complexity is what kills software.

Indeed.

Daniel Armak

Posts: 2
Nickname: danarmak
Registered: Jan, 2009

Re: Mixins considered harmful/2 Posted: Jan 7, 2009 6:09 AM
Reply to this message Reply
> You are right, a functionality as generic as a logger should be left outside a specific class (this is exactly my point when I talk about generic functions). However, here I assumed there was some reason to have a .log method: typically it could be an interface requirement coming from an external framework.

Wouldn't it still be neater to do:

class Foo(object):
logger = Logger(...)
log = logger.log # expose to external consumers

Daniel Armak

Posts: 2
Nickname: danarmak
Registered: Jan, 2009

Re: Mixins considered harmful/2 Posted: Jan 7, 2009 6:21 AM
Reply to this message Reply
...the idea being that the class may want to use other methods of logger besides log().

To handle the subclassing problem of having different loggers we might do instead:
def log(self, *a): self.logger.log(*a)

Which is heavier, admitted, but still readable.

I don't have any experience with large systems using generic functions. My intuition may well be wrong, but what I feel is that generic functions only move the namespace clutter to a different namespace. Instead of having a lot of mixed-in methods in each object, we end up with even more generic functions in the namespace of every module using those objects. This tends to lead to import * statements, too.

What's needed IMO is a way to compromise in the direction of plain old object composition.

robert young

Posts: 361
Nickname: funbunny
Registered: Sep, 2003

Re: Mixins considered harmful/2 Posted: Jan 7, 2009 2:34 PM
Reply to this message Reply
> Maybe I'm just getting batty in my old age...

No, just maturing. Read Allen Holub, one of the few who write about OO as if objects were not either Action objects or Data objects (there be dragons). He's not popular with the framework folk, of course. Some others are starting to get the point; Dave Thomas off the top of my head.

Of course, for most folk, bifurcating the world into Action and Data objects permits one to claim to be an OO programmer while still doing the same old procedural stuff. Only, the spaghetti gets denser when you do it that way.

OO is about semantics, not syntax. And now we have Haskell making some noise, all the while some corners of the OO world claim that FP was replaced by OO. Yet they embrace xml as datastore, which is just a bad rehash of the hierarchical model of IMS, from 1964. Religion.

Jeff Heon

Posts: 40
Nickname: jfheon
Registered: Feb, 2005

Re: Mixins considered harmful/2 Posted: Jan 8, 2009 12:02 PM
Reply to this message Reply
> No, just maturing. Read Allen Holub, one of the few who...
Ok, found him:
http://www.holub.com/

Very interesting!

> ...are starting to get the point; Dave Thomas off the top of
> my head.
Which one?
http://pragdave.pragprog.com/
http://www.davethomas.net/

Jules Jacobs

Posts: 119
Nickname: jules2
Registered: Mar, 2006

Re: Mixins considered harmful/2 Posted: Jan 8, 2009 2:05 PM
Reply to this message Reply
Would integrating mixins with the module system solve the problems? You could make methods from mixins visible only in a particular namespace (just like generic functions without the additional dispatch).

Daniel Berger

Posts: 1377
Nickname: djberg96
Registered: Sep, 2004

Re: Mixins considered harmful/2 Posted: Jan 8, 2009 2:22 PM
Reply to this message Reply
In Ruby I solved this "problem" by redefining include so that you can selectively include or exclude methods from any given module. You can even alias them on the fly.

A method redefinition will cause a warning, so you at least get a clue that something bad may have happened.

http://rubyforge.org/docman/view.php/735/2472/examples.html

Nope, no need for generic functions just because mixins broke on one huge project.

Michael Askew

Posts: 1
Nickname: ransom
Registered: Aug, 2005

Grammar Nazi Posted: Jan 22, 2009 1:43 PM
Reply to this message Reply
Good article. I won't stop writing mixins any time soon, but it is important to remember that any technique can be overused and misused.

Your excellent writing and vocabulary notwithstanding, please proofread before publishing. I found a number of mistakes distracting.

"a week design"
"I find most unpythonic" ("I find it most unpythonic")
"people coming from other languages does not know" ("do not know")
"functionalies" ("functionalities")
"I will ave to pass"
"want the external method to ends up" ("end")

Flat View: This topic has 10 replies on 1 page
Topic: Yahoo Groups Set a Standard for User Frustration Previous Topic   Next Topic Topic: Video Interview


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us