My two posts on adding optional static typing to Python have been widely misunderstood, and spurred some flames from what I'll call the NIMPY (Not In My PYthon) crowd. In this post I'm describing a scaled-down proposal with run-time semantics based on interfaces and adaptation.
Let's go back to the basics. A function is defined like this:
def foo(x: t1, y: t2) -> t3:
is more or less equivalent to this:
def foo__(x, y): # original function
def foo(x, y): # wrapper function
x = adapt(x, t1)
y = adapt(y, t2)
r = foo__(x, y)
return adapt(r, t3)
Here t1, t2 and t3 are expressions that are evaluated once, at function definition time (i.e. at the same time as argument default values). Also, the wrapper doesn't really access the original by name; more likely it is an object that has a reference to the original function somehow.
The types are standard Python expressions, there's no separate syntax for type expressions. A type can be combined with a default value:
def foo(x: int = 42):
The default value gets adapted to the given type at function declaration time.
A function has a __signature__ attribute from which the names, types, and default values of the arguments can be introspected, as well as the return type (and the types for *args and **kwds, if specified).
Until the parser has been modified to accept this syntax, we can experiment with decorators like this:
The metaclass gives C.foo the __signature__ attribute and adaptation wrappers gleaned from the interface at run time. You don't have to specify argument and return types in the interface declaration; if the type is absent adapt() is not called.
The interfaces don't show up in the __bases__ attribute of the class; rather, they show up in a new __implements__ attribute (the metaclass can tell the difference between an interface and a class).
This proposal is even simpler than PEP 245 (Python Interface Syntax); I don't think the "implements" keyword proposed there is needed.
This has received an inordinate amount of attention in the discussion forum, but I'm not very impressed with the resulting designs. There are basically two styles of proposals: statement-based and expression-based.
I think the expression-based proposals are too limited: they don't handle guards involving multiple arguments very well, and the proposed overloading of type expressions and boolean guards feels error-prone (what if I make my guard 'True' while awaiting inspiration for something better?). Also, there are clear use cases for guards that (in Python) can only be expressed using multiple statements.
But the statement-based designs are pretty cumbersome too, and I expect that in practice these will be used only in large projects. At the moment I am leaning towards not defining any new syntax for these, but instead use a decorator until we've got more usage experience. Here's a strawman proposal:
def _pre_foo(self, a, b): # The pre-condition has the same signature as the function
assert a > 0
assert b > a
def _post_foo(self, rv, a, b): # The signature inserts the return value in front!
assert rv > b
@dbc(_pre_foo, _post_foo) # design-by-contract decorator
def foo(self, a, b):
In this example, _pre_foo and _post_foo are just names I picked; they are associated with the foo method by the @dbc decorator.
An alternative proposal could use an implicit binding based on naming conventions; then pre- and post-conditions could automatically be inherited, but the metaclass has to do more work.
But if you really want my opinion, I think these should not become a part of standard Python just yet -- I'd rather see others experiment with the ideas sketched here, write a PEP, and then we can talk about standardization.
I'm dropping the advanced and untried ideas for now, such as overloaded methods, parameterized types, variable declarations, and 'where' clauses. I'm also dropping things like unions and cartesian products, and explicit references to duck typing (the adapt() function can default to duck typing). Most of these (except for 'where' clauses) can be added back later without introducing new syntax when people feel the need, but right now they just act as red flags for the NIMPY (Not In My PYthon) crowd.
Most importantly, I'm dropping any direct connection to compile-time type checking or generating more efficient code. The adaptation wrappers will slow things down -- a price some people will gladly pay for the flexibility offered by adaptation and better run-time error checking. I expect that interface declarations will be helpful to PyChecker-like static bug finders and to optimizers using type inferencing, but these will have to deal with pretty much the entire range of dynamic usage that's possible in Python, or they will have to explicitly say that certain programming styles are not supported.
PEP 246 (Object Adaptation) has lots of good things to say about adaptation that I won't repeat.
When deferring to adapt() for all our type checking needs, we could give built-in types like int and list a suitably wide meaning. For example:
def foo(a: int, b: list) -> list:
This should accept a long value for a, because (presumably) adapt(x, int) returns x when x is a long; and it should accept any sequence object for b. But what if I have a sequence object that doesn't implement sort()? That method isn't used here, but it's defined by the built-in list type, so won't the default duck adaptation to list fail here?
There are a few interesting ideas here (e.g. Eiffel conformance), but in practice we'll likely end up declaring a bunch of standard interfaces that finally define carefully what it means to be an integer, sequence, mapping, or file-like object (etc.), and we'll be writing things like this instead:
The interface portion will need some fleshing out, as far as whether people will be able to define their own interface types, or whether it's even 100% necessary to have a special interface type at all. For "pure" PEP 246 purposes, plain old abstract classes would suffice, and no special syntax is needed for that. (And by implication, neither is any of the potential bickering over what features a built-in interface type should have. :)
Then there are little things like the distinction between instance.__implements__ and class.__implements__ which bedeviled both Zope and Twisted's implementations at one time or another. (Not that they do now; but it looks like your proposal might be based on Zope interfaces circa PEP 245, which is a few generations ago for Zope's interface implementation, and PEP 245 hasn't been kept up to date.)
But I think all of the open issues are quite resolvable, and even without an official interface type, this sounds at least like a blessing on PEP 246 - which means that maybe I should polish off the C implementation of 'adapt()' I have floating around in PyProtocols and prepare a patch for 2.5. :)
An interesting question there is whether __conform__/__adapt__ should actually become type slots, not just special names. Becoming slots would probably be beneficial for performance if adaptation is widely used. On the other hand, the non-slot nature of __conform__ currently allows PyProtocols to define per-instance adapters for modules and functions, to indicate that they implement a particular interface or to define how to adapt to that interface. Zope also does per-instance interface declarations like these, but using a different mechanism. (Because Zope considers "implements" to be the fundamental concept and "adapts" as derived from that, while PyProtocols goes the other way and says that "implements" is just a special case of "adapts to" where no adapter is needed.)
Anyway, I think these are all *good* questions to have, because I think they're reasonably answerable ones in the near term. :)
Btw, my condolences on being misunderstood/flamed; if it's any consolation, I think it's inherent to blogging, as I have been discovering in the last few months. The bigger your audience, the more likely that you will be misunderstood, misquoted, and flamed for things you didn't even say, or get figuratively called on the carpet for ideas you were just exploring but people took as some kind of gospel proclamation. On the other hand, you'll also get insightful helpful comments from quarters you never expected. Welcome to the internet! :)
> I hope you will consider the syntax again though. Notice > this sample, posted earlier: > > def draw(items: iter, canvas): > for item: Shape in items: > item.draw(canvas) > > Your eyes stop at each colon, which isn't a good thing in > this case.
He's not proposing it for arbitrary variable bindings; just instance attribute definitions in class bodies. Not for class attributes (unless you put them on a metaclass), and not for local or global variables. Just functions, methods, and instance attributes. These are the only places where it makes sense to add them; in normal code 'adapt(x,IFoo)' or 'IFoo(x)' suffices to indicate your intent, and it's not going to give a type inferencer any new problems to solve that it doesn't already have.
(The latter syntax, btw, is a shortcut currently used in Twisted, Zope, and PyProtocols for 'adapt(x,IFoo)' when 'IFoo' is an interface object.)
First off, I am disappointed people flipped out so much over your blog posts. People just didn't seem to grasp they were just brain dumps on your part for ideas. There were not PEPs, they had not been thrown to python-dev to be torn apart, nore officially presented to the python community to comment upon. Hopefully people will come to realize this and let up a little so that you can at least share any and all ideas, reasonable or not, with us all.
Anyway, on to the topics covered:
- Argument and return type declarations
I already liked the idea and this just keeps it going. Moving it explicitly to use 'adapt' is great and making it all run-time is also good since that will help discourage extraneous usage.
- Attribute declarations (maybe)
I think these are going to be important if thorough coverage of type checking is wanted. If people go with heavy OOP design then checking function signatures will not be enough; checking instance attributes will definitely be needed.
- Interface declarations
Once again I was already happy with what you were initially laying out and this version is still good with me.
I think once we have a stdlib module with common interfaces for sequences, iterables, iterators, lists, ints, etc. these will be used extensively with run-time type checking and help alleviate defensive hasattr checks and even EAFP code by doing it at the entry and exit points of methods.
- Design by contract (maybe)
I say give a decorator a shot (should probably start thinking about a decorator collection module in the stdlib, huh?). The only reason I am thinking that above a metaclass is that random chance of reusable conditions are then possible. Some people might come up with auto-generated conditions and thus want to be able to pass that in to a decorator than do a step of wrapping that generation in an explicit method definition.
I have to say proposing controversial changes to Python and then complaining when people react is a little disingenuous. Especially by using an emotive phrase like NIMPY.
I have a huge problem with adding typing to Python. Python's strength is its agility and its dynamic behavior. Adding code to essentially help the compiler increases the amount of code I write, decreases the adaptability of the code (if as I have often done decide that the way I pass values around has changed - I now will need to go round and change types, etc. etc.).
You may argue that this is all optional. I think that this is also a very disingenuous argument - it might be optional in the next version of python but as people start to use it , it will be required. Style guides will recommend it , new comers from other languages will use it out of habit and like design patterns and over-engineering in java it will become the culture of the language.
The question that has to be asked is where is all this going? Is Python stepping up to the plate to become the next Java (or C#) and take on enterprise culture. Or is it a slick and different way of doing things that gives a real edge.
Having worked in many languages for more years than I care to remember it seems to me language culture is just as important as the language itself. Part of my problem with Java (in which I have designed and built a number of enterprise scale products) is the culture that guides it is one that assumes the programmer is essentially stupid - and that he or she needs to be protected from this. This ends up with a language that can only be efficiently written in an IDE that in turn protects you from the quite considerable amount of work needed to protect the programmer from themselves "help": types, structure, etc.
Python is *not* like this at the moment. It assumes that its practitioners can think for themselves and in return you get to write less code which you can create faster. I have seen no evidence either from direct experience or from research that shows this produces less production ready code than Java. In fact my personal experience (which obviously not real evidence) is that its the other way round.
This is not just a "not in my python" argument: I am embarking on a big product soon - I have currently chosen Python because of the edge it will give our company. I have to say I am now worried about this choice - if python goes down this route and I loose the commercial edge that writing in Python gives us then I need to rethink our whole strategy.
I've been doing this far to long not to recognize warning signs when I see them.
Hey, Guido... I just wanted to say two things. First of all, this sounds pretty good! I'm still having difficulty fully grocking how adaption will work. Yes, I see the obvious uses, but once those are in place, how will it influence people? Will people feel forced to use adaption? How will it alter their behavior? But even allowing for such reservations, it's pretty clear that adaption is useful, and this syntax would make it quite useful. It seems to me that you started out with a wide and ambitious set of ideas and over the course of a week managed to pare it down to the simplest and best ideas at the core. (That doesn't preclude adding more ideas later, just starting with the core.)
The second point I wanted to make is that this is a pretty good way of working. I want to publicly THANK you for putting up these postings, and for putting up with the flames and complaints that resulted. It may be an annoying process, but it's a productive one. Please don't let the flames scare you off.
(PS: I've got some other thoughts, particularly around the definitions of standard types for Python, but I think perhaps I'll wait until there's a PEP before chiming in with those.)
> Most of these (except for > 'where' clauses) can be added back later without > introducing new syntax
Just to demonstrate this point, Guido's previous post, and some helpful feedback I got from PJE led me to come up with the following classes:
class strict(object): # Explicit type check def __init__(self, face): object.__init__(self) self.face = face
def __adapt__(self, obj, params = None): obj = adapt(obj, self.face) # True strictness would need a check that disallows # instances of subclasses in the next line if not isinstance(obj, self.face): raise AdaptError("Strict adaptation requires instance") return obj
class any_of(object): # Interface union def __init__(self): object.__init__(self) self.faces = faces
def __adapt__(self, obj): for face in self.faces: try: return adapt(obj, face) catch AdaptError: pass raise AdaptError("Unable to adapt to any interface")
def __adapt__(self, seq): return tuple(adapt(x, face) for x, face in zip(seq, self.faces))
class any(object): def __adapt__(self, obj): return obj
any = any()
Hopefully what these are for is fairly self-explanatory.
Parameterised types can also be done without any new syntax. However, Guido's suggested __getitem__ method for type (which looks like new syntax, even though it isn't really) makes them a lot easier to use. Using list and dict as examples:
class parameterised_face(object): def __init__(self, face, *params): # Hopefully the next line can be replaced with # something based on __signature__ if face.__adapt__.func_code.co_argcount < 3: raise TypeError("Unparameterisable interface") self.face = face self.params = params
class type(object): # Rest of type's definition. . . def __getitem__(self, params): if isinstance(params, tuple): return parameterised_face(self, *params) else: return parameterised_face(self, params)
class list(object): # Rest of list's definition. . . def __adapt__(self, obj, item_face = None): try: obj = list(obj) catch StandardError, ex: raise AdaptError(ex) if item_face is not None: return [adapt(x, item_face) for x in obj)]
class dict(object): # Rest of dict's definition. . . def __adapt__(self, obj, key_face = None, value_face = None): try: obj = dict(obj) catch StandardError, ex: raise AdaptError(ex) if key_face is not None: if value_face is None: return dict((adapt(k, item_face), v) for k, v in obj.iteritems())) else: return dict((adapt(k, item_face), adapt(v, value_face)) for k, v in obj.iteritems()))
This seems a reasonable proposal. Calling it "Optional Static Typing" doesn't seem especially accurate, though :)
As for your previous posts being misunderstood, I thought your first post on the topic was pretty hard to understand! That's not to say people were justified in seeing all their pet hates in it and ranting, but I certainly wasn't sure what your real goal was.
I don't really see the point of attribute declarations. Something about the auto-adaptation of return values strikes me as a bit odd (not sure just why, though).
And also, the real issue is clearly going to be the details of interfaces and adaptation. I don't know so much about that, but I guess the twisted and zope crowd, at least, do by now...
I really like the direction typing in Python is heading with the focus on adaption.
The one issue I have is about the proposed syntax of "subclassing" for signalling that an interface is implemented by a class. I prefer something more like the zope.interface style:
class Foo(Base): implements(IFoo)
class Foo(Base) implements(IFoo): pass
(although I prefer the first version).
implements() makes it clear that Base is a class and IFoo is an interface. This is especially true if the interface name does not begin with an 'I'.
I also suspect that API documentation generators that only parse the source code (i.e. don't walk the AST) will have a much easier time with implements(). Unless IFoo is already known to be an interface, the parser has no way of knowing what to do with it and will probably have to assume it's a class. If IFoo is listed inside implements() it is unambigous.
Mark Williamson:"I have to say proposing controversial changes to Python and then complaining when people react is a little disingenuous. Especially by using an emotive phrase like NIMPY."
Sorry to sound defensive, but you're still missing the point. I was (and still am) thinking out loud and (I thought) made that pretty clear. I was hoping to spur people's thoughts. Instead, several folks (e.g. Chris Petrilli) wrote their own blog representing my post in the worst possible light and then tearing it apart to look cool.
"Adding code to essentially help the compiler increases the amount of code I write, decreases the adaptability of the code"
Did you read and understand the proposal? It is not there for helping the compiler, it is a concise way to cause adaptation to happen automatically at run-time. Adaptation is something that large projects like Zope and Twisted already use with great success.
Michael Hudson:"Calling it "Optional Static Typing" doesn't seem especially accurate, though :)"
I know; I figured it would be better to keep it in the title for continuity with the previous two posts. The next installment may be titled "Optional (Not So) Static Typing" :-)
Doug Holton:"I hope you will consider the syntax again though."
In my first two posts I tried to discourage a syntax debate because it tends to never end until I break the tie, so I might as well just stick to my own preference and avoid the whole hullabaloo. :-)
But since 'as' keeps being proposed: the problem with using 'as' is that Python already uses 'as' with a very different meaning: like in SQL, "import X as Y" is a local renaming of X to Y. Using the same keyword for type declarations is confusing; it would also preclude adding optional type declarations to imports in the distant future (not an entirely unreasonable extension).
Besides, I'd rather inherit from Pascal than from VB. :-)
I am a Python neophyte. I have been researching and genuinly enthuastic about Python for the past month. I am a professional Delphi developer, and have used Java, C/C++, x86 asm, basic etc for nearly 10 years.
I have one question and possible suggestion:
One of the things that I REALLY like about Python as I am exploring it, and, ultimatly what attracted me to it, is the common-sense nature of the language. The main turning point was indention based scoping. A lot of developers hear this and freak out, but the more I thought about it, the more it just made sense. Humans read the scope that way, why not require the compiler to do the same? I love it.
I am not arguing the debate about putting OPTIONAL static typing in the language, there are FAR more people with much more experience in that arena. I hope it stays optional. I do have an idea about implementation, that, on the surface seems "pythonic".
Most developers use a prefix notation when declaring a variable of a certain type. Why not us this as part of the language, as indention scoping is? I don't know how this would effect function signatures and such, but just ponder:
iSomeVariable: int or def gcd(a: int, b: int) -> int:
def int_gcd(int_a, int_b) or int_SomeVariable
Obviously int_[code] could be anything ... [code]i_ ... but I didn't want to promote a cryptic syntax.