The Artima Developer Community
Sponsored Link

Weblogs Forum
Adding Optional Static Typing to Python -- Part II

59 replies on 4 pages. Most recent reply: Jun 21, 2005 2:42 PM by A. Ellerton

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 59 replies on 4 pages [ « | 1 2 3 4 | » ]
Ka-Ping Yee

Posts: 24
Nickname: ping
Registered: Dec, 2004

Re: no trailing colon means no implementation Posted: Jan 4, 2005 7:27 AM
Reply to this message Reply
Advertisement
True, it's better to have a docstring in the method body, so we can drop the no-trailing-colon idea.

This reminds me of an idea i heard about in a conversation a while ago... how about defining "..." as a shorthand for "raise NotImplementedError"? Perhaps making it very easy to type this particular idiom might encourage people to be explicit about parts of a program that were intentionally left unimplemented.

Rich Salz

Posts: 1
Nickname: rsalz
Registered: Jan, 2005

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 4, 2005 7:38 AM
Reply to this message Reply
Holy crap. Python has become very succesful with the current type system. And you are now considering doubling the complexity of the language. (I mean that seriously; by the time you nail down all the dark corners of your proposal it will be comparable to C++. Not equivalent, but comparable.)

Don't do this. It might help a few, and it will hurt most. You won't be able to wrap your brain around it any more. They "pythonic" way of doing things will fork into Python folks and C++ (okay, maybe C#) folks. Other important implementations (Jython, IronPython) may *never* be able to catch up, further splitting the community.

Something as drastic and involved as this should have overwhelming, screaming, everyone-agrees demand. Have you seen that?

Instead, please invest the significant time and brain-power into working on the runtime library. For a long time the batteries were included, but they're old and leaking now. Why doesn't python have world-class XML support bundled, as an "obvious" first thing?

Todd Blanchard

Posts: 316
Nickname: tblanchard
Registered: May, 2003

Optional typing Posted: Jan 4, 2005 9:13 AM
Reply to this message Reply
Just thought I'd throw in a suggestion that you look at how ObjectiveC does typing. Its completely optional but if you use it then the compiler will generate warnings when you are doing something obviously bad. Untyped object references are of type id. Typed ones are pointers like NSString*.

Interfaces are called protocols. If your class claims to implement a protocol, then the compiler will warn if you don't implement the whole protocol. Additionally, you can perform a runtime check by sending an instance the conformsToProtocol: aProtocol message.

That seems to provide the type checking benefits desired by the static crowd while allowing totally dynamic development when appropriate. Seems like a sweet spot to me.

Pierre-Yves Martin

Posts: 17
Nickname: pym
Registered: Jan, 2005

Static stuff in python Posted: Jan 4, 2005 9:33 AM
Reply to this message Reply
Very interesting posts... and I would like to summarise everything and cleanup some few thing I said/understand.

Adding static typing and other static stuff in python

All the posts here are in fact about static addition to python (not only static typing). There are many different static improvement that have been discussed:
- static typing
- static interface (that is liked with dynamic adaptation)
- static contract for methods
- static definition for nested methods

All those stuff could be very usefull to improve the python language but have some important drawbacks sometimes... let's study each of them first.

Static typing

Static typing as proposed by Guido seems attracting but some very big drawbacks could appear...
advantages:
- give some "easy read" documentations about the params
- enable some very good optimisation
- allow to detect/correct some bugs with very few effort
disavantages:
- hard cohabitation between dynamic/static typing
- risk of splitting community
- isn't interface checking more "pythonic"
- could increase the complexity of python language with poor benefit

static interface

Static interface consist of defining behaviour that some object have to implement so that they could be used in peticular situations.
This could be achieved by using an interface hierarchy and by allowing classes to implement those interfaces.
Then at runtime or compile time you could think of a way to check (compile time) or adapt (run time) the object interface.
advantages:
- allow to define the behaviour of a class independently of the class inheritance.
- it provide a very smart type system to python
- widely discussed in PEP 246 and some zope implementation of adaptation
disavantages:
- this kind of "typing" is not really easy to use... it must be considered as an advanced feature in python
- if we adapt at run time we cannot do any "static type checking" this way... losing all the advantages of it

static contract for methods

Another static stuff: defining contract (pre/postconditions) for methods. It is something static that is not easy to do in python except if you considere the pre/post condition has a part of the method code itself (what most languages do).

We have many implementation proposed in this topics... but all of them share the same principle: definiting pre/postcondition for methods so that we can check in/output. This must be easy to do and could be easyly separated from the code itself... for documentation purpose for exemple.

advantages:
- increase readability of code
- allow to extract some interessant doc from code (such documentation is always uptodate because if not the code do not works properly)
- increase reliability of code (due to Design By Contract)
disavantages:
- is it so useful to add a new language feature for this... it is possible to do such thing using some assert statements
- could increase the complexity of python language with poor benefit

static definition for nested methods

it appeared that it was also interesting to have the possibility to define staticaly nested methods whereas for the moment any nested declaration is done everytime we call the method where it is nested (for backward compatibility it would not be a good idea to change this).

I proposed a way to do this by an extension of the "global" keyword that would apply also on nested method definition.

advantages:
- this allow clean definition of well scoped nested function
- could be used for defining pre post condition
- could be a solution for many static stuff... (see further)
disavantages:
- heavy syntax
- maybe I'm the only one in the world to think it's a good idea (maybe my roomy agree with me.... he's still thinking about it...)

All I wrote here is not my opinion or what I personnaly fight for... it's just a summary of the topic (I may have forgotten a lot of things sorry for that) so that we can discuss thing in a well known context: static improvements in python!

(in my next post I will give my opinion and possible solutions)

Michael Chermside

Posts: 17
Nickname: mcherm
Registered: Jan, 2004

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 4, 2005 10:08 AM
Reply to this message Reply
I've been watching this discussion, and I'd like to chime in (but only if it's helpful). Unfortunately, this is a very hard problem, and I don't think I can do more than just peck away at the edges of the problem. Perhaps such input will be helpful.

First, a low-hanging fruit. You write Let's get rid of unbound methods.... That's a good idea, it's PEPable, and it's not tied in with the rest of type checking. I'd say do it, but don't relate it ty type checking.

A few misc thoughts. My initial impression is that I particularly like the use of '|' '&' and '*' operators to combine types. I like 'list[T]' as a syntax for collections, but I think it's important that user-defined collections have first class support. At first glance, this is easy: 'list[T]' generalizes to 'myClass[T]' when myClass is a container. But it's NOT so easy... for instance, dict has TWO types (that of keys and of values), and user-defined "containers" may be arbitrarily complex. For that matter, I'm not even sure what the definition of a "container" is (other, perhaps, than a type which is parameterized over some other type or types). After the effort to unify types and classes, I wouldn't want to go back and make classes second-class citizens again. Sorry... no good answers here, just questions.

Another thought: I like interfaces... they seem like the right tool for the job. But I think you might be making them a bit too strict... in a language "trust that the programmer knows what she's doing" is a design principle. For one thing, I know all about covariant arguments and contravariant return types, but do we really need to enforce something like that? What if we said that sub-interfaces must simply have the same methods (I suppose I mean methods and instance variables), without enforcing details about whether such methods are perfectly Lyskov substitutable? I'm not expecting full type inference a la ML to ever work in Python (as you say, the main goals are expresivity, documentation, etc)... so do we really need the full machinery?

Okay, enough questions without answers for now. I think it's an interesting start, but it seems FAR from PEP stage right now.

Michael Bernstein

Posts: 2
Nickname: webmaven
Registered: Jan, 2005

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 4, 2005 11:29 AM
Reply to this message Reply
I'm not sure what to think of this proposal, but I have a feeling it will largely increase the complexity of code produced by people coming from other languages, with no attendant benefits (for that particular code anyway).

In short, it will just make it easier to write unmaintainable Java code in Python.

That said, I should stress that my perspective is that I definitely want to reduce the complexity of other people's code that I encounter (and occasionally have to maintain), and the prospect of encountering gratuitous (and potentially automatically generated) type declarations does not fill me with joy.

Can we just try out interfaces first, and then revisit type declarations after they've been digested by the community?

"Note: I'm not quite sure that I really like this operator syntax; I've toyed with alternatives, like union[T1, T2] instead of "T1 | T2", but I'm not keen on intersection[T1, T2] or cartesian[T1, T2] (both too long) and I can't seem to find shorter words. For now, the operators will do."

How about using and[T1, T2], or[T1, T2], & xor[T1, T2] ?

Pierre-Yves Martin

Posts: 17
Nickname: pym
Registered: Jan, 2005

Re: Static stuff in python Posted: Jan 4, 2005 11:55 AM
Reply to this message Reply
For me a very important thing is to keep python simple and to leave the choice to the user to use or not all the stuff.

So here is a suggestion: why not use a static definition system that would allow to define nested method and use that system to do most of the static stuff we've talked about?

Static method definition

Static nested definition syntax exemple (here the syntax is not important but I have to use one to explain what I mean):

def foo(arg1, arg2):
global def nested_method(a, b, c):
#code of the nested method
#code of the foo method
#you can use the the nested method here if you want

in fact it is the equivalent of:

def foo(arg1, arg2):
#code of the foo method
def quite_nested_method(a, b, c):
#code of the quite nested method
foo.nested_method = quite_nested_method

except that here we can not use the nested method inside the foo method itself...

In fact what we did is a definition a static method in the function object (some kind of...). But it has been done without the need of defining a callable class.

What would be the implementation of this? A good thing would be that any "global" definition in a method would define an attribute to that method (if this is two too diferent concept then we need another syntax to define function attribute).

Usage for static nested definition

For exemple we want to have pre and post conditions we just have to use reserved noun of methods:

def foo(arg1, arg2):
"""docstring"""
global def __pre__():
assert arg1 > 0
assert arg2 > 0
global def __post__(result):
assert result is not nil
#here we have the method body

All this is perfectly compatible with a interface definition because interface definition is at compile time so everything in it have to be static and the "global" new statement *is* static.

interface MyInterface:
"""docstring"""
def foo(arg1, arg2):
"""docstring"""
global def __pre__():
assert arg1 > 0
assert arg2 > 0
global def __post__(result):
assert result is not nil


how to deal with type checking with no changes in the language
How to impleme,nt a good type checking system without changing the language itself? It is possible to use decorators to make some type checking cf PEP 318 --> accepts and return decorators.
If those 2 decorator are just made builtins... it would be possible to use them for optimisation, documentation...
here we have a static declaration of type but runtime checking...

@accepts(int, int)
@returns(list)
def foo(arg1, arg2):
"""docstring"""
global def __pre__():
assert arg1 > 0
assert arg2 > 0
global def __post__(result):
assert result is not nil
#code of foo here

Like this it's further from the C/C++/java syntax and therefore is less prone to create a war between the pro and anti static typing... than the Guido proposal:

def foo(arg1: int, arg2: int)-> list:
"""docstring"""
global def __pre__():
assert arg1 > 0
assert arg2 > 0
global def __post__(result):
assert result is not nil
#code of foo here

You may think it's just syntax stuff... but it is also a different view of the type checking... python *must* stay a dynamicaly typed language (according to somes) but we want to *optionnaly* specify the type or interface accepted as input or output (for many purpose). If we change the syntax we have something to near of static language syntax.

Ka-Ping Yee

Posts: 24
Nickname: ping
Registered: Dec, 2004

Does "optional" really mean optional? Posted: Jan 4, 2005 12:21 PM
Reply to this message Reply
I see several negative reactions to potential complexity here, which may be a useful warning flag. Even though the syntax may be an optional feature in the sense that an author can choose not to use it, a syntax feature is never completely optional since anyone who looks at code is going to have to know the feature to understand what the code means. The reference manual will grow, the implementation will grow, tutorials and classes will grow, and the "fits in my head" footprint will grow. So staying conservative in terms of new concepts is probably wise. (Or as i mentioned before, try to make sure each step in increased complexity is driven by a burning practical need.)

1. Adaptation is perhaps the lowest hanging fruit, since it requires no syntax changes and provides useful flexibility.

2. Interfaces may be a possible burning need. It's worth considering how far you can get just by adding interfaces without type declarations. The ambiguity of "standard" protocols (sequence, mapping, etc.) has been bugging people for some time, and it would be nice to settle some of those things cleanly, which is what motivated my suggestion of including mixin code in interfaces.

3. Preconditions and postconditions might be the next low-hanging fruit after that. (I consider methods defined on the interface a smaller step in complexity than nested functions that aren't redefined each time like normal nested functions.) I don't know how badly anyone wants this, though.

4. Types seem to me a lower priority than the above...

5. ...and parameterized types even lower than that.

Robert Brewer

Posts: 7
Nickname: fumanchu
Registered: Jan, 2005

Re: Preconditions and Postconditions Posted: Jan 4, 2005 1:35 PM
Reply to this message Reply
Ka-Ping Yee wrote:
> Suppose you establish a convention that calling any method >foo first calls the method pre_foo with the arguments to foo
> (if pre_foo exists); then calls foo; then calls the method
> post_foo with the value returned by foo (if post_foo
> exists).

I had the same idea. But IMO you shouldn't use special names--make new keywords (e.g. predef and postdef).

Michael Chermside wrote:
> You write "Let's get rid of unbound methods...."
> That's a good idea, it's PEPable, and it's not
> tied in with the rest of type checking. I'd say
> do it, but don't relate it to type checking.

Completely concur. Note that this is a case of *removing* unnecessary/impractical typechecking. ;)

Brett C.

Posts: 4
Nickname: drifty
Registered: Jan, 2005

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 4, 2005 1:46 PM
Reply to this message Reply
"There might also be a way to declare and add overloaded methods a la Java."

Perhaps, but I wouldn't want it. To start, I don't like overloaded methods; I have personally never found them useful enough to warrant having to search through multiple method definitions to find the one I care about.

But in terms of Python, I can see a major complexity issue with implementation. In Java you just have positional arguments (although in 1.5 I believe they have an equivalent varargs). In Python, though, we get to add keyword arguments plus kwargs. With so many options decided which direction to go could be messy. Do two methods defined with the same number of arguments always equate out to be the same, even if one has default args? What if two methods have the same signature thanks to the default args on one and the user calls using only the default args?

It just seems hairy without enough enough benefit to me.

Ka-Ping Yee

Posts: 24
Nickname: ping
Registered: Dec, 2004

Re: Preconditions and Postconditions Posted: Jan 4, 2005 1:55 PM
Reply to this message Reply
> I had the same idea. But IMO you shouldn't use special
> names--make new keywords (e.g. predef and postdef).

I thought about that a bit. That would be cleaner in one sense but uglier in another. You wouldn't have to invent a name-mangling convention for the method names, which would be nice. But how would you access the functions (e.g. so that you could write one precondition for a few methods, or have a subclass call the precondition in its superclass)? Unless you add corresponding keywords (or an operator) for that, which seems very unlikely to me, you're stuck having to invent another name-mangling convention anyway.

Actually it's worse than that. If you want the preconditions and postconditions to be inherited (which i currently believe they should), then either (a) they have to be attributes of the class, or (b) you have to add another inheritance mechanism. I assume that adding another inheritance mechanism just for this feature is out of the question. If they are attributes of the class, then they must have names — and that brings us back to name-mangling. I wish there were a cleaner way, but i can't think of one.

There are definitely shades of aspect-oriented programming going on here. It might be best to look at a general, simple AOP mechanism and let preconditions/postconditions be just one special case of that.

Marek Baczyński

Posts: 3
Nickname: imbaczek
Registered: Jan, 2005

Re: Does "optional" really mean optional? Posted: Jan 4, 2005 2:03 PM
Reply to this message Reply
> 1. Adaptation [...]
> 2. Interfaces [...]
> 3. Preconditions and postconditions [...]

As I see it, having 1, 2 and 3 gives you 4 (= static typing) and, to some extent, 5 (= parametrized types) for free (except for syntax.) That is, if we're going to go "the types way", let's do it in that order - features first, syntax later (just like with decorators.)

What I'm trying to say is that a feature only takes off when there's (standard) syntax for it, decorators being a prime example, True/False being another. That means that even if the type system is availible, it won't be overused, which I believe is it's biggest problem.

BTW: a way of doing macros would be of great help in testing proposed syntax variants... But just pretend I didn't say that ;)

mike bayer

Posts: 22
Nickname: zzzeek
Registered: Jan, 2005

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 4, 2005 2:55 PM
Reply to this message Reply
> Basically, if you require an actual type check, PEP 246 is
> DOA with respect to declarations. Here's why. Let's say
> I have a routine that takes an IFoo parameter, that I
> currently adapt to *inside* the routine, so that callers
> can pass a variety of items. For example, in PEAK there
> are certain interfaces that can accept either a filename,
> a URL, or a stream factory. There's a
> 'config.IStreamFactory' interface that many methods adapt
> their arguments to. This allows the caller freedom to
> pass whatever makes sense in context; they don't have to
> explicitly wrap the arguments in an 'IStreamFactory()'
> call.

why not:

do_something(stream : str | URL | IStreamFactory)

and again do the adaptation inside of do_something() ?

Pierre-Yves Martin

Posts: 17
Nickname: pym
Registered: Jan, 2005

Roadmap... Posted: Jan 4, 2005 5:26 PM
Reply to this message Reply
Interesting... after discussing quite a bit about the feature and their issue most of us now focus to the roadmap to follow. We're on the right way, we still don't know witch way... but it seems to be the right one.

I agree with many thing told before:
- first do not add new syntax but prefer using current syntax to do the stuff... then add a new syntax when needed by the intensive usage ^_^ (cf the decorator evolution)
- adaptation is the most flexible thing and it is really advanced right now so it's must be the first thing to do
- pre/postconditions are quite hard to deal with *but* could be a major improvement if well done (what means clean inheritance of pre/postcondition just as in eiffel or planned for the D language)

so here is the roadmap with exemple for each step... it's easier for me to understand like this:

1 - adaptation adaptation at runtime of parameters is really very flexible and the PEP is quite advanced... so let's do it first!

def foo(arg1, arg2):
#do the adaptation here at run time

2 - standard type checking decorators it is very easy to have such stuff (exemple implementation (but not standardized could be found in PEP 318)

@accepts(int, int)
@returns(list)
def foo(arg1, arg2):
#foo code here


now we have two major features... for low cost... let's deal then with harder stuff...

3 - advanced type operations let's make the type behave like sets (what is academicaly not so dumb ^_^ ). It would allow very good compliance between adaptation at run time and typechecking.

@accepts(str | url , int)
@returns(list)
def foo(arg1, arg2):
#do the adaptation here at run time


4 - interface/protocolshave a nice way to define protocols (I think this term is nearer of the exact purpose: definig a behaviour). This for me should define a new type (but not a class) and therefore it needs a new syntax... let's deal with the Guido's one (I like it it I do not see any other).

interface MyInterface:
def foo1(arg1, arg2):
"""do some stuff 1"""
def foo2():
"""do some stuff 2"""
def foo3(arg1):
"""do some stuff 3"""

And here MyInterface is a type not a class and it could be used like a type (including the "sets-like" operations)

5 - pre/post conditionwhat kind of syntax could be used without changing the syntax at all? Really I don't know... if I did I would have used it for a long time... here is a proposition I'm currently working on using decorator (yes decorators are my favorite toys ^_^ ). It allows definition of condition on the fly using lamda function or to reuse previously defined function.

def invariant1():
assert stuff is not nil

@precondition(invariant1)
@precondition(lambda arg1 : assert arg1 > 0)
@precondition(lambda arg2 : assert arg2 > 0)
@postcondition(lambda result : assert result is not nil)
def foo():
#foo code here

(this code is just here to show that it is possible to do something without adding new feature but implementation syntax *have to* be discussed.

5 - parametrised type It comes last because (for me) it's the less useful and it is quite complicated to have somthing good : syntax, usefulness, readability...

my_list1 = list[int]()
my_list2 = list[int | str]()


Now let's go further
So now it would be interesting to discuss where and how each point have to evolve... here all the exemples were simple exemples to show that we could tests many stuff before dealing with language change. For exemple it would be interesting to first follow existing PEP and create new ones. Then many feature could be include in the lib and then maybe the most useful feature could appear in the language it self.
I know it could be a long long way... but it is a safe and smart way!

sample of feature evolution

Here is an exemple of feature evolution: the static typing:
- proposition of implementation using decorators:

@accepts(int, int)
def foo(arg1, arg2):
#code here

- discussion/PEP leads to a modification of the prototype to have a more flexible thing:

@accepts(int, int)
def foo1(arg1, arg2):
#code here

@accepts(arg1 = int, arg2 = int)
def foo2(arg1, arg2):
#code here

- feature added to the library (yeepee!):

from decorators import *

@accepts(int, int)
def foo1(arg1, arg2):
#code here

@accepts(arg1 = int, arg2 = int)
def foo2(arg1, arg2):
#code here

- as it is commonly used by a lot of people a new PEP opens about adding this feature to the language or not and with witch syntax...
- if everything goes well the feature is added to the language:

def foo1(arg1 : int, arg2: int):
#code here


But it is just one scenario... anything could stop earlier... maybe it's not a good idea to go so far. Maybe decorator is enough, maybe noone will use this, maybe even the decorator version will be refused as a standard... (for me it would be stupid because it fulfill the requirement for static type checking/information and so enable any kind of documentation/optimisation... but it's just my opinion).

So let's discuss all those things!!!

daniel suarez

Posts: 2
Nickname: danielsu
Registered: Jan, 2005

Re: Roadmap... Posted: Jan 4, 2005 7:18 PM
Reply to this message Reply
> 2 - standard type checking decorators it is very
> easy to have such stuff (exemple implementation (but not
> standardized could be found in PEP 318)
>

> @accepts(int, int)
> @returns(list)
> def foo(arg1, arg2):
> #foo code here
>

>
> now we have two major features... for low cost... let's
> deal then with harder stuff...

adding any kind of "hack" to do something that should go in the core of the language for both,esthetic and tecnical reasons would be wrong, i have to agree with guido that new syntax is needed, this should not be a problem since anyone already familiar with a static typed language can get it right away, besides the syntax proposed by guido is quite elegant and easy to read.

And having a method bloated with 10 decorators is unnecesary verbose and hard to read, this things has to be used with measure.


> 5 - pre/post conditionwhat kind of syntax could be
> used without changing the syntax at all? Really I don't
> know... if I did I would have used it for a long time...
> here is a proposition I'm currently working on using
> decorator (yes decorators are my favorite toys ^_^ ). It
> allows definition of condition on the fly using lamda
> function or to reuse previously defined function.
>

> def invariant1():
> assert stuff is not nil
>
> @precondition(invariant1)
> @precondition(lambda arg1 : assert arg1 > 0)
> @precondition(lambda arg2 : assert arg2 > 0)
> @postcondition(lambda result : assert result is not nil)
> def foo():
> #foo code here
>

> (this code is just here to show that it is possible to do
> something without adding new feature but implementation
> syntax *have to* be discussed.

way too much decorators!! I think an elegant solution would be a strong name convention for functions which represents pre/post conditions.For example appending pre_ and post_


def foo(a:int):
pre_Something
#foo code
post_Something

Flat View: This topic has 59 replies on 4 pages [ « | 1  2  3  4 | » ]
Topic: Typesafe Enums in C++ Previous Topic   Next Topic Topic: Enhancing agile planning with abuser stories

Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2019 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use