The Artima Developer Community
Sponsored Link

Weblogs Forum
Adding Optional Static Typing to Python -- Part II

59 replies on 4 pages. Most recent reply: Jun 21, 2005 2:42 PM by A. Ellerton

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 59 replies on 4 pages [ « | 1 2 3 4 ]
Lenny Domnitser

Posts: 1
Nickname: ldrhcp
Registered: Jan, 2005

Re: Perspective from a Java developer Posted: Jan 5, 2005 6:52 PM
Reply to this message Reply
Advertisement
"if I had been learning Python and read that I could declare types for function parameters, but I didn't have to, I certainly would have assumed that it's better to do so. I think you'll find that people moving to Python will take advantage of whatever language features look most familiar. This might be great for experts who already understand the advantages of programming the Python way, but you might be throwing the newbies for a loop. A great way to teach Python philosophy is to require it in the language."

This is by far the largest problem I see with adding static typing to Python. Python should not try to be the best overall programming language for "frameworks or large applications" at the expense of polluting available Python libraries with un-Pythonic code.

For such large-scale projects there is Java, and if a large Java system must be scripted in Python, there is Jython. There is no need to introduce new features to Python to satisfy needs that are suited perfectly by Java.

Pierre-Yves Martin

Posts: 17
Nickname: pym
Registered: Jan, 2005

Let's find the possible solutions to have contracted functions Posted: Jan 5, 2005 7:51 PM
Reply to this message Reply
Here is the problem

- We want to define a contract (pre- and/or postconditions) for a function
- this contract is a static data about the function. It have to be defined once when we declare the function (if it's possible to change this the notion of contract is useless)
- in python a function definition means an instanciation of a function object with feeding of the different attribute of the object.
- the code of the function itself is just a data that is put in the func_code attribute of the function object. So nothing static could be put in there.

conclusion
- it is not possible to put any static stuff for a function in it's own code... (it's possible to use some hack to do so but in the idea it is useless: the code is a data that is executed when the __call__ method is executed on the function object that's all)
- we have to found a place to put static stuff for a function definition

let's keep hope !
We still have some static data access in a function definition. first let's look at them:
- the function name
- the parameters numbers and names
- the docstring
- the code
- we can add/access to the function attribute after it's definition
- the decorators

So how is it possible to use them for pre- post condition static definition?
- the function name --> uh? ok leave that as it is we can not change the function name of the cantracted function. But it is possible to use a naming convention over other methods so that they could be considered hass contract for this method.
- the parameter number and name --> they are too much used outside the function definition... and I see no way to use them
- the docstring --> sometimes it is used to add metadata to the documentation using a peticular syntax (cf doxygen for the C++ it's the one I know the most... but there are exemples in any languages). Soo we could use such syntax to type our contract code... but docstring is just a comment region so you should not put code in there it won't be executed and so won't be maintained.
- the code --> ok we already have deal with him some few line before: it's content is not staticaly executed... only the code itself is copied staticaly in an attibute of the function object...
- we can add/access to the function attribute after it's definition --> the problem here is that it is done after the function definition... so it's hard to have something very readable, a method body could be (even if should not be) very very long. It was the problem of the staticmethod() builtin function before python 2.4 decorator syntax.
- the decorators --> they are not very elegant I can admit it but one of their aim *is* to execute code at the function definition.

[final conclusion of this analysis[/b]
So after this quite deep analysis what do we have left:
- naming convention over other same-scoped function to make them cantract for the contracted method.

def pre_foo(arg1, arg2):
assert arg1 < 0
assert arg2 > 0

def post_foo(result):
assert result is not nil

def foo(arg1, arg2):
#foo code here

- decorators using extaernaly defined function as cantracts. This would prevent any kind of naming convention and enable the reusability of one contract function for many cantracts.

def invariant():
assert g != 0

def pre_foo(arg1, arg2):
assert arg1 < 0
assert arg2 > 0

def post_foo(result):
assert result is not nil

#syntax version 1: all-in-one contract
@contract(pre=pre_foo, post=post_foo, inv=invariant)
def foo(arg1, arg2):
#foo code here

#syntax version 2: one by one contract
@precondition(pre_foo)
@postcondition(post_foo)
@invariant(invariant)
def foo(arg1, arg2):
#foo code here

#syntax version 3: on-the-fly declaration using lambda
@precondition(lambda arg1: arg1 < 0)
@precondition(lambda arg2: arg2 > 0)
@postcondition(lambda result: result is not nil)
@invariant(lambda: g != 0)
def foo(arg1, arg2):
#foo code here

Note that the 3 different versions are perfectly compatible... in fact they use the same code (it's what I'm currently using in my own code). I think that this possibility to use the decorators "the way you want" is a good thing, it gives more freedom to the programmer but the underneath mecanism is standardised and so it allow a wide usage (what is good since contract are a very nice way to improve software reliability).

Implementation
Whatever the syntax (naming convention or decorators) the underlying implementation is the same:
1) on definition of the function attributes are populated with methods.
__pre__ with preconditions methods
__post__ with postconditions methods
__inva__ with invariant methods
2) the __call__ is modified so that it calls all the methods in the __pre__ and __inva__ lists (the parameters must be recopied according to their names) before calling the actual method code and then calls all the methods in the __post__ and __inva__ lists (first the result of the function is passed as a parameter then the other parameters are recopied according to their names) after the call.
3) on run time if any exception is raised in one of the contract method it is catched and encapsuled in a ContractFailed assertion so that we could exactly understand what the problem is.

All this mecanism should be disabled if optimisation is activated of course.

One thing: why did I choose to change the __call__ rather than using the common decorator idiom that consist in calling the decorated method in a wrapper method that would do the stuff... because doing this add a big overhead if many decorator are called on the same function since the cost of a function call in python is high... but of course it is possible to use that classic way (it what I did before and I'm currently testing the new version that is a bit harder to code...).

Pierre-Yves Martin

Posts: 17
Nickname: pym
Registered: Jan, 2005

Re: Let's find the possible solutions to have contracted functions Posted: Jan 5, 2005 8:06 PM
Reply to this message Reply
Errata:
My __call_ changing technic is stupidly complicated for no gain and just discored it today while reading my own post (pathetic I know). The overhead is negligeable since each contract method have to be called... so here is my modified version:

> 1) on definition of the function attributes are populated
> with methods.
> __pre__ with preconditions methods
> __post__ with postconditions methods
> __inva__ with invariant methods
2) instead of the contracted function a wrapper function is used that calls all the methods in the __pre__ and __inva__ lists (the parameters must be recopied according to their names) before calling the actual method code and then calls all the methods in the __post__ and __inva__ lists (first the result of the function is passed as a parameter then the other parameters are recopied according to their names) after the call.
> 3) on run time if any exception is raised in one of the
> contract method it is catched and encapsuled in a
> ContractFailed assertion so that we could exactly
> understand what the problem is.

Brian Slesinsky

Posts: 43
Nickname: skybrian
Registered: Sep, 2003

Warning: duck typing with recursion is hard Posted: Jan 5, 2005 8:37 PM
Reply to this message Reply
I once tried to implement a python-like language using duck typing. I didn't get very far, but I learned that duck typing is harder than it looks.

A simple example:

interface Parent {
Child getChild(int index);
}

interface Child {
Parent getParent();
}

Suppose there is some class C and we want to figure out whether implements(C, Parent) is true using duck typing. It's not enough to look at C's class definition. We also need to figure out what class C's getChild() method returns (suppose it is D), and then determine whether D implements Child using duck typing. But in order to do that, we need to know what type D's getParent() method might return. A simple depth-first recursive type-checking algorthm will never terminate.

Furthermore, if we somehow fix that and recursively iterate over all the classes mentioned in method parameters and return types, it tends to pull in a large number of classes. For example, in Java, every class implements Object. Object has methods referring to String and Class. String refers to StringBuffer and Locale. Class refers to ClassLoader, Constructor, Field, Method, Package, and URL. URL refers to URLStreamHandler and URLConnection. And so on.

In the end we're going to end up looking at every method of a large number of classes just to figure out whether A implements B. That's pretty expensive, especially if you're planning to do type-checking at runtime. And suppose the type-check fails? Generating a meaningful error message doesn't look trivial either.

Figuring out how to implement implements() efficiently in the presence of duck typing looks to me like a nice topic for a thesis paper, not something that a practical language designer should attempt.

With an explicitly declared type hierarchy (such as in Java), things are much simpler. If you want to know whether A implements B, you just need to know the names of A's ancestors.

Pierre-Yves Martin

Posts: 17
Nickname: pym
Registered: Jan, 2005

Static typing or not... Posted: Jan 5, 2005 9:01 PM
Reply to this message Reply
After reading all the post here and thinking a little bit... I think addint a static typing system to python (even optional) would have a bad impact on the language itself and the community. I won't write all the arguments that leads me to that change in my mind because they are all written before in the other posts.

But...(there always a "but") I think that a simple but flexible adaptation system + a good interface/protocol declaration system + an simple and optionnal interface requirement sytem would do the job better.

Explanations (sorry if my posts are always long I try to be concise... I swear it could be realy worst !):

adaptation system: the runtime adaptation system is according to me the most flexible system to deal with the problem of behaviour requirement. It is very pythonic/dynamic/Object Oriented and it do not comes with problems regarding to the current duck typing schema of python.

interface/protocol declaration: It would be really nice if we add an easy and simple way to define a reusable behaviour schema (I mean what call an interface or protocol definition). An interface is neither a type (like "int" or "float") nor a class (that store data and method corresponding to those data) it is an encapsulation of many behaviours so they can be easily manipulated (for exemple to chack that a bundle of 30 methods needed to control a electronic drone are present in a object passed as a parametrer to a function in charge of controling this drone). Dealing with those methods one by one could be quite long and hard... and of course error prone (human being are no good at repeating task).

optionnal interface requirement: interface requirement declaration is an easy way to "present" to the world what we expect from him. Anyone that have one day deal with RPC or CORBA systems understand what I mean. We could win here in readability of code, documentation, bug detection. Adaptation is good but remember that it could be done deep inside the function code... so it's not necessary obvious for the function user to know what kind of object to pass as a parameter.
We have argument name to help us (but they have so much thing to express and have to be quite short) or documentation (but it to be kept up to date... what is not the easiest thing in developpement).
And keeping this feature optional could be easier than static typing since it is further from other language typing and it would not deal with classes or type but only with interface (remember that a int type or a class is not an behaviour definition !!! It may implies one but it is not one.).

Sample of code the clarify my thought

For me a piece of code is always easier to understand so here are samples... interface specification are static information and so there is only some very few way to express them (cf my previous post)"decorators" and "parameter definition" (and maybe "docstring hack" but I won't deal with this last one).

adaptation system
--> no code sample please refer to PEP 246

interface/protocol declaration

interface TripleFooAble:
def foo1(arg1):
"""docstring1"""
def foo2(arg1, arg2)
"""docstring2"""
def foo3(arg1, arg2, arg3)
"""docstring3"""


optionnal interface requirement

#version 1: one-at-a-time decorator
@implements(arg1,filelike)
@implements(arg2,numeric | text)
def foo(arg1, arg2):
#foo code here with adaptation if needed

#version 2: all-in-one decorator
@implements((arg1,filelike),(arg2,numeric | text))
def foo(arg1, arg2):
#foo code here with adaptation if needed

#version 3: static-typing-like syntax
def foo(arg1: filelike, arg2: numeric | text):
#foo code here with adaptation if needed

I think the last syntax exemple is too near from static typing to prevent possible confusion between static type checking and static interface specification... but I had to show it since it is also the easier to use (let's be pragmatic sometimes).

Ka-Ping Yee

Posts: 24
Nickname: ping
Registered: Dec, 2004

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 6, 2005 2:42 AM
Reply to this message Reply
If you look at it a certain way, conformance to an interface, adaptation, preconditions, and postconditions are all the same thing: some languages call them guards.

A guard is a thing that looks at a value and returns something to be used in its place. It can return the original value or a replacement; or it can throw an exception.

If there is a lightweight syntax for defining guards, i think we can achieve most of what we want. Assume for the moment that we use Guido's syntax, but with a slightly modified interpretation: where the original proposal has a "type expression", let's consider that expression the body of a lambda with one argument, which is the name on the left side. Here are two examples:

A. def gcd(a: int, b: int) -> int
B. def deduct(self, amount: 0 <= amount <= self.balance)

The parts to the right of the colons and arrow are the guards. In order to check an argument arg against a guard, we do the following:

1. Evaluate the lambda expression on arg to yield a value, val.
2. Invoke adapt(arg, val).

Before a function body is executed, we check the argument values against their argument guards in this way; and after the function body is executed, we check the return value against the return guard in the same way.

This works fine for example A: each guard adapts its argument to an int, and the return guard adapts the result to an int. The call to adapt() should raise an exception if the argument cannot be converted to an int.

To make example B work as well, we add one special case to the adapt() function. Let adapt(x, True) return x and let adapt(x, False) raise an exception. Now we can also use any test as a guard and it will work as a precondition or postcondition. That also means we can reuse predicates: just define a function that returns a boolean value, then use it over and over again in a guard.

Okay. So that's one proposal for addressing the problem.

Now let's take this one step further and generalize a bit. Suppose that instead of the colon being just a part of function signature syntax, it becomes an operator: the guard operator. (Maybe we would have to use something else, like :: or ->, to distinguish it from the block-introducing colon. But we can argue about that later. Assume for the moment that colon-end-of-line introduces a block, where as a colon in the middle of a line is the guard operator.)

When the guard operator is used in a function signature, it behaves just like described above. When it is used in an expression, it guards the left-hand side in the same way it would guard an argument. Thus we could express more interesting guards, and also make code more self-documenting:
def deduct(self, amount: 0 <= amount: int <= self.balance): self.balance >= 0:
self.balance -= amount
def draw(items, canvas):
for item in items:
shape = item: Shape
shape.draw(canvas)
or even
def draw(items, canvas):
for shape: Shape in items:
shape.draw(canvas)
When we're reading the code, the line shape = item: Shape tells us that, after that point, we can be assured that shape conforms to the Shape interface.

Pierre-Yves Martin

Posts: 17
Nickname: pym
Registered: Jan, 2005

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 6, 2005 5:12 AM
Reply to this message Reply
I really really like the generic guard concept... and the proposed syntax (no matter for the moment if we use a colon an arrow or any other symbol to do this). I think it's a very elegant way to formalise adaptation and static restrictions that we want for a function.

Some problems still keep in my mind:
1) how could we express a very simple post condition like this one: "result of function foo must be positive"?
Here is the solution with decorator syntax of precondition (any other postcondition syntax would fit).

@postcondition(lambda result : result > 0)
def foo(arg1, arg2):
#foo code here

With the guard syntax we have a problem: the result of the function as no name and can not be tested except if we use a new reserved token "result":

def foo(arg1, arg2) -> result > 0:
#foo code here


2) How should we express a predicate that do not apply to one peticular parameter but to many of them or to data that are not parameters (attribute of the object itself for exemple). It's difficult to do this and keep the declaration static....

#is it good to do so ?
def foo1(arg1: arg1 > arg2, arg2):
#foo code here

#...or so maybe ?
def foo2(arg1, arg2: arg1 > arg2):
#foo code here

#...and so ?
def foo3(arg1, arg2: self.amout > 0):
#foo code here


3) If we use many predicate for one argument the syntax could be quite heavy... but it is still possible to express it in a multiline way or put many predicates in one external function.

#raw syntax with many predicates
def foo(arg1: numeric: arg1 > 0, arg2: numeric: arg2 < 0):
#foo code here

#already defined predicates
def predicate1(arg):
arg = adapt(arg, numeric)
arg = adapt(arg, arg > 0)
return arg

def predicate2(arg):
arg = adapt(arg, numeric)
arg = adapt(arg, arg < 0)
return arg

def foo(arg1: predicate1, arg2: predicate2):
#foo code here

4) I think we should not allow to have a type as second param to the adapt() function so that we do not allow static typing (what for me is not a good thing for python). But interface should be allowed (once again interfaces are not types but only behaviour definitions !!!!!).

But do not think I do not like your idea... I found it really elegant. If one can find solution for the 2 first problems (the 3rd is more a personnal problem about too long lines ^_^ ).

But once again here what I think could be a good road map to that feature because I do not like the idea to directly add a new syntax element with no wide usage/testing before:

1) Add a good adapt() function/mecanism to the library (it should support the "adapt(a, True)" syntax).

def foo(arg1, arg2):
arg1 = adapts(arg1, numeric)
arg1 = adapts(arg1, arg1 > 0)
arg2 = adapts(arg2, numeric)
arg2 = adapts(arg2, arg2 < 0)
#foo code here

2) Add a standard decorator to enable the use of guards for function using adaptation.

@adapts(arg1, numeric)
@adapts(arg1, lambda arg1: arg1 > 0)
@adapts(arg2, numeric)
@adapts(arg2, lambda arg2: arg2 < 0)
def foo(arg1, arg2):
#foo code here

3) Add a new syntax to have an heavyweight way to express such guards using a new operator(I choose the arrow syntax to prevent confusion with) :

def foo(arg1 -> numeric -> arg1 > 0, \
arg2 -> numeric -> arg2 < 0):
#foo code here

4) (maybe it could happens in the same time as point 3) Extends the new syntax so that the operator could be use anywhere in the code.

def foo(arg1 -> numeric -> arg1 > 0, \
arg2 -> numeric -> arg2 < 0):
#some code
for item in items:
shape = item -> Shape #equivalent to shape = adapts(item, Shape)
#some other code

Noel Welsh

Posts: 1
Nickname: noelwelsh
Registered: Jan, 2005

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 6, 2005 5:52 AM
Reply to this message Reply
There are some problems with Eiffel's contract system. Robby Findler, of PLT, has done a lot of work fixing the problems with Eiffel's system and creating a better system that is used for interface definitions in PLT Scheme.

His publications are at http://people.cs.uchicago.edu/~robby/pubs/

The main PLT publications page is at http://www.ccs.neu.edu/scheme/pubs/

The documentation for PLT Scheme's contract system is at http://download.plt-scheme.org/doc/209/html/mzlib/mzlib-Z-H-13.html#node_chap_13

Pierre-Yves Martin

Posts: 17
Nickname: pym
Registered: Jan, 2005

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 6, 2005 6:24 AM
Reply to this message Reply
The PLT Scheme's contract sytem is really interesting... maybe a little bit complex... but very powerful. The Ka-Ping Yee proposal to add an adapt/guard operator to python is really near of it in the concept and I think it is more pythonic.

I suggeste the reading of the last article about PLT Scheme it is very instructive... and it enforce me in my opinion that the adapt/gard operator is a very smart and elegant solution.

Phillip J. Eby

Posts: 28
Nickname: pje
Registered: Dec, 2004

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 6, 2005 8:57 AM
Reply to this message Reply
> If there is a lightweight syntax for defining guards, i
> think we can achieve most of what we want.

Indeed. Not only that, but we could incorporate generic functions, predicate dispatch, and maybe even pattern matching... Well, okay, it might be hard to convince Guido to add all that stuff. But PyProtocols' dispatch package is doing a pretty decent job right now with arbitrary guard expressions using Python's existing syntax. It would do even better if there was a compact adaptation or conformance operator, and even better still if there was a binding syntax (for pattern matching).


> Assume for the
> moment that we use Guido's syntax, but with a slightly
> modified interpretation: where the original proposal has a
> "type expression", let's consider that expression the body
> of a lambda with one argument, which is the name on the
> left side. Here are two examples:
>
> A. def gcd(a: int, b: int) -> int
> B. def deduct(self, amount: 0 <= amount <=
> self.balance)

>
> The parts to the right of the colons and arrow are the
> guards. In order to check an argument arg
> against a guard, we do the following:
>
> 1. Evaluate the lambda expression on arg to
> yield a value, val.
> 2. Invoke adapt(arg, val).

There's a simpler way to do this; if the expression doesn't refer to the variable, it can be interpreted at function definition time, and used as an adaptation target. If it does refer to the variable, it's a call-time true/false guard.

The extension I'd like to see at that point is the ability to add a general pre-execution guard to the function that isn't specific to a single argument, e.g. "arg1 < arg2". Once you have that, you can build generic functions PyProtocols style, with optimal decision trees built automatically.

Bruce Eckel

Posts: 875
Nickname: beckel
Registered: Jun, 2003

Re: Adding Optional Static Typing to Python -- Part II Posted: Jan 6, 2005 12:26 PM
Reply to this message Reply
What you seem to be talking about is really a different language, or perhaps a preprocessor or something orthogonal like that. Once you go down the rabbit hole it's like the const correctness problem, and you play whack-a-mole trying to fill gaps in your type-checking scheme. I cannot see how traditional static type checking could be applied to Python and have anything like Python come out the other end.

On the other hand, I think there may be some techniques that could be applied that could make a big difference. The trick would be in discovering what they are and judiciously selecting them to get the ones that produce the most effect with the smallest programmer impact. Things along the lines of type inference, perhaps coverage tools or other things that we don't normally think of as typical type checking features.

I would also suggest an incremental approach -- start by adding a feature that can be optionally used, such as type inference (if that's possible), that might have a big improvement with very little downside. Or coverage testing or something like that, or interfaces with contract support.

I've noticed something interesting with Visual Basic; if you don't declare variables it falls back to dynamic mode, but if you declare any variables then it makes you declare all of them. However, I don't know if type declaration fits my definition of a feature that would give a lot of bang for the buck; I suspect there are other things that would do much more in that direction, with much less negative impact on the programmer.

In short, I think that improvements can be made, but I think that the truly pythonic approach will require deeper insights into the issue of finding errors in programs than simply adding traditional static type checking features.

Bengt Richter

Posts: 2
Nickname: bokr
Registered: Jan, 2005

Re: Interface and design by contract Posted: Jan 6, 2005 1:49 PM
Reply to this message Reply
> Sorry Bengt... but your proposition do not work better
> than mine (I tested it a long time ago in fact but did not
> mention it).
>

> >>> def foo():
> ... global nested
> ... def nested():
> ... pass
> ... return nested
> ...
> >>> foo() == foo()
> False
>

> the variable "nested" here is global but the function in
> it is redefined at each call... what is the same problem
> than with my current implementation... you just keep a
> track of the previous defined nested function before
> redefiniting a new one...
Ok, how about this fix:

>>> def foo():
... global nested
... try: nested
... except NameError:
... def nested():
... pass
... return nested
...
>>> foo() == foo()
True

Guido van van Rossum

Posts: 359
Nickname: guido
Registered: Apr, 2003

Topic Closed Posted: Jan 6, 2005 5:09 PM
Reply to this message Reply
Please read my later post: http://www.artima.com/weblogs/viewpost.jsp?thread=87182

Ka-Ping Yee

Posts: 24
Nickname: ping
Registered: Dec, 2004

Guards, guards! Posted: Jan 6, 2005 5:54 PM
Reply to this message Reply
I forgot to mention three things about the guard idea.

1. For return guards, i was thinking of using the name of the function to refer to the returned value.

2. One advantage of a guard operator is that they let you achieve some of the expressive power of parameterized types, without having to invent a parameterized type system. The following example checks and uses the fact that the first argument is a sequence of Shapes, but it doesn't have to declare them with a parameterized type like list[Shape]:
def draw(items: iter, canvas):
for item: Shape in items:
item.draw(canvas)

3. I stole the whole guard thing from E. http://erights.org/

A. Ellerton

Posts: 1
Nickname: ellers
Registered: Jun, 2005

Re: Adding Optional Static Typing to Python -- Part II Posted: Jun 21, 2005 2:42 PM
Reply to this message Reply
A small observation...

"->" and "|" are new syntactic elements.

I think it would be good if existing elements could be reused - iff its reasonable to do so.

For example:

def add(int a, int b) -> int:
return a+b

could use "as" instead of "->":

def add(int a, int b) as int:
return a+b

WDYT?

Flat View: This topic has 59 replies on 4 pages [ « | 1  2  3  4 ]
Topic: Typesafe Enums in C++ Previous Topic   Next Topic Topic: Enhancing agile planning with abuser stories

Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2019 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use