The Artima Developer Community
Sponsored Link

Weblogs Forum
Python Optional Typechecking Redux

36 replies on 3 pages. Most recent reply: Mar 28, 2006 11:03 PM by James Merrill

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 36 replies on 3 pages [ « | 1 2 3 | » ]
Tony Lownds

Posts: 2
Nickname: tlownds
Registered: Jan, 2005

Re: Implementation Posted: Jan 19, 2005 4:24 PM
Reply to this message Reply
Advertisement

> def foo(a: t1, b: t2) -> t3:
> ...

> would be compiled as if it said

> @typechecking(a=t1, b=t2, __return__=t3)
> def foo(a, b):
> ...

> and the @typechecking decorator would produce a wrapper
> that holds references to t1, t2, t3 and the wrapped
> function object. (It will have to use sys._getframe() to
> access __typecheck__ and bind that to the wrapper too.)

I don't know how much of the expression syntax from previous blog posts you were thinking about keeping. If "def(t1, t2) -> t3" is going to be valid syntax for declaring a callable interface, and a "signature" object is created from that syntax, then perhaps it would be easier to pass a signature object to the @typecheck function.

It would be nice to let __typecheck__ actually get the function's value.
That way implementations could do a byte-code check.


def typecheck(signature):
def maketypecheckedfunction(fn):
...look up __typecheck__...
...make wrapper...
return __typecheck__(fn, signature)
return maketypecheckedfunction


I hope that a lot of the type expression operators come back from the previous proposals. I really like | and & and [] functionality. It wouldn't be hard to have attribute lookups type checked, which could help support duck types.


O = VoodooType()

# assumes syntax for "signature" literals: def() -> type
# the rest uses operator overloading hacks
readable = (O.read == def() -> str) & (O.readlines == def() -> iter[str])
writeable = O.write == def(str) -> None
printf = def(str, *any) -> str


Without builtin support, that would become something like the following. The names of types wouldn't be as canonical or as nice to write.


from some_type_system import *
O = VoodooType()

# no special syntax.
readable = (O.read() == IStr) & (O.readlines() == IIter[IStr])
writeable = O.write(IStr) == INone
# star would wrap IAny for the code inside O.__call__ to detect
printf = O(str, *star(IAny)) == None

Nicolas Lehuen

Posts: 2
Nickname: nlehuen
Registered: Jan, 2005

Re: Python Optional Typechecking Redux Posted: Jan 19, 2005 6:28 PM
Reply to this message Reply
I dread the day when I'll be forced to use some arbitrary parameter type for methods defined in a strictly typed API using isinstance checks...

I don't want to loose the wonderful liberty of duck-typing that allows me to integrate code in Python way faster than in Java.

Checking types is very well, but if an API designer is doomed with short-sightedness, the type checks may well end up being too much restrictive, and the API users will soon enough find themselves piles and piles of wrapping code.

Stupid 2-seconds example of a badly designed API :

def count_words(f:file) -> int:
return len(re.split('\W+',f.read()))

"OH, this method wants a file instance, not a file-like object. Damn, this means I must write my memory buffer to a real file so that it can be accessed ???"

Without type checks, the API would pose no problem here. Checking types implies a lot of extra thinking for the API designers.

What I mean is that introducing type checks will have three consequences :

1) poor API design will lead to unnecessary type restrictions, with a lot a wrapping code required from the users to work around the limitations introduced by the API designer.

2) good API design will require a plethora of abstract types or interfaces (what is precisely a file-like object ? What about an 'Readable' interface with a single read() method, from which file-like objects would inherit ? Yada yada ad nauseum) and you'll end up with Java-esque APIs. A lot of abstractions, but the job does not get done any faster.

3) This leads us to the current state of Java, which tries to break its static typing limitations by introducing genericity through generic types. At last you can write generic algorithms in Java ! Great ! But you could have done this long ago without static typing...

That's why I flipped when I read your first proposal about type checking, interface and generic types. It's the snake who bites its own tail ! You decide to add type checks, then you need interfaces for flexibility, then you notice that interfaces are not enough, you need generic types, and eventually you cool down and you realize that you'd better dump all those types checks ASAP.

There is a certain dynamically typed language out there, that's been developed and used heavily for more than 20 years, whose name begins by Small and ends by Talk... I'm not 100% sure, but I think that 20 years after, type checks are still not persona grata in this language. And I think there's a good reason for it.

Implementing types checks, a new syntax, unifying the various interfaces proposals, introducing a generic typing system, all this is very sexy, but it won't improve the 'get the job done simply, quickly and clearly' mission of Python.

I think there are other improvements which can be as sexy AND provide a strong value to the users' community, like (those are the one that fly around my head now, but there are many others) :

- improvements in multi-threading, dropping the GIL ?
- dropping reference counting to simplify C extension development
- endorsing the development of Psyco, including it in the standard Python distribution
- ditto for Pyrex - hey, in Pyrex you even get some type checks :)
- unifying the DBAPI parameter passing style across the various drivers (what's this mess with question marks, python-style, named style and so on ? How are we supposed to write portable SQL queries ?)
- cleaning up the XML situation
- find a clever way to concatenate strings so that we stop writing ''.join(list_of_strings), which is very non intuitive.
- making sure that sys.getdefaultencoding() returns a sensible value on each and every computer instead of 'ascii' everywhere. OK, this one is definitely not sexy, and in fact you just have to remove a comment or a "if 0" somewhere, or add something in sitecustomize.py . But why the hell is this required ? This should be automatically done !

And so on and so forth. Those improvements need to be tackled, and all the community energy should not be wasted by endlessly discussing typing or syntax issues. Well, that's my opinion, at least ;)

I understand that as the BDFL you may want to do other things than tackling unicode codecs issues, but if you tell the community 'we should really improve the standard library' instead of 'how can we complicate the language even further ?', you can make a huge difference, especially if you consider that there are much fewer people that can change the language than people who can improve the standard library. You choose where to turn the spotlight...

Best regards,
Nicolas

Blake Winton

Posts: 4
Nickname: bwinton
Registered: Jan, 2005

Re: Python Optional Typechecking Redux Posted: Jan 19, 2005 10:11 PM
Reply to this message Reply
It sounds to me as if many of the complaints about typechecking would be eliminated if the decision about whether to typecheck or not was left to the user of the library. So if I didn't care what those Java programmers thought they wanted, I would just set "notypechecking = true" (or pass an option to the interpreter, or whatever!) in my file, and I wouldn't have to worry about it. And if I wanted speed, or enhanced checks, or whatever it is I think typechecking gets me, I could set the opposite flag.

For backwards compatibility with older Python programs, I recommend that the default behaviour is no-type-checking, but either way, this should make everyone happy.

Thoughts? (I'm interested in thoughts from everyone. Would this allay your fears that the language would change for the worse? Would it completely fail for some reason I haven't thought of?)

Jay Parlar

Posts: 1
Nickname: jayp
Registered: Jan, 2005

Re: Python Optional Typechecking Redux Posted: Jan 20, 2005 8:39 AM
Reply to this message Reply
My question is this: Why are we (and by "we", I mean long time Python users/advocates) even considering static type checking?

Most blogs I read about it *seem* to be saying it's to placate the new-comers (often refered to as "Java People"). And why would we want to do that? Well, to bring more people to this wonderful language of course.

But if static type checking were just thrown in, and a "Java People" person decides to check out Python, are they really going to see it as *that* different from Java? Sure, they'll see some more flexibility here or there, but will it be enough to get them to really learn Python, to keep it as a long term tool in their utility-belt? Even though the static checking would be optional, you just KNOW that people used to it would use it off the bat.

I don't know. I think one of the great things about Python is that when you start learning, you can see the dynamism right away, you see that as one of the key points of language.

If all newcomers see is "static typing with some more flexibility", then they're probably not going to stick around for the rest of the show, figuring the time required to learn a new language is not worth the (perceived) few extra features

Just my thoughts on the topic...

Blake Winton

Posts: 4
Nickname: bwinton
Registered: Jan, 2005

Re: Python Optional Typechecking Redux Posted: Jan 20, 2005 9:31 AM
Reply to this message Reply
From http://www.artima.com/weblogs/viewpost.jsp?thread=86641

"they can be useful for documentation, for runtime introspection (including adaptation), to help intelligent IDEs do their magic (name completion, find uses, refactoring, etc.), and to help find certain bugs earlier"

Also, they could help automatic optimization, and they assist in binding Python to other more statically-typed languages.

Daniel Arbuckle

Posts: 1
Nickname: djarb
Registered: Jan, 2005

Re: Python Optional Typechecking Redux Posted: Jan 20, 2005 11:35 AM
Reply to this message Reply
The advantages of including static type information are claimed to be documentation, introspection, source automation (IDEs, refactoring tools, etc), bug finding, automatic optimization, and binding to other languages.

Of these, documentation, introspection, source automation and binding to other languages are all fully available even if __typecheck__ doesn't do anything!

That means the only purposes that should be considered when designing the default typechecking function are bug finding and automatic optimization. Of these, optimization has already been discarded as a concern, rightly in my opinion. This leaves only bug finding as an influence on the design of the default __typecheck__.

It's been my experience that Java-style type checking does not aid in finding bugs, so I would recommend not using isinstance by default. In fact, the only typing system that I've ever found useful is the one that already exists in Python, which tells me if I try to do something impossible.

For this reason, I suggest that you extend the parser to handle your proposed typing syntax, but by default simply store the type information in a variable of the function, perhaps __signature__, and otherwise ignore it.

Writing adapters that use this information, in whatever way a module creator deems appropriate, wouldn't be hard. The standard library could even include common ones such as @strict and @adapted

Shalabh Chaturvedi

Posts: 5
Nickname: shalabh
Registered: Jun, 2004

Re: Python Optional Typechecking Redux Posted: Jan 20, 2005 1:01 PM
Reply to this message Reply
Perhaps a typo? Shouldn't the example be:

def foo(a, b): # Typechecking wrapper
a = __typecheck__(a, t1)
b = __typecheck__(b, t2)
r = foo__(a, b)
r = __typecheck__(r, t3) ## <-- instead of r = __typecheck__(t3)
return r


Of all proposals I've seen, this one seems best because:

* It is minimal

* Extensible (by future stdlib and other developers)

* Does not attempt to solve a complicated problem in one go

Some observations and questions:

* The typechecking (and also adaptation) control lies entirely with the 'callee' i.e. the module or class called. Should there be something to let the passed object specify how it wants to be adapted to a type T (before being typechecked by the callee __typecheck__). By passed object I mean the object b in a call f(b).

* Would it be possible to define a metaclass with a method __typecheck__ that would be used as the typechecker for all its instances? For example:

def MyType(type):
def __typecheck__(a,T):
"typechecking code here"

Nicolas Lehuen

Posts: 2
Nickname: nlehuen
Registered: Jan, 2005

Re: Python Optional Typechecking Redux Posted: Jan 20, 2005 3:11 PM
Reply to this message Reply
I agree that a declarative way to tell "here is what this function expects for the types of its parameters" could be beneficial, for documentation, IDEs etc.

But please, please, don't enforce those types ! This would lead us straight to Bondage & Discipline programming, and Python would loose all its interest.

So this could be implemented quite rapidly with a decorator like :

@signature(size=int,coordinates=(int,int),text=union(file,str))
def paint_text(size,coordinates,text):
# ...

Why not... But again, DON'T ENFORCE THOSE TYPES :) ! Alas, I think the temptation will be too strong for people coming from Java or C++... Even Guido was tempted !

Kay Schluehr

Posts: 302
Nickname: schluehk
Registered: Jan, 2005

Re: Python Optional Typechecking Redux Posted: Jan 20, 2005 4:31 PM
Reply to this message Reply
> I think this makes it too easy to change the "flavor" of
> Python, because people used to Java will get the idea that
> you should define __typecheck__ to do isinstance()
> checking and then we'll all have to put up with it.

IMHO the only reason to use type-guards are type-checks based on isinstance(). You may argue that adapt() permits more advanced "interface conformity transformations", but I think most of the use-cases are still conservative and the user does not want to generate wrapper-objects on the fly and produce side-effects.

If one wants to check interface-conformity of the input parameters of a certain function the most likely approach to succeed and to let the compiler do all the work is to derive (partial) interfaces from the use of the parameters in the function itself. This "interface-inference" will always only approximate a type. It should be simpler to implement though not be trivial keeping properties and overloaded __setattr__ in mind.

Imagine a function derive_interface(), that keeps a function and returns a dictionary that contains the names of the input parameters and the related attributes of those parameters used within the function ( aliasing should not be hard to grasp ):

def driveSomeCar(car):
    car.drive()


>>> derive_interface(f)
{"car":["drive"]}

This information could be supplied to the interface-conformity checker using hasattr(), that is called whenever driveSomeCar() is called. In the current approach the implicit call of __typecheck__() depends on the explicit definition of type-guards which imply a completely different top-down ( Java-like ) perspective.

Conclusion: Python will obviously become more Java-like, but without the advantage of catching type-errors by the compiler because it remains dynamically typed. It is a kind of a funny mixture: "Nix Halbes, Nix Ganzes" as people say in German.

Regards Kay

Chad Crabtree

Posts: 1
Nickname: flaxeater
Registered: Jan, 2005

Re: Python Optional Typechecking Redux Posted: Jan 20, 2005 4:55 PM
Reply to this message Reply
Well it seems to me that there is a lot of emotional outporing over something that is not that likely to happen. I keep reading about how some people are terrified of badly written libraries.

Also the assumption is that Java people will come over and pollute the pythonosphere with type checked code like a tsunami. There are not really that many very active python developers now. Would this number of very productive people be tripled or quadrupled by the admission of "Java people"? I think not, would that really be a BAD Thing?.

There are plenty of times that only a certain type is needed in a function. In a library this is very appropriate. How many sane people are going to insist that it be only a string when a string like object will work. Many times in my own projects I write functions that take only the object I created and I forget sometimes, I think it would be nice for Python to tell me exactly what I did wrong by using this checking mechanism. I don't really care if python grows this feature or not. It still remains to be seen but decorators have not destroyed "python as we know it." It just seems very reactionary and why have a BDFL if we cannot trust him.

Greg Jorgensen

Posts: 65
Nickname: gregjor
Registered: Feb, 2004

Re: Python Optional Typechecking Redux Posted: Jan 20, 2005 11:27 PM
Reply to this message Reply
I vote no on anything to do with static type checking.

I've been programming for over 25 years, and wrote C++ for almost ten years. I have developed several real-world applications using Python, since version 1.5. Not once have I wished for static typing in Python. I can't recall a single bug that would have been prevented if I had used static typing.

Static typing may be useful for some very specific applications, but I think it will come at a huge cost for regular Python programmers like me. If I want better documentation, runtime introspection, adaption, intelligent IDEs, debugging, etc. I will use Java. Right now, Python is better than Java and C++ in so many ways I'm surprised to see this idea come up.

In the last few releases of Python stability, completeness, and performance have all improved to the point that I can talk my CIO and the programmers I manage into switching to Python without much of a fuss; Python is now perceived as a serious language. Unfortunately the last few releases have included a lot of esoterica that only a handful of initiates understand or use: metaclasses, generators, decorators, etc. I don't doubt that these features have real applications (especially for library designers). But I think Python needs to solidify and grow horizontally -- over the application space -- and stop growing vertically -- in the language design space.

The static type checking proposals bother me on another level: they are exacerbating the trend in Python to have invisible code and side-effects. I don't even like __xxx__ names in my classes or functions. It seems like Python is acquiring macros in bits and pieces, and the compiler is turning into the black box that templates bestowed on C++.

Documentation? Try something like this:

def drive(vehicle, miles):
# vehicle: instance of Vehicle or subclass
# miles: positive number
...

Comments are optional and don't have side-effects. Comment conventions work for the unit testing frameworks, why not for IDEs?

Runtime introspection? The introspection available now has always worked for me, the few times I've needed it, and isinstance() obviates most of the introspection I ever needed.

Adaptation? Looks useful, but that will be a library we can use explicitly if we need it, right?

Intelligent IDEs? We already have them. I know everyone is in love with Eclipse, but it's my hope that Python programmers never need that much help because the language is simple and the libraries make sense and just work.

Debugging? For those of us who come from C++ or Java backgrounds, Python is already a thousand times easier to debug -- if you don't have a lot of invisible code monkeying going on, that is.

Somehow Zope and Plone and Twisted and many other best-of-breed applications and frameworks were developed without static typing. I hope real Python users get involved in this discussion, not just people who are into language design and debating tricky implementations and marginal cases.

Kay Schluehr

Posts: 302
Nickname: schluehk
Registered: Jan, 2005

Re: Python Optional Typechecking Redux Posted: Jan 21, 2005 2:49 AM
Reply to this message Reply
> Static typing may be useful for some very specific
> applications, but I think it will come at a huge cost for
> regular Python programmers like me. If I want better
> documentation, runtime introspection, adaption,
> intelligent IDEs, debugging, etc. I will use Java. Right
> now, Python is better than Java and C++ in so many ways
> I'm surprised to see this idea come up.

To be fair: the proposed typechecking in Python is not very static at all. It is still the interpreter that typechecks at runtime, not the bytecode-compiler. It is not even clear, what the proposed __typecheck__ function does, because it is user dependend ( wether global, per module or per class is an open issue ). If You want, it can do something very different from performing isinstance() which may be otherwise helpfull and is used in each users-code today.

On the other hand i would prefer three different functions in the standard-library used for function decoration:

- adapt()
which is transformative as proposed in PEP246

- typecheck()
performing checks based on isinstance()

- interfacecheck()
derives implicit interfaces from the function and checks against the input-parameters ( duck-type checking ).


Those functions have fixed semantics which would be clearer than having one __typeckeck__ function with opaque semantics. "Better explicit than implicit" so to say.

Regards Kay

Jayson Vantuyl

Posts: 7
Nickname: kagato
Registered: Jan, 2005

Re: Python Optional Typechecking Redux Posted: Jan 21, 2005 12:02 PM
Reply to this message Reply
I don't object to the syntax, but it seems a bit ugly, and I'm not sure how useful it would be.

I don't know about anyone else, but this is starting to feel like metaclassing. Metaclassing in Python is ugly, underdocumented, and hardly used (possibly because of being ugly and underdocumented). I fear typechecking done this way will be the same. I also question its usefulness over assert isinstance.

I think that making this choice on a per module level is probably a poor one.

GUIDO! READ FROM HERE ON! PLEASE! This entire discussion shows the need for interfaces and a post-hoc syntax for associating types/classes with them. I think that the needs of PyCheck and type-inferencing can be served AT THE SAME TIME!

Type-checking and instance-checking both tie into the fundamental definitions of what an object is (what type of coredata does it have, what class is it, what are its parent classes). The dynamicism most valuable in Python (duck-typing and the like) comes from only concerning ones self with what an object does (is there a read method that takes an integer and returns a sequence-of-characters?).

With that in mind, consider the advantages of:

interface worker:
work(time: number):
"""Work for an amount of time"""
wage -> number:
"""How much do we pay this fellow?"""
write_resume(resumefile: file_like):
"""Prepare for a new job"""
complain() -> iterator_of_strings:
"""Complaints"""
pay(paycheck: number):
"""Actually pay the worker!"""

class human_employee(person) implements worker:
def work(self,time: number):
...

def complain(self) -> iterator_of_strings:
while 1:
yield "pay me more"

def write_resume(self,resumefile: file_like):
resumefile.write(self._resume)
return two_weeks_notice(self)

def pay(self,paycheck: number):
...

def routine(w: worker):
w.work()
pay(w,w.wage())

What you see above provided little over interfaces in Java or really any implementation in Python (but can still give adaptation and type inferencing a boost). Many people will immediately mention that there appears to be no support for duck-typing either.

However, that's only how it APPEARS. In the above example, the symbols "number", "file" and "iterator_of_strings" are references to interface objects. In fact, they are REQUIRED to be interfaces to be used in an interface.

The value of this shines in the following code (likely written by someone completely different than the first code, also providing completely different functionality).

class robot(person):
def work(self,time):
...

def wage(self):
return 0

def pay(self,paycheck):
pass

def write_resume(self,resumefile):
resumefile.write(self._manufacturers_brochure)

def complain():
return iter( () )

# Note the following syntax, this allows much easier
# integration in post
implement worker with robot

The magic is in the last line. With the capability to arbitrarily define your data (duck) types to match the interface, any type-checking arguments have a simple, obvious, one-line fix.

This has many other advantages.

It supports duck-typing.

It never restricts coders post-facto from complying with an interface trivially. There are no intrusive changes.

It still provides the ability to type-check compliance with regular-typing AND duck-typing.

It provides a good foundation for type-inferencing.

It provides an infrastructure for retrofitting sanity-checking onto duck-typing already heavily present in the standard library and in existing user code.

It provides the interface infrastructure necessary for adaptation.

It looks good (or at least doesn't look too bad).

It allows (but not requires) implementation and specification/documentation to be in separate files, encouraging better design/readability.

Interestingly, it provides for run-time type modification, example:


magic_implement_list = [ (number, int), (number, float), ]

for iface,datatype in magic_implement_list:
implement iface with datatype


One could make the argument that this makes simple situations more complex since a stub interface must be designed to do the most simple of type checks. I think that most people here would agree that those situations are so simple that you should not use a type-check at all.

I would also claim that this is NOT OPTIONAL (not that anyone completely believes that the previous proposals would eventally be optional either). It is, however, REASONABLE (as your suggestion tries to be). It also also READABLE, something that the first two proposals kind of achieved, but that __typecheck__ doesn't.

I fear that __typecheck__ will take on the down-sides of __metaclass__. Ugly. Underused. Underdocumented. More careful maintenance for the core developers.

For __metaclass__, this is acceptable, since it elegantly solves a hard problem as well as can be done (well, maybe super could be done better). However, __typecheck__ tries-to-solve an almost-problem fairly poorly, so it's a little different.

Therefore, please don't add __typecheck__ or either of the other types of typechecking proposed. Instead, solve the real problem that typechecking is trying to address. Use interface associations to validate data as it flows through the program. Allow it to be modified at runtime. Allow it to be done by programmers at other places than the declaration.

Kay Schluehr

Posts: 302
Nickname: schluehk
Registered: Jan, 2005

Syntax proposals Posted: Jan 21, 2005 2:21 PM
Reply to this message Reply
Goal: specifying declarations that are both expressive and straight enough to distinguish between different kind of conformity checks.

No implicite check:

def f(a,b): ...

> - adapt()
> which is transformative as proposed in PEP246

def f(a: adapt(_,t1), b: t2) -> t3: ...

This can be extended to arbitrary functions that take a as a first argument.

> - typecheck()
> performing checks based on isinstance()

def f(a: t1, b: t2) -> t3: ...

> - interfacecheck()
> derives implicit interfaces from the function and checks
> against the input-parameters ( "duck-type" checking ).

def f(a:_, b:_) -> t3: ...

Regards Key

Shalabh Chaturvedi

Posts: 5
Nickname: shalabh
Registered: Jun, 2004

Partial Type Checking Posted: Jan 23, 2005 6:29 PM
Reply to this message Reply
Will 'partial' type checking be allowed? For example:

def f(a, b:t1) -> t2:
...

should be equivalent to

def f(a:object, b:t1) -> t2:
...

Flat View: This topic has 36 replies on 3 pages [ « | 1  2  3 | » ]
Topic: Think from the User In Previous Topic   Next Topic Topic: The Thinking in Java Conference


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us