Constrained TypeVar fails to narrow on left hand side of conditional expression

[alanhdu]

When I run mypy --python-version 3.6 on the following code:

from typing import AnyStr
def f(x: AnyStr) -> bytes:
    return x.encode("ascii") if isinstance(x, str) else x

I get:

test.py:3: error: "bytes" has no attribute "encode"; maybe "decode"?

But as far as I can tell, this is backwards! x.encode("ascii") is only called when isinstance(x, str), not in the else case (where x is a bytes object).

This is with the latest mypy==0.530

[JelleZijlstra]

I think this is because you're using AnyStr incorrectly. As a TypeVar, it really makes sense only when used multiple times in a signature (or in a few other cases not relevant here). I believe we're going to introduce a warning for incorrect usages like yours, although I can't find the relevant issue now.

Your code should have been written as:

from typing import Union
def f(x: Union[bytes, str]) -> bytes:
    return x.encode("ascii") if isinstance(x, str) else x

which passes mypy with no errors for me.

I don't know why your code gives an error talking about bytes though, and maybe there's a real bug there. I think it's related to the ternary expression, because rewriting the code to use an if-else block removes the error.

[alanhdu]

Ah... I see. Thanks, Union[bytes, str] works perfectly!

I'll leave this open because I think the inference that x is a bytes object probably represents a bug somewhere:

from typing import Union, Text
def f(x: bytes) -> bytes:
    return x.encode("ascii") if isinstance(x, str) else x

returns the same MyPy error FWIW.

[gvanrossum]

Hm, I do think there's a bug somewhere, since this gives no errors -- the key difference being that it uses an if statement instead of an if expression:

def f(x: AnyStr) -> bytes:
    if isinstance(x, str):
        return x.encode('ascii')
    else:
        return x

UPDATE: Your second example also points in the same direction -- again if that's rewritten to use a statement it gives no errors (presumably treating the True branch as unreachable code).

[ilevkivskyi]

I don't think there is any mypy bug here, although this may be a usability problem. What is going on here consists of two parts:

1. For constrained type variables (like AnyStr) the function body is checked for all constraints (str and bytes in this case).
2. For ternary expression, mypy can't mark first part as "unreachable", but it doesn't perform any type narrowing either if the types are non-overlapping, this can be seen form following:

x: int
y = reveal_type(x) if isinstance(x, str) else ''  # Revealed type is 'int'

I think the solution here should be also twofold:

1. Potentially problematic function signatures, like in the original example should get a warning (at least with a note and an exit 0). This is already mentioned.
2. In addition we should error on attempts of narrowing with non-overlapping types. For user-defined types it is OK to keep current behaviour (there might be custom __instancecheck__ or other dynamic things), but for built-in types - especially for more error-prone situations like str vs bytes vs unicode vs basestring etc. - we should give an error:

# Python 3 mode
x: bytes
...
if isinstance(x, str):  # This should be an error, this is likely a confusion with Python 2
    ...

[gvanrossum]

I think there are several red herrings in the examples (use of AnyStr for a single argument, or isinstance() with a non-overlapping type, but I still think it's a bug that the code with the ternary expression gives an error while the equivalent code with an if-expression.

def f(x: AnyStr, y: AnyStr) -> bytes:
    x += y
    return x.encode('ascii') if isinstance(x, str) else x  # Error here

def g(x: AnyStr, y: AnyStr) -> bytes:
    x += y
    if isinstance(x, str):
        return x.encode('ascii')  # No error here
    else:
        return x

We used to have more of these, and we solved them by making the binder smarter -- I think there's more of that to be done to make the first version here pass.

[ilevkivskyi]

@gvanrossum

but I still think it's a bug that the code with the ternary expression gives an error while the equivalent code with an if-expression

This is not easy, since every part of the ternary expression is an expression. It can't just be unreachable like a statement, and should have some type. A possible strategy is to give that "unreachable" expression UninhabitedType (this one has all attributes) and infer a simplified union from ternary expressions, not a join (the second is an independent problem, but I think we already have an issue for this).

[gvanrossum]

So just to clarify, generic functions over type variables with "value constraints" are type-checked multiple times, once for each possible constraint? (Where for AnyStr the constraints are "str" and "bytes".) And isinstance() checks involving those constraints can make specific branches at the statement level unreachable, but there's no concept of an unreachable expression?

[ilevkivskyi]

So just to clarify [...]

I think this is exactly what happens.

[petergaultney]

What would be the appropriate place in the docs to document this behavior? https://mypy.readthedocs.io/en/latest/common_issues.html ? I'll submit a PR (unless I'm just missing where this is already documented!)

[ilevkivskyi]

@petergaultney This depends on what exactly you want to document (i.e. what is "this behavior").

[petergaultney]

I think the correct section of the docs would be here: https://mypy.readthedocs.io/en/latest/common_issues.html?highlight=isinstance#complex-type-tests

And the documentation would be something along the lines of "Ternary expressions will not allow type inference in the same way as using if statements."

[ilevkivskyi]

OK, I see yes, this makes sense (this is not a permanent limitation, but likely to be around for a while). You can make a PR and then we can polish the wording.