Repository navigation
Tests specifically for tricky, white-lie signatures #1339
Description
Activity
/cc @JukkaL @gvanrossum @vlasovskikh @matthiaskramm @JelleZijlstra @ambv from previous discussions
I like this idea. This would interact nicely with python/typing#436: the same test runner proposed there could run tests in typeshed.
I'm ±0 on this.
Pro: Recently, we've had a number of pull requests with commit messages along the lines of
"Make[some example code]work with typeshed". Clearly, it would have been good to put[some example code]somewhere more meaningful.Contra: It's a slippery slope. Testing leads to tests, which leads to more tests, which leads to complete test coverage, and look! We've just duplicated https://github.com/python/cpython/tree/master/Lib/test.
If we go down this road, we would need very clear wording that unit tests for
pyiare reserved for special circumstances and not mandatory. That's a message that's really hard to convey, in a language culture as addicted to testing as Python.I agree with what Matthias said (and it's why previously I was against such tests). But there are only a few committers on the project and together we can set the bar high for when tests are accepted in PRs. It will help if there are clear words in README.md and CONTRIBUTING.md about this (reviewers can then refer contributors to those words).
I like the general idea of having tests for stubs. So having them at least for tricky signatures looks like a reasonable compromise to me.
An easy and hopefully unobjectionable way to do this is to create a repo with whatever code you want to test and add the repo to mypy_primer.
Reacted by Guido van Rossum, Akuli and Christopher DignamI think that now that we have the new directory structure, we can reopen this discussion.
One way to do this would be to suport an optional
@testdirectory under each stub directory. The@testdirectory could contain any number of.pyfiles. The simplest approach would to be type check these files and fail the test if there are any errors. We can also support specifying that certain lines must generate errors, for example by using a magic comment (for example,# ERROR).Here's an example of how this could work. To add tests for
foobar, we can add the filestubs/foobar/@test/test.py:from foobar import complex_function complex_function(1, arg="x") # On this line there must not be an error complex_function(1, arg=["x"]) # ERROR # On this line there must be an error
We'd run the tests in Python 2 and/or Python 3 modes, depending on what's supported by the stubs. It should be possible to use Python version checks to have different tests for Python 2 and Python 3.
In addition to optionally having tests for tricky signatures, I believe that there are a few other valid use cases for tests:
- For each package, we can have a small number of tests that verify some of the most common use cases. These can act as sanity checks, and they can protect against accidentally making the stubs completely unusable (after major refactorings or after changes to semantics of stubs, for example).
- Any PR that fixes some signature can add a test to validate the fix. This can prevent regressions and make it easier to review the fix. It can also make the contributor more confident that their fix works as expected.
- Non-trivial re-exported definitions can be useful to test, since these are easy to break.
As discussed above, we don't want to have exhaustive tests that test every single signature in stubs. Also PRs that add dozens of annotations probably shouldn't add tests for all of the new signatures (but it may be fine to test a few of the signatures). We'd need to make the conventions explicit and enforce them in code reviews. I believe that after we've had the conventions established for some time, contributors will figure this out by looking at existing tests as examples. It will be important that we don't let poor-quality or too wide tests through code review, as it's possible (even likely) that contributors will start emulating these examples.
Reacted by Christopher Dignam and Guido van RossumNow that we have a package version in METADATA.toml, we could also run stubtest. This would totally take care of 1.
Reacted by AkuliBoth @JukkaL's and @hauntsaninja's suggestions sound good to me.
In December 2020, @hauntsaninja wrote:
An easy and hopefully unobjectionable way to do this is to create a repo with whatever code you want to test and add the repo to mypy_primer.
I would very much like to do something like this to add some regression tests for
pow, which is exceedingly difficult to type. However, as @JelleZijlstra noted, it would probably be better for these mypy_primer-based regression tests to be put inside typeshed rather than maintained as an external repo.Some questions, therefore:
- Do we agree that regression tests for the thorniest cases, such as
pow, would help typeshed? I agree with the consensus that regression tests on the level of the cpython repo are wholly unnecessary, but it seems to me that the corner cases ofpowjust don't appear often enough in open-source code for mypy_primer/stubtest to be sufficient with this function, wonderful tools though they are. - Would something along these lines be an acceptable starting point for some mypy_primer-based regression tests for
pow? - Do we agree that it would be better to have these tests inside typeshed, rather than in a different respository?
If the answer to all of these is "yes", I would be happy to submit a PR, leading to...
- Where should these tests go in the typeshed repo? What should the folder be called? Etc., etc.
- Do we agree that regression tests for the thorniest cases, such as
Thank you! I agree on "yes" for these questions.
I don't feel strongly about the structure or naming, but here's a proposal:
- Create a new top-level directory
test_cases/ - Within this directory, the directory structure mirrors the top level
- Every stub file is tested in a file prefixed with
test_ - So, for example, the
pow()tests would live intest_cases/stdlib/test_builtins.py.
Reacted by Alex Waygood and Akuli- Create a new top-level directory
One complication is that a regression tests for
pow(for example) won't work unless we have the absolute strictest mypy settings applied. That includes--disallow-any-expr. For some builtins tests, however, that might not be practical -- we might want the revealed type to beAny, or it might be impossible in some cases to write a test without a type being inferred by mypy asAny. I can think of two solutions to this problem:- Have two sets of tests (in two separate directories?), with different levels of strictness applied.
- Have one set of tests, and apply the strictest mypy settings to all tests. If we want an inferred type to be
Any, we add a# type: ignorecomment accordingly. Then, if we see any mypy_primer output telling us about an unused# type: ignorecomment in the tests directory, we "know" internally that this signifies a potential failing test case.
Option (2) probably won't work until #6307 is fixed, however.
Another problem is how to ensure the type matches. This doesn't really work:
def expects_float(arg: float) -> None: ... expects_float(pow(5, -7))If
pow(5, -7)returnsint, it would implicitly convert tofloatand cause no errors. This doesn't work either, because a list literal like[2]can belist[float]orlist[int], depending on how it is used:def expects_float(arg: list[float]) -> None: ... expects_float([pow(5, -7)])Maybe we should use mypy's test suite instead? It makes it possible to assert the output of
reveal_type().This doesn't really work:
I would say it "partially works", since we are at least verifying that the inferred type is not
Anyorcomplex. But yes, I agree that this is far from an ideal way of testing these things. Being able to assert the output ofreveal_typesounds much more promising, though I don't really know anything about how mypy's test suite works.though I don't really know anything about how mypy's test suite works
It mostly works like this:
reveal_type(1) # E: Revealed type is 'int'And the framework complains if the revealed type is something different. I don't think it's easy to use that syntax outside of mypy itself, though there are apparently a few tools like pytest plugins doing that.
You can work around the float/int thing by taking advantage of list invariance. This errors for mypy:
from typing import List def f(x: List[float]): pass y = [3] f(y)Reacted by Alex WaygoodReacted by Christopher DignamI think we shouldn't let the perfect be the enemy of the good. If nobody objects, I'll go ahead and draft a PR with some mypy_primer-based regression tests.
If somebody else drafts a PR proposing superior tests based on mypy's testing framework, I'll happily abandon my PR (/revert my PR if it's already merged) 🙂
You can work around the float/int thing by taking advantage of list invariance.
Very nice!
If somebody else drafts a PR proposing superior tests based on mypy's testing framework
I cannot refrain from a shameless plug: https://github.com/typeddjango/pytest-mypy-plugins
It is way more advanced. It supports parametrization, regex matchers (to problems with numeric typevar ids), etc.Reacted by Alex Waygood and AkuliI cannot refrain from a shameless plug: https://github.com/typeddjango/pytest-mypy-plugins
This looks great! If others also agree that this would be a better way to go, I'll hold off making a PR and let @sobolevn propose a PR using this framework.
We now have such tests: #7663.
Reacted by Alex WaygoodReacted by Alex Waygood
We've had some useful, long discussions about writing tests for stubs (#754, #917). Most of the discussion has been around having tests for the bulk of the functionality in stubs and the libraries they describe. I think we're all broadly in agreement that
This issue is for a narrower proposal: sometimes when a library's real signature is something complex or hard to express in the type system, our signature ends up pretty tricky or we end up telling sort of a white lie to the type-checkers. For these, even when looking right at the signature in one window and the library's docs (or an experiment in the REPL) in the other, it can be hard to tell whether they match. In these cases, I think a hand-written test would be useful -- probably looking a lot like a mypy unit test, with some code to be type-checked and an indication that certain lines should have errors and the rest shouldn't, and then we could e.g. run mypy and pytype against it.
@JukkaL suggested just this in one bullet point of a comment on #754, and gave some examples:
A new example today is #1331:
autodoesn't really inherit fromIntFlag, and we may find a new way of refining how we describe that interaction in the stubs. When we do it'll be useful to have a test for the functionality that that little lie in the stubs is intended to provide.