Add a pyright-primer

[JelleZijlstra]

In addition to mypy-primer, it would be useful to have a pyright-primer to see how changes we make affect pyright's type checking.

@erictraut do you agree? Would your team be able to help set this up?

[AlexWaygood]

+1, I think this would be really useful

[erictraut]

Changes in typeshed haven't been a big source of problems for pyright users. There have been a few occasional problems (perhaps 1 or 2 a year), but these are typically fixed in less than a week. So we're not feeling much pain here. It sounds like you think it's potentially a bigger problem though?

Let's brainstorm about what a "pyright-primer" would look like. Would it run on the same projects used for mypy-primer, or would it target a different set of code bases — those that are regularly type checked using pyright? Or some combination?

If it's the same set of code bases, I wonder if we could repurpose "mypy-primer" and simply add a new commmand-line option so it invokes pyright rather than mypy? That might require some refactoring in the mypy-primer script, but it would be easier to maintain one tool rather than two. Thoughts?

[JelleZijlstra]

Changes in typeshed haven't been a big source of problems for pyright users. There have been a few occasional problems (perhaps 1 or 2 a year), but these are typically fixed in less than a week. So we're not feeling much pain here. It sounds like you think it's potentially a bigger problem though?

Thanks, that's good to know. Maybe this idea isn't too useful then. Still, it's better to learn about such issues early, and there may be things we miss because people never report it.

Let's brainstorm about what a "pyright-primer" would look like. Would it run on the same projects used for mypy-primer, or would it target a different set of code bases — those that are regularly type checked using pyright? Or some combination?

My impression is that fewer major open source projects run pyright in CI; I looked through usages of Jake's GH Action yesterday and the only project that I recognized was discord.py. Running on code that isn't regularly checked with pyright could also be interesting, because it gives a sense of what people who are new to type checking could encounter.

If it's the same set of code bases, I wonder if we could repurpose "mypy-primer" and simply add a new commmand-line option so it invokes pyright rather than mypy? That might require some refactoring in the mypy-primer script, but it would be easier to maintain one tool rather than two. Thoughts?

Yes, as an implementation strategy, it probably makes sense to just use mypy-primer. We might have to rename it though :)

[hauntsaninja]

Can confirm that there aren't yet many open source projects that run pyright in CI (I previously spent some time trying to finding these since it seemed like a promising source of repos that mypy does badly on). The fact that most of the projects in mypy-primer maintain their own mypy configuration helps keep mypy-primer easier to maintain — this would be different if we were to support pyright.

My sense is that a pyright primer running on the same set of projects wouldn't be the most useful for typeshed. I see a lot of the potential value from a pyright primer in checking all the (not-so) new type system features that mypy doesn't yet support — but most of those projects don't use those features (and typeshed typically doesn't use features mypy doesn't support either).

I'm curious if we have examples of things this would catch. I went through issues filed by Eric (since the advent of mypy-primer), and I found two potential ones:

• Recent change to concurrent/futures/_base.pyi broke a bunch of code #6033 it looks like mypy-primer actually caught this one (one hit) and we ignored it
• Regression: Cannot combine @property decorator with @abstractclassmethod #7134 mypy-primer had complaints about this one too, but haven't confirmed if they match up with the issue report

I'll also note that if someone has time to work on primer, there is lower hanging fruit in that we haven't had a primer at all for third party stubs since the unbundling of typeshed.

[kkirsche]

I'd be hesitant about this if the open source libraries use libraries such as SQLAlchemy ORM, as pyright is rather incompatible with SQLAlchemy's recommend approach to typing which leverages a Mypy plugin to accomplish things. This leads to Pyright complaining about the recommend way of typing SQLAlchemy via the Mapped attribute as described here:

https://docs.sqlalchemy.org/en/14/orm/extensions/mypy.html

This incompatability can cause some major headaches if you want cross-typechecker validation.

In the codebases I work on at work, attempts to mix typeshed types and SQLAlchemy's plugin is sufficient to cause mypy to hard crash

[jab]

Recently hit another regression in typeshed that a pyright_primer would have caught, so linking to it here: #9296

I think a pyright_primer could also have caught this regression from a few months ago: microsoft/pyright#3534