Optionally run tests with mypy nightly (#512)

* CI: option to use mypy nightly

* update pre-commit

* skip ignores to run on CI

* docs

Author: twoertwein

.github/pull_request_template.md

@@ -1,2 +1,6 @@
+<!--
+Two CI tests are using not yet released versions of pandas ("nightly") and mypy ("mypy_nightly"). It is okay if they fail.
+-->
+
 - [ ] Closes #xxxx (Replace xxxx with the Github issue number)
 - [ ] Tests added: Please use `assert_type()` to assert the type of any return value

.github/workflows/test.yml

@@ -66,3 +66,19 @@ jobs:
     - uses: actions/checkout@v3
 
     - uses: pre-commit/action@v3.0.0
+
+  mypy_nightly:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - name: Install project dependencies
+      uses: ./.github/setup
+      with:
+        os: ubuntu-latest
+        python-version: '3.10'
+
+    - name: Run mypy tests with mypy nightly
+      run: poetry run poe mypy --mypy_nightly

.pre-commit-config.yaml

@@ -7,7 +7,7 @@ repos:
     hooks:
     -   id: black
 -   repo: https://github.com/PyCQA/isort
-    rev: 5.11.2
+    rev: 5.11.4
     hooks:
     -   id: isort
 -   repo: https://github.com/asottile/pyupgrade
@@ -27,7 +27,7 @@ repos:
     -   id: flake8
         name: flake8 (pyi)
         additional_dependencies:
-        - flake8-pyi==22.11.0
+        - flake8-pyi==23.1.0
         types: [pyi]
         args: [
           --ignore=E301 E302 E305 E402 E501 E701 E704 F401 F811 W503 Y019 Y034 Y037 Y041 Y042,

docs/tests.md

@@ -11,7 +11,11 @@ Here are the most important options. Fore more details, please use `poe --help`.
   - Run only pytest: `poe pytest`
   - Run only pre-commit: `poe style`
 - Run tests against the installed stubs (this will install and uninstall the stubs): `poe test_dist`
-- Optional: run pytest against pandas nightly (this might fail): `poe pytest --nightly`
-- Optional: Run stubtest to compare the installed pandas-stubs against pandas (this will fail): `poe stubtest`. If you have created an allowlist to ignore certain errors: `poe stubtest path_to_the_allow_list`
 
 These tests originally came from https://github.com/VirtusLab/pandas-stubs.
+
+The following tests are **optional**. Some of them are run by the CI but it is okay if they fail.
+
+- Run pytest against pandas nightly: `poe pytest --nightly`
+- Use mypy nightly to validate the annotations: `poe mypy --mypy_nightly`
+- Run stubtest to compare the installed pandas-stubs against pandas (this will fail): `poe stubtest`. If you have created an allowlist to ignore certain errors: `poe stubtest path_to_the_allow_list`

pyproject.toml

@@ -90,7 +90,8 @@ script = "scripts.test.run:style"
 
 [tool.poe.tasks.mypy]
 help = "Run mypy on 'tests' (using the local stubs) and on the local stubs"
-script = "scripts.test.run:mypy_src"
+args = [{name = "mypy_nightly", positional = false, default = false, type = "boolean", required = false, help= "Use mypy nightly (off by default)"}]
+script = "scripts.test:mypy_src(mypy_nightly)"
 
 [tool.poe.tasks.mypy_dist]
 help = "Run mypy on 'tests' using the installed stubs"

scripts/test/__init__.py

@@ -51,3 +51,8 @@ def stubtest(allowlist: str, check_missing: bool, nightly: bool) -> None:
 def pytest(nightly: bool) -> None:
     steps = [_step.nightly] if nightly else []
     run_job(steps + [_step.pytest])
+
+
+def mypy_src(mypy_nightly: bool) -> None:
+    steps = [_step.mypy_nightly] if mypy_nightly else []
+    run_job(steps + [_step.mypy_src])

scripts/test/_step.py

@@ -32,3 +32,6 @@
 nightly = Step(
     name="Install pandas nightly", run=run.nightly_pandas, rollback=run.released_pandas
 )
+mypy_nightly = Step(
+    name="Install mypy nightly", run=run.nightly_mypy, rollback=run.released_mypy
+)

scripts/test/run.py

@@ -95,13 +95,49 @@ def nightly_pandas():
     subprocess.run(cmd, check=True)
 
 
-def released_pandas():
-    # query pandas version
+def _get_version_from_pyproject(program: str) -> str:
     text = Path("pyproject.toml").read_text()
     version_line = next(
-        line for line in text.splitlines() if line.startswith("pandas = ")
+        line for line in text.splitlines() if line.startswith(f"{program} = ")
     )
-    version = version_line.split('"')[1]
+    return version_line.split('"')[1]
+
 
+def released_pandas():
+    version = _get_version_from_pyproject("pandas")
     cmd = [sys.executable, "-m", "pip", "install", f"pandas=={version}"]
     subprocess.run(cmd, check=True)
+
+
+def nightly_mypy():
+    cmd = [
+        sys.executable,
+        "-m",
+        "pip",
+        "install",
+        "--upgrade",
+        "git+https://github.com/python/mypy.git",
+    ]
+    subprocess.run(cmd, check=True)
+
+    # ignore unused ignore errors
+    config_file = Path("pyproject.toml")
+    config_file.write_text(
+        config_file.read_text().replace(
+            "warn_unused_ignores = true", "warn_unused_ignores = false"
+        )
+    )
+
+
+def released_mypy():
+    version = _get_version_from_pyproject("mypy")
+    cmd = [sys.executable, "-m", "pip", "install", f"mypy=={version}"]
+    subprocess.run(cmd, check=True)
+
+    # check for unused ignores again
+    config_file = Path("pyproject.toml")
+    config_file.write_text(
+        config_file.read_text().replace(
+            "warn_unused_ignores = false", "warn_unused_ignores = true"
+        )
+    )