Codecov failing on 3.11

[ezio-melotti]

In https://github.com/python/bedevere/runs/6651826354?check_suite_focus=true the test fail because codecov doesn't reach 100%:

See full pytest output

Run pytest --cov --cov-report=xml
  pytest --cov --cov-report=xml
  shell: /usr/bin/bash -e {0}
  env:
    FORCE_COLOR: 1
    pythonLocation: /opt/hostedtoolcache/Python/3.11.0-beta.1/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib
============================= test session starts ==============================
platform linux -- Python 3.11.0b1, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/runner/work/bedevere/bedevere, configfile: pytest.ini, testpaths: tests
plugins: cov-3.0.0, asyncio-0.18.3, aiohttp-1.0.4
asyncio: mode=Mode.AUTO
collected 172 items

tests/test___main__.py ...                                               [  1%]
tests/test_backport.py ..........................                        [ 16%]
tests/test_close_pr.py .......                                           [ 20%]
tests/test_filepaths.py ............                                     [ 27%]
tests/test_gh_issue.py ....................................              [ 48%]
tests/test_news.py .................................................     [ 77%]
tests/test_prtype.py .........                                           [ 82%]
tests/test_stage.py ....................                                 [ 94%]
tests/test_util.py ..........                                            [100%]

----------- coverage: platform linux, python 3.11.0-beta-1 -----------
Coverage XML written to file coverage.xml

FAIL Required test coverage of 100.0% not reached. Total coverage: 99.02%

============================= 172 passed in 6.27s ==============================
Error: Process completed with exit code 1.

This however only seems to affect 3.11.0-apha, and not the other Python versions. This should be fixed however -- apparently because of #462 -- the uploaded report doesn't include the failure:

See full codecov output

Run codecov/codecov-action@v2
  with:
    file: ./coverage.xml
  env:
    FORCE_COLOR: 1
    pythonLocation: /opt/hostedtoolcache/Python/3.11.0-beta.1/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (648b599397548e4bb92429eec6391374c2cbb0edb835e3b3f03d4281c011f401  codecov)
==> Running version latest
==> Running version v0.2.3
/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov -n  -Q github-action-2.1.0 -f ./coverage.xml -C 571de6f456fc8379015f221d8b2d796f1bc88818
[2022-05-30T09:11:54.662Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.2.3
[2022-05-30T09:11:54.669Z] ['info'] => Project root located at: /home/runner/work/bedevere/bedevere
[2022-05-30T09:11:54.672Z] ['info'] -> No token specified or token is empty
[2022-05-30T09:11:54.781Z] ['info'] Searching for coverage files...
[2022-05-30T09:11:54.839Z] ['info'] => Found 1 possible coverage files:
  ./coverage.xml
[2022-05-30T09:11:54.839Z] ['info'] Processing ./coverage.xml...
[2022-05-30T09:11:54.845Z] ['info'] Detected GitHub Actions as the CI provider.
[2022-05-30T09:11:54.847Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.2.3&token=*******&branch=dependabot%2Fpip%2Fcachetools-5.2.0&build=2407989696&build_url=https%3A%2F%2Fgithub.com%2Fpython%2Fbedevere%2Factions%2Fruns%2F2407989696&commit=571de6f456fc8379015f221d8b2d796f1bc88818&job=Tests&pr=460&service=github-actions&slug=python%2Fbedevere&name=&tag=&flags=&parent=
[2022-05-30T09:11:55.423Z] ['info'] https://codecov.io/github/python/bedevere/commit/571de6f456fc8379015f221d8b2d796f1bc88818
https://storage.googleapis.com/codecov/v4/raw/2022-05-30/D77050ECB3DC286B3CD9265D83DDA42F/571de6f456fc8379015f221d8b2d796f1bc88818/e482bb87-1a14-49a8-998d-c64096a45b86.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EKKHVKCKHW7KBCGM7IHR55T63V2PAVJWLVFNITJHDU5G6R5IRN3LMWJA%2F20220530%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20220530T091155Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ba144246e7b0cc69b67e35caad1afce4ce9fbdc60902a813d9854590416d23e9
[2022-05-30T09:11:55.423Z] ['info'] Uploading...
[2022-05-30T09:11:55.624Z] ['info'] {"status":"success","resultURL":"https://codecov.io/github/python/bedevere/commit/571de6f456fc8379015f221d8b2d796f1bc88818"}

[hugovk]

Technically Codecov isn't failing, the pytest/coverage check is failing because there's less than 100% coverage for each individual build (fail_under = 100 in setup.cfg).

When all the coverage uploaded to Codecov and combined, the aggregate coverage is 100%.

If we add --cov-report=term to the pytest run (hugovk@186818e), we can 100% coverage for 3.10 but some missing coverage for 3.11:

Name                      Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------------
bedevere/__init__.py          0      0      0      0   100%
bedevere/__main__.py         37      0      4      1    98%
bedevere/backport.py         63      0     36      0   100%
bedevere/close_pr.py         22      0     14      0   100%
bedevere/filepaths.py        15      0     12      0   100%
bedevere/gh_issue.py         72      0     34      1    99%
bedevere/news.py             51      0     28      0   100%
bedevere/prtype.py           33      0     19      0   100%
bedevere/stage.py           117      0     60      9    95%
bedevere/util.py             67      0     24      1    99%
tests/__init__.py             0      0      0      0   100%
tests/test___main__.py       25      0      0      0   100%
tests/test_backport.py      146      0     22      0   100%
tests/test_close_pr.py       95      0     16      0   100%
tests/test_filepaths.py     106      0      8      0   100%
tests/test_gh_issue.py      249      0     96      9    97%
tests/test_news.py          155      0     28      0   100%
tests/test_prtype.py        103      0      2      0   100%
tests/test_stage.py         291      0     10      0   100%
tests/test_util.py           73      0      6      0   100%
-----------------------------------------------------------
TOTAL                      1720      0    419     21    99%
Coverage XML written to file coverage.xml

FAIL Required test coverage of 100.0% not reached. Total coverage: 99.02%

(Aside: PR #461 would have prevented me from running the above tests without (a) creating a spammy PR, (b) temporarily editing the workflow to allow the feature branch to run or (c) making the changes in main.)

htmlcov.zip is the output of running locally with --cov-report html on Python 3.11. The missing lines are things like:

[ezio-melotti]

Is there a reason why 3.10 and 3.11 report different coverage for the same code?

[hugovk]

That's a very good question... I've not seen these partial coverage results in Coverage.py before.

It could be a new feature only in 3.11, or (more likely?) a bug in Coverage.py for 3.11, or possibly in Python 3.11 itself?

[ezio-melotti]

It could be a new feature only in 3.11, or (more likely?) a bug in Coverage.py for 3.11, or possibly in Python 3.11 itself?

This is what I'm wondering 🤔

All the issues seem to be with async with, async for, and if await lines.

[AlexWaygood]

It could be a new feature only in 3.11, or (more likely?) a bug in Coverage.py for 3.11, or possibly in Python 3.11 itself?

This is what I'm wondering 🤔

All the issues seem to be with async with, async for, and if await lines.

Maybe this bug is the cause? (I think it will be fixed in beta 2.)

• Incorrect tracing of "if" inside async-for cpython#93061

[hugovk]

Good find!

Using https://github.com/deadsnakes/action to test a nightly build of 3.11: hugovk@c045c8f

And 3.11 passes with 100% coverage :)

https://github.com/hugovk/bedevere/runs/6659010706?check_suite_focus=true#step:7:48

3.11 beta 2 is due today, so all should be good tomorrow or the day after when it reaches GitHub Actions.

[ezio-melotti]

I'll leave this open for now and check back tomorrow to see if the issue is solved.

[ezio-melotti]

It took yet another beta, but this seems now fixed.

[AlexWaygood]

It took yet another beta, but this seems now fixed.

I remember a time when you had to wait a month before your next beta. Now? A dozen a day. Smdh.