Skip to content

Fix pytest result parsing counting more tests than were run - #5

Open
KTanmay1 wants to merge 1 commit into
DeepSoftwareAnalytics:mainfrom
KTanmay1:fix/pytest-result-count
Open

Fix pytest result parsing counting more tests than were run#5
KTanmay1 wants to merge 1 commit into
DeepSoftwareAnalytics:mainfrom
KTanmay1:fix/pytest-result-count

Conversation

@KTanmay1

Copy link
Copy Markdown

Problem

RepoTransAgent/test_analyzer.py:126

test_pattern = r'(\S+)::\S+\s+(PASSED|FAILED|ERROR|SKIPPED)'
matches = re.findall(test_pattern, stdout)

\s matches newlines, so a test id ending one line pairs with a status token beginning the next. pytest's short summary emits exactly that shape for setup and teardown errors, whose lines carry no trailing reason:

ERROR tests/test_smaz.py::test_one
ERROR tests/test_smaz.py::test_two

Line 1 ends in ::test_one, line 2 begins with ERROR, and the pair matches. Chained across the block, N entries are counted as 2N - 1.

Observed on antirez_smaz (C → Python) against an empty tree: 18 tests run, 35 counted.

Reproduction

import re
pat = r'(\S+)::\S+\s+(PASSED|FAILED|ERROR|SKIPPED)'
out = """
tests/test_smaz.py::test_one ERROR                                       [ 33%]
tests/test_smaz.py::test_two ERROR                                       [ 66%]
tests/test_smaz.py::test_three ERROR                                     [100%]

=========================== short test summary info ============================
ERROR tests/test_smaz.py::test_one
ERROR tests/test_smaz.py::test_two
ERROR tests/test_smaz.py::test_three
============================== 3 errors in 0.02s ===============================
"""
print(len(re.findall(pat, out)))                 # 5  -- 3 tests ran
print(len(re.findall('^' + pat, out, re.M)))     # 3  -- correct

Scope

Only entries with no trailing reason chain:

  • setup/teardown errors chain — ERROR tests/x.py::test_one is bare
  • collection/import errors do not — they print ERROR tests/x.py, with no :: to match
  • failures do not — FAILED tests/x.py::test_y - AssertionError: ... has text after the id

Because Ni inflates while Ti does not, APR and AMPR are depressed on affected tasks. A fully passing run is unaffected, so SR is unchanged.

Fix

Anchor to line start and restrict the separator to spaces and tabs.

Verification

Through PythonTestAnalyzer.analyze_output, not the regex alone:

input before after
3 setup errors total=5 total=3, rate 0.0%
2 passed, 1 failed total=3 total=3, rate 66.7%
2 passed total=2 total=2, rate 100.0%

This changes reported APR/AMPR on tasks with setup or teardown errors. Flagging that explicitly since it affects published results — happy to hold if you would rather land it alongside a re-run.

`test_pattern` uses `\s+` between the test id and the status token. `\s`
matches newlines, so a test id ending one line pairs with a status token
beginning the next. pytest's short summary emits exactly that shape for
setup and teardown errors, whose lines carry no trailing reason:

    ERROR tests/test_smaz.py::test_one
    ERROR tests/test_smaz.py::test_two

Line 1 ends in `::test_one`, line 2 begins with `ERROR`, and the pair
matches. Chained across the block, N entries are counted as 2N-1.

Observed on antirez_smaz (C -> Python) against an empty tree: 18 tests
run, 35 counted. Reproduced standalone at 3 -> 5.

Only entries with no trailing reason chain. Collection errors print
`ERROR tests/x.py` with no `::` and never matched. Failures print
`FAILED tests/x.py::test_y - AssertionError: ...`, so nothing chains onto
them. A fully passing run is unaffected.

Because Ni inflates while Ti does not, APR and AMPR are depressed on
affected tasks; SR is unchanged.

Fix: anchor the match to the start of a line and restrict the separator
to spaces and tabs.

Verified against PythonTestAnalyzer.analyze_output:

    3 setup errors  total=3 (was 5)  rate=0.0%
    2 pass 1 fail   total=3          rate=66.7%
    all passing     total=2          rate=100.0%

This changes reported APR/AMPR on tasks with setup or teardown errors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant