Skip to content

chore: repository health check (tests, security, deps, CI) - #297

Draft
shenxianpeng wants to merge 7 commits into
mainfrom
chore/repo-health-check
Draft

shenxianpeng wants to merge 7 commits into
mainfrom
chore/repo-health-check

Conversation

@shenxianpeng

Copy link
Copy Markdown
Member

Summary

Repository health check: main.py to 100% test coverage, a security review of main.py, action.yml and the workflows, a dependency review, and CI warning cleanup. One commit per category. The three fix: commits change behavior and are listed separately below.

Coverage

pytest --cov=main --cov-branch main_test.py, scope main.py (the shipped code):

Lines Branches
Before 95% (607/637) 219/228
After 100% (649/649) 232/232

No new exclusions. The existing # pragma: no cover on the delimiter-collision loop in _write_output is unchanged. The project has no coverage gate. Now that the suite is at 100%, adding --cov-branch --cov-fail-under=100 to coverage.yml would hold it there. I left that as a suggestion.

Tests added

25 tests in main_test.py (207 → 232). All run offline, with git, the CLI and the API mocked.

  • Paths that had no test: log_env_vars, reading the PR number from the event payload (pull_request_target), a missing PyGithub, token or repository, cleanup of stale reports, an unparsable scope in the report table, the footer without a version, the all-skipped step-log verdict, and running main.py as a script.
  • A regression test for each fix below.
  • if __name__ == "__main__": unittest.main() sat in the middle of main_test.py. As a result, python main_test.py ran only 158 tests, and two of those errored on helpers defined further down. The guard is now at the end, and all 232 tests also run under plain unittest.

Fixes (behavior)

  • PR comments on GitHub Enterprise Server (bed3b4c): add_pr_comments built its PyGithub client without base_url. On GHES the comment went to api.github.com, which rejected the token. The client now uses GITHUB_API_URL, as the commit listing already did. Nothing changes on github.com.
  • Commit text in the step log (c0f5a57): the runner reads every output line for workflow commands. It trims indentation before looking for :: and finds ##[ anywhere in a line, so a commit body quoting a command (::add-mask::…, ::stop-commands::…) was acted on. When a group contains such a line, its lines are now wrapped in ::stop-commands::<random token>. Ordinary logs are byte-for-byte unchanged.
  • Report comment ownership (623ec3e): a marked comment is now edited or deleted only if a bot posted it, the same rule legacy comments already followed. A comment someone else wrote is never rewritten or deleted.

Security hardening

  • Workflows (9815be8):
    • actions/checkout in commit-check.yml is now pinned to its v7.0.1 SHA, like the other workflows.
    • persist-credentials: false where no later step needs authenticated git: commit-check, coverage, test and used-by. create-pull-request sets up its own auth.
    • release-drafter.yml now grants exactly contents: write and pull-requests: write, as commit-check/commit-check does, instead of the repository default.
    • In release.yaml and used-by.yml, the write scopes moved from workflow level to job level.
  • README (25171a9): the result example now passes the JSON through env: and reads it with jq. It used to expand toJSON(...) inside run:, and the scopes quote commit text.
  • Reviewed, no change needed:
    • action.yml passes every input through env:, with no ${{ }} inside run:. It installs wheels only, offline, from the verified download.
    • The report escapes user text for Markdown: code spans with | escaped, and fences longer than any backtick run. Annotations are escaped too.
    • GITHUB_OUTPUT uses a random heredoc delimiter.
    • The CLI never sends GITHUB_TOKEN anywhere, and fetches parent configs over HTTPS only, with a timeout.
  • Tools:
    • zizmor: remaining findings are listed under decisions (org reusable workflows at @main, the release job's needed credentials, Dependabot cooldown).
    • actionlint: clean.
    • bandit: 11 low findings, all subprocess calls without a shell, by design.
    • pip-audit on the installed closure: no known vulnerabilities.
    • No open Dependabot, code-scanning or secret-scanning alerts.

Dependencies

Everything is current. commit-check 2.18.2 and PyGithub 2.10.0 are the latest releases, every action pin matches its latest release tag, and Python 3.14 is the newest stable release (3.15 is at rc2). Bot PRs worth merging:

CI warnings

  • release.yaml: removed branches: under the release event (cea587a). actionlint flags it because it isn't a filter for that event, and GitHub ignored it.
  • Not fixable here:
    • Release Drafter's deprecation warnings come from the org config commit-check/.github:.github/release-drafter.yml (categories[*].labels, exclude-labels, version-resolver.major.labels).
    • The notices about ubuntu-latest moving to Ubuntu 26 on Oct 19 and about macOS queue times are informational.

Needs a maintainer decision

  • GHES attestation: gh attestation verify in action.yml reads GITHUB_TOKEN, which gh uses for github.com. On a GHES runner that variable holds the server's token, so verification would fail with "Bad credentials" before main.py runs. The README lists GHES as a reason to pick the action. I left this unchanged because it needs a design choice, for example downloading the attestation bundle separately without auth.
  • Transitive dependencies: only commit-check and PyGithub are pinned. Their dependencies resolve fresh on each run and are not hash-checked, yet the README says they are "pinned by requirements.txt". A hash-locked requirements file (e.g. uv pip compile --universal --generate-hashes) would make that statement true.
  • Python 3.10 reaches end of life on 2026-10-31. It is the floor in test.yml, coverage.yml and the README. Dropping it is a separate change.
  • Dependabot cooldown (zizmor): github-actions has no cooldown and pip has default-days: 0. I left both as they are because they look deliberate.
  • Coverage gate: see Coverage.

How it was verified

  • pytest --cov=main --cov-branch main_test.py on Python 3.10: 232 passed, 100% of lines and branches. The same suite with -W error on Python 3.14: 232 passed.
  • python main_test.py: 232 tests OK.
  • pre-commit run --all-files: all hooks pass.
  • actionlint: clean. zizmor: only the items above remain.
  • Each fix's tests fail on the code before the fix and pass after it.
  • End to end: ran main.py with the real commit-check 2.18.2 on a scratch repo whose failing commit body quotes ::add-mask:: and ##[warning]. The quoted tree is wrapped in the stop-commands block, while ::group::/::endgroup:: and the ::error annotation sit outside it.
  • Every commit message, the branch name and the PR title pass commit-check with this repository's config.

Bring main.py to 100% line and branch coverage (95% before): the debug
log of the inputs, the PR number read from the event payload, a missing
PyGithub, token or repository, stale report clean-up, an unparsable scope
in the report table, the footer without a version, and running the
script itself.

Also move `if __name__ == "__main__": unittest.main()` to the end of the
file. It sat in the middle, so `python main_test.py` ran 158 of the tests,
and two of those errored on helpers defined after it.
add_pr_comments built its PyGithub client without a base_url, so on
GitHub Enterprise Server the comment went to api.github.com, which
rejected the server's token, and the report was never posted. The
commit listing already read GITHUB_API_URL; both clients now take it
from one helper.
…ng on it

The runner reads every line of a step's output for workflow commands. It
trims indentation before looking for `::`, and it finds the older `##[`
syntax anywhere in a line, so the tree's quoted commit bodies, PR titles
and branch names could be taken as commands: a commit documenting
`::add-mask::` or `::stop-commands::` changed the log it appeared in.

A group whose lines could be read that way is now fenced with
`::stop-commands::` and a random token; the CLI notices relayed to stderr
get the same treatment. Ordinary output carries no fence and is unchanged.
The hidden marker alone decided which comments the action edits and
deletes. A marker pasted into someone's own comment leaves it that
person's comment, so marked comments now need a bot author too, as
legacy ones already did: the action never rewrites or deletes a comment
it did not post.
…missions

- commit-check.yml: pin actions/checkout to its v7.0.1 SHA, like every
  other workflow, and give the workflow a read-only default.
- checkout in commit-check, coverage, test and used-by no longer leaves
  the token in .git/config: nothing after it needs authenticated git, and
  create-pull-request sets up its own auth for the push.
- release-drafter: grant the reusable workflow the two write scopes it
  needs rather than the repository default, as commit-check/commit-check
  does.
- release and used-by: the write scopes move from the workflow to the one
  job that uses them.
actionlint: "branches" is not a filter for the release event. GitHub
ignores it, so the workflow already ran for every published release;
removing it changes nothing but the warning.
The example expanded `toJSON(...scopes)` directly inside `run:`. The
scopes quote commit text as it is, so follow GitHub's guidance for such
values: pass the JSON in through `env:` and read it with jq.
@shenxianpeng shenxianpeng added the chore Choses update label Oct 1, 2026
@coderabbitai

coderabbitai Bot commented Oct 1, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Commit Check

✅ All 11 checks passed

Show all 11 checks
Commit message
  ✔ PR title (chore: repository health check (tests, security, deps, CI))
  ✔ Commit 1/7 (0bfc57d) (test: cover the remaining paths in main.py)
  ✔ Commit 2/7 (bed3b4c) (fix: post PR comments to the API of the instance the run ...)
  ✔ Commit 3/7 (c0f5a57) (fix: print quoted commit text to the step log without the...)
  ✔ Commit 4/7 (623ec3e) (fix: only adopt or delete report comments a bot posted)
  ✔ Commit 5/7 (9815be8) (ci(security): pin checkout, drop persisted tokens and sco...)
  ✔ Commit 6/7 (cea587a) (ci: drop the branches filter the release event never applied)
  ✔ Commit 7/7 (25171a9) (docs: read the result output through env in the README ex...)
Branch
  ✔ Branch (chore/repo-health-check)
Author
  ✔ Author name (Xianpeng Shen)
  ✔ Author email (xianpeng.shen@gmail.com)

commit-check 2.18.2 · Rules reference

@codecov

codecov Bot commented Oct 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (1ad9eb5) to head (25171a9).

Additional details and impacted files
@@             Coverage Diff             @@
##             main      #297      +/-   ##
===========================================
+ Coverage   95.29%   100.00%   +4.70%     
===========================================
  Files           1         1              
  Lines         637       649      +12     
===========================================
+ Hits          607       649      +42     
+ Misses         30         0      -30     
Flag Coverage Δ
unittests 100.00% <100.00%> (+4.70%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Choses update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant