Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .github/workflows/issue-difficulty.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Issue difficulty

# Every issue needs a `difficulty: hard|medium|low` label so that the PR
# closing it can be scored on the leaderboard. This workflow creates those
# labels if they're missing and marks any issue that lacks one with
# `needs difficulty`, so untriaged issues are easy to find:
# https://github.com/secfoo-com/secfoo/issues?q=is:open+label:"needs+difficulty"

on:
issues:
types: [opened, reopened, labeled, unlabeled]
workflow_dispatch:
inputs:
backfill:
description: "Flag every open issue that has no difficulty label"
type: boolean
default: true

permissions: read-all

jobs:
triage:
name: Check difficulty label
# Ignore the events this workflow causes itself (adding/removing
# `needs difficulty`), so it can't loop.
if: github.event_name == 'workflow_dispatch' || github.event.label.name != 'needs difficulty'
runs-on: ubuntu-latest
permissions:
issues: write
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}

steps:
- name: Ensure the labels exist
run: |
gh label create "difficulty: hard" --force --color b60205 --description "Leaderboard: 10 points when the PR closing this is merged"
gh label create "difficulty: medium" --force --color d93f0b --description "Leaderboard: 6 points when the PR closing this is merged"
gh label create "difficulty: low" --force --color 0e8a16 --description "Leaderboard: 3 points when the PR closing this is merged"
gh label create "needs difficulty" --force --color fbca04 --description "Maintainer: add difficulty: hard, medium or low so this can be scored"

- name: Flag or clear this issue
if: github.event_name == 'issues'
env:
ISSUE: ${{ github.event.issue.number }}
run: |
if gh issue view "$ISSUE" --json labels --jq '.labels[].name' | grep -qiE '^difficulty'; then
gh issue edit "$ISSUE" --remove-label "needs difficulty" || true
else
gh issue edit "$ISSUE" --add-label "needs difficulty"
fi

- name: Backfill open issues
if: github.event_name == 'workflow_dispatch' && inputs.backfill
run: |
gh issue list --state open --limit 500 --json number,labels \
--jq '.[] | select(([.labels[].name] | map(test("^difficulty"; "i")) | any) | not) | .number' \
| while read -r number; do
echo "#$number needs a difficulty label"
gh issue edit "$number" --add-label "needs difficulty"
done
93 changes: 93 additions & 0 deletions .github/workflows/leaderboard.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
name: Leaderboard

# Scores merged PRs by the difficulty of the issues they close and publishes
# the result to the `leaderboard` branch (LEADERBOARD.md, leaderboard.json,
# badge.json). Rules live in scripts/leaderboard.py and CONTRIBUTING.md.
#
# main is branch-protected, so the generated files go to their own branch
# instead: https://github.com/secfoo-com/secfoo/blob/leaderboard/LEADERBOARD.md

on:
pull_request_target:
# pull_request_target (not pull_request) so PRs from forks -- which is
# most hackathon PRs -- get a token that can comment and push. Safe
# because this workflow never checks out or runs the PR's code; it only
# reads merge metadata through the API.
types: [closed]
branches: [main]
issues:
# Re-score when a maintainer adds a difficulty label after the fact.
types: [labeled, unlabeled]
schedule:
# Every 6 hours, as a catch-all.
- cron: "15 */6 * * *"
workflow_dispatch:

permissions: read-all

# One publish at a time so two merges in quick succession don't race on the
# leaderboard branch.
concurrency:
group: leaderboard
cancel-in-progress: false

jobs:
publish:
name: Score and publish
if: >-
(github.event_name != 'pull_request_target' && github.event_name != 'issues')
|| (github.event_name == 'pull_request_target' && github.event.pull_request.merged == true)
|| (github.event_name == 'issues' && startsWith(github.event.label.name, 'difficulty'))
runs-on: ubuntu-latest
permissions:
contents: write # push to the leaderboard branch
pull-requests: write # comment the award on the merged PR
env:
# Optional tuning. LEADERBOARD_SINCE counts only PRs merged at/after an
# ISO-8601 instant (e.g. "2026-11-08T00:00:00Z" for hackathon-only);
# LEADERBOARD_EXCLUDE is a comma-separated list of logins to leave off
# (e.g. maintainers).
LEADERBOARD_SINCE: ""
LEADERBOARD_EXCLUDE: ""

steps:
- name: Checkout default branch
# No `ref:` on purpose -- for pull_request_target this is the base
# branch, never the PR head. Credentials stay so the push step works.
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Build leaderboard
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
MERGED_PR: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.number || '' }}
run: |
args=(--repo "$GITHUB_REPOSITORY" --out build/leaderboard)
[ -n "$LEADERBOARD_SINCE" ] && args+=(--since "$LEADERBOARD_SINCE")
[ -n "$LEADERBOARD_EXCLUDE" ] && args+=(--exclude "$LEADERBOARD_EXCLUDE")
[ -n "$MERGED_PR" ] && args+=(--announce-pr "$MERGED_PR")
python3 scripts/leaderboard.py "${args[@]}"

- name: Publish to the leaderboard branch
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
if git ls-remote --exit-code --heads origin leaderboard >/dev/null; then
git fetch --depth=1 origin leaderboard
git checkout -B leaderboard origin/leaderboard
else
# First run: start the branch empty so it carries only the
# generated files, not a copy of main.
git checkout --orphan leaderboard
git rm -rfq --cached .
git clean -fdxq -e build
printf '# secfoo leaderboard branch\n\nGenerated by the Leaderboard workflow on main. See LEADERBOARD.md.\n' > README.md
git add README.md
fi
cp build/leaderboard/LEADERBOARD.md build/leaderboard/leaderboard.json build/leaderboard/badge.json .
git add LEADERBOARD.md leaderboard.json badge.json
if git diff --cached --quiet; then
echo "Leaderboard unchanged; nothing to publish."
exit 0
fi
git commit -q -m "leaderboard: update $(date -u +%Y-%m-%dT%H:%MZ)"
git push origin leaderboard
33 changes: 33 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,39 @@ fine with the generic activity page (`activities/detail.html`).
- Explain the *why* in the PR description, not just the what — what
broke, or what was missing, that this fixes.

## Hackathon points

Merged PRs earn points on the
[contributor leaderboard](https://github.com/secfoo-com/secfoo/blob/leaderboard/LEADERBOARD.md),
based on the difficulty of the issue they close:

| Issue label | Points |
|----------------------|-------:|
| `difficulty: hard` | 10 |
| `difficulty: medium` | 6 |
| `difficulty: low` | 3 |

| Level | Points needed |
|---------|--------------:|
| Level 1 | 100 |
| Level 2 | 50 |
| Level 3 | 25 |

How it works:

- Maintainers put one `difficulty:` label on every issue. Issues without
one carry `needs difficulty` until that's done; if you pick one up, ask
for a difficulty in the issue before you start so you know its value.
- Link the issue from your PR (`Fixes #123` in the description, or the
"Development" sidebar). Only issues GitHub records as closed by the
merged PR are scored. A PR that closes several issues scores each one.
- Each issue is scored once, for the first merged PR that closes it.
- The board is rebuilt from scratch after every merge and every label
change, so an issue labelled after your PR merged still counts. The
workflow leaves a comment on your PR with the points it earned.
- Scoring lives in `scripts/leaderboard.py`; the rules above are the
constants at the top of that file.

## Reporting bugs

Open a GitHub issue with the exact command you ran, what you expected,
Expand Down
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,5 +148,12 @@ PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) and
[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). Security issues go through
[SECURITY.md](SECURITY.md), not a public issue.

Hackathon? Issues tagged [`hackathon`](https://github.com/secfoo-com/secfoo/issues?q=is%3Aopen+label%3Ahackathon)
are open for contribution, and merged PRs earn points on the
[contributor leaderboard](https://github.com/secfoo-com/secfoo/blob/leaderboard/LEADERBOARD.md)
(10 / 6 / 3 for hard / medium / low issues; levels at 25, 50 and 100).

<a href="https://github.com/secfoo-com/secfoo/blob/leaderboard/LEADERBOARD.md"><img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fsecfoo-com%2Fsecfoo%2Fleaderboard%2Fbadge.json" alt="Hackathon leaderboard"></a>

Choose a model for one assessment with `secfoo run --skill sast --agent claude --model <model-id>`,
or set a per-agent default in [configuration](docs/REFERENCE.md#configuration-secfooconfigtoml).
Loading
Loading