Skip to content

Drop weakly dominated rows from the Pareto front - #78

Open
0z5a wants to merge 2 commits into
NVIDIA:mainfrom
0z5a:fix/pareto-weak-dominance
Open

0z5a wants to merge 2 commits into
NVIDIA:mainfrom
0z5a:fix/pareto-weak-dominance

Conversation

@0z5a

@0z5a 0z5a commented Nov 17, 2026

Copy link
Copy Markdown

Description

SearchResult.calculate_pareto_front dropped a row only when the current point was strictly better on every objective (np.any(scores <= point) kept everything else). Rows that tie the point on one objective and are worse on another stayed on the front. That also affects pareto_front(), get_best_result() (default for multi-objective) and Search.best_result.

This PR keeps a row when it is strictly better than the point on some objective, or equal to it on every objective. Identical rows still all stay on the front, as test_identical_points_both_on_front expects. That is the definition written in the test class docstrings: A dominates B if A is no worse on every objective and strictly better on at least one.

Two existing assertions encoded the old behavior and are updated:

  • MIN: (1,5) next to (1,1) is now excluded (it was asserted to stay "equal on dim 0").
  • MAX: (5,1) next to (5,5) is now excluded (it was asserted to "trade off").

Closes #77

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Validation

  • pytest tests/unit/test_results_pareto.py: on main 6 of 14 tests fail (the 2 updated assertions plus 4 new tests). With this change all 14 pass.
  • New tests:
    • tie-on-one-objective for MIN and MAX
    • identical rows kept while a row they dominate is dropped
    • comparison with a brute-force dominance check over 50 seeded sets of 30 rows with 3 integer objectives
  • Unit suite: the same set of tests passes and fails before and after, plus the 4 new tests passing.
  • Brute-force agreement over 1,000 random integer score sets (MIN and MAX, 2–3 objectives): main 24/1000, this PR 1000/1000.
  • calculate_pareto_front is only called after the search finishes, so this is not a hot path. With 5,000 rows of (time, spills), the median of 5 runs goes from 50.49 ms to 2.45 ms, because fewer rows survive each pass.

Bug fix

import pandas as pd
from compileiq.results import SearchResult
from compileiq.types import ProblemType

df = pd.DataFrame(
    [[0, 0, 1, 1, {"cfg": "a"}], [1, 0, 1, 5, {"cfg": "b"}]],
    columns=["metadata", "generation", "score_1", "score_2", "params"],
)
r = SearchResult(df=df, problem_type=ProblemType.MIN, num_scores=2)
print([(row["score_1"], row["score_2"]) for row in r.pareto_front()])
# main:    [(1, 1), (1, 5)]
# this PR: [(1, 1)]

This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.

calculate_pareto_front dropped a row only when the current point was
strictly better on every objective, so rows that tie on one objective
and are worse on another stayed on the front, and with it in
pareto_front(), get_best_result() and Search.best_result. Keep a row
only when it is strictly better on some objective or equal on all of
them; identical rows still all stay. Update the two assertions that
encoded the old behavior.

Closes NVIDIA#77

Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@pnwamk

pnwamk commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the fix and the clear reproducer. Agreed that this is the preferred behavior. I'll run the CI.

@pnwamk

pnwamk commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

/ok to test b5d4a3f

@pnwamk

pnwamk commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 160652c

@pnwamk
pnwamk enabled auto-merge (squash) November 18, 2026 02:30

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pareto_front() returns rows that another row dominates when scores tie on one objective

2 participants