Skip to content

fix(metrics): rank a NaN combined_score as the worst score - #603

Open
Bhaveshmeghwal21 wants to merge 2 commits into
algorithmicsuperintelligence:mainfrom
Bhaveshmeghwal21:fix/nan-combined-score
Open

Bhaveshmeghwal21 wants to merge 2 commits into
algorithmicsuperintelligence:mainfrom
Bhaveshmeghwal21:fix/nan-combined-score

Conversation

@Bhaveshmeghwal21

@Bhaveshmeghwal21 Bhaveshmeghwal21 commented Nov 19, 2026 •

Copy link
Copy Markdown

get_fitness_score returns combined_score as it is, so a NaN score (a numpy 0/0, the mean of an empty array) gives the program a NaN fitness. Every comparison with NaN is False, so _is_better is never true against it. If that program is the first one in the database, a cell or an island, nothing can replace it and it stays the best program:

db.add(Program(id="nan", code="x = 1", metrics={"combined_score": float("nan")}))
db.add(Program(id="good", code="x = 2", metrics={"combined_score": 0.9}))
db.best_program_id  # "nan" before, "good" after

A NaN combined_score now gives -inf, so that program ranks below every other one, negative scores included. Valid scores, inf and -inf are unchanged.

I first fell back to averaging the other metrics, as for a combined_score that cannot be converted to a float. That let a NaN program that also reported eval_time or similar outrank valid programs, so I dropped it.

Not changed: process_parallel.py sorts the programs shown in the prompt by the raw combined_score, so a NaN still sorts oddly there. Using get_fitness_score for that sort would be a small follow-up.

tests/test_nan_combined_score.py has four tests. Two fail without the change; the other two check that valid scores are unchanged and that a NaN program does not replace a valid one. The full unittest run has the same 33 failing tests with and without the change (all environment-related on my Windows checkout).

get_fitness_score returned a NaN combined_score as it was. NaN loses every
comparison, so a program with that fitness could never be beaten and stayed
the best program. Fall back to the other metrics, as for any other unusable
combined_score.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

… other metrics

Averaging the other metrics for a NaN combined_score let a program with a NaN
score outrank valid programs when it reported another metric such as eval_time.
Return -inf so it ranks below every other program, and test that a valid
program is not replaced by one.
@Bhaveshmeghwal21 Bhaveshmeghwal21 changed the title fix(metrics): ignore a NaN combined_score when computing fitness fix(metrics): rank a NaN combined_score as the worst score Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants