Repository navigation
Support discrete constraints in the Client API - #5295
Open
saitcakmak wants to merge 3 commits into
Open
saitcakmak wants to merge 3 commits into
saitcakmak wants to merge 3 commits into
Conversation
Summary: Problem: Random generation encoded ordered numeric choices as integer indices while retaining parameter constraints in value space. Constraint checks could therefore accept candidates that became invalid after transforming back, especially for irregularly spaced values. What changed: This foundation teaches RandomGenerator to sample every discrete dimension uniformly by rank, decode the original value before feasibility and deduplication checks, and project polytope samples onto supported choices. It also handles pure-discrete and mixed discrete/continuous equalities, including correctly shaped equality right-hand sides. Stack structure: 1. D113627036 -- generator mechanics and direct generator tests. 2. D122426289 -- wires the behavior into Sobol and Uniform adapters and safely relaxes integer ranges with more than 1000 values. 3. D122426291 -- public Client parsing and end-to-end coverage. 4. D122426292 -- internal BOPE candidate-generation integration. End-to-end evidence: The public end-to-end tests intentionally live in D122426291 because they require both this generator foundation and the adapter wiring in D122426289. They execute Client.configure_experiment -> GenerationStrategy(random_search) -> registry transforms -> RandomAdapter -> SobolGenerator without mocking candidate generation. Coverage verifies constrained irregular choices, constrained integer ranges, fixed discrete parameters, JSON snapshot and restore, mixed discrete/continuous equality, large integer ranges, exact trial counts, and the Sobol generator key. A three-seed local benchmark on the complete stack improved valid constrained Client candidates from a 0.688 mean to 1.0 and valid mixed-equality candidates from 0.0 to 1.0. Differential Revision: D113627036
Summary: Problem: The generator support in D113627036 is not user-visible until Sobol and Uniform adapters preserve numeric choices. Applying discrete transforms indiscriminately would also eagerly materialize very large integer ranges. What changed: Introduce a dedicated random-adapter transform chain. Numeric choices and integer ranges with at most 1000 values remain discrete, while larger integer ranges use IntToFloat. Sobol and Uniform share the configuration without aliasing the MBM transform list. Stack structure: 1. D113627036 -- generator mechanics and direct generator tests. 2. D122426289 -- this adapter wiring and high-cardinality escape hatch. 3. D122426291 -- public Client parsing and end-to-end coverage. 4. D122426292 -- internal BOPE candidate-generation integration. End-to-end evidence: This diff verifies registry construction and generation-strategy dispatch locally. The public end-to-end tests are in D122426291, where Client.get_next_trials traverses this registry configuration into RandomAdapter and SobolGenerator. Those tests assert actual supported choice values, integer types, satisfied constraints, exact trial counts, and the Sobol generator key. The complete-stack benchmark raises constrained Client validity from a 0.688 mean to 1.0. Differential Revision: D122426289
Summary: Problem: The lower stack layers add discrete-aware random generation and adapter wiring, but public Client usage also needs equality parsing and end-to-end proof that generated parameters remain valid after the full transform round trip. What changed: Recognize == expressions when constructing Client parameter constraints. Add public Client coverage for constrained irregular choices, constrained integer ranges, mixed discrete/continuous equalities, fixed discrete parameters, JSON snapshot and restore, large integer ranges, exact trial counts, and explicit random-search dispatch. Stack structure: 1. D113627036 -- generator mechanics and direct generator tests. 2. D122426289 -- Sobol and Uniform adapter wiring and the high-cardinality escape hatch. 3. D122426291 -- this public API and end-to-end validation layer. 4. D122426292 -- internal BOPE candidate-generation integration. End-to-end evidence: These tests execute Client.configure_experiment -> GenerationStrategy(random_search) -> registry transforms -> RandomAdapter -> SobolGenerator without mocking candidate generation. They verify that returned choices belong to the declared set, integer ranges return integers, linear and equality constraints hold after inverse transforms, fixed values survive generation, restored clients continue generating valid trials, large integer ranges do not require enumeration, and the selected generator key is Sobol. A three-seed local benchmark improved constrained Client validity from [0.734, 0.656, 0.672] to [1.0, 1.0, 1.0] and mixed-equality validity from [0.0, 0.0, 0.0] to [1.0, 1.0, 1.0]. Differential Revision: D122426291
meta-cla
Bot
added
the
CLA Signed
|
@saitcakmak has exported this pull request. If you are a Meta employee, you can view the originating Diff in D122426291. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5295 +/- ##
========================================
Coverage 96.59% 96.59%
========================================
Files 621 621
Lines 70952 71110 +158
========================================
+ Hits 68536 68691 +155
- Misses 2416 2419 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Problem:
The lower stack layers add discrete-aware random generation and adapter wiring, but public Client usage also needs equality parsing and end-to-end proof that generated parameters remain valid after the full transform round trip.
What changed:
Recognize == expressions when constructing Client parameter constraints. Add public Client coverage for constrained irregular choices, constrained integer ranges, mixed discrete/continuous equalities, fixed discrete parameters, JSON snapshot and restore, large integer ranges, exact trial counts, and explicit random-search dispatch.
Stack structure:
End-to-end evidence:
These tests execute Client.configure_experiment -> GenerationStrategy(random_search) -> registry transforms -> RandomAdapter -> SobolGenerator without mocking candidate generation. They verify that returned choices belong to the declared set, integer ranges return integers, linear and equality constraints hold after inverse transforms, fixed values survive generation, restored clients continue generating valid trials, large integer ranges do not require enumeration, and the selected generator key is Sobol.
A three-seed local benchmark improved constrained Client validity from [0.734, 0.656, 0.672] to [1.0, 1.0, 1.0] and mixed-equality validity from [0.0, 0.0, 0.0] to [1.0, 1.0, 1.0].
Differential Revision: D122426291