Repository navigation
Conversation
The add helper stored a new node's 'type' as a string while the graph and the prompt use a list, and named a new switch's port after the dict repr. Store 'type' as a list, name the port '<switch name>.p1', and state the new port and its capacity (1000) in the prompt. Also fix the RK_CONTROLS edge type name in the prompt and add the missing name attribute in the add example.
The graph comparison ignored edge direction and edge type, so a remove answer with reversed edges passed. Compare as directed graphs and match edge 'type'.
Graph answers are compared with edge types since the previous commit, but the prompt did not say which edge joins a CONTROL_DOMAIN to a PACKET_SWITCH, and 133 of the 500 level-1 add queries add a switch to a domain. State 'CONTROL_DOMAIN contains PACKET_SWITCH', as in the graph (the 8 domains contain all 320 switches through RK_CONTAINS edges) and in error_check.py, and test that the hierarchy covers the parent of every level-1 add query.
The three check_graph_equal tests built the same kind of graph pair and one of them mixed a negative and a positive case. Use one parametrized test with a case per variant: copy, extra node, reversed edge, wrong edge type and list edge type.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #21.
typeis a list, and the port created with a new switch is<switch name>.p1. The prompt now says how that port is named and that new ports getphysical_capacity_bps1000. I also fixedRK_CONTROL->RK_CONTROLSin the prompt (the graph has noRK_CONTROLedges) and added the missingnameto the few-shot add example.typewritten as a list still matches.CONTROL_DOMAIN contains PACKET_SWITCH(133 of the 500 level-1 add queries add a switch to a domain).With these, the reference answer from #21 passes all 5,000 queries in
sampled_500_benchmark_malt.jsonl, and one reversed or mistyped edge fails all 500 level-1 remove queries. 4 new tests (8 cases), all passing:Old and new MALT scores aren't directly comparable, since the prompt and scoring change for every query.
mainhas the same code if you want the fix there too. And if you'd prefer 1e9 over 1000 for new port capacity, it's a small change.