Repository navigation
Conversation
2 tasks
gaurav
force-pushed
the
setid-post-list
branch
from
October 14, 2026 17:35
85e22d6 to
22f1970
Compare
Tornado's write() refuses a list ("Lists not accepted for security
reasons"), a guard against JSON hijacking through <script> tags in very
old browsers. That made GET /get_allowed_conflations return HTTP 500,
and made POST /get_setid answer with an object keyed "0", "1", ...
instead of the list NodeNorm Redis returns. NodeNormalizationBaseHandler
now encodes a list the way Tornado encodes a dict, so every handler can
finish() one.
/get_allowed_conflations keeps its {"conflations": [...]} wrapper, which
is the shape NodeNorm Redis returns; only its comment changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /get_setid answered with an object keyed "0", "1", ... because Tornado would not write a list. Now that the base handler can, it returns the list of SetIDResponse objects in input order, which is what NodeNorm Redis returns and what the OpenAPI document now says. This is a breaking change for any client that reads body["0"]. The "if not set_identifiers: 500" check after the loop could never fire (the body is already known to be non-empty and each set appends one result), so it is gone. The unit test that mounts the handler with a fake generate_setid() used to assign the module attribute and never put it back, so under pytest-randomly the fake could leak into tests/test_set_identifiers.py. It now patches with unittest.mock and undoes the patch after each test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit used to add the POST /get_setid body checks themselves: a body that wasn't valid JSON, was a single object, or held anything other than objects reached group.get() and failed with HTTP 500. biothings#42 now makes those checks (through load_json_body()) and tests them, so after rebasing onto it all that is left is the empty-body case, which its test didn't cover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each set awaited its own Elasticsearch lookups before the next began, so a POST with many sets took one round trip per set in series. The sets are independent, so asyncio.gather() now runs them together; it returns the results in input order. This raises the number of concurrent Elasticsearch requests one POST can make, in proportion to the number of sets (see biothings#22 on how much load the cluster takes). It is a separate commit so it can be dropped on its own if that is a concern. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gaurav
force-pushed
the
setid-post-list
branch
from
November 9, 2026 19:42
22f1970 to
bc8fb41
Compare
gaurav
added a commit
to TranslatorSRI/babel-validation
that referenced
this pull request
Nov 9, 2026
…document (#153) NodeNorm Redis and NodeNorm ES are meant to be interchangeable for the endpoints they share, but three NodeNorm ES endpoints answered differently from Redis and from ES's own OpenAPI document: `GET /get_allowed_conflations` returned HTTP 500, `POST /get_setid` returns an object keyed `"0"`, `"1"`, … instead of a list, and `/get_semantic_types` lower-cases Biolink classes. This PR adds `tests/nodenorm/test_nodenorm_endpoints.py`, which runs against every target and would have caught all three. It also checks that each deployment documents the endpoints both backends share, and that every path its OpenAPI document lists is actually served. biothings/NodeNormalizationAPI#58, released as NodeNorm ES 1.2.0, fixed the conflations 500; `ci` and `test` run 1.2.0, prod still runs 1.0.0. The setid shape and the semantic-types casing are fixed by biothings/NodeNormalizationAPI#48 and #59, both still open. What is still wrong is a strict xfail on the ES targets, so an unexpected pass reports when a marker can come out. ## Calls worth overruling - **Each xfail passes `raises=KnownElasticsearchDiscrepancy`, and that class is raised only on the bug's own signature.** That means only lower-cased `biolink:` classes, only an object keyed `"0"`..`"n-1"` with a set ID under each key, and only Tornado's bare 500 page while `get_normalized_nodes` still works. Anything else the same check catches fails with a plain assert. A bare strict xfail, or a class raised for "not a list" or "any 500", would also absorb an outage or an unrelated regression. [`CLAUDE.md`](https://github.com/TranslatorSRI/babel-validation/blob/46e69ea/CLAUDE.md#testing-patterns) records the pattern, and why an imperative `pytest.xfail()` doesn't work with the dashboard. - **The conflations xfail is keyed on the version `/status` reports**, not just the backend: ES targets older than 1.2.0 expect the 500, later ones must pass. `nodenorm_version()` in `service_checks.py` reads `1.2.0`, `v1.2.1`, `1.3`, `1.3.0-rc1` and PEP 440's `1.3.0.dev0` or `1.2.0.post1`; a suffix is ignored, so a pre-release counts as its release. A target whose `/status` has no release number at all fails rather than guessing. - **The path sweep skips `/get_allowed_conflations` where its 500 is expected**, instead of being a second xfail for the same bug. `test_get_allowed_conflations` tracks it, and the sweep still checks every other path on prod. - **The shared-endpoint list is the five both backends serve**, not everything Redis documents. NodeNorm ES deliberately leaves out `/query`, `/asyncquery` and `/get_curie_prefixes` (biothings/NodeNormalizationAPI#36). - **A path counts as served unless a bare GET returns 404 or 5xx.** A POST-only or parameter-requiring path legitimately answers 405, 400 or 422. A timeout or refused connection is recorded against its path rather than raised, so one hung path doesn't hide the rest. - **The OpenAPI document is treated as untrusted.** Its path keys must be plain `/segment` paths, and a document may list at most 20, before any URL is built from them, so a key such as `//host/x` cannot send a request elsewhere. The sweep requests five paths at a time with a (5s, 10s) connect/read timeout, so even a hung deployment fails inside pytest-timeout's 300s with each path named. Every message goes through `truncated_repr()`. - **The setid test checks only the shape**, with a single POST. `test_setid_basic` already compares the set IDs from GET and POST. It now asserts the POST succeeded and returned a list, showing the status and body, rather than failing with a `TypeError`. It gets no xfail, because it also fails on Redis targets today (water now normalizes to `CHEBI:15377`), and an xfail would hide that. ## Also here - `service_checks.py` gains `get_openapi()`, which caches each target's OpenAPI document per process the way `get_status()` caches `/status`. Both `test_openapi_json` tests and the new tests share it. - The shared fetch checks for a JSON object inside the cache, so a brief 200 with `null` is fetched again by the next test rather than remembered. - Every request in `test_nodenorm_setid.py` now has a timeout. ## Verification On 46e69ea, against `prod`, `test`, `ci`, `ci-nameres-es`, `dev` and `exp`, every test in the new file passes or xfails as expected. The nine xfails are semantic types and setid on the four ES targets, plus conflations on prod. The unit suite passes. Outside the new file, the touched tests fail on exactly the targets they fail on `main`, for reasons in the deployments: missing or wrong `x-translator` blocks and a `maxEdges` schema error in `test_openapi_json`, non-JSON setid bodies on the ES targets, and water now normalizing to `CHEBI:15377` in `test_setid_basic`. Faked responses confirmed that an error object, a non-Biolink type, or a 500 during an outage or on 1.2.0 fails outright. ## After merging Nothing blocks merging. The markers come out as the strict XPASSes report: conflations once prod reaches 1.2.0, setid with biothings/NodeNormalizationAPI#48, and semantic types with biothings/NodeNormalizationAPI#59. Whether lower-cased Biolink classes would also be acceptable is a separate question, #158. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
POST /get_setidanswers with an object keyed by input index ({"0": {...}, "1": {...}}) because Tornado refuses to write a top-level list. NodeNorm Redis returns a list, and the clients were written against it. This PR letsNodeNormalizationBaseHandlerwrite lists, and then returns theSetIDResponseobjects as a list in input order, as the OpenAPI document now says. This is a breaking change for any client of NodeNorm ES that readsbody["0"].The
babel_versionquery parameter still selects one Babel release for every set in the POST, and the response still carriesX-Babel-Version.Calls worth overruling
<script>tags in very old browsers, which doesn't matter for a public JSON API that allows any origin. Allowing lists in the base handler means the next handler that returns a list doesn't hit the same 500/get_allowed_conflationsdid. The comment onwrite()inhandlers/base.pysays why.asyncio.gather. One POST can now make as many concurrent Elasticsearch requests as it has sets (see Is there a limit to how many simultaneous CURIEs NodeNorm ES can support? #22 on cluster load). It's the last commit, so it can be dropped on its own.What's here
POST /get_setidreturns a JSON array; the unreachable "empty result → 500" check after the loop is gone. The OpenAPI document's 200 response for it is now an array ofSetIDResponse.tests/test_openapi.pythat mounts the handler with a fakegenerate_setid()patched throughunittest.mock, so the fake doesn't leak intotests/test_set_identifiers.pywhen pytest-randomly runs that later. The existing tests of POST errors and of multi-Babel release selection (tests/test_babel_endpoints.py) now read the list, and an empty body is tested to get a 400.tests/test_set_identifiers.pyexpects a list.Before merging
info.versionis still 1.2.0, the version Update OpenAPI for SmartAPI registration and preserve multi-Babel behavior #58 set. This PR changes the shape of a response, and/statusreportsinfo.version, so if 1.2.0 has been or will be deployed without this, it should become 1.3.0, with a line in the README's "OpenAPI document" section.Verification
On bc8fb41, all 290-odd tests pass except the four integration tests that need the
su10host (test_node_normalization_getandtest_node_normalization_postintest_normalized_nodes.py,test_get_endpointandtest_post_endpointintest_set_identifiers.py), which weren't run. So the list-expecting integration test hasn't been run.🤖 Generated with Claude Code