Repository navigation
fix(chronicle): prevent response data loss in nl_search and search_udm when :udmSearch returns stats - #228
Open
dandye wants to merge 2 commits into
Open
fix(chronicle): prevent response data loss in nl_search and search_udm when :udmSearch returns stats#228dandye wants to merge 2 commits into
dandye wants to merge 2 commits into
Conversation
Attach normalized stats results via process_stats_results along with too_many_results when :udmSearch returns a stats payload on Chronicle SIEM YARA-L 2.0 statistical queries. Plain event filter queries without a stats payload continue returning only events, total_events, and more_data_available. Fixes google#227
Match ChronicleClient.get_stats() 3-key stats dict, fold tooManyResults into more_data_available, guard non-dict stats payloads, and clarify search_udm and nl_search docstrings.
Contributor
Author
|
@prachib29 Could you please review this backward-compatible stability fix when you have a chance? It resolves HTTP 200 response data loss in |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #227
Summary
This stability fix prevents HTTP 200 response data loss and false-negative empty results in
ChronicleClient.nl_search,ChronicleClient.search_udm, andsecops search --nl-query.When a caller runs a natural language prompt such as
"Count failed user logins by user in the last 24 hours"viaclient.nl_search(...)orsecops search --nl-query, Chronicle's:translateUdmQueryendpoint emits a Chronicle SIEM YARA-L 2.0 query withmatch:andoutcome:sections.nl_searchpasses that translated query directly toclient.search_udm(...), andsecops statshas no--nl-queryflag.GET /v1alpha/{instance}:udmSearchreturns HTTP 200 withresult["stats"]and omitsevents. Previously,search_udmdiscardedresult["stats"]and returned{"events": [], "total_events": 0, "more_data_available": False}, reporting zero matches even when the API returned matching security records. Passingtranslate_nl_to_udm(...)output tosearch_udm(...)as shown inREADME.mdhit the same bug.Changes
as_list=Falseandisinstance(result.get("stats"), dict)holds,search_udmsetsresponse["stats"] = process_stats_results(raw_stats)using the existing helper insecops.chronicle.stats, matching the exactcolumns,rows, andtotal_rowsdictionary structure ofChronicleClient.get_stats.raw_stats.get("tooManyResults", False)into the existingmore_data_availableboolean so callers still detect truncated statistical results without extra schema fields.search.py,nl_search.py,client.py, andREADME.md, noting thatChronicleClient.get_statsremains the dedicated method for direct statistical queries.Backward compatibility
search_udm,ChronicleClient.search_udm,nl_search,ChronicleClient.nl_search,ChronicleClient.get_stats, andsecops searchkeep identical parameters, defaults, and type annotations.:udmSearchomitsstats,search_udmreturns the exact 3-key dictionary{"events": [...], "total_events": int, "more_data_available": bool}with nostatskey added.as_list=Trueunchanged.search_udmwithas_list=Truecontinues returninglist[dict[str, Any]].statskey on YARA-L 2.0outcome:queries. Existing top-level keysevents,total_events, andmore_data_availableremain present with identical types, whileresponse["stats"]is purely additive and matchesChronicleClient.get_stats1:1.secops searchpasses the returned dictionary directly tooutput_formatter, which serializesjsonandtextoutput cleanly without schema assumptions.Maintenance mode alignment
This PR is a backward-compatible stability fix for HTTP 200 response data loss in existing SDK and CLI methods. It adds no new endpoints, methods, parameters, or dependencies.
Testing
tests/chronicle/test_search.pycoveringstatsnormalization,more_data_availablepropagation fromtooManyResults, zero-rowstatspayloads, non-dictstatsguards,as_list=Truebehavior onstatsresponses, and absence ofstatson standard event responses.tests/chronicle/test_nl_search.pyverifying end-to-endnl_searchpreservation ofstatswhen:translateUdmQueryemits a YARA-L 2.0outcome:query.pytest -m "not integration" --cov=secopsandpylint --rcfile=pylintrc src/secops.