[AUTOCUT] Integration Test Failed for sql-3.7.0 #5391 #3

Activity

1. RyanL1997

   added

   bugSomething isn't workingSomething isn't working

   untriaged

   on May 4, 2026
2. RyanL1997 commented on May 4, 2026

   

   RyanL1997

   on May 4, 2026

   OwnerAuthor

   More actions

   @ryan-gh-bot /rca
3. ryan-gh-bot commented on May 4, 2026

   

   ryan-gh-bot

   on May 4, 2026

   More actions

   Working on /rca (invoked by @RyanL1997) — posting result when done.
4. ryan-gh-bot commented on May 4, 2026

   

   ryan-gh-bot

   on May 4, 2026

   More actions

   [RCA] VectorSearchIT#testExecutionWithoutKnnPluginReturnsCapabilityError fails in release-distribution integ-test because the bundled cluster ships the k-NN plugin, invalidating the test's "no k-NN" precondition

   The test asserts that vectorSearch() returns a "k-NN plugin not installed" message from KnnPluginCapability, which only happens on a cluster where k-NN is absent. The OpenSearch 3.7.0 release distribution tar bundles opensearch-knn, so the capability probe succeeds, execution proceeds to the cluster, and the cluster rejects the query with Field 'embedding' is not knn_vector type. — a different 400 error, causing the assertion to fail. This is a test-environment mismatch introduced with the vector-search feature in opensearch-project#5394, not a product regression.

   Environment

   • Tenant: sql (opensearch-project/sql)
   • Refs checked:
     • upstream/main @ d3bdca82a (current HEAD; adds the failing test and the KnnPluginCapability class in the same commit)
   • Failing test: org.opensearch.sql.sql.VectorSearchIT#testExecutionWithoutKnnPluginReturnsCapabilityError (both with-security and without-security, both linux/x64 build 11131 and linux/arm64 build 11130 of the 3.7.0 integ-test pipeline).

   Suspected location

   • integ-test/src/test/java/org/opensearch/sql/sql/VectorSearchIT.java L348-L373 — the test itself, whose comment declares the precondition "The default integ-test cluster does not have the k-NN plugin installed". That precondition holds for the gradle-managed cluster (see below) but not for the release-distribution cluster.
   • integ-test/build.gradle L273-L281 — the testClusters.integTest definition installs only opensearch-job-scheduler, opensearch-geospatial, and opensearch-sql-plugin. No k-NN. This is the cluster used by ./gradlew :integ-test:integTest and is where the precondition is true.
   • opensearch/src/main/java/org/opensearch/sql/opensearch/storage/capability/KnnPluginCapability.java L47-L67 — requireInstalled(); throws the expected message only when the Nodes Info probe confirms the plugin is absent. When the probe reports "installed", this returns normally and execution continues.

   Analysis

   The vector-search feature ships a capability probe (KnnPluginCapability) that fails fast with a fixed user-facing sentence when the k-NN plugin is missing. The failing test was designed around this path and assumes the test cluster lacks k-NN — which is accurate for the sql repo's own gradle integ-test harness (integ-test/build.gradle wires only job-scheduler, geospatial, and the sql plugin into the cluster).

   The opensearch-build release pipeline runs integ-tests against a different cluster: the full OpenSearch release tar, which bundles every core plugin including opensearch-knn. Under that cluster, the probe reports "installed", requireInstalled() returns without throwing, and the query reaches the cluster, which correctly rejects it because the accounts index the test seeds (Index.ACCOUNT) has no knn_vector field named embedding. OpenSearch returns SearchPhaseExecutionException with Field 'embedding' is not knn_vector type., wrapped by the SQL plugin as HTTP 400. The test's containsString check for the capability-probe sentence then fails.

   The adjacent test testExplainWithoutKnnPluginStillWorks passes in the same run (see the stdout timeline around testExplainWithoutKnnPluginStillWorks), which is consistent: _explain never opens the scan, so the capability probe is never invoked and no cluster-side k-NN is touched. That rules out a broken or misregistered probe.

   The test is simply not portable across the two integ-test harnesses. Any cluster that has k-NN installed (which is the expected topology for real deployments, and is what the release integ-test simulates) will hit this failure.

   Evidence

   Failing assertion from the release integ-test stdout (sql/without-security/stdout.txt, lines 157-173 of the stdout capture):

   VectorSearchIT > testExecutionWithoutKnnPluginReturnsCapabilityError FAILED
       java.lang.AssertionError:
       Expected: a string containing "vectorSearch() requires the k-NN plugin, which is not installed on this cluster."
            but: was "method [POST], host [http://localhost:9200], URI [/_plugins/_sql?format=jdbc], status line [HTTP/1.1 400 Bad Request]
       {
         "error": {
           "reason": "Error occurred in OpenSearch engine: all shards failed",
           "details": "Shard[0]: [opensearch-sql_test_index_account/...] QueryShardException[failed to create query: Field 'embedding' is not knn_vector type.]; nested: IllegalArgumentException[Field 'embedding' is not knn_vector type.]; ...",
           "type": "SearchPhaseExecutionException"
         },
         "status": 400
       }"
           at org.opensearch.sql.sql.VectorSearchIT.testExecutionWithoutKnnPluginReturnsCapabilityError(VectorSearchIT.java:369)
   

   Release-distribution cluster log confirming k-NN is loaded (from sql/without-security/local-cluster-logs/id-1/stdout.txt):

   [2026-05-04T08:11:42,394][INFO ][o.o.p.PluginsService] [node_name_9200] loaded plugin [opensearch-alerting]
   [2026-05-04T08:11:42,394][INFO ][o.o.p.PluginsService] [node_name_9200] loaded plugin [opensearch-anomaly-detection]
   ...
   [2026-05-04T08:11:42,395][INFO ][o.o.p.PluginsService] [node_name_9200] loaded plugin [opensearch-knn]
   ...
   [truncated — 19 plugins total in the release distribution]
   

   Also from the cluster log, line 18: k-NN libraries not found in LD_LIBRARY_PATH. Updating path to: ... /plugins/opensearch-knn/lib. — further confirming the plugin is present on disk at cluster startup.

   Regression scope: both the test and KnnPluginCapability were introduced in the same commit, d3bdca82a [Feature] Support SQL Vector Search (#5394), which is the current upstream/main HEAD. This is the first release that includes the feature, so there is no earlier good state to bisect to — the test has never been exercised against a k-NN–enabled cluster before.

   $ git log --oneline -- integ-test/src/test/java/org/opensearch/sql/sql/VectorSearchIT.java
   d3bdca82a [Feature] Support SQL Vector Search (#5394)
   
   $ git log --oneline -- opensearch/src/main/java/org/opensearch/sql/opensearch/storage/capability/KnnPluginCapability.java
   d3bdca82a [Feature] Support SQL Vector Search (#5394)

   Local verification: not performed. The test's precondition is stated in-line and the two harness configurations are unambiguous from the build scripts and cluster startup logs, which is sufficient for High confidence.

   Confidence: High
   Three independent signals converge: (1) the test's own comment states the "no k-NN" precondition; (2) the release-distribution cluster log shows loaded plugin [opensearch-knn], invalidating that precondition; (3) the actual failure string is exactly the downstream error you get when KnnPluginCapability.requireInstalled() returns normally and execution reaches a real OpenSearch cluster that lacks a knn_vector mapping on the seeded index. The sibling testExplainWithoutKnnPluginStillWorks passes in the same run, ruling out broader breakage in the vector-search pipeline.

   Next steps

   • Preferred: make the test cluster-topology aware so it does the right thing on both harnesses. Unverified suggestion: probe the _nodes/plugins endpoint at test start; if opensearch-knn is present, use assumeFalse / assumeTrue to skip this particular test (the capability-error path is still covered by KnnPluginCapabilityTest unit tests in opensearch/src/test/.../capability/), or instead assert the cluster-side knn_vector-type error as the expected outcome for that environment.
   • Alternative: restrict this test to the gradle integ-test harness only, e.g. mark it with a test-category annotation that the release-distribution integ-test excludes, and add a distinct release-distribution test that asserts the k-NN–installed behavior (seed a knn_vector index, expect a real top-k result).
   • Unverified suggestion: in the opensearch-build integ-test manifest, the sql component could be run against a cluster flavor that omits k-NN — but this cuts against the project's convention of testing against the full bundled distribution and is not recommended.
   • No product code change is required: KnnPluginCapability behaves correctly in both environments. The fix is in the test.

   ---

   Automated response from @ryan-gh-bot (maintainer-triggered agent). About this agent.