Repository navigation
Add regression tests for known defects, tagged known-defect - #265
Conversation
Tests only; no production code changes. Adds 21 test classes covering 56 findings. Tests for defects that are still present are tagged @category(KnownDefect.class). The default `test` task (and so `build` and CI) excludes that tag, so the build stays green. `./gradlew knownDefectTests` runs only the tagged tests and is expected to fail until each defect is fixed; remove the tag when a fix lands so the test joins the normal suite. Tests that pass today (fixed defects, guards, already-correct cases) run in the default suite and protect the current behaviour. Repository-health checks skip instead of failing when the tests are not run from a full checkout. Build change: buildSrc/src/main/kotlin/b2sdk.gradle.kts excludes the category from `test` and registers `knownDefectTests` for modules that contain the marker interface com.backblaze.b2.testing.KnownDefect. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
testGuardC18_closeReturnsPromptly failed intermittently (about 2 of 12 isolated runs): close() right after build() can land before the cleaner thread reaches its first wait. The guard now waits (bounded) for the thread to be parked first, so it only measures close() latency. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ffumero2003
left a comment
There was a problem hiding this comment.
This review focuses on how reliably the staged functional tests hand off to the eventual fixes. Several cases currently allow an unrelated failure to satisfy the assertion, depend on scheduling or wall time, or use doubles and source-text patterns that do not preserve the behavior being exercised. Tightening those fixtures and assertions will make the intended failure paths identifiable now and give the fixes dependable regression checks once the marked tests are promoted.
The broad failure handling noted below appears in separate test catches; the KnownDefect category is an exclusion marker rather than a shared expected-failure matcher. Where tests share a fixture, constrain the common setup once and then require each test to observe its own intended response or failure path. The KnownDefect staging itself can remain in place for issues that are still open.
The ordinary build passed on 26b400e; the separate known-defect task ran 57 tests and all failed. With IPv6 preferred, I reproduced two failures in the ordinary HTTP tests and false passes in the response-path tests noted below.
Findings
HTTP fixtures use different loopback addresses — blocking
What the test does now: At B2WebApiHttpClientFindingsTest.java:42, the server binds to InetAddress.getLoopbackAddress() but line 51 builds a client URL using 127.0.0.1. SampleFindingsTest.java:47,55 uses the same pattern.
Why it is inadequate: With -Djava.net.preferIPv6Addresses=true, the server can bind to ::1 while the client connects to IPv4. On the current PR head, the default C46 and C47 HTTP tests failed with connection errors under that setting even though the HTTP behavior under test was unchanged. The sample range test can instead pass on that connection error.
Fix: Bind both fixtures explicitly with InetAddress.getByName("127.0.0.1") to match their existing URLs, or construct each URL from its server's actual bound address (including IPv6 brackets when needed). Run the HTTP and sample tests with IPv6 preference both enabled and disabled.
Tag: blocking
Connection failures can satisfy response assertions — blocking
What the test does now: At SampleFindingsTest.java:57–64, the C28 range test accepts any IOException as evidence that UrlContentSource rejected the server's 200 response. Similarly, B2WebApiHttpClientFindingsTest.java:150–160 accepts any B2Exception with a null or nonnegative retry delay.
Why it is inadequate: A refused connection throws an IOException before the range server responds, making C28 pass even though the current implementation accepts a 200 response. Likewise, a connection failure with no retry delay makes the negative Retry-After test pass without processing the configured 503 response. Both false passes were reproduced with IPv6 preferred on this PR head.
Fix: Give each handler a CountDownLatch that it counts down after receiving the request, and assert the latch completes within a bounded timeout. In C28, capture the Range header and assert it equals bytes=200-299; require an IOException specifically identifying rejection of the unexpected 200 response (for example, a message containing 200). In the Retry-After test, assert the caught B2Exception has status 503 before asserting its retry delay is null or nonnegative. Fail on connection errors rather than treating them as the expected response path.
Tag: blocking
Cancellation test races worker startup — blocking
What the test does now: At B2LargeFileStorerFindingsTest.java:132–150, C6 pre-interrupts the calling thread, invokes storeParts, and polls until the first fake worker's recorded token is cancelled.
Why it is inadequate: A correct implementation may notice the existing interrupt and cancel queued futures before either worker enters storePart; then the fake records no token and the test fails even though no upload remains running. It can also cancel an in-flight future by interrupting the worker rather than setting this particular token.
Fix: Have the fake workers count down a CountDownLatch on entry to storePart, run storeParts on a separate caller thread, and wait with a timeout until the uploads under observation have started before interrupting that caller. Record worker termination or interruption with another latch, wait for it with a timeout, and assert that no started upload remains active; release all latches and shut down executors in finally. Assert token cancellation only if the public contract specifically requires that mechanism.
Tag: blocking
Close guard can hang and observe the wrong worker — blocking
What the test does now: At HttpClientFactoryFindingsTest.java:43–65, C18 polls the state of any thread whose class name contains IdleConnectionMonitorThread, then calls factory.close() synchronously and checks the elapsed time afterward. The polling helper silently returns when its two-second deadline expires.
Why it is inadequate: If close() blocks indefinitely, the test itself never reaches its assertion. If another factory's monitor is waiting, or this factory's monitor has not reached its wait when the helper times out, the test can exercise the startup race it says it excludes and fail for scheduling reasons.
Fix: Identify the monitor created for the test factory, wait with a bounded deadline until that specific worker is waiting, and fail explicitly if readiness is not observed. Run factory.close() on a daemon-backed executor and require Future.get(4, TimeUnit.SECONDS) to complete; cancel and clean up in finally so a stuck call is reported as a timeout rather than blocking the test indefinitely.
Tag: blocking
Parts-listing stub ignores the continuation cursor — blocking
What the test does now: At B2ListIterablesFindingsTest.java:99–109, C14 stubs listParts(anyObject()) to return three successive pages regardless of the requests, and asserts that iteration produces parts 1 and 3.
Why it is inadequate: An implementation that keeps requesting the first page without setting startPartNumber still receives the mock's second and third pages and passes. A real service would receive the wrong continuation cursor, so the mock hides a pagination regression.
Fix: Stub or capture requests according to their actual B2ListPartsRequest: the initial request has a null start part, the next must have start part 2, and the next must have start part 3, all for the original file ID. Return the corresponding page only for each matching request and retain the assertion that the resulting part numbers are [1, 3].
Tag: blocking
Source-text checks do not establish sample behavior — blocking
What the test does now: At SampleFindingsTest.java:146–148, C41 checks that B2.java does not contain the exact source text pair.split("="); the other methods marked Static use similar source patterns for sample behavior.
Why it is inadequate: Changing the implementation to pair.split("=", 0) leaves values containing = split into more than two pieces, but makes this assertion pass. Equivalent formatting or an unrelated matching source fragment can likewise change a source-pattern result without changing the behavior.
Fix: Replace the C41 source search with a behavioral test of the sample's --info processing using key=a=b, and assert that the resulting file-info map contains key key with the complete value a=b. Where direct invocation is blocked by the command's structure, make the argument-parsing portion callable through a small testable entry point and drive the command through that path. Apply the same approach to the other static behavior checks, asserting their documented outputs or calls rather than exact source snippets.
Tag: blocking
Interrupted-retry test relies on elapsed wall time — blocking
What the test does now: At B2RetryerFindingsTest.java:32–47, C10 pre-interrupts the current thread and accepts either at most two attempts or at least one second of elapsed wall-clock time.
Why it is inadequate: A broken retryer that burns through every attempt without pausing passes if the test process is descheduled for one second during the run. Conversely, timing variation can make the result depend on the machine rather than the retry behavior.
Fix: Use a controlled retry policy and a test sleeper that exposes each sleep attempt and its interruption outcome. For an interrupted sleep that returns immediately, assert that the retry loop stops within the allowed two attempts; if the intended behavior is to keep waiting, use a latch-controlled sleeper and assert no subsequent attempt occurs until the latch is released. Bound any waits and clear the interrupt flag during cleanup.
Tag: blocking
Non-retry tests do not assert failure propagation — non-blocking
What the test does now: At B2StorageClientImplFindingsTest.java:61–66, the create-key test catches a B2Exception if one occurs, but also accepts normal return and only verifies one webifier invocation. The start-large-file, create-bucket, delete-key and copy-file tests use the same pattern.
Why it is inadequate: A client implementation that calls the webifier once, catches the lost-reply exception, and returns normally passes the one-call assertion even though it reports the failed operation as successful.
Fix: In each of the five tests, explicitly fail if the public client call returns normally, catch the expected B2ConnectionBrokenException or its intended public B2Exception wrapper, and assert that the lost-reply cause or identifying message survives. Retain the existing times(1) verification for the corresponding webifier call.
Tag: non-blocking
Known-defect tests need promotion into the default suite — non-blocking
What the test does now: At b2sdk.gradle.kts:47–60, the default test task excludes KnownDefect tests and the separate knownDefectTests task includes only those tests. The separate task is expected to remain red while any marked defect remains.
Why it is inadequate: When one defect is fixed, its test can begin passing inside the still-red separate task while remaining excluded from the default build. A later regression in that behavior would therefore not be caught by the normal suite unless the marker is removed.
Fix: Add per-test tracking for expected failures and flag an unexpectedly passing marked test, or establish an explicit fix-PR step that removes @Category(KnownDefect.class) from the fixed test and verifies it runs under the ordinary test task. Preserve the intentionally staged status of defects that are still open.
Tag: non-blocking
|
|
||
| @Before | ||
| public void startServer() throws Exception { | ||
| server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 10); |
There was a problem hiding this comment.
HTTP fixtures use different loopback addresses
What the test does now: At B2WebApiHttpClientFindingsTest.java:42, the server binds to InetAddress.getLoopbackAddress() but line 51 builds a client URL using 127.0.0.1. SampleFindingsTest.java:47,55 uses the same pattern.
Why it is inadequate: With -Djava.net.preferIPv6Addresses=true, the server can bind to ::1 while the client connects to IPv4. On the current PR head, the default C46 and C47 HTTP tests failed with connection errors under that setting even though the HTTP behavior under test was unchanged. The sample range test can instead pass on that connection error.
Fix: Bind both fixtures explicitly with InetAddress.getByName("127.0.0.1") to match their existing URLs, or construct each URL from its server's actual bound address (including IPv6 brackets when needed). Run the HTTP and sample tests with IPv6 preference both enabled and disabled.
Tag: blocking
| try { | ||
| final String url = "http://127.0.0.1:" + server.getAddress().getPort() + "/f"; | ||
| final UrlContentSource part = new UrlContentSource.Builder(url, 100).setStart(200).build(); | ||
| try (InputStream in = part.createInputStream()) { |
There was a problem hiding this comment.
Connection failures can satisfy response assertions
What the test does now: At SampleFindingsTest.java:57–64, the C28 range test accepts any IOException as evidence that UrlContentSource rejected the server's 200 response. Similarly, B2WebApiHttpClientFindingsTest.java:150–160 accepts any B2Exception with a null or nonnegative retry delay.
Why it is inadequate: A refused connection throws an IOException before the range server responds, making C28 pass even though the current implementation accepts a 200 response. Likewise, a connection failure with no retry delay makes the negative Retry-After test pass without processing the configured 503 response. Both false passes were reproduced with IPv6 preferred on this PR head.
Fix: Give each handler a CountDownLatch that it counts down after receiving the request, and assert the latch completes within a bounded timeout. In C28, capture the Range header and assert it equals bytes=200-299; require an IOException specifically identifying rejection of the unexpected 200 response (for example, a message containing 200). In the Retry-After test, assert the caught B2Exception has status 503 before asserting its retry delay is null or nonnegative. Fail on connection errors rather than treating them as the expected response path.
Tag: blocking
| /** Finding C6: after the caller is interrupted, the part uploads in progress must be told to stop. */ | ||
| @Category(KnownDefect.class) | ||
| @Test | ||
| public void testFindingC6_partUploadsAreCancelledWhenCallerInterrupted() throws Exception { |
There was a problem hiding this comment.
Cancellation test races worker startup
What the test does now: At B2LargeFileStorerFindingsTest.java:132–150, C6 pre-interrupts the calling thread, invokes storeParts, and polls until the first fake worker's recorded token is cancelled.
Why it is inadequate: A correct implementation may notice the existing interrupt and cancel queued futures before either worker enters storePart; then the fake records no token and the test fails even though no upload remains running. It can also cancel an in-flight future by interrupting the worker rather than setting this particular token.
Fix: Have the fake workers count down a CountDownLatch on entry to storePart, run storeParts on a separate caller thread, and wait with a timeout until the uploads under observation have started before interrupting that caller. Record worker termination or interruption with another latch, wait for it with a timeout, and assert that no started upload remains active; release all latches and shut down executors in finally. Assert token cancellation only if the public contract specifically requires that mechanism.
Tag: blocking
| * this guard fail intermittently if it were not excluded here. | ||
| */ | ||
| @Test | ||
| public void testGuardC18_closeReturnsPromptly() { |
There was a problem hiding this comment.
Close guard can hang and observe the wrong worker
What the test does now: At HttpClientFactoryFindingsTest.java:43–65, C18 polls the state of any thread whose class name contains IdleConnectionMonitorThread, then calls factory.close() synchronously and checks the elapsed time afterward. The polling helper silently returns when its two-second deadline expires.
Why it is inadequate: If close() blocks indefinitely, the test itself never reaches its assertion. If another factory's monitor is waiting, or this factory's monitor has not reached its wait when the helper times out, the test can exercise the startup race it says it excludes and fail for scheduling reasons.
Fix: Identify the monitor created for the test factory, wait with a bounded deadline until that specific worker is waiting, and fail explicitly if readiness is not observed. Run factory.close() on a daemon-backed executor and require Future.get(4, TimeUnit.SECONDS) to complete; cancel and clean up in finally so a stuck call is reported as a timeout rather than blocking the test indefinitely.
Tag: blocking
| @Test | ||
| public void testFindingC14_partsIteratorContinuesPastEmptyMiddlePage() throws B2Exception { | ||
| final B2ListPartsRequest request = B2ListPartsRequest.builder(fileId(1)).build(); | ||
| when(client.listParts(anyObject())).thenReturn( |
There was a problem hiding this comment.
Parts-listing stub ignores the continuation cursor
What the test does now: At B2ListIterablesFindingsTest.java:99–109, C14 stubs listParts(anyObject()) to return three successive pages regardless of the requests, and asserts that iteration produces parts 1 and 3.
Why it is inadequate: An implementation that keeps requesting the first page without setting startPartNumber still receives the mock's second and third pages and passes. A real service would receive the wrong continuation cursor, so the mock hides a pagination regression.
Fix: Stub or capture requests according to their actual B2ListPartsRequest: the initial request has a null start part, the next must have start part 2, and the next must have start part 3, all for the original file ID. Return the corresponding page only for each matching request and retain the assertion that the resulting part numbers are [1, 3].
Tag: blocking
| /** Finding C41 (static): --info must accept values containing '=' (split must be limited to 2 parts). */ | ||
| @Category(KnownDefect.class) | ||
| @Test | ||
| public void testFindingC41_infoOptionAllowsEqualsInValueStatic() throws IOException { |
There was a problem hiding this comment.
Source-text checks do not establish sample behavior
What the test does now: At SampleFindingsTest.java:146–148, C41 checks that B2.java does not contain the exact source text pair.split("="); the other methods marked Static use similar source patterns for sample behavior.
Why it is inadequate: Changing the implementation to pair.split("=", 0) leaves values containing = split into more than two pieces, but makes this assertion pass. Equivalent formatting or an unrelated matching source fragment can likewise change a source-pattern result without changing the behavior.
Fix: Replace the C41 source search with a behavioral test of the sample's --info processing using key=a=b, and assert that the resulting file-info map contains key key with the complete value a=b. Where direct invocation is blocked by the command's structure, make the argument-parsing portion callable through a small testable entry point and drive the command through that path. Apply the same approach to the other static behavior checks, asserting their documented outputs or calls rather than exact source snippets.
Tag: blocking
| } catch (B2Exception expected) { | ||
| // expected | ||
| } | ||
| final long elapsedMillis = (System.nanoTime() - start) / 1_000_000; |
There was a problem hiding this comment.
Interrupted-retry test relies on elapsed wall time
What the test does now: At B2RetryerFindingsTest.java:32–47, C10 pre-interrupts the current thread and accepts either at most two attempts or at least one second of elapsed wall-clock time.
Why it is inadequate: A broken retryer that burns through every attempt without pausing passes if the test process is descheduled for one second during the run. Conversely, timing variation can make the result depend on the machine rather than the retry behavior.
Fix: Use a controlled retry policy and a test sleeper that exposes each sleep attempt and its interruption outcome. For an interrupted sleep that returns immediately, assert that the retry loop stops within the allowed two attempts; if the intended behavior is to keep waiting, use a latch-controlled sleeper and assert no subsequent attempt occurs until the latch is released. Bound any waits and clear the interrupt flag during cleanup.
Tag: blocking
| public void testFindingC3_createKeyNotRetried() throws B2Exception { | ||
| authorize(); | ||
| when(webifier.createKey(anyObject(), anyObject())).thenThrow(lostReply()); | ||
| try { |
There was a problem hiding this comment.
Non-retry tests do not assert failure propagation
What the test does now: At B2StorageClientImplFindingsTest.java:61–66, the create-key test catches a B2Exception if one occurs, but also accepts normal return and only verifies one webifier invocation. The start-large-file, create-bucket, delete-key and copy-file tests use the same pattern.
Why it is inadequate: A client implementation that calls the webifier once, catches the lost-reply exception, and returns normally passes the one-call assertion even though it reports the failed operation as successful.
Fix: In each of the five tests, explicitly fail if the public client call returns normally, catch the expected B2ConnectionBrokenException or its intended public B2Exception wrapper, and assert that the lost-reply cause or identifying message survives. Retain the existing times(1) verification for the corresponding webifier call.
Tag: non-blocking
| // expected to fail until the defects are fixed. Only modules that have the marker interface take part. | ||
| val knownDefectMarker = "com.backblaze.b2.testing.KnownDefect" | ||
| if (file("src/test/java/com/backblaze/b2/testing/KnownDefect.java").exists()) { | ||
| tasks.test { |
There was a problem hiding this comment.
Known-defect tests need promotion into the default suite
What the test does now: At b2sdk.gradle.kts:47–60, the default test task excludes KnownDefect tests and the separate knownDefectTests task includes only those tests. The separate task is expected to remain red while any marked defect remains.
Why it is inadequate: When one defect is fixed, its test can begin passing inside the still-red separate task while remaining excluded from the default build. A later regression in that behavior would therefore not be caught by the normal suite unless the marker is removed.
Fix: Add per-test tracking for expected failures and flag an unexpectedly passing marked test, or establish an explicit fix-PR step that removes @Category(KnownDefect.class) from the fixed test and verifies it runs under the ordinary test task. Preserve the intentionally staged status of defects that are still open.
Tag: non-blocking
…r order dependence - bind HTTP fixtures to 127.0.0.1 to match client URLs (works with IPv6 preferred) - C28, C11: handlers signal a latch; assert the request was seen, the Range header, a 200-specific IOException, and status 503 - C6 cancellation: caller on its own thread, wait for uploads to start, then for each to stop (bounded) - C18/C17: use this factory's own cleaner thread; close() runs on a daemon thread with a bounded wait - C14 parts: stub pages per request cursor and verify the requests - C10: controlled sleeper instead of wall-clock time - C3: tests fail if the call returns normally and check the lost-reply failure is propagated - samples C38-C41: drive the B2 command with a mocked client instead of reading source text (C43/C45 stay source checks) - document the fix-PR step of removing @category(KnownDefect) in b2sdk.gradle.kts Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Add an independent equals-only file ID and assert the exact percent-encoded query. Preserve the existing ampersand cases and known-defect staging.
ffumero2003
left a comment
There was a problem hiding this comment.
Re-reviewed at 78e3ac7. The known-defect regression tests look good. The file-ID gap is properly closed — the added independent '='-only assertion fails against a broken encoder and passes against the correct one, so it guards behavior rather than restating the implementation. known-defect stays a distinct category and each staged test asserts its documented, test-specific failure. C43 and C45 remain source-text checks; converting them to runtime checks needs a production change, so leaving them as-is here is correct — I'll track that conversion as a separate issue rather than block this PR. Approving; the only red check is the non-required code/snyk (Backblaze Projects) status, a known false positive.
|
Holding this on merge pending internal coordination on this repo's ownership. The approval stands; we're just not landing it yet. Will revisit once that's settled. |
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Summary
Tests only. No production code changes.
Adds regression tests for 56 known findings, written against the current
master. Each test name carries its finding id (for exampletestFindingC21_...).@Category(KnownDefect.class). They are excluded from the defaulttesttask, so./gradlew buildand CI stay green../gradlew knownDefectTestsruns only those tests. It is expected to fail until the defects are fixed. When a fix lands, remove the tag so the test joins the normal suite.@Ignore; the exclusion is the visible category plus the task above.How to run
Default suite (known-defect tests excluded; 730 tests, all pass):
Known-defect tests only (58 tests; the task fails by design, every test fails today):
A class whose tests are all tagged reports "No tests found" if run with the plain
testtask; useknownDefectTestsfor it.What each status means
KnownDefect, run only inknownDefectTests, each fails for the reason the finding describes.Total 56 findings. The repository-health checks (changelog, README, security policy, one workflow command, one sample option) read repository files and are skipped, not failed, when run outside a full checkout.
Notes: a few tests read the sample source text because the CLI cannot run without a B2 account; these are source-pattern checks, not behaviour tests. One default-suite guard (C44) reflects JDK 11/17 behaviour and creates a sparse 2 GiB file that is deleted afterwards. The C18 guard waits (bounded) for the cleaner thread to park before timing
close(), so it is not timing-sensitive at startup, but it can still pass while the race it mentions exists.Existing tests are unchanged. Build change:
buildSrc/src/main/kotlin/b2sdk.gradle.kts(category exclusion and theknownDefectTeststask), plus a small marker interfacecom.backblaze.b2.testing.KnownDefectin the test source of core, httpclient and samples.The samples module had no tests before. No test needs credentials or network access beyond loopback.
🤖 Generated with Claude Code
Note:
knownDefectTestsis not part ofbuildor CI, so the known defects are not checked automatically until someone runs that task.