feat: support plain HTTP forward proxy for private IP endpoints #155

Activity

1. johntmyers

   added

   feat

   area:policyPolicy engine and policy lifecycle workPolicy engine and policy lifecycle work

   area:supervisorProxy and routing-path workProxy and routing-path work

   state:review-readyReady for human reviewReady for human review

   area:sandboxSandbox runtime and isolation workSandbox runtime and isolation work

   spike

   on Mar 6, 2026
2. johntmyers commented on Mar 6, 2026

   

   johntmyers

   on Mar 6, 2026

   CollaboratorAuthor

   More actions

   🔬 spike-agent

   Technical Investigation

   Architecture Overview

   The sandbox proxy (crates/navigator-sandbox/src/proxy.rs, 1715 lines) is a CONNECT-tunnel-only HTTP proxy bound to 10.200.0.1:3128 inside the sandbox. Every TCP connection is dispatched to handle_tcp_connection (line 160), which reads the HTTP request line, extracts the method and target, and branches:

   1. Non-CONNECT → 403 Forbidden (line 199-208)
   2. CONNECT → OPA policy eval → SSRF check → establish tunnel → relay

   The OPA engine (opa.rs) returns a tri-state NetworkAction: Allow, InspectForInference, or Deny. For Allow, the proxy resolves DNS, validates against SSRF checks (is_internal_ip or allowed_ips), connects upstream, and relays traffic — either raw bidirectional copy (L4) or protocol-aware inspection (L7).

   The L7 relay subsystem (l7/relay.rs, l7/rest.rs) already handles full HTTP request/response parsing, forwarding, and per-request policy evaluation. It supports Content-Length, chunked transfer encoding, and EOF-terminated bodies.

   Code References

   Location	Description
   proxy.rs:160-524	handle_tcp_connection — main per-connection handler
   proxy.rs:199-208	Non-CONNECT rejection block — primary insertion point
   proxy.rs:210-224	CONNECT target parsing + OPA evaluation
   proxy.rs:284-287	Deny handling
   proxy.rs:292-331	InspectForInference handling
   proxy.rs:336-386	allowed_ips check + SSRF validation + upstream connect
   proxy.rs:388-521	L7 config check + relay dispatch (TLS terminate, plaintext, L4-only)
   proxy.rs:979-997	is_internal_ip() — blocks all RFC 1918 + loopback + link-local
   proxy.rs:1035-1053	is_always_blocked_ip() — blocks only loopback + link-local (used with allowed_ips)
   proxy.rs:1062-1098	resolve_and_check_allowed_ips() — DNS resolve + CIDR allowlist validation
   proxy.rs:1134-1166	query_allowed_ips() — queries OPA for endpoint's allowed_ips config
   proxy.rs:1173-1193	extract_host_from_uri() — parses hostname from absolute-form URI (currently logging-only)
   proxy.rs:1195-1203	parse_target() — parses host:port from CONNECT target
   l7/relay.rs:41-67	relay_with_inspection() — L7 relay dispatcher
   l7/relay.rs:70-154	relay_rest() — REST relay loop: parse request → evaluate policy → relay/deny
   l7/rest.rs:119-161	relay_http_request() — forwards request + relays response
   l7/rest.rs:218-239	relay_fixed() — relay fixed-length body
   l7/rest.rs:250-352	relay_chunked() — relay chunked transfer encoding
   l7/rest.rs:365-488	relay_response() — parse + relay HTTP response with framing detection
   l7/rest.rs:551-564	looks_like_http() — peek-based HTTP protocol detection

   Current Behavior

   When a standard HTTP client (e.g., httpx with proxy=) makes a request to http://10.86.8.223:8000/screenshot/, it sends:

   GET http://10.86.8.223:8000/screenshot/ HTTP/1.1
   Host: 10.86.8.223:8000
   

   The proxy reads this at line 196, sees method = "GET" (not "CONNECT"), and immediately returns 403 at line 206. The OPA engine is never consulted. The existing extract_host_from_uri function (line 1173) already parses the hostname from these absolute-form URIs but is only used for logging the rejected request.

   What Would Need to Change

   1. proxy.rs — Replace the deny block (lines 199-208)

   Instead of blanket 403, dispatch to a new handle_forward_proxy function. This function:

   • Calls a new parse_proxy_uri to extract scheme, host, port, and path from the absolute-form URI
   • Rejects https:// scheme (must use CONNECT)
   • Runs evaluate_opa_tcp — same OPA call as CONNECT, identical identity binding
   • Rejects Deny and InspectForInference — forward proxy only works for explicitly Allowed endpoints
   • Queries allowed_ips — if empty, reject (no implicit private IP access)
   • Calls resolve_and_check_allowed_ips — same SSRF validation as CONNECT
   • Adds a private-IP gate: all_addrs_are_private() — ensures resolved IPs are RFC 1918 (the forward proxy security boundary)
   • Connects upstream via TcpStream::connect
   • Rewrites the request-line from absolute-form to origin-form: GET http://host:port/path HTTP/1.1 → GET /path HTTP/1.1
   • Forwards the request and relays the response

   2. proxy.rs — New helper functions

   • parse_proxy_uri(uri: &str) -> Result<(&str, String, u16, String)> — parse absolute-form URI into (scheme, host, port, path)
   • rewrite_forward_request(raw_request: &[u8], path: &str) -> Vec<u8> — rewrite request-line, strip hop-by-hop headers, add Via
   • all_addrs_are_private(addrs: &[SocketAddr]) -> bool — ensure all resolved IPs are RFC 1918

   3. proxy.rs — Logging

   Add a distinct log prefix for forward proxy requests (e.g., "FORWARD" instead of "CONNECT") so they're distinguishable in sandbox logs.

   4. Optional: L7 integration

   After connecting upstream, check query_l7_config for the matched endpoint. If L7 is configured, the existing relay_with_inspection can handle per-request policy evaluation on the forward-proxied traffic. This is possible because the traffic is already plaintext HTTP — no TLS termination needed. The L7 REST provider's relay_rest loop already handles multi-request connections.

   Alternative Approaches Considered

   A. Synthetic CONNECT: Convert the forward proxy request into an internal CONNECT flow — parse the URI, create a synthetic tunnel, inject the original request bytes. This reuses the entire CONNECT code path but is hacky and harder to reason about. The separate handle_forward_proxy function is cleaner.

   B. Client-side fix only: Keep the proxy CONNECT-only and have client libraries (like sandbox_http.py) always use CONNECT for HTTP. This is what exists today but pushes complexity to every user and every language. Fragile and poor DX.

   C. Accept all forward proxy requests: Handle forward proxy for any allowed endpoint, not just private IPs. This would allow plain HTTP to the public internet, which is a security concern. The private-IP restriction is the right boundary.

   Patterns to Follow

   • The CONNECT handling flow at lines 210-521 is the template. The forward proxy function should mirror its structure: parse → OPA eval → log → SSRF check → connect → relay.
   • Logging should follow the existing structured format: info!(src_addr, dst_host, dst_port, binary, action, policy, ...).
   • Error handling should use the existing respond(&mut client, b"HTTP/1.1 4xx ...\r\n\r\n") pattern.
   • The L7 relay integration should follow the same pattern as lines 390-516 (check query_l7_config, build L7EvalContext, dispatch).

   Test Coverage Notes

   Proxy tests are primarily in crates/navigator-sandbox/testdata/ (policy fixtures) and integration/e2e tests in e2e/. The OPA engine has unit tests in opa.rs using test fixtures from testdata/sandbox-policy.yaml. For this feature:

   • Unit tests for parse_proxy_uri and rewrite_forward_request can live in proxy.rs as #[cfg(test)] modules
   • Integration tests should cover the matrix: allowed private (200), public IP (403), no allowed_ips (403), no policy match (403), HTTPS scheme (403)
   • A regression test confirming CONNECT still works unchanged

   ---

   This investigation provides context for implementation. Next step: review the issue, refine if needed, then use build-from-issue to create a plan and implement.
3. johntmyers commented on Mar 6, 2026

   

   johntmyers

   on Mar 6, 2026

   Last edited by johntmyers

   CollaboratorAuthor

   More actions

   🏗️ build-plan

   Implementation Plan

   Issue type: feat
   Complexity: Medium
   Confidence: High — reuses existing OPA, SSRF, and relay infrastructure

   Summary

   Add HTTP forward proxy mode to the sandbox proxy, restricted to private IP endpoints with explicit allowed_ips policy. When the proxy receives a non-CONNECT request (e.g. GET http://10.86.8.223:8000/path HTTP/1.1), it parses the absolute-form URI, evaluates OPA policy, validates the destination is a private IP with allowed_ips, connects upstream, rewrites the request to origin-form, and relays the response using copy_bidirectional for full streaming support. This eliminates the need for users to write custom CONNECT tunnel code for plain HTTP calls to internal services.

   Decisions on Risks & Open Questions

   Connection reuse / keep-alive: v1 will inject Connection: close into forwarded requests. Each forward proxy request opens a fresh upstream connection and closes after the response. This is simpler and sufficient for the primary use cases (tool server calls, LLM inference). Keep-alive can be added later if profiling shows it matters.

   Streaming (LLM SSE / chunked responses): After forwarding the rewritten request, the proxy uses tokio::io::copy_bidirectional between client and upstream. This has no idle timeout and copies bytes as they arrive — suitable for long-lived chunked/SSE streams from LLMs that can run for minutes. This avoids the 5-second RELAY_EOF_IDLE_TIMEOUT in the L7 relay_until_eof path, which only applies to the L7-inspected codepath (not used here).

   Request body forwarding: The initial request buffer may contain overflow body bytes beyond the header \r\n\r\n boundary. The rewrite function includes these in the forwarded payload. Any remaining body bytes (beyond what was buffered) flow through copy_bidirectional from client→upstream. This handles Content-Length, Transfer-Encoding: chunked, and no-body requests.

   Hop-by-hop headers: Strip Proxy-Connection, Proxy-Authorization, and Proxy-Authenticate. Replace Connection header with Connection: close. Preserve Transfer-Encoding and all other end-to-end headers.

   Via header: Add Via: 1.1 navigator-sandbox per RFC 7230 §5.7.

   L7 inspection: Not included in v1. The forward proxy is already gated by OPA endpoint+binary match, allowed_ips, and the private-IP requirement. Per-request L7 rules (e.g. access: read-only) can be added as a follow-up by integrating relay_with_inspection after the initial request rewrite.

   Scope

   • crates/navigator-sandbox/src/proxy.rs: Primary change — new handle_forward_proxy function, parse_proxy_uri and rewrite_forward_request helpers, update dispatch in handle_tcp_connection
   • e2e/python/test_sandbox_policy.py: E2E tests for forward proxy allow/deny behavior
   • architecture/security-policy.md: Document that allowed_ips enables forward proxy mode for plain HTTP to private endpoints
   • architecture/plans/plain-http-forward-proxy.md: Already exists, update with final decisions

   Implementation Steps

   1. Add parse_proxy_uri helper (proxy.rs)

      • Parse absolute-form URI http://host:port/path?query into (scheme, host, port, path_with_query)
      • Handle: default port 80 for http, IPv6 bracket notation, missing path (default /), query strings
      • Unit test: various URI forms including edge cases

   2. Add rewrite_forward_request helper (proxy.rs)

      • Input: raw request bytes buf[..used], target path
      • Find header end (\r\n\r\n)
      • Replace first line: METHOD http://host:port/path HTTP/1.1 → METHOD /path HTTP/1.1
      • Strip hop-by-hop headers: Proxy-Connection, Proxy-Authorization, Proxy-Authenticate
      • Replace/add Connection: close
      • Add Via: 1.1 navigator-sandbox
      • Append overflow body bytes (anything after header end in original buffer)
      • Unit test: verify rewriting, header stripping, body preservation

   3. Add handle_forward_proxy function (proxy.rs)

      • Called from handle_tcp_connection when method != CONNECT
      • Flow:
        a. parse_proxy_uri(target) — reject if scheme is https (respond 400: "Use CONNECT for HTTPS")
        b. evaluate_opa_tcp(peer_addr, ...) — same identity binding as CONNECT
        c. Match on NetworkAction: only proceed on Allow; reject Deny and InspectForInference with 403
        d. Structured log with "FORWARD" tag (mirrors CONNECT log format)
        e. query_allowed_ips(...) — if empty, reject 403 (forward proxy requires explicit allowed_ips)
        f. parse_allowed_ips + resolve_and_check_allowed_ips — same SSRF validation as CONNECT
        g. Private-IP gate: verify all resolved addresses pass is_internal_ip() — reject 403 if any is public
        h. TcpStream::connect(addrs) to upstream — on failure, respond 502 Bad Gateway and return
        i. rewrite_forward_request(buf, path) — transform headers
        j. upstream.write_all(rewritten_bytes) — forward request
        k. tokio::io::copy_bidirectional(&mut client, &mut upstream) — relay response (supports streaming)

   4. Update dispatch in handle_tcp_connection (proxy.rs)

      • Replace lines 199-208 (blanket 403 for non-CONNECT) with a call to handle_forward_proxy
      • Pass: method, target URI, raw buffer, client stream, OPA engine, identity cache, entrypoint PID

   5. Add unit tests (proxy.rs #[cfg(test)])

      • parse_proxy_uri: standard URI, default port, IPv6, missing path, query string, HTTPS scheme, malformed
      • rewrite_forward_request: GET rewrite, POST with body overflow, hop-by-hop stripping, Via addition, Connection: close injection

   6. Add E2E tests (e2e/python/test_sandbox_policy.py)

      • See E2E Test Plan section below

   7. Update documentation

      • architecture/security-policy.md: Add section explaining forward proxy behavior for allowed_ips endpoints
      • architecture/plans/plain-http-forward-proxy.md: Mark decisions resolved

   E2E Test Plan

   New tests in e2e/python/test_sandbox_policy.py, following the existing patterns (_proxy_connect closure factory, sb.exec_python(), assert on stdout).

   New closure factory: _forward_proxy_with_server()

   A closure factory (module-level, like _proxy_connect) that:

   1. Starts a minimal http.server.HTTPServer in a background daemon thread on 0.0.0.0:19876 inside the sandbox
   2. Sends a raw forward proxy request (GET http://10.200.0.2:19876/test HTTP/1.1) to the sandbox proxy at 10.200.0.1:3128
   3. Returns the raw HTTP response string

   def _forward_proxy_with_server():
       def fn(proxy_host, proxy_port, target_host, target_port):
           import socket, threading
           from http.server import HTTPServer, BaseHTTPRequestHandler
   
           class H(BaseHTTPRequestHandler):
               def do_GET(self):
                   self.send_response(200)
                   body = b'forward-proxy-ok'
                   self.send_header('Content-Length', str(len(body)))
                   self.end_headers()
                   self.wfile.write(body)
               def log_message(self, *a): pass
   
           srv = HTTPServer(('0.0.0.0', int(target_port)), H)
           threading.Thread(target=srv.handle_request, daemon=True).start()
   
           import time; time.sleep(0.3)
   
           conn = socket.create_connection((proxy_host, int(proxy_port)), timeout=10)
           req = (f"GET http://{target_host}:{target_port}/test HTTP/1.1\r\n"
                  f"Host: {target_host}:{target_port}\r\n\r\n")
           conn.sendall(req.encode())
           data = b""
           while True:
               chunk = conn.recv(4096)
               if not chunk:
                   break
               data += chunk
           conn.close()
           srv.server_close()
           return data.decode("latin1")
       return fn

   New closure factory: _forward_proxy_raw()

   A simpler closure (no server) for testing deny cases — just sends the forward proxy request and returns the proxy's response:

   def _forward_proxy_raw():
       def fn(proxy_host, proxy_port, target_url):
           import socket
           from urllib.parse import urlparse
           conn = socket.create_connection((proxy_host, int(proxy_port)), timeout=10)
           parsed = urlparse(target_url)
           host_header = parsed.netloc or parsed.hostname
           req = (f"GET {target_url} HTTP/1.1\r\n"
                  f"Host: {host_header}\r\n\r\n")
           conn.sendall(req.encode())
           return conn.recv(4096).decode("latin1")
       return fn

   Test cases

   #	Test name	Policy	Request	Expected	Notes
   1	test_forward_proxy_allows_private_ip_with_allowed_ips	allowed_ips=["10.200.0.0/24"], host=10.200.0.2, port=19876, binary=/**	GET http://10.200.0.2:19876/test via proxy	"200" + "forward-proxy-ok" in response	Happy path with in-sandbox HTTP server via _forward_proxy_with_server()
   2	test_forward_proxy_denied_without_allowed_ips	host=10.200.0.2, port=19876, binary=/**, NO allowed_ips	GET http://10.200.0.2:19876/test via proxy	"403" in response	Forward proxy requires explicit allowed_ips. Uses _forward_proxy_raw().
   3	test_forward_proxy_rejects_https_scheme	allowed_ips=["10.200.0.0/24"], host=10.200.0.2, port=19876, binary=/**	GET https://10.200.0.2:19876/test via proxy	"400" in response	HTTPS must use CONNECT. Uses _forward_proxy_raw().
   4	test_forward_proxy_denied_no_policy_match	Policy for different host/port	GET http://10.200.0.2:19876/test via proxy	"403" in response	No matching network policy. Uses _forward_proxy_raw().
   5	test_forward_proxy_public_ip_denied	allowed_ips=["93.184.0.0/16"], host=example.com, port=80, binary=/**	GET http://example.com/ via proxy	"403" in response	Private-IP gate rejects public destinations. Uses _forward_proxy_raw().
   6	test_l4_non_connect_method_rejected (existing)	Existing policy (no allowed_ips for example.com)	GET http://example.com/ via proxy	"403" in response	Regression — unchanged, still passes.

   Log verification test

   #	Test name	What it checks
   7	test_forward_proxy_log_fields	Read /var/log/navigator.log via _read_navigator_log(). Assert: log contains FORWARD tag, action=allow for permitted request, dst_host=10.200.0.2, dst_port=19876, matched policy name. Also verify denied case logged with action=deny.

   Test Plan (Summary)

   • Unit tests: parse_proxy_uri (8+ cases) and rewrite_forward_request (5+ cases) in proxy.rs #[cfg(test)] module
   • E2E tests: 6 new tests + 1 existing regression in e2e/python/test_sandbox_policy.py covering: happy path with real HTTP relay, deny without allowed_ips, HTTPS scheme rejection, no policy match, public IP rejection, log field verification
   • Manual verification: Push to sandbox and test with standard Python httpx (no custom tunnel code) against an internal service

   Risks & Open Questions

   • All resolved — see Decisions section above
   • Low risk: the dispatch change (step 4) touches the hot path for ALL proxy connections. The check is a simple string comparison (method != "CONNECT") that already exists — we're just changing the else branch from 403 to handle_forward_proxy. No performance impact.
   • E2E test 1 (happy path) depends on the HTTP server starting within the exec_python closure before the forward proxy request arrives. The 300ms sleep should be sufficient but could be flaky on very slow CI. Mitigation: increase sleep or add a retry wrapper if CI shows flakiness.

   Documentation Impact

   • architecture/security-policy.md: Document forward proxy mode and its security constraints
   • architecture/plans/plain-http-forward-proxy.md: Update with resolved decisions

   ---

   Revision 2 — added E2E test plan with 7 test cases covering happy path (with in-sandbox HTTP server), deny paths, HTTPS rejection, public IP rejection, and log verification
   Revision 1 — initial plan
4. johntmyers

   added

   state:agent-readyApproved for agent implementationApproved for agent implementation

   state:in-progressWork is currently in progressWork is currently in progress

   on Mar 6, 2026
5. johntmyers commented on Mar 6, 2026

   

   johntmyers

   on Mar 6, 2026

   CollaboratorAuthor

   More actions

   Implementation Complete

   PR #158 implements the plain HTTP forward proxy for private IP endpoints.

   What was built

   • Forward proxy handler (handle_forward_proxy) in proxy.rs — handles non-CONNECT HTTP methods with absolute-form URIs
   • URI parser (parse_proxy_uri) — extracts scheme, host, port, path from proxy requests
   • Request rewriter (rewrite_forward_request) — converts absolute-form to origin-form, strips hop-by-hop headers, adds Via and Connection: close
   • Streaming support — uses tokio::io::copy_bidirectional with no idle timeout for chunked/SSE/LLM streaming responses
   • Updated dispatch — non-CONNECT methods route to forward proxy instead of returning 403

   Security constraints enforced

   All three must be true for a forward proxy request to succeed:

   1. OPA policy explicitly allows the destination
   2. Matched endpoint has allowed_ips configured
   3. All resolved IPs are RFC 1918 private

   Tests

   • 14 unit tests (9 for parse_proxy_uri, 5 for rewrite_forward_request)
   • 6 E2E tests (FWD-1 through FWD-6) — all passing in cluster
   • Regression: test_l4_non_connect_method_rejected still passes

   V1 simplifications

   • Connection: close injected (no keep-alive)
   • No L7 inspection on forwarded traffic
   • https:// forward proxy rejected (must use CONNECT)
6. johntmyers

   added

   state:pr-openedPR has been opened for this issuePR has been opened for this issue

   and removed

   state:in-progressWork is currently in progressWork is currently in progress

   on Mar 6, 2026
7. johntmyers

   added a commit that references this issue on Mar 6, 2026

   feat(proxy): support plain HTTP forward proxy for private IP endpoints

   ...

   This issue will close once commit 48b8dfe is merged into the 'main' branch.

   48b8dfe
8. johntmyers

   closed this as completedin #158on Mar 7, 2026
9. johntmyers

   added a commit that references this issue on Mar 7, 2026

   feat(proxy): support plain HTTP forward proxy for private IP endpoints (

   ...

   This issue will close once commit 0dcc165 is merged into the 'main' branch.

   0dcc165
10. drew

    added a commit that references this issue on Mar 16, 2026

    feat(proxy): support plain HTTP forward proxy for private IP endpoints (

    ...

    This issue will close once commit b8d873f is merged into the 'main' branch.

    b8d873f