diff --git a/guides/testing-on-v4-hardware.md b/guides/testing-on-v4-hardware.md index 1bb919c..effa628 100644 --- a/guides/testing-on-v4-hardware.md +++ b/guides/testing-on-v4-hardware.md @@ -17,6 +17,7 @@ related: - core/cpu-68080/performance-counters.md sources: - "lessons/2026-11-akb-verification.md" + - "lessons/2026-11-apolloexplorer.md" - "ApolloKnowledgeBase tools/v4/v4test.py and tools/v4/tests/ (2026-11-09)" - "ApolloOS-Port project, mk/card.mk and tools/ (WDrijver, 2026-10)" - "github.com/WDrijver/ApolloCrossDev Projects/ApolloLib/ApolloCrossDev_Debug.c" @@ -88,8 +89,15 @@ This is how the knowledge base's own hardware tests run (`tools/v4/`). names. ffmpeg needs about a second to start: start it before the program. 7. **Recover a hung V4.** Switch the plug off, wait 10 s, switch it on, and - wait until step 1 answers again. The tested V4 answered 15 s after - power-on (M). + wait until step 1 answers again. A V4 boots fast: the tested V4 answered + `ash` 15 s after power-on under ApolloOS R9.5, and answered ping 34 s + after power-on under AmigaOS 3.9 (M). Poll every second and give up + after 60 s; a V4 that is not back by then needs a look, not more + waiting. + - DON'T expect step 1 to answer after a power cycle unless + ApolloExplorerSrv starts at boot (`WBStartup` or `S:User-Startup`). + Without that, the V4 answers ping, but no remote command works until + someone starts the server by hand. `tools/v4/v4test.py` in the knowledge base does all of this: it builds the test with ApolloCrossDev, runs variants alternately (see @@ -110,4 +118,6 @@ picture and core number per run. | `ash` hangs with the older ApolloExplorerSrv, works after the update; `acp` works with both | (M) | Core 12556 | V4 UniCorn, ApolloOS R9.5, 2026-11-09 | | Direct UART writes at 115200 arrive; `Echo >SER:` does not | (M) | Core 12556 | `tools/v4/tests/serial-out`; `SER:` listened for at 9600, 19200, 38400, 57600, 115200 | | The V4 answers `ash` 15 s after power-on | (M) | Core 12556 | `v4test.py power-cycle`, Tasmota plug | +| The V4 answers ping 34.4 s after power-on | (M) | Core 12856 | V4 UniCorn, AmigaOS 3.9, Tasmota plug off 5 s then on, ping polled every 0.5 s, 2026-11-16 | +| Without ApolloExplorerSrv in the startup, nothing answers on port 30302 after a power cycle | (M) | Core 12856 | Same power cycle: port 30302 refused 90 s after power-on while ping answered | | The capture device needs `-pixel_format uyvy422` and offers 1920x1080 at up to 60 fps | (M) | any | ffmpeg 9.0.2, macOS | diff --git a/lessons/2026-11-apolloexplorer.md b/lessons/2026-11-apolloexplorer.md new file mode 100644 index 0000000..2211515 --- /dev/null +++ b/lessons/2026-11-apolloexplorer.md @@ -0,0 +1,137 @@ +--- +title: ApolloExplorer 1.4.4, an "unstable network" that was a driver, and server fixes +type: lesson +owner: WDrijver +status: draft +origin: ai-assisted +basis: [measured, source-code] +components: [devs, libraries, general] +applies_to: + core: "Core 12856" + os: [apolloos-r10, amigaos-3.9] + mode: [os-friendly] + board: [v4-unicorn] +tags: [network, tcp, bsdsocket, arostcp, amitcp-ng, v4net, mtu, systemtaglist, priority, testing] +related: + - os/pitfalls/v4net-full-size-frames-lost.md + - os/pitfalls/v4-rom-modules-and-reset.md + - guides/testing-on-v4-hardware.md +sources: + - "WDrijver/ApolloExplorer pull request #9, commits 5d466e9 and 4f4b504 (2026-11-16)" + - "Test harness tests/aetest.py and tests/mockserver.py in WDrijver/ApolloExplorer" + - "Measured on a V4 UniCorn, Core 12856, v4net.device 3.0, under ApolloOS R9.6 (AROSTCP) and AmigaOS 3.9 (AmiTCP_NG 4.1.8), 2026-11-16" + - "ApolloOS source: AROSTCP tcp_usrreq.c, tcp_output.c, kern/uipc_socket.c; rom/dos/systemtaglist.c, createnewproc.c" + - "AmiTCP_NG source (github.com/MW0MWZ/AmiTCP_NG): src/netinet/tcp_usrreq.c, src/net/ng_ifconfig.c, src/kern/uipc_socket.c" +--- + +## Project + +ApolloExplorer (github.com/WDrijver/ApolloExplorer, fork of +ronybeck/ApolloExplorer): a TCP server on the V4 (`ApolloExplorerSrv`) with +PC clients, the GUI, `acp` (copy) and `ash` (remote shell). Reported +problem: `acp` connections were unstable and `ash` mostly did not respond. +Worked on by Claude (AI) with WDrijver on 2026-11-16, first under ApolloOS, +then under AmigaOS 3.9; result: server 1.4.4 in pull request #9, plus +`mtu=1492` in the AmigaOS interface configuration. + +## What went well + +- **A protocol-level test harness first.** `tests/aetest.py` speaks the wire + protocol directly, so a failure points at the server or the network, not + at the Qt clients. A mock server with fault injection (split messages, + pings, latency) showed the clients were fine before anyone touched the V4. +- **Narrowing down by size.** Growing a directory listing one entry at a + time showed replies up to 1435 bytes arrive and 1497 bytes never do; a + `ping -D -s` binary search then proved it outside the application. +- **Reading the stack and OS sources instead of assuming.** The AROSTCP, + AmiTCP_NG and DOS sources answered questions that classic Amiga knowledge + would have answered wrongly or not at all (see the evidence table). +- **Before and after numbers for every fix.** Each fix got a test scenario + first (`busy`, `churn --op cd`, `fault split-header`), measured on the old + server, then on the new one, then again under the second OS. +- **Remote configuration without seeing the output.** With large replies + lost, `Search >NIL:` return codes (0 found, 5 not found) and output-free + commands (`Copy`, `Echo >>`) were enough to back up, check and change + `DEVS:NetInterfaces/V4net` on a V4 without a working keyboard. + +## What went wrong + +- Two earlier server fixes were reverted without a recorded reason. One of + them set `NP_Priority` in the `SystemTagList()` tags, which does nothing on + ApolloOS (below), so it could not have worked as intended. +- Server threads stuck sending to the V4's lost packets could not be shut + down; switching server versions needed power cycles. A soft reset was + avoided because of `os/pitfalls/v4-rom-modules-and-reset.md`. +- Each failed test left a stuck session on the old server until it was + restarted. +- The fix that passed every test under ApolloOS failed under AmigaOS, because + it depended on a socket option one stack has and the other lacks. +- ApolloExplorerSrv does not start at boot on the test V4, so after a power + cycle no remote work is possible until someone starts it by hand. + +## What the AI got wrong + +- **Blamed the server code first.** From the code review alone, the AI + ranked five server bugs as the likely causes. They were real, but the main + cause was the V4's Ethernet driver (`os/pitfalls/v4net-full-size-frames-lost.md`). + Only measuring on the V4 found it. +- **Assumed `TCP_MAXSEG` works on every stack.** It read the AROSTCP source + and stopped there; the server also ignored the `setsockopt()` result. + AmiTCP_NG does not implement the option, so under AmigaOS 3.9 the fix did + nothing and failed silently. +- **Blamed the driver for loss it did not cause.** Under ApolloOS, about one + packet per 4 MB was lost in bursts; the AI attributed that to the driver. + Under AmigaOS 3.9 with the same driver and core, 4 MB downloads showed no + loss at all, so it is not the driver alone. The cause is still unknown. +- **Misread a test failure.** A download that failed with "0 of 12 header + bytes" was first read as the server not sending its 16-byte + acknowledgement. Reading the server code showed the acknowledgement was + sent; what never arrived was the next, larger message. +- **Took the OS version from the conversation instead of the machine.** The + developer said ApolloOS R10; the server's discovery reply said R9.6. The AI + first wrote "R10" in a code comment. +- **Proposed an ApolloOS issue for the driver.** `v4net.device` is not in the + ApolloOS repository; it lives in the apollo-core SVN + (`apollo_software/net/V4Net`) and also runs on AmigaOS. +- **Waited 3 minutes after a power cycle.** The V4 is up in 15 to 35 s + (`guides/testing-on-v4-hardware.md`); a timeout should be based on that. +- **Got a protocol constant wrong in its own test tool.** It treated 0 as + success for `PMT_MKDIR`; the server answers `AT_OK` (1). + +## Evidence + +| Claim | Source | Core | Method | +|---|---|---|---| +| The V4 never sends a 1500-byte IP packet; 1498 bytes works; receiving 1500 works | (M) | Core 12856 | `ping -D -s` binary search, size sweep 0 to 1470, TCP upload; v4net.device 3.0, ApolloOS R9.6 and AmigaOS 3.9 | +| `TCP_MAXSEG` 1400 on the accepted socket fixes large replies under AROSTCP | (M) | Core 12856 | `tests/aetest.py smoke/shell/transfer` on ApolloOS R9.6: all timed out before, all pass after | +| `TCP_MAXSEG` can only lower the MSS a connection negotiated, so it must be set after `accept()`/`connect()` | (SRC) | any | AROSTCP `netinet/tcp_usrreq.c` | +| AmiTCP_NG 4.1.8 rejects `TCP_MAXSEG` with `EINVAL`; it reads `mtu=` from the interface file | (SRC) | any | AmiTCP_NG `src/netinet/tcp_usrreq.c` ("not yet"), `src/net/ng_ifconfig.c` | +| `mtu=1492` in `DEVS:NetInterfaces/V4net` fixes large replies under AmiTCP_NG | (M) | Core 12856 | AmigaOS 3.9: replies up to 3771 bytes arrive; every `tests/aetest.py` scenario passes | +| AROSTCP and AmiTCP_NG both support `SO_RCVTIMEO`/`SO_SNDTIMEO`; non-blocking only via `FIONBIO` (no `MSG_DONTWAIT`) | (SRC) | any | AROSTCP and AmiTCP_NG `kern/uipc_socket.c` | +| `NP_Priority` in `SystemTagList()` tags is ignored: its own `NP_Priority` (the caller's) comes first and `CreateNewProc()` uses the first match | (SRC) | any | ApolloOS `rom/dos/systemtaglist.c` (`TAG_MORE` after its defaults), `createnewproc.c` (`ApplyTagChanges`) | +| A shell command started from a priority-8 task starves AROSTCP: no new connection for 9.5 s during `List SYS: ALL` | (M) | Core 12856 | `tests/aetest.py busy` on ApolloOS R9.6; with `SetTaskPri(0)` around `SystemTagList()`: 46 replies, median 56 ms; on AmigaOS 3.9: 89 replies, median 36 ms | +| A finished session left its server thread alive ~60 s: 30 sessions held 4.6 MB, all back after 90 s | (M) | Core 12856 | `tests/aetest.py churn --avail`; after the fix -19 KB (ApolloOS), -15 KB (AmigaOS) | +| A header split with a 200 ms gap dropped the connection (50 `FIONREAD` polls without waiting) | (M) | Core 12856 | `tests/aetest.py fault split-header`, before and after | +| About one lost packet per 4 MB (1.2 to 1.5 s stall) under ApolloOS; none under AmigaOS | (M) | Core 12856 | Per-chunk arrival times of 4 MB downloads: ApolloOS R9.6 with `TCP_MAXSEG` 1400, 1.1 to 1.25 MB/s; AmigaOS 3.9 with `mtu=1492`, 4.2 to 4.3 MB/s, largest gap 15 ms | + +## Candidates for distillation + +- `os/pitfalls/v4net-full-size-frames-lost.md`: added together with this + lesson. Deprecate it once a fixed `v4net.device` is released. +- **`SystemTagList()` ignores `NP_Priority`.** First check AmigaOS 3.x: if it + honours the tag, this is an ApolloOS bug for an issue in + ApolloTeam-dev/ApolloOS.R10 (CONTRIBUTING.md section 2); if it does not, + it becomes an `os/pitfalls/` file for `os: [any]`. Either way: lower the + calling task with `SetTaskPri()` while the command runs. +- **A reference file for bsdsocket on V4** (`os/libraries/`): which TCP/IP + stack each OS uses (AROSTCP on ApolloOS, AmiTCP_NG or Roadshow on + AmigaOS), which socket options each supports, and the interface + configuration keywords such as `mtu=`. The knowledge base has nothing on + networking yet. +- **The lost packets under ApolloOS.** Not the driver alone (none under + AmigaOS with the same driver). Compare AROSTCP and AmiTCP_NG at the same + MSS before filing anything. +- **A guide for testing network services on V4** (`guides/`): a + protocol-level harness separate from the client, a mock server for the + client side, ping size sweeps, `Search >NIL:` return codes when output + cannot come back, and before/after numbers per fix. diff --git a/os/pitfalls/v4net-full-size-frames-lost.md b/os/pitfalls/v4net-full-size-frames-lost.md new file mode 100644 index 0000000..4eb5f9b --- /dev/null +++ b/os/pitfalls/v4net-full-size-frames-lost.md @@ -0,0 +1,124 @@ +--- +title: v4net.device 3.0-beta loses full-size Ethernet frames +type: pitfall +owner: WDrijver +status: draft +origin: ai-assisted +basis: [measured, source-code] +components: [devs] +applies_to: + core: "Core 12856" + os: [apolloos-r10, amigaos-3.9] + mode: [os-friendly] + board: [v4-unicorn] +tags: [network, ethernet, v4net, mtu, mss, tcp, arostcp, amitcp-ng, bsdsocket, stall] +related: + - os/pitfalls/v4-rom-modules-and-reset.md + - guides/testing-on-v4-hardware.md +sources: + - "Measured on a V4 UniCorn, Core 12856, v4net.device 3.0, under ApolloOS R9.6 (AROSTCP) and AmigaOS 3.9 (AmiTCP_NG 4.1.8), 2026-11-16 (ping size sweeps, tests/aetest.py of WDrijver/ApolloExplorer)" + - "ApolloOS source: workbench/network/stacks/AROSTCP/bsdsocket/netinet/tcp_usrreq.c, tcp_output.c" + - "AmiTCP_NG source (github.com/MW0MWZ/AmiTCP_NG): src/netinet/tcp_usrreq.c, src/net/ng_ifconfig.c" + - "WDrijver/ApolloExplorer pull request #9" + - "lessons/2026-11-apolloexplorer.md" +--- + +## Symptom + +A TCP connection from the V4 stalls as soon as the V4 sends more than about +1.4 KB in one go. Small replies keep arriving, and data sent *to* the V4 +arrives fine, so the connection looks alive but "unstable". In +ApolloExplorer, every `ash` command with output and every download hung, +while uploads worked. Seen under ApolloOS and under AmigaOS 3.9 alike. + +From any other host on the LAN: + +```sh +ping -D -s 1470 # 1498-byte IP packet: reply +ping -D -s 1472 # 1500-byte IP packet: no reply, ever +``` + +## Cause + +`v4net.device` 3.0 (beta) does not transmit a 1514-byte Ethernet frame +(a 1500-byte IP packet); 1512 bytes and smaller work (M). It still reports +an MTU of 1500, so the TCP/IP stack builds 1500-byte packets: TCP uses an +MSS of 1460, and every full-size segment is lost and retransmitted at the +same size forever. Receiving full-size frames works (M). The same happens +under ApolloOS (AROSTCP) and AmigaOS 3.9 (AmiTCP_NG), so it is the driver, +not the OS or the stack. + +Not yet known: whether `v4net.device` 2.991 and older behave the same, and +AmigaOS 3.2. The fix belongs in the driver; this pitfall stops applying once +a fixed version is released. + +## Wrong + +```c +/* Server: accept and send. Every reply over ~1.4 KB never arrives. */ +SOCKET client = accept( listenSocket, (struct sockaddr *)&addr, &addrLen ); +send( client, reply, replyLength, 0 ); +``` + +## Right + +Keep packets below 1500 bytes. How depends on the TCP/IP stack. + +**Every program at once, by configuration (preferred where supported).** +AmiTCP_NG reads `mtu=` from the interface file (SRC: `src/net/ng_ifconfig.c`; +it is clamped to the device MTU). Add a line to `DEVS:NetInterfaces/V4net` +and restart the network (power-cycle the V4): + +```text +mtu=1492 +``` + +The stack then sends at most 1492-byte packets and uses an MSS of 1452 (M). + +**One program, in code: `TCP_MAXSEG` on the connected socket.** AROSTCP +(ApolloOS) only lets `TCP_MAXSEG` lower the MSS a connection has negotiated +(SRC: `tcp_usrreq.c` accepts a value only when it is `<= tp->t_maxseg`). On a +listen socket `t_maxseg` is still the 512-byte default, so 1400 is refused +there with `EINVAL`. `tcp_output.c` caps every segment at `t_maxseg` (SRC). +AmiTCP_NG 4.1.8 does not implement `TCP_MAXSEG` at all: setting it returns +`EINVAL` ("not yet", SRC: `src/netinet/tcp_usrreq.c`), so check the result +and fall back to the configuration above. + +```c +/* Server: after accept(). A client: after connect(). */ +SOCKET client = accept( listenSocket, (struct sockaddr *)&addr, &addrLen ); +int maxSegmentSize = 1400; /* 1400 + 40 bytes headers + TCP options < 1498 */ +if( setsockopt( client, IPPROTO_TCP, TCP_MAXSEG, &maxSegmentSize, sizeof( maxSegmentSize ) ) < 0 ) +{ + /* Not supported by this stack (AmiTCP_NG): large sends will stall + unless the interface MTU is lowered. */ +} +send( client, reply, replyLength, 0 ); +``` + +`TCP_MAXSEG` is `0x02` in the ApolloCrossDev NDK header (`netinet/tcp.h`), +AROSTCP and AmiTCP_NG. + +## Detection + +- `ping -D -s 1472 ` from another host gets no reply, while + `ping -D -s 1470` does. With `mtu=1492` set this test still fails (the + V4 must then fragment its reply); test with TCP instead. +- Check the driver version: `Version DEVS:Networks/v4net.device`. +- A TCP service on the V4 whose small replies work but whose large ones + never arrive (the client receives not even the first byte). + +## Evidence + +| Claim | Source | Core | Method | +|---|---|---|---| +| A 1500-byte IP packet from the V4 never arrives; 1498 bytes does | (M) | Core 12856 | `ping -D -s` binary search from a Mac on the same gigabit LAN, v4net.device 3.0, ApolloOS R9.6 and AmigaOS 3.9 | +| Every payload size 0 to 1470 bytes gets a reply | (M) | Core 12856 | One ping per size, spaced out, ApolloOS R9.6 | +| The V4 receives 1500-byte packets | (M) | Core 12856 | TCP upload with 1460-byte segments, 200 KB in 0.33 to 0.36 s, both OSes | +| Other hosts on the same LAN send 1500-byte packets fine | (M) | n/a | `ping -D -s 1472` to three other hosts | +| TCP replies up to 1435 bytes arrive, 1497 bytes never | (M) | Core 12856 | Directory listing grown one entry at a time, both OSes | +| `TCP_MAXSEG` 1400 on the accepted socket makes large replies arrive under AROSTCP | (M) | Core 12856 | ApolloExplorer 1.4.4 on ApolloOS R9.6: shell output, 1 MB transfers and parallel sessions pass | +| `TCP_MAXSEG` has no effect under AmiTCP_NG 4.1.8 | (M) | Core 12856 | ApolloExplorer 1.4.4 on AmigaOS 3.9: replies over 1435 bytes still lost | +| `mtu=1492` in `DEVS:NetInterfaces/V4net` fixes it under AmiTCP_NG | (M) | Core 12856 | AmigaOS 3.9: replies up to 3771 bytes arrive; all ApolloExplorer tests pass; 4 MB downloads at 4.2 MB/s | +| `TCP_MAXSEG` can only lower `t_maxseg`; `tcp_output()` caps segments at it | (SRC) | any | AROSTCP `tcp_usrreq.c`, `tcp_output.c` | +| AmiTCP_NG rejects `TCP_MAXSEG` with `EINVAL`; reads `mtu=` per interface | (SRC) | any | AmiTCP_NG `src/netinet/tcp_usrreq.c`, `src/net/ng_ifconfig.c` | diff --git a/tools/v4/v4test.py b/tools/v4/v4test.py index aa57057..9781b1e 100644 --- a/tools/v4/v4test.py +++ b/tools/v4/v4test.py @@ -106,12 +106,16 @@ def power_cycle(): urllib.request.urlopen("http://%s/cm?cmnd=%s" % (plug, cmd), timeout=5).read() if cmd.endswith("Off"): time.sleep(10) - for i in range(24): - time.sleep(5) + # A V4 answers 15 to 35 s after power-on (guides/testing-on-v4-hardware.md): + # poll every second, and stop after 60 s instead of waiting minutes for a + # V4 that will not come back by itself. + start = time.time() + while time.time() - start < 60: + time.sleep(1) if alive(): - print("V4 up after %d s" % ((i + 1) * 5)) + print("V4 up after %d s" % (time.time() - start)) return True - print("V4 not answering 120 s after power-on") + print("V4 not answering 60 s after power-on") return False