Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions guides/testing-on-v4-hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ related:
- core/cpu-68080/performance-counters.md
sources:
- "lessons/2026-11-akb-verification.md"
- "lessons/2026-11-apolloexplorer.md"
- "ApolloKnowledgeBase tools/v4/v4test.py and tools/v4/tests/ (2026-11-09)"
- "ApolloOS-Port project, mk/card.mk and tools/ (WDrijver, 2026-10)"
- "github.com/WDrijver/ApolloCrossDev Projects/ApolloLib/ApolloCrossDev_Debug.c"
Expand Down Expand Up @@ -88,8 +89,15 @@ This is how the knowledge base's own hardware tests run (`tools/v4/`).
names. ffmpeg needs about a second to start: start it before the
program.
7. **Recover a hung V4.** Switch the plug off, wait 10 s, switch it on, and
wait until step 1 answers again. The tested V4 answered 15 s after
power-on (M).
wait until step 1 answers again. A V4 boots fast: the tested V4 answered
`ash` 15 s after power-on under ApolloOS R9.5, and answered ping 34 s
after power-on under AmigaOS 3.9 (M). Poll every second and give up
after 60 s; a V4 that is not back by then needs a look, not more
waiting.
- DON'T expect step 1 to answer after a power cycle unless
ApolloExplorerSrv starts at boot (`WBStartup` or `S:User-Startup`).
Without that, the V4 answers ping, but no remote command works until
someone starts the server by hand.

`tools/v4/v4test.py` in the knowledge base does all of this: it builds the
test with ApolloCrossDev, runs variants alternately (see
Expand All @@ -110,4 +118,6 @@ picture and core number per run.
| `ash` hangs with the older ApolloExplorerSrv, works after the update; `acp` works with both | (M) | Core 12556 | V4 UniCorn, ApolloOS R9.5, 2026-11-09 |
| Direct UART writes at 115200 arrive; `Echo >SER:` does not | (M) | Core 12556 | `tools/v4/tests/serial-out`; `SER:` listened for at 9600, 19200, 38400, 57600, 115200 |
| The V4 answers `ash` 15 s after power-on | (M) | Core 12556 | `v4test.py power-cycle`, Tasmota plug |
| The V4 answers ping 34.4 s after power-on | (M) | Core 12856 | V4 UniCorn, AmigaOS 3.9, Tasmota plug off 5 s then on, ping polled every 0.5 s, 2026-11-16 |
| Without ApolloExplorerSrv in the startup, nothing answers on port 30302 after a power cycle | (M) | Core 12856 | Same power cycle: port 30302 refused 90 s after power-on while ping answered |
| The capture device needs `-pixel_format uyvy422` and offers 1920x1080 at up to 60 fps | (M) | any | ffmpeg 9.0.2, macOS |
137 changes: 137 additions & 0 deletions lessons/2026-11-apolloexplorer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
---
title: ApolloExplorer 1.4.4, an "unstable network" that was a driver, and server fixes
type: lesson
owner: WDrijver
status: draft
origin: ai-assisted
basis: [measured, source-code]
components: [devs, libraries, general]
applies_to:
core: "Core 12856"
os: [apolloos-r10, amigaos-3.9]
mode: [os-friendly]
board: [v4-unicorn]
tags: [network, tcp, bsdsocket, arostcp, amitcp-ng, v4net, mtu, systemtaglist, priority, testing]
related:
- os/pitfalls/v4net-full-size-frames-lost.md
- os/pitfalls/v4-rom-modules-and-reset.md
- guides/testing-on-v4-hardware.md
sources:
- "WDrijver/ApolloExplorer pull request #9, commits 5d466e9 and 4f4b504 (2026-11-16)"
- "Test harness tests/aetest.py and tests/mockserver.py in WDrijver/ApolloExplorer"
- "Measured on a V4 UniCorn, Core 12856, v4net.device 3.0, under ApolloOS R9.6 (AROSTCP) and AmigaOS 3.9 (AmiTCP_NG 4.1.8), 2026-11-16"
- "ApolloOS source: AROSTCP tcp_usrreq.c, tcp_output.c, kern/uipc_socket.c; rom/dos/systemtaglist.c, createnewproc.c"
- "AmiTCP_NG source (github.com/MW0MWZ/AmiTCP_NG): src/netinet/tcp_usrreq.c, src/net/ng_ifconfig.c, src/kern/uipc_socket.c"
---

## Project

ApolloExplorer (github.com/WDrijver/ApolloExplorer, fork of
ronybeck/ApolloExplorer): a TCP server on the V4 (`ApolloExplorerSrv`) with
PC clients, the GUI, `acp` (copy) and `ash` (remote shell). Reported
problem: `acp` connections were unstable and `ash` mostly did not respond.
Worked on by Claude (AI) with WDrijver on 2026-11-16, first under ApolloOS,
then under AmigaOS 3.9; result: server 1.4.4 in pull request #9, plus
`mtu=1492` in the AmigaOS interface configuration.

## What went well

- **A protocol-level test harness first.** `tests/aetest.py` speaks the wire
protocol directly, so a failure points at the server or the network, not
at the Qt clients. A mock server with fault injection (split messages,
pings, latency) showed the clients were fine before anyone touched the V4.
- **Narrowing down by size.** Growing a directory listing one entry at a
time showed replies up to 1435 bytes arrive and 1497 bytes never do; a
`ping -D -s` binary search then proved it outside the application.
- **Reading the stack and OS sources instead of assuming.** The AROSTCP,
AmiTCP_NG and DOS sources answered questions that classic Amiga knowledge
would have answered wrongly or not at all (see the evidence table).
- **Before and after numbers for every fix.** Each fix got a test scenario
first (`busy`, `churn --op cd`, `fault split-header`), measured on the old
server, then on the new one, then again under the second OS.
- **Remote configuration without seeing the output.** With large replies
lost, `Search >NIL:` return codes (0 found, 5 not found) and output-free
commands (`Copy`, `Echo >>`) were enough to back up, check and change
`DEVS:NetInterfaces/V4net` on a V4 without a working keyboard.

## What went wrong

- Two earlier server fixes were reverted without a recorded reason. One of
them set `NP_Priority` in the `SystemTagList()` tags, which does nothing on
ApolloOS (below), so it could not have worked as intended.
- Server threads stuck sending to the V4's lost packets could not be shut
down; switching server versions needed power cycles. A soft reset was
avoided because of `os/pitfalls/v4-rom-modules-and-reset.md`.
- Each failed test left a stuck session on the old server until it was
restarted.
- The fix that passed every test under ApolloOS failed under AmigaOS, because
it depended on a socket option one stack has and the other lacks.
- ApolloExplorerSrv does not start at boot on the test V4, so after a power
cycle no remote work is possible until someone starts it by hand.

## What the AI got wrong

- **Blamed the server code first.** From the code review alone, the AI
ranked five server bugs as the likely causes. They were real, but the main
cause was the V4's Ethernet driver (`os/pitfalls/v4net-full-size-frames-lost.md`).
Only measuring on the V4 found it.
- **Assumed `TCP_MAXSEG` works on every stack.** It read the AROSTCP source
and stopped there; the server also ignored the `setsockopt()` result.
AmiTCP_NG does not implement the option, so under AmigaOS 3.9 the fix did
nothing and failed silently.
- **Blamed the driver for loss it did not cause.** Under ApolloOS, about one
packet per 4 MB was lost in bursts; the AI attributed that to the driver.
Under AmigaOS 3.9 with the same driver and core, 4 MB downloads showed no
loss at all, so it is not the driver alone. The cause is still unknown.
- **Misread a test failure.** A download that failed with "0 of 12 header
bytes" was first read as the server not sending its 16-byte
acknowledgement. Reading the server code showed the acknowledgement was
sent; what never arrived was the next, larger message.
- **Took the OS version from the conversation instead of the machine.** The
developer said ApolloOS R10; the server's discovery reply said R9.6. The AI
first wrote "R10" in a code comment.
- **Proposed an ApolloOS issue for the driver.** `v4net.device` is not in the
ApolloOS repository; it lives in the apollo-core SVN
(`apollo_software/net/V4Net`) and also runs on AmigaOS.
- **Waited 3 minutes after a power cycle.** The V4 is up in 15 to 35 s
(`guides/testing-on-v4-hardware.md`); a timeout should be based on that.
- **Got a protocol constant wrong in its own test tool.** It treated 0 as
success for `PMT_MKDIR`; the server answers `AT_OK` (1).

## Evidence

| Claim | Source | Core | Method |
|---|---|---|---|
| The V4 never sends a 1500-byte IP packet; 1498 bytes works; receiving 1500 works | (M) | Core 12856 | `ping -D -s` binary search, size sweep 0 to 1470, TCP upload; v4net.device 3.0, ApolloOS R9.6 and AmigaOS 3.9 |
| `TCP_MAXSEG` 1400 on the accepted socket fixes large replies under AROSTCP | (M) | Core 12856 | `tests/aetest.py smoke/shell/transfer` on ApolloOS R9.6: all timed out before, all pass after |
| `TCP_MAXSEG` can only lower the MSS a connection negotiated, so it must be set after `accept()`/`connect()` | (SRC) | any | AROSTCP `netinet/tcp_usrreq.c` |
| AmiTCP_NG 4.1.8 rejects `TCP_MAXSEG` with `EINVAL`; it reads `mtu=` from the interface file | (SRC) | any | AmiTCP_NG `src/netinet/tcp_usrreq.c` ("not yet"), `src/net/ng_ifconfig.c` |
| `mtu=1492` in `DEVS:NetInterfaces/V4net` fixes large replies under AmiTCP_NG | (M) | Core 12856 | AmigaOS 3.9: replies up to 3771 bytes arrive; every `tests/aetest.py` scenario passes |
| AROSTCP and AmiTCP_NG both support `SO_RCVTIMEO`/`SO_SNDTIMEO`; non-blocking only via `FIONBIO` (no `MSG_DONTWAIT`) | (SRC) | any | AROSTCP and AmiTCP_NG `kern/uipc_socket.c` |
| `NP_Priority` in `SystemTagList()` tags is ignored: its own `NP_Priority` (the caller's) comes first and `CreateNewProc()` uses the first match | (SRC) | any | ApolloOS `rom/dos/systemtaglist.c` (`TAG_MORE` after its defaults), `createnewproc.c` (`ApplyTagChanges`) |
| A shell command started from a priority-8 task starves AROSTCP: no new connection for 9.5 s during `List SYS: ALL` | (M) | Core 12856 | `tests/aetest.py busy` on ApolloOS R9.6; with `SetTaskPri(0)` around `SystemTagList()`: 46 replies, median 56 ms; on AmigaOS 3.9: 89 replies, median 36 ms |
| A finished session left its server thread alive ~60 s: 30 sessions held 4.6 MB, all back after 90 s | (M) | Core 12856 | `tests/aetest.py churn --avail`; after the fix -19 KB (ApolloOS), -15 KB (AmigaOS) |
| A header split with a 200 ms gap dropped the connection (50 `FIONREAD` polls without waiting) | (M) | Core 12856 | `tests/aetest.py fault split-header`, before and after |
| About one lost packet per 4 MB (1.2 to 1.5 s stall) under ApolloOS; none under AmigaOS | (M) | Core 12856 | Per-chunk arrival times of 4 MB downloads: ApolloOS R9.6 with `TCP_MAXSEG` 1400, 1.1 to 1.25 MB/s; AmigaOS 3.9 with `mtu=1492`, 4.2 to 4.3 MB/s, largest gap 15 ms |

## Candidates for distillation

- `os/pitfalls/v4net-full-size-frames-lost.md`: added together with this
lesson. Deprecate it once a fixed `v4net.device` is released.
- **`SystemTagList()` ignores `NP_Priority`.** First check AmigaOS 3.x: if it
honours the tag, this is an ApolloOS bug for an issue in
ApolloTeam-dev/ApolloOS.R10 (CONTRIBUTING.md section 2); if it does not,
it becomes an `os/pitfalls/` file for `os: [any]`. Either way: lower the
calling task with `SetTaskPri()` while the command runs.
- **A reference file for bsdsocket on V4** (`os/libraries/`): which TCP/IP
stack each OS uses (AROSTCP on ApolloOS, AmiTCP_NG or Roadshow on
AmigaOS), which socket options each supports, and the interface
configuration keywords such as `mtu=`. The knowledge base has nothing on
networking yet.
- **The lost packets under ApolloOS.** Not the driver alone (none under
AmigaOS with the same driver). Compare AROSTCP and AmiTCP_NG at the same
MSS before filing anything.
- **A guide for testing network services on V4** (`guides/`): a
protocol-level harness separate from the client, a mock server for the
client side, ping size sweeps, `Search >NIL:` return codes when output
cannot come back, and before/after numbers per fix.
124 changes: 124 additions & 0 deletions os/pitfalls/v4net-full-size-frames-lost.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
---
title: v4net.device 3.0-beta loses full-size Ethernet frames
type: pitfall
owner: WDrijver
status: draft
origin: ai-assisted
basis: [measured, source-code]
components: [devs]
applies_to:
core: "Core 12856"
os: [apolloos-r10, amigaos-3.9]
mode: [os-friendly]
board: [v4-unicorn]
tags: [network, ethernet, v4net, mtu, mss, tcp, arostcp, amitcp-ng, bsdsocket, stall]
related:
- os/pitfalls/v4-rom-modules-and-reset.md
- guides/testing-on-v4-hardware.md
sources:
- "Measured on a V4 UniCorn, Core 12856, v4net.device 3.0, under ApolloOS R9.6 (AROSTCP) and AmigaOS 3.9 (AmiTCP_NG 4.1.8), 2026-11-16 (ping size sweeps, tests/aetest.py of WDrijver/ApolloExplorer)"
- "ApolloOS source: workbench/network/stacks/AROSTCP/bsdsocket/netinet/tcp_usrreq.c, tcp_output.c"
- "AmiTCP_NG source (github.com/MW0MWZ/AmiTCP_NG): src/netinet/tcp_usrreq.c, src/net/ng_ifconfig.c"
- "WDrijver/ApolloExplorer pull request #9"
- "lessons/2026-11-apolloexplorer.md"
---

## Symptom

A TCP connection from the V4 stalls as soon as the V4 sends more than about
1.4 KB in one go. Small replies keep arriving, and data sent *to* the V4
arrives fine, so the connection looks alive but "unstable". In
ApolloExplorer, every `ash` command with output and every download hung,
while uploads worked. Seen under ApolloOS and under AmigaOS 3.9 alike.

From any other host on the LAN:

```sh
ping -D -s 1470 <v4-ip> # 1498-byte IP packet: reply
ping -D -s 1472 <v4-ip> # 1500-byte IP packet: no reply, ever
```

## Cause

`v4net.device` 3.0 (beta) does not transmit a 1514-byte Ethernet frame
(a 1500-byte IP packet); 1512 bytes and smaller work (M). It still reports
an MTU of 1500, so the TCP/IP stack builds 1500-byte packets: TCP uses an
MSS of 1460, and every full-size segment is lost and retransmitted at the
same size forever. Receiving full-size frames works (M). The same happens
under ApolloOS (AROSTCP) and AmigaOS 3.9 (AmiTCP_NG), so it is the driver,
not the OS or the stack.

Not yet known: whether `v4net.device` 2.991 and older behave the same, and
AmigaOS 3.2. The fix belongs in the driver; this pitfall stops applying once
a fixed version is released.

## Wrong

```c
/* Server: accept and send. Every reply over ~1.4 KB never arrives. */
SOCKET client = accept( listenSocket, (struct sockaddr *)&addr, &addrLen );
send( client, reply, replyLength, 0 );
```

## Right

Keep packets below 1500 bytes. How depends on the TCP/IP stack.

**Every program at once, by configuration (preferred where supported).**
AmiTCP_NG reads `mtu=` from the interface file (SRC: `src/net/ng_ifconfig.c`;
it is clamped to the device MTU). Add a line to `DEVS:NetInterfaces/V4net`
and restart the network (power-cycle the V4):

```text
mtu=1492
```

The stack then sends at most 1492-byte packets and uses an MSS of 1452 (M).

**One program, in code: `TCP_MAXSEG` on the connected socket.** AROSTCP
(ApolloOS) only lets `TCP_MAXSEG` lower the MSS a connection has negotiated
(SRC: `tcp_usrreq.c` accepts a value only when it is `<= tp->t_maxseg`). On a
listen socket `t_maxseg` is still the 512-byte default, so 1400 is refused
there with `EINVAL`. `tcp_output.c` caps every segment at `t_maxseg` (SRC).
AmiTCP_NG 4.1.8 does not implement `TCP_MAXSEG` at all: setting it returns
`EINVAL` ("not yet", SRC: `src/netinet/tcp_usrreq.c`), so check the result
and fall back to the configuration above.

```c
/* Server: after accept(). A client: after connect(). */
SOCKET client = accept( listenSocket, (struct sockaddr *)&addr, &addrLen );
int maxSegmentSize = 1400; /* 1400 + 40 bytes headers + TCP options < 1498 */
if( setsockopt( client, IPPROTO_TCP, TCP_MAXSEG, &maxSegmentSize, sizeof( maxSegmentSize ) ) < 0 )
{
/* Not supported by this stack (AmiTCP_NG): large sends will stall
unless the interface MTU is lowered. */
}
send( client, reply, replyLength, 0 );
```

`TCP_MAXSEG` is `0x02` in the ApolloCrossDev NDK header (`netinet/tcp.h`),
AROSTCP and AmiTCP_NG.

## Detection

- `ping -D -s 1472 <v4-ip>` from another host gets no reply, while
`ping -D -s 1470` does. With `mtu=1492` set this test still fails (the
V4 must then fragment its reply); test with TCP instead.
- Check the driver version: `Version DEVS:Networks/v4net.device`.
- A TCP service on the V4 whose small replies work but whose large ones
never arrive (the client receives not even the first byte).

## Evidence

| Claim | Source | Core | Method |
|---|---|---|---|
| A 1500-byte IP packet from the V4 never arrives; 1498 bytes does | (M) | Core 12856 | `ping -D -s` binary search from a Mac on the same gigabit LAN, v4net.device 3.0, ApolloOS R9.6 and AmigaOS 3.9 |
| Every payload size 0 to 1470 bytes gets a reply | (M) | Core 12856 | One ping per size, spaced out, ApolloOS R9.6 |
| The V4 receives 1500-byte packets | (M) | Core 12856 | TCP upload with 1460-byte segments, 200 KB in 0.33 to 0.36 s, both OSes |
| Other hosts on the same LAN send 1500-byte packets fine | (M) | n/a | `ping -D -s 1472` to three other hosts |
| TCP replies up to 1435 bytes arrive, 1497 bytes never | (M) | Core 12856 | Directory listing grown one entry at a time, both OSes |
| `TCP_MAXSEG` 1400 on the accepted socket makes large replies arrive under AROSTCP | (M) | Core 12856 | ApolloExplorer 1.4.4 on ApolloOS R9.6: shell output, 1 MB transfers and parallel sessions pass |
| `TCP_MAXSEG` has no effect under AmiTCP_NG 4.1.8 | (M) | Core 12856 | ApolloExplorer 1.4.4 on AmigaOS 3.9: replies over 1435 bytes still lost |
| `mtu=1492` in `DEVS:NetInterfaces/V4net` fixes it under AmiTCP_NG | (M) | Core 12856 | AmigaOS 3.9: replies up to 3771 bytes arrive; all ApolloExplorer tests pass; 4 MB downloads at 4.2 MB/s |
| `TCP_MAXSEG` can only lower `t_maxseg`; `tcp_output()` caps segments at it | (SRC) | any | AROSTCP `tcp_usrreq.c`, `tcp_output.c` |
| AmiTCP_NG rejects `TCP_MAXSEG` with `EINVAL`; reads `mtu=` per interface | (SRC) | any | AmiTCP_NG `src/netinet/tcp_usrreq.c`, `src/net/ng_ifconfig.c` |
12 changes: 8 additions & 4 deletions tools/v4/v4test.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,12 +106,16 @@ def power_cycle():
urllib.request.urlopen("http://%s/cm?cmnd=%s" % (plug, cmd), timeout=5).read()
if cmd.endswith("Off"):
time.sleep(10)
for i in range(24):
time.sleep(5)
# A V4 answers 15 to 35 s after power-on (guides/testing-on-v4-hardware.md):
# poll every second, and stop after 60 s instead of waiting minutes for a
# V4 that will not come back by itself.
start = time.time()
while time.time() - start < 60:
time.sleep(1)
if alive():
print("V4 up after %d s" % ((i + 1) * 5))
print("V4 up after %d s" % (time.time() - start))
return True
print("V4 not answering 120 s after power-on")
print("V4 not answering 60 s after power-on")
return False


Expand Down
Loading