[stefanskotte]

Hi Rob, this one is optional, and yours to take, change or decline.

It adds LZ4 as a third frame encoding next to MFM and RLE. It is used only when the board says it supports LZ4, so with your current Pico firmware nothing changes.

Which encoding is used

CPU	board -> Amiga	Amiga -> board
68000 / 68010	LZ4	RLE (as today)
68020 / 68030	LZ4	LZ4 (hash table 1,024 entries)
68040 / 68060	LZ4	LZ4 (hash table 4,096 entries)
a driver without this change	MFM	RLE

The driver reads SysBase->AttnFlags at open. A 68000 decodes LZ4 quickly but encodes it too slowly to gain anything, so its uploads stay RLE.

Uploads use LZ4 only in place of RLE: with FLAGS_AMIGA_COMPRESSION off they stay MFM.

Wire format

Bit 14 of a frame's size word marks a standard LZ4 block, and bits 0-13 give its length.

• Toward the Amiga the block is MFM-encoded (0xC000 | n, 2n bytes on the wire), because raw bytes toward the Amiga are misread (see the RLE -> MFM PR).
• Toward the board it goes raw (0x4000 | n), padded to even, with no run of more than 5 zero bytes.

A frame's matches may reach into earlier frames of the same batch, which lie back to back in a 16,128-byte window, but never across batches. Every frame starts at an even window offset: after an odd-length frame both sides lay one zero pad byte, which is never sent.

LZ4 is used only when the block is shorter than the frame's present encoding, so incompressible data is never larger than before.

Capabilities

The driver sets 0x10 in DataHeader.compressed ("I decode LZ4"). The board states what it does in LiveStatus.reserved, byte 155 of the status: bit 0 means it sends LZ4, bit 1 means it accepts LZ4.

Your current Pico firmware leaves byte 155 at 0, so with it this driver behaves exactly as today: MFM down, RLE up. The only change it sees is the 0x10 bit, and your firmware ignores it, because for Ethernet batches it reads only FLAGS_FROM_DRIVER (PaulaNET.cpp:583). We checked that in your source; we have not run it on your hardware.

Code

webadf_lz4.c/.h (the logic) and lz4_68k.asm/.h (a 68000 LZ4 decoder and encoder) are new files, MIT-licensed (LICENSE-lz4), written by us. The 68k code is checked under the vamos emulator against liblz4 and against its own C model.

Your files change by a few lines, all marked webadf LZ4:

• paulanetio.c: decode 0xC000 frames into the window, the LZ4 choice in paulaNetAddPacket, the 0x10 bit, and the bounds check using the wire length of a 0xC000 frame.
• Makefile: two objects and a vasm rule.

The driver makes one extra AllocMem at open (MEMF_REVERSE on V39+). If that allocation fails, it opens as your stock driver. NetDevice/LZ4.md describes the format.

Board half

webadf's board firmware 1.16.0 implements the other side in wifi-floppy/firmware/src/pnet_lz4.c (the C LZ4 codec, linked within a batch) and pnet_pack.c (choosing an encoding for each frame and packing the batch, with a 500 us encode budget for each batch). Both are MIT, there for you to port to the Pico firmware.

What has been tested on hardware

Only downloads (board to Amiga), on a standard A500 (68000, 8 MHz) with fast RAM. A 68000 never sends LZ4, so the upload path has not run on a real Amiga at all. The 68020+ upload path has passed only under the vamos emulator (68020 and 68040). We will add A4000D numbers when we have them.

Measurements

On that A500, with webadf board firmware 1.16.0, sending files to the Amiga. "Your driver" is your driver as it is; "this driver" is this change, built with your Makefile.

file	your driver	this driver
880 KB ADF	140.4 s	112.6 s	-20%
LHA archive (already compressed)	65.5 s	62.5 s	not slower
text	97.9 s	84.6 s	-14%
HTML	10.0 s	8.2 s	-18%

Every file arrived intact. Across the four sends, 687 frames went LZ4 and 766 MFM, saving 428,778 bytes on the wire, with no late reads and no duplicates.

The gain is smaller than the compression ratio, because on a 68000 the TCP stack and the disk writes become the limit once the link is faster.

Build

Tested against your commit 0afd5fa. Built with your Makefile and vbcc (stefanreinauer/amiga-gcc image, NDK 3.2 R4 headers).

This produces a 19,704-byte PaulaNET.device; with your own sources the same toolchain gives 12,808 bytes. version.h is left as yours.

Stefan Skotte (webadf)