ecm 2026-07: Use LZSA1 or LZSA2 compression for crab-type flags

[ecm-pushbx]

As previously discussed in #4 I want to replace the lz4_8088 depacker, mainly due to the incompatible usage conditions.

This PR switches to use the LZSA2 or LZSA1 formats. The depackers are provided in https://hg.pushbx.org/ecm/vgadepak/file/356876ae6503

The usage conditions of these are inherited by the original 8086 LZSA depacker, reading as a zlib license as follows:

; 8086 Assembly LZSA2/LZSA1 depacker
;  by E. C. Masloch, 2021, 2026
;
;  based on:
;  decompress_small.S - space-efficient decompressor implementation for 8088
;
;  Copyright (C) 2019 Emmanuel Marty
;
;  This software is provided 'as-is', without any express or implied
;  warranty.  In no event will the authors be held liable for any damages
;  arising from the use of this software.
;
;  Permission is granted to anyone to use this software for any purpose,
;  including commercial applications, and to alter it and redistribute it
;  freely, subject to the following restrictions:
;
;  1. The origin of this software must not be misrepresented; you must not
;     claim that you wrote the original software. If you use this software
;     in a product, an acknowledgment in the product documentation would be
;     appreciated but is not required.
;  2. Altered source versions must be plainly marked as such, and must not be
;     misrepresented as being the original software.
;  3. This notice may not be removed or altered from any source distribution.

To assemble the depackers, use either of the following commands:

nasm -f obj lzsa.asm -D_LZSA1=1 -D_LZSA2=0 -l lzsa1.lst
nasm -f obj lzsa.asm -D_LZSA1=0 -D_LZSA2=1 -l lzsa2.lst

Changes in the VGAPride repo include:

• The planize.py script calls LZSA with -f 1 or -f 2, and emits .sa files. It also uses the -N switch currently, which requires a patch I submitted to ecm eod marker, capital -M and capital -N switches, and -S switch (more compatible block limit) emmanuel-marty/lzsa#74
• display.cpp calls lzsa_depack with as yet same protocol as lz4_depack
• vgapride.cpp refers to lzsa rather than lz4
• Turbo C project files include linking of lzsa.obj rather than assembling lz4_8088.asm
• planize.py also emits the uncompressed files, which was useful during debugging
• I had to set the compiler include and library directories to E:\TC\ to work in my setup. You may want to not include that.

I prepared a diskette image in https://pushbx.org/ecm/test/20260728/ with VGAPRIDE.EXE from https://github.com/foone/VGAPride/releases/tag/v0.6 and vgalzsa1.exe and vgalzsa2.exe which I tested as a remote diskette in https://www.pcjs.org/machines/pcx86/compaq/deskpro386/vga/ The LZSA2 build runs noticeably slower than the lz4_8088 one, LZSA1 is much closer in speed.

Here's the file sizes:

test/20260728$ ls -lgG
total 2516
-rw-r----- 1  224573 Nov 25  2022 20221125.exe
-rw-r--r-- 1 1474560 Jul 28 20:09 diskette.img
-rw-r--r-- 1      26 Jul 28 20:09 revision.txt
-rw-r----- 1  245399 Jul 28 19:16 vgalzsa1.exe
-rw-r----- 1  239239 Jul 28 20:08 vgalzsa2.exe
-rw-r--r-- 1  126580 Jul 12  2023 vgapride_0.6.zip
-rw-r--r-- 1  255283 Jul 12  2023 VGAPRIDE.EXE

255 kB for lz4_8088, 245 kB for LZSA1, and 239 kB for LZSA2.

Using raw LZSA blocks would allow to save a bit of data and some code, and we wouldn't have to depend on the -N switch. On the other hand, non-raw LZSA streams allow dispatching for the variant (LZSA2 vs LZSA1) so a single depacker function could handle both formats.

I am also interested in benchmarking the depackers on several different systems, for which I may re-use some of the code I added to inicomp. Further, display.cpp and the VGA commands format could be extended to pass along the buffer sizes to the depacker, which would allow a useful _UNSAFE=0 build which actually checks for buffer overflows. Finally, the display.cpp caller currently does not check the depacker's return value.

[ecm-pushbx]

Latest change to planize.py allows specifying the format out of lz4old, lz4, lzsa1, or lzsa2. The old format uses lz4demo from 2013 as I described in #6

I also added a benchmark program, which assembles with an incbin of a packed file (eg transcrab_plane0.sa2) and an unpacked file (eg transcrab_plane0.bin), and links to one of the depackers using our revision of WarpLink (this must be run on a DOS, my script uses dosemu2).

The benchmark files are all built in https://pushbx.org/ecm/test/20260729/vgadepak/ and cmd.log contains the exact commands that I used to generate them. The results, only from the non-KVM dosemu2 on our server yet:

       0.34ms/run, 8192 runs, sum 2.80s=51t    lzsa1
       0.46ms/run, 8192 runs, sum 3.79s=69t    lzsa2
      12.77ms/run, 8192 runs, sum 1.7min=1905t   lz4ecm
       0.24ms/run, 8192 runs, sum 2.03s=37t lz4_8088
       0.22ms/run, 8192 runs, sum 1.81s=33t lz4_8088

The last result is from assembling the test with the -D_TEST_PROGRESS=0 switch instead of the default.

[ecm-pushbx]

Added raw variants of LZSA1 and LZSA2 to the planize.py script. And the corresponding define to the depackers.

This is not noticeably faster than plain LZSA1, but the code required shrinks quite some.

$ ~/local.new/bin/dosemu -dumb -td -kt -q -quiet -K "$PWD" -E "lzsa1.exe b 8192"
Time:        0.30ms/run, 8192 runs, sum 2.52s=46t    lzsa1
$ ~/local.new/bin/dosemu -dumb -td -kt -q -quiet -K "$PWD" -E "lzsa1r.exe b 8192"
Time:        0.30ms/run, 8192 runs, sum 2.47s=45t    lzsa1 raw
$ nasm -f obj lzsa.asm -D_LZSA1=1 -D_LZSA2=0 -o lzsa1.obj -D_RAW=0
lzsa.asm:1295: warning: inilzsa1: 432 bytes used for depacker [-w+user]
$ nasm -f obj lzsa.asm -D_LZSA1=1 -D_LZSA2=0 -o lzsa1r.obj -D_RAW=1
lzsa.asm:1295: warning: inilzsa1: 176 bytes used for depacker [-w+user]

[ecm-pushbx]

I added a benchmarking mode that actually repeatedly displays a flag from the main VGAPride executable, then determines how many ms (with two decimal places) a single display run took. I'm not sure whether this is a valid check but it does seem to work.

Files in https://pushbx.org/ecm/test/20260731/ - note how lzsa1r is nearly as fast, if not faster, than the old lz4_8088 depacker.

Running the image https://pushbx.org/ecm/test/20260731/diskette.img as a remote disk in https://www.pcjs.org/machines/pcx86/compaq/deskpro386/vga/ results, for bench trans-crab 100, results in 107.69ms/run 196t/sum for lz4_8088 and 122.52ms/run 223t/sum for lzsa1r.

I also added two string constants to all six depacker variants, which identify the depacker (_depackername) with a short name and specify the depacker attribution line for the help screen (_depackerattribution).

[ecm-pushbx]

By the way, at some point the bench.asm benchmark resulted in 0.30ms for lzsa1r normally but 0.20ms when the debugger was loaded. That's what the _STARTPAD define is about. The same speedup could be observed by adding 128 KiB of padding after the PSP. However, the current lzsa1r does not show this defect (0.19ms throughout) and lz4_8088 never acted differently under these conditions (always around 0.22ms).

[ecm-pushbx]

Added Jim Leonard's lzsa1 jump-table based depacker to my vgadepak repo: https://hg.pushbx.org/ecm/vgadepak/file/3d9f2c185f3e/lzsa1jmp.asm

This is compatible to the raw LZSA1 format (hence lzsa1jmp is an alias to lzsa1raw for planize.py). The depacker including all jump tables comes in at 2349 bytes but for that cost it is faster than even the lz4_8088 code.

The usage conditions for this are the zlib license, curing my original problem with the lz4_8088 depacker.

bench.asm result: Time: 0.16ms/run, 8192 runs, sum 1.37s=25t lzsa1 raw/jmp

Running files from https://pushbx.org/ecm/test/20260801/ on https://www.pcjs.org/machines/pcx86/compaq/deskpro386/vga/ time for vgapride bench trans-crab 100: 77.47ms/run 141t/sum for lzsa1j vs 107.14ms/run 195t/sum for lz4_8088