diff --git a/README.md b/README.md index b24b513..b0a8db4 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Every coded stream is decoded again and compared sample for sample. - `ada/block_codec.ads`, `.adb`: block codec. SPARK decoder with no heap and no exceptions; malformed input returns `Truncated_Input` or `Bad_Data`. Encoder with no heap and no shared state; linear prediction analysis in 64-bit floating point. - `ada/canonical_decoding`, `bit_streams`, `checksums`, `range_coding`: canonical Huffman decoding, bit access, CRC-16 and the binary range coder, in SPARK. `ada/checksums_test.adb` tests the CRC; `ada/corrupt_test.adb` walks a written stream and checks its framing and every CRC against flipped bits. `ada/fuzz_decode.adb` decodes changed copies of the test vectors and fails on any exception. -- `ada/codec_main.adb`: stream of split trees and stereo modes around the block codec, run on a WAV file. `ada/stream_reading` and `ada/stream_format`: header, groups and damage handling, read from a stream's own bytes. `ada/stream_writing` with `ada/stream_encode.adb`: audio file to stream. `ada/stream_decode.adb`: stream to audio file, group by group. `ada/byte_files`: files and pipes. `ada/block_codec_ffi` and `ada/codec_lib.gpr`: block codec as a C library, `ada/lib/libblockcodec.so`. +- `ada/codec_main.adb`: stream of split trees and stereo modes around the block codec, run on a WAV file. `ada/stream_reading` and `ada/stream_format`: header, groups and damage handling, read from a stream's own bytes. `ada/stream_writing` with `ada/blkc_encode.adb`: audio file to stream. `ada/blkc_decode.adb`: stream to audio file, group by group. `ada/byte_files`: files and pipes. `ada/block_codec_ffi` and `ada/codec_lib.gpr`: block codec as a C library, `ada/lib/libblockcodec.so`. - `ada/codec_api` with `include/blockcodec.h`: the library's C interface for streams; `capi/` tests it. - `ada/wav_files`: input from WAV, RF64, Wave64 and AIFF, WAV output. `ada/audio_huffman.adb` with `huffman_codes`, `predictors` and `residual_coders`: experiment coders. - `haskell/src/BlockCodec.hs`, `Stream.hs`, `Checksum.hs`, `RangeCoder.hs`: block codec, stream, CRC-16 and range coder in Haskell, groups coded in parallel. `haskell/hybrid/`: same stream around the SPARK library. `haskell/src/Blocks.hs`, `Tiers.hs`: parallel versions of experiment schemes. @@ -217,7 +217,7 @@ The encoder codes every node of each group's split tree, up to 15 blocks per gro Haskell decodes every block that does not reach back before itself in parallel. Pure Haskell times at 16 threads vary between runs, 59.56 s and 40.57 s in two runs of an earlier build, while Ada moved by 0.3 s. -`tool_timing.sh` times `flac`, `wavpack` and the release build of `stream_encode` and `stream_decode`, on one thread, one file at a time. Each tool's decoder writes a WAV file, and its samples are checked against the input's. FLAC does not code 32-bit float, so the table sums every file but the three float ones. +`tool_timing.sh` times `flac`, `wavpack` and the release build of `blkc-encode` and `blkc-decode`, on one thread, one file at a time. Each tool's decoder writes a WAV file, and its samples are checked against the input's. FLAC does not code 32-bit float, so the table sums every file but the three float ones.
Size and time against FLAC and WavPack (13 rows) @@ -231,12 +231,12 @@ Haskell decodes every block that does not reach back before itself in parallel. | `wavpack` | 52,446,880 | 0.72 s | 0.37 s | | `wavpack -hh` | 52,082,680 | 1.10 s | 0.57 s | | `wavpack -hh -x4` | 43,358,134 | 12.26 s | 0.65 s | -| `stream_encode --level 0` | 42,436,904 | 2.78 s | 1.23 s | -| `stream_encode --level 1` | 37,914,752 | 4.72 s | 1.42 s | -| `stream_encode --level 2` | 37,462,799 | 15.54 s | 2.32 s | -| `stream_encode --level 4` | 37,109,813 | 110.49 s | 2.58 s | -| `stream_encode --level 4`, every processor | 37,109,813 | 15.98 s | 2.34 s | -| `stream_encode --level 5`, every processor | 36,954,023 | 96.32 s | 9.48 s | +| `blkc-encode --level 0` | 42,436,904 | 2.78 s | 1.23 s | +| `blkc-encode --level 1` | 37,914,752 | 4.72 s | 1.42 s | +| `blkc-encode --level 2` | 37,462,799 | 15.54 s | 2.32 s | +| `blkc-encode --level 4` | 37,109,813 | 110.49 s | 2.58 s | +| `blkc-encode --level 4`, every processor | 37,109,813 | 15.98 s | 2.34 s | +| `blkc-encode --level 5`, every processor | 36,954,023 | 96.32 s | 9.48 s |
@@ -244,7 +244,7 @@ Level 0 is smaller than every FLAC and WavPack mode and encodes in 2.78 s, again Coding groups on all 16 threads, level 4 encodes in 15.98 s against 110.49 s on one, to the same bytes. Level 5 is 0.4% smaller than level 4 and decodes in 9.48 s, about four times as long, since its adaptive filter does up to 256 multiply-adds per sample where it is kept. WavPack without `-x` is larger than `flac -0` here: the 192 kHz recording alone costs 4.9 MB more than with `-x4`. Its size is measured with a seek point every second, and its times come from the run before that change. -The release build leaves out run-time checks and assertions and inlines across units at `-O3`. Its streams and decoded files are identical to the checked build's. Over the whole corpus at level 0 on one thread, best of three runs, release `stream_encode` takes 2.51 s against 3.54 s for the checked build, and `stream_decode` 1.32 s against 1.88 s. +The release build leaves out run-time checks and assertions and inlines across units at `-O3`. Its streams and decoded files are identical to the checked build's. Over the whole corpus at level 0 on one thread, best of three runs, release `blkc-encode` takes 2.51 s against 3.54 s for the checked build, and `blkc-decode` 1.32 s against 1.88 s. Haskell uses unboxed vectors whose loops fuse, strict unpacked writer state, `countLeadingZeros` for unary and bit-length codes, and decoding from a 32-bit peek. Parallelism comes from sparks over pure functions, so output cannot depend on scheduling. Executables run with `-N -A4m`; a 4 MB allocation area measured faster than 64 MB. @@ -297,7 +297,7 @@ Installed sizes of the tools compared, and this work's binaries. Decoder rows co | Read-only data of the five units, `-Os -gnatp` | 521 (637 with checks on) | | `libblockcodec.so`, x86-64, stripped, Ada runtime included, blocks and streams | 777,816 | | `codec_main`, x86-64, stripped, needs only libc | 777,984 | -| `stream_decode`, x86-64, stripped, needs only libc | 670,528 | +| `blkc-decode`, x86-64, stripped, needs only libc | 670,528 | | `huffman-audio`, Haskell, x86-64, stripped, needs libgmp | 6,543,248 | | `huffman-audio-hybrid`, x86-64, stripped, needs `libblockcodec.so` and libgmp | 6,495,776 | @@ -363,9 +363,9 @@ Input is WAV, RF64, Wave64 or AIFF, 8 to 32-bit integer, 32-bit float, or 64-bit | `--no-pairs` | code every channel alone | | `--features ` | write each group's features and whether the search split its root | -`stream_encode [--level <0 to 5>] [--tag ] [--seek-every ] [--independent] [--no-pairs] [--threads ] [--verify] ` codes a file into a stream, byte for byte the stream `codec_main --stream` writes. It codes groups on every processor, or on `--threads` of them, and the stream is the same for any count. Either program takes `-` for standard input or output. With `--verify` it decodes the stream before writing it and checks every frame against the audio MD5. On a mismatch it writes nothing and exits with status 1. +`blkc-encode [--level <0 to 5>] [--tag ] [--seek-every ] [--independent] [--no-pairs] [--threads ] [--verify] ` codes a file into a stream, byte for byte the stream `codec_main --stream` writes. It codes groups on every processor, or on `--threads` of them, and the stream is the same for any count. Either program takes `-` for standard input or output. With `--verify` it decodes the stream before writing it and checks every frame against the audio MD5. On a mismatch it writes nothing and exits with status 1. -`stream_decode [--skip ] [--until ] ` reads a stream from its own bytes and writes the input file back, container, chunks and samples byte for byte (`docs/container.md`). It decodes group by group and keeps only the history later blocks may read, so its memory does not grow with the audio's length. With `--skip` or `--until` it writes frames from the first named up to the second as a WAV file, decoding only the groups those frames need, as `flac --skip --until` does. +`blkc-decode [--skip ] [--until ] ` reads a stream from its own bytes and writes the input file back, container, chunks and samples byte for byte (`docs/container.md`). It decodes group by group and keeps only the history later blocks may read, so its memory does not grow with the audio's length. With `--skip` or `--until` it writes frames from the first named up to the second as a WAV file, decoding only the groups those frames need, as `flac --skip --until` does. Exit status 0 means the stream was intact. Status 1 means damage: frames of unreadable groups, and blocks that did not decode, are written as zeros and counted on stderr. Status 2 means the file is not a readable stream. @@ -384,7 +384,7 @@ The library builds the units proved free of run-time errors without run-time che ### Release -`release.sh [out] [name]` builds the release programs and the library with the version `git describe` gives, and packs them with the header, the specification, the README and the licenses into `--linux-x86_64.tar.gz` with its SHA-256. The name is `blockcodec` when none is given. `stream_encode --version` and `stream_decode --version` report that version, the format version and the C API version. +`release.sh [out] [name]` builds the release programs and the library with the version `git describe` gives, and packs them with the header, the specification, the README and the licenses into `--linux-x86_64.tar.gz` with its SHA-256. The name is `blockcodec` when none is given. `blkc-encode --version` and `blkc-decode --version` report that version, the format version and the C API version. Once the repository variable `RELEASE_NAME` is set, every merge to main runs `.github/workflows/release.yml` on a hosted runner. It tags the merge as the next version, runs `release.sh` with that name, and publishes a GitHub release with generated notes. Releases are numbered from v0.1.0, and each one raises the last number only: v0.1.1, v0.1.2, and on past v0.1.9 to v0.1.10. diff --git a/ada/stream_decode.adb b/ada/blkc_decode.adb similarity index 92% rename from ada/stream_decode.adb rename to ada/blkc_decode.adb index e75a531..014ae8c 100644 --- a/ada/stream_decode.adb +++ b/ada/blkc_decode.adb @@ -1,8 +1,8 @@ --- Decode a stream written by codec_main --stream or stream_encode into the file it came from, from the stream's own +-- Decode a stream written by codec_main --stream or blkc-encode into the file it came from, from the stream's own -- bytes, group by group, so memory does not grow with the audio's length. Frames of a group that cannot be read -- are written as zeros and reported. "-" names standard input or standard output. --skip and --until write frames -- from the first named up to the second as a WAV file, decoding from the groups those frames need. --- Usage: stream_decode [--skip ] [--until ] , or stream_decode --version +-- Usage: blkc-decode [--skip ] [--until ] , or blkc-decode --version -- Exit status: 0 decoded intact, 1 written with damage or with audio that differs from its MD5, 2 not a readable -- stream or wrong arguments. with Ada.Command_Line; @@ -18,12 +18,12 @@ with Stream_Format; use Stream_Format; with Stream_Reading; use Stream_Reading; with Wav_Files; -procedure Stream_Decode is +procedure Blkc_Decode is use Ada.Text_IO; procedure Fail (Message : String) is begin - Put_Line (Standard_Error, "stream_decode: " & Message); + Put_Line (Standard_Error, "blkc-decode: " & Message); Ada.Command_Line.Set_Exit_Status (2); end Fail; @@ -127,12 +127,12 @@ procedure Stream_Decode is begin if Ada.Command_Line.Argument_Count = 1 and then Ada.Command_Line.Argument (1) = "--version" then - Put_Line ("stream_decode " & Build_Version.Version & ", format version" & Integer'Image (Version) & ", C API version 1"); + Put_Line ("blkc-decode " & Build_Version.Version & ", format version" & Integer'Image (Version) & ", C API version 1"); return; end if; declare use Ada.Strings.Unbounded; - Usage : constant String := "usage: stream_decode [--skip ] [--until ] "; + Usage : constant String := "usage: blkc-decode [--skip ] [--until ] "; Names : array (1 .. 2) of Unbounded_String; Named : Natural := 0; Skip : Natural := 0; @@ -175,6 +175,6 @@ begin when Constraint_Error => Fail (Usage); end; -end Stream_Decode; +end Blkc_Decode; diff --git a/ada/stream_encode.adb b/ada/blkc_encode.adb similarity index 91% rename from ada/stream_encode.adb rename to ada/blkc_encode.adb index 27cf817..788f56c 100644 --- a/ada/stream_encode.adb +++ b/ada/blkc_encode.adb @@ -1,5 +1,5 @@ -- Code an audio file into a stream with the block codec. "-" names standard input or standard output. --- Usage: stream_encode [--level <0 to 5>] [--tag ]... [--seek-every ] [--independent] +-- Usage: blkc-encode [--level <0 to 5>] [--tag ]... [--seek-every ] [--independent] -- [--no-pairs] [--threads ] [--verify] -- --threads sets how many groups are coded at once, every processor when left out; the stream is the same. -- --verify decodes the stream before writing it and checks every frame against the audio's MD5. @@ -20,7 +20,7 @@ with Stream_Writing; use Stream_Writing; with System.Multiprocessors; with Wav_Files; -procedure Stream_Encode is +procedure Blkc_Encode is use Ada.Text_IO; use Ada.Streams; @@ -33,7 +33,7 @@ procedure Stream_Encode is procedure Fail (Message : String) is begin - Put_Line (Standard_Error, "stream_encode: " & Message); + Put_Line (Standard_Error, "blkc-encode: " & Message); Ada.Command_Line.Set_Exit_Status (2); end Fail; @@ -59,7 +59,7 @@ procedure Stream_Encode is end Value; begin if Ada.Command_Line.Argument_Count = 1 and then Ada.Command_Line.Argument (1) = "--version" then - Put_Line ("stream_encode " & Build_Version.Version & ", format version" & Integer'Image (Version) & ", C API version 1"); + Put_Line ("blkc-encode " & Build_Version.Version & ", format version" & Integer'Image (Version) & ", C API version 1"); return; end if; while I <= Ada.Command_Line.Argument_Count loop @@ -97,7 +97,7 @@ begin I := I + 1; end loop; if Bad or else Name_Count /= 2 then - Fail ("usage: stream_encode [--level <0 to 5>] [--tag ]... [--seek-every ] [--independent] " + Fail ("usage: blkc-encode [--level <0 to 5>] [--tag ]... [--seek-every ] [--independent] " & "[--no-pairs] [--threads ] [--verify] "); return; end if; @@ -154,7 +154,7 @@ begin end; end if; if not Ok then - Put_Line (Standard_Error, "stream_encode: " & In_Name & ": verification failed, the stream does not decode to " + Put_Line (Standard_Error, "blkc-encode: " & In_Name & ": verification failed, the stream does not decode to " & "the input; nothing written"); Ada.Command_Line.Set_Exit_Status (1); return; @@ -173,4 +173,4 @@ begin & Input.Rate'Image & " Hz, level" & Opt.Level'Image & "," & Length'Image & " bytes" & (if Verify then ", verified" else "")); end; -end Stream_Encode; +end Blkc_Encode; diff --git a/ada/fuzz_decode.adb b/ada/fuzz_decode.adb index 453f51e..680c1fd 100644 --- a/ada/fuzz_decode.adb +++ b/ada/fuzz_decode.adb @@ -1,5 +1,5 @@ -- Mutation fuzzer for the stream decoder. Each iteration takes one of the given streams, changes it at random and --- decodes it as stream_decode does: header, walker, audio MD5 and the writer, into /dev/null, then seeks to a frame +-- decodes it as blkc-decode does: header, walker, audio MD5 and the writer, into /dev/null, then seeks to a frame -- at random. Some changes rewrite a header or group field and recompute its CRC, so the change reaches the parsers -- behind the CRCs. Any exception is a failure: the changed stream is saved as fuzz_failure.blkc and the exit status -- is 1. With 0 iterations, each stream is decoded once as given, to replay a saved failure. @@ -173,7 +173,7 @@ procedure Fuzz_Decode is return S; end Mutate; - -- Decode as stream_decode does, into /dev/null. + -- Decode as blkc-decode does, into /dev/null. procedure Decode (S : Byte_Array_Access) is H : Header; begin diff --git a/ada/huffman_audio.gpr b/ada/huffman_audio.gpr index 27c1c0e..188da02 100644 --- a/ada/huffman_audio.gpr +++ b/ada/huffman_audio.gpr @@ -10,9 +10,14 @@ project Huffman_Audio is when "check" => for Object_Dir use "obj"; when "release" => for Object_Dir use "obj-release"; end case; - for Main use ("audio_huffman.adb", "codec_main.adb", "checksums_test.adb", "corrupt_test.adb", "stream_decode.adb", "float_bits_test.adb", "stream_encode.adb", + for Main use ("audio_huffman.adb", "codec_main.adb", "checksums_test.adb", "corrupt_test.adb", "blkc_decode.adb", "float_bits_test.adb", "blkc_encode.adb", "fuzz_decode.adb"); + package Builder is + for Executable ("blkc_encode.adb") use "blkc-encode"; + for Executable ("blkc_decode.adb") use "blkc-decode"; + end Builder; + package Compiler is case Build is when "check" => for Default_Switches ("Ada") use ("-O2", "-gnat2022", "-gnata", "-gnatwa"); diff --git a/docs/container.md b/docs/container.md index ca5c6d9..4d9ac3a 100644 --- a/docs/container.md +++ b/docs/container.md @@ -47,7 +47,7 @@ Metadata block 3 describes the input file so a decoder can write it back byte fo | 2 | audio: `data` or `SSND` | what precedes the samples, AIFF's offset and block size | | 3 | RF64's `ds64` size table | whole, its sizes rewritten on decode | -`stream_decode` writes the container and chunks back in order, the samples into the audio chunk, and every size field computed again. A size field that held 0 or all ones, as a program writing to a pipe leaves it, is written back as it was, so a file piped from ffmpeg also comes back byte for byte. +`blkc-decode` writes the container and chunks back in order, the samples into the audio chunk, and every size field computed again. A size field that held 0 or all ones, as a program writing to a pipe leaves it, is written back as it was, so a file piped from ffmpeg also comes back byte for byte. The decoder writes plain WAV when the stream has no description, or when the format chunk does not describe the decoded samples. Chunks other than format, audio and the size table are left out when all of them would not fit one 16 MB metadata block, and `codec_main` reports that. @@ -55,7 +55,7 @@ The decoder writes plain WAV when the stream has no description, or when the for The MD5 covers the samples as the input file held them, frames in order with channels interleaved. Each integer sample is shifted down to its valid bits and written little-endian in as many bytes as those bits need. Each float sample is its 32-bit or 64-bit pattern. This is FLAC's layout, so a FLAC file of the same audio records the same digest. `metaflac --show-md5sum` gives the stream's digest for every FLAC file of the corpus, and for a 20-bit file in 24-bit words. -Per-block CRCs show that coded bytes arrived intact. The MD5 shows that the decoder rebuilt the input: `stream_decode` checks the decoded audio against it and exits with status 1 on a difference. +Per-block CRCs show that coded bytes arrived intact. The MD5 shows that the decoder rebuilt the input: `blkc-decode` checks the decoded audio against it and exits with status 1 on a difference. Readers take version 2 only. Version 1 had a 32-bit frame count and no valid bits, channel mask or MD5. The header is 25 bytes longer for them. @@ -65,13 +65,13 @@ A seek to frame F decodes from the group that contains F, or from an earlier gro From the last seek point at or before F, the decoder skips group by group, reading each group's length, to the group that contains F. It then walks back while a block of the groups kept reads an earlier frame, and stops at the seek point at the latest. Each group decoded after F is checked the same way. A group that reads before the first frame decoded since the seek sends the decoder back to an earlier group. -`stream_decode --skip --until `, `blkc_seek` and `blkc_decode_range` seek this way. `codec_main --seek ` decodes from the seek point after the full decode and compares the samples from F to the end of that group with the input. +`blkc-decode --skip --until `, `blkc_seek` and `blkc_decode_range` seek this way. `codec_main --seek ` decodes from the seek point after the full decode and compares the samples from F to the end of that group with the input. ## Damage A decoder reads the header's CRC, then each group's. A group whose sync word, fields or CRC are wrong is skipped. The decoder scans forward for the next sync word whose group header passes its CRC, resumes at that group's first frame, and counts the frames between as lost. A block whose CRC fails is counted and decoded anyway. -`stream_decode ` decodes this way from the stream's bytes alone, group by group, writes lost frames as zeros and exits with status 1 when anything was lost. A header that claims more frames than its groups hold costs output length, not memory. A truncated stream decodes up to its last whole group. `codec_main --flip ` flips one bit of the written stream before decoding and reports what was lost. `corrupt_test ` checks the same on a written stream, with the group found again after a damaged length. +`blkc-decode ` decodes this way from the stream's bytes alone, group by group, writes lost frames as zeros and exits with status 1 when anything was lost. A header that claims more frames than its groups hold costs output length, not memory. A truncated stream decodes up to its last whole group. `codec_main --flip ` flips one bit of the written stream before decoding and reports what was lost. `corrupt_test ` checks the same on a written stream, with the group found again after a damaged length. Within a segment, blocks after a lost group that read the lost samples decode to wrong values without a CRC failure. The next segment is exact again. With every group a segment, damage is contained to the lost group. @@ -96,4 +96,4 @@ Segments, measured by `segments.sh` at level 4 on the corpus, where a block may A seek point every 10 s costs nothing worth measuring, because few blocks reach more than 16,384 samples back and fewer start a segment. Every group its own segment costs 1.8 percent, which is what the history buys: the delay predictor's reach into earlier samples and the adaptive stage's warm-up, both lost by the first blocks of every segment. A stream for random access or for links that drop data chooses between these by how much it must be able to lose at once. -Measured per file with `stream_encode` on the corpus and the wider corpus, that cost falls on generated signals. At level 4, every group a segment left each recording's stream the same size, one header byte apart, and made generated signals 5.3 percent larger. At level 5 it made recordings 0.95 percent larger, the adaptive stage's training lost at every group. A seek point every second, level 5's default, made recordings 0.076 percent and generated signals 0.44 percent larger than one every 10 s. +Measured per file with `blkc-encode` on the corpus and the wider corpus, that cost falls on generated signals. At level 4, every group a segment left each recording's stream the same size, one header byte apart, and made generated signals 5.3 percent larger. At level 5 it made recordings 0.95 percent larger, the adaptive stage's training lost at every group. A seek point every second, level 5's default, made recordings 0.076 percent and generated signals 0.44 percent larger than one every 10 s. diff --git a/docs/integrity.md b/docs/integrity.md index 20a4496..968a6a8 100644 --- a/docs/integrity.md +++ b/docs/integrity.md @@ -17,7 +17,7 @@ Blocks later in the same segment that read lost samples decode to wrong values w - `corrupt_test` on a written stream: every CRC matches as written, groups are contiguous and end at the file's end, and every seek point names a group. A flipped bit in the header fails the header CRC. A flipped bit in group 0's length fails that group's CRC, and group 1 is found again by its sync word and CRC at its own offset. A flipped bit in block 0 fails block 0's CRC and leaves block 1's and group 0's matching. Every check passes on a 40-group mono stream. - `codec_main rec_speech_8k.wav 1 --independent --flip 3 2` (a bit of group 3's length): 1 group failed, 4096 frames lost, 0 samples wrong outside them, and a seek to frame 100000 after the damage still matches. Without `--independent` the groups after it in the same 10 s segment read the lost samples and decode wrong until the next seek point. - Proof: the `Checksums` package proves at gnatprove level 2 with the decoder, no check unproved. -- `fuzz_decode` changes copies of the test vectors at random and decodes each as `stream_decode` does, with run-time checks and assertions on. Changes include header and group fields with their CRCs recomputed, so the change reaches the parsers. It found two faults that only damaged streams reach. A stereo mode rebuilt a channel outside its depth from damaged blocks, which handed later blocks history outside the proved decoder's precondition. A resync to a group at frame 0 asked to zero an empty range. Both are fixed, and their streams are regression vectors. +- `fuzz_decode` changes copies of the test vectors at random and decodes each as `blkc-decode` does, with run-time checks and assertions on. Changes include header and group fields with their CRCs recomputed, so the change reaches the parsers. It found two faults that only damaged streams reach. A stereo mode rebuilt a channel outside its depth from damaged blocks, which handed later blocks history outside the proved decoder's precondition. A resync to a group at frame 0 asked to zero an empty range. Both are fixed, and their streams are regression vectors. - After the fixes, 50,000 changed streams from one seed raised no exception: 6,345 refused, 4,294 intact, 39,305 decoded around damage, 56 skipped for claiming over 2^18 frames. ## Cost diff --git a/extra.sh b/extra.sh index 8b44e48..383f286 100755 --- a/extra.sh +++ b/extra.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Check the block codec on the wider corpus in corpus/extra: each file must come back byte for byte from -# stream_encode and stream_decode at levels 0, 2 and 4, and the Ada, Haskell and hybrid programs must write one +# blkc-encode and blkc-decode at levels 0, 2 and 4, and the Ada, Haskell and hybrid programs must write one # stream at level 4. Writes results/extra.tsv with each file's coded bytes beside flac -8 and wavpack -hh -x4. Needs # the Ada release programs, the Ada and Haskell builds of run.sh, flac, wavpack and ffprobe. set -uo pipefail @@ -22,8 +22,8 @@ for f in corpus/extra/*.flac; do pcm=$((duration_ts * channels * ((bits_per_sample + 7) / 8))) sizes=() for level in 0 2 4; do - ada/obj-release/stream_encode --level $level "$wav" "$work/s.bin" 2> /dev/null - if ada/obj-release/stream_decode "$work/s.bin" "$work/d.wav" 2> /dev/null && cmp -s "$wav" "$work/d.wav"; then + ada/obj-release/blkc-encode --level $level "$wav" "$work/s.bin" 2> /dev/null + if ada/obj-release/blkc-decode "$work/s.bin" "$work/d.wav" 2> /dev/null && cmp -s "$wav" "$work/d.wav"; then sizes+=("$(stat -c %s "$work/s.bin")") else echo "extra: $name level $level does not come back byte for byte" >&2 diff --git a/release.sh b/release.sh index 4c4f212..3da8c25 100755 --- a/release.sh +++ b/release.sh @@ -12,17 +12,17 @@ trap 'rm -rf "$work"' EXIT mkdir -p "$work/version" printf -- '-- Written by release.sh.\npackage Build_Version is\n Version : constant String := "%s";\nend Build_Version;\n' \ "$version" > "$work/version/build_version.ads" -if ! (cd ada && gprbuild -q -P huffman_audio.gpr -XBUILD=release -XVERSION_SOURCE="$work/version" stream_encode \ - stream_decode && gprbuild -q -P codec_lib.gpr) > "$work/build.log" 2>&1; then +if ! (cd ada && gprbuild -q -P huffman_audio.gpr -XBUILD=release -XVERSION_SOURCE="$work/version" blkc_encode \ + blkc_decode && gprbuild -q -P codec_lib.gpr) > "$work/build.log" 2>&1; then cat "$work/build.log" >&2 exit 1 fi root=$work/$name-$version mkdir -p "$root/bin" "$root/lib" "$root/include" "$root/share/doc/$name" -install -m 755 ada/obj-release/stream_encode ada/obj-release/stream_decode "$root/bin/" +install -m 755 ada/obj-release/blkc-encode ada/obj-release/blkc-decode "$root/bin/" install -m 755 ada/lib/libblockcodec.so "$root/lib/" -strip "$root/bin/stream_encode" "$root/bin/stream_decode" +strip "$root/bin/blkc-encode" "$root/bin/blkc-decode" strip --strip-unneeded "$root/lib/libblockcodec.so" install -m 644 include/blockcodec.h "$root/include/" cp README.md docs/spec.md LICENSE LICENSING.md "$root/share/doc/$name/" diff --git a/seek_timing.sh b/seek_timing.sh index 840ec03..96c3ed5 100755 --- a/seek_timing.sh +++ b/seek_timing.sh @@ -4,7 +4,7 @@ # then seeks to 200 frames drawn from one seed and reads 4096 frames from each; the median, 90th percentile and # largest times go to results/seek_timing.tsv with a check sum that must equal FLAC's. Runs on one pinned processor # and prints the load average, since a seek competes with the rest of the machine. Needs flac, wavpack, ffmpeg, the -# FLAC and WavPack development headers, the library (gprbuild -P ada/codec_lib.gpr) and the release stream_encode. +# FLAC and WavPack development headers, the library (gprbuild -P ada/codec_lib.gpr) and the release blkc-encode. set -uo pipefail cd "$(dirname "$0")" work=$(mktemp -d -p "${TMPDIR:-/var/tmp}") @@ -24,7 +24,7 @@ for name in joined_48k vinyl_192k; do wav="$work/$name.wav" flac -s -f -5 -o "$work/a.flac" "$wav"; flac -s -f -8 -o "$work/b.flac" "$wav" wavpack -q -y "$wav" -o "$work/c.wv"; wavpack -q -y -hh "$wav" -o "$work/d.wv" - for level in 2 4 5; do ada/obj-release/stream_encode --level "$level" "$wav" "$work/l$level.blkc" 2> /dev/null; done + for level in 2 4 5; do ada/obj-release/blkc-encode --level "$level" "$wav" "$work/l$level.blkc" 2> /dev/null; done reference= for m in "flac -5:a.flac" "flac -8:b.flac" "wavpack:c.wv" "wavpack -hh:d.wv" "block level 2:l2.blkc" \ "block level 4:l4.blkc" "block level 5:l5.blkc"; do diff --git a/test.do b/test.do index 05bdc99..07585f7 100755 --- a/test.do +++ b/test.do @@ -12,7 +12,7 @@ FAIL=0 echo "== 1/10 Build" (cd ada && gprbuild -q -P huffman_audio.gpr && gprbuild -q -P codec_lib.gpr) || { echo " Ada build failed"; exit 1; } -(cd ada && gprbuild -q -P huffman_audio.gpr -XBUILD=release stream_encode stream_decode) \ +(cd ada && gprbuild -q -P huffman_audio.gpr -XBUILD=release blkc_encode blkc_decode) \ || { echo " Ada release build failed"; exit 1; } (cd haskell && stack build 2>&1 | grep -E "rror" && exit 1; true) || { echo " Haskell build failed"; exit 1; } echo " built" @@ -36,21 +36,21 @@ echo "== 3/10 Stream decoder" ada/obj/codec_main corpus/depth_pink_32f.wav 1 --stream build/pink_32f.bin > /dev/null || exit 1 for s in speech_8k:build/wav/rec_speech_8k.wav pink_32f:corpus/depth_pink_32f.wav; do name=${s%%:*}; wav=${s#*:} - if ada/obj/stream_decode "build/$name.bin" "build/${name}_decoded.wav" 2>&1 | grep -q "audio MD5 matches" \ + if ada/obj/blkc-decode "build/$name.bin" "build/${name}_decoded.wav" 2>&1 | grep -q "audio MD5 matches" \ && cmp -s "$wav" "build/${name}_decoded.wav"; then echo " $name: decoded from the stream alone, byte-identical file, audio MD5 checked" else - echo " $name: stream_decode FAILED"; FAIL=1 + echo " $name: blkc-decode FAILED"; FAIL=1 fi done -ada/obj/stream_encode --level 1 --tag title=speech build/wav/rec_speech_8k.wav build/speech_8k_tool.bin 2> /dev/null -cmp -s build/speech_8k.bin build/speech_8k_tool.bin && echo " stream_encode writes codec_main's stream" \ - || { echo " stream_encode: FAILED"; FAIL=1; } +ada/obj/blkc-encode --level 1 --tag title=speech build/wav/rec_speech_8k.wav build/speech_8k_tool.bin 2> /dev/null +cmp -s build/speech_8k.bin build/speech_8k_tool.bin && echo " blkc-encode writes codec_main's stream" \ + || { echo " blkc-encode: FAILED"; FAIL=1; } # The release programs, built without run-time checks, write the same stream and the same file, and --verify passes. -ada/obj-release/stream_encode --level 4 --verify build/wav/rec_speech_8k.wav build/speech_8k_release.bin 2> /dev/null \ - && ada/obj/stream_encode --level 4 build/wav/rec_speech_8k.wav build/speech_8k_checked.bin 2> /dev/null \ +ada/obj-release/blkc-encode --level 4 --verify build/wav/rec_speech_8k.wav build/speech_8k_release.bin 2> /dev/null \ + && ada/obj/blkc-encode --level 4 build/wav/rec_speech_8k.wav build/speech_8k_checked.bin 2> /dev/null \ && cmp -s build/speech_8k_release.bin build/speech_8k_checked.bin \ - && ada/obj-release/stream_decode build/speech_8k_release.bin build/speech_8k_release.wav 2> /dev/null \ + && ada/obj-release/blkc-decode build/speech_8k_release.bin build/speech_8k_release.wav 2> /dev/null \ && cmp -s build/wav/rec_speech_8k.wav build/speech_8k_release.wav \ && echo " release programs: same stream, verified, byte-identical file" \ || { echo " release programs: FAILED"; FAIL=1; } @@ -76,22 +76,22 @@ b[26] ^= 1 b[pos:pos + 2] = crc16(b[:pos]).to_bytes(2, 'big') open(sys.argv[2], 'wb').write(b) EOF -out=$(ada/obj/stream_decode build/wrong_md5.bin build/wrong_md5.wav 2>&1); st=$? +out=$(ada/obj/blkc-decode build/wrong_md5.bin build/wrong_md5.wav 2>&1); st=$? [ $st = 1 ] && grep -q "AUDIO MD5 DIFFERS" <<< "$out" && echo " changed audio MD5: reported" \ || { echo " changed audio MD5: not reported"; FAIL=1; } head -c 60000 build/speech_8k.bin > build/truncated.bin -ada/obj/stream_decode build/truncated.bin build/truncated.wav 2> /dev/null +ada/obj/blkc-decode build/truncated.bin build/truncated.wav 2> /dev/null [ $? = 1 ] && echo " truncated stream: damage status" || { echo " truncated stream: wrong status"; FAIL=1; } head -c 3000 build/wav/rec_speech_8k.wav > build/not_a_stream.bin -ada/obj/stream_decode build/not_a_stream.bin build/not_a_stream.wav 2> /dev/null +ada/obj/blkc-decode build/not_a_stream.bin build/not_a_stream.wav 2> /dev/null [ $? = 2 ] && echo " foreign file: refused" || { echo " foreign file: wrong status"; FAIL=1; } # --skip and --until write the same frames as a full decode, here on streams whose blocks read earlier groups. for v in delay_in_history:0:1:5000:9000:6208:10000:19000:20000 adaptive_stage:0:4096:4095:8193:30000:34000:44000:44100; do name=${v%%:*}; ranges=${v#*:}; ok=1 - ada/obj/stream_decode "vectors/$name.blkc" "build/${name}_full.wav" 2> /dev/null || ok=0 + ada/obj/blkc-decode "vectors/$name.blkc" "build/${name}_full.wav" 2> /dev/null || ok=0 while [ -n "$ranges" ]; do skip=${ranges%%:*}; ranges=${ranges#*:}; until=${ranges%%:*}; ranges=${ranges#"$until"}; ranges=${ranges#:} - ada/obj/stream_decode --skip "$skip" --until "$until" "vectors/$name.blkc" "build/${name}_range.wav" 2> /dev/null || ok=0 + ada/obj/blkc-decode --skip "$skip" --until "$until" "vectors/$name.blkc" "build/${name}_range.wav" 2> /dev/null || ok=0 python3 -I - "build/${name}_full.wav" "build/${name}_range.wav" "$skip" "$until" <<'EOF' || ok=0 import sys def pcm(path): @@ -130,7 +130,7 @@ for s in build/wav/rec_speech_8k.wav:build/speech_8k_rf64.wav build/wav/rec_spee corpus/depth_pink_32f.wav:build/pink_32f.aifc build/wav/rec_speech_8k.wav:build/speech_8k_piped.wav; do wav=${s%%:*}; file=${s#*:}; name=$(basename "$file") a=$(audio_md5 "$wav" build/source.bin); b=$(audio_md5 "$file" build/container.bin) - if [ -n "$a" ] && [ "$a" = "$b" ] && ada/obj/stream_decode build/container.bin "build/back_$name" 2> /dev/null \ + if [ -n "$a" ] && [ "$a" = "$b" ] && ada/obj/blkc-decode build/container.bin "build/back_$name" 2> /dev/null \ && cmp -s "$file" "build/back_$name"; then echo " $name: WAV's audio MD5, byte-identical after decoding" else @@ -144,7 +144,7 @@ ffmpeg -v error -y -i corpus/depth_pink_32f.wav -c:a pcm_f64be -f aiff build/pin for file in build/pink_64f.wav build/pink_64f.aifc; do name=$(basename "$file") if ada/obj/codec_main "$file" 0 --stream build/float64.bin | grep -q 'round trip OK' \ - && ada/obj/stream_decode build/float64.bin "build/back_$name" 2> /dev/null && cmp -s "$file" "build/back_$name"; then + && ada/obj/blkc-decode build/float64.bin "build/back_$name" 2> /dev/null && cmp -s "$file" "build/back_$name"; then echo " $name: 64-bit float, byte-identical after decoding" else echo " $name: FAILED"; FAIL=1 @@ -222,16 +222,16 @@ ha=$(grep -oE 'hash [0-9a-f]+' <<< "$a"); hh=$(grep -oE 'hash [0-9a-f]+' <<< "$h if [ "$ok" = 3 ] && [ "$ha" = "$hh" ] && [ "$ha" = "$hy" ]; then echo " four channels, level 1: three round trips, one stream ($(awk '{print $6}' <<< "$a") bits per sample)" ada/obj/codec_main build/four_48k.wav 1 --stream build/four_48k.bin > /dev/null - if ada/obj/stream_decode build/four_48k.bin build/four_decoded.wav 2> /dev/null && cmp -s build/four_48k.wav build/four_decoded.wav; then + if ada/obj/blkc-decode build/four_48k.bin build/four_decoded.wav 2> /dev/null && cmp -s build/four_48k.wav build/four_decoded.wav; then echo " four channels: decoded from the stream alone, byte-identical file" else - echo " four channels: stream_decode FAILED"; FAIL=1 + echo " four channels: blkc-decode FAILED"; FAIL=1 fi - ada/obj/stream_encode --level 1 - - < build/four_48k.wav 2> /dev/null | ada/obj/stream_decode - - 2> /dev/null > build/four_piped.wav + ada/obj/blkc-encode --level 1 - - < build/four_48k.wav 2> /dev/null | ada/obj/blkc-decode - - 2> /dev/null > build/four_piped.wav cmp -s build/four_48k.wav build/four_piped.wav && echo " through pipes: four channels byte-identical" \ || { echo " through pipes: FAILED"; FAIL=1; } - ada/obj/stream_encode --level 2 --threads 1 build/four_48k.wav build/four_t1.bin 2> /dev/null - ada/obj/stream_encode --level 2 --threads 3 build/four_48k.wav build/four_t3.bin 2> /dev/null + ada/obj/blkc-encode --level 2 --threads 1 build/four_48k.wav build/four_t1.bin 2> /dev/null + ada/obj/blkc-encode --level 2 --threads 3 build/four_48k.wav build/four_t3.bin 2> /dev/null cmp -s build/four_t1.bin build/four_t3.bin && echo " one thread and three: same stream" \ || { echo " one thread and three: FAILED"; FAIL=1; } else @@ -240,8 +240,8 @@ else fi echo "== 7/10 Test vectors" -vectors/check.sh ada/obj/stream_decode || FAIL=1 -vectors/check.sh ada/obj-release/stream_decode > /dev/null || { echo " test vectors: release decoder FAILED"; FAIL=1; } +vectors/check.sh ada/obj/blkc-decode || FAIL=1 +vectors/check.sh ada/obj-release/blkc-decode > /dev/null || { echo " test vectors: release decoder FAILED"; FAIL=1; } echo "== 8/10 Fuzzing" # Changed copies of every vector through the decoder with checks on; a failing stream is kept as build/fuzz_failure.blkc. @@ -252,7 +252,7 @@ failed=0 differ=0 for seed in $(seq 1 20); do python3 -I vectors/edge_audio.py "$seed" build/edge.wav > /dev/null for level in 0 2 4; do - ada/obj/stream_encode --level $level --verify build/edge.wav build/edge.bin 2> /dev/null || failed=$((failed + 1)) + ada/obj/blkc-encode --level $level --verify build/edge.wav build/edge.bin 2> /dev/null || failed=$((failed + 1)) done if [ "$seed" -le 8 ]; then a=$(ada/obj/codec_main build/edge.wav 2 | grep -oE 'round trip OK, hash [0-9a-f]+') @@ -276,9 +276,9 @@ tarball=$(bash release.sh build/dist 2> /dev/null | tail -1) version=$(git describe --tags --match 'v*' --abbrev=8 --always --dirty) mkdir -p build/dist/unpacked && tar -xzf "$tarball" -C build/dist/unpacked root=$(echo build/dist/unpacked/*) -if [ -x "$root/bin/stream_encode" ] && [ -f "$root/lib/libblockcodec.so" ] && [ -f "$root/include/blockcodec.h" ] \ +if [ -x "$root/bin/blkc-encode" ] && [ -f "$root/lib/libblockcodec.so" ] && [ -f "$root/include/blockcodec.h" ] \ && [ -f "$root/share/doc/blockcodec/spec.md" ] && (cd build/dist && sha256sum -c --quiet ./*.sha256) \ - && "$root/bin/stream_decode" --version | grep -q "^stream_decode $version, format version 2, C API version 1$"; then + && "$root/bin/blkc-decode" --version | grep -q "^blkc-decode $version, format version 2, C API version 1$"; then echo " release package: programs, library, header and documents, version $version" else echo " release package: FAILED"; FAIL=1 diff --git a/tool_timing.sh b/tool_timing.sh index 07fd890..4a08beb 100755 --- a/tool_timing.sh +++ b/tool_timing.sh @@ -29,14 +29,14 @@ for wav in build/wav/*.wav; do flac*) coded="$work/c.flac"; $method -s -f -o "$coded" "$wav" 2> /dev/null ;; wavpack*) coded="$work/c.wv"; $method -q -y "$wav" -o "$coded" 2> /dev/null ;; block*parallel) coded="$work/c.bin"; level=${method#block level } - ada/obj-release/stream_encode --level "${level%% *}" "$wav" "$coded" 2> /dev/null ;; - block*) coded="$work/c.bin"; ada/obj-release/stream_encode --threads 1 --level "${method##* }" "$wav" "$coded" 2> /dev/null ;; + ada/obj-release/blkc-encode --level "${level%% *}" "$wav" "$coded" 2> /dev/null ;; + block*) coded="$work/c.bin"; ada/obj-release/blkc-encode --threads 1 --level "${method##* }" "$wav" "$coded" 2> /dev/null ;; esac t1=$(now) case $method in flac*) flac -s -d -f -o "$work/d.wav" "$coded" 2> /dev/null ;; wavpack*) wvunpack -q -y "$coded" -o "$work/d.wav" 2> /dev/null ;; - block*) ada/obj-release/stream_decode "$coded" "$work/d.wav" 2> /dev/null ;; + block*) ada/obj-release/blkc-decode "$coded" "$work/d.wav" 2> /dev/null ;; esac t2=$(now) if [[ -s $coded && -s $work/d.wav ]] \ diff --git a/vectors/README.md b/vectors/README.md index 85b2880..2720d04 100644 --- a/vectors/README.md +++ b/vectors/README.md @@ -1,6 +1,6 @@ # Test vectors -Streams in format version 2, as [docs/spec.md](../docs/spec.md) defines it, with what a decoder writes from each. `MANIFEST.tsv` lists every vector: its name, the exit status of `stream_decode`, the MD5 of the file it writes, and what the vector covers. For intact vectors that includes the block kinds and stereo modes `codec_main` reported when the vector was written. +Streams in format version 2, as [docs/spec.md](../docs/spec.md) defines it, with what a decoder writes from each. `MANIFEST.tsv` lists every vector: its name, the exit status of `blkc-decode`, the MD5 of the file it writes, and what the vector covers. For intact vectors that includes the block kinds and stereo modes `codec_main` reported when the vector was written. - Intact vectors decode with status 0 to their input file, byte for byte. - Damaged vectors are copies of `split_tree.blkc` with one change each. Status 1 means damage was found and decoded around, with lost frames written as zeros. Status 2 means the stream was refused. @@ -8,6 +8,6 @@ Streams in format version 2, as [docs/spec.md](../docs/spec.md) defines it, with To check a decoder that takes a stream and an output file: - vectors/check.sh [decoder] # ada/obj/stream_decode when none is named + vectors/check.sh [decoder] # ada/obj/blkc-decode when none is named `make_vectors.sh` wrote the vectors from generated signals and short corpus excerpts, and checked that each intact vector decodes to its input. The committed streams are the reference: an encoder that searches differently writes other streams that decode to the same audio. diff --git a/vectors/check.sh b/vectors/check.sh index 2cfaa54..ebb5d6b 100755 --- a/vectors/check.sh +++ b/vectors/check.sh @@ -1,9 +1,9 @@ #!/usr/bin/env bash -# Decode every test vector with a decoder (ada/obj/stream_decode when none is named) and compare its exit status and +# Decode every test vector with a decoder (ada/obj/blkc-decode when none is named) and compare its exit status and # the MD5 of the file it writes with MANIFEST.tsv. The decoder takes . Exit status 1 on any # difference. set -uo pipefail -dec=$(realpath "${1:-$(dirname "$0")/../ada/obj/stream_decode}") +dec=$(realpath "${1:-$(dirname "$0")/../ada/obj/blkc-decode}") cd "$(dirname "$0")" || exit 1 work=$(mktemp -d -p "${TMPDIR:-/var/tmp}") trap 'rm -rf "$work"' EXIT diff --git a/vectors/make_vectors.sh b/vectors/make_vectors.sh index 277c17e..18273a3 100755 --- a/vectors/make_vectors.sh +++ b/vectors/make_vectors.sh @@ -2,7 +2,7 @@ # Write the format version 2 test vectors into vectors/: streams that together use every block kind, residual code, # stereo mode, depth, float mode, container and metadata block, and damaged copies of one stream. Regression streams # the fuzzer found stay as found. MANIFEST.tsv lists each stream with the exit status and the MD5 of the file that -# ada/obj/stream_decode writes from it, and the block kinds codec_main reports for the intact ones. Sources are +# ada/obj/blkc-decode writes from it, and the block kinds codec_main reports for the intact ones. Sources are # generated signals and short corpus excerpts. Needs the Ada programs built, ffmpeg and python3. The vectors are kept # as written: rerun this only to add vectors. set -euo pipefail @@ -10,8 +10,8 @@ cd "$(dirname "$0")/.." out=vectors work=$(mktemp -d -p "${TMPDIR:-/var/tmp}") trap 'rm -rf "$work"' EXIT -enc=ada/obj/stream_encode -dec=ada/obj/stream_decode +enc=ada/obj/blkc-encode +dec=ada/obj/blkc-decode mkdir -p "$out" find "$out" -name '*.blkc' ! -name 'regression_*' -delete manifest=$out/MANIFEST.tsv