Skip to content

Built-in engines W24: Word/ODT → TXT, XPS and OpenXPS, XLSB, PDF nesting - #1699

Merged
KnifeLemon merged 20 commits into
builtin-enginesfrom
p4/w24
Nov 19, 2026
Merged

KnifeLemon merged 20 commits into
builtin-enginesfrom
p4/w24

Conversation

@KnifeLemon

Copy link
Copy Markdown
Owner

Wave 24 of the built-in engines. Built on top of #1692 (archive speed round 3 and .zst), so merge that first.

Word, ODT and DOC → TXT without PDF

  • With the default settings (renderer on) DOCX, DOCM, DOTX, DOTM, DOC, DOT, WPS and ODT went to TXT through PDF text, like RTF did before Built-in engines W23: Word floating tables, UYAP UDF, RTF → TXT, Excel short date #1693: a blank line after every paragraph, leading spaces, tabs and double spaces lost.
  • They now have a direct built-in edge (cost 24) that writes one line per paragraph with PlainTextWriter.WriteLines, the same as RTF. Lists get 1. / • , table rows are tab separated.
  • Text in a table nested inside a table cell used to be dropped; it is now joined to the cell's line (HWP → TXT too).
  • LibreOffice moved to the top of the engine order still takes these conversions. Word 6.0/95 DOC files still go another way (checked edge).
  • XLSX writer: a date in the PC's short date pattern is written as built-in number format 14, so Excel shows it in the reader's own locale ([Bug]:excel转pdf的图片问题 #1315).

XPS and OpenXPS (#613)

  • New reader for XPS 1.0 and OpenXPS (ECMA-388), both namespaces: interleaved parts, Canvas / Path / Glyphs with transforms, clips, opacity and opacity masks, abbreviated path syntax and PathGeometry, solid, gradient, image and visual brushes with tiling, resources and AlternateContent, obfuscated fonts (.odttf).
  • Pages go through the drawing engine (DrawingFormat.ReadPages), so PDF uses PdfPageWriter with font subsets and PNG / SVG the existing writers.
  • JPEG XR and broken or oversized pictures are left out and counted in the left-out notice (notice.renderer_left_out.pictures, 23 languages).
  • Limits for pages, nesting (64), elements, path points and data length, dashes, page size and picture pixels; hostile-input tests and XpsFuzzTests.

XLSB (#1535)

  • New reader for Excel binary workbooks from [MS-XLSB]: workbook, shared strings, styles and worksheets into the same sheet model as XLSX, so every sheet output works. Formulas give their cached values, like XLSX and XLS.
  • A misnamed XLSX or XLS keeps its own reader; encrypted files ask for the password to be removed.
  • Records counted like XML elements (5 million), parsed records up to 1 MB, fields read inside their record, row / column ranges as for XLSX.
  • Fuzzing found an overflow on huge column widths in SheetDocument (XLSX had it too); widths are now capped at 255 characters.
  • Checked only against XLSX twins built in code: no Excel-saved .xlsb yet.

Safety

  • PDF objects nested deeper than 256 levels are refused with the usual "too complex" error before PDFsharp opens the file. Object streams are checked too.
  • DIB pictures decode straight into the bitmap: an 8192² picture no longer allocates 512 MB of managed copies.
  • Metafiles drawn for pictures get the conversion's cancellation token.

Settings

Tests

  • Merged branch: Filee.Core.Tests 1780 passed; Engines filter (XPS, XLSB, drawings, renderer, built-in text, PDF nesting, metafiles, sheets, archives in process) 512 passed, 15 skipped; App localization and left-out 261 passed. dotnet format --severity warn clean on every touched project.

🤖 Generated with Claude Code

KnifeLemon and others added 19 commits November 19, 2026 13:40
ZST and TAR.ZST become archive formats like XZ and TAR.XZ: recognised by
extension, unpacked in-process (the TAR in a second stage), converted to
every archive target, and ZST to the other single-file formats. ZstdReader
puts SharpCompress's decoder behind Filee's checks: the first frame's magic
(other data is "not an archive", not an empty file), frames one after
another and skippable frames, damage and early ends as "damaged", windows
over 128 MiB left to 7-Zip. RPM payloads in zstd use the same reader.

Read only: writing would add a developer-only output, which FORMAT-RULES
rule 5 keeps out while the developer field is on by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- LZMA decoder: a match's source is asked into the cache (prefetch hint,
  x86) when the match is decoded, so it arrives while the next symbols are
  decoded and before the deferred copy; the three older repeat distances
  live in fields, which leaves registers to the coder. Same bytes.
- Hash-chain search (Fast): a search of its own with its values in
  registers and one two-byte check per candidate (same matches), a 10-bit
  hash for the two-byte table (4 KB instead of 256 KB per thread), and the
  next position's table slots prefetched. Output stays deterministic for
  any number of threads; Normal and Maximum are unchanged.
- bzip2: the move-to-front shift moves 16 bytes at a time (decoder and
  encoder; the decoder's symbol thread sets the pace), and the transform
  writes single rows without a span fill. Same bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The converter took the main stream byte by byte through a buffered reader;
it now scans its own buffer for the next E8 / E9 / 0F 8x opcode and copies
the run before it at once. Same bytes, same checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source that wraps around the window's end is hinted only up to it, so no
reference past the array is ever formed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…chives-3

# Conflicts:
#	tests/Filee.Core.Tests/FormatFieldsTests.cs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-in route

With the Filee renderer on, DOCX (and DOCM, DOTX, DOTM), DOC (DOT, WPS) and
ODT → TXT went through the renderer's PDF and the PDF's text, which put an
empty line between all paragraphs and dropped leading spaces, tabs, runs of
spaces and empty paragraphs. They now have direct built-in edges in the HWP
text engine, written by PlainTextWriter.WriteLines like RTF → TXT: a line per
paragraph, list numbers and bullets, table rows as tab-separated cells.

The edges cost 24: cheaper than the PDF route and the routes through Pandoc
or the HWPX writer, dearer than LibreOffice moved to the top of the engine
list, which keeps these files there. DOC, DOT and WPS edges are checked, so a
Word 6.0 / 95 file goes another way. A table inside a table cell now keeps
its text in the cell's line instead of being left out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1315)

Format 14 is read as the PC's short date. The XLSX writer wrote that code back
as a format of the file's own, so XLS / ODS / CSV → XLSX fixed the date to the
converting PC's order. A date in the PC's short date is now written as format
14 and Excel shows it in the short date of whoever opens the file. Where the
PC's short date can't be an Excel code (other calendars), the ISO date read
for 14 stays a format of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PDFsharp's object parser reads arrays and dictionaries recursively and
reads every object (object streams included) when it opens a file, so a
few thousand nested brackets overflowed the stack. Every PDFsharp open of
an input now goes through PdfObjectNesting, which reads each object from
its "obj" / "trailer" keyword and each object-stream entry from its
offset, and refuses nesting deeper than 256 levels with
error.document_too_complex. The bytes it reads are capped at a few times
the file.

Tests build the hostile PDFs in code; a fuzz target covers the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dib decoded pixels into a managed array, copied it, and copied it again
into the Skia bitmap; it now checks the pixel data against the header
first and decodes straight into the bitmap. ImageReader passes the
conversion's cancellation token to the metafile renderer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sheet whose file claims a column millions of characters wide made the
table width overflow while it was written as a HWPX / DOCX table. Widths
are now held to Excel's widest column (255 characters) there. Found by
fuzzing the new XLSB reader; an XLSX with such a width hit it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XLSB is the XLSX package with the workbook, shared strings, styles and
sheets stored as BIFF12 records instead of XML. The new built-in XLSB
reader, written from Microsoft's published [MS-XLSB], fills the same
sheet model as the XLSX reader, so every sheet output works: XLSX, ODS,
CSV, TSV through the spreadsheet engine, PDF through the sheet printer,
HWPX, DOCX, ODT and HTML through the document writers.

Read: visible sheets, the 1904 date system, print areas and titles,
shared strings with rich text, number formats, fonts, fills (solid,
patterns, gradients), borders, alignment, numbers (RK and real), text,
booleans, errors, formulas as their cached values, widths, heights,
hidden rows and columns, merges, frozen rows, right-to-left sheets,
page breaks, page setup, margins, print options, headers and footers,
and the pictures and shapes of each sheet's drawing. The printer names
charts and other objects it doesn't draw as for XLSX, and LibreOffice
(when installed) takes such a workbook. An XLSX or XLS with an .xlsb
name is read as what it is; an encrypted one asks for the password to
be removed.

The input is read within DocumentLimits: part and package sizes and
the inflate ratio while unpacking, records counted like XML elements,
a record the reader takes apart holds at most 1 MB, every field is read
within its record, and rows, columns and column ranges are bounded as
in the XLSX reader. Tests build each workbook twice in code (XLSB and
XLSX) and compare what the two readers read; hostile records are
refused quickly, and a fuzz target mutates the records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sb, #1535)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Filee's own XPS reader (XPS 1.0 and ECMA-388 OpenXPS, both namespaces)
draws .xps and .oxps pages through the drawing engine, page by page:
the fixed document sequence and its documents (interleaved pieces
joined, part names resolved inside the package), Canvas, Path and Glyphs
with RenderTransform, Clip, Opacity, OpacityMask, links and link targets,
the abbreviated path syntax and PathGeometry, solid, linear and radial
gradient, image (PNG, JPEG, GIF, BMP, TIFF, sized by their resolution)
and visual brushes with tiling, static resources (inline and remote
dictionaries) and AlternateContent. Glyphs use the embedded fonts, with
obfuscated .odttf fonts restored and checked by Filee's SFNT parser;
Indices, cluster maps, BidiLevel and style simulations are applied.
Plain glyph runs go to the page model, so the PDF writer subsets their
fonts and maps their text.

JPEG XR pictures and damaged or oversized pictures are left out and
named in the file's result (notice.renderer_left_out.pictures, all
locales). The drawing engine's description names XPS in every locale.

Reading is bounded: package limits, pages, canvas and visual brush
nesting, elements, path and glyph points, path data length, dash
patterns, page size and picture pixels. A hostile-input test and a fuzz
target cover the reader and the drawing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…613)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	src/Filee.Core/Settings/AppSettings.cs
#	src/Filee.Core/Settings/UserDataStore.cs
#	tests/Filee.Core.Tests/FormatFieldsTests.cs
# Conflicts:
#	README.md
#	src/Filee.Core/Formats/FormatFields.cs
#	src/Filee.Core/Settings/AppSettings.cs
#	src/Filee.Core/Settings/UserDataStore.cs
#	tests/Filee.Core.Tests/FormatFieldsTests.cs
@KnifeLemon KnifeLemon added the built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet label Nov 19, 2026
…gine test crashes on Unix

The amount and date columns were 12.5 characters wide; with the wider CJK
fallback font on Linux the printed cells came out as ######.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon
KnifeLemon merged commit 1501ac0 into builtin-engines Nov 19, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant