Skip to content

Built-in engines W25: charts, document tables to XLSX, Markdown math, archives round 4 - #1708

Merged
KnifeLemon merged 12 commits into
builtin-enginesfrom
p4/w25
Nov 19, 2026
Merged

KnifeLemon merged 12 commits into
builtin-enginesfrom
p4/w25

Conversation

@KnifeLemon

Copy link
Copy Markdown
Owner

Wave 25 of the built-in engines: DrawingML charts, document tables to spreadsheets for every document format, Markdown math, and archive speed round 4. Also brings in main's CI fix for the FFmpeg install.

Charts

  • New chart reader and drawer (src/Filee.Engines/Charts/). Charts are drawn from the cached values in the chart part, so no formulas are evaluated.
  • Types drawn:
    • bar and column (clustered, stacked, 100 %, horizontal)
    • line (markers, smooth, stacked)
    • area, pie, doughnut and scatter
    • combinations on two pairs of axes
  • Parts drawn: titles, axes, gridlines, legend, data labels and manual layouts. Series colours come from the chart part or from the theme's accents. Text stays text in PDF.
  • Used by:
    • the sheet printer (XLSX, XLSB): a workbook whose charts are all drawn stays on the built-in printer
    • the Word renderer
    • the slide renderer
  • What's still left out: 3D, radar, bubble, stock and surface charts, chartex types, and chart sheets. These are still named as left out, or fall back to a stored picture.
  • Limits on series, points and per-document points, refused with error.document_too_complex.
  • New chart fuzz target. It found an XLSX font size of 1e-308 that crashed the sheet printer; font sizes are now clamped to Excel's 1–409.

Document tables → XLSX / ODS / CSV (#1016)

  • DOC, DOT, WPS, ODT, RTF, HWP and HWPX tables convert with the same rules as DOCX. Before, these formats had no route at all.
  • HWP/HWPX are read the way the renderer reads them, so a number typed as an equation becomes a number.
  • Cells keep italic and centred or right alignment.
  • Rows under a merged cell get the height their text needs.
  • Limits on sheets, rows and total cells. Nested tables are walked without recursion.

Markdown math (#1652)

  • $…$, $$…$$, \(…\) and \[…\] follow Pandoc's dollar rules. Our own TeX reader covers fractions, roots, scripts and primes, big operators with limits, \left…\right, matrices, cases, aligned, accents, \text and font commands. Unknown commands stay as written.
  • Output by format:
    • PDF/PNG: typeset by the existing equation layout
    • DOCX: Word math
    • HTML and EPUB: MathML, with the TeX kept as an annotation
    • HWPX: 한글 equations
  • Equations from DOCX/HWP/HWPX now also come out as Word math and MathML instead of linear text.
  • Primes are laid out after their base at full size.
  • Length and nesting limits; flood refused through DocumentLimits.MaxObjects. New fuzz target.

Archive speed round 4

  • bzip2 suffix sorting rewritten (SA-IS without the type array): sorting 13–15 % faster.
  • xz CRC-64 folds 16 bytes at a time with PCLMULQDQ on x64; other CPUs keep the table path.
  • Prefetch hint for the Fast hash chain.
  • Output bytes unchanged. Profiling shows the remaining gap on short jobs is mostly .NET tiering at first run; that is the next round.

CI

portable.yml already passes --blame-crash to the engine tests (from #1699).

Tests

Run locally on the merged branch:

  • Release build
  • Filtered engine tests (charts, math, tables, sheets, slides, DOCX, XLSB, XPS, archives, hostile input): 958 passed, 43 skipped, 0 failed
  • Core: 1780 passed
  • App localization and left-out tests: 261 passed
  • dotnet format --verify-no-changes --severity warn clean on Filee.Engines and Filee.Engines.Tests

🤖 Generated with Claude Code

KnifeLemon and others added 12 commits November 19, 2026 11:03
… and retries

The metadata job hit its 30-minute limit on #1663, #1669, #1671, #1672 and #1679: apt took 23-30 minutes to
install Ubuntu's FFmpeg. Skip the runner's man-db index rebuild, skip recommends, give apt network timeouts and
three attempts, and limit the step to 12 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ci: stop the FFmpeg apt install from eating the metadata job's 30 minutes
…#1016)

The document tables route now takes every document the built-in readers
read with tables: DOC (and DOT, WPS), ODT, RTF, HWP and HWPX, like DOCX and
HTML. Each table is one sheet, merged cells stay merged, numbers and dates
become real values and header rows are frozen. Before, these files had no
route to a spreadsheet at all.

- Cells keep italic and centred or right alignment besides bold, fill and
  borders; left and justified text keeps Excel's own alignment.
- Rows under a merged cell get the height its lines need: Excel and Calc
  don't fit a row to a merged cell, so its text was cut off.
- 한글 files are read as the renderer reads them, so a number typed as an
  equation is a number ("40"), not its script ("40~~~").
- DOC edges are checked, as on DOC's other routes.
- Bound the workbook: at most 10,000 sheets, Excel's row count per sheet
  and 10 million cell positions in all, counted before anything is
  written; nested tables are walked without recursion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Markdown's $...$, $$...$$, \(...\) and \[...\] are read into the equation
tree Word and Hangul equations use, so the built-in engines typeset them
instead of printing the TeX source.

- MarkdownMath: a Markdig inline parser for the four delimiters with
  Pandoc's dollar rules (no space inside, no digit after the closing $,
  escaped $ is text, code spans win); Markdown -> HTML writes MathML.
- LatexMath: our own reader for the TeX subset Markdown uses: fractions,
  roots, scripts, Greek and symbols, sums/integrals/lim with limits,
  \left...\right, matrices, cases, aligned/gather, accents, \text,
  \mathbf/\mathrm/\mathbb, spacing, \not, \tag. Unknown commands stay as
  their text. Formulas longer than 32,768 characters or nested deeper
  than 100 levels stay their source; equations count against MaxObjects.
- MarkdownReader: equations for math; display math alone in its
  paragraph is a centred equation.
- DOCX writer: equations as Word math (OmmlWriter, m:oMath/m:oMathPara).
- HTML and EPUB writers: equations as MathML (MathMLWriter, TeX kept as
  the annotation); EPUB pages with MathML get properties="mathml".
- Writers without equations keep the TeX source as written; the Markdown
  writer writes it back unescaped.
- MathLayout: TeX's further relations and binary operators get their
  spacing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Charts/ reads a chart part (c:chartSpace) from its cached values and
draws it with Skia, text as glyph runs: bar and column (clustered,
stacked, 100 %), line, area, pie, doughnut and scatter plots with
titles, axes, gridlines, legend and data labels, series colours from
c:spPr or the theme's accents in Office's order.

- Sheet printer: chart frames become SheetChart drawings; a workbook
  whose charts are all drawn stays on the built-in printer.
- Word: a chart becomes a picture of it for the writers that carries
  the chart for the renderer; the check only names charts it can't draw.
- Slides: chart frames are drawn by the slide renderer; the ODP writer
  and the deck's text take a picture of the chart.
- 3D, radar, stock, surface, bubble and chartex charts are still named.
- Bound series and points per chart and points drawn per document;
  clamp sheet font sizes to Excel's range (a zero size crashed printing).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- f', f'', f^\prime: MathLayout draws a superscript that is only primes
  at the base's size on its baseline after the base (the prime character
  is a raised mark already), instead of shrinking and raising it over an
  italic letter. Word math and MathML keep it as a superscript.
- An unknown TeX command keeps its [ ] and { } arguments literally
  (\foo{x}, not \foo followed by x); brackets alone stay the formula's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r chain prefetch

- bzip2: SA-IS rewritten to read less per step: no suffix-type array (types
  follow from two characters and the scan, a bit per position marks LMS ones),
  entries carry by their sign whether the scan has a suffix to place from them,
  LMS substrings named with their lengths in the unused half of the array and
  compared in one go. About 13 % less sorting time, same transform.
- bzip2: the smallest-rotation race also tells whether the block is periodic,
  so the prefix function only runs for blocks that are.
- xz: CRC-64 folds 16 bytes at a time with PCLMULQDQ on x64 (constants worked
  out from the polynomial at start-up); tables elsewhere and for short input.
- LZMA fast level: the chain's first candidate (link and bytes) is asked into
  the cache before the two- and three-byte candidates are checked. Same bytes.
- Tests: large blocks against rotations sorted by doubling, smallest rotation
  and periodicity against a plain search, CRC-64 folding against the tables.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon KnifeLemon added the built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet label Nov 19, 2026
@KnifeLemon
KnifeLemon merged commit 5935010 into builtin-engines Nov 19, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant