diff --git a/CHANGELOG.md b/CHANGELOG.md index afae34f..2e7fdfd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,60 @@ All notable changes to this project are documented here. The format follows - `install.sh`, a one-line installer for macOS and Linux that downloads the latest release binary. - Agent skill (`skills/sheetgraph/SKILL.md`) and a Claude Code plugin marketplace (`.claude-plugin/`), so agents run sheetgraph on their own when asked about a spreadsheet. - README: before/after demo image (`assets/demo.png`) and the script that generates its sample workbook. +- Macros: each VBA procedure's trigger (workbook and sheet events, `Auto_Open`), the sheets and ranges it writes and + uses, the procedures it calls, and the formula cells that call it as a function, in REPORT.md and as `macro` nodes + with `writes` / `uses` / `calls` edges in graph.json. Apps Script functions are listed with their triggers. +- `sheetgraph macro [name]` lists macros or prints one module's or procedure's code; `find` also searches + macro code. +- Sheets' VBA code names are read from xlsx/xlsm (`Sheet.code_name`). +- File-level settings: document and custom properties, calculation mode and iterative calculation (Excel and + Google), and whether the sheet structure is locked, at the top of REPORT.md. +- Data connections (ODBC, OLE DB, web, text, Power Query) with passwords removed, and Power Query (M) code as + `power_query` scripts that `macro` and `find` read. +- Pictures, shapes, text boxes and form controls over the grid (`Sheet.shapes`), with their text, alt text, the macro + a button runs and a control's linked cell; graph.json gets `shape` nodes with `runs` and `writes` edges. +- Pictures placed in cells and Excel data types (Stocks, Geography) with their fields (`Cell.obj`); Google people and + file chips the same way. +- Text with mixed formatting inside a cell (`Cell.runs`), from Excel and Google (`textFormatRuns`). +- Locked/unlocked cells and hidden formulas (`Style.unlocked`, `Style.hidden_formula`), listed per sheet as the + inputs left editable under protection. +- Numbers in xlsx/xlsm show as their format displays them (`25%`, `$12.50`, `(1,234)`), not only dates. +- `tests/fixtures/audit.xlsx` and `tests/audit.rs`: one workbook with every item on the coverage checklist. +- Very hidden sheets (only macro code can unhide them) are marked as such (`Sheet.very_hidden`). +- Names local to one sheet keep their scope (`NamedRange.scope`), and a sheet's own definition wins over a + workbook-wide name of the same name. Names defined by a formula, LAMBDA functions included, link to what the + formula reads, through other names too. +- Named cell styles ("Input", "Calculation", custom ones) as `Style.cell_style`, and which sides of a cell have a + border and its line (`Style.border`: `bottom double`, `box thin`), from Excel and Google. +- Dropdowns, validation formulas and conditional formats are graph nodes with `choices` / `reads` edges to the ranges + they read; REPORT.md lists those that read another sheet, and `trace` shows them both ways. +- `trace` follows links into other workbooks extracted to the same folder (`'[Other.xlsx]Data'!A1`, IMPORTRANGE), + in both directions. +- Merged cells: `cells` and `trace` say a cell is merged or hidden under a merge and show what the merge displays; + REPORT.md folds repeating merges (`12× A:C, 1 row each, rows 8–30`); formulas reading a cell hidden under a merge + (always blank) are flagged. +- Google Sheets slicers (as filters) and Tables. +- Column widths and row heights in pixels where they differ from the default (`Sheet.col_widths`, + `Sheet.row_heights`); a column of width 0 or a row of height 0 counts as hidden. Gridlines turned off + (`Sheet.no_gridlines`) show in the sheet's REPORT.md header. +- Grouped rows and columns with their level and whether they're collapsed (`Sheet.outlines`), from Excel and Google, + listed per sheet in REPORT.md. +- Notes and comment threads carry their authors (`Analyst: Check with finance`, replies after `↳`, resolved threads + marked). Google Drive comments, which the Sheets API doesn't return, are read too (`Workbook.comments`) and tied to + a cell when exactly one cell shows the text they quote; REPORT.md lists them, `cells` shows them on their cell and + `find` searches them. + +### Fixed + +- A reference across sheets (`=SUM(Jan:Dec!B2)`) linked only the last sheet; it now links every sheet in the span. +- A reference to another workbook (`'[Other.xlsx]Data'!A1`) also produced a broken `reads` edge in graph.json. +- An xlsm whose VBA uses compression calamine doesn't expect no longer aborts the extraction; the macros are reported + missing and everything else is kept. + +### Changed + +- `Style.borders` (has any border) is replaced by `Style.border`, which says which sides and their line. +- REPORT.md counts modules with no code (the empty module Excel gives every sheet) instead of listing each one. ## [0.1.0] diff --git a/README.md b/README.md index 4baf46a..b9b08bc 100644 --- a/README.md +++ b/README.md @@ -102,8 +102,9 @@ sheetgraph extract ... -o sheetgraph-out --creds service-accou # drill down (all read workbook.json.gz; no network) sheetgraph cells sheetgraph-out/ "'Revenue Input'!A1:F20" -sheetgraph trace sheetgraph-out/ "Monthly!E10" --depth 3 # precedents + dependents -sheetgraph find sheetgraph-out/ "VAT" +sheetgraph trace sheetgraph-out/ "Monthly!E10" --depth 3 # precedents + dependents, into other extracted workbooks too +sheetgraph find sheetgraph-out/ "VAT" # values, formulas, notes, comments, macro code +sheetgraph macro sheetgraph-out/ [Module1 | RollUp] # macros and what they touch; code with a name sheetgraph sheet sheetgraph-out/ "Ledger" --full # untruncated section sheetgraph report sheetgraph-out/ [--full] # re-render without refetching ``` @@ -126,14 +127,18 @@ let graph = sheetgraph::graph::build(&wb, &a); // serde_json::Value sheetgraph-out/ INDEX.md every workbook, sizes, links between workbooks (IMPORTRANGE, Excel external links) /REPORT.md the map: read this first - /graph.json nodes (sheets, blocks, formula groups, charts, pivots, externals) + edges - /workbook.json.gz every cell (see Known gaps for very large files): value, formula, display text, style, validation, note, link + /graph.json nodes (sheets, blocks, formula groups, charts, pivots, dropdowns and rules, names, externals, macros) + edges + /workbook.json.gz every cell (see Known gaps for very large files): value, formula, display text, style, validation, note, link; plus column widths, row heights and file comments /meta.json ``` -REPORT.md contains, per workbook: sheet list with used ranges and sheet→sheet data flow; external links (IMPORTRANGE / IMPORTDATA / … with targets resolved from cells, Excel links to other workbooks, refs to deleted sheets, Connected-Sheets data sources); named ranges; function census incl. **non-builtin functions** (Apps Script custom functions / Named Functions / VBA UDFs), dynamic refs (INDIRECT/OFFSET) and volatile functions; scripts (VBA modules from xlsm; Apps Script on request); developer metadata (left by add-ons/integrations). +REPORT.md contains, per workbook: sheet list with used ranges and sheet→sheet data flow; external links (IMPORTRANGE / IMPORTDATA / … with targets resolved from cells, Excel links to other workbooks, refs to deleted sheets, Connected-Sheets data sources); dropdowns, validation and conditional formats that read another sheet; named ranges, with the sheet a local name belongs to and which names are LAMBDA functions; function census incl. **non-builtin functions** (Apps Script custom functions / Named Functions / VBA UDFs), dynamic refs (INDIRECT/OFFSET) and volatile functions; scripts (VBA modules from xlsm; Apps Script on request), with each procedure's trigger (`Workbook_Open`, `Worksheet_Change`, `onEdit`), the sheets and ranges it writes and uses, the procedures it calls, and the formula cells that call it as a function; modules with no code (the empty one Excel gives every sheet) are counted, not listed; developer metadata (left by add-ons/integrations). -Per sheet: blocks (tables with column profiles + samples, or small blocks verbatim), formula copy-groups and families with flags (`[array/spills]`, `[external]`, `[dynamic ref]`, `[#REF!]`, error counts), data validation (dropdowns, checkboxes, range-sourced lists), conditional formats, visual formatting regions, merges, filters/filter views, protected ranges, hidden rows/cols, banding, Excel tables, charts and pivots (with source ranges), notes and comments, hyperlinks. +REPORT.md also starts with what changes how the whole file behaves: document properties, calculation mode (manual calculation means stored values may be stale; iterative means circular references are intended), a locked sheet structure, and data connections and Power Query code. + +graph.json edges: `reads` (formula group → the blocks and groups its references land on), `plots` (chart → source), `summarizes` (pivot → source), `imports` (formula → external workbook or URL), `writes` / `uses` (macro → what it changes or reads; a sheet the macro creates has `"to": null` and `missing_sheet`), `calls` (macro → macro, and formula group → the VBA function it calls), `runs` (button or shape → the macro it runs), `writes` from a form control to its linked cell, `choices` (dropdown → the range its list comes from), and `reads` from validation and conditional-format rules and from names defined by a formula (LAMBDA included) to what they read. A formula reading a name defined by a formula gets `reads` edges to what that formula reads, with `via` naming it; a name local to one sheet is the node `name:Sheet!Name`. + +Per sheet: blocks (tables with column profiles + samples, or small blocks verbatim), formula copy-groups and families with flags (`[array/spills]`, `[external]`, `[dynamic ref]`, `[#REF!]`, error counts, reads of a cell hidden under a merge), data validation (dropdowns, checkboxes, range-sourced lists), conditional formats, visual formatting regions (with named cell styles such as "Input" and which sides have a border), merges (repeats folded into patterns), filters/filter views and slicers, protected ranges, hidden rows/cols, banding, Excel tables, charts and pivots (with source ranges), pictures, shapes, text boxes and buttons over the grid (with their text and the macro they run), pictures in cells and data types (Excel Stocks/Geography, Google people and file chips), text with mixed formatting, unlocked input cells and hidden formulas, notes and comments with their authors, hyperlinks, grouped rows and columns (collapsed or not), and whether gridlines are off. Column widths and row heights are kept in workbook.json.gz only; a zero width or height counts as hidden. Google Drive comments (with replies and whether they're resolved) get their own section. ## Giving it to an AI agent @@ -144,30 +149,37 @@ Point the agent at `INDEX.md`, then the `REPORT.md` it needs, and let it run the - "typed arithmetic" is hand-entered sums like `=3820+3770`: data entry, not logic. - `▸ range table …` is a data block, one line per column: type and range, `enum` for small value sets, `fmt` for the number format, `ƒN` when the column is computed. - "same layout as X" means a sheet is a structural clone of an earlier one with different data; `sheetgraph sheet "" --full` shows it in full. +- `[reads B8 inside merge A8:C8: blank]` flags a formula pointing at a cell a merge hides: it always reads blank, usually a mistake for the merge's top-left cell. - `⋯ rows A–B summarized` means a very large sheet's middle rows were summarized per column rather than kept: the column profiles still count every row, but sample rows and `cells` don't show those rows. - Flags: `[external]` another file, `[dynamic ref]` INDIRECT/OFFSET (graph edges incomplete), `[array/spills]`, `[#REF!]`, `[N errors]`. +- Under Scripts, `RollUp: writes Summary!A1:B1,B2:B5 · uses Data!E10,… · calls Totals` is one procedure. `,…` means it also reaches cells whose address is built at run time (`Cells(i, 2)`); `(not in file)` is a sheet the macro creates; `ActiveSheet` is whichever sheet is active when it runs. `sheetgraph macro RollUp` prints its code. ## Layout | module | role | |---|---| | `model` | source-agnostic full-fidelity workbook (sparse cells, deduped styles/validations) | -| `gsheets` | auth (service account JWT / ADC refresh / token), chunked parallel grid fetch with retry, conversion | +| `gsheets` | auth (service account JWT / ADC refresh / token), chunked parallel grid fetch with retry, conversion, Drive comments | | `xlsx` | calamine: values, formulas, defined names, merges, VBA; csv | -| `ooxml` | xlsx/xlsm package parts calamine skips: styles, validation, conditional formats, comments, hyperlinks, frozen/hidden rows and cols, filters, protection, tables, charts, pivots, external workbook links | +| `ooxml` | xlsx/xlsm package parts calamine skips: styles, validation, conditional formats, comments and their authors, hyperlinks, frozen/hidden rows and cols, column widths, row heights, outline groups, gridlines, filters, protection, tables, charts, pivots, external workbook links, rich text, pictures and data types in cells, shapes and form controls, document properties, calculation settings, connections, Power Query | +| `numfmt` | Excel number formats applied to numbers (`25%`, `$12.50`) | | `formula` | reference lexer (quoted and non-English sheet names, open-ended `A2:A`, whole rows/cols, `'[Book.xlsx]Sheet'`), R1C1 keys, function census | -| `analyze` | copy-groups, families, blocks, cross-sheet edges, externals | +| `analyze` | copy-groups, families, blocks, cross-sheet edges, externals, names, dropdown and rule refs | +| `macros` | VBA / Apps Script procedures: triggers, sheets and ranges each writes and uses, calls | | `report` | REPORT.md | | `graph` | graph.json | -| `query` | cells / trace / find | +| `query` | cells / trace / find / macro | | `store` | on-disk layout, INDEX.md | ## Known gaps - **Apps Script / macros / add-ons on Google Sheets**: the Sheets API doesn't expose bound scripts, installed add-ons or triggers, and the Apps Script API doesn't work with service accounts. `--apps-script ` fetches a project with user OAuth (scope `script.projects.readonly`). Custom functions are still detected from formulas. -- **Local files**: xlsx/xlsm get the same coverage as Google Sheets, plus Excel tables and links to other workbooks (`'[Budget.xlsx]Data'!B2`). Images, shapes, sparklines and slicers aren't read (the report says when a file has them). xls, xlsb and ods get values, formulas, names and merges only. -- Number-format display text (e.g. `$1,234.50`) is only rendered for dates in local files; the format itself is on every cell. +- **Local files**: xlsx/xlsm get the same coverage as Google Sheets, plus Excel tables, links to other workbooks (`'[Budget.xlsx]Data'!B2`), shapes and pictures, connections and Power Query. Sparklines, Excel slicers and ActiveX controls aren't read (the report says when a file has them). xls, xlsb and ods get values, formulas, names and merges only. +- **Comments on Google Sheets**: the Drive API doesn't say which cell a comment is on, only the text that cell showed. A comment is placed on a cell only when exactly one cell shows that text; otherwise REPORT.md lists it with the text it quotes. +- **Pictures on Google Sheets**: the Sheets API doesn't return pictures placed over the grid or inserted into cells (an `=IMAGE()` formula is kept). +- Number-format display text covers the common formats (percent, currency, thousands, decimals, scientific, sections); fractions, conditional sections and formats that mix digits with literals (`000-00-0000`) show the plain value. The format itself is on every cell. - Spill ranges of array formulas are flagged on the formula but the spilled cells look like plain values. +- **What a macro touches** is read from its code, not by running it: literal addresses, sheets held in variables, `With` blocks, sheets passed to other procedures, code names, string constants and named ranges are followed; an address built at run time counts as the whole sheet, and a sheet picked at run time (from a cell, a loop over all sheets) isn't known. Apps Script code is listed by function without the ranges it touches. - INDIRECT/OFFSET targets are dynamic, so those edges are missing (flagged in the report). - **Very large local files**: up to 10 million cells are kept whole (about 0.2–0.4 GB of memory per million). Past that, a big sheet keeps its last 1,000 rows and its formulas, and summarizes the rows in between per column (counts, min and max, most common values; no sums), so memory stays near 2–3 GB however big the file is (a 160 MB xlsx with 44 million cells: 3 GB, 84 s). REPORT.md's column profiles still count every row and say which rows were summarized; `cells` only lists kept rows, and `find` reports text matches in summarized rows by column. The budget is `xlsx::CELL_BUDGET`; `xlsx::read_limited` takes another. - **Very large Google Sheets** are read whole, without that budget. Google allows up to 20 million cells, so a sheet near the limit can take several GB of memory. @@ -178,7 +190,7 @@ Point the agent at `INDEX.md`, then the `REPORT.md` it needs, and let it run the cargo test ``` -`tests/fixtures/features.xlsx` exercises every xlsx feature the reader handles. It is generated by `tests/fixtures/make_features.py` (openpyxl 3.1.5); rerun the script after changing it. +`tests/fixtures/features.xlsx` exercises every xlsx feature the reader handles. It is generated by `tests/fixtures/make_features.py` (openpyxl 3.1.5); rerun the script after changing it. `tests/fixtures/macros.xlsm` is the VBA fixture, generated the same way by `tests/fixtures/make_macros.py`, which also writes its VBA project from scratch. See [CONTRIBUTING.md](CONTRIBUTING.md) before opening an issue or pull request, and [SECURITY.md](SECURITY.md) to report a vulnerability. diff --git a/skills/sheetgraph/SKILL.md b/skills/sheetgraph/SKILL.md index 20367bd..b94adc0 100644 --- a/skills/sheetgraph/SKILL.md +++ b/skills/sheetgraph/SKILL.md @@ -41,14 +41,23 @@ Notation: - "typed arithmetic" (`=3820+3770`) is hand-entered data, not logic. - `▸ range table …` is a data block, one line per column: type and range, `enum` for small value sets, `fmt` for the number format, `ƒN` when computed. - `⋯ rows A–B summarized`: a very large sheet's middle rows are summarized per column, not kept. -- Flags: `[external]` another file, `[dynamic ref]` INDIRECT/OFFSET (graph edges incomplete), `[array/spills]`, `[#REF!]`, `[N errors]`. +- Flags: `[external]` another file, `[dynamic ref]` INDIRECT/OFFSET (graph edges incomplete), `[array/spills]`, `[#REF!]`, `[N errors]`, `[reads B8 inside merge A8:C8: blank]` (a formula pointing at a cell a merge hides: it always reads blank, usually a bug). +- `[very hidden]` sheets can only be unhidden by macro code: someone meant them to stay out of sight. `Name (Calc only)` is a name local to one sheet; `(LAMBDA function)` names are custom functions defined in the workbook. +- `gridlines off` in a sheet's header means it's laid out as a page or form, not a grid of records. `grouped rows: 5-8 (collapsed)` are detail rows folded under a summary; collapsed ones are hidden on screen. +- Notes read `Author: text`, with replies after `↳`. `## Comments` lists Google Drive comments; those not tied to a cell name the text the commented cell showed. +- Formatting lines name cell styles (`style "Input"` marks intended inputs, `"Calculation"` computed cells) and borders (`border bottom double` is usually a totals line). +- The lines under the title are file settings. `calculation: manual` means stored values may be stale; `iterative` means circular references are intended, not errors. +- `▣` lines are pictures in cells and data types (stocks, places, people chips); `[image: …]` in a cell is a picture, not text. `✎ mixed formatting` shows runs as `[text]{bold fg#FF0000}`. `unlocked cells` are the inputs a protected sheet leaves editable. +- Shape lines (`button "Run roll-up" at G9:H10 → runs RollUp`, `text box "…"`) are things drawn over the grid; text boxes often hold instructions. +- Scripts: one line per macro with its trigger, what it `writes` and `uses`, and what it `calls`. `,…` means more cells whose address is built at run time; `(not in file)` is a sheet the macro creates. Values a macro writes are pasted values in the sheet, not formulas: check here before calling a number hardcoded. ## 4. Drill down for exact values (offline, fast) ```bash sheetgraph cells sheetgraph-out/ "'Sheet Name'!A1:F20" # exact values, formulas, formats -sheetgraph trace sheetgraph-out/ "Summary!B4" --depth 3 # precedents and dependents -sheetgraph find sheetgraph-out/ "VAT" # search text, formulas and names +sheetgraph trace sheetgraph-out/ "Summary!B4" --depth 3 # precedents and dependents, across extracted workbooks +sheetgraph find sheetgraph-out/ "VAT" # search text, formulas, names, notes, comments and macro code +sheetgraph macro sheetgraph-out/ RollUp # a macro's code (no name: every macro and what it touches) sheetgraph sheet sheetgraph-out/ "Ledger" --full # one sheet's section, untruncated ``` diff --git a/src/analyze.rs b/src/analyze.rs index 6b59430..8168693 100644 --- a/src/analyze.rs +++ b/src/analyze.rs @@ -3,6 +3,7 @@ //! - families: groups whose formulas differ only in refs/numbers //! - blocks: connected regions of non-empty cells (tables, label clusters) //! - edges: group → referenced range, resolved to sheets / named ranges / externals +//! - rules: dropdowns, validation and conditional formats, with the ranges they read use crate::a1::{parse_ref, Rect}; use crate::formula::{self, call_args, lex, unquote, Lexed}; @@ -21,7 +22,40 @@ pub struct Analysis { pub func_cells: BTreeMap, /// References to sheets that do not exist in this workbook. pub broken_refs: BTreeMap, - pub named: HashMap, Rect)>, + /// Defined names and tables by upper-cased name: workbook-wide and + /// sheet-local definitions of the same name side by side. + pub names: HashMap>, + /// Procedures in the workbook's macro / script modules. + pub macros: Vec, +} + +/// What a defined name (or an Excel table) stands for. +#[derive(Debug, Clone, Default)] +pub struct NameDef { + /// As defined (names match case-insensitively). + pub name: String, + /// The sheet the name is local to; None when workbook-wide. + pub scope: Option, + /// The ranges it covers: its reference, or every range its formula reads + /// (through other names too). A None sheet is the sheet using the name. + pub targets: Vec<(Option, Rect)>, + /// Defined by a formula (a constant, an expression or a LAMBDA) rather + /// than a reference. + pub formula: bool, + pub lambda: bool, +} + +impl Analysis { + /// The definition `n` has on sheet `si`: the sheet's own, else the workbook's. + pub fn name(&self, si: Option, n: &str) -> Option<&NameDef> { + let defs = self.names.get(&n.to_ascii_uppercase())?; + defs.iter().find(|d| d.scope.is_some() && d.scope == si).or_else(|| defs.iter().find(|d| d.scope.is_none())) + } + + /// The single range a plain-reference name covers. + pub fn name_range(&self, si: Option, n: &str) -> Option<(Option, Rect)> { + self.name(si, n).filter(|d| !d.formula && d.targets.len() == 1).map(|d| d.targets[0]) + } } #[derive(Debug, Clone, Default)] @@ -39,6 +73,19 @@ pub struct SheetAnalysis { /// Sheet index this sheet is a structural clone of (same layout, other data). pub clone_of: Option, pub clone_similarity: f32, + /// Dropdowns, validation and conditional-format rules that read ranges. + pub rules: Vec, +} + +/// A dropdown, validation or conditional-format rule and what it reads. +#[derive(Debug, Clone)] +pub struct Rule { + /// "dropdown" (choices from a range), "validation" (a formula), "cond_format" + pub kind: &'static str, + pub ranges: Vec, + /// The rule as REPORT.md shows it. + pub desc: String, + pub refs: Vec, } #[derive(Debug, Clone)] @@ -58,6 +105,17 @@ pub struct Group { pub external: bool, /// No refs, names or functions: hand-typed arithmetic like `=3000+1577`. pub constant: bool, + /// Cells of the group whose formula reads a cell hidden under a merge + /// (blank: a merge keeps only its top-left cell), with the first such read. + pub covered: u32, + pub covered_eg: Option, +} + +#[derive(Debug, Clone)] +pub struct CoveredRead { + pub sheet: usize, + pub cell: (u32, u32), + pub merge: Rect, } #[derive(Debug, Clone)] @@ -88,19 +146,6 @@ pub struct External { } pub fn analyze(wb: &Workbook) -> Analysis { - let mut named = HashMap::new(); - for nr in &wb.named_ranges { - if let Some((sh, rect)) = parse_ref(&nr.range) { - let si = sh.as_deref().and_then(|s| sheet_index(wb, s)); - named.insert(nr.name.to_ascii_uppercase(), (si, rect)); - } - } - // Excel tables resolve like names: `Sales[Amount]` lexes as the name `Sales`. - for (si, sh) in wb.sheets.iter().enumerate() { - for t in &sh.tables { - named.entry(t.name.to_ascii_uppercase()).or_insert((Some(si), t.range)); - } - } let mut a = Analysis { sheets: Vec::new(), sheet_edges: BTreeMap::new(), @@ -108,16 +153,82 @@ pub fn analyze(wb: &Workbook) -> Analysis { custom_funcs: BTreeMap::new(), func_cells: BTreeMap::new(), broken_refs: BTreeMap::new(), - named, + names: names(wb), + macros: Vec::new(), }; for (si, sh) in wb.sheets.iter().enumerate() { let sa = analyze_sheet(wb, si, sh, &mut a); a.sheets.push(sa); } find_clones(wb, &mut a); + a.macros = crate::macros::analyze(wb, &a); a } +/// Every defined name and table. A name defined by a formula covers what the +/// formula reads, other names included (resolved a few levels deep). +fn names(wb: &Workbook) -> HashMap> { + let mut out: HashMap> = HashMap::new(); + // (key, index in out[key], names its formula uses) for the second pass. + let mut pending: Vec<(String, usize, Vec)> = Vec::new(); + for nr in &wb.named_ranges { + let scope = nr.scope.as_deref().and_then(|s| sheet_index(wb, s)); + let key = nr.name.to_ascii_uppercase(); + let mut d = NameDef { name: nr.name.clone(), scope, ..Default::default() }; + let mut uses = Vec::new(); + match parse_ref(&nr.range) { + Some((sh, rect)) => d.targets.push((sh.as_deref().and_then(|s| sheet_index(wb, s)).or(scope), rect)), + None => { + let lx = lex(&nr.range); + d.formula = true; + d.lambda = lx.funcs.iter().any(|f| f.name == "LAMBDA"); + for t in &lx.refs { + let Some(r) = t.rect() else { continue }; + match &t.sheet { + None => d.targets.push((scope, r)), + Some(n) => d.targets.extend(sheet_span(wb, n).into_iter().map(|s| (Some(s), r))), + } + } + uses = lx.names.iter().chain(lx.funcs.iter().map(|f| &f.name)).map(|n| n.to_ascii_uppercase()).filter(|n| *n != key).collect(); + } + } + let defs = out.entry(key.clone()).or_default(); + if !uses.is_empty() { + pending.push((key, defs.len(), uses)); + } + defs.push(d); + } + // Excel tables resolve like names: `Sales[Amount]` lexes as the name `Sales`. + for (si, sh) in wb.sheets.iter().enumerate() { + for t in &sh.tables { + out.entry(t.name.to_ascii_uppercase()).or_insert_with(|| vec![NameDef { name: t.name.clone(), targets: vec![(Some(si), t.range)], ..Default::default() }]); + } + } + for _ in 0..3 { + let mut changed = false; + for (key, i, uses) in &pending { + let scope = out[key][*i].scope; + let mut add = Vec::new(); + for u in uses { + let Some(defs) = out.get(u) else { continue }; + let d = defs.iter().find(|d| d.scope.is_some() && d.scope == scope).or_else(|| defs.iter().find(|d| d.scope.is_none())); + add.extend(d.into_iter().flat_map(|d| d.targets.iter().copied())); + } + let d = &mut out.get_mut(key).unwrap()[*i]; + for t in add { + if !d.targets.contains(&t) { + d.targets.push(t); + changed = true; + } + } + } + if !changed { + break; + } + } + out +} + /// Structural fingerprint: formula families (copied ≥2 cells, ignoring typed /// arithmetic) plus the text labels in the top rows / first columns. pub fn fingerprint(sh: &Sheet, sa: &SheetAnalysis) -> std::collections::HashSet { @@ -169,6 +280,18 @@ pub fn sheet_index(wb: &Workbook, name: &str) -> Option { }) } +/// Sheets a reference's sheet part covers: the one sheet, or every sheet from +/// first to last (in tab order) for a 3D ref like `Jan:Dec`. Empty if unknown. +pub fn sheet_span(wb: &Workbook, name: &str) -> Vec { + if let Some(i) = sheet_index(wb, name) { + return vec![i]; + } + match name.split_once(':').map(|(a, b)| (sheet_index(wb, a), sheet_index(wb, b))) { + Some((Some(x), Some(y))) => (x.min(y)..=x.max(y)).collect(), + _ => Vec::new(), + } +} + /// Index of the cell at (r, c) in a sheet's sorted cell list. pub fn find_cell(sh: &Sheet, r: u32, c: u32) -> Option<&Cell> { sh.cells.binary_search_by_key(&(r, c), |x| (x.r, x.c)).ok().map(|i| &sh.cells[i]) @@ -283,6 +406,8 @@ fn analyze_sheet(wb: &Workbook, si: usize, sh: &Sheet, a: &mut Analysis) -> Shee spill: false, external: false, constant: lx.refs.is_empty() && lx.names.is_empty() && lx.funcs.is_empty(), + covered: 0, + covered_eg: None, }; for fu in &g.funcs { g.dynamic |= formula::DYNAMIC_FUNCS.contains(&fu.as_str()); @@ -293,6 +418,7 @@ fn analyze_sheet(wb: &Workbook, si: usize, sh: &Sheet, a: &mut Analysis) -> Shee g.external |= !lx.books.is_empty(); g.errors = cells_in(sh, &rect).filter(|c| matches!(c.v, Value::Err { .. })).count() as u32; g.refs = resolve_refs(wb, si, &lx, &rect, a); + covered_reads(wb, si, &lx, &mut g); let mut targets: Vec = g.refs.iter().filter_map(|r| r.sheet).filter(|t| *t != si).collect(); targets.sort_unstable(); targets.dedup(); @@ -317,9 +443,87 @@ fn analyze_sheet(wb: &Workbook, si: usize, sh: &Sheet, a: &mut Analysis) -> Shee } sa.blocks = find_blocks(sh); + sa.rules = rules(wb, si, sh, sa.used, a); sa } +/// A merge shows its top-left cell over the whole range; every other cell in +/// it is blank. Count the group's cells whose single-cell refs land on one. +fn covered_reads(wb: &Workbook, si: usize, lx: &Lexed, g: &mut Group) { + let rect = g.rect; + let (dr, dc) = (rect.r1 as i64 - rect.r0 as i64, rect.c1 as i64 - rect.c0 as i64); + let mut hit = vec![false; (rect.area() as usize).min(1 << 16)]; + for t in &lx.refs { + if t.rect_at(0, 0).is_none_or(|r| r.area() != 1) { + continue; + } + let sheets = match &t.sheet { + None => vec![si], + Some(n) => sheet_span(wb, n), + }; + for ts in sheets { + let merges = &wb.sheets[ts].merges; + let (Some(a0), Some(a1)) = (t.rect_at(0, 0), t.rect_at(dr, dc)) else { continue }; + let span = a0.union(&a1); + let near: Vec<&Rect> = merges.iter().filter(|m| m.area() > 1 && m.intersects(&span)).collect(); + if near.is_empty() { + continue; + } + let cols = rect.cols() as usize; + for (i, h) in hit.iter_mut().enumerate() { + let Some(x) = t.rect_at((i / cols) as i64, (i % cols) as i64) else { continue }; + if let Some(m) = near.iter().find(|m| m.contains(x.r0, x.c0) && (x.r0, x.c0) != (m.r0, m.c0)) { + *h = true; + g.covered_eg.get_or_insert(CoveredRead { sheet: ts, cell: (x.r0, x.c0), merge: **m }); + } + } + } + } + g.covered = hit.iter().filter(|h| **h).count() as u32; +} + +/// Dropdowns, validation formulas and conditional formats that read ranges. +fn rules(wb: &Workbook, si: usize, sh: &Sheet, used: Option, a: &Analysis) -> Vec { + let mut out = Vec::new(); + // Whole-column rules: only as far as the sheet has cells. + let clamp = |r: Rect| match used { + Some(u) => Rect { r1: r.r1.min(u.r1.max(r.r0)), c1: r.c1.min(u.c1.max(r.c0)), ..r }, + None => r, + }; + let dv_pts: Vec<(u32, u32, u32)> = sh.cells.iter().filter_map(|c| c.dv.map(|d| (d, c.r, c.c))).collect(); + let mut by: BTreeMap> = BTreeMap::new(); + for (d, r) in cover(dv_pts) { + by.entry(d).or_default().push(r); + } + for (d, ranges) in by { + let v = &wb.validations[d as usize]; + let (kind, f) = match (v.kind.as_str(), v.values.first()) { + ("ONE_OF_RANGE", Some(f)) => ("dropdown", f), + (_, Some(f)) if f.starts_with('=') => ("validation", f), + _ => continue, + }; + let f = if f.starts_with('=') { f.clone() } else { format!("={f}") }; + let lx = lex(&f); + let bbox = ranges.iter().skip(1).fold(ranges[0], |b, r| b.union(r)); + // A dropdown's list is one fixed range for every cell; a formula shifts. + let span = if kind == "dropdown" { Rect::cell(bbox.r0, bbox.c0) } else { bbox }; + let refs = resolve_refs(wb, si, &lx, &span, a); + if !refs.is_empty() { + out.push(Rule { kind, desc: f, ranges, refs }); + } + } + for cf in &sh.cond_formats { + let Some(first) = cf.ranges.first() else { continue }; + let bbox = cf.ranges.iter().skip(1).fold(clamp(*first), |b, r| b.union(&clamp(*r))); + let span = Rect { r0: first.r0, c0: first.c0, ..bbox }; + let refs: Vec = cf.formulas.iter().flat_map(|f| resolve_refs(wb, si, &lex(f), &span, a)).collect(); + if !refs.is_empty() { + out.push(Rule { kind: "cond_format", ranges: cf.ranges.clone(), desc: cf.rule.clone(), refs }); + } + } + out +} + fn dedup(mut v: Vec) -> Vec { let mut seen = std::collections::HashSet::new(); v.retain(|x| seen.insert(x.clone())); @@ -331,15 +535,25 @@ fn resolve_refs(wb: &Workbook, si: usize, lx: &Lexed, rect: &Rect, a: &Analysis) let mut out = Vec::new(); for t in &lx.refs { let (Some(a0), Some(a1)) = (t.rect_at(0, 0), t.rect_at(dr, dc)) else { continue }; - let (sheet, sheet_name) = match &t.sheet { - None => (Some(si), None), - Some(n) => (sheet_index(wb, n), Some(n.clone())), + let rect = a0.union(&a1); + let Some(n) = &t.sheet else { + out.push(GRef { sheet: Some(si), sheet_name: None, rect, via_name: None }); + continue; }; - out.push(GRef { sheet, sheet_name, rect: a0.union(&a1), via_name: None }); + let span = sheet_span(wb, n); + if span.is_empty() { + out.push(GRef { sheet: None, sheet_name: Some(n.clone()), rect, via_name: None }); + } + for s in span { + out.push(GRef { sheet: Some(s), sheet_name: Some(n.clone()), rect, via_name: None }); + } } - for n in &lx.names { - if let Some((sh, r)) = a.named.get(&n.to_ascii_uppercase()) { - out.push(GRef { sheet: sh.or(Some(si)), sheet_name: None, rect: *r, via_name: Some(n.clone()) }); + // Names, and functions defined as LAMBDA names (`=AddTax(10)`). + let lambdas = lx.funcs.iter().filter(|f| !formula::is_builtin(&f.name)).map(|f| &f.name); + for n in lx.names.iter().chain(lambdas) { + let Some(d) = a.name(Some(si), n) else { continue }; + for (sh, r) in &d.targets { + out.push(GRef { sheet: sh.or(Some(si)), sheet_name: None, rect: *r, via_name: Some(d.name.clone()) }); } } out diff --git a/src/formula.rs b/src/formula.rs index 87e4fd5..b53e43d 100644 --- a/src/formula.rs +++ b/src/formula.rs @@ -127,6 +127,17 @@ pub fn lex(f: &str) -> Lexed { i = lex_ref_after_sheet(f, start, i + 1, Some(word.to_string()), &mut out); continue; } + // 3D ref across a run of sheets: `Jan:Dec!B2` (sheet "Jan:Dec"). + if i < b.len() && b[i] == b':' { + let mut k = i + 1; + while k < b.len() && is_ident(b[k]) { + k += 1; + } + if k > i + 1 && k < b.len() && b[k] == b'!' { + i = lex_ref_after_sheet(f, start, k + 1, Some(f[start..k].to_string()), &mut out); + continue; + } + } if i < b.len() && b[i] == b'[' { // Table1[Col] out.names.push(word.to_string()); @@ -435,6 +446,16 @@ mod tests { assert_eq!(args, vec!["\"abc\"", "\"S!A1:B2\""]); } + #[test] + fn three_d_refs() { + for f in ["=SUM(Jan:Dec!B2)", "=SUM('Jan:Dec'!B2)"] { + let lx = lex(f); + assert_eq!(lx.refs.len(), 1, "{f}"); + assert_eq!(lx.refs[0].sheet.as_deref(), Some("Jan:Dec"), "{f}"); + assert!(lx.names.is_empty(), "{f}"); + } + } + #[test] fn external_workbook_refs() { let lx = lex("='[Budget 2026.xlsx]Data'!B2+'[Rates.xlsx]'!Margin*C1"); diff --git a/src/graph.rs b/src/graph.rs index 3a18aef..a5b89b4 100644 --- a/src/graph.rs +++ b/src/graph.rs @@ -2,6 +2,7 @@ use crate::a1::{quote_sheet, Rect}; use crate::analyze::Analysis; +use crate::macros::At; use crate::model::Workbook; use serde_json::{json, Value as J}; @@ -15,6 +16,9 @@ pub fn group_id(si: usize, gi: usize) -> String { pub fn block_id(si: usize, bi: usize) -> String { format!("s{}.b{}", si + 1, bi + 1) } +pub fn macro_id(wb: &Workbook, p: &crate::macros::Proc) -> String { + format!("macro:{}.{}", wb.scripts[p.script].name, p.name) +} pub fn build(wb: &Workbook, a: &Analysis) -> J { let mut nodes = Vec::new(); @@ -22,7 +26,7 @@ pub fn build(wb: &Workbook, a: &Analysis) -> J { for (si, sh) in wb.sheets.iter().enumerate() { let sa = &a.sheets[si]; nodes.push(json!({ - "id": format!("s{}", si + 1), "type": "sheet", "name": sh.name, "hidden": sh.hidden, + "id": format!("s{}", si + 1), "type": "sheet", "name": sh.name, "hidden": sh.hidden, "very_hidden": sh.very_hidden, "used": sa.used.map(|u| u.a1()), "cells": sa.n_cells, "formulas": sa.n_formulas, })); for (bi, b) in sa.blocks.iter().enumerate() { @@ -33,7 +37,7 @@ pub fn build(wb: &Workbook, a: &Analysis) -> J { } for (gi, g) in sa.groups.iter().enumerate() { let mut flags = Vec::new(); - for (on, name) in [(g.external, "external"), (g.dynamic, "dynamic"), (g.volatile, "volatile"), (g.spill, "spill")] { + for (on, name) in [(g.external, "external"), (g.dynamic, "dynamic"), (g.volatile, "volatile"), (g.spill, "spill"), (g.covered > 0, "reads_merged_blank")] { if on { flags.push(name); } @@ -50,6 +54,10 @@ pub fn build(wb: &Workbook, a: &Analysis) -> J { _ => r.rect.a1(), }; let Some(t) = r.sheet else { + // `[Book.xlsx]Sheet` refs are `imports` edges to the external node. + if r.sheet_name.as_deref().is_some_and(|n| n.starts_with('[')) { + continue; + } edges.push(json!({"from": group_id(si, gi), "to": null, "kind": "reads", "range": range_txt, "broken": true})); continue; }; @@ -70,6 +78,50 @@ pub fn build(wb: &Workbook, a: &Analysis) -> J { link_a1(wb, a, &id, src, "plots", &mut edges); } } + for (xi, x) in sh.shapes.iter().enumerate().filter(|(_, x)| x.on_click.is_some() || x.linked_cell.is_some()) { + let id = format!("s{}.shape{}", si + 1, xi + 1); + nodes.push(json!({ + "id": id, "type": "shape", "sheet": format!("s{}", si + 1), "kind": x.kind, "text": x.text, + "range": x.anchor.map(|r| r.a1()), + })); + if let Some(m) = &x.on_click { + for p in a.macros.iter().filter(|p| crate::macros::clicks(m, wb, p)) { + edges.push(json!({"from": id, "to": macro_id(wb, p), "kind": "runs"})); + } + } + if let Some(cell) = &x.linked_cell { + let a1 = if cell.contains('!') { cell.clone() } else { format!("{}!{cell}", quote_sheet(&sh.name)) }; + link_a1(wb, a, &id, &a1, "writes", &mut edges); + } + } + // Dropdowns pick from a range; validation and conditional formats read their formula's refs. + let (mut ndv, mut ncf) = (0, 0); + for r in &sa.rules { + let (id, ty) = if r.kind == "cond_format" { + ncf += 1; + (format!("s{}.cf{ncf}", si + 1), "cond_format") + } else { + ndv += 1; + (format!("s{}.dv{ndv}", si + 1), "validation") + }; + nodes.push(json!({ + "id": id, "type": ty, "kind": r.kind, "sheet": format!("s{}", si + 1), + "ranges": r.ranges.iter().map(|x| x.a1()).collect::>(), "rule": r.desc, + })); + let kind = if r.kind == "dropdown" { "choices" } else { "reads" }; + for x in &r.refs { + let Some(t) = x.sheet else { continue }; + let range = format!("{}!{}", quote_sheet(&wb.sheets[t].name), x.rect); + let ts = targets(a, t, &x.rect, None); + if ts.is_empty() || ts.len() > MAX_TARGETS { + edges.push(json!({"from": id, "to": format!("s{}", t + 1), "kind": kind, "range": range, "via": x.via_name})); + } else { + for to in ts { + edges.push(json!({"from": id, "to": to, "kind": kind, "range": range, "via": x.via_name})); + } + } + } + } for p in &sh.pivots { let id = format!("s{}.pivot{}", si + 1, crate::a1::cell_name(p.anchor.0, p.anchor.1)); nodes.push(json!({"id": id, "type": "pivot", "sheet": format!("s{}", si + 1), "rows": p.rows, "cols": p.cols, "values": p.values})); @@ -84,7 +136,67 @@ pub fn build(wb: &Workbook, a: &Analysis) -> J { edges.push(json!({"from": group_id(e.sheet, e.group), "to": id, "kind": "imports", "range": e.range})); } for n in &wb.named_ranges { - nodes.push(json!({"id": format!("name:{}", n.name), "type": "named_range", "range": n.range})); + // A sheet-local name is `name:Sheet!Name`: other sheets can hold their own. + let id = match &n.scope { + Some(sc) => format!("name:{sc}!{}", n.name), + None => format!("name:{}", n.name), + }; + let scope = n.scope.as_deref().and_then(|x| crate::analyze::sheet_index(wb, x)); + let def = a.name(scope, &n.name).filter(|d| d.scope == scope); + nodes.push(json!({ + "id": id, "type": "named_range", "range": n.range, "scope": n.scope, + "formula": def.is_some_and(|d| d.formula), "lambda": def.is_some_and(|d| d.lambda), + })); + // What a formula name (or LAMBDA) reads. + for (t, r) in def.filter(|d| d.formula).map(|d| d.targets.as_slice()).unwrap_or_default() { + let Some(t) = t.or(scope) else { continue }; + let range = format!("{}!{r}", quote_sheet(&wb.sheets[t].name)); + for to in targets(a, t, r, None).into_iter().take(MAX_TARGETS) { + edges.push(json!({"from": id, "to": to, "kind": "reads", "range": range})); + } + } + } + for p in &a.macros { + let id = macro_id(wb, p); + let sc = &wb.scripts[p.script]; + nodes.push(json!({ + "id": id, "type": "macro", "lang": sc.kind, "module": sc.name, "name": p.name, "kind": p.kind, + "lines": format!("{}-{}", p.lines.0, p.lines.1), + "trigger": p.trigger.as_ref().map(|t| t.event.clone()), + "trigger_sheet": p.trigger.as_ref().and_then(|t| t.sheet).map(|s| format!("s{}", s + 1)), + })); + for t in &p.touches { + let kind = if t.write { "writes" } else { "uses" }; + let range = crate::macros::place(wb, t); + match &t.at { + At::Sheet(s) => { + let ts = t.rect.map(|r| targets(a, *s, &r, None)).unwrap_or_default(); + if ts.is_empty() || ts.len() > MAX_TARGETS { + edges.push(json!({"from": id, "to": format!("s{}", s + 1), "kind": kind, "range": range})); + } else { + for to in ts { + edges.push(json!({"from": id, "to": to, "kind": kind, "range": range})); + } + } + } + At::Missing(name) => edges.push(json!({"from": id, "to": null, "kind": kind, "range": range, "missing_sheet": name})), + // Which sheet is active when it runs isn't known. + At::Active => {} + } + } + for &c in &p.calls { + edges.push(json!({"from": id, "to": macro_id(wb, &a.macros[c]), "kind": "calls"})); + } + } + // Worksheet formulas calling VBA functions (UDFs). + for (si, sa) in a.sheets.iter().enumerate() { + for (gi, g) in sa.groups.iter().enumerate() { + for f in &g.funcs { + if let Some(p) = a.macros.iter().find(|p| p.kind == "function" && p.name.eq_ignore_ascii_case(f)) { + edges.push(json!({"from": group_id(si, gi), "to": macro_id(wb, p), "kind": "calls"})); + } + } + } } json!({ "schema": SCHEMA, diff --git a/src/gsheets.rs b/src/gsheets.rs index 3d4cff8..f279d8d 100644 --- a/src/gsheets.rs +++ b/src/gsheets.rs @@ -5,7 +5,7 @@ use crate::a1::{col_name, quote_sheet, Rect, OPEN}; use crate::model::*; use anyhow::{anyhow, bail, Context, Result}; use serde::Deserialize; -use serde_json::Value as J; +use serde_json::{json, Value as J}; use std::collections::HashMap; use std::sync::{mpsc, Mutex}; use std::time::{Duration, Instant}; @@ -164,6 +164,7 @@ struct RowData { #[serde(rename_all = "camelCase", default)] struct DimMeta { hidden_by_user: bool, + pixel_size: Option, } #[derive(Deserialize, Default)] #[serde(rename_all = "camelCase", default)] @@ -172,6 +173,8 @@ struct CellData { effective_value: Option, formatted_value: Option, user_entered_format: Option, + text_format_runs: Vec, + chip_runs: Vec, data_validation: Option, note: Option, hyperlink: Option, @@ -226,8 +229,8 @@ impl ExtValue { pub fn fetch(auth: &Auth, id_or_url: &str, verbose: bool) -> Result { let id = crate::formula::spreadsheet_id(id_or_url); - let meta_fields = "spreadsheetId,spreadsheetUrl,properties(title,locale,timeZone),namedRanges,dataSources,developerMetadata,\ - sheets(properties,merges,conditionalFormats,protectedRanges,basicFilter,filterViews,charts,bandedRanges,developerMetadata)"; + let meta_fields = "spreadsheetId,spreadsheetUrl,properties(title,locale,timeZone,autoRecalc,iterativeCalculationSettings),namedRanges,dataSources,developerMetadata,\ + sheets(properties,merges,conditionalFormats,protectedRanges,basicFilter,filterViews,charts,bandedRanges,developerMetadata,slicers,tables,rowGroups,columnGroups)"; let meta: J = serde_json::from_str(&auth.get(&format!("{SHEETS}/{id}"), &[("fields", meta_fields)])?)?; let mut wb = Workbook { @@ -242,7 +245,9 @@ pub fn fetch(auth: &Auth, id_or_url: &str, verbose: bool) -> Result { }, ..Default::default() }; - // Drive metadata is best-effort: needs the drive scope and visibility. + wb.calc = calc(&meta["properties"]); + // Drive metadata and comments are best-effort: they need the drive scope and visibility. + let mut drive_ok = false; match auth.get( &format!("https://www.googleapis.com/drive/v3/files/{id}"), &[("fields", "modifiedTime,owners(emailAddress),lastModifyingUser(emailAddress)"), ("supportsAllDrives", "true")], @@ -251,8 +256,9 @@ pub fn fetch(auth: &Auth, id_or_url: &str, verbose: bool) -> Result { let d: J = serde_json::from_str(&t).unwrap_or_default(); wb.source.modified = d["modifiedTime"].as_str().map(Into::into); wb.source.owner = d["owners"][0]["emailAddress"].as_str().map(Into::into); + drive_ok = true; } - Err(e) => wb.warnings.push(format!("drive metadata unavailable: {}", short(&e.to_string()))), + Err(e) => wb.warnings.push(format!("drive metadata and comments unavailable: {}", short(&e.to_string()))), } let sheets_j = meta["sheets"].as_array().cloned().unwrap_or_default(); @@ -278,13 +284,18 @@ pub fn fetch(auth: &Auth, id_or_url: &str, verbose: bool) -> Result { frozen_rows: gp["frozenRowCount"].as_u64().unwrap_or(0) as u32, frozen_cols: gp["frozenColumnCount"].as_u64().unwrap_or(0) as u32, tab_color: color_hex(&p["tabColorStyle"]["rgbColor"]).or_else(|| color_hex(&p["tabColor"])), + no_gridlines: gp["hideGridlines"].as_bool().unwrap_or(false), ..Default::default() }; + sh.outlines = arr(&s["rowGroups"]).iter().chain(arr(&s["columnGroups"])).filter_map(outline).collect(); + sh.outlines.sort_by_key(|o| (o.cols, o.span.0, o.depth)); for m in arr(&s["merges"]) { sh.merges.push(gr(m)); } for cf in arr(&s["conditionalFormats"]) { - sh.cond_formats.push(CondFormat { ranges: arr(&cf["ranges"]).iter().map(gr).collect(), rule: cond_rule(cf) }); + // Formula values (`=$C2>100`, `=Inputs!B2`) are what the rule reads. + let formulas = cond_values(&cf["booleanRule"]["condition"]).into_iter().filter(|v| v.starts_with('=')).collect(); + sh.cond_formats.push(CondFormat { ranges: arr(&cf["ranges"]).iter().map(gr).collect(), rule: cond_rule(cf), formulas }); } if s["basicFilter"].is_object() { sh.filters.push(filter(&s["basicFilter"], None)); @@ -292,6 +303,12 @@ pub fn fetch(auth: &Auth, id_or_url: &str, verbose: bool) -> Result { for fv in arr(&s["filterViews"]) { sh.filters.push(filter(fv, fv["title"].as_str().map(Into::into))); } + for sl in arr(&s["slicers"]) { + sh.filters.push(slicer(&sl["spec"])); + } + for t in arr(&s["tables"]) { + sh.tables.push(table(t)); + } for pr in arr(&s["protectedRanges"]) { sh.protected.push(Protected { range: pr.get("range").map(gr), @@ -316,6 +333,7 @@ pub fn fetch(auth: &Auth, id_or_url: &str, verbose: bool) -> Result { wb.named_ranges.push(NamedRange { name: nr["name"].as_str().unwrap_or_default().into(), range: format!("{}!{}", quote_sheet(&sheet), grid_range(r)), + scope: None, }); } wb.data_sources = arr(&meta["dataSources"]).to_vec(); @@ -336,8 +354,9 @@ pub fn fetch(auth: &Auth, id_or_url: &str, verbose: bool) -> Result { } } let total = jobs.len(); - let fields = "sheets(data(startRow,startColumn,rowMetadata(hiddenByUser),columnMetadata(hiddenByUser),\ - rowData(values(userEnteredValue,effectiveValue,formattedValue,userEnteredFormat,dataValidation,note,hyperlink,pivotTable))))"; + let fields = "sheets(data(startRow,startColumn,rowMetadata(hiddenByUser,pixelSize),columnMetadata(hiddenByUser,pixelSize),\ + rowData(values(userEnteredValue,effectiveValue,formattedValue,userEnteredFormat,dataValidation,note,hyperlink,pivotTable,\ + textFormatRuns,chipRuns))))"; let names: Vec<(String, u32)> = wb.sheets.iter().map(|s| (s.name.clone(), s.grid_cols)).collect(); let queue = Mutex::new(jobs.into_iter()); let (tx, rx) = mpsc::channel::>(); @@ -388,10 +407,17 @@ pub fn fetch(auth: &Auth, id_or_url: &str, verbose: bool) -> Result { for sh in &mut wb.sheets { sh.cells.sort_by_key(|c| (c.r, c.c)); sh.hidden_rows = merge_spans(std::mem::take(&mut sh.hidden_rows)); + sh.row_heights = merge_sized(std::mem::take(&mut sh.row_heights)); } wb.styles = styles.items; wb.validations = dvs.items; canonicalize(&mut wb); + if drive_ok { + match drive_comments(auth, &id) { + Ok(c) => wb.comments = place_comments(&wb, c), + Err(e) => wb.warnings.push(format!("comments unavailable: {}", short(&e.to_string()))), + } + } wb.warnings.push( "Apps Script (bound scripts, macros, add-ons, triggers) is not exposed by the Sheets API; \ pass --apps-script with user credentials to include a project" @@ -400,6 +426,79 @@ pub fn fetch(auth: &Auth, id_or_url: &str, verbose: bool) -> Result { Ok(wb) } +/// Comment threads on the file, from the Drive API. Their anchors are opaque +/// ids, so [`place_comments`] finds the cell from the text it quotes. +fn drive_comments(auth: &Auth, id: &str) -> Result> { + let fields = "nextPageToken,comments(author(displayName),content,resolved,quotedFileContent(value),replies(author(displayName),content))"; + let mut out = Vec::new(); + let mut page = String::new(); + for _ in 0..20 { + let mut q = vec![("fields", fields), ("pageSize", "100")]; + if !page.is_empty() { + q.push(("pageToken", page.as_str())); + } + let j: J = serde_json::from_str(&auth.get(&format!("https://www.googleapis.com/drive/v3/files/{id}/comments"), &q)?)?; + out.extend(arr(&j["comments"]).iter().map(drive_comment)); + match j["nextPageToken"].as_str() { + Some(t) => page = t.to_string(), + None => break, + } + } + Ok(out) +} + +fn drive_comment(c: &J) -> Comment { + let who = |m: &J| m["author"]["displayName"].as_str().unwrap_or_default().to_string(); + Comment { + author: who(c), + text: c["content"].as_str().unwrap_or_default().trim().into(), + quoted: c["quotedFileContent"]["value"].as_str().map(unescape_html).filter(|q| !q.is_empty()), + cell: None, + resolved: c["resolved"].as_bool().unwrap_or(false), + // Resolving or reopening a thread adds a reply with no text. + replies: arr(&c["replies"]) + .iter() + .filter_map(|r| Some((who(r), r["content"].as_str().map(str::trim).filter(|t| !t.is_empty())?.to_string()))) + .collect(), + } +} + +/// A row or column group (DimensionGroup). +fn outline(g: &J) -> Option { + let r = &g["range"]; + let (a, b) = (r["startIndex"].as_u64().unwrap_or(0) as u32, r["endIndex"].as_u64()? as u32); + (b > a).then(|| Outline { + cols: r["dimension"] == "COLUMNS", + span: (a, b - 1), + depth: g["depth"].as_u64().unwrap_or(1) as u32, + collapsed: g["collapsed"].as_bool().unwrap_or(false), + }) +} + +/// Tie each comment to the one cell showing the text it quotes, if exactly one does. +fn place_comments(wb: &Workbook, mut comments: Vec) -> Vec { + let mut seen: HashMap<&str, (u32, Option)> = comments.iter().filter_map(|c| Some((c.quoted.as_deref()?, (0, None)))).collect(); + if !seen.is_empty() { + for sh in &wb.sheets { + for c in &sh.cells { + if let Some(hit) = seen.get_mut(c.display().as_str()) { + hit.0 += 1; + hit.1 = Some(format!("{}!{}", quote_sheet(&sh.name), crate::a1::cell_name(c.r, c.c))); + } + } + } + } + let cells: HashMap> = seen.into_iter().map(|(q, (n, at))| (q.to_string(), at.filter(|_| n == 1))).collect(); + for c in &mut comments { + c.cell = c.quoted.as_ref().and_then(|q| cells.get(q).cloned().flatten()); + } + comments +} + +fn unescape_html(s: &str) -> String { + s.replace("<", "<").replace(">", ">").replace(""", "\"").replace("'", "'").replace("&", "&") +} + /// Fetch an Apps Script project's source (needs user OAuth with script.projects.readonly). pub fn fetch_apps_script(auth: &Auth, script_id: &str) -> Result> { let body = auth.get(&format!("https://script.googleapis.com/v1/projects/{script_id}/content"), &[])?; @@ -427,12 +526,19 @@ fn absorb( if m.hidden_by_user { push_span(&mut sh.hidden_rows, sr + i as u32); } + if let Some(px) = m.pixel_size.filter(|p| *p != DEFAULT_ROW_PX) { + push_sized(&mut sh.row_heights, sr + i as u32, px); + } } - if sh.hidden_cols.is_empty() && sr == 0 { + // Every chunk spans all columns; the first one reports them. + if sr == 0 { for (i, m) in gd.column_metadata.iter().enumerate() { if m.hidden_by_user { push_span(&mut sh.hidden_cols, sc + i as u32); } + if let Some(px) = m.pixel_size.filter(|p| *p != DEFAULT_COL_PX) { + push_sized(&mut sh.col_widths, sc + i as u32, px); + } } } for (ri, row) in gd.row_data.into_iter().enumerate() { @@ -454,7 +560,13 @@ fn absorb( }; let s = cd.user_entered_format.as_ref().map(style_of).filter(|s| *s != Style::default()).map(|s| styles.intern(s)); let dv = cd.data_validation.as_ref().map(|d| dvs.intern(validation(d))); - let cell = Cell { r, c, v, f, t, s, dv, note: cd.note, link: cd.hyperlink }; + let text = cd.formatted_value.as_deref().or(match &v { + Value::Str(s) => Some(s.as_str()), + _ => None, + }); + let runs = text.map(|t| runs(t, &cd.text_format_runs)).unwrap_or_default(); + let obj = chip(text, &cd.chip_runs); + let cell = Cell { r, c, v, f, t, s, dv, note: cd.note, link: cd.hyperlink, runs, obj }; if cell.is_blank() && cell.s.is_none() && cell.dv.is_none() && cell.note.is_none() { continue; } @@ -463,6 +575,23 @@ fn absorb( } } +/// Google's default row height and column width. +const DEFAULT_ROW_PX: u32 = 21; +const DEFAULT_COL_PX: u32 = 100; + +/// Sizes from chunks fetched in any order, sorted and joined again. +fn merge_sized(mut v: Vec<(u32, u32, u32)>) -> Vec<(u32, u32, u32)> { + v.sort(); + let mut out: Vec<(u32, u32, u32)> = Vec::new(); + for (a, b, px) in v { + match out.last_mut() { + Some(l) if l.1 + 1 == a && l.2 == px => l.1 = b, + _ => out.push((a, b, px)), + } + } + out +} + fn merge_spans(mut v: Vec<(u32, u32)>) -> Vec<(u32, u32)> { v.sort(); let mut out: Vec<(u32, u32)> = Vec::new(); @@ -530,7 +659,20 @@ fn style_of(f: &J) -> Style { let fg = color_of(&tf["foregroundColorStyle"], &tf["foregroundColor"]).filter(|c| c != "#000000"); let font = tf["fontFamily"].as_str().filter(|f| *f != "Arial").map(Into::into); let size = tf["fontSize"].as_u64().map(|s| s as u32).filter(|s| *s != 10); - let borders = f["borders"].as_object().map(|b| b.values().any(|v| v["style"].as_str().is_some_and(|s| s != "NONE"))).unwrap_or(false); + // Line names as Excel's, so both sources read alike. + let sides: Vec<(&str, String)> = ["top", "bottom", "left", "right"] + .into_iter() + .filter_map(|k| { + let line = match f["borders"][k]["style"].as_str()? { + "NONE" => return None, + "SOLID" => "thin", + "SOLID_MEDIUM" => "medium", + "SOLID_THICK" => "thick", + other => return Some((k, other.to_ascii_lowercase())), + }; + Some((k, line.to_string())) + }) + .collect(); Style { num_fmt: f["numberFormat"]["pattern"].as_str().map(Into::into), num_type: f["numberFormat"]["type"].as_str().map(Into::into), @@ -540,12 +682,68 @@ fn style_of(f: &J) -> Style { italic: tf["italic"].as_bool().unwrap_or(false), strike: tf["strikethrough"].as_bool().unwrap_or(false), underline: tf["underline"].as_bool().unwrap_or(false), - borders, + border: crate::model::border_desc(&sides), h_align: f["horizontalAlignment"].as_str().filter(|a| *a != "LEFT").map(Into::into), font, size, wrap: f["wrapStrategy"].as_str() == Some("WRAP"), + ..Default::default() + } +} + +/// Recalculation settings, when they differ from Google's default (on change, no iteration). +fn calc(p: &J) -> Option { + let mode = match p["autoRecalc"].as_str() { + Some("MINUTE") => "on change and every minute", + Some("HOUR") => "on change and every hour", + _ => "on change", + }; + let it = &p["iterativeCalculationSettings"]; + let iterative = it.is_object(); + if !iterative && mode == "on change" { + return None; } + Some(Calc { + mode: mode.into(), + iterative, + max_iterations: it["maxIterations"].as_u64().map(|n| n as u32), + max_change: it["convergenceThreshold"].as_f64(), + }) +} + +/// A cell's text split where its formatting changes (`textFormatRuns`). +/// Indexes count UTF-16 code units. +fn runs(text: &str, fmt_runs: &[J]) -> Vec { + let units: Vec = text.encode_utf16().collect(); + let mut out: Vec = Vec::new(); + for (i, r) in fmt_runs.iter().enumerate() { + let start = r["startIndex"].as_u64().unwrap_or(0) as usize; + let end = fmt_runs.get(i + 1).and_then(|n| n["startIndex"].as_u64()).map_or(units.len(), |e| e as usize); + let Some(slice) = units.get(start.min(units.len())..end.min(units.len())).filter(|s| !s.is_empty()) else { continue }; + let f = style_of(&json!({ "textFormat": r["format"] })); + out.push(Run { text: String::from_utf16_lossy(slice), f }); + } + if !runs_differ(&out) { + return Vec::new(); + } + out +} + +/// Smart chips: people, Drive files and other links. +fn chip(text: Option<&str>, chips: &[J]) -> Option { + let mut objs = chips.iter().filter_map(|c| { + let c = &c["chip"]; + let (kind, target) = if let Some(email) = c["personProperties"]["email"].as_str() { + ("person", email) + } else { + let uri = c["richLinkProperties"]["uri"].as_str()?; + (if uri.contains("docs.google.com") || uri.contains("drive.google.com") { "file" } else { "link" }, uri) + }; + Some((kind, target.to_string())) + }); + let (kind, target) = objs.next()?; + let props = objs.map(|(k, t)| (k.to_string(), t)).collect(); + Some(CellObject { kind: kind.into(), text: text.map(Into::into), target: Some(target), props }) } fn cond_values(cond: &J) -> Vec { @@ -611,24 +809,41 @@ fn cond_rule(cf: &J) -> String { } fn filter(f: &J, title: Option) -> Filter { - let mut criteria = Vec::new(); - for spec in arr(&f["filterSpecs"]) { - let col = spec["columnIndex"].as_u64().unwrap_or(0) as u32; - let c = &spec["filterCriteria"]; - let mut bits = Vec::new(); - if let Some(h) = c["hiddenValues"].as_array() { - bits.push(format!("hide {} values", h.len())); - } - if c["condition"].is_object() { - bits.push(format!("{} {}", c["condition"]["type"].as_str().unwrap_or("?"), cond_values(&c["condition"]).join(","))); - } - if !bits.is_empty() { - criteria.push(format!("{}: {}", col_name(col), bits.join("; "))); - } - } + let criteria = arr(&f["filterSpecs"]).iter().filter_map(|spec| criterion(spec["columnIndex"].as_u64().unwrap_or(0) as u32, &spec["filterCriteria"])).collect(); Filter { range: grid_range(&f["range"]), title, criteria } } +/// One column's filter criteria, e.g. `C: hide 3 values`. +fn criterion(col: u32, c: &J) -> Option { + let mut bits = Vec::new(); + if let Some(h) = c["hiddenValues"].as_array() { + bits.push(format!("hide {} values", h.len())); + } + if c["condition"].is_object() { + bits.push(format!("{} {}", c["condition"]["type"].as_str().unwrap_or("?"), cond_values(&c["condition"]).join(","))); + } + (!bits.is_empty()).then(|| format!("{}: {}", col_name(col), bits.join("; "))) +} + +/// A slicer is a filter control on one column of a range. +fn slicer(spec: &J) -> Filter { + let range = grid_range(&spec["dataRange"]); + let col = range.c0 + spec["columnIndex"].as_u64().unwrap_or(0) as u32; + let title = spec["title"].as_str().filter(|t| !t.is_empty()).map_or_else(|| format!("slicer on {}", col_name(col)), |t| format!("slicer {t}")); + Filter { range, title: Some(title), criteria: criterion(col, &spec["filterCriteria"]).into_iter().collect() } +} + +/// A Google Sheets table (Format → Convert to table). +fn table(t: &J) -> Table { + Table { + name: t["name"].as_str().unwrap_or_default().into(), + range: grid_range(&t["range"]), + columns: arr(&t["columnProperties"]).iter().map(|c| c["columnName"].as_str().unwrap_or_default().to_string()).collect(), + // The API doesn't say whether the last row is a totals row. + totals: false, + } +} + fn chart(c: &J, id_to_name: &HashMap) -> Chart { let spec = &c["spec"]; let kind = spec @@ -687,3 +902,89 @@ fn pivot(p: &J, anchor: (u32, u32), id_to_name: &HashMap) -> Pivot filters: arr(&p["filterSpecs"]).iter().map(col).collect(), } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn text_runs_and_chips() { + // Shapes as the Sheets API returns them; indexes are UTF-16 units. + let fmt = json!([ + {"startIndex": 0, "format": {}}, + {"startIndex": 6, "format": {"bold": true, "foregroundColorStyle": {"rgbColor": {"red": 1}}}}, + ]); + let r = runs("Café: due", fmt.as_array().unwrap()); + assert_eq!(r.iter().map(|r| r.text.as_str()).collect::>(), ["Café: ", "due"]); + assert!(r[1].f.bold && r[1].f.fg.as_deref() == Some("#FF0000")); + let same = json!([{"startIndex": 0, "format": {"italic": true}}, {"startIndex": 2, "format": {"italic": true}}]); + assert!(runs("abcd", same.as_array().unwrap()).is_empty()); + + let chips = json!([ + {"startIndex": 0, "chip": {"personProperties": {"email": "ana@example.com", "displayFormat": "DEFAULT"}}}, + {"startIndex": 4, "chip": {"richLinkProperties": {"uri": "https://docs.google.com/document/d/abc", "mimeType": "application/vnd.google-apps.document"}}}, + ]); + let o = chip(Some("Ana and the plan"), chips.as_array().unwrap()).unwrap(); + assert_eq!((o.kind.as_str(), o.target.as_deref()), ("person", Some("ana@example.com"))); + assert_eq!(o.props, [("file".to_string(), "https://docs.google.com/document/d/abc".to_string())]); + + assert_eq!(calc(&json!({"autoRecalc": "ON_CHANGE"})), None); + let c = calc(&json!({"autoRecalc": "HOUR", "iterativeCalculationSettings": {"maxIterations": 50, "convergenceThreshold": 0.01}})).unwrap(); + assert_eq!((c.mode.as_str(), c.iterative, c.max_iterations, c.max_change), ("on change and every hour", true, Some(50), Some(0.01))); + } + + #[test] + fn borders_slicers_and_tables() { + let st = style_of(&json!({"borders": {"top": {"style": "SOLID"}, "bottom": {"style": "DOUBLE"}, "left": {"style": "NONE"}}})); + assert_eq!(st.border.as_deref(), Some("top thin, bottom double")); + let all = json!({"style": "SOLID_MEDIUM"}); + assert_eq!(style_of(&json!({"borders": {"top": all, "bottom": all, "left": all, "right": all}})).border.as_deref(), Some("box medium")); + + let sl = slicer(&json!({ + "dataRange": {"sheetId": 0, "startRowIndex": 0, "endRowIndex": 50, "startColumnIndex": 1, "endColumnIndex": 5}, + "columnIndex": 2, "title": "Region", + "filterCriteria": {"hiddenValues": ["East", "West"]}, + })); + assert_eq!((sl.range.a1(), sl.title.as_deref(), sl.criteria.as_slice()), ("B1:E50".to_string(), Some("slicer Region"), ["D: hide 2 values".to_string()].as_slice())); + + let t = table(&json!({ + "tableId": "t1", "name": "Orders", + "range": {"sheetId": 0, "startRowIndex": 0, "endRowIndex": 10, "startColumnIndex": 0, "endColumnIndex": 2}, + "columnProperties": [{"columnIndex": 0, "columnName": "Order"}, {"columnIndex": 1, "columnName": "Amount", "columnType": "CURRENCY"}], + })); + assert_eq!((t.name.as_str(), t.range.a1(), t.columns.as_slice()), ("Orders", "A1:B10".to_string(), ["Order".to_string(), "Amount".to_string()].as_slice())); + + let cond = json!({"type": "CUSTOM_FORMULA", "values": [{"userEnteredValue": "=$C2>Inputs!B1"}]}); + assert_eq!(cond_values(&cond), ["=$C2>Inputs!B1"]); + } + + #[test] + fn layout_groups_and_comments() { + // Sizes and hidden rows from a chunk's dimension metadata. + let mut sh = Sheet { name: "Plan".into(), ..Default::default() }; + let gd: GridData = serde_json::from_value(json!({ + "rowMetadata": [{"pixelSize": 21}, {"pixelSize": 40}, {"pixelSize": 40}, {"pixelSize": 21, "hiddenByUser": true}], + "columnMetadata": [{"pixelSize": 100}, {"pixelSize": 12}, {"pixelSize": 250}], + "rowData": [{"values": [{"formattedValue": "45", "effectiveValue": {"numberValue": 45}}, {"formattedValue": "Total", "effectiveValue": {"stringValue": "Total"}}]}, + {"values": [{"formattedValue": "45", "effectiveValue": {"numberValue": 45}}]}], + })) + .unwrap(); + absorb(&mut sh, 0, gd, &mut Interner::default(), &mut Interner::default(), &HashMap::new()); + assert_eq!((sh.row_heights.as_slice(), sh.col_widths.as_slice(), sh.hidden_rows.as_slice()), ([(1, 2, 40)].as_slice(), [(1, 1, 12), (2, 2, 250)].as_slice(), [(3, 3)].as_slice())); + assert_eq!(merge_sized(vec![(5, 6, 40), (0, 1, 40), (2, 4, 40), (7, 7, 30)]), [(0, 6, 40), (7, 7, 30)]); + + let g = outline(&json!({"range": {"dimension": "COLUMNS", "startIndex": 3, "endIndex": 6}, "depth": 2, "collapsed": true})).unwrap(); + assert_eq!(g, Outline { cols: true, span: (3, 5), depth: 2, collapsed: true }); + + let wb = Workbook { sheets: vec![sh], ..Default::default() }; + let c = |q: &str| drive_comment(&json!({ + "author": {"displayName": "Ana"}, "content": "why? ", "resolved": true, + "quotedFileContent": {"mimeType": "text/html", "value": q}, + "replies": [{"author": {"displayName": "Bo"}, "content": "fixed"}, {"author": {"displayName": "Bo"}, "content": ""}], + })); + let placed = place_comments(&wb, vec![c("Total"), c("45"), c("gone")]); + let at: Vec> = placed.iter().map(|c| c.cell.as_deref()).collect(); + assert_eq!(at, [Some("Plan!B1"), None, None], "45 shows twice; nothing shows `gone`"); + assert_eq!(crate::query::comment_text(&placed[0]), "Ana: why? ↳ Bo: fixed (resolved)"); + } +} diff --git a/src/lib.rs b/src/lib.rs index 1db0156..4fa20f0 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -10,7 +10,9 @@ pub mod analyze; pub mod formula; pub mod graph; pub mod gsheets; +pub mod macros; pub mod model; +pub mod numfmt; pub mod ooxml; pub mod query; pub mod report; diff --git a/src/macros.rs b/src/macros.rs new file mode 100644 index 0000000..f9b0a37 --- /dev/null +++ b/src/macros.rs @@ -0,0 +1,874 @@ +//! Macro code: the procedures in each script module and, for VBA, which sheets +//! and ranges each one reads or changes, which procedures it calls, and which +//! event runs it. +//! +//! This reads the source without running it, so it follows what is written +//! out: `Worksheets("Data").Range("A1:B5")`, a sheet or range held in a +//! variable (`Set ws = …`), a `With` block, a sheet passed to another +//! procedure's parameter, a code name (`Sheet1.Range`), a named range. An +//! address built at run time (`Range("A" & i)`, `Cells(i, 2)`) counts as +//! somewhere on that sheet. + +use crate::a1::{col_index, parse_rect, quote_sheet, Rect}; +use crate::analyze::{sheet_index, Analysis}; +use crate::model::Workbook; +use std::collections::{BTreeMap, BTreeSet, HashMap}; + +#[derive(Debug, Clone)] +pub struct Proc { + /// Index into `wb.scripts`. + pub script: usize, + pub name: String, + /// "sub" | "function" | "property" + pub kind: &'static str, + /// First and last line in the module, 1-based. + pub lines: (usize, usize), + pub trigger: Option, + /// Sorted by (sheet, range). + pub touches: Vec, + /// Indices into the procedure list, sorted. + pub calls: Vec, + /// Declared `As Worksheet`: a helper that hands back a sheet by name. + pub returns_sheet: bool, +} + +/// An event that runs a procedure without anyone calling it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Trigger { + /// "Open", "Change", "BeforeClose", … (Apps Script: "onEdit", "doGet", …) + pub event: String, + /// The sheet whose event it is; None for the workbook. + pub sheet: Option, +} + +/// The sheet a reference lands on. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum At { + Sheet(usize), + /// Named in the code but not in the file: one the macro creates, or one + /// since deleted or renamed. + Missing(String), + /// Whichever sheet is active when it runs. + Active, +} + +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Touch { + pub at: At, + /// None: somewhere on the sheet (an address built at run time). + pub rect: Option, + /// Assigns to it or changes it (`.Value = …`, `.ClearContents`, `.Delete`, …). + pub write: bool, +} + +/// Methods that change the range or sheet they're called on. +const MUTATORS: &[&str] = &[ + "add", "autofill", "autofit", "clear", "clearcomments", "clearcontents", "clearformats", "copypicture", "delete", "filldown", + "fillright", "insert", "merge", "pastespecial", "removeduplicates", "replace", "sort", "texttocolumns", "unmerge", +]; +/// Members that move away from the literal address, so only the sheet is known. +const RELATIVE: &[&str] = &["columns", "currentregion", "end", "entirecolumn", "entirerow", "offset", "resize", "rows", "specialcells", "usedrange"]; +/// Statements whose `=` compares rather than assigns. +const NO_ASSIGN: &[&str] = &[ + "call", "case", "debug", "do", "exit", "for", "goto", "loop", "msgbox", "next", "on", "resume", "return", "select", "until", "wend", "while", +]; +/// Apps Script simple triggers and web-app entry points. +const JS_TRIGGERS: &[&str] = &["onOpen", "onEdit", "onChange", "onSelectionChange", "onFormSubmit", "onInstall", "doGet", "doPost"]; + +pub fn analyze(wb: &Workbook, a: &Analysis) -> Vec { + let mut procs = Vec::new(); + let mut bodies: Vec>> = Vec::new(); + let mut params: Vec> = Vec::new(); + let code_names: HashMap = + wb.sheets.iter().enumerate().filter_map(|(i, s)| Some((s.code_name.as_ref()?.to_ascii_lowercase(), i))).collect(); + // String constants stand in for sheet names and addresses + // (`Const RAW As String = "Raw Data"`); a module's own come first. + let mut global_consts = HashMap::new(); + let mut module_consts = Vec::new(); + for sc in &wb.scripts { + let mut mine = HashMap::new(); + if sc.kind == "vba" { + for (_, st) in statements(&sc.source) { + if let Some((name, value, public)) = string_const(&st) { + if public { + global_consts.entry(name.clone()).or_insert_with(|| value.clone()); + } + mine.insert(name, value); + } + } + } + module_consts.push(mine); + } + for (si, sc) in wb.scripts.iter().enumerate() { + if sc.kind == "vba" { + let me = code_names.get(&sc.name.to_ascii_lowercase()).copied(); + let mut open: Option<(Proc, Vec, Vec>)> = None; + let mut last = 0; + for (line, mut st) in statements(&sc.source) { + last = line; + for t in &mut st { + if let Tok::Id(w) = t { + let lw = w.to_ascii_lowercase(); + if let Some(v) = module_consts[si].get(&lw).or_else(|| global_consts.get(&lw)) { + *t = Tok::Str(v.clone()); + } + } + } + if let Some((kind, name, ps, returns_sheet)) = proc_header(&st) { + let trigger = vba_trigger(&name, me); + let p = Proc { script: si, name, kind, lines: (line, line), trigger, touches: Vec::new(), calls: Vec::new(), returns_sheet }; + open = Some((p, ps, Vec::new())); + } else if id_is(st.first(), "end") && ["sub", "function", "property"].iter().any(|k| id_is(st.get(1), k)) { + if let Some((mut p, ps, body)) = open.take() { + p.lines.1 = line; + procs.push(p); + params.push(ps); + bodies.push(body); + } + } else if let Some((_, _, body)) = &mut open { + body.push(st); + } + } + // A module cut off before `End Sub` still has the procedure. + if let Some((mut p, ps, body)) = open { + p.lines.1 = last.max(p.lines.0); + procs.push(p); + params.push(ps); + bodies.push(body); + } + } else { + let total = sc.source.lines().count(); + let start = procs.len(); + for (n, line) in sc.source.lines().enumerate() { + if let Some(name) = js_function(line) { + let trigger = JS_TRIGGERS.contains(&name.as_str()).then(|| Trigger { event: name.clone(), sheet: None }); + procs.push(Proc { script: si, name, kind: "function", lines: (n + 1, total), trigger, touches: Vec::new(), calls: Vec::new(), returns_sheet: false }); + params.push(Vec::new()); + bodies.push(Vec::new()); + } + } + // Each function runs until the next one starts. + let starts: Vec = procs[start..].iter().skip(1).map(|p| p.lines.0).collect(); + for (p, next) in procs[start..].iter_mut().zip(starts) { + p.lines.1 = next - 1; + } + } + } + + let mut by_name: HashMap> = HashMap::new(); + for (i, p) in procs.iter().enumerate() { + by_name.entry(p.name.to_ascii_lowercase()).or_default().push(i); + } + // Sheets passed as arguments bind the callee's parameters, which can pass + // them on again: repeat until no binding grows. + let mut bindings: Vec> = vec![HashMap::new(); procs.len()]; + for _ in 0..10 { + let mut next = bindings.clone(); + for (pi, body) in bodies.iter().enumerate() { + if wb.scripts[procs[pi].script].kind != "vba" { + continue; + } + let me = code_names.get(&wb.scripts[procs[pi].script].name.to_ascii_lowercase()).copied(); + let mut sc = Scope { wb, a, code_names: &code_names, by_name: &by_name, procs: &procs, me, this: pi, vars: bindings[pi].clone(), with: Vec::new() }; + let mut out = Out::default(); + for st in body { + sc.statement(st, &mut out); + } + let mut touches: BTreeMap<(At, Option), bool> = BTreeMap::new(); + for t in out.touches { + *touches.entry((t.at, t.rect)).or_default() |= t.write; + } + procs[pi].touches = touches.into_iter().map(|((at, rect), write)| Touch { at, rect, write }).collect(); + procs[pi].calls = out.calls.iter().map(|(c, _)| *c).collect::>().into_iter().collect(); + for (c, args) in out.calls { + for (k, b) in args.into_iter().enumerate() { + if let (Some(b), Some(p)) = (b, params[c].get(k)) { + next[c].entry(p.clone()).or_default().merge(&b); + } + } + } + } + if next == bindings { + break; + } + bindings = next; + } + procs +} + +/// Formula cells that call this procedure (a VBA function used as a worksheet function). +pub fn formula_cells(a: &Analysis, p: &Proc) -> u64 { + if p.kind != "function" { + return 0; + } + a.custom_funcs.iter().filter(|(f, _)| f.eq_ignore_ascii_case(&p.name)).map(|(_, n)| n).sum() +} + +fn sheet_label(wb: &Workbook, at: &At) -> String { + match at { + At::Sheet(s) => quote_sheet(&wb.sheets[*s].name), + At::Missing(n) => quote_sheet(n), + At::Active => "ActiveSheet".into(), + } +} + +/// `Sheet!A1`, `Sheet`, `Out!A1 (not in file)`, `ActiveSheet!A1`. +pub fn place(wb: &Workbook, t: &Touch) -> String { + let mut s = sheet_label(wb, &t.at); + if let Some(r) = t.rect { + s += &format!("!{r}"); + } + if matches!(t.at, At::Missing(_)) { + s += " (not in file)"; + } + s +} + +/// One line on what a procedure does: its trigger, formula use, the ranges it +/// writes and uses, the procedures it calls. `max` caps each list. None when +/// there's nothing to say beyond its name. +/// Whether a shape's `on_click` (`RollUp` or `Module1.RollUp`) names this procedure. +pub fn clicks(on_click: &str, wb: &Workbook, p: &Proc) -> bool { + match on_click.rsplit_once('.') { + Some((m, n)) => n.eq_ignore_ascii_case(&p.name) && m.eq_ignore_ascii_case(&wb.scripts[p.script].name), + None => on_click.eq_ignore_ascii_case(&p.name), + } +} + +pub fn describe(wb: &Workbook, a: &Analysis, p: &Proc, max: usize) -> Option { + let list = |v: Vec, sep: &str| { + let more = v.len().saturating_sub(max); + let mut s = v.into_iter().take(max).collect::>().join(sep); + if more > 0 { + s += &format!(" +{more} more"); + } + s + }; + let mut parts = Vec::new(); + if let Some(t) = &p.trigger { + match t.sheet { + Some(s) => parts.push(format!("runs on {} {}", quote_sheet(&wb.sheets[s].name), t.event)), + None if wb.scripts[p.script].kind == "vba" => parts.push(format!("runs on workbook {}", t.event)), + None => parts.push(format!("trigger {}", t.event)), + } + } + for sh in &wb.sheets { + for x in sh.shapes.iter().filter(|x| x.on_click.as_deref().is_some_and(|m| clicks(m, wb, p))) { + let label = x.text.as_deref().or(Some(x.name.as_str()).filter(|n| !n.is_empty())).map(|t| format!(" \"{}\"", crate::report::trunc(t, 40))); + parts.push(format!("run by {}{} on {}", x.kind, label.unwrap_or_default(), quote_sheet(&sh.name))); + } + } + let n = formula_cells(a, p); + if n > 0 { + parts.push(format!("used in {n} formula cell{}", if n == 1 { "" } else { "s" })); + } + // Per sheet: `Sheet!A1,B2:C5` (ranges inside another listed one dropped), + // with `,…` when it also reaches cells whose address is built at run time. + for (write, verb) in [(true, "writes"), (false, "uses")] { + let mut by_sheet: BTreeMap<&At, (bool, Vec)> = BTreeMap::new(); + for t in p.touches.iter().filter(|t| t.write == write) { + let e = by_sheet.entry(&t.at).or_default(); + match t.rect { + Some(r) => e.1.push(r), + None => e.0 = true, + } + } + let inside = |r: &Rect, o: &Rect| o != r && o.r0 <= r.r0 && r.r1 <= o.r1 && o.c0 <= r.c0 && r.c1 <= o.c1; + let sheets: Vec = by_sheet + .into_iter() + .map(|(at, (whole, rects))| { + let rects: Vec = rects.iter().filter(|r| !rects.iter().any(|o| inside(r, o))).map(|r| r.a1()).collect(); + let mut s = sheet_label(wb, at); + if !rects.is_empty() { + let more = rects.len().saturating_sub(max); + s += &format!("!{}", rects.into_iter().take(max).collect::>().join(",")); + if more > 0 { + s += &format!(",+{more}"); + } + if whole { + s += ",…"; + } + } + if matches!(at, At::Missing(_)) { + s += " (not in file)"; + } + s + }) + .collect(); + if !sheets.is_empty() { + parts.push(format!("{verb} {}", list(sheets, "; "))); + } + } + if !p.calls.is_empty() { + parts.push(format!("calls {}", list(p.calls.iter().map(|&c| a.macros[c].name.clone()).collect(), ", "))); + } + (!parts.is_empty()).then(|| parts.join(" · ")) +} + +fn vba_trigger(name: &str, me: Option) -> Option { + let lower = name.to_ascii_lowercase(); + if let Some(ev) = lower.strip_prefix("workbook_") { + return Some(Trigger { event: name[name.len() - ev.len()..].to_string(), sheet: None }); + } + if let Some(ev) = lower.strip_prefix("worksheet_") + && me.is_some() + { + return Some(Trigger { event: name[name.len() - ev.len()..].to_string(), sheet: me }); + } + match lower.as_str() { + "auto_open" => Some(Trigger { event: "Open".into(), sheet: None }), + "auto_close" => Some(Trigger { event: "BeforeClose".into(), sheet: None }), + _ => None, + } +} + +fn js_function(line: &str) -> Option { + let t = line.trim_start(); + let rest = ["async function ", "function "].iter().find_map(|k| t.strip_prefix(k))?; + let name: String = rest.trim_start().chars().take_while(|c| c.is_alphanumeric() || *c == '_' || *c == '$').collect(); + (!name.is_empty()).then_some(name) +} + +// ---------------------------------------------------------------- tokens + +#[derive(Debug, Clone, PartialEq)] +enum Tok { + Id(String), + Str(String), + Num(f64), + Op(String), +} + +fn id_is(t: Option<&Tok>, w: &str) -> bool { + matches!(t, Some(Tok::Id(s)) if s.eq_ignore_ascii_case(w)) +} + +fn op_is(t: Option<&Tok>, o: &str) -> bool { + matches!(t, Some(Tok::Op(s)) if s == o) +} + +/// One line of VBA, up to its comment. +fn tokenize(line: &str) -> Vec { + let cs: Vec = line.chars().collect(); + let mut out = Vec::new(); + let mut i = 0; + while i < cs.len() { + let c = cs[i]; + if c.is_whitespace() { + i += 1; + } else if c == '\'' { + break; + } else if c == '"' { + let mut s = String::new(); + i += 1; + while i < cs.len() { + if cs[i] == '"' { + if cs.get(i + 1) == Some(&'"') { + s.push('"'); + i += 2; + continue; + } + i += 1; + break; + } + s.push(cs[i]); + i += 1; + } + out.push(Tok::Str(s)); + } else if c.is_ascii_digit() { + let st = i; + while i < cs.len() && (cs[i].is_ascii_digit() || cs[i] == '.') { + i += 1; + } + out.push(Tok::Num(cs[st..i].iter().collect::().parse().unwrap_or(f64::NAN))); + } else if c.is_alphabetic() || c == '_' { + let st = i; + while i < cs.len() && (cs[i].is_alphanumeric() || cs[i] == '_') { + i += 1; + } + let id: String = cs[st..i].iter().collect(); + if cs.get(i) == Some(&'$') { + i += 1; // Left$(…) + } + if id.eq_ignore_ascii_case("rem") && out.is_empty() { + break; + } + out.push(Tok::Id(id)); + } else { + let two: String = cs[i..(i + 2).min(cs.len())].iter().collect(); + if [":=", "<=", ">=", "<>"].contains(&two.as_str()) { + out.push(Tok::Op(two)); + i += 2; + } else { + out.push(Tok::Op(c.to_string())); + i += 1; + } + } + } + out +} + +/// A module's statements with the line each starts on: continuation lines +/// (` _`) joined, `:`-separated statements split. +fn statements(src: &str) -> Vec<(usize, Vec)> { + let mut out = Vec::new(); + let mut cur: Vec = Vec::new(); + let mut start = 0; + for (n, line) in src.lines().enumerate() { + if cur.is_empty() { + start = n + 1; + } + let mut toks = tokenize(line); + let cont = matches!(toks.last(), Some(Tok::Id(s)) if s == "_"); + if cont { + toks.pop(); + } + cur.extend(toks); + if !cont { + for st in std::mem::take(&mut cur).split(|t| op_is(Some(t), ":")) { + if !st.is_empty() { + out.push((start, st.to_vec())); + } + } + } + } + if !cur.is_empty() { + out.push((start, cur)); + } + out +} + +/// `[Public|Private] Const Name [As String] = "text"`: (lowercased name, text, public). +fn string_const(t: &[Tok]) -> Option<(String, String, bool)> { + let public = id_is(t.first(), "public") || id_is(t.first(), "global"); + let i = usize::from(public || id_is(t.first(), "private")); + if !id_is(t.get(i), "const") { + return None; + } + let Some(Tok::Id(name)) = t.get(i + 1) else { return None }; + let eq = t.iter().position(|x| op_is(Some(x), "="))?; + match &t[eq + 1..] { + [Tok::Str(v)] => Some((name.to_ascii_lowercase(), v.clone(), public)), + _ => None, + } +} + +/// `[Public|Private|Friend|Static] Sub|Function|Property Get|Let|Set Name(params)`. +fn proc_header(t: &[Tok]) -> Option<(&'static str, String, Vec, bool)> { + let mut i = 0; + while ["public", "private", "friend", "static"].iter().any(|k| id_is(t.get(i), k)) { + i += 1; + } + let kind = if id_is(t.get(i), "sub") { + "sub" + } else if id_is(t.get(i), "function") { + "function" + } else if id_is(t.get(i), "property") { + i += 1; + "property" + } else { + return None; + }; + let Some(Tok::Id(name)) = t.get(i + 1) else { return None }; + let mut params = Vec::new(); + let mut close = i + 1; + if op_is(t.get(i + 2), "(") { + close = close_paren(t, i + 2); + for arg in split_args(&t[i + 3..close]) { + let mut j = 0; + while ["optional", "byval", "byref", "paramarray"].iter().any(|k| id_is(arg.get(j), k)) { + j += 1; + } + if let Some(Tok::Id(p)) = arg.get(j) { + params.push(p.to_ascii_lowercase()); + } + } + } + let sheet = id_is(t.get(close + 1), "as") && id_is(t.get(close + 2), "worksheet"); + Some((kind, name.clone(), params, sheet)) +} + +/// Index of the `)` closing the `(` at `open`, or the end. +fn close_paren(t: &[Tok], open: usize) -> usize { + let mut depth = 0; + for (i, tok) in t.iter().enumerate().skip(open) { + if op_is(Some(tok), "(") { + depth += 1; + } else if op_is(Some(tok), ")") { + depth -= 1; + if depth == 0 { + return i; + } + } + } + t.len() +} + +fn split_args(t: &[Tok]) -> Vec<&[Tok]> { + if t.is_empty() { + return Vec::new(); + } + let mut out = Vec::new(); + let (mut depth, mut start) = (0i32, 0); + for (i, tok) in t.iter().enumerate() { + match tok { + Tok::Op(o) if o == "(" => depth += 1, + Tok::Op(o) if o == ")" => depth -= 1, + Tok::Op(o) if o == "," && depth == 0 => { + out.push(&t[start..i]); + start = i + 1; + } + _ => {} + } + } + out.push(&t[start..]); + out +} + +// ---------------------------------------------------------------- resolution + +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)] +struct Place { + at: At, + rect: Option, +} + +/// What a variable, parameter or expression refers to. +#[derive(Debug, Clone, Default, PartialEq)] +struct Binding { + places: BTreeSet, + /// A range rather than a whole sheet, so using it bare reads or writes cells. + range: bool, +} + +impl Binding { + fn sheet(at: At) -> Binding { + Binding { places: [Place { at, rect: None }].into(), range: false } + } + fn merge(&mut self, o: &Binding) { + self.places.extend(o.places.iter().cloned()); + self.range |= o.range; + } +} + +struct Hit { + b: Binding, + write: bool, + /// Reaches cells or the sheet itself, not just names a sheet to pass on. + counts: bool, + end: usize, +} + +#[derive(Default)] +struct Out { + touches: Vec, + /// (callee, what each argument refers to) + calls: Vec<(usize, Vec>)>, +} + +struct Scope<'a> { + wb: &'a Workbook, + a: &'a Analysis, + code_names: &'a HashMap, + by_name: &'a HashMap>, + procs: &'a [Proc], + /// The sheet whose module this is: `Me`, and where a bare `Range` points. + me: Option, + this: usize, + vars: HashMap, + with: Vec, +} + +impl Scope<'_> { + fn statement(&mut self, t: &[Tok], out: &mut Out) { + let head = match t.first() { + Some(Tok::Id(w)) => w.to_ascii_lowercase(), + _ => String::new(), + }; + match head.as_str() { + // The target of `With` / `Set` is reached through what follows, + // so only calls in it are recorded here. + "with" => { + let b = self.whole(&t[1..]); + self.scan(&t[1..], None, out, b.is_none()); + self.with.push(b.unwrap_or_default()); + } + "end" => { + if id_is(t.get(1), "with") { + self.with.pop(); + } + } + "set" if matches!(t.get(1), Some(Tok::Id(_))) && op_is(t.get(2), "=") => { + let Some(Tok::Id(v)) = t.get(1) else { return }; + let b = self.whole(&t[3..]); + self.scan(&t[3..], None, out, b.is_none()); + match b { + Some(b) => self.vars.insert(v.to_ascii_lowercase(), b), + None => self.vars.remove(&v.to_ascii_lowercase()), + }; + } + "dim" | "redim" | "const" | "attribute" | "option" | "declare" | "public" | "private" | "global" => {} + "if" | "elseif" => { + let then = t.iter().position(|x| id_is(Some(x), "then")).unwrap_or(t.len()); + self.scan(&t[1..then], None, out, true); + if then + 1 < t.len() { + self.statement(&t[then + 1..], out); + } + } + "else" => self.statement(&t[1..], out), + h if NO_ASSIGN.contains(&h) => self.scan(t, None, out, true), + _ => { + // A single-line If's `… Else …` tail is its own statement. + if let Some(e) = t.iter().position(|x| id_is(Some(x), "else")) { + self.statement(&t[..e], out); + self.statement(&t[e..], out); + return; + } + // `ws.Name = "Out"`: ws now holds the sheet called Out. + if let [Tok::Id(v), Tok::Op(d), Tok::Id(n), Tok::Op(e), Tok::Str(name)] = t + && d == "." + && n.eq_ignore_ascii_case("name") + && e == "=" + { + let at = sheet_index(self.wb, name).map_or_else(|| At::Missing(name.clone()), At::Sheet); + self.vars.insert(v.to_ascii_lowercase(), Binding::sheet(at)); + } + let mut depth = 0; + let eq = t.iter().position(|x| { + match x { + Tok::Op(o) if o == "(" => depth += 1, + Tok::Op(o) if o == ")" => depth -= 1, + _ => {} + } + depth == 0 && op_is(Some(x), "=") + }); + self.scan(t, eq, out, true); + } + } + } + + /// Record the calls in `t` and, with `touches`, every reference; those + /// before `eq` are assigned to. + fn scan(&self, t: &[Tok], eq: Option, out: &mut Out, touches: bool) { + let mut i = 0; + while i < t.len() { + if let Tok::Id(w) = &t[i] + && !op_is(i.checked_sub(1).and_then(|p| t.get(p)), ".") + && let Some(callees) = self.callees(w) + { + let args = if op_is(t.get(i + 1), "(") { + split_args(&t[i + 2..close_paren(t, i + 1).min(t.len())]) + } else if i == 0 || id_is(t.get(i - 1), "call") { + split_args(&t[i + 1..]) + } else { + Vec::new() + }; + let bound: Vec> = args.iter().map(|a| self.whole(a)).collect(); + for c in callees { + out.calls.push((c, bound.clone())); + } + } + match self.resolve(t, i) { + Some(h) => { + if touches && h.counts { + let write = h.write || eq.is_some_and(|e| i < e); + out.touches.extend(h.b.places.iter().map(|p| Touch { at: p.at.clone(), rect: p.rect, write })); + } + i = h.end.max(i + 1); + } + None => i += 1, + } + } + } + + /// Procedures `w` may call: the one in this module, else any of that name. + fn callees(&self, w: &str) -> Option> { + let all = self.by_name.get(&w.to_ascii_lowercase())?; + let mine = self.procs[self.this].script; + let same: Vec = all.iter().copied().filter(|&c| self.procs[c].script == mine).collect(); + let v: Vec = if same.is_empty() { all.clone() } else { same }; + let v: Vec = v.into_iter().filter(|&c| c != self.this).collect(); + (!v.is_empty()).then_some(v) + } + + /// What `t` as a whole refers to, allowing a workbook prefix + /// (`ThisWorkbook.Worksheets("Data")`). + fn whole(&self, t: &[Tok]) -> Option { + for k in 0..t.len() { + if let Some(h) = self.resolve(t, k) { + return (h.end == t.len()).then_some(h.b); + } + if !matches!(t[k], Tok::Id(_)) && !op_is(Some(&t[k]), ".") { + return None; + } + } + None + } + + /// A reference starting at `t[i]`: its base (sheet expression, variable, + /// code name, `With` target, bare `Range`) and the member chain after it. + fn resolve(&self, t: &[Tok], i: usize) -> Option { + let prev_dot = i > 0 && op_is(t.get(i - 1), "."); + let (mut b, mut j, mut counts, mut write) = match &t[i] { + Tok::Op(o) if o == "." => { + let leading = i == 0 || matches!(&t[i - 1], Tok::Op(p) if p != ")"); + let top = self.with.last().filter(|b| leading && !b.places.is_empty())?; + (top.clone(), i, false, false) + } + Tok::Id(w) => { + let lw = w.to_ascii_lowercase(); + if lw == "worksheets" || lw == "sheets" { + let at = self.sheet_arg(t, i + 1, true, true)?; + (Binding::sheet(at), i + 4, true, false) + } else if prev_dot { + return None; + } else if lw == "range" || lw == "cells" { + let mut b = Binding::sheet(self.me_at()); + let mut write = false; + let j = self.member(&mut b, &mut write, t, i); + (b, j, true, write) + } else if lw == "me" { + (Binding::sheet(At::Sheet(self.me?)), i + 1, false, false) + } else if lw == "activesheet" { + (Binding::sheet(At::Active), i + 1, false, false) + } else if let Some(v) = self.vars.get(&lw) { + (v.clone(), i + 1, v.range, false) + } else if let Some(&s) = self.code_names.get(&lw) { + (Binding::sheet(At::Sheet(s)), i + 1, false, false) + } else { + // A helper given a sheet's name: SheetByName("Data") + let creates = self.callees(w).is_some_and(|c| c.iter().any(|&c| self.procs[c].returns_sheet)); + (Binding::sheet(self.sheet_arg(t, i + 1, false, creates)?), i + 4, true, false) + } + } + _ => return None, + }; + while op_is(t.get(j), ".") && matches!(t.get(j + 1), Some(Tok::Id(_))) { + counts = true; + j = self.member(&mut b, &mut write, t, j + 1); + } + Some(Hit { b, write, counts, end: j }) + } + + /// Apply the member at `t[m]` (with its arguments) to `b`; returns the index after it. + fn member(&self, b: &mut Binding, write: &mut bool, t: &[Tok], m: usize) -> usize { + let Some(Tok::Id(name)) = t.get(m) else { return m + 1 }; + let name = name.to_ascii_lowercase(); + let (args, next) = if op_is(t.get(m + 1), "(") { + let c = close_paren(t, m + 1); + (Some(&t[m + 2..c.min(t.len())]), c + 1) + } else { + (None, m + 1) + }; + if (name == "range" || name == "cells") && !b.range { + let (sheet, rect) = self.range_args(&name, args); + b.places = b.places.iter().map(|p| Place { at: sheet.map_or_else(|| p.at.clone(), At::Sheet), rect }).collect(); + b.range = true; + } else if RELATIVE.contains(&name.as_str()) || ((name == "range" || name == "cells") && args.is_some()) { + b.places = b.places.iter().map(|p| Place { at: p.at.clone(), rect: None }).collect(); + b.range = true; + } else if MUTATORS.contains(&name.as_str()) { + *write = true; + } + next + } + + fn me_at(&self) -> At { + self.me.map_or(At::Active, At::Sheet) + } + + /// `("Name")` naming a sheet of this workbook (or, when `sure` it names a + /// sheet, any name), or `(3)` with `sure`. + fn sheet_arg(&self, t: &[Tok], k: usize, numeric: bool, sure: bool) -> Option { + if !op_is(t.get(k), "(") || !op_is(t.get(k + 2), ")") { + return None; + } + match t.get(k + 1)? { + Tok::Str(s) => match sheet_index(self.wb, s) { + Some(i) => Some(At::Sheet(i)), + None => sure.then(|| At::Missing(s.clone())), + }, + Tok::Num(n) if numeric && *n >= 1.0 && (*n as usize) <= self.wb.sheets.len() => Some(At::Sheet(*n as usize - 1)), + _ => None, + } + } + + /// The sheet (from a named range) and address of `Range(…)` / `Cells(…)` + /// arguments, when they are written out. + fn range_args(&self, kind: &str, args: Option<&[Tok]>) -> (Option, Option) { + let Some(args) = args else { return (None, None) }; + let parts = split_args(args); + if kind == "cells" { + return (None, cells_rect(&parts)); + } + let (mut sheet, mut acc): (Option, Option) = (None, None); + for p in &parts { + let r = match p { + [Tok::Str(s)] => match parse_rect(s) { + Some(r) => r, + None => { + let Some((s, r)) = self.a.name_range(None, s) else { return (None, None) }; + sheet = s.or(sheet); + r + } + }, + // Cells(1, 2), or .Cells(1, 2) inside a With + [Tok::Id(c), Tok::Op(o), rest @ ..] | [Tok::Op(_), Tok::Id(c), Tok::Op(o), rest @ ..] + if c.eq_ignore_ascii_case("cells") && o == "(" && !matches!(&p[0], Tok::Op(x) if x != ".") => + { + match cells_rect(&split_args(&rest[..rest.len().saturating_sub(1)])) { + Some(r) => r, + None => return (None, None), + } + } + _ => return (None, None), + }; + acc = Some(acc.map_or(r, |a| a.union(&r))); + } + (sheet, acc) + } +} + +/// `Cells(2, 3)` / `Cells(2, "C")` with literal arguments. +fn cells_rect(parts: &[&[Tok]]) -> Option { + let [[Tok::Num(r)], [col]] = parts else { return None }; + let c = match col { + Tok::Num(c) if *c >= 1.0 => *c as u32 - 1, + Tok::Str(s) => col_index(s)?, + _ => return None, + }; + (*r >= 1.0).then(|| Rect::cell(*r as u32 - 1, c)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn statements_join_continuations_and_drop_comments() { + let st = statements("x = \"it's \"\"quoted\"\"\" ' comment\nIf a Then _\n b = 1: c = 2\nRem gone"); + assert_eq!(st.len(), 3); + assert_eq!(st[0].1[2], Tok::Str("it's \"quoted\"".into())); + assert_eq!((st[1].0, st[1].1.len()), (2, 6)); + assert_eq!((st[2].0, st[2].1.len()), (2, 3)); + } + + #[test] + fn string_consts() { + let c = |src: &str| string_const(&statements(src)[0].1); + assert_eq!(c("Private Const RAW As String = \"Raw Data\""), Some(("raw".into(), "Raw Data".into(), false))); + assert_eq!(c("Public Const OUT = \"Out\""), Some(("out".into(), "Out".into(), true))); + assert_eq!(c("Const N As Long = 252"), None); + assert_eq!(c("Const A = \"x\" & \"y\""), None); + } + + #[test] + fn headers() { + let (kind, name, ps, _) = proc_header(&tokenize("Private Sub Totals(ByVal ws As Worksheet, Optional n As Long = 2)")).unwrap(); + assert_eq!((kind, name.as_str(), ps), ("sub", "Totals", vec!["ws".to_string(), "n".to_string()])); + assert!(proc_header(&tokenize("Private Function FreshSheet(nm As String) As Worksheet")).unwrap().3); + assert_eq!(proc_header(&tokenize("Public Property Get Rate() As Double")).unwrap().1, "Rate"); + assert!(proc_header(&tokenize("Private Declare PtrSafe Function GetTickCount Lib \"kernel32\" () As Long")).is_none()); + assert_eq!(js_function(" async function onEdit(e) {").as_deref(), Some("onEdit")); + } +} diff --git a/src/main.rs b/src/main.rs index 2baf460..3446ec3 100644 --- a/src/main.rs +++ b/src/main.rs @@ -55,7 +55,14 @@ enum Cmd { #[arg(long)] down: bool, }, - /// Search values, formulas and notes. + /// List macro / script modules and procedures with the sheets and ranges each + /// writes or uses; with NAME (module, procedure or Module.Procedure), print its code. + #[command(visible_alias = "macros")] + Macro { + dir: PathBuf, + name: Option, + }, + /// Search values, formulas, notes and macro code. Find { dir: PathBuf, text: String, @@ -171,13 +178,19 @@ fn main() -> Result<()> { let wb = store::load(&dir)?; let a = analyze::analyze(&wb); let (u, d) = if !up && !down { (true, true) } else { (up, down) }; - emit(&query::trace(&wb, &a, &cell, depth, u, d)?)?; + let books = store::Siblings::of(&dir); + emit(&query::trace(&wb, &a, &cell, depth, u, d, Some(&books))?)?; } Cmd::Find { dir, text, limit } => { let wb = store::load(&dir)?; let a = analyze::analyze(&wb); emit(&query::find(&wb, &a, &text, limit))?; } + Cmd::Macro { dir, name } => { + let wb = store::load(&dir)?; + let a = analyze::analyze(&wb); + emit(&query::macros(&wb, &a, name.as_deref())?)?; + } Cmd::Rebuild { out, full } => { let opts = if full { report::Opts::full() } else { report::Opts::default() }; for e in std::fs::read_dir(&out)? { diff --git a/src/model.rs b/src/model.rs index 51ad275..222974b 100644 --- a/src/model.rs +++ b/src/model.rs @@ -24,6 +24,22 @@ pub struct Workbook { /// name them as `'[file]Sheet'!A1`. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub external_links: Vec, + /// Document properties: title, author, description, … then custom ones. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub properties: Vec<(String, String)>, + /// Calculation settings, when not the default (automatic, no iteration). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub calc: Option, + /// Sheets can't be added, removed, renamed or unhidden. + #[serde(default, skip_serializing_if = "is_false")] + pub structure_locked: bool, + /// Data connections: databases, web queries, text imports, Power Query. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub connections: Vec, + /// Discussion comments the source doesn't tie to a cell (Google Drive + /// comments). Excel comments are cell notes ([`Cell::note`]). + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub comments: Vec, /// Anything the backend could not extract, so consumers know what is missing. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub warnings: Vec, @@ -54,6 +70,9 @@ pub struct Sheet { pub index: usize, #[serde(default, skip_serializing_if = "is_false")] pub hidden: bool, + /// Hidden so that only macro code can unhide it (Excel `veryHidden`); also `hidden`. + #[serde(default, skip_serializing_if = "is_false")] + pub very_hidden: bool, /// GRID, OBJECT (chart sheet), DATA_SOURCE #[serde(default, skip_serializing_if = "String::is_empty")] pub kind: String, @@ -66,6 +85,10 @@ pub struct Sheet { pub frozen_cols: u32, #[serde(default, skip_serializing_if = "Option::is_none")] pub tab_color: Option, + /// The sheet's VBA code name (`Sheet1`), which macros can use in place of + /// `Worksheets("Data")` and which names the sheet's own code module. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub code_name: Option, /// Sparse, sorted by (row, col). pub cells: Vec, #[serde(default, skip_serializing_if = "Vec::is_empty")] @@ -86,9 +109,24 @@ pub struct Sheet { pub hidden_cols: Vec<(u32, u32)>, #[serde(default, skip_serializing_if = "Vec::is_empty")] pub banded: Vec, + /// Columns not at the sheet's default width: (first, last, pixels). + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub col_widths: Vec<(u32, u32, u32)>, + /// Rows not at the default height: (first, last, pixels). + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub row_heights: Vec<(u32, u32, u32)>, + /// Gridlines turned off: laid out as a page or form rather than a grid. + #[serde(default, skip_serializing_if = "is_false")] + pub no_gridlines: bool, + /// Grouped rows and columns, which can be collapsed to their summary. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub outlines: Vec, /// Excel tables (ListObjects); formulas reach them as `Name[Column]`. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub tables: Vec, + /// Pictures, shapes, text boxes and form controls placed over the grid. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub shapes: Vec, /// Rows summarized instead of kept, when the workbook was too big to hold /// whole (see [`crate::xlsx::CELL_BUDGET`]). #[serde(default, skip_serializing_if = "Option::is_none")] @@ -178,25 +216,72 @@ pub struct Cell { pub s: Option, #[serde(default, skip_serializing_if = "Option::is_none")] pub dv: Option, + /// Note or comment thread, each message as `Author: text` when the + /// author is known, replies after `\n↳ `. #[serde(default, skip_serializing_if = "Option::is_none")] pub note: Option, #[serde(default, skip_serializing_if = "Option::is_none")] pub link: Option, + /// Text whose formatting changes partway (one word bold, say), as runs. + /// Only kept when the runs differ from each other. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub runs: Vec, + /// A picture in the cell, or a data type (stock, place, person chip). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub obj: Option, } impl Cell { pub fn is_blank(&self) -> bool { - self.v.is_empty() && self.f.is_none() + self.v.is_empty() && self.f.is_none() && self.obj.is_none() } /// What a person would see in the cell. pub fn display(&self) -> String { - match &self.t { - Some(t) => t.clone(), - None => self.v.to_string(), + match (&self.t, &self.obj) { + (Some(t), _) => t.clone(), + (None, Some(o)) if self.v.is_empty() => match &o.text { + Some(t) => format!("[{}: {t}]", o.kind), + None => format!("[{}]", o.kind), + }, + _ => self.v.to_string(), } } } +/// A stretch of a cell's text with its own formatting (only the font +/// properties of [`Style`] are used). +#[derive(Debug, Clone, Serialize, Deserialize, Default, PartialEq)] +pub struct Run { + pub text: String, + #[serde(default, skip_serializing_if = "Style::is_plain")] + pub f: Style, +} + +/// Whether runs differ in emphasis (bold, italic, underline, strike, colour). +/// Runs that only switch font or size are mostly font fallback for another +/// script, and aren't worth keeping. +pub fn runs_differ(runs: &[Run]) -> bool { + let key = |r: &Run| (r.f.bold, r.f.italic, r.f.underline, r.f.strike, r.f.fg.clone()); + runs.iter().any(|r| key(r) != key(&runs[0])) +} + +/// What a cell holds besides a plain value. +#[derive(Debug, Clone, Serialize, Deserialize, Default, PartialEq)] +pub struct CellObject { + /// "image" (a picture in the cell), "entity" (Excel linked data type such + /// as Stocks or Geography), "person", "file", "link" (Google smart chips). + pub kind: String, + /// Alt text of a picture, or the data type's display name. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub text: Option, + /// Picture file inside the package, or a chip's URL / email. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub target: Option, + /// A data type's fields (`Price = 412.5`, `Population = 67750000`). + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub props: Vec<(String, String)>, +} + #[derive(Debug, Clone, Serialize, Deserialize, Default, PartialEq)] #[serde(untagged)] pub enum Value { @@ -255,8 +340,10 @@ pub struct Style { pub strike: bool, #[serde(default, skip_serializing_if = "is_false")] pub underline: bool, - #[serde(default, skip_serializing_if = "is_false")] - pub borders: bool, + /// Which sides have a border and its line: `box thin`, `bottom double`, + /// `top thin, bottom double`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub border: Option, #[serde(default, skip_serializing_if = "Option::is_none")] pub h_align: Option, #[serde(default, skip_serializing_if = "Option::is_none")] @@ -265,9 +352,41 @@ pub struct Style { pub size: Option, #[serde(default, skip_serializing_if = "is_false")] pub wrap: bool, + /// Left editable when the sheet is protected (cells are locked by default). + #[serde(default, skip_serializing_if = "is_false")] + pub unlocked: bool, + /// The formula is hidden from the formula bar when the sheet is protected. + #[serde(default, skip_serializing_if = "is_false")] + pub hidden_formula: bool, + /// Named cell style other than Normal ("Input", "Calculation", "Good", a custom one). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub cell_style: Option, +} + +/// `box thin` when all four sides share a line, else the sides grouped by +/// line: `top thin, bottom double`. None without any side. +pub fn border_desc(sides: &[(&str, String)]) -> Option { + if sides.is_empty() { + return None; + } + if sides.len() == 4 && sides.iter().all(|(_, l)| *l == sides[0].1) { + return Some(format!("box {}", sides[0].1)); + } + let mut groups: Vec<(Vec<&str>, &str)> = Vec::new(); + for (side, line) in sides { + match groups.iter_mut().find(|(_, l)| l == line) { + Some(g) => g.0.push(side), + None => groups.push((vec![side], line)), + } + } + Some(groups.iter().map(|(s, l)| format!("{} {l}", s.join("+"))).collect::>().join(", ")) } impl Style { + pub fn is_plain(&self) -> bool { + *self == Style::default() + } + pub fn is_date(&self) -> bool { matches!(self.num_type.as_deref(), Some("DATE" | "DATE_TIME" | "TIME")) } @@ -290,8 +409,12 @@ pub struct Validation { #[derive(Debug, Clone, Serialize, Deserialize, Default)] pub struct NamedRange { pub name: String, - /// A1 reference including sheet, e.g. `'Revenue Input'!A1:R100`. + /// A1 reference including sheet, e.g. `'Revenue Input'!A1:R100`, or the + /// formula a name stands for (a constant, an expression, a LAMBDA). pub range: String, + /// The sheet a name is local to; None for workbook-wide names. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub scope: Option, } #[derive(Debug, Clone, Serialize, Deserialize, Default)] @@ -299,6 +422,10 @@ pub struct CondFormat { pub ranges: Vec, /// Human-readable rule, e.g. `NUMBER_LESS 0 → fg#FF0000` or `=$C2>100 → bg#F4CCCC`. pub rule: String, + /// Formulas the rule evaluates, each with a leading `=`, relative to the + /// top-left cell of the first range. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub formulas: Vec, } #[derive(Debug, Clone, Serialize, Deserialize, Default)] @@ -365,14 +492,161 @@ pub struct ExternalLink { pub sheets: Vec, } +/// How the workbook recalculates. +#[derive(Debug, Clone, Serialize, Deserialize, Default, PartialEq)] +pub struct Calc { + /// "automatic", "manual", "automatic except tables" (Excel); "on change", + /// "on change and every minute", "on change and every hour" (Google). + pub mode: String, + /// Circular references are allowed and iterated. + #[serde(default, skip_serializing_if = "is_false")] + pub iterative: bool, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_iterations: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_change: Option, +} + +/// A run of grouped rows or columns (an outline level). +#[derive(Debug, Clone, Serialize, Deserialize, Default, PartialEq)] +pub struct Outline { + #[serde(default, skip_serializing_if = "is_false")] + pub cols: bool, + /// First and last row (or column), 0-based and inclusive. + pub span: (u32, u32), + /// 1 for an outer group, 2 for one inside it, … + pub depth: u32, + /// Collapsed: the group's rows or columns are hidden. + #[serde(default, skip_serializing_if = "is_false")] + pub collapsed: bool, +} + +/// Grouped spans from outline levels: `levels` holds (first, last, level) +/// with level > 0, sorted. A group is collapsed when `hidden` covers all of it. +pub fn outlines(levels: &[(u32, u32, u32)], hidden: &[(u32, u32)], cols: bool) -> Vec { + let mut hidden = hidden.to_vec(); + hidden.sort_unstable(); + let covered = |(a, b): (u32, u32)| { + let mut next = a; + for h in hidden.iter().filter(|h| h.1 >= a && h.0 <= b) { + if h.0 > next { + return false; + } + next = next.max(h.1.saturating_add(1)); + } + next > b + }; + let max = levels.iter().map(|l| l.2).max().unwrap_or(0); + let mut out = Vec::new(); + for depth in 1..=max { + let mut run: Option<(u32, u32)> = None; + let mut flush = |run: Option<(u32, u32)>| { + if let Some(span) = run { + out.push(Outline { cols, span, depth, collapsed: covered(span) }); + } + }; + for &(a, b, _) in levels.iter().filter(|l| l.2 >= depth) { + run = match run { + Some((s, e)) if e + 1 == a => Some((s, b)), + prev => { + flush(prev); + Some((a, b)) + } + }; + } + flush(run); + } + out.sort_by_key(|o| (o.span.0, o.depth)); + out +} + +/// Add index `i` with value `v` to (first, last, value) spans built in order. +pub fn push_sized(spans: &mut Vec<(u32, u32, u32)>, i: u32, v: u32) { + match spans.last_mut() { + Some(last) if last.1 + 1 == i && last.2 == v => last.1 = i, + _ => spans.push((i, i, v)), + } +} + +/// A comment thread on the file (see [`Workbook::comments`]). +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +pub struct Comment { + pub author: String, + pub text: String, + /// What the commented cell showed when the comment was made. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub quoted: Option, + /// The cell (`Sheet!A1`) that shows `quoted`, when exactly one does. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub cell: Option, + #[serde(default, skip_serializing_if = "is_false")] + pub resolved: bool, + /// (author, text) of each reply. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub replies: Vec<(String, String)>, +} + +/// A data connection the workbook refreshes from. +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +pub struct Connection { + pub name: String, + /// "odbc", "oledb", "web", "text", "power query", … + pub kind: String, + /// Connection string or URL, with passwords removed. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source: Option, + /// SQL, table name or query run on the source. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub command: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub description: Option, + #[serde(default, skip_serializing_if = "is_false")] + pub refresh_on_open: bool, +} + +/// Something drawn over the grid. +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +pub struct Shape { + /// "picture", "shape", "text box", "button", "checkbox", "dropdown", … + pub kind: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub name: String, + /// Cells it covers. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub anchor: Option, + /// Text shown on it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub text: Option, + /// Alt text (descriptions of pictures). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub alt: Option, + /// Macro run when it's clicked. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub on_click: Option, + /// Cell a form control reads and writes. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub linked_cell: Option, +} + #[derive(Debug, Clone, Serialize, Deserialize, Default)] pub struct Script { - /// "vba" | "apps_script" | "macro" + /// "vba" | "apps_script" | "power_query" pub kind: String, pub name: String, pub source: String, } +impl Script { + /// Whether the module holds any code. Excel gives every sheet and the + /// workbook a module, and most stay `Attribute …` / `Option …` lines only. + pub fn has_code(&self) -> bool { + self.source.lines().map(str::trim).any(|l| { + let lower = l.to_ascii_lowercase(); + !(l.is_empty() || l.starts_with('\'') || lower.starts_with("attribute ") || lower.starts_with("option ") || lower.starts_with("rem ")) + }) + } +} + fn is_false(b: &bool) -> bool { !*b } diff --git a/src/numfmt.rs b/src/numfmt.rs new file mode 100644 index 0000000..0a99e48 --- /dev/null +++ b/src/numfmt.rs @@ -0,0 +1,278 @@ +//! Excel number formats applied to numbers, so a cell reads as it shows on +//! screen (`25%`, `$12.50`, `(1,234)`) rather than as its stored value. +//! +//! Covers the common shapes: sections (`pos;neg;zero`), literals, `%`, +//! thousands separators and `,` scaling, fixed and optional decimals, +//! scientific notation, colour and currency tags. Anything else (dates, +//! fractions, conditions, digits mixed with literals) returns None, and the +//! caller keeps the plain value. + +use crate::model::fmt_num; + +/// `n` formatted with `code`, or None when the format isn't one this handles. +pub fn format(n: f64, code: &str) -> Option { + if !n.is_finite() { + return None; + } + let sections = split_sections(code); + if sections.iter().any(|s| s.contains("[>") || s.contains("[<") || s.contains("[=")) { + return None; + } + let (sec, signed) = match (sections.len(), n) { + (_, n) if n > 0.0 => (sections[0].as_str(), false), + (1, n) if n < 0.0 => (sections[0].as_str(), true), + (_, n) if n < 0.0 => (sections[1].as_str(), false), + (l, _) if l >= 3 => (sections[2].as_str(), false), + _ => (sections[0].as_str(), false), + }; + let body = render(n.abs(), sec)?; + Some(if signed && body.chars().any(|c| c.is_ascii_digit() && c != '0') { format!("-{body}") } else { body }) +} + +fn split_sections(code: &str) -> Vec { + let mut out = vec![String::new()]; + let mut chars = code.chars(); + while let Some(c) = chars.next() { + match c { + ';' => out.push(String::new()), + '"' => { + let last = out.last_mut().unwrap(); + last.push('"'); + for q in chars.by_ref() { + last.push(q); + if q == '"' { + break; + } + } + } + '\\' => { + let last = out.last_mut().unwrap(); + last.push('\\'); + last.extend(chars.next()); + } + _ => out.last_mut().unwrap().push(c), + } + } + out +} + +enum Piece { + Lit(String), + Num, +} + +/// One section: literals around a single run of digit placeholders. +fn render(v: f64, sec: &str) -> Option { + let mut pieces: Vec = Vec::new(); + let mut pat = String::new(); // digit placeholders, '.', ',', 'E+00' + let mut lit = String::new(); + let mut general = false; + let mut percent = 0; + let sec = replace_general(sec); + let mut chars = sec.chars().peekable(); + let flush = |lit: &mut String, pieces: &mut Vec| { + if !lit.is_empty() { + pieces.push(Piece::Lit(std::mem::take(lit))); + } + }; + while let Some(c) = chars.next() { + match c { + '"' => { + for q in chars.by_ref() { + if q == '"' { + break; + } + lit.push(q); + } + } + '\\' => lit.extend(chars.next()), + '_' => { + chars.next(); + lit.push(' '); + } + '*' => { + chars.next(); + } + '[' => { + let tag: String = chars.by_ref().take_while(|c| *c != ']').collect(); + // [$€-407] is a currency symbol; [$-409] only a locale; [Red] a colour. + if let Some(cur) = tag.strip_prefix('$') { + lit.push_str(cur.split('-').next().unwrap_or("")); + } else if tag.chars().next().is_some_and(|c| "hHmMsS".contains(c)) { + return None; // elapsed time + } + } + '0' | '#' | '?' | '.' => { + if pieces.iter().any(|p| matches!(p, Piece::Num)) { + return None; // digits broken up by literals (phone numbers, fractions) + } + flush(&mut lit, &mut pieces); + pat.push(c); + while let Some(&d) = chars.peek() { + if "0#?.,".contains(d) { + pat.push(d); + chars.next(); + } else if (d == 'E' || d == 'e') && !pat.contains('E') { + chars.next(); + let sign = chars.next().filter(|s| *s == '+' || *s == '-')?; + pat.push('E'); + pat.push(sign); + } else { + break; + } + } + pieces.push(Piece::Num); + } + '%' => { + percent += 1; + lit.push('%'); + } + GENERAL => { + flush(&mut lit, &mut pieces); + general = true; + pieces.push(Piece::Num); + } + '@' | '/' => return None, + c if c.is_ascii_alphabetic() && !matches!(c, 'E' | 'e') => { + // y m d h s and AM/PM belong to dates and times. + if "yYmMdDhHsSaA".contains(c) { + return None; + } + lit.push(c); + } + _ => lit.push(c), + } + } + flush(&mut lit, &mut pieces); + let v = v * 100f64.powi(percent); + let num = if general { + fmt_num(v) + } else if pat.is_empty() { + String::new() + } else { + number(v, &pat) + }; + // `_)` and `* ` only pad for alignment. + let out: String = pieces.into_iter().map(|p| match p { + Piece::Lit(s) => s, + Piece::Num => num.clone(), + }).collect(); + Some(out.trim().to_string()) +} + +/// Stands in for the word `General` (outside quotes) while a section is read. +const GENERAL: char = '\u{1}'; + +fn replace_general(sec: &str) -> String { + let mut out = String::new(); + let mut rest = sec; + let mut quoted = false; + while let Some(c) = rest.chars().next() { + if !quoted && rest.get(..7).is_some_and(|w| w.eq_ignore_ascii_case("general")) { + out.push(GENERAL); + rest = &rest[7..]; + continue; + } + quoted ^= c == '"'; + out.push(c); + rest = &rest[c.len_utf8()..]; + } + out +} + +/// `v` (not negative) through a placeholder pattern like `#,##0.00` or `0.0E+00`. +fn number(mut v: f64, pat: &str) -> String { + let (mant, exp) = match pat.find('E') { + Some(i) => (&pat[..i], Some(&pat[i + 1..])), + None => (pat, None), + }; + // Commas right after the last digit placeholder scale by 1000 each. + let trimmed = mant.trim_end_matches(','); + for _ in 0..mant.len() - trimmed.len() { + v /= 1000.0; + } + let (int_pat, dec_pat) = trimmed.split_once('.').unwrap_or((trimmed, "")); + let thousands = int_pat.contains(','); + let min_int = int_pat.chars().filter(|c| *c == '0').count(); + let (min_dec, max_dec) = (dec_pat.chars().filter(|c| *c == '0' || *c == '?').count(), dec_pat.chars().filter(|c| "0#?".contains(*c)).count()); + + let mut exponent = 0i32; + if exp.is_some() && v != 0.0 { + exponent = v.log10().floor() as i32; + v /= 10f64.powi(exponent); + // Rounding can carry the mantissa to 10.0. + if format!("{v:.max_dec$}").starts_with("10") { + v /= 10.0; + exponent += 1; + } + } + let fixed = format!("{v:.max_dec$}"); + let (int_s, dec_s) = fixed.split_once('.').unwrap_or((&fixed, "")); + let mut dec = dec_s.to_string(); + while dec.len() > min_dec && dec.ends_with('0') { + dec.pop(); + } + let int_s = int_s.trim_start_matches('0'); + let mut int = format!("{:0>min_int$}", int_s); + if thousands && int.len() > 3 { + let mut grouped = String::new(); + for (i, ch) in int.chars().enumerate() { + if i > 0 && (int.len() - i) % 3 == 0 { + grouped.push(','); + } + grouped.push(ch); + } + int = grouped; + } + let mut out = int; + if !dec.is_empty() || (dec_pat.is_empty() && trimmed.contains('.')) { + out.push('.'); + out.push_str(&dec); + } + if let Some(e) = exp { + let (sign, digits) = e.split_at(1); + let width = digits.chars().filter(|c| *c == '0').count(); + let s = if exponent < 0 { "-" } else if sign == "+" { "+" } else { "" }; + out.push_str(&format!("E{s}{:0>width$}", exponent.abs())); + } + out +} + +#[cfg(test)] +mod tests { + use super::format; + + #[test] + fn common_formats() { + let f = |n: f64, code: &str| format(n, code); + assert_eq!(f(0.25, "0%").as_deref(), Some("25%")); + assert_eq!(f(0.1234, "0.0%").as_deref(), Some("12.3%")); + assert_eq!(f(12.5, "$#,##0.00").as_deref(), Some("$12.50")); + assert_eq!(f(1234567.0, "#,##0").as_deref(), Some("1,234,567")); + assert_eq!(f(-1234.5, "$#,##0.00_);($#,##0.00)").as_deref(), Some("($1,234.50)")); + assert_eq!(f(1234.5, "$#,##0.00_);($#,##0.00)").as_deref(), Some("$1,234.50")); + assert_eq!(f(0.0, "_(* #,##0_);_(* \\(#,##0\\);_(* \"-\"??_);_(@_)").as_deref(), Some("-")); + assert_eq!(f(-1500.0, "_(* #,##0_);_(* \\(#,##0\\);_(* \"-\"??_);_(@_)").as_deref(), Some("(1,500)")); + assert_eq!(f(-5.0, "#,##0").as_deref(), Some("-5")); + assert_eq!(f(0.0, "#,##0;(#,##0);\"-\"").as_deref(), Some("-")); + assert_eq!(f(-7.0, "#,##0;[Red](#,##0)").as_deref(), Some("(7)")); + assert_eq!(f(0.00012, "0.00E+00").as_deref(), Some("1.20E-04")); + assert_eq!(f(123456.0, "0.0E+0").as_deref(), Some("1.2E+5")); + assert_eq!(f(1_500_000.0, "0.0,,\"M\"").as_deref(), Some("1.5M")); + assert_eq!(f(1234.5, "[$€-407] #,##0.00").as_deref(), Some("€ 1,234.50")); + assert_eq!(f(5.0, "0.00").as_deref(), Some("5.00")); + assert_eq!(f(0.5, "#.00").as_deref(), Some(".50")); + assert_eq!(f(1.23456, "0.0#").as_deref(), Some("1.23")); + assert_eq!(f(3.1, "0.0#").as_deref(), Some("3.1")); + assert_eq!(f(42.0, "General\" units\"").as_deref(), Some("42 units")); + assert_eq!(f(42.0, ";;;").as_deref(), Some("")); + assert_eq!(f(-0.001, "0.0").as_deref(), Some("0.0")); + } + + #[test] + fn declines_what_it_cannot_render() { + for code in ["yyyy-mm-dd", "h:mm", "# ?/?", "000-00-0000", "[>100]0;0.0", "@", "[h]:mm"] { + assert_eq!(format(1.0, code), None, "{code}"); + } + } +} diff --git a/src/ooxml.rs b/src/ooxml.rs index b5cc6e7..26555bf 100644 --- a/src/ooxml.rs +++ b/src/ooxml.rs @@ -7,8 +7,9 @@ //! failed extraction. Values and formulas still come from calamine. use crate::a1::{col_name, parse_pt, parse_rect, quote_sheet, Rect, OPEN}; -use crate::model::{ - Cell, Chart, CondFormat, ExternalLink, Filter, Interner, Pivot, Protected, Sheet, Style, Table, Validation, Workbook, +use crate::model::{fmt_num, outlines, push_sized, Value, + Calc, Cell, CellObject, Chart, CondFormat, Connection, ExternalLink, Filter, Interner, NamedRange, Pivot, Protected, Run, Script, Shape, Sheet, Style, + Table, Validation, Workbook, }; use anyhow::{anyhow, Result}; use quick_xml::events::{BytesRef, BytesStart, Event}; @@ -35,13 +36,30 @@ pub fn enrich(path: &Path, wb: &mut Workbook) -> Result<()> { styles.xfs.iter().map(|s| if *s == Style::default() { None } else { Some(interner.intern(s.clone())) }).collect(); let mut dvs = Interner::::default(); let mut skipped: BTreeSet<&str> = BTreeSet::new(); + let rich = part_of("sharedStrings").map(|p| rich_strings(&mut pkg, &p, &styles)).unwrap_or_default(); + let objects = rich_values(&mut pkg); + let persons: HashMap = part_of("person") + .and_then(|p| pkg.node(&p)) + .map(|n| n.children("person").filter_map(|x| Some((x.attr("id")?.to_string(), x.attr("displayName")?.to_string()))).collect()) + .unwrap_or_default(); wb.external_links = external_links(&mut pkg, &root, &rels); + // calamine drops which sheet a name belongs to; read the names again. + if let Some(names) = defined_names(&root) { + wb.named_ranges = names; + } + wb.calc = calc(&root); + wb.structure_locked = root.child("workbookProtection").is_some_and(|p| p.flag("lockStructure")); + wb.properties = properties(&mut pkg); + if let Some(p) = part_of("connections") { + wb.connections = connections(&mut pkg, &p); + } + wb.scripts.extend(power_query(&mut pkg)); for s in root.child("sheets").map(|s| s.children("sheet").collect::>()).unwrap_or_default() { let (Some(name), Some(rel)) = (s.attr("name"), s.attr("id").and_then(|id| rels.get(id))) else { continue }; let Some(si) = wb.sheets.iter().position(|x| x.name == name) else { continue }; - let mut cx = SheetCx { styles: &styles, xf_ids: &xf_ids, dvs: &mut dvs, skipped: &mut skipped }; + let mut cx = SheetCx { styles: &styles, xf_ids: &xf_ids, dvs: &mut dvs, skipped: &mut skipped, rich: &rich, objects: &objects, persons: &persons }; if let Err(e) = read_sheet(&mut pkg, &rel.target, &rel.kind, &mut wb.sheets[si], &mut cx) { wb.warnings.push(format!("{name}: {e:#}")); } @@ -69,7 +87,7 @@ pub fn enrich(path: &Path, wb: &mut Workbook) -> Result<()> { } } - for (prefix, what) in [("xl/media/", "images"), ("xl/slicers/", "slicers"), ("xl/timelines/", "timelines")] { + for (prefix, what) in [("xl/slicers/", "slicers"), ("xl/timelines/", "timelines"), ("xl/activex/", "ActiveX controls")] { if pkg.names.keys().any(|n| n.starts_with(prefix)) { skipped.insert(what); } @@ -114,6 +132,13 @@ impl Pkg { Some(r) } + fn bytes(&mut self, part: &str) -> Option> { + let name = self.names.get(&part.to_ascii_lowercase())?.clone(); + let mut out = Vec::new(); + std::io::Read::read_to_end(&mut self.zip.by_name(&name).ok()?, &mut out).ok()?; + Some(out) + } + fn node(&mut self, part: &str) -> Option { parse(&mut self.reader(part)?).ok() } @@ -202,6 +227,14 @@ impl Node { walk(self, name, &mut out); out } + /// All text below, in document order. + fn all_text(&self) -> String { + let mut s = self.text.clone(); + for k in &self.kids { + s.push_str(&k.all_text()); + } + s + } /// Concatenated text of every `` below (rich-text runs). fn runs(&self) -> String { self.all("t").iter().map(|t| t.text.as_str()).collect() @@ -274,6 +307,12 @@ struct SheetCx<'a> { xf_ids: &'a [Option], dvs: &'a mut Interner, skipped: &'a mut BTreeSet<&'static str>, + /// Shared strings with mixed formatting, by index. + rich: &'a HashMap>, + /// Pictures and data types by cell `vm` (value metadata) index, from 0. + objects: &'a [Option], + /// Threaded-comment authors by person id. + persons: &'a HashMap, } /// One pass over a worksheet part: cell style ids and hidden rows are taken @@ -282,16 +321,28 @@ struct SheetCx<'a> { struct SheetXml { styled: Vec<(u32, u32, u32)>, hidden_rows: Vec<(u32, u32)>, + /// Rows with a stored height: (first, last, pixels). + row_heights: Vec<(u32, u32, u32)>, + /// Rows in outline groups: (first, last, level). + row_levels: Vec<(u32, u32, u32)>, nodes: Vec, /// Corrected text for shared formulas whose defining cell isn't the top-left /// of their range (see [`shared_formula`]). shared_fixes: Vec<(u32, u32, String)>, /// Summarized rows ([`crate::model::Elided`]): their cell styles are skipped. gap: Option<(u32, u32)>, + /// Read shared-string indexes (only when some shared strings are rich). + want_sst: bool, + /// Cells holding a shared string: (row, col, index). + sst_cells: Vec<(u32, u32, u32)>, + /// Inline rich strings (`…`). + inline: Vec<(u32, u32, Node)>, + /// Cells with value metadata (`vm`): pictures in cells, data types. + vm: Vec<(u32, u32, u32)>, } -fn read_sheet_xml(rd: &mut Reader, gap: Option<(u32, u32)>) -> Result { - let mut out = SheetXml { gap, ..Default::default() }; +fn read_sheet_xml(rd: &mut Reader, gap: Option<(u32, u32)>, want_sst: bool) -> Result { + let mut out = SheetXml { gap, want_sst, ..Default::default() }; let mut buf = Vec::new(); loop { buf.clear(); @@ -319,9 +370,23 @@ fn read_sheet_data(rd: &mut Reader, out: &mut SheetXml, buf: &mut // `r` is optional on rows and cells; position then follows the previous one. let (mut row, mut next_col, mut started) = (0u32, 0u32, false); let (mut cell, mut masters) = ((0u32, 0u32), HashMap::new()); + let mut shared_str = false; loop { buf.clear(); + let kept = |out: &SheetXml, r: u32| !out.gap.is_some_and(|(a, b)| (a..=b).contains(&r)); match rd.read_event_into(buf)? { + Event::Start(e) if e.local_name().as_ref() == "v" && shared_str && out.want_sst => { + let v = read_children(rd, "v".into(), Vec::new(), buf)?; + if let (Ok(i), true) = (v.text.trim().parse::(), kept(out, cell.0)) { + out.sst_cells.push((cell.0, cell.1, i)); + } + } + Event::Start(e) if e.local_name().as_ref() == "is" => { + let n = read_children(rd, "is".into(), Vec::new(), buf)?; + if n.child("r").is_some() && kept(out, cell.0) { + out.inline.push((cell.0, cell.1, n)); + } + } Event::Start(e) if e.local_name().as_ref() == "f" => { let f = read_children(rd, "f".into(), attrs_of(&e), buf)?; shared_formula(&f, cell, &mut masters, &mut out.shared_fixes); @@ -332,33 +397,48 @@ fn read_sheet_data(rd: &mut Reader, out: &mut SheetXml, buf: &mut } Event::Start(e) | Event::Empty(e) => match e.local_name().as_ref() { "row" => { - let (mut r, mut hidden) = (None, false); + let (mut r, mut hidden, mut px, mut level) = (None, false, None, 0); for a in e.attributes().flatten() { match a.key.local_name().as_ref() { "r" => r = a.value.parse::().ok().filter(|r| *r > 0), "hidden" => hidden = matches!(a.value.as_ref(), "1" | "true"), + "ht" => px = a.value.parse::().ok().filter(|h| *h >= 0.0).map(pt_to_px), + "outlineLevel" => level = a.value.parse::().unwrap_or(0), _ => {} } } row = r.map(|r| r - 1).unwrap_or(if started { row + 1 } else { 0 }); started = true; next_col = 0; - if hidden { + // A row of height 0 is hidden too. + if hidden || px == Some(0) { push_span(&mut out.hidden_rows, row); } + if let Some(px) = px.filter(|p| *p > 0) { + push_sized(&mut out.row_heights, row, px); + } + if level > 0 { + push_sized(&mut out.row_levels, row, level); + } } "c" => { - let (mut pos, mut s) = (None, None); + let (mut pos, mut s, mut vm) = (None, None, None); + shared_str = false; for a in e.attributes().flatten() { match a.key.local_name().as_ref() { "r" => pos = parse_pt(&a.value).and_then(|p| Some((p.row?, p.col?))), "s" => s = a.value.parse::().ok(), + "t" => shared_str = a.value.as_ref() == "s", + "vm" => vm = a.value.parse::().ok(), _ => {} } } let (r, c) = pos.unwrap_or((row, next_col)); next_col = c + 1; cell = (r, c); + if let Some(vm) = vm.filter(|v| *v > 0) { + out.vm.push((r, c, vm)); + } if let Some(s) = s.filter(|s| *s != 0) && !out.gap.is_some_and(|(a, b)| (a..=b).contains(&r)) { @@ -399,6 +479,17 @@ fn shared_formula(f: &Node, at: (u32, u32), masters: &mut Masters, fixes: &mut V } } +/// Row height in points → pixels at 96 dpi. +fn pt_to_px(pt: f64) -> u32 { + (pt * 4.0 / 3.0).round() as u32 +} + +/// Column width in characters (Excel's unit, padding included) → pixels, +/// for the default font's 7-pixel digits. +fn chars_to_px(w: f64) -> u32 { + (w * 7.0).round() as u32 +} + fn push_span(spans: &mut Vec<(u32, u32)>, i: u32) { match spans.last_mut() { Some(last) if last.1 + 1 == i => last.1 = i, @@ -409,7 +500,7 @@ fn push_span(spans: &mut Vec<(u32, u32)>, i: u32) { fn read_sheet(pkg: &mut Pkg, part: &str, kind: &str, sh: &mut Sheet, cx: &mut SheetCx) -> Result<()> { let sx = { let mut rd = pkg.reader(part).ok_or_else(|| anyhow!("missing part {part}"))?; - read_sheet_xml(&mut rd, sh.elided.as_ref().map(|e| e.rows))? + read_sheet_xml(&mut rd, sh.elided.as_ref().map(|e| e.rows), !cx.rich.is_empty())? }; let rels = pkg.rels(part); if kind == "chartsheet" { @@ -426,14 +517,57 @@ fn read_sheet(pkg: &mut Pkg, part: &str, kind: &str, sh: &mut Sheet, cx: &mut Sh && let Some(x) = cell_at(&mut sh.cells, &mut at, r, c) { x.s = Some(*id); + // Numbers as the format shows them (25%, $12.50); dates already are. + if let (Value::Num(n), None, Some(st)) = (&x.v, &x.t, cx.styles.xfs.get(xf as usize)) + && !st.is_date() + && let Some(t) = st.num_fmt.as_deref().and_then(|code| crate::numfmt::format(*n, code)) + && t != fmt_num(*n) + { + x.t = Some(t); + } } } + for &(r, c, i) in &sx.sst_cells { + if let Some(runs) = cx.rich.get(&i) + && let Some(x) = cell_at(&mut sh.cells, &mut at, r, c) + { + x.runs = runs.clone(); + } + } + for (r, c, n) in &sx.inline { + let runs = rich_runs(n, cx.styles); + if !runs.is_empty() + && let Some(x) = cell_at(&mut sh.cells, &mut at, *r, *c) + { + x.runs = runs; + } + } + for &(r, c, vm) in &sx.vm { + let Some(Some(o)) = cx.objects.get(vm as usize - 1) else { continue }; + let Some(x) = cell_at(&mut sh.cells, &mut at, r, c) else { continue }; + // The stored value of a picture or data type cell is #VALUE!. + if x.f.is_none() || matches!(x.v, Value::Err { .. }) { + x.v = match (&o.text, o.kind.as_str()) { + (Some(t), "entity") => Value::Str(t.clone()), + _ => Value::Empty, + }; + } + x.obj = Some(o.clone()); + } for (r, c, f) in sx.shared_fixes { if let Some(x) = cell_at(&mut sh.cells, &mut at, r, c) { x.f = Some(f); } } sh.hidden_rows = sx.hidden_rows; + // Sizes are kept where they differ from the sheet's defaults (Excel's are + // 15pt rows and 8.43-character columns). + let format = sx.nodes.iter().find(|n| n.name == "sheetFormatPr"); + let size = |k: &str| format.and_then(|f| f.attr(k)).and_then(|v| v.parse::().ok()); + let default_row = size("defaultRowHeight").map_or(20, pt_to_px); + let default_col = size("defaultColWidth").map_or(64, chars_to_px); + sh.row_heights = sx.row_heights.into_iter().filter(|h| h.2 != default_row).collect(); + let mut col_levels = Vec::new(); // Rules often cover whole columns; apply them only as far as the sheet has cells. let ext = sh.cells.iter().chain(&at.cells).fold((0u32, 0u32), |(r, c), x| (r.max(x.r), c.max(x.c))); let clamp = |rect: Rect| Rect { r1: rect.r1.min(ext.0.max(rect.r0)), c1: rect.c1.min(ext.1.max(rect.c0)), ..rect }; @@ -449,17 +583,32 @@ fn read_sheet(pkg: &mut Pkg, part: &str, kind: &str, sh: &mut Sheet, cx: &mut Sh for n in &sx.nodes { match n.name.as_str() { - "sheetPr" => sh.tab_color = n.child("tabColor").and_then(|c| cx.styles.colors.of(c)), + "sheetPr" => { + sh.tab_color = n.child("tabColor").and_then(|c| cx.styles.colors.of(c)); + sh.code_name = n.attr("codeName").map(str::to_string); + } "sheetViews" => { - if let Some(p) = n.child("sheetView").and_then(|v| v.child("pane")).filter(|p| p.attr("state").is_some_and(|s| s.starts_with("frozen"))) { + let view = n.child("sheetView"); + if let Some(p) = view.and_then(|v| v.child("pane")).filter(|p| p.attr("state").is_some_and(|s| s.starts_with("frozen"))) { sh.frozen_rows = p.num("ySplit").unwrap_or(0); sh.frozen_cols = p.num("xSplit").unwrap_or(0); } + sh.no_gridlines = view.is_some_and(|v| matches!(v.attr("showGridLines"), Some("0" | "false"))); } "cols" => { - for col in n.children("col").filter(|c| c.flag("hidden")) { - if let (Some(a), Some(b)) = (col.num("min"), col.num("max")) { - sh.hidden_cols.push((a.saturating_sub(1), b.saturating_sub(1))); + for col in n.children("col") { + let (Some(a), Some(b)) = (col.num("min"), col.num("max")) else { continue }; + let (a, b) = (a.saturating_sub(1), b.saturating_sub(1)); + let px = col.attr("width").and_then(|w| w.parse::().ok()).filter(|w| *w >= 0.0).map(chars_to_px); + // A column of width 0 is hidden too. + if col.flag("hidden") || px == Some(0) { + sh.hidden_cols.push((a, b)); + } + if let Some(px) = px.filter(|p| *p > 0 && *p != default_col) { + sh.col_widths.push((a, b, px)); + } + if let Some(l) = col.num("outlineLevel").filter(|l| *l > 0) { + col_levels.push((a, b, l)); } } } @@ -472,7 +621,7 @@ fn read_sheet(pkg: &mut Pkg, part: &str, kind: &str, sh: &mut Sheet, cx: &mut Sh "conditionalFormatting" => { let ranges = sqref(n.attr("sqref").unwrap_or("")); for rule in n.children("cfRule") { - sh.cond_formats.push(CondFormat { ranges: ranges.clone(), rule: cf_rule(rule, cx.styles) }); + sh.cond_formats.push(CondFormat { ranges: ranges.clone(), rule: cf_rule(rule, cx.styles), formulas: cf_formulas(rule) }); } } "hyperlinks" => { @@ -511,6 +660,13 @@ fn read_sheet(pkg: &mut Pkg, part: &str, kind: &str, sh: &mut Sheet, cx: &mut Sh if let Some(rel) = n.attr("id").and_then(|id| rels.get(id)) { let target = rel.target.clone(); sh.charts.extend(charts(pkg, &target)); + sh.shapes.extend(drawing_shapes(pkg, &target)); + } + } + "legacyDrawing" => { + if let Some(rel) = n.attr("id").and_then(|id| rels.get(id)) { + let target = rel.target.clone(); + sh.shapes.extend(form_controls(pkg, &target)); } } "extLst" => { @@ -522,7 +678,7 @@ fn read_sheet(pkg: &mut Pkg, part: &str, kind: &str, sh: &mut Sheet, cx: &mut Sh for cf in n.all("conditionalFormatting") { let ranges = sqref(&cf.child("sqref").map(|s| s.text.clone()).unwrap_or_default()); for rule in cf.children("cfRule").filter(|r| !r.attr("id").is_some_and(|id| main_ids.iter().any(|m| m == id))) { - sh.cond_formats.push(CondFormat { ranges: ranges.clone(), rule: cf_rule(rule, cx.styles) }); + sh.cond_formats.push(CondFormat { ranges: ranges.clone(), rule: cf_rule(rule, cx.styles), formulas: cf_formulas(rule) }); } } if !n.all("sparklineGroup").is_empty() { @@ -533,27 +689,37 @@ fn read_sheet(pkg: &mut Pkg, part: &str, kind: &str, sh: &mut Sheet, cx: &mut Sh } } + sh.outlines = outlines(&sx.row_levels, &sh.hidden_rows, false); + sh.outlines.extend(outlines(&col_levels, &sh.hidden_cols, true)); + // Notes: legacy comments, replaced by threaded comments where a cell has them. let mut notes: BTreeMap<(u32, u32), String> = BTreeMap::new(); for rel in rels.values().filter(|r| r.kind == "comments") { let Some(n) = pkg.node(&rel.target) else { continue }; + let authors: Vec = n.child("authors").map(|a| a.children("author").map(|x| x.text.trim().to_string()).collect()).unwrap_or_default(); for c in n.all("comment") { if let Some(p) = c.attr("ref").and_then(cell_pos) { - notes.insert(p, c.runs().trim().to_string()); + // Placeholders (`tc={…}`) stand for a threaded comment's author. + let author = c.attr("authorId").and_then(|i| authors.get(i.parse::().ok()?)).filter(|a| !a.starts_with("tc=")); + notes.insert(p, signed(author.map(String::as_str), c.runs().trim())); } } } - let mut threads: BTreeMap<(u32, u32), Vec> = BTreeMap::new(); + let mut threads: BTreeMap<(u32, u32), (Vec, bool)> = BTreeMap::new(); for rel in rels.values().filter(|r| r.kind == "threadedComment") { let Some(n) = pkg.node(&rel.target) else { continue }; for c in n.all("threadedComment") { if let Some(p) = c.attr("ref").and_then(cell_pos) { - threads.entry(p).or_default().push(c.child("text").map(|t| t.text.trim().to_string()).unwrap_or_default()); + let author = c.attr("personId").and_then(|id| cx.persons.get(id)).map(String::as_str); + let text = c.child("text").map(|t| t.text.trim().to_string()).unwrap_or_default(); + let t = threads.entry(p).or_default(); + t.0.push(signed(author, &text)); + t.1 |= c.flag("done"); } } } - for (p, msgs) in threads { - notes.insert(p, msgs.join("\n↳ ")); + for (p, (msgs, done)) in threads { + notes.insert(p, format!("{}{}", msgs.join("\n↳ "), if done { "\n(resolved)" } else { "" })); } // Notes are few and say things nothing else does: keep them in summarized rows too. at.gap = None; @@ -574,6 +740,15 @@ fn read_sheet(pkg: &mut Pkg, part: &str, kind: &str, sh: &mut Sheet, cx: &mut Sh Ok(()) } +/// `Author: text`, unless the text already opens with the author's name +/// (Excel writes `Author:` as a note's first line). +fn signed(author: Option<&str>, text: &str) -> String { + match author.filter(|a| !a.is_empty()) { + Some(a) if !text.starts_with(a) => format!("{a}: {text}"), + _ => text.to_string(), + } +} + /// Cells `read_sheet` creates (styled blanks, notes, links...), kept apart and /// appended once at the end: pushing them one by one onto a sheet's millions /// of cells would double that Vec's allocation. @@ -660,7 +835,7 @@ fn validation(n: &Node, f1: Option, f2: Option) -> Option ("ONE_OF_RANGE".to_string(), vec![f.replace('$', "")]), None => ("ONE_OF_LIST".to_string(), Vec::new()), }, - "custom" => ("CUSTOM_FORMULA".to_string(), f1.map(|f| format!("={f}")).into_iter().collect()), + "custom" => ("CUSTOM_FORMULA".to_string(), f1.map(|f| format!("={}", f.trim_start_matches('='))).into_iter().collect()), "whole" | "decimal" | "date" | "time" | "textLength" => { let pre = match ty { "whole" => "WHOLE_NUMBER", @@ -703,6 +878,11 @@ fn validation(n: &Node, f1: Option, f2: Option) -> Option`, or `` in x14 rules), as `=…`. +fn cf_formulas(rule: &Node) -> Vec { + rule.kids.iter().filter(|k| k.name == "formula" || k.name == "f").map(|k| format!("={}", k.text.trim().trim_start_matches('='))).collect() +} + fn cf_rule(rule: &Node, st: &Styles) -> String { let ty = rule.attr("type").unwrap_or("?"); // Main rules hold ``, x14 rules ``. @@ -780,6 +960,263 @@ fn table(n: &Node) -> Option
{ }) } +// ---------------------------------------------------------------- rich text & rich values + +/// Shared strings whose runs are formatted differently, by index. Streams the +/// part so a large plain string table costs nothing kept. +fn rich_strings(pkg: &mut Pkg, part: &str, st: &Styles) -> HashMap> { + match pkg.reader(part) { + Some(mut rd) => rich_string_table(&mut rd, st), + None => HashMap::new(), + } +} + +fn rich_string_table(rd: &mut Reader, st: &Styles) -> HashMap> { + let mut out = HashMap::new(); + let (mut buf, mut i) = (Vec::new(), 0u32); + loop { + buf.clear(); + match rd.read_event_into(&mut buf) { + Ok(Event::Start(e)) if e.local_name().as_ref() == "si" => { + let Ok(n) = read_children(rd, "si".into(), Vec::new(), &mut buf) else { break }; + if n.child("r").is_some() { + let runs = rich_runs(&n, st); + if !runs.is_empty() { + out.insert(i, runs); + } + } + i += 1; + } + Ok(Event::Empty(e)) if e.local_name().as_ref() == "si" => i += 1, + Ok(Event::Eof) | Err(_) => break, + _ => {} + } + } + out +} + +/// Runs of an `` / ``, merged where formatting repeats; empty when +/// every run looks the same. +fn rich_runs(n: &Node, st: &Styles) -> Vec { + let mut out: Vec = Vec::new(); + for k in &n.kids { + let (text, f) = match k.name.as_str() { + "t" => (k.text.clone(), Style::default()), + "r" => (k.children("t").map(|t| t.text.as_str()).collect(), k.child("rPr").map(|p| st.font_style(p)).unwrap_or_default()), + _ => continue, + }; + match out.last_mut() { + Some(last) if last.f == f => last.text.push_str(&text), + _ if text.is_empty() => {} + _ => out.push(Run { text, f }), + } + } + if !crate::model::runs_differ(&out) { + out.clear(); + } + out +} + +/// A rich value's structure type and its (key, key type, value) fields. +type RichFields<'a> = (&'a str, Vec<(&'a str, &'a str, &'a str)>); + +/// What each cell `vm` index points at: a picture placed in the cell or a +/// linked data type. Follows metadata.xml → rich value → its structure (and, +/// for pictures, richValueRel.xml → the image part). +fn rich_values(pkg: &mut Pkg) -> Vec> { + let Some(meta) = pkg.node("xl/metadata.xml") else { return Vec::new() }; + let types: Vec<&str> = meta.child("metadataTypes").map(|t| t.children("metadataType").map(|m| m.attr("name").unwrap_or("")).collect()).unwrap_or_default(); + let rv_index: Vec> = meta + .children("futureMetadata") + .find(|f| f.attr("name") == Some("XLRICHVALUE")) + .map(|f| f.children("bk").map(|bk| bk.all("rvb").first().and_then(|r| r.num("i"))).collect()) + .unwrap_or_default(); + let structs: Vec<(String, Vec<(String, String)>)> = pkg + .node("xl/richData/rdrichvaluestructure.xml") + .map(|n| { + n.children("s") + .map(|s| { + let keys = s.children("k").map(|k| (k.attr("n").unwrap_or("").to_string(), k.attr("t").unwrap_or("").to_string())).collect(); + (s.attr("t").unwrap_or("").to_string(), keys) + }) + .collect() + }) + .unwrap_or_default(); + let values: Vec<(usize, Vec)> = pkg + .node("xl/richData/rdrichvalue.xml") + .map(|n| n.children("rv").map(|rv| (rv.num("s").unwrap_or(0) as usize, rv.children("v").map(|v| v.text.clone()).collect())).collect()) + .unwrap_or_default(); + let rel_part = "xl/richData/richValueRel.xml"; + let rels = pkg.rels(rel_part); + let images: Vec> = pkg + .node(rel_part) + .map(|n| n.children("rel").map(|r| r.attr("id").and_then(|id| rels.get(id)).map(|x| x.target.clone())).collect()) + .unwrap_or_default(); + + // A rich value's fields by key name. + let fields = |i: usize| -> Option> { + let (s, vs) = values.get(i)?; + let (t, keys) = structs.get(*s)?; + Some((t.as_str(), keys.iter().zip(vs).map(|((k, kt), v)| (k.as_str(), kt.as_str(), v.as_str())).collect())) + }; + let object = |i: usize| -> Option { + let (t, kv) = fields(i)?; + let get = |name: &str| kv.iter().find(|(k, _, _)| *k == name).map(|(_, _, v)| v.to_string()); + if t.to_ascii_lowercase().contains("image") { + let target = get("_rvRel:LocalImageIdentifier").and_then(|r| r.parse::().ok()).and_then(|r| images.get(r).cloned().flatten()); + return Some(CellObject { kind: "image".into(), text: get("Text").filter(|t| !t.is_empty()), target, props: Vec::new() }); + } + let display = get("_DisplayString")?; + let mut props = Vec::new(); + for (k, kt, v) in &kv { + if k.starts_with('_') || k.starts_with('%') || props.len() >= 30 { + continue; + } + let v = match *kt { + "" | "s" | "b" | "d" => v.to_string(), + // Nested rich value: show its display name. + "r" => match v.parse::().ok().and_then(fields) { + Some((_, nested)) => match nested.iter().find(|(k, _, _)| *k == "_DisplayString") { + Some((_, _, d)) => d.to_string(), + None => continue, + }, + None => continue, + }, + _ => continue, + }; + props.push((k.to_string(), v)); + } + Some(CellObject { kind: "entity".into(), text: Some(display), target: None, props }) + }; + meta.child("valueMetadata") + .map(|vm| { + vm.children("bk") + .map(|bk| { + let rc = bk.child("rc")?; + let ty = types.get(rc.num("t")?.checked_sub(1)? as usize)?; + if *ty != "XLRICHVALUE" { + return None; + } + object((*rv_index.get(rc.num("v")? as usize)?)? as usize) + }) + .collect() + }) + .unwrap_or_default() +} + +// ---------------------------------------------------------------- shapes + +/// Pictures, shapes and text boxes on a drawing part (charts are read apart). +fn drawing_shapes(pkg: &mut Pkg, drawing: &str) -> Vec { + let Some(d) = pkg.node(drawing) else { return Vec::new() }; + let mut out = Vec::new(); + for a in d.kids.iter().filter(|k| k.name.ends_with("Anchor")) { + let rect = anchor_rect(a); + for k in &a.kids { + drawn(k, rect, &mut out); + } + } + out +} + +fn drawn(k: &Node, anchor: Option, out: &mut Vec) { + let props = |nv: &str| k.child(nv).and_then(|n| n.child("cNvPr")); + let mut shape = |kind: &str, p: Option<&Node>, text: Option| { + let alt = p.and_then(|p| p.attr("descr")).filter(|d| !d.is_empty()).map(String::from); + let on_click = k.attr("macro").and_then(macro_name); + if kind != "picture" && text.is_none() && on_click.is_none() && alt.is_none() { + return; // plain lines and boxes + } + out.push(Shape { + kind: kind.into(), + name: p.and_then(|p| p.attr("name")).unwrap_or("").into(), + anchor, + text, + alt, + on_click, + linked_cell: None, + }); + }; + match k.name.as_str() { + "pic" => shape("picture", props("nvPicPr"), None), + "sp" => { + let text_box = k.child("nvSpPr").and_then(|n| n.child("cNvSpPr")).is_some_and(|c| c.flag("txBox")); + let text = k.child("txBody").map(|b| b.children("p").map(|p| p.runs()).collect::>().join("\n").trim().to_string()).filter(|t| !t.is_empty()); + shape(if text_box { "text box" } else { "shape" }, props("nvSpPr"), text); + } + "grpSp" => { + for kid in &k.kids { + drawn(kid, anchor, out); + } + } + _ => {} + } +} + +/// Cells a drawing anchor covers. An edge at offset 0 sits on the cell's +/// border, so that row or column isn't covered. +fn anchor_rect(a: &Node) -> Option { + let pos = |n: &Node| -> Option<(u32, u32, bool)> { + let num = |k: &str| n.child(k)?.text.trim().parse::().ok(); + Some((num("row")?, num("col")?, num("rowOff").unwrap_or(0) == 0 && num("colOff").unwrap_or(0) == 0)) + }; + let (r0, c0, _) = pos(a.child("from")?)?; + let (mut r1, mut c1) = (r0, c0); + if let Some((r, c, on_edge)) = a.child("to").and_then(pos) { + let back = |v: u32, start: u32| if on_edge && v > start { v - 1 } else { v }; + (r1, c1) = (back(r, r0), back(c, c0)); + } + Some(Rect::new(r0, c0, r1, c1)) +} + +/// `[0]!RollUp`, `'Book.xlsm'!Module1.RollUp` → `RollUp`, `Module1.RollUp`. +fn macro_name(s: &str) -> Option { + let s = s.trim().trim_start_matches('='); + let name = s.rsplit('!').next().unwrap_or(s).trim_matches('\'').trim(); + (!name.is_empty()).then(|| name.to_string()) +} + +/// Form controls (buttons, checkboxes, dropdowns) from a VML drawing. +/// Comment boxes live there too and are skipped. +fn form_controls(pkg: &mut Pkg, part: &str) -> Vec { + let Some(n) = pkg.node(part) else { return Vec::new() }; + let mut out = Vec::new(); + for sh in n.all("shape") { + let Some(cd) = sh.child("ClientData") else { continue }; + let kind = match cd.attr("ObjectType").unwrap_or("") { + "Note" => continue, + "Button" => "button", + "Checkbox" => "checkbox", + "Drop" => "dropdown", + "List" => "list box", + "Radio" => "option button", + "Spin" => "spinner", + "Scroll" => "scroll bar", + "GBox" => "group box", + "Label" => "label", + "Edit" => "edit box", + other => other, + } + .to_ascii_lowercase(); + let text = sh.child("textbox").map(|t| t.all_text().trim().to_string()).filter(|t| !t.is_empty()); + let anchor = cd.child("Anchor").and_then(|a| { + let v: Vec = a.text.split(',').filter_map(|x| x.trim().parse().ok()).collect(); + (v.len() == 8).then(|| Rect::new(v[2], v[0], v[6], v[4])) + }); + let fmla = |k: &str| cd.child(k).map(|x| x.text.trim().to_string()).filter(|t| !t.is_empty()); + out.push(Shape { + kind, + name: sh.attr("alt").unwrap_or("").into(), + anchor, + text, + alt: None, + on_click: fmla("FmlaMacro").as_deref().and_then(macro_name), + linked_cell: fmla("FmlaLink"), + }); + } + out +} + // ---------------------------------------------------------------- charts & pivots /// Charts on a drawing part, in the order they're anchored. @@ -876,6 +1313,224 @@ fn pivot(pkg: &mut Pkg, part: &str) -> Option { }) } +// ---------------------------------------------------------------- workbook settings + +/// Defined names with their scope (`localSheetId` counts sheets in +/// workbook order). `_FilterDatabase` is Excel's hidden name for every autofilter. +fn defined_names(root: &Node) -> Option> { + let sheets: Vec<&str> = root.child("sheets")?.children("sheet").filter_map(|s| s.attr("name")).collect(); + let names = root.child("definedNames")?; + Some( + names + .children("definedName") + .filter_map(|n| { + let name = n.attr("name")?; + (!name.ends_with("_FilterDatabase")).then(|| NamedRange { + name: name.to_string(), + range: n.text.trim().trim_start_matches('=').to_string(), + scope: n.num("localSheetId").and_then(|i| sheets.get(i as usize)).map(|s| s.to_string()), + }) + }) + .collect(), + ) +} + +fn calc(root: &Node) -> Option { + let c = root.child("calcPr")?; + let mode = match c.attr("calcMode") { + Some("manual") => "manual", + Some("autoNoTable") => "automatic except tables", + _ => "automatic", + }; + let iterative = c.flag("iterate"); + if !iterative && mode == "automatic" { + return None; + } + Some(Calc { + mode: mode.into(), + iterative, + max_iterations: iterative.then(|| c.num("iterateCount").unwrap_or(100)), + max_change: iterative.then(|| c.attr("iterateDelta").and_then(|d| d.parse().ok()).unwrap_or(0.001)), + }) +} + +/// Core, extended and custom document properties, in that order. +fn properties(pkg: &mut Pkg) -> Vec<(String, String)> { + let rels = pkg.rels(""); + let part = |kind: &str| rels.values().find(|r| r.kind == kind).map(|r| r.target.clone()); + let mut out: Vec<(String, String)> = Vec::new(); + let mut add = |k: &str, v: &str| { + let v = v.trim(); + if !v.is_empty() { + out.push((k.to_string(), v.to_string())); + } + }; + if let Some(n) = part("core-properties").and_then(|p| pkg.node(&p)) { + for (tag, name) in [ + ("title", "title"), + ("subject", "subject"), + ("creator", "author"), + ("description", "description"), + ("keywords", "keywords"), + ("category", "category"), + ("contentStatus", "status"), + ("lastModifiedBy", "last modified by"), + ("created", "created"), + ("modified", "modified"), + ] { + if let Some(x) = n.child(tag) { + add(name, &x.text); + } + } + } + if let Some(n) = part("extended-properties").and_then(|p| pkg.node(&p)) { + for (tag, name) in [("Company", "company"), ("Manager", "manager")] { + if let Some(x) = n.child(tag) { + add(name, &x.text); + } + } + } + if let Some(n) = part("custom-properties").and_then(|p| pkg.node(&p)) { + for p in n.children("property") { + if let (Some(name), Some(v)) = (p.attr("name"), p.kids.first()) { + add(name, &v.text); + } + } + } + out +} + +fn connections(pkg: &mut Pkg, part: &str) -> Vec { + let Some(n) = pkg.node(part) else { return Vec::new() }; + let mut out = Vec::new(); + for c in n.children("connection") { + let mut x = Connection { + name: c.attr("name").unwrap_or("").into(), + description: c.attr("description").map(String::from), + refresh_on_open: c.flag("refreshOnLoad"), + kind: match c.attr("type") { + Some("1") => "odbc", + Some("2") => "dao", + Some("3") => "file", + Some("4") => "web", + Some("5") => "oledb", + Some("6") => "text", + Some("7") => "ado", + Some("8") => "dsp", + _ => "connection", + } + .into(), + ..Default::default() + }; + if let Some(db) = c.child("dbPr") { + let conn = db.attr("connection").unwrap_or(""); + // Power Query loads go through the Mashup provider; Location names the query. + if conn.contains("Microsoft.Mashup.OleDb") { + x.kind = "power query".into(); + x.source = conn_field(conn, "location"); + } else { + x.source = Some(scrub_secrets(conn)).filter(|s| !s.is_empty()); + } + x.command = db.attr("command").map(String::from); + } else if let Some(w) = c.child("webPr") { + x.source = w.attr("url").map(String::from); + } else if let Some(t) = c.child("textPr") { + x.source = t.attr("sourceFile").map(String::from); + } else if let Some(o) = c.child("olapPr") { + x.kind = "olap".into(); + x.source = o.attr("connection").map(scrub_secrets); + } + out.push(x); + } + out +} + +fn conn_field(conn: &str, key: &str) -> Option { + conn.split(';').find_map(|kv| kv.split_once('=').filter(|(k, _)| k.trim().eq_ignore_ascii_case(key)).map(|(_, v)| v.trim().to_string())) +} + +/// A connection string without its password. +fn scrub_secrets(conn: &str) -> String { + conn.split(';') + .filter(|kv| { + let k = kv.split('=').next().unwrap_or("").trim().to_ascii_lowercase(); + !matches!(k.as_str(), "pwd" | "password" | "jet oledb:database password") + }) + .collect::>() + .join(";") +} + +/// Power Query (M) code: base64 in a `customXml` part, holding a zip with +/// `Formulas/Section1.m`. One script per query. +fn power_query(pkg: &mut Pkg) -> Vec