diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fa9d253..ca534a9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -20,7 +20,7 @@ jobs: with: persist-credentials: false - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 - - name: Compile and inspect Go packages + - name: Test, compile, and inspect Go packages uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0 with: context: . diff --git a/BUILD.md b/BUILD.md index f1fc252..0cc75d1 100644 --- a/BUILD.md +++ b/BUILD.md @@ -212,6 +212,7 @@ limitations, and redistribution requirements. ```sh go build ./... go vet ./... +go test ./... ``` Frontend HTML, CSS, and JavaScript live in `internal/webui` and are embedded with diff --git a/Dockerfile b/Dockerfile index 135f813..f301c58 100644 --- a/Dockerfile +++ b/Dockerfile @@ -8,7 +8,9 @@ RUN go mod download FROM dependencies AS checks COPY . . -RUN CGO_ENABLED=0 go build ./... && CGO_ENABLED=0 go vet ./... \ +RUN CGO_ENABLED=0 go test ./... \ + && CGO_ENABLED=0 go build ./... \ + && CGO_ENABLED=0 go vet ./... \ && CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build ./... \ && CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go vet ./... diff --git a/MAINTAINER.md b/MAINTAINER.md index 55e6a67..9bf7ef7 100644 --- a/MAINTAINER.md +++ b/MAINTAINER.md @@ -78,6 +78,14 @@ itself on the destination machine. The Go toolchain is specified in [go.mod](go.mod) and the container build toolchain in [Dockerfile](Dockerfile). Keep them compatible when updating either one. +Release inputs pin Go modules and bundled OCR/WebView2 assets by version and +checksum. The Go container image is selected by version tag, while Debian package +repositories are live and package installation includes current upgrades. Thus a +rebuild uses the same source-level dependencies but may resolve different base +image and native system package revisions; release artifacts are not guaranteed +to be bit-for-bit reproducible. This intentionally favors current Debian security +updates over snapshot-based repeatability. + Release source collection must resolve the exact dependency versions used in the build. Missing sources stop publication rather than substituting a different version. Preserve the source and notice downloads alongside published binaries; diff --git a/README.md b/README.md index 5fadaf9..86593d4 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,10 @@ Shelfmark manages metadata for a local ebook collection. Scan a folder, find mat catalog records, edit metadata, and review selected changes before writing them to your books. Drafts are stored in SQLite and survive restarts. -The interface is a single page web application embedded in the Go executable. -Shelfmark can serve it in a browser or display it in an optional desktop window. -It does not require Calibre. +The web interface is embedded in the executable and can be used in a browser or +desktop window. + +Shelfmark does not require Calibre. ## Features @@ -37,19 +38,17 @@ chmod +x shelfmark-linux-amd64 The executable includes the HTTP server, UI, desktop helper, and Tesseract OCR with English and French language models. Linux releases bundle GTK/WebKitGTK; -Windows releases bundle a Fixed Version WebView2 runtime. Shelfmark starts its -HTTP server first and attempts to open a desktop window. If the window cannot -start, the server continues running so you can use a browser. For source builds, -see [BUILD.md](BUILD.md). +Windows releases bundle a Fixed Version WebView2 runtime. Shelfmark starts in +desktop mode by default. Use `-headless` to run the HTTP server without opening +a desktop window. If desktop mode is unavailable, use the browser interface. +For source builds, see [BUILD.md](BUILD.md). Open **http://127.0.0.1:8766**. Choose a folder on the computer running Shelfmark and scan it. **Back up your books before saving changes: Shelfmark does not create backup copies.** -1. Select a book and find catalog matches, or edit its metadata directly. - Searches use the title, subtitle, original title and author, plus ISBNs when - available. Sparse metadata falls back to a cleaned filename. You can edit the - terms to run a custom search. +1. Select a book and search for catalog matches, or edit its metadata directly. + Search terms come from available book metadata and can be edited. 2. Review the proposed metadata and the explanation for each field. Check the title, authors, series, volume, and edition details. Select only the values you want to apply; uncertain or conflicting suggestions start unchecked. @@ -59,84 +58,39 @@ backup copies.** **Discard selected drafts** for several. Discarding removes the whole draft without modifying the ebook file; closing the review keeps drafts for later. -Matching tries ISBN lookups and title/author variations, using cleaned filenames -when metadata is sparse. Results are ranked using title similarity, authors, -language, and ISBN agreement. Suggestions include explanations for individual -fields; uncertain or conflicting values start unchecked. Catalog records can be -incomplete or describe a different edition, so review them before applying changes. -Matching never modifies files automatically. +Catalog records can be incomplete or describe a different edition. Review +suggestions before applying them. Matching does not modify book files. -Choosing a folder in the folder picker starts a scan. Background matching searches -the selected folder and its scanned subfolders, filling the cache without applying -changes. Manual searches take priority. You can stop background matching from the -library; closing a browser tab does not stop it. A new scan or shutting down -Shelfmark cancels it. +Selecting a folder starts a scan. Background matching searches scanned books and +stores results without applying changes. It continues if the browser tab closes +and stops when cancelled, a new scan starts, or Shelfmark shuts down. ## Downloads Download release files from [GitHub Releases](https://github.com/Nejaa/ShelfMark/releases). -Choose the executable for your system, or the Docker archive for a container. -The checksums and source archives are accompanying downloads; they are not needed -to launch Shelfmark. - -The release assets contain: - -| File | Contents | -| --- | --- | -| `shelfmark-linux-amd64` | Linux desktop executable with bundled OCR | -| `shelfmark-windows-amd64.exe` | Windows desktop executable with bundled OCR | -| `shelfmark-linux-amd64-licenses.tar.gz` | Linux executable's dependency notices | -| `shelfmark-windows-amd64-licenses.tar.gz` | Windows executable's dependency notices | -| `shelfmark-docker-linux-amd64.tar.gz` | Compressed `docker save` archive for `docker load` | -| `shelfmark-source.tar.gz` | Project source from the release tag | -| `shelfmark-third-party-sources.tar.gz` | Corresponding dependency sources and models | -| `SHA256SUMS` | SHA-256 checksums for the release assets | +Choose the Linux or Windows executable, or the Docker archive. Releases also +include checksums, license notices, and source archives. ## Running ### Runtime requirements -Use an executable matching the operating system and CPU architecture. Go, Python, -a C/C++ compiler, and a separate SQLite installation are not needed to run it. +Release executables are available for Linux x64 and Windows x64. Go, Python, a +C/C++ compiler, and a separate SQLite installation are not required. -| System | HTTP-only server | Additional requirements for a desktop window | -| --- | --- | --- | -| Linux x64 release | A 64-bit Linux system; no GTK or WebKit installation. Bundled OCR requires glibc 2.36 or newer. | X11 or Wayland, glibc 2.36 or newer with its ELF loader, and kernel support for WebKit's sandbox. GTK, WebKitGTK, fonts, and software rendering libraries are bundled. | -| Windows x64 release | Windows 10/11 or a compatible Windows Server version | A graphical desktop session. Desktop builds bundle a compatible WebView2 runtime; no separate installation is needed. Current WebView2 targets Windows 10/11 and supported Windows Server editions; see [Microsoft's OS requirements](https://learn.microsoft.com/en-us/deployedge/microsoft-edge-supported-operating-systems). | - -Shelfmark needs a writable data directory, permission to read the library, and an -available TCP listen port. Saving metadata also needs permission to write book -files and create temporary files in their folders. A modern browser, local or on -another device, is needed to use the HTTP interface. - -Desktop mode needs enough writable temporary disk space to extract the bundled -runtime, and permission to execute the extracted helpers. On Linux, the temporary -filesystem must not be mounted `noexec`; use a short `TMPDIR` if path relocation -fails. The desktop and OCR helpers require glibc 2.36 or newer. WSL -needs WSLg or another display server for a Linux window. HTTP-only operation does -not need a display. - -On Windows, Shelfmark extracts the bundled Fixed Version WebView2 runtime onto -a local filesystem. An installed WebView2 runtime is not required. On first -desktop launch, read and agree to Microsoft's separate runtime terms in the -native confirmation dialog. Declining leaves HTTP mode available. The terms are -served at `/api/webview2-license`; acceptance is recorded beside the executable. -The runtime does not update itself; install a newer Shelfmark version to update it. - -Internet access is needed for enabled catalog searches and downloading remote -covers, but not for scanning or editing local books. - -Release executables include Tesseract and its dependencies, with English and -French language models. OCR works offline without a separate installation, -including in headless mode. The Linux OCR helper needs compatible glibc and its -loader; Windows uses OS system libraries. - -On first launch the bundled models are installed into `tessdata/` beside the -executable. Existing nonempty model files are preserved. To restore a damaged -model, remove its `eng.traineddata` or `fra.traineddata` file and restart. The -private OCR helper is extracted to a temporary directory and removed on shutdown; -that directory must permit execution. OCR failures leave the rest of the app -available. +| System | Requirements | +| --- | --- | +| Linux x64 | 64-bit Linux. Bundled desktop and OCR helpers require glibc 2.36 or newer. Desktop mode requires X11 or Wayland and kernel support for WebKit's sandbox. | +| Windows x64 | Windows 10/11 or a compatible Windows Server version. Desktop mode requires a graphical session; the release includes WebView2. | + +Shelfmark needs permission to read the library and a writable data directory. +Saving metadata also requires write access to book files and their folders. +Desktop mode requires writable temporary storage. WSL requires WSLg or another +display server for a Linux window. + +Release executables include offline Tesseract OCR with English and French +language models. Internet access is required for catalog searches and remote +cover downloads, but not for scanning or editing local books. ### Command-line options @@ -155,13 +109,10 @@ To access the interface from another device on a trusted network: Use the server's address from the other device. Folder paths always refer to the server's filesystem. Shelfmark has no user authentication; its API can browse -folders and update book files. Keep the default loopback binding unless you trust -the devices that can reach it. For remote access, use an authenticated reverse -proxy or an SSH tunnel. +folders and update book files. Bind to loopback unless network clients are +trusted. Use an authenticated reverse proxy or an SSH tunnel for remote access. -The HTTP server starts before a desktop window is attempted. If a window is -unavailable, the server stays running. Stop the server with Ctrl+C. Closing an -opened desktop window stops Shelfmark. +Stop an HTTP-only server with Ctrl+C. Closing the desktop window stops Shelfmark. ### Data and logs @@ -171,41 +122,17 @@ in a writable folder. `-data-dir` changes the database location; the log file stays beside the executable. Logs are appended across launches and may be removed when Shelfmark is stopped. -The Windows release executable opens no command console when launched from -Explorer. When launched from a terminal, Shelfmark reuses that console and also -writes its logs there. Redirected terminal output is preserved. Native GUI helper -diagnostics are included in `shelfmark.log`. - -Log records include their severity and structured details. The default `info` -level shows startup, scans, searches, draft saves, and feature availability. -`warn` includes recoverable failures and missing optional dependencies; `error` -includes failed saves and fatal startup errors. Enable `debug` for HTTP timings, -cache activity, generated search plans, catalog calls, candidate ranking, metadata -processing, and OCR progress: +Choose the minimum log level in **Settings → Logging**. The default is `info`; +`debug` adds diagnostic details: ```sh ./shelfmark-linux-amd64 -log-level debug ``` -Change the minimum level in **Settings → Logging** and save to apply it immediately. -The choice is stored in SQLite and restored on launch. `-log-level` overrides the -saved level at startup; Settings shows the active level and can change it for the -running session. The flag alone does not change the saved preference. - -The selected level includes all higher severities. Flag help and crash reports -remain available independently of the filter. OCR engine or language-data failures -produce a warning with troubleshooting steps when OCR is enabled. Settings also -shows a red warning if OCR is selected but unavailable on the server. - -Log records can contain local file paths. Debug logging also includes search terms -and candidate titles. API keys, request bodies, book text, and OCR output are not -logged. - -Because the Windows executable uses the GUI subsystem, `cmd.exe` can return to its -prompt while Shelfmark is running. Use `start /wait "" shelfmark-windows-amd64.exe` -if you want the shell to wait for it, or run it with `Start-Process -Wait` in -PowerShell. A windowless launch can be stopped through Task Manager if no console -is attached. +The log level is saved in the local database. `-log-level` overrides it for the +current launch. Logs may include local file paths; debug logs may include search +terms and candidate titles. API keys, request bodies, book text, and OCR output +are not logged. ## Docker @@ -225,10 +152,10 @@ docker run --rm --name shelfmark \ Open and scan `/books`. The container runs the HTTP server with bundled OCR and no desktop window. Its base image is `debian:bookworm-slim`. `/data` holds SQLite, logs, and OCR models. -The books mount needs write access for saving metadata; add `readonly` if you only -want to scan. The process uses UID/GID `10001:10001`: grant that identity access to -bind-mounted books and, if using a bind mount for `/data`, write access there. -A Docker named data volume is initialized with the image's ownership. +The books mount needs write access to save metadata; mount it as `readonly` for +scanning only. The process uses UID/GID `10001:10001`. Grant that identity access +to bind-mounted books and, when `/data` is a bind mount, write access there. A +Docker named data volume is initialized with the image's ownership. On Windows with Docker Desktop's Linux containers, use a Windows source path for the books mount. Paths entered in Shelfmark still refer to the container, such as @@ -241,8 +168,8 @@ the application does not provide authentication. Stop it with `docker stop shelf ## Settings -Settings are grouped by purpose and stored on the server. Changes affect future -operations; scanning preferences require a new scan. +Settings are stored on the server. Changes affect future operations; scanning +preferences take effect on the next scan. | Category | Controls | | --- | --- | @@ -251,25 +178,19 @@ operations; scanning preferences require a new scan. | Matching | Local content inspection, optional OCR, maximum candidates, cache duration, background search delay | | Logging | Minimum severity (debug, info, warning, error), applied immediately when saved | -Catalogs receive search terms and discovered ISBNs. Book text is inspected locally -and is not uploaded. OCR of opening images uses the bundled English and French -models in release executables. Turning off content inspection -also disables OCR for searches. OCR is used for opening images in EPUB and CBZ; -PDF matching reads the text layer and does not OCR scanned PDF pages. Custom -search terms bypass local content inspection. - -Cached searches expire after seven days by default; set the duration to zero to -disable caching. The cache holds at most 500 search entries. **Find matches** retries a -full search when the cached result is empty. Check **Ignore cache** beside the -button to force a fresh search even when cached matches exist. Successful fresh -searches update the cache. Background matching can reuse empty results. Provider -failures are shown separately from empty results and are not cached. Matching scores are heuristics; -verify the edition before using a record. Open Library results can describe a -work rather than the specific edition of your file. - -API keys are stored unencrypted in the local database and are not returned to the -browser. Leave the key field blank to keep a saved key, or use the remove control -to delete it from the active settings. +Catalog searches use search terms and discovered ISBNs. Book text is inspected +locally and is not uploaded. OCR searches opening images in EPUB and CBZ files; +PDF matching reads its text layer. Release executables include English and +French OCR models. Custom search terms bypass local content inspection. + +Cached results appear when a book is selected. Use **Find matches** to search +catalogs, or **Ignore cache** to run a fresh search. Cache duration and size are +configurable. Catalog results can describe a different edition; Open Library +results may describe a work rather than a specific edition. + +API keys are stored unencrypted in the local database and are not sent to the +browser. Leave the key field blank to retain a saved key, or remove it in +settings. ## Files and data @@ -280,22 +201,19 @@ to delete it from the active settings. | CBZ | `ComicInfo.xml` | Not supported; archive pages remain intact | | PDF | Document properties | Not supported | -Additional properties may not be displayed by every ebook reader. Some formats -store fewer details than others: FB2 has one language, for example. DRM-protected, -encrypted, or malformed books may not be editable. +Additional properties may not be displayed by every ebook reader. FB2 supports +one language field. DRM-protected, encrypted, or malformed books may not be +editable. Selected updates replace the original file through a temporary file in the same folder. File permissions are preserved. **Shelfmark does not create backup -copies.** Back up your collection before editing it. A save checks whether the -file or draft changed since the review was opened and asks for a new review when -needed. Saving several books is not a transaction: failures are reported per -book, and earlier successful updates remain saved. - -`shelfmark.sqlite3` contains settings, catalog caches, and drafts. SQLite may also -create `-wal` and `-shm` files while the app is running. Stop Shelfmark before -copying its data directory. Original book files are stored separately. Existing -databases are migrated automatically when opened; preserve a copy before using a -newer version if you need to return to an older executable. +copies.** Back up your collection before editing. Saves check for changes made +since review; books are saved independently, with failures reported per book. + +`shelfmark.sqlite3` contains settings, catalog caches, and drafts. Stop Shelfmark +before copying its data directory. Book files are stored separately. Databases +are migrated automatically when opened; keep a copy if you need to use an older +version later. ## Project documentation diff --git a/cmd/shelfmark/options_test.go b/cmd/shelfmark/options_test.go new file mode 100644 index 0000000..737d723 --- /dev/null +++ b/cmd/shelfmark/options_test.go @@ -0,0 +1,70 @@ +package main + +import ( + "errors" + "flag" + "io" + "os" + "testing" + + "log/slog" +) + +func TestParseOptions(t *testing.T) { + originalArgs := os.Args + originalOutput := diagnosticOutput + t.Cleanup(func() { + os.Args = originalArgs + diagnosticOutput = originalOutput + }) + diagnosticOutput = io.Discard + + for _, test := range []struct { + name string + args []string + want options + wantErr bool + wantHelp bool + }{ + { + name: "defaults", + want: options{listen: "127.0.0.1:8766", dataDir: "/app", logLevel: slog.LevelInfo}, + }, + { + name: "explicit options normalize level", + args: []string{"-headless", "-listen", "localhost:9000", "-data-dir", "/data", "-log-level", " WARN "}, + want: options{headless: true, listen: "localhost:9000", dataDir: "/data", logLevel: slog.LevelWarn, logLevelSet: true}, + }, + { + name: "empty data directory uses executable directory", + args: []string{"-data-dir="}, + want: options{listen: "127.0.0.1:8766", dataDir: "/app", logLevel: slog.LevelInfo}, + }, + {name: "invalid level", args: []string{"-log-level", "trace"}, wantErr: true}, + {name: "unknown option", args: []string{"-unknown"}, wantErr: true}, + {name: "help", args: []string{"-h"}, wantHelp: true}, + } { + t.Run(test.name, func(t *testing.T) { + os.Args = append([]string{"shelfmark"}, test.args...) + got, err := parseOptions("/app") + if test.wantHelp { + if !errors.Is(err, flag.ErrHelp) { + t.Fatalf("parseOptions() error = %v, want flag.ErrHelp", err) + } + return + } + if test.wantErr { + if err == nil { + t.Fatalf("parseOptions() = %+v, want error", got) + } + return + } + if err != nil { + t.Fatalf("parseOptions() error = %v", err) + } + if got != test.want { + t.Fatalf("parseOptions() = %+v, want %+v", got, test.want) + } + }) + } +} diff --git a/docs/architecture.md b/docs/architecture.md index 3a13976..c106c04 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -119,7 +119,8 @@ ISBNs available for matching and a note about edition ambiguity. ISBN lookups also fetch Open Library edition JSON; the work record supplies fallback author names, while the edition supplies its actual title, subtitle, languages, series labels, publisher and date. Missing edition records are normal; -transport failures remain warnings and partial results are not cached. Explicit +transport failures remain warnings; useful partial results are cached, while +failed searches with no candidates are not. Explicit contributor roles and labelled translation statements provide translator evidence. EPUB readers recognize inline and refined contributor roles, and writes preserve unselected roles. diff --git a/docs/ux.md b/docs/ux.md index 6700ede..8885158 100644 --- a/docs/ux.md +++ b/docs/ux.md @@ -11,6 +11,8 @@ Matching. New groups can be added without expanding a single API-key dialog. - Navigate books and matches with the keyboard; dialogs manage focus and Escape. - Edit descriptions over several lines and clear metadata fields intentionally. - Stop background matching, and see catalog failures alongside useful results. +- Review and discard staged drafts without changing ebook files; background matching + is owned by the server and continues if the browser closes. - Keep API keys masked with a password field; saving other settings preserves the key. - Use small screens with stacked panels and scrollable metadata tables. - See folder paths as paths on the server, including when browsing remotely. @@ -23,9 +25,8 @@ Matching. New groups can be added without expanding a single API-key dialog. 2. **Scan progress.** Show discovered/read/skipped counts and cancellation for large collections. Separate job state from the current selection so users can keep reviewing while a scan runs. -3. **Draft management.** Add a drafts-only library filter, discard actions, and - recovery for books that were moved or deleted. Discard should show what will - be lost and leave book files unchanged. +3. **Draft management.** Add a drafts-only library filter and recovery for books + that were moved or deleted. Discard actions are available from the draft review. 4. **Metadata profiles.** Allow users to choose visible fields and default field selection for catalog matches. Keep manual editing available for every supported field and show format-specific limitations beside the control. @@ -37,5 +38,5 @@ Matching. New groups can be added without expanding a single API-key dialog. states. Keep these browser preferences separate from server-wide settings. For larger libraries, use a paginated or virtualized list before adding more -client-side filters. Background work currently belongs to the open browser; -continuing jobs after it closes would require server-owned job state. +client-side filters. Background matching is server-owned, but its job status is +held in memory and is lost when Shelfmark shuts down. diff --git a/internal/app/proposal.go b/internal/app/proposal.go index 0288717..b7243dc 100644 --- a/internal/app/proposal.go +++ b/internal/app/proposal.go @@ -77,7 +77,12 @@ func (s *Server) proposal(w http.ResponseWriter, r *http.Request) { if edited { evidence = "Manual edit" } - fields = append(fields, proposalField{Evidence: evidence, ReviewRequired: review, Name: name, Current: library.Text(current[name]), Value: library.Text(proposed[name]), + inputValue, err := library.InputText(name, proposed[name]) + if err != nil { + jsonError(w, err) + return + } + fields = append(fields, proposalField{Evidence: evidence, ReviewRequired: review, Name: name, Current: library.Text(current[name]), Value: inputValue, Changed: changed, Supported: supported, Suggested: changed && supported && !review && (edited || library.Text(proposed[name]) != "" || (name == "subtitle" && evidence != ""))}) } notes := "" diff --git a/internal/app/search.go b/internal/app/search.go index 7d2833b..9fd6026 100644 --- a/internal/app/search.go +++ b/internal/app/search.go @@ -37,30 +37,52 @@ func (s *Server) search(w http.ResponseWriter, r *http.Request) { jsonOut(w, result) } +// cachedSearch loads suggestions on selection without scheduling a catalog +// request or running content inspection. A miss leaves explicit search to the user. +func (s *Server) cachedSearch(w http.ResponseWriter, r *http.Request) { + var in struct { + ID string `json:"id"` + } + if err := decode(r, &in); err != nil { + jsonError(w, err) + return + } + result, err := s.findMatches(r.Context(), in.ID, nil, lookupOptions{CacheOnly: true}) + if err != nil { + jsonError(w, err) + return + } + jsonOut(w, result) +} + // lookupOptions controls scheduling and cache reuse for one search action. type lookupOptions struct { Background bool IgnoreCache bool + CacheOnly bool } // findMatches is shared by explicit lookups and the server-owned background job. // A nil query asks the backend to derive terms from the current working metadata. func (s *Server) findMatches(ctx context.Context, id string, query *string, options lookupOptions) (catalog.Result, error) { background := options.Background - if !background { + if !background && !options.CacheOnly { s.manualSearches.Add(1) defer s.manualSearches.Add(-1) } if query != nil && strings.TrimSpace(*query) == "" { return catalog.Result{}, errors.New("enter a title, author or ISBN") } - select { - case s.searches <- struct{}{}: - defer func() { - <-s.searches - }() - case <-ctx.Done(): - return catalog.Result{}, ctx.Err() + // Reading cached matches must not wait behind a background catalog request. + if !options.CacheOnly { + select { + case s.searches <- struct{}{}: + defer func() { + <-s.searches + }() + case <-ctx.Done(): + return catalog.Result{}, ctx.Err() + } } path := s.lookupPath(id) @@ -73,8 +95,15 @@ func (s *Server) findMatches(ctx context.Context, id string, query *string, opti return catalog.Result{}, err } - slog.Info("metadata search started", "book_id", id, "background", background, "ignore_cache", options.IgnoreCache) + if options.CacheOnly { + slog.Debug("loading cached matches for book selection", "book_id", id) + } else { + slog.Info("metadata search started", "book_id", id, "background", background, "ignore_cache", options.IgnoreCache) + } if !prefs.GoogleBooks && !prefs.OpenLibrary && !prefs.InternetArchive { + if options.CacheOnly { + return catalog.Result{Candidates: []catalog.Candidate{}}, nil + } slog.Warn("metadata search unavailable: no catalogs enabled", "hint", "enable at least one catalog in Settings") return catalog.Result{ Candidates: []catalog.Candidate{}, @@ -110,17 +139,25 @@ func (s *Server) findMatches(ctx context.Context, id string, query *string, opti }{catalogOptions, prefs.InspectContent, prefs.OCR, current, identity, query != nil}) key := catalog.CacheKey(fp+string(encoded), terms) // Bypassing cache reads still allows a successful fresh search to replace - // the old entry. Failures retain the existing cache and are never cached. + // the old entry. Useful partial results retain their provider warnings; a + // failed search with no candidates must not replace an existing entry. if options.IgnoreCache { slog.Debug("search cache bypassed by request", "book_id", id) } else { var cached catalog.Result ok, err := s.db.CacheGet(ctx, key, time.Duration(prefs.CacheDays)*24*time.Hour, &cached) if err != nil { + if options.CacheOnly { + slog.Warn("could not load cached catalog matches", "book_id", id, "error", err) + return catalog.Result{}, err + } slog.Warn("search cache read failed; querying catalogs", "error", err) } if ok && err == nil { slog.Debug("search cache hit", "book_id", id, "candidates", len(cached.Candidates)) + if options.CacheOnly { + return cached, nil + } // Empty results avoid repeated background work, but an explicit lookup // must retry content inspection and all enabled catalogs for new matches. if background || len(cached.Candidates) > 0 { @@ -132,6 +169,9 @@ func (s *Server) findMatches(ctx context.Context, id string, query *string, opti slog.Debug("search cache miss or disabled", "book_id", id) } } + if options.CacheOnly { + return catalog.Result{Candidates: []catalog.Candidate{}}, nil + } // Indexed content is read locally. It is never submitted to a catalog service. clues := "" if prefs.InspectContent && query == nil { @@ -156,9 +196,11 @@ func (s *Server) findMatches(ctx context.Context, id string, query *string, opti results := s.catalog.Search(ctx, catalog.SearchRequest{ Query: terms, CustomQuery: query != nil, Identity: identity, ISBNs: isbns, Background: background, }, catalogOptions) - if len(results.Warnings) == 0 && ctx.Err() == nil && prefs.CacheDays > 0 { + if (len(results.Candidates) > 0 || len(results.Warnings) == 0) && ctx.Err() == nil && prefs.CacheDays > 0 { if err := s.db.CachePut(ctx, key, results); err != nil { slog.Warn("could not cache search results", "error", err) + } else { + slog.Debug("catalog results cached", "book_id", id, "candidates", len(results.Candidates), "warnings", len(results.Warnings)) } } diff --git a/internal/app/server.go b/internal/app/server.go index da11e6b..8d28501 100644 --- a/internal/app/server.go +++ b/internal/app/server.go @@ -100,6 +100,7 @@ func (s *Server) Handler() http.Handler { mux.HandleFunc("GET /api/background", s.backgroundStatus) mux.HandleFunc("POST /api/cover", s.cover) mux.HandleFunc("POST /api/search", s.search) + mux.HandleFunc("POST /api/search/cached", s.cachedSearch) mux.HandleFunc("POST /api/ui-error", s.uiError) mux.HandleFunc("POST /api/apply", s.apply) mux.HandleFunc("POST /api/staged-count", s.stagedCount) diff --git a/internal/app/server_test.go b/internal/app/server_test.go new file mode 100644 index 0000000..f017321 --- /dev/null +++ b/internal/app/server_test.go @@ -0,0 +1,512 @@ +package app + +import ( + "archive/zip" + "context" + "encoding/json" + "encoding/xml" + "fmt" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + "shelfmark/internal/catalog" + "shelfmark/internal/library" + "shelfmark/internal/metadata" + "shelfmark/internal/settings" +) + +type memoryRepository struct { + mu sync.Mutex + settings map[string]string + states map[string]library.State + cache map[string]json.RawMessage +} + +func newMemoryRepository() *memoryRepository { + return &memoryRepository{settings: map[string]string{}, states: map[string]library.State{}, cache: map[string]json.RawMessage{}} +} + +func (m *memoryRepository) GetSetting(_ context.Context, key string) (string, error) { + m.mu.Lock() + defer m.mu.Unlock() + return m.settings[key], nil +} + +func (m *memoryRepository) SetSetting(_ context.Context, key, value string) error { + m.mu.Lock() + defer m.mu.Unlock() + m.settings[key] = value + return nil +} + +func (m *memoryRepository) CacheGet(_ context.Context, key string, _ time.Duration, dst any) (bool, error) { + m.mu.Lock() + defer m.mu.Unlock() + raw, ok := m.cache[key] + if !ok { + return false, nil + } + return true, json.Unmarshal(raw, dst) +} + +func (m *memoryRepository) CachePut(_ context.Context, key string, value any) error { + raw, err := json.Marshal(value) + if err != nil { + return err + } + m.mu.Lock() + defer m.mu.Unlock() + m.cache[key] = raw + return nil +} + +func (m *memoryRepository) ReadBook(_ context.Context, path, fingerprint string, metadata map[string]any) (map[string]any, bool, error) { + m.mu.Lock() + defer m.mu.Unlock() + state, ok := m.states[path] + if !ok || state.Fingerprint != fingerprint { + state = library.State{Path: path, Fingerprint: fingerprint, Metadata: cloneMap(metadata), Staged: map[string]any{}} + m.states[path] = state + } + return library.Working(state.Metadata, state.Staged), len(state.Staged) > 0, nil +} + +func (m *memoryRepository) Stage(_ context.Context, path, fingerprint string, metadata, staged map[string]any) error { + m.mu.Lock() + defer m.mu.Unlock() + m.states[path] = library.State{Path: path, Fingerprint: fingerprint, Metadata: cloneMap(metadata), Staged: cloneMap(staged)} + return nil +} + +func (m *memoryRepository) State(_ context.Context, path string) (library.State, error) { + m.mu.Lock() + defer m.mu.Unlock() + state, ok := m.states[path] + if !ok { + return library.State{Path: path, Metadata: map[string]any{}, Staged: map[string]any{}}, nil + } + state.Metadata = cloneMap(state.Metadata) + state.Staged = cloneMap(state.Staged) + return state, nil +} + +func (m *memoryRepository) Staged(_ context.Context) ([]library.State, error) { + m.mu.Lock() + defer m.mu.Unlock() + result := make([]library.State, 0) + for _, state := range m.states { + if len(state.Staged) > 0 { + state.Metadata = cloneMap(state.Metadata) + state.Staged = cloneMap(state.Staged) + result = append(result, state) + } + } + return result, nil +} + +func (m *memoryRepository) DiscardDrafts(_ context.Context, paths []string) error { + m.mu.Lock() + defer m.mu.Unlock() + for _, path := range paths { + state := m.states[path] + state.Staged = map[string]any{} + m.states[path] = state + } + return nil +} + +func (m *memoryRepository) FinishSave(_ context.Context, path, fingerprint string, metadata, staged map[string]any) error { + m.mu.Lock() + defer m.mu.Unlock() + m.states[path] = library.State{Path: path, Fingerprint: fingerprint, Metadata: cloneMap(metadata), Staged: cloneMap(staged)} + return nil +} + +func cloneMap(src map[string]any) map[string]any { + out := make(map[string]any, len(src)) + for key, value := range src { + out[key] = value + } + return out +} + +type fakeCatalog struct { + mu sync.Mutex + calls int + request catalog.SearchRequest + result catalog.Result +} + +func (f *fakeCatalog) Search(_ context.Context, request catalog.SearchRequest, _ catalog.Options) catalog.Result { + f.mu.Lock() + defer f.mu.Unlock() + f.calls++ + f.request = request + return f.result +} + +func (f *fakeCatalog) callCount() int { + f.mu.Lock() + defer f.mu.Unlock() + return f.calls +} + +func newTestServer(t *testing.T) (*Server, *memoryRepository, string, string) { + t.Helper() + root := t.TempDir() + path := filepath.Join(root, "book.epub") + writeTestEPUB(t, path, "Old title") + db := newMemoryRepository() + s := New(db, nil) + response := request(t, s.Handler(), http.MethodPost, "/api/scan", `{"root":`+mustJSON(t, root)+`}`, "application/json") + if response.Code != http.StatusOK { + t.Fatalf("scan failed: %d %s", response.Code, response.Body.String()) + } + return s, db, root, path +} + +func writeTestEPUB(t *testing.T, path, title string) { + t.Helper() + file, err := os.Create(path) + if err != nil { + t.Fatal(err) + } + zw := zip.NewWriter(file) + mimetype, err := zw.CreateHeader(&zip.FileHeader{Name: "mimetype", Method: zip.Store}) + if err != nil { + t.Fatal(err) + } + if _, err := io.WriteString(mimetype, "application/epub+zip"); err != nil { + t.Fatal(err) + } + var escapedTitle strings.Builder + if err := xml.EscapeText(&escapedTitle, []byte(title)); err != nil { + t.Fatal(err) + } + entries := []struct{ name, data string }{ + {"META-INF/container.xml", ``}, + {"OEBPS/content.opf", fmt.Sprintf(`%surn:isbn:9780306406157enTest Author`, escapedTitle.String())}, + } + for _, entry := range entries { + writer, err := zw.Create(entry.name) + if err != nil { + t.Fatal(err) + } + if _, err := io.WriteString(writer, entry.data); err != nil { + t.Fatal(err) + } + } + chapter, err := zw.Create("OEBPS/chapter.xhtml") + if err != nil { + t.Fatal(err) + } + if _, err := io.WriteString(chapter, `Chapter

Test chapter.

`); err != nil { + t.Fatal(err) + } + if err := zw.Close(); err != nil { + t.Fatal(err) + } + if err := file.Close(); err != nil { + t.Fatal(err) + } +} + +func request(t *testing.T, handler http.Handler, method, target, body, contentType string) *httptest.ResponseRecorder { + t.Helper() + req := httptest.NewRequest(method, target, strings.NewReader(body)) + if contentType != "" { + req.Header.Set("Content-Type", contentType) + } + response := httptest.NewRecorder() + handler.ServeHTTP(response, req) + return response +} + +func mustJSON(t *testing.T, value any) string { + t.Helper() + raw, err := json.Marshal(value) + if err != nil { + t.Fatal(err) + } + return string(raw) +} + +func TestHandlerRejectsMalformedJSONAndCrossOriginRequests(t *testing.T) { + db := newMemoryRepository() + server := New(db, nil) + handler := server.Handler() + + for _, test := range []struct { + name, method, path, body, contentType string + origin, fetchSite string + status int + }{ + {name: "unknown field", method: http.MethodPost, path: "/api/scan", body: `{"root":"unused","unexpected":true}`, contentType: "application/json", status: http.StatusBadRequest}, + {name: "trailing json", method: http.MethodPost, path: "/api/scan", body: `{"root":"unused"} {}`, contentType: "application/json", status: http.StatusBadRequest}, + {name: "wrong content type", method: http.MethodPost, path: "/api/scan", body: `{}`, contentType: "text/plain", status: http.StatusUnsupportedMediaType}, + {name: "foreign origin", method: http.MethodPost, path: "/api/scan", body: `{}`, contentType: "application/json", origin: "https://evil.example", status: http.StatusForbidden}, + {name: "cross site fetch", method: http.MethodPost, path: "/api/scan", body: `{}`, contentType: "application/json", fetchSite: "cross-site", status: http.StatusForbidden}, + } { + t.Run(test.name, func(t *testing.T) { + req := httptest.NewRequest(test.method, test.path, strings.NewReader(test.body)) + req.Header.Set("Content-Type", test.contentType) + if test.origin != "" { + req.Header.Set("Origin", test.origin) + } + if test.fetchSite != "" { + req.Header.Set("Sec-Fetch-Site", test.fetchSite) + } + response := httptest.NewRecorder() + handler.ServeHTTP(response, req) + if response.Code != test.status { + t.Fatalf("status = %d, want %d (%s)", response.Code, test.status, response.Body.String()) + } + if response.Header().Get("X-Content-Type-Options") != "nosniff" || response.Header().Get("Referrer-Policy") != "no-referrer" || response.Header().Get("Content-Security-Policy") == "" { + t.Fatalf("security headers missing: %#v", response.Header()) + } + if response.Header().Get("Cache-Control") != "no-store" { + t.Fatalf("API response Cache-Control = %q", response.Header().Get("Cache-Control")) + } + }) + } +} + +func TestApplyStageSaveSelectedFieldAndDiscard(t *testing.T) { + server, db, _, path := newTestServer(t) + id := idFor(path) + proposal := request(t, server.Handler(), http.MethodPost, "/api/proposal", `{"id":`+mustJSON(t, id)+`,"values":{"title":"Manual proposal"}}`, "application/json") + if proposal.Code != http.StatusOK { + t.Fatalf("proposal failed: %d %s", proposal.Code, proposal.Body.String()) + } + var proposalResult struct { + Fields []proposalField `json:"fields"` + CoverEditable bool `json:"cover_editable"` + } + if err := json.Unmarshal(proposal.Body.Bytes(), &proposalResult); err != nil { + t.Fatal(err) + } + if len(proposalResult.Fields) == 0 || !proposalResult.CoverEditable { + t.Fatalf("proposal response missing editor fields: %+v", proposalResult) + } + foundTitle := false + for _, field := range proposalResult.Fields { + if field.Name == "title" { + foundTitle = field.Value == "Manual proposal" && field.Changed + } + } + if !foundTitle { + t.Fatalf("manual title edit not reflected in proposal: %+v", proposalResult.Fields) + } + apply := request(t, server.Handler(), http.MethodPost, "/api/apply", `{"id":`+mustJSON(t, id)+`,"metadata":{"title":"Reviewed title","subtitle":"Unselected subtitle"}}`, "application/json") + if apply.Code != http.StatusOK { + t.Fatalf("apply failed: %d %s", apply.Code, apply.Body.String()) + } + if state, _ := db.State(context.Background(), path); state.Staged["title"] != "Reviewed title" || state.Staged["subtitle"] != "Unselected subtitle" { + t.Fatalf("apply did not stage supplied metadata: %#v", state.Staged) + } + + stageView := request(t, server.Handler(), http.MethodPost, "/api/staged", `{}`, "application/json") + if stageView.Code != http.StatusOK { + t.Fatalf("staged review failed: %s", stageView.Body.String()) + } + var review struct { + Books []struct { + Fingerprint string `json:"fingerprint"` + DraftVersion string `json:"draft_version"` + } `json:"books"` + } + if err := json.Unmarshal(stageView.Body.Bytes(), &review); err != nil { + t.Fatal(err) + } + if len(review.Books) != 1 { + t.Fatalf("staged review books = %d", len(review.Books)) + } + first := review.Books[0] + saveBody := mustJSON(t, map[string]any{"books": []any{map[string]any{"id": id, "fingerprint": first.Fingerprint, "draft_version": first.DraftVersion, "fields": []string{"title"}}}}) + saved := request(t, server.Handler(), http.MethodPost, "/api/save", saveBody, "application/json") + if saved.Code != http.StatusOK { + t.Fatalf("save failed: %d %s", saved.Code, saved.Body.String()) + } + fileMetadata, err := readFileMetadata(path) + if err != nil { + t.Fatal(err) + } + if fileMetadata["title"] != "Reviewed title" { + t.Fatalf("saved file title = %#v", fileMetadata["title"]) + } + state, _ := db.State(context.Background(), path) + if state.Staged["subtitle"] != "Unselected subtitle" || state.Staged["title"] != nil { + t.Fatalf("save should retain only unselected fields, got %#v", state.Staged) + } + + stageView = request(t, server.Handler(), http.MethodPost, "/api/staged", `{}`, "application/json") + if err := json.Unmarshal(stageView.Body.Bytes(), &review); err != nil { + t.Fatal(err) + } + second := review.Books[0] + saveBody = mustJSON(t, map[string]any{"books": []any{map[string]any{"id": id, "fingerprint": second.Fingerprint, "draft_version": second.DraftVersion, "fields": []string{"subtitle"}}}}) + saved = request(t, server.Handler(), http.MethodPost, "/api/save", saveBody, "application/json") + if saved.Code != http.StatusOK { + t.Fatalf("second save failed: %d %s", saved.Code, saved.Body.String()) + } + state, _ = db.State(context.Background(), path) + if len(state.Staged) != 0 { + t.Fatalf("all selected fields should be cleared after save, got %#v", state.Staged) + } + + apply = request(t, server.Handler(), http.MethodPost, "/api/apply", `{"id":`+mustJSON(t, id)+`,"metadata":{"title":"Discard me"}}`, "application/json") + if apply.Code != http.StatusOK { + t.Fatalf("second apply failed: %s", apply.Body.String()) + } + state, _ = db.State(context.Background(), path) + discardBody := mustJSON(t, map[string]any{"books": []any{map[string]string{"id": id, "draft_version": draftVersion(state.Staged)}}}) + discard := request(t, server.Handler(), http.MethodPost, "/api/drafts/discard", discardBody, "application/json") + if discard.Code != http.StatusOK { + t.Fatalf("discard failed: %d %s", discard.Code, discard.Body.String()) + } + state, _ = db.State(context.Background(), path) + if len(state.Staged) != 0 { + t.Fatalf("discard left staged fields: %#v", state.Staged) + } +} + +func TestSaveRejectsStaleFileAndDraftVersionsWithoutWriting(t *testing.T) { + server, db, _, path := newTestServer(t) + id := idFor(path) + response := request(t, server.Handler(), http.MethodPost, "/api/apply", `{"id":`+mustJSON(t, id)+`,"metadata":{"title":"Staged title"}}`, "application/json") + if response.Code != http.StatusOK { + t.Fatalf("apply failed: %s", response.Body.String()) + } + response = request(t, server.Handler(), http.MethodPost, "/api/staged", `{}`, "application/json") + var review struct { + Books []struct { + Fingerprint string `json:"fingerprint"` + DraftVersion string `json:"draft_version"` + } `json:"books"` + } + if err := json.Unmarshal(response.Body.Bytes(), &review); err != nil { + t.Fatal(err) + } + if len(review.Books) != 1 { + t.Fatalf("review failed: %s", response.Body.String()) + } + preconditions := review.Books[0] + writeTestEPUB(t, path, "External title") + body := mustJSON(t, map[string]any{"books": []any{map[string]any{"id": id, "fingerprint": preconditions.Fingerprint, "draft_version": preconditions.DraftVersion, "fields": []string{"title"}}}}) + response = request(t, server.Handler(), http.MethodPost, "/api/save", body, "application/json") + if response.Code != http.StatusBadRequest { + t.Fatalf("stale file save status = %d, want 400: %s", response.Code, response.Body.String()) + } + meta, err := readFileMetadata(path) + if err != nil { + t.Fatal(err) + } + if meta["title"] != "External title" { + t.Fatalf("stale save overwrote external update: %#v", meta["title"]) + } + if state, _ := db.State(context.Background(), path); state.Staged["title"] != "Staged title" { + t.Fatalf("stale save cleared draft: %#v", state.Staged) + } +} + +func TestSaveAndDiscardRejectStaleDraftVersion(t *testing.T) { + server, db, _, path := newTestServer(t) + id := idFor(path) + response := request(t, server.Handler(), http.MethodPost, "/api/apply", `{"id":`+mustJSON(t, id)+`,"metadata":{"title":"First draft"}}`, "application/json") + if response.Code != http.StatusOK { + t.Fatalf("apply failed: %s", response.Body.String()) + } + response = request(t, server.Handler(), http.MethodPost, "/api/staged", `{}`, "application/json") + var review struct { + Books []struct { + Fingerprint string `json:"fingerprint"` + DraftVersion string `json:"draft_version"` + } `json:"books"` + } + if err := json.Unmarshal(response.Body.Bytes(), &review); err != nil || len(review.Books) != 1 { + t.Fatalf("staged review invalid: %v %s", err, response.Body.String()) + } + oldReview := review.Books[0] + response = request(t, server.Handler(), http.MethodPost, "/api/apply", `{"id":`+mustJSON(t, id)+`,"metadata":{"title":"Updated draft"}}`, "application/json") + if response.Code != http.StatusOK { + t.Fatalf("revised apply failed: %s", response.Body.String()) + } + oldSelection := map[string]any{"books": []any{map[string]any{"id": id, "fingerprint": oldReview.Fingerprint, "draft_version": oldReview.DraftVersion, "fields": []string{"title"}}}} + response = request(t, server.Handler(), http.MethodPost, "/api/save", mustJSON(t, oldSelection), "application/json") + if response.Code != http.StatusBadRequest { + t.Fatalf("stale draft save status = %d, want 400: %s", response.Code, response.Body.String()) + } + discard := map[string]any{"books": []any{map[string]string{"id": id, "draft_version": oldReview.DraftVersion}}} + response = request(t, server.Handler(), http.MethodPost, "/api/drafts/discard", mustJSON(t, discard), "application/json") + if response.Code != http.StatusBadRequest { + t.Fatalf("stale draft discard status = %d, want 400: %s", response.Code, response.Body.String()) + } + state, _ := db.State(context.Background(), path) + if state.Staged["title"] != "Updated draft" { + t.Fatalf("stale selection changed newer draft: %#v", state.Staged) + } +} + +func TestSearchCacheAndBackgroundCompletion(t *testing.T) { + root := t.TempDir() + path := filepath.Join(root, "book.epub") + writeTestEPUB(t, path, "Cacheable title") + db := newMemoryRepository() + prefs := settings.Defaults() + prefs.BackgroundDelay = 1 + raw, _ := json.Marshal(prefs) + db.settings["preferences"] = string(raw) + catalogue := &fakeCatalog{result: catalog.Result{Candidates: []catalog.Candidate{{Title: "Cacheable title", Authors: []string{"Test Author"}, Source: "test", Confidence: .9}}}} + server := New(db, catalogue) + response := request(t, server.Handler(), http.MethodPost, "/api/scan", `{"root":`+mustJSON(t, root)+`}`, "application/json") + if response.Code != http.StatusOK { + t.Fatalf("scan failed: %s", response.Body.String()) + } + id := idFor(path) + body := `{"id":` + mustJSON(t, id) + `,"query":"Cacheable title"}` + response = request(t, server.Handler(), http.MethodPost, "/api/search", body, "application/json") + if response.Code != http.StatusOK { + t.Fatalf("search failed: %d %s", response.Code, response.Body.String()) + } + response = request(t, server.Handler(), http.MethodPost, "/api/search/cached", `{"id":`+mustJSON(t, id)+`}`, "application/json") + if response.Code != http.StatusOK { + t.Fatalf("cached search failed: %d %s", response.Code, response.Body.String()) + } + if catalogue.callCount() != 1 { + t.Fatalf("cached selection triggered catalog request; calls = %d", catalogue.callCount()) + } + + response = request(t, server.Handler(), http.MethodPost, "/api/background/start", `{"folder":`+mustJSON(t, root)+`}`, "application/json") + if response.Code != http.StatusOK { + t.Fatalf("background start failed: %d %s", response.Code, response.Body.String()) + } + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + response = request(t, server.Handler(), http.MethodGet, "/api/background", "", "") + var status matchProgress + if err := json.Unmarshal(response.Body.Bytes(), &status); err != nil { + t.Fatal(err) + } + if status.Status == "completed" { + if status.Completed != 1 || status.Failures != 0 { + t.Fatalf("unexpected background progress: %+v", status) + } + server.Close() + return + } + time.Sleep(10 * time.Millisecond) + } + server.Close() + t.Fatal("background matching did not complete in time") +} + +func readFileMetadata(path string) (map[string]any, error) { + return metadata.Read(path) +} diff --git a/internal/catalog/catalog_test.go b/internal/catalog/catalog_test.go new file mode 100644 index 0000000..0cb6b7c --- /dev/null +++ b/internal/catalog/catalog_test.go @@ -0,0 +1,434 @@ +package catalog + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "net/url" + "reflect" + "strings" + "testing" + "time" + + "shelfmark/internal/library" +) + +func TestNormalizeContributors(t *testing.T) { + tests := []struct { + name string + authors []string + translators []string + wantAuthors []string + wantTrans []string + }{ + { + name: "explicit translated role moves names and deduplicates normalized spellings", + authors: []string{" Jane Doe (translator) ", "Jane Doe", "Émile Zola [translator]", "A. Writer (traductrice)"}, + translators: []string{"jane doe", " Existing Translator "}, + wantAuthors: []string{"Jane Doe"}, + wantTrans: []string{"jane doe", "Existing Translator", "Émile Zola", "A. Writer"}, + }, + { + name: "prose in a name does not invent a role", + authors: []string{"Translated by Alex Author", "Mary Author"}, + wantAuthors: []string{"Translated by Alex Author", "Mary Author"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := normalizeContributors(Candidate{Authors: tt.authors, Translators: tt.translators}) + if !reflect.DeepEqual(got.Authors, tt.wantAuthors) { + t.Errorf("authors = %#v, want %#v", got.Authors, tt.wantAuthors) + } + if !reflect.DeepEqual(got.Translators, tt.wantTrans) { + t.Errorf("translators = %#v, want %#v", got.Translators, tt.wantTrans) + } + }) + } +} + +func TestCanonicalISBN(t *testing.T) { + tests := []struct { + name, value, want string + }{ + {name: "valid ISBN-10 converts to ISBN-13", value: "0-306-40615-2", want: "9780306406157"}, + {name: "valid ISBN-10 X check digit", value: "0-8044-2957-X", want: "9780804429573"}, + {name: "valid ISBN-13", value: "978-0-306-40615-7", want: "9780306406157"}, + {name: "valid 979 ISBN-13", value: "9791090636071", want: "9791090636071"}, + {name: "invalid ISBN-10 checksum", value: "0306406153", want: ""}, + {name: "invalid ISBN-13 checksum", value: "9780306406158", want: ""}, + {name: "X is only valid as the ISBN-10 check digit", value: "X306406152", want: ""}, + {name: "wrong length", value: "306406152", want: ""}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := canonicalISBN(tt.value); got != tt.want { + t.Errorf("canonicalISBN(%q) = %q, want %q", tt.value, got, tt.want) + } + }) + } +} + +func TestRankCandidateUsesIndependentTitleAuthorAndISBNEvidence(t *testing.T) { + request := SearchRequest{ + Identity: library.SearchIdentity{Titles: []string{"Dune"}, Authors: []string{"Frank Herbert"}, Languages: []string{"en"}}, + ISBNs: []string{"0-306-40615-2"}, + } + tests := []struct { + name string + candidate Candidate + wantISBN bool + wantStrong bool + minScore float64 + maxScore float64 + }{ + { + name: "exact title author language and ISBN", + candidate: Candidate{Title: "Dune", Authors: []string{"Frank Herbert"}, Language: "eng", ISBN: "9780306406157"}, + wantISBN: true, wantStrong: true, minScore: .99, maxScore: 1, + }, + { + name: "title and author without ISBN remains strong", + candidate: Candidate{Title: "Dune", Authors: []string{"Frank Herbert"}, Language: "en"}, + wantStrong: true, minScore: .99, maxScore: 1, + }, + { + name: "title match with unrelated author is discounted", + candidate: Candidate{Title: "Dune", Authors: []string{"Someone Else"}, Language: "en"}, + maxScore: .7, + }, + { + name: "invalid ISBN does not count as a hit", + candidate: Candidate{Title: "Dune", ISBN: "9780306406158"}, + maxScore: .7, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := rankCandidate(request, tt.candidate) + if got.ISBN != tt.wantISBN { + t.Errorf("ISBN match = %v, want %v", got.ISBN, tt.wantISBN) + } + if got.Strong != tt.wantStrong { + t.Errorf("Strong = %v, want %v (rank %#v)", got.Strong, tt.wantStrong, got) + } + if tt.minScore != 0 && got.Total < tt.minScore || tt.maxScore != 0 && got.Total > tt.maxScore { + t.Errorf("score %v outside [%v, %v]", got.Total, tt.minScore, tt.maxScore) + } + }) + } +} + +func TestRankCandidatePenalizesDifferentNumberedTitle(t *testing.T) { + request := SearchRequest{Identity: library.SearchIdentity{Titles: []string{"Dune 1"}}} + correct := rankCandidate(request, Candidate{Title: "Dune 1"}) + wrongVolume := rankCandidate(request, Candidate{Title: "Dune 2"}) + if !(correct.Total > wrongVolume.Total) { + t.Errorf("same-volume score %v should exceed wrong-volume score %v", correct.Total, wrongVolume.Total) + } +} + +func TestPlanSearch(t *testing.T) { + tests := []struct { + name string + request SearchRequest + want []searchQuery + maxLen int + }{ + { + name: "automatic searches start with a bounded set of ISBNs and specific title-author queries", + request: SearchRequest{ + Query: "noisy file name", ISBNs: []string{"isbn-one", "isbn-two", "isbn-three"}, + Identity: library.SearchIdentity{Titles: []string{"Title A", "Title B", "Title C", "Title D"}, Authors: []string{"Author Name"}}, + }, + want: []searchQuery{ + {Kind: "isbn", ISBN: "isbn-one"}, {Kind: "isbn", ISBN: "isbn-two"}, + {Kind: "title-author", Title: "Title A", Author: "Author Name"}, + {Kind: "title-author", Title: "Title B", Author: "Author Name"}, + {Kind: "title-author", Title: "Title C", Author: "Author Name"}, + }, + maxLen: 8, + }, + { + name: "custom query is preserved and prevents automatic variants", + request: SearchRequest{Query: " edited query ", CustomQuery: true, ISBNs: []string{"0-306-40615-2"}, Identity: library.SearchIdentity{Titles: []string{"Ignored"}}}, + want: []searchQuery{{Kind: "isbn", ISBN: "0-306-40615-2"}, {Kind: "text", Text: "edited query"}}, + }, + { + name: "no identity falls back to the supplied query", + request: SearchRequest{Query: "some search terms"}, + want: []searchQuery{{Kind: "text", Text: "some search terms"}}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := planSearch(tt.request) + if tt.maxLen == 0 && !reflect.DeepEqual(got, tt.want) { + t.Errorf("planSearch() = %#v, want %#v", got, tt.want) + return + } + if tt.maxLen > 0 { + if len(got) > tt.maxLen { + t.Fatalf("plan has %d queries, want at most %d: %#v", len(got), tt.maxLen, got) + } + if !reflect.DeepEqual(got[:len(tt.want)], tt.want) { + t.Errorf("leading queries = %#v, want %#v", got[:len(tt.want)], tt.want) + } + } + }) + } +} + +func TestReconcilePreservesSeriesAndSeparatesRemoteTitle(t *testing.T) { + current := map[string]any{ + "title": "Discworld 3", "series": "Discworld", "series_index": "3", + "authors": []string{"Terry Pratchett"}, + } + candidate := Candidate{ + Title: "Discworld #3 - Equal Rites", Authors: []string{"Terry Pratchett"}, + Publisher: "Corgi", Pubdate: "1987", ISBN: "9780552131063", + CatalogURL: "https://catalog.example/book/3", Source: "Example Catalog", + } + got := Reconcile(current, candidate, "Terry Pratchett - Discworld - 3 - Equal Rites.epub") + for key, want := range map[string]any{ + "title": "Equal Rites", "series": "Discworld", "series_index": "3", + "authors": []string{"Terry Pratchett"}, "publisher": "Corgi", "pubdate": "1987", + "catalog_url": "https://catalog.example/book/3", + } { + if !reflect.DeepEqual(got.Values[key], want) { + t.Errorf("Values[%q] = %#v, want %#v", key, got.Values[key], want) + } + } + if got.Review["title"] { + t.Errorf("title review should not be required when the local title is only a series label: %#v", got.Review) + } + if got.Evidence["title"] != "Example Catalog; book title separated from series/volume" { + t.Errorf("title evidence = %q", got.Evidence["title"]) + } +} + +func TestReconcilePreservesEditionFieldsAcrossLanguageConflict(t *testing.T) { + current := map[string]any{ + "title": "Le Petit Prince", "authors": []string{"Antoine de Saint-Exupéry"}, + "publisher": "Local Publisher", "pubdate": "1943", "isbn": "local-isbn", "languages": []string{"fr"}, + } + candidate := Candidate{ + Title: "The Little Prince", Authors: []string{"Antoine de Saint-Exupéry"}, Publisher: "Remote Publisher", + Pubdate: "1945", ISBN: "remote-isbn", Language: "en", Source: "Work Catalog", WorkRecord: true, + } + got := Reconcile(current, candidate, "Le Petit Prince.epub") + for _, field := range []string{"title", "publisher", "pubdate", "isbn", "languages"} { + if !reflect.DeepEqual(got.Values[field], current[field]) { + t.Errorf("Values[%q] = %#v, want existing value %#v", field, got.Values[field], current[field]) + } + } + if got.Evidence["publisher"] == "" || got.Evidence["languages"] == "" { + t.Errorf("expected preservation evidence for conflicting edition data, got %#v", got.Evidence) + } + if got.Review["title"] { + t.Errorf("local title in its existing language should be preserved without review, got %#v", got.Review) + } +} + +func TestReconcileTransfersOnlySupportedCorroboratedEvidence(t *testing.T) { + current := map[string]any{"title": "Unknown"} + candidate := Candidate{ + Title: "Unknown", Source: "Target Catalog", + Evidence: []MetadataEvidence{ + {Field: "title", Value: "The Specific Title", Source: "Donor Catalog", URL: "https://donor.example/record"}, + {Field: "publisher", Value: "Unrelated Publisher", Source: "Donor Catalog", URL: "https://donor.example/record"}, + }, + } + got := Reconcile(current, candidate, "untitled.epub") + if got.Values["title"] != "The Specific Title" { + t.Errorf("transferred title = %#v, want specific corroborated title", got.Values["title"]) + } + if got.Evidence["title"] != "Corroborated by Donor Catalog: https://donor.example/record" || !got.Review["title"] { + t.Errorf("title provenance/review = %q / %v", got.Evidence["title"], got.Review["title"]) + } + if _, ok := got.Values["publisher"]; ok { + t.Errorf("publisher must not be copied from cross-record evidence: %#v", got.Values) + } +} + +func TestCorroborateCandidatesRequiresSharedValidISBNAndCompatibleLanguage(t *testing.T) { + tests := []struct { + name string + isbn string + donorLang string + wantEvidence bool + }{ + {name: "shared valid ISBN", isbn: "9780306406157", donorLang: "en", wantEvidence: true}, + {name: "invalid ISBN is not a link", isbn: "9780306406158", donorLang: "en"}, + {name: "different language is not a link", isbn: "9780306406157", donorLang: "fr"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + request := SearchRequest{Identity: library.SearchIdentity{Titles: []string{"Wanted Book"}, Languages: []string{"en"}}} + candidates := []Candidate{ + {Title: "Unknown", ISBN: tt.isbn, Source: "Target", URL: "https://target.example/book"}, + {Title: "Wanted Book", ISBN: tt.isbn, Language: tt.donorLang, Source: "Donor", URL: "https://donor.example/book"}, + } + corroborateCandidates(request, candidates) + gotEvidence := len(candidates[0].Evidence) == 1 && candidates[0].Evidence[0].Field == "title" + if gotEvidence != tt.wantEvidence { + t.Errorf("title evidence = %#v, want present %v", candidates[0].Evidence, tt.wantEvidence) + } + }) + } +} + +func TestCandidateMarshalJSONKeepsOmittedListsAsArrays(t *testing.T) { + encoded, err := json.Marshal(Candidate{Title: "A Book"}) + if err != nil { + t.Fatal(err) + } + var value map[string]json.RawMessage + if err := json.Unmarshal(encoded, &value); err != nil { + t.Fatal(err) + } + for _, field := range []string{"authors", "translators", "categories", "isbns", "languages"} { + if got := string(value[field]); got != "[]" { + t.Errorf("JSON field %q = %q, want []", field, got) + } + } +} + +// rewriteTransport routes fixed production API URLs to a local httptest server. +type rewriteTransport struct { + target string + base http.RoundTripper +} + +func (r rewriteTransport) RoundTrip(request *http.Request) (*http.Response, error) { + clone := request.Clone(request.Context()) + target, err := url.Parse(r.target) + if err != nil { + return nil, err + } + clone.URL.Scheme, clone.URL.Host = target.Scheme, target.Host + clone.Host = target.Host + return r.base.RoundTrip(clone) +} + +func newTestService(server *httptest.Server) *Service { + service := New() + service.http = &http.Client{Transport: rewriteTransport{target: server.URL, base: http.DefaultTransport}} + return service +} + +func allowImmediateRequest(service *Service) { + service.mu.Lock() + service.lastRequest = time.Now().Add(-2 * time.Second) + service.mu.Unlock() +} + +func TestCatalogProviderParsingWithHTTPFixtures(t *testing.T) { + seen := make(map[string]bool) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + seen[r.URL.Path] = true + if r.Header.Get("User-Agent") != "Shelfmark/1.0 (local ebook metadata helper)" { + t.Errorf("User-Agent = %q", r.Header.Get("User-Agent")) + } + w.Header().Set("Content-Type", "application/json") + switch r.URL.Path { + case "/books/v1/volumes": + if got := r.URL.Query().Get("q"); got != `intitle:"The Hobbit" inauthor:"J. R. R. Tolkien"` { + t.Errorf("Google Books q = %q", got) + } + _, _ = w.Write([]byte(`{"items":[{"id":"hobbit-1","volumeInfo":{"title":"The Hobbit","subtitle":"There and Back Again","authors":["J. R. R. Tolkien"],"publisher":"Allen & Unwin","publishedDate":"1937","industryIdentifiers":[{"type":"ISBN_10","identifier":"054792822X"},{"type":"ISBN_13","identifier":"9780547928227"}],"imageLinks":{"thumbnail":"http://images.example/cover"},"language":"en","categories":["Fantasy"]}}]}`)) + case "/advancedsearch.php": + if got := r.URL.Query().Get("q"); got != `title:"The Hobbit" AND creator:"J. R. R. Tolkien"` { + t.Errorf("Internet Archive q = %q", got) + } + _, _ = w.Write([]byte(`{"response":{"docs":[{"identifier":"the-hobbit","title":"The Hobbit","isbn":["9780547928227","054792822X"],"creator":["J. R. R. Tolkien"],"year":"1937","language":["eng"],"description":"A hobbit goes on an adventure."},{"identifier":"","title":"Invalid row"}]}}`)) + case "/search.json": + if r.URL.Query().Get("title") != "The Hobbit" || r.URL.Query().Get("author") != "J. R. R. Tolkien" { + t.Errorf("Open Library query parameters = %v", r.URL.Query()) + } + _, _ = w.Write([]byte(`{"docs":[{"key":"/works/OL123W","title":"The Hobbit","author_name":["J. R. R. Tolkien"],"first_publish_year":1937,"language":["eng"],"isbn":["054792822X","9780547928227"],"publisher":["Allen & Unwin"],"cover_i":123,"subject":["Fantasy"]}]}`)) + default: + http.NotFound(w, r) + } + })) + defer server.Close() + service := newTestService(server) + query := searchQuery{Kind: "title-author", Title: "The Hobbit", Author: "J. R. R. Tolkien"} + + allowImmediateRequest(service) + google, err := service.google(context.Background(), query, "") + if err != nil { + t.Fatalf("Google Books search: %v", err) + } + if len(google) != 1 { + t.Fatalf("Google Books candidates = %d, want 1", len(google)) + } + if google[0].ISBN != "9780547928227" || !reflect.DeepEqual(google[0].ISBNs, []string{"054792822X", "9780547928227"}) { + t.Errorf("Google Books ISBN data = %q / %#v", google[0].ISBN, google[0].ISBNs) + } + if google[0].Cover != "https://images.example/cover" || google[0].FieldSources["title"] != "Google Books" { + t.Errorf("Google Books cover/provenance = %q / %#v", google[0].Cover, google[0].FieldSources) + } + + allowImmediateRequest(service) + archive, err := service.internetArchive(context.Background(), query) + if err != nil { + t.Fatalf("Internet Archive search: %v", err) + } + if len(archive) != 1 { + t.Fatalf("Internet Archive candidates = %d, want 1", len(archive)) + } + if archive[0].Title != "The Hobbit" || !reflect.DeepEqual(archive[0].Authors, []string{"J. R. R. Tolkien"}) || archive[0].ISBN != "9780547928227" || archive[0].Pubdate != "1937" { + t.Errorf("Internet Archive candidate = %#v", archive[0]) + } + + allowImmediateRequest(service) + openLibrary, err := service.openLibrary(context.Background(), query) + if err != nil { + t.Fatalf("Open Library search: %v", err) + } + if len(openLibrary) != 1 { + t.Fatalf("Open Library candidates = %d, want 1", len(openLibrary)) + } + if !openLibrary[0].WorkRecord || openLibrary[0].ISBN != "054792822X" || openLibrary[0].Cover != "https://covers.openlibrary.org/b/id/123-M.jpg" || openLibrary[0].Pubdate != "1937" { + t.Errorf("Open Library candidate = %#v", openLibrary[0]) + } + for _, endpoint := range []string{"/books/v1/volumes", "/advancedsearch.php", "/search.json"} { + if !seen[endpoint] { + t.Errorf("fixture endpoint %q was not called", endpoint) + } + } +} + +func TestOpenLibraryISBNEditionParsingWithHTTPFixture(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/isbn/0306406152.json" { + t.Errorf("path = %q", r.URL.Path) + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"key":"/books/OL123M","title":"Edition Title","subtitle":"An Edition","publishers":["Publisher"],"publish_date":"2001","isbn_10":["0306406152"],"isbn_13":["9780306406157"],"series":["A Series"],"covers":[456],"by_statement":"Translated from the French by Jane Doe; illustrated by Sam Artist","languages":[{"key":"/languages/eng"}],"contributors":[{"name":"J. Smith","role":"trl"},{"name":"Editor Name","role":"editor"}]}`)) + })) + defer server.Close() + service := newTestService(server) + allowImmediateRequest(service) + got, err := service.openLibraryISBN(context.Background(), "0306406152", []string{"Author Name"}) + if err != nil { + t.Fatalf("openLibraryISBN() error = %v", err) + } + if got == nil { + t.Fatal("openLibraryISBN() returned no edition") + } + if got.Title != "Edition Title" || got.Publisher != "Publisher" || got.Pubdate != "2001" || got.Language != "eng" { + t.Errorf("edition fields = %#v", got) + } + if !reflect.DeepEqual(got.Authors, []string{"Author Name"}) || !reflect.DeepEqual(got.Translators, []string{"J. Smith", "Jane Doe"}) { + t.Errorf("edition contributors = authors %#v translators %#v", got.Authors, got.Translators) + } + if got.Cover != "https://covers.openlibrary.org/b/id/456-M.jpg" || !strings.Contains(got.URL, "/books/OL123M") { + t.Errorf("edition cover/URL = %q / %q", got.Cover, got.URL) + } + if got.ISBN != "0306406152" || !reflect.DeepEqual(got.ISBNs, []string{"9780306406157", "0306406152"}) { + t.Errorf("edition ISBN data = %q / %#v", got.ISBN, got.ISBNs) + } +} diff --git a/internal/catalog/proposal.go b/internal/catalog/proposal.go index b0295f3..08f060c 100644 --- a/internal/catalog/proposal.go +++ b/internal/catalog/proposal.go @@ -155,6 +155,11 @@ func Reconcile(current map[string]any, candidate Candidate, filename string) Met series, volume, reason = parts.Series, parts.Volume, source+"; edition series label" break } + // Edition series fields may contain just the name, with no numbering. + // Retain that evidence instead of requiring a complete series/volume pair. + if series == "" && strings.TrimSpace(label) != "" { + series, reason = strings.TrimSpace(label), source+"; edition series label" + } } if series == "" && fileLabel.Series != "" { series, volume, reason = fileLabel.Series, fileLabel.Volume, "Filename series/volume label" @@ -162,13 +167,20 @@ func Reconcile(current map[string]any, candidate Candidate, filename string) Met if series == "" && localLabel.Series != "" { series, volume, reason = localLabel.Series, localLabel.Volume, "Existing numbered main title" } + volumeReason := reason + if volume == "" && fileLabel.Volume != "" && library.ClassifyBookLabel(fileLabel.Series, series).Series != "" { + volume, volumeReason = fileLabel.Volume, "Filename series/volume label" + } seriesConflict := identity.Series != "" && series != "" && library.ClassifyBookLabel(series, identity.Series).Series == "" if identity.Series == "" || seriesConflict { set("series", series, reason, seriesConflict || reason == "Filename series/volume label") } if volume != "" { conflict := identity.SeriesIndex != "" && !sameVolume(identity.SeriesIndex, volume) - set("series_index", volume, reason, conflict || seriesConflict || reason == "Filename series/volume label") + if fileLabel.Volume != "" && library.ClassifyBookLabel(fileLabel.Series, series).Series != "" && !sameVolume(fileLabel.Volume, volume) { + conflict = true + } + set("series_index", volume, volumeReason, conflict || seriesConflict || volumeReason == "Filename series/volume label") } set("original_title", candidate.OriginalTitle, source, library.Text(current["original_title"]) != "" && textSimilarity(library.Text(current["original_title"]), candidate.OriginalTitle) < .65) diff --git a/internal/desktop/desktop_test.go b/internal/desktop/desktop_test.go new file mode 100644 index 0000000..147b2f6 --- /dev/null +++ b/internal/desktop/desktop_test.go @@ -0,0 +1,89 @@ +package desktop + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "os" + "path/filepath" + "testing" +) + +func TestExtractCreatesPrivateFilesAndDirectories(t *testing.T) { + original := payload + t.Cleanup(func() { payload = original }) + payload = desktopArchive(t, []desktopEntry{{name: "runtime/bin/helper", data: []byte("helper"), mode: 0700}, {name: "runtime/data/file", data: []byte("data")}}) + dir := t.TempDir() + if err := extract(dir); err != nil { + t.Fatalf("extract() error = %v", err) + } + for name, want := range map[string]string{"runtime/bin/helper": "helper", "runtime/data/file": "data"} { + got, err := os.ReadFile(filepath.Join(dir, name)) + if err != nil { + t.Fatalf("read %s: %v", name, err) + } + if string(got) != want { + t.Errorf("%s = %q, want %q", name, got, want) + } + } + info, err := os.Stat(filepath.Join(dir, "runtime/bin/helper")) + if err != nil { + t.Fatal(err) + } + if info.Mode().Perm() != 0700 { + t.Errorf("helper permissions = %o, want 700", info.Mode().Perm()) + } +} + +func TestExtractRejectsTraversalLinksAndDuplicates(t *testing.T) { + for _, test := range []struct { + name string + entries []desktopEntry + }{ + {name: "path traversal", entries: []desktopEntry{{name: "../outside", data: []byte("escape")}}}, + {name: "symbolic link", entries: []desktopEntry{{name: "link", typeflag: tar.TypeSymlink}}}, + {name: "duplicate member", entries: []desktopEntry{{name: "same", data: []byte("first")}, {name: "same", data: []byte("second")}}}, + } { + t.Run(test.name, func(t *testing.T) { + original := payload + t.Cleanup(func() { payload = original }) + payload = desktopArchive(t, test.entries) + if err := extract(t.TempDir()); err == nil { + t.Fatal("extract() succeeded for invalid archive") + } + }) + } +} + +type desktopEntry struct { + name string + data []byte + mode int64 + typeflag byte +} + +func desktopArchive(t *testing.T, entries []desktopEntry) []byte { + t.Helper() + var buffer bytes.Buffer + compressed := gzip.NewWriter(&buffer) + archive := tar.NewWriter(compressed) + for _, entry := range entries { + typeflag := entry.typeflag + if typeflag == 0 { + typeflag = tar.TypeReg + } + if err := archive.WriteHeader(&tar.Header{Name: entry.name, Mode: entry.mode, Size: int64(len(entry.data)), Typeflag: typeflag}); err != nil { + t.Fatalf("write tar header: %v", err) + } + if _, err := archive.Write(entry.data); err != nil { + t.Fatalf("write tar entry: %v", err) + } + } + if err := archive.Close(); err != nil { + t.Fatalf("close tar writer: %v", err) + } + if err := compressed.Close(); err != nil { + t.Fatalf("close gzip writer: %v", err) + } + return buffer.Bytes() +} diff --git a/internal/library/book_label.go b/internal/library/book_label.go index 23cbc62..e2582b8 100644 --- a/internal/library/book_label.go +++ b/internal/library/book_label.go @@ -3,12 +3,14 @@ package library import ( "path/filepath" "regexp" + "strconv" "strings" ) var ( numberedLabel = regexp.MustCompile(`(?i)^(.+?)\s+(?:[-:]\s*)?(?:t(?:ome)?|vol(?:ume)?\.?|book|livre|#)\s*([0-9]+(?:[.,][0-9]+)?)\s*(?:[-:–—]\s*)?(.*)$`) bareNumberedLabel = regexp.MustCompile(`^(.+?)\s+(?:[-:#]+\s*)?([0-9]+(?:[.,][0-9]+)?)(?:\s*[-:–—]\s*(.*))?$`) + filenameVolume = regexp.MustCompile(`^[0-9]{1,3}(?:[.,][0-9]+)?$`) ) // BookLabel separates explicitly numbered series labels from actual titles. @@ -50,6 +52,9 @@ func seriesKey(value string) string { // FilenameBookLabel extracts a possible series/volume/title from the filename. // Known author segments are excluded before classifying the remaining label. +// A separate numeric segment between series and title is a reviewable volume +// clue even without existing series metadata. Bare numbers in titles remain +// untouched, and four-digit years are not treated as volume numbers. func FilenameBookLabel(filename string, identity SearchIdentity) BookLabel { filename = strings.TrimSuffix(filename, filepath.Ext(filename)) filename = strings.NewReplacer("_", " ", " – ", " - ", " — ", " - ").Replace(filename) @@ -64,5 +69,14 @@ func FilenameBookLabel(filename string, identity SearchIdentity) BookLabel { filename = strings.Join(remaining, " - ") } filename = fileNoise.ReplaceAllString(filename, " ") + parts := strings.Split(filename, " - ") + if len(parts) >= 3 && filenameVolume.MatchString(strings.TrimSpace(parts[1])) { + series := strings.TrimSpace(parts[0]) + title := strings.TrimSpace(strings.Join(parts[2:], " - ")) + if series != "" && title != "" { + volume, _ := strconv.ParseFloat(strings.ReplaceAll(strings.TrimSpace(parts[1]), ",", "."), 64) + return BookLabel{Title: title, Series: series, Volume: strconv.FormatFloat(volume, 'f', -1, 64)} + } + } return ClassifyBookLabel(filename, identity.Series) } diff --git a/internal/library/library_test.go b/internal/library/library_test.go new file mode 100644 index 0000000..792d409 --- /dev/null +++ b/internal/library/library_test.go @@ -0,0 +1,194 @@ +package library + +import ( + "reflect" + "strings" + "testing" +) + +func TestNormalizeSearchText(t *testing.T) { + tests := []struct { + name string + in string + want string + }{ + {name: "accents punctuation and case", in: " L'Été — Tome 2! ", want: "l ete tome 2"}, + {name: "preserves meaningful digits", in: "Catch-22, Part 1.5", want: "catch 22 part 1 5"}, + {name: "empty input", in: " \t\n", want: ""}, + {name: "caps normalized input length", in: strings.Repeat("a", 520), want: strings.Repeat("a", 512)}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := NormalizeSearchText(tt.in); got != tt.want { + t.Errorf("NormalizeSearchText(%q) = %q, want %q", tt.in, got, tt.want) + } + }) + } +} + +func TestClassifyBookLabel(t *testing.T) { + tests := []struct { + name, value, known string + want BookLabel + }{ + {name: "explicit volume marker", value: "Discworld #3 - Equal Rites", want: BookLabel{Title: "Equal Rites", Series: "Discworld", Volume: "3"}}, + {name: "known series and bare volume", value: "The Expanse 2", known: "Expanse", want: BookLabel{Series: "Expanse", Volume: "2"}}, + {name: "known series label", value: "The Expanse", known: "Expanse", want: BookLabel{Series: "Expanse"}}, + {name: "unmarked number remains title", value: "Catch 22", known: "Discworld", want: BookLabel{Title: "Catch 22"}}, + {name: "ordinary title remains unchanged", value: "The Left Hand of Darkness", want: BookLabel{Title: "The Left Hand of Darkness"}}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := ClassifyBookLabel(tt.value, tt.known); got != tt.want { + t.Errorf("ClassifyBookLabel(%q, %q) = %#v, want %#v", tt.value, tt.known, got, tt.want) + } + }) + } +} + +func TestFilenameBookLabel(t *testing.T) { + tests := []struct { + name string + filename string + identity SearchIdentity + want BookLabel + }{ + { + name: "author and format excluded while separate volume is retained", + filename: "Ursula Le Guin - Earthsea - 2,5 - Tehanu (epub).epub", + identity: SearchIdentity{Authors: []string{"Ursula Le Guin"}}, + want: BookLabel{Title: "Tehanu", Series: "Earthsea", Volume: "2.5"}, + }, + { + name: "known series marker label", + filename: "Earthsea Tome 2 - The Tombs of Atuan.pdf", + identity: SearchIdentity{Series: "Earthsea", Authors: []string{"Ursula K. Le Guin"}}, + want: BookLabel{Title: "The Tombs of Atuan", Series: "Earthsea", Volume: "2"}, + }, + { + name: "year in title is not an invented volume", + filename: "Catch 22 (1961).epub", + want: BookLabel{Title: "Catch 22 (1961)"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := FilenameBookLabel(tt.filename, tt.identity); got != tt.want { + t.Errorf("FilenameBookLabel(%q, %#v) = %#v, want %#v", tt.filename, tt.identity, got, tt.want) + } + }) + } +} + +func TestIdentifySearchUsesMetadataAndCleanFilename(t *testing.T) { + tests := []struct { + name string + values map[string]any + filename string + want SearchIdentity + }{ + { + name: "specific metadata precedes original and filename titles", + values: map[string]any{ + "title": "Dune", "subtitle": "A Novel", "original_title": "Dune Messiah", + "authors": []string{"Frank Herbert"}, "languages": []any{"en", "fr"}, + }, + filename: "Frank Herbert - Dune.epub", + want: SearchIdentity{Titles: []string{"Dune A Novel", "Dune Messiah", "Dune", "A Novel"}, Authors: []string{"Frank Herbert"}, Languages: []string{"en", "fr"}}, + }, + { + name: "placeholder title falls back to specific separated filename title", + values: map[string]any{"title": "Unknown", "authors": []string{"Ursula Le Guin"}, "series": "Earthsea"}, + filename: "Ursula Le Guin - Earthsea - Tehanu [epub].epub", + want: SearchIdentity{Titles: []string{"Tehanu"}, Authors: []string{"Ursula Le Guin"}, Series: "Earthsea"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := IdentifySearch(tt.values, tt.filename); !reflect.DeepEqual(got, tt.want) { + t.Errorf("IdentifySearch() = %#v, want %#v", got, tt.want) + } + }) + } +} + +func TestMetadataFieldParsingAndChanges(t *testing.T) { + tests := []struct { + name string + values map[string]string + current map[string]any + wantParsed map[string]any + wantChange map[string]any + wantErr bool + }{ + { + name: "list fields split on commas while scalar fields remain intact", + values: map[string]string{"authors": ` Ada Lovelace, "Martin Luther King, Jr." `, "title": " Notes, from the Engine ", "tags": "math, history"}, + current: map[string]any{"title": "Notes, from the Engine", "authors": []any{"Ada Lovelace", "Martin Luther King, Jr."}, "tags": []string{"math"}, "unrelated": true}, + wantParsed: map[string]any{"authors": []string{"Ada Lovelace", "Martin Luther King, Jr."}, "title": " Notes, from the Engine ", "tags": []string{"math", "history"}}, + wantChange: map[string]any{"tags": []string{"math", "history"}}, + }, + { + name: "malformed quoted list value is rejected", + values: map[string]string{"authors": `Ada Lovelace, "unfinished`}, + wantErr: true, + }, + { + name: "unknown field is rejected", + values: map[string]string{"private": "value"}, + wantErr: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + parsed, err := ParseInputs(tt.values) + if (err != nil) != tt.wantErr { + t.Fatalf("ParseInputs() error = %v, wantErr %v", err, tt.wantErr) + } + if tt.wantErr { + return + } + if !reflect.DeepEqual(parsed, tt.wantParsed) { + t.Errorf("ParseInputs() = %#v, want %#v", parsed, tt.wantParsed) + } + if got := Changes(tt.current, parsed); !reflect.DeepEqual(got, tt.wantChange) { + t.Errorf("Changes() = %#v, want %#v", got, tt.wantChange) + } + }) + } +} + +func TestInputTextRoundTripsCommaAndQuoteInContributorNames(t *testing.T) { + want := []string{"Ada Lovelace", `Martin Luther King, Jr.`, `Alex "Lex" Smith`} + encoded, err := InputText("authors", want) + if err != nil { + t.Fatalf("InputText() error = %v", err) + } + parsed, err := ParseInputs(map[string]string{"authors": encoded}) + if err != nil { + t.Fatalf("ParseInputs() error = %v", err) + } + if !reflect.DeepEqual(parsed["authors"], want) { + t.Fatalf("round-trip authors = %#v, want %#v (encoded %q)", parsed["authors"], want, encoded) + } +} + +func TestEqualTreatsUnsetAndEmptyRepresentationsAlike(t *testing.T) { + tests := []struct { + name string + a, b any + want bool + }{ + {name: "nil and empty list", a: nil, b: []string{}, want: true}, + {name: "blank scalar and nil", a: " \t", b: nil, want: true}, + {name: "JSON list representations", a: []string{"a", "b"}, b: []any{"a", "b"}, want: true}, + {name: "list order remains meaningful", a: []string{"a", "b"}, b: []string{"b", "a"}, want: false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := Equal(tt.a, tt.b); got != tt.want { + t.Errorf("Equal(%#v, %#v) = %v, want %v", tt.a, tt.b, got, tt.want) + } + }) + } +} diff --git a/internal/library/presentation.go b/internal/library/presentation.go index c5a7bb9..7ccfce4 100644 --- a/internal/library/presentation.go +++ b/internal/library/presentation.go @@ -1,6 +1,8 @@ package library import ( + "bytes" + "encoding/csv" "fmt" "strings" ) @@ -25,6 +27,29 @@ func Text(value any) string { } } +// InputText encodes list values as CSV so values containing commas or quotes +// remain one metadata item when submitted from the editor. +func InputText(name string, value any) (string, error) { + list, known := Field(name) + if !known || !list { + return Text(value), nil + } + values := Strings(value) + if len(values) == 0 { + return "", nil + } + var buffer bytes.Buffer + writer := csv.NewWriter(&buffer) + if err := writer.Write(values); err != nil { + return "", err + } + writer.Flush() + if err := writer.Error(); err != nil { + return "", err + } + return strings.TrimSuffix(buffer.String(), "\n"), nil +} + // DisplayTitle falls back to the file name when metadata has no usable title. func DisplayTitle(values map[string]any, filename string) string { if title := strings.TrimSpace(Text(values["title"])); title != "" { @@ -67,10 +92,19 @@ func ParseInputs(values map[string]string) (map[string]any, error) { result[name] = value continue } + reader := csv.NewReader(strings.NewReader(strings.TrimSpace(value))) + reader.FieldsPerRecord = -1 + reader.TrimLeadingSpace = true + records, err := reader.ReadAll() + if err != nil { + return nil, fmt.Errorf("%s must be valid comma-separated text: %w", name, err) + } items := make([]string, 0) - for _, item := range strings.Split(value, ",") { - if item = strings.TrimSpace(item); item != "" { - items = append(items, item) + for _, record := range records { + for _, item := range record { + if item = strings.TrimSpace(item); item != "" { + items = append(items, item) + } } } result[name] = items diff --git a/internal/metadata/comic.go b/internal/metadata/comic.go index 48c2983..70221ba 100644 --- a/internal/metadata/comic.go +++ b/internal/metadata/comic.go @@ -9,7 +9,9 @@ import ( "os" "path/filepath" "regexp" + "strconv" "strings" + "time" "github.com/beevik/etree" ) @@ -34,13 +36,7 @@ func readCBZFile(filename string) (map[string]any, error) { defer func() { _ = archive.Close() }() for _, file := range archive.File { if strings.EqualFold(file.Name, "ComicInfo.xml") { - r, e := file.Open() - if e != nil { - return nil, e - } - - data, e := io.ReadAll(r) - _ = r.Close() + data, e := readComicInfo(file) if e != nil { return nil, e } @@ -73,14 +69,12 @@ func writeCBZFile(ctx context.Context, filename string, fields map[string]any) e var info *etree.Element for _, file := range archive.File { if strings.EqualFold(file.Name, "ComicInfo.xml") { - r, e := file.Open() + data, e := readComicInfo(file) if e != nil { return e } - doc := etree.NewDocument() - _, e = doc.ReadFrom(r) - _ = r.Close() + e = doc.ReadFromBytes(data) if e != nil { return e } @@ -94,40 +88,41 @@ func writeCBZFile(ctx context.Context, filename string, fields map[string]any) e info = etree.NewElement("ComicInfo") } - // Only selected ComicInfo elements change; preserve other comic attributes. - mapping := map[string]string{ - "title": "Title", - "subtitle": "Subtitle", - "authors": "Writer", - "translators": "Translator", - "series": "Series", - "series_index": "Number", - "publisher": "Publisher", - "pubdate": "Year", - "comments": "Summary", - "languages": "LanguageISO", - "tags": "Genre", - "isbn": "ISBN", - "catalog_url": "Web", - "rating": "CommunityRating", - "rights": "Rights", - "original_title": "OriginalTitle", - "edition": "Format", - "collection": "SeriesGroup", - "identifiers": "Notes", - } - for key, tag := range mapping { + // Preserve the element sequence defined by the ComicInfo v2.0 schema. + for _, field := range []struct{ key, tag string }{ + {"title", "Title"}, + {"series", "Series"}, + {"series_index", "Number"}, + {"comments", "Summary"}, + {"authors", "Writer"}, + {"publisher", "Publisher"}, + {"tags", "Genre"}, + {"catalog_url", "Web"}, + {"languages", "LanguageISO"}, + {"edition", "Format"}, + {"collection", "SeriesGroup"}, + {"rating", "CommunityRating"}, + } { + key, tag := field.key, field.tag value, ok := fields[key] if !ok { continue } - node := info.FindElement("./" + tag) - if node == nil { - node = info.CreateElement(tag) + setComicInfoElement(info, tag, Text(value)) + } + if value, ok := fields["pubdate"]; ok { + dateFields, err := comicInfoDateFields(Text(value)) + if err != nil { + return err + } + for _, tag := range []string{"Year", "Month", "Day"} { + if field, present := dateFields[tag]; present { + setComicInfoElement(info, tag, field) + } else { + removeChildren(info, tag) + } } - - node.SetText(Text(value)) } var buf bytes.Buffer @@ -212,6 +207,25 @@ func writeCBZFile(ctx context.Context, filename string, fields map[string]any) e return replaceFile(tmp, filename) } +const maxComicInfoSize = 8 << 20 + +func readComicInfo(file *zip.File) ([]byte, error) { + r, err := file.Open() + if err != nil { + return nil, err + } + defer func() { _ = r.Close() }() + + data, err := io.ReadAll(io.LimitReader(r, maxComicInfoSize+1)) + if err != nil { + return nil, err + } + if len(data) > maxComicInfoSize { + return nil, fmt.Errorf("ComicInfo.xml exceeds 8 MiB") + } + return data, nil +} + func mapComicInfo(meta map[string]any, root *etree.Element) { get := func(tag string) string { e := root.FindElement("./" + tag) @@ -222,21 +236,17 @@ func mapComicInfo(meta map[string]any, root *etree.Element) { } for _, entry := range [][2]string{ {"Title", "title"}, - {"Subtitle", "subtitle"}, {"Series", "series"}, {"Number", "series_index"}, - {"Publisher", "publisher"}, - {"Year", "pubdate"}, {"Summary", "comments"}, - {"LanguageISO", "language"}, - {"ISBN", "isbn"}, + {"Writer", "authors"}, + {"Publisher", "publisher"}, + {"Genre", "tags"}, {"Web", "catalog_url"}, - {"CommunityRating", "rating"}, - {"Rights", "rights"}, - {"OriginalTitle", "original_title"}, + {"LanguageISO", "language"}, {"Format", "edition"}, {"SeriesGroup", "collection"}, - {"Notes", "identifiers"}, + {"CommunityRating", "rating"}, } { if value := get(entry[0]); value != "" { if entry[1] == "language" { @@ -246,6 +256,9 @@ func mapComicInfo(meta map[string]any, root *etree.Element) { } } } + if date := comicInfoDate(root); date != "" { + meta["pubdate"] = date + } for _, k := range [][2]string{{"Writer", "authors"}, {"Translator", "translators"}, {"Genre", "tags"}} { if v := get(k[0]); v != "" { parts := strings.Split(v, ",") @@ -257,6 +270,93 @@ func mapComicInfo(meta map[string]any, root *etree.Element) { } } +var comicInfoElementOrder = map[string]int{ + "Title": 0, "Series": 1, "Number": 2, "Count": 3, "Volume": 4, + "AlternateSeries": 5, "AlternateNumber": 6, "AlternateCount": 7, + "Summary": 8, "Notes": 9, "Year": 10, "Month": 11, "Day": 12, + "Writer": 13, "Penciller": 14, "Inker": 15, "Colorist": 16, + "Letterer": 17, "CoverArtist": 18, "Editor": 19, "Publisher": 20, + "Imprint": 21, "Genre": 22, "Web": 23, "PageCount": 24, + "LanguageISO": 25, "Format": 26, "BlackAndWhite": 27, "Manga": 28, + "Characters": 29, "Teams": 30, "Locations": 31, "ScanInformation": 32, + "StoryArc": 33, "SeriesGroup": 34, "AgeRating": 35, "Pages": 36, + "CommunityRating": 37, "MainCharacterOrTeam": 38, "Review": 39, +} + +func setComicInfoElement(parent *etree.Element, tag, value string) { + if node := parent.FindElement("./" + tag); node != nil { + node.SetText(value) + return + } + node := etree.NewElement(tag) + node.SetText(value) + order, known := comicInfoElementOrder[tag] + if known { + for _, child := range parent.ChildElements() { + childOrder, childKnown := comicInfoElementOrder[child.Tag] + if childKnown && childOrder > order { + parent.InsertChildAt(child.Index(), node) + return + } + } + } + parent.AddChild(node) +} + +func comicInfoDateFields(value string) (map[string]string, error) { + if value == "" { + return map[string]string{}, nil + } + for _, layout := range []string{"2006", "2006-01", "2006-01-02"} { + parsed, err := time.Parse(layout, value) + if err != nil || parsed.Year() < 1 { + continue + } + fields := map[string]string{"Year": strconv.Itoa(parsed.Year())} + if len(value) >= len("2006-01") { + fields["Month"] = strconv.Itoa(int(parsed.Month())) + } + if len(value) == len("2006-01-02") { + fields["Day"] = strconv.Itoa(parsed.Day()) + } + return fields, nil + } + return nil, fmt.Errorf("CBZ publication date must be YYYY, YYYY-MM, or YYYY-MM-DD") +} + +func comicInfoDate(root *etree.Element) string { + fieldText := func(name string) string { + if element := root.FindElement("./" + name); element != nil { + return strings.TrimSpace(element.Text()) + } + return "" + } + year, err := strconv.Atoi(fieldText("Year")) + if err != nil || year < 1 || year > 9999 { + return "" + } + date := strconv.Itoa(year) + monthText := fieldText("Month") + if monthText == "" { + return date + } + month, err := strconv.Atoi(monthText) + if err != nil || month < 1 || month > 12 { + return date + } + date += fmt.Sprintf("-%02d", month) + dayText := fieldText("Day") + if dayText == "" { + return date + } + day, err := strconv.Atoi(dayText) + parsed, parseErr := time.Parse("2006-01-02", fmt.Sprintf("%04d-%02d-%02d", year, month, day)) + if err != nil || parseErr != nil || parsed.Day() != day { + return date + } + return fmt.Sprintf("%04d-%02d-%02d", year, month, day) +} + func setElement(parent *etree.Element, tag, value string) { node := parent.FindElement("./" + tag) if node == nil { diff --git a/internal/metadata/comic_test.go b/internal/metadata/comic_test.go new file mode 100644 index 0000000..90f7248 --- /dev/null +++ b/internal/metadata/comic_test.go @@ -0,0 +1,268 @@ +package metadata + +import ( + "archive/zip" + "bytes" + "context" + "encoding/xml" + "io" + "os" + "path/filepath" + "reflect" + "strings" + "testing" +) + +func TestReadComicInfoSizeLimit(t *testing.T) { + for _, test := range []struct { + name string + size int + wantError bool + }{ + {name: "at limit", size: maxComicInfoSize}, + {name: "over limit", size: maxComicInfoSize + 1, wantError: true}, + } { + t.Run(test.name, func(t *testing.T) { + file := comicInfoZipEntry(t, bytes.Repeat([]byte("x"), test.size)) + got, err := readComicInfo(file) + if test.wantError { + if err == nil || !strings.Contains(err.Error(), "exceeds 8 MiB") { + t.Fatalf("readComicInfo() error = %v, want size-limit error", err) + } + return + } + if err != nil { + t.Fatalf("readComicInfo() error = %v", err) + } + if len(got) != test.size { + t.Fatalf("readComicInfo() returned %d bytes, want %d", len(got), test.size) + } + }) + } +} + +func TestCBZReadWritePreservesOtherArchiveMembers(t *testing.T) { + filename := filepath.Join(t.TempDir(), "comic.cbz") + writeZipFixture(t, filename, map[string][]byte{ + "ComicInfo.xml": []byte(`Old titleEarthsea2A story.Original notes202074Old authorPublisherDrama, Mysteryhttps://example.org/comicen-USDigitalFantasy4.5`), + "pages/001.jpg": []byte("image bytes"), + }) + + metadata, err := readCBZFile(filename) + if err != nil { + t.Fatalf("readCBZFile() error = %v", err) + } + if got := Text(metadata["title"]); got != "Old title" { + t.Fatalf("title = %q, want Old title", got) + } + if got := Strings(metadata["authors"]); len(got) != 1 || got[0] != "Old author" { + t.Fatalf("authors = %#v, want [Old author]", got) + } + if got := metadata["pubdate"]; got != "2020-07-04" { + t.Fatalf("publication date = %#v, want 2020-07-04 from ComicInfo Year/Month/Day", got) + } + if got := Strings(metadata["languages"]); len(got) != 1 || got[0] != "en-US" { + t.Fatalf("languages = %#v, want [en-US]", got) + } + + if err := Write(context.Background(), filename, map[string]any{"title": "New title", "pubdate": "2024-02-29"}); err != nil { + t.Fatalf("Write() error = %v", err) + } + metadata, err = readCBZFile(filename) + if err != nil { + t.Fatalf("readCBZFile() after write error = %v", err) + } + if got := Text(metadata["title"]); got != "New title" { + t.Fatalf("title after write = %q, want New title", got) + } + if got := metadata["pubdate"]; got != "2024-02-29" { + t.Fatalf("publication date after write = %#v, want 2024-02-29", got) + } + assertComicInfoElementsInSchemaOrder(t, filename, []string{"Title", "Series", "Number", "Summary", "Notes", "Year", "Month", "Day", "Writer", "Publisher", "Genre", "Web", "LanguageISO", "Format", "SeriesGroup", "CommunityRating"}) + + archive, err := zip.OpenReader(filename) + if err != nil { + t.Fatalf("open rewritten archive: %v", err) + } + defer func() { _ = archive.Close() }() + for _, member := range archive.File { + if member.Name != "pages/001.jpg" { + continue + } + reader, err := member.Open() + if err != nil { + t.Fatalf("open preserved member: %v", err) + } + var actual bytes.Buffer + _, copyErr := actual.ReadFrom(reader) + _ = reader.Close() + if copyErr != nil { + t.Fatalf("read preserved member: %v", copyErr) + } + if !bytes.Equal(actual.Bytes(), []byte("image bytes")) { + t.Fatalf("preserved page bytes = %q", actual.Bytes()) + } + return + } + t.Fatal("rewritten archive is missing pages/001.jpg") +} + +func TestCBZPublicationDatePrecisionAndClearing(t *testing.T) { + filename := filepath.Join(t.TempDir(), "dated.cbz") + writeZipFixture(t, filename, map[string][]byte{"ComicInfo.xml": []byte(`Book19991231`)}) + for _, test := range []struct { + value string + want []string + }{ + {value: "2025", want: []string{"Year"}}, + {value: "2025-04", want: []string{"Year", "Month"}}, + {value: "", want: nil}, + } { + if err := Write(context.Background(), filename, map[string]any{"pubdate": test.value}); err != nil { + t.Fatalf("Write(pubdate=%q): %v", test.value, err) + } + got, err := comicInfoElementValues(t, filename) + if err != nil { + t.Fatal(err) + } + var dateElements []string + for _, field := range []string{"Year", "Month", "Day"} { + if _, exists := got[field]; exists { + dateElements = append(dateElements, field) + } + } + if !reflect.DeepEqual(dateElements, test.want) { + t.Errorf("date elements after pubdate %q = %v, want %v", test.value, dateElements, test.want) + } + } +} + +func assertComicInfoElementsInSchemaOrder(t *testing.T, filename string, want []string) { + t.Helper() + got, err := comicInfoElementNames(t, filename) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("ComicInfo element order = %v, want schema order %v", got, want) + } +} + +func comicInfoElementValues(t *testing.T, filename string) (map[string]string, error) { + t.Helper() + archive, err := zip.OpenReader(filename) + if err != nil { + return nil, err + } + defer func() { _ = archive.Close() }() + for _, member := range archive.File { + if !strings.EqualFold(member.Name, "ComicInfo.xml") { + continue + } + reader, err := member.Open() + if err != nil { + return nil, err + } + defer func() { _ = reader.Close() }() + values := map[string]string{} + decoder := xml.NewDecoder(reader) + var current string + for { + token, err := decoder.Token() + if err == io.EOF { + return values, nil + } + if err != nil { + return nil, err + } + switch value := token.(type) { + case xml.StartElement: + current = value.Name.Local + case xml.CharData: + if current != "" { + values[current] = string(bytes.TrimSpace(value)) + } + case xml.EndElement: + current = "" + } + } + } + return nil, os.ErrNotExist +} + +func comicInfoElementNames(t *testing.T, filename string) ([]string, error) { + t.Helper() + archive, err := zip.OpenReader(filename) + if err != nil { + return nil, err + } + defer func() { _ = archive.Close() }() + for _, member := range archive.File { + if !strings.EqualFold(member.Name, "ComicInfo.xml") { + continue + } + reader, err := member.Open() + if err != nil { + return nil, err + } + defer func() { _ = reader.Close() }() + decoder := xml.NewDecoder(reader) + var result []string + depth := 0 + for { + token, err := decoder.Token() + if err == io.EOF { + return result, nil + } + if err != nil { + return nil, err + } + switch value := token.(type) { + case xml.StartElement: + depth++ + if depth == 2 { + result = append(result, value.Name.Local) + } + case xml.EndElement: + depth-- + } + } + } + return nil, os.ErrNotExist +} + +func comicInfoZipEntry(t *testing.T, data []byte) *zip.File { + t.Helper() + filename := filepath.Join(t.TempDir(), "metadata.zip") + writeZipFixture(t, filename, map[string][]byte{"ComicInfo.xml": data}) + archive, err := zip.OpenReader(filename) + if err != nil { + t.Fatalf("open test ZIP: %v", err) + } + t.Cleanup(func() { _ = archive.Close() }) + return archive.File[0] +} + +func writeZipFixture(t *testing.T, filename string, members map[string][]byte) { + t.Helper() + file, err := os.Create(filename) + if err != nil { + t.Fatalf("create ZIP fixture: %v", err) + } + writer := zip.NewWriter(file) + for name, data := range members { + member, err := writer.Create(name) + if err != nil { + t.Fatalf("create ZIP member %q: %v", name, err) + } + if _, err := member.Write(data); err != nil { + t.Fatalf("write ZIP member %q: %v", name, err) + } + } + if err := writer.Close(); err != nil { + t.Fatalf("close ZIP fixture: %v", err) + } + if err := file.Close(); err != nil { + t.Fatalf("close ZIP fixture file: %v", err) + } +} diff --git a/internal/metadata/epub_contributors.go b/internal/metadata/epub_contributors.go index 5c8ee79..518f498 100644 --- a/internal/metadata/epub_contributors.go +++ b/internal/metadata/epub_contributors.go @@ -23,6 +23,14 @@ func epubContributorRoles(meta *etree.Element) map[*etree.Element]string { continue } role := node.SelectAttrValue("role", "") + if role == "" { + for _, attr := range node.Attr { + if attr.Key == "role" && attr.NamespaceURI() == "http://www.idpf.org/2007/opf" { + role = attr.Value + break + } + } + } if role == "" { role = refined[node.SelectAttrValue("id", "")] } diff --git a/internal/metadata/fields.go b/internal/metadata/fields.go index 287da6e..ee8f48d 100644 --- a/internal/metadata/fields.go +++ b/internal/metadata/fields.go @@ -3,11 +3,17 @@ package metadata import ( "fmt" "path/filepath" + "regexp" + "strconv" "strings" "shelfmark/internal/library" + + "golang.org/x/text/language" ) +var comicRatingPattern = regexp.MustCompile(`^\+?[0-9]+(?:\.[0-9]{1,2})?$`) + // ValidateFields rejects values that cannot be represented by a format writer. func ValidateFields(path string, values map[string]any) error { for key, value := range values { @@ -19,6 +25,39 @@ func ValidateFields(path string, values map[string]any) error { if key == "cover_url" && !strings.EqualFold(filepath.Ext(path), ".epub") { return fmt.Errorf("cover writing requires an EPUB") } + if key == "pubdate" && strings.EqualFold(filepath.Ext(path), ".pdf") { + return fmt.Errorf("publication date writing is not supported for PDF") + } + if strings.EqualFold(filepath.Ext(path), ".cbz") { + switch key { + case "title", "series", "series_index", "publisher", "pubdate", "authors", "comments", "languages", "tags", "catalog_url", "rating", "edition", "collection": + default: + return fmt.Errorf("%s metadata is not supported for CBZ", key) + } + if key == "pubdate" { + if _, err := comicInfoDateFields(Text(value)); err != nil { + return err + } + } + if key == "rating" { + rating := Text(value) + number, err := strconv.ParseFloat(rating, 64) + if err != nil || !comicRatingPattern.MatchString(rating) || number < 0 || number > 5 { + return fmt.Errorf("ComicInfo rating must be a decimal between 0 and 5 with at most two fractional digits") + } + } + if key == "languages" { + values := Strings(value) + if len(values) > 1 { + return fmt.Errorf("ComicInfo supports one language tag") + } + if len(values) == 1 { + if _, err := language.Parse(values[0]); err != nil { + return fmt.Errorf("invalid ComicInfo language tag %q", values[0]) + } + } + } + } if list { switch v := value.(type) { diff --git a/internal/metadata/fields_test.go b/internal/metadata/fields_test.go new file mode 100644 index 0000000..b4d9577 --- /dev/null +++ b/internal/metadata/fields_test.go @@ -0,0 +1,57 @@ +package metadata + +import ( + "path/filepath" + "strings" + "testing" +) + +func TestValidateFieldsEnforcesFormatCapabilities(t *testing.T) { + for _, test := range []struct { + name string + path string + fields map[string]any + wantErr string + }{ + {name: "PDF publication date unsupported", path: "book.pdf", fields: map[string]any{"pubdate": "2020"}, wantErr: "publication date writing is not supported for PDF"}, + {name: "EPUB publication date supported", path: "book.epub", fields: map[string]any{"pubdate": "2020"}}, + {name: "CBZ year publication date supported", path: "book.cbz", fields: map[string]any{"pubdate": "2020"}}, + {name: "CBZ full ISO publication date supported", path: "book.cbz", fields: map[string]any{"pubdate": "2020-02-29"}}, + {name: "invalid CBZ date rejected", path: "book.cbz", fields: map[string]any{"pubdate": "2021-02-29"}, wantErr: "CBZ publication date"}, + {name: "CBZ unsupported ISBN rejected", path: "book.cbz", fields: map[string]any{"isbn": "9780306406157"}, wantErr: "isbn metadata is not supported for CBZ"}, + {name: "CBZ unsupported subtitle rejected", path: "book.cbz", fields: map[string]any{"subtitle": "A novel"}, wantErr: "subtitle metadata is not supported for CBZ"}, + {name: "CBZ language uses BCP 47 tag", path: "book.cbz", fields: map[string]any{"languages": []string{"zh-Hant"}}}, + {name: "CBZ language only allows one tag", path: "book.cbz", fields: map[string]any{"languages": []string{"en", "fr"}}, wantErr: "supports one language tag"}, + {name: "CBZ rating follows ComicInfo bounds", path: "book.cbz", fields: map[string]any{"rating": "4.5"}}, + {name: "CBZ rating over five rejected", path: "book.cbz", fields: map[string]any{"rating": "5.01"}, wantErr: "between 0 and 5"}, + {name: "cover only supported for EPUB", path: "book.cbz", fields: map[string]any{"cover_url": "https://example.org/cover.jpg"}, wantErr: "cover writing requires an EPUB"}, + {name: "unknown field rejected", path: "book.fb2", fields: map[string]any{"unknown": "x"}, wantErr: "unknown metadata field"}, + {name: "invalid list item rejected", path: "book.epub", fields: map[string]any{"authors": []any{"valid", 12}}, wantErr: "authors must contain strings"}, + } { + t.Run(test.name, func(t *testing.T) { + err := ValidateFields(filepath.Clean(test.path), test.fields) + if test.wantErr == "" { + if err != nil { + t.Fatalf("ValidateFields() error = %v", err) + } + return + } + if err == nil || !strings.Contains(err.Error(), test.wantErr) { + t.Fatalf("ValidateFields() error = %v, want substring %q", err, test.wantErr) + } + }) + } +} + +func TestISBNsExtractsValidDistinctValues(t *testing.T) { + got := ISBNs("ISBN-10: 0-306-40615-2; ISBN-13: 978-0-306-40615-7; ISBN 0-306-40615-2; invalid 9780306406158") + want := []string{"0306406152", "9780306406157"} + if len(got) != len(want) { + t.Fatalf("ISBNs() = %#v, want %#v", got, want) + } + for index := range want { + if got[index] != want[index] { + t.Fatalf("ISBNs() = %#v, want %#v", got, want) + } + } +} diff --git a/internal/metadata/formats_test.go b/internal/metadata/formats_test.go new file mode 100644 index 0000000..a62273f --- /dev/null +++ b/internal/metadata/formats_test.go @@ -0,0 +1,233 @@ +package metadata + +import ( + "archive/zip" + "bytes" + "context" + "encoding/xml" + "io" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "github.com/pdfcpu/pdfcpu/pkg/api" + "github.com/pdfcpu/pdfcpu/pkg/pdfcpu/model" +) + +func TestEPUBMetadataRoundTripPreservesUnselectedMembers(t *testing.T) { + filename := filepath.Join(t.TempDir(), "book.epub") + chapter := []byte(`Chapter

Chapter text.

`) + writeEPUBFixture(t, filename, "Original title", chapter) + + if err := Write(context.Background(), filename, map[string]any{"title": "Updated title", "pubdate": "2024"}); err != nil { + t.Fatalf("Write() error = %v", err) + } + got, err := Read(filename) + if err != nil { + t.Fatalf("Read() after write error = %v", err) + } + if got["title"] != "Updated title" || got["pubdate"] != "2024" { + t.Fatalf("updated metadata = title %q, publication date %q", got["title"], got["pubdate"]) + } + if !reflect.DeepEqual(got["authors"], []string{"Original author"}) || !reflect.DeepEqual(got["tags"], []string{"Historical fiction"}) { + t.Fatalf("unselected metadata changed: authors=%#v tags=%#v", got["authors"], got["tags"]) + } + if !reflect.DeepEqual(got["translators"], []string{"Jane Translator"}) { + t.Fatalf("EPUB 2 opf:role translator = %#v, want [Jane Translator]", got["translators"]) + } + assertEPUBPackageMetadata(t, filename) + + archive, err := zip.OpenReader(filename) + if err != nil { + t.Fatalf("open rewritten EPUB: %v", err) + } + defer func() { _ = archive.Close() }() + if len(archive.File) == 0 || archive.File[0].Name != "mimetype" || archive.File[0].Method != zip.Store { + t.Fatal("EPUB output must begin with an uncompressed mimetype entry") + } + for _, member := range archive.File { + if member.Name != "OEBPS/chapter.xhtml" { + continue + } + reader, err := member.Open() + if err != nil { + t.Fatalf("open chapter: %v", err) + } + data, readErr := io.ReadAll(reader) + _ = reader.Close() + if readErr != nil { + t.Fatalf("read chapter: %v", readErr) + } + if !bytes.Equal(data, chapter) { + t.Fatalf("chapter bytes = %q, want %q", data, chapter) + } + return + } + t.Fatal("rewritten EPUB is missing its chapter") +} + +func writeEPUBFixture(t *testing.T, filename, title string, chapter []byte) { + t.Helper() + file, err := os.Create(filename) + if err != nil { + t.Fatal(err) + } + writer := zip.NewWriter(file) + mimetype, err := writer.CreateHeader(&zip.FileHeader{Name: "mimetype", Method: zip.Store}) + if err != nil { + t.Fatal(err) + } + if _, err := io.WriteString(mimetype, "application/epub+zip"); err != nil { + t.Fatal(err) + } + entries := []struct{ name, content string }{ + {"META-INF/container.xml", ``}, + {"OEBPS/content.opf", `` + title + `urn:isbn:9780306406157enOriginal authorJane TranslatorHistorical fiction1999`}, + } + for _, entry := range entries { + member, err := writer.Create(entry.name) + if err != nil { + t.Fatal(err) + } + if _, err := io.WriteString(member, entry.content); err != nil { + t.Fatal(err) + } + } + member, err := writer.Create("OEBPS/chapter.xhtml") + if err != nil { + t.Fatal(err) + } + if _, err := member.Write(chapter); err != nil { + t.Fatal(err) + } + if err := writer.Close(); err != nil { + t.Fatal(err) + } + if err := file.Close(); err != nil { + t.Fatal(err) + } +} + +func assertEPUBPackageMetadata(t *testing.T, filename string) { + t.Helper() + archive, err := zip.OpenReader(filename) + if err != nil { + t.Fatal(err) + } + defer func() { _ = archive.Close() }() + for _, member := range archive.File { + if member.Name != "OEBPS/content.opf" { + continue + } + reader, err := member.Open() + if err != nil { + t.Fatal(err) + } + var packageDoc struct { + XMLName xml.Name `xml:"http://www.idpf.org/2007/opf package"` + UniqueIdentifier string `xml:"unique-identifier,attr"` + Metadata struct { + Title string `xml:"http://purl.org/dc/elements/1.1/ title"` + Identifier []struct { + ID string `xml:"id,attr"` + Value string `xml:",chardata"` + } `xml:"http://purl.org/dc/elements/1.1/ identifier"` + Creator []string `xml:"http://purl.org/dc/elements/1.1/ creator"` + Date []struct { + Event string `xml:"http://www.idpf.org/2007/opf event,attr"` + Value string `xml:",chardata"` + } `xml:"http://purl.org/dc/elements/1.1/ date"` + } `xml:"http://www.idpf.org/2007/opf metadata"` + } + decodeErr := xml.NewDecoder(reader).Decode(&packageDoc) + _ = reader.Close() + if decodeErr != nil { + t.Fatalf("decode OPF package: %v", decodeErr) + } + if packageDoc.XMLName.Space != "http://www.idpf.org/2007/opf" || packageDoc.UniqueIdentifier != "BookId" { + t.Fatalf("OPF package identity is invalid: %+v", packageDoc) + } + if packageDoc.Metadata.Title != "Updated title" || len(packageDoc.Metadata.Creator) != 1 || packageDoc.Metadata.Creator[0] != "Original author" { + t.Fatalf("OPF Dublin Core metadata = %+v", packageDoc.Metadata) + } + if len(packageDoc.Metadata.Identifier) != 1 || packageDoc.Metadata.Identifier[0].ID != packageDoc.UniqueIdentifier || packageDoc.Metadata.Identifier[0].Value != "urn:isbn:9780306406157" { + t.Fatalf("OPF unique identifier is missing or changed: %+v", packageDoc.Metadata.Identifier) + } + if len(packageDoc.Metadata.Date) != 1 || packageDoc.Metadata.Date[0].Event != "publication" || packageDoc.Metadata.Date[0].Value != "2024" { + t.Fatalf("OPF publication date event/value = %+v", packageDoc.Metadata.Date) + } + return + } + t.Fatal("rewritten EPUB is missing its package document") +} + +func TestFB2MetadataRoundTripPreservesBookBody(t *testing.T) { + filename := filepath.Join(t.TempDir(), "book.fb2") + body := `Keep this body intact.` + original := `Original titleOldAuthorenPress2001

` + body + `

` + if err := os.WriteFile(filename, []byte(original), 0600); err != nil { + t.Fatal(err) + } + + if err := Write(context.Background(), filename, map[string]any{"title": "Updated title", "authors": []string{"New Author"}, "pubdate": "2025"}); err != nil { + t.Fatalf("Write() error = %v", err) + } + got, err := Read(filename) + if err != nil { + t.Fatalf("Read() after write error = %v", err) + } + if got["title"] != "Updated title" || got["pubdate"] != "2025" || !reflect.DeepEqual(got["authors"], []string{"New Author"}) { + t.Fatalf("updated metadata = %#v", got) + } + data, err := os.ReadFile(filename) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(data), body) { + t.Fatalf("FB2 body was not preserved: %s", data) + } +} + +func TestPDFPublicationDateWriteRejectedWithoutModifyingFile(t *testing.T) { + filename := filepath.Join(t.TempDir(), "book.pdf") + var pdf bytes.Buffer + configuration := model.NewDefaultConfiguration() + configuration.CreationDate = "2000-01-02 03:04" + if err := api.Create(context.Background(), nil, strings.NewReader(`{"pages":{"1":{"content":{}}}}`), &pdf, configuration); err != nil { + t.Fatalf("create PDF fixture: %v", err) + } + if err := os.WriteFile(filename, pdf.Bytes(), 0600); err != nil { + t.Fatal(err) + } + before := readPDFProperties(t, filename) + metadata, err := Read(filename) + if err != nil { + t.Fatalf("Read() PDF: %v", err) + } + if _, exists := metadata["pubdate"]; exists { + t.Fatalf("PDF CreationDate was incorrectly exposed as publication date: %#v", metadata["pubdate"]) + } + if err := Write(context.Background(), filename, map[string]any{"pubdate": "2020"}); err == nil { + t.Fatal("Write() accepted a publication date for PDF") + } + after := readPDFProperties(t, filename) + if after["CreationDate"] != before["CreationDate"] { + t.Fatalf("rejected publication-date edit changed CreationDate: before=%q after=%q", before["CreationDate"], after["CreationDate"]) + } +} + +func readPDFProperties(t *testing.T, filename string) map[string]string { + t.Helper() + file, err := os.Open(filename) + if err != nil { + t.Fatal(err) + } + defer func() { _ = file.Close() }() + properties, err := api.Properties(context.Background(), file, model.NewDefaultConfiguration()) + if err != nil { + t.Fatalf("read PDF properties: %v", err) + } + return properties +} diff --git a/internal/metadata/pdf.go b/internal/metadata/pdf.go index e980c7b..85be245 100644 --- a/internal/metadata/pdf.go +++ b/internal/metadata/pdf.go @@ -29,7 +29,6 @@ func readPDFFile(filename string) (map[string]any, error) { "subtitle": values["Subtitle"], "authors": splitAuthors(authors), "publisher": values["Publisher"], - "pubdate": values["CreationDate"], "comments": values["Subject"], "isbn": values["ISBN"], "series": values["Series"], @@ -54,7 +53,6 @@ func writePDFFile(ctx context.Context, filename string, fields map[string]any) e "subtitle": "Subtitle", "authors": "Author", "publisher": "Publisher", - "pubdate": "CreationDate", "comments": "Subject", "isbn": "ISBN", "series": "Series", diff --git a/internal/notices/notices_test.go b/internal/notices/notices_test.go new file mode 100644 index 0000000..9bcb73d --- /dev/null +++ b/internal/notices/notices_test.go @@ -0,0 +1,51 @@ +package notices + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "io" + "testing" +) + +func TestReadNoticeFromArchive(t *testing.T) { + original := payload + t.Cleanup(func() { payload = original }) + payload = noticeArchive(t, map[string]string{"LICENSE.txt": "license text", "THIRD-PARTY.txt": "dependency notices"}) + + got, err := Read("LICENSE.txt") + if err != nil { + t.Fatalf("Read() error = %v", err) + } + if string(got) != "license text" { + t.Fatalf("Read() = %q, want license text", got) + } + if _, err := Read("missing.txt"); err == nil { + t.Fatal("Read() succeeded for a missing notice") + } + if !bytes.Equal(Archive(), payload) { + t.Fatal("Archive() does not return the bundled archive") + } +} + +func noticeArchive(t *testing.T, files map[string]string) []byte { + t.Helper() + var buffer bytes.Buffer + compressed := gzip.NewWriter(&buffer) + archive := tar.NewWriter(compressed) + for name, content := range files { + if err := archive.WriteHeader(&tar.Header{Name: name, Mode: 0600, Size: int64(len(content)), Typeflag: tar.TypeReg}); err != nil { + t.Fatalf("write tar header: %v", err) + } + if _, err := io.WriteString(archive, content); err != nil { + t.Fatalf("write tar entry: %v", err) + } + } + if err := archive.Close(); err != nil { + t.Fatalf("close tar archive: %v", err) + } + if err := compressed.Close(); err != nil { + t.Fatalf("close gzip archive: %v", err) + } + return buffer.Bytes() +} diff --git a/internal/ocrruntime/runtime_test.go b/internal/ocrruntime/runtime_test.go new file mode 100644 index 0000000..d8ca77e --- /dev/null +++ b/internal/ocrruntime/runtime_test.go @@ -0,0 +1,141 @@ +package ocrruntime + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "context" + "os" + "path/filepath" + "testing" +) + +func TestExtractWritesOnlyLocalRegularFiles(t *testing.T) { + original := payload + t.Cleanup(func() { payload = original }) + payload = ocrArchive(t, []archiveEntry{{name: "tessdata/eng.traineddata", data: []byte("model")}, {name: "tesseract", data: []byte("engine"), mode: 0700}}) + dir := t.TempDir() + if err := extract(context.Background(), dir); err != nil { + t.Fatalf("extract() error = %v", err) + } + for name, want := range map[string][]byte{"tessdata/eng.traineddata": []byte("model"), "tesseract": []byte("engine")} { + got, err := os.ReadFile(filepath.Join(dir, name)) + if err != nil { + t.Fatalf("read extracted %s: %v", name, err) + } + if !bytes.Equal(got, want) { + t.Errorf("%s = %q, want %q", name, got, want) + } + } + info, err := os.Stat(filepath.Join(dir, "tesseract")) + if err != nil { + t.Fatal(err) + } + if info.Mode().Perm() != 0700 { + t.Errorf("executable mode = %o, want 700", info.Mode().Perm()) + } +} + +func TestExtractRejectsUnsafeArchiveEntries(t *testing.T) { + for _, test := range []struct { + name string + entry archiveEntry + }{ + {name: "path traversal", entry: archiveEntry{name: "../outside", data: []byte("no")}}, + {name: "symbolic link", entry: archiveEntry{name: "link", typeflag: tar.TypeSymlink}}, + } { + t.Run(test.name, func(t *testing.T) { + original := payload + t.Cleanup(func() { payload = original }) + payload = ocrArchive(t, []archiveEntry{test.entry}) + dir := t.TempDir() + if err := extract(context.Background(), dir); err == nil { + t.Fatal("extract() succeeded for an unsafe archive entry") + } + }) + } +} + +func TestExtractHonorsCancellation(t *testing.T) { + original := payload + t.Cleanup(func() { payload = original }) + payload = ocrArchive(t, []archiveEntry{{name: "model", data: []byte("model")}}) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if err := extract(ctx, t.TempDir()); err != context.Canceled { + t.Fatalf("extract() error = %v, want context.Canceled", err) + } +} + +func TestInstallModelPreservesExistingAndInstallsAtomically(t *testing.T) { + dir := t.TempDir() + source := filepath.Join(dir, "bundled.traineddata") + destination := filepath.Join(dir, "tessdata", "eng.traineddata") + if err := os.WriteFile(source, []byte("bundled"), 0600); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Dir(destination), 0700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(destination, []byte("user model"), 0600); err != nil { + t.Fatal(err) + } + if err := installModel(context.Background(), source, destination); err != nil { + t.Fatalf("installModel() existing: %v", err) + } + got, err := os.ReadFile(destination) + if err != nil || string(got) != "user model" { + t.Fatalf("existing model = %q, error %v; user model should be preserved", got, err) + } + + if err := os.Remove(destination); err != nil { + t.Fatal(err) + } + if err := installModel(context.Background(), source, destination); err != nil { + t.Fatalf("installModel() missing model: %v", err) + } + got, err = os.ReadFile(destination) + if err != nil || string(got) != "bundled" { + t.Fatalf("installed model = %q, error %v", got, err) + } + + if err := os.WriteFile(destination, nil, 0600); err != nil { + t.Fatal(err) + } + if err := installModel(context.Background(), source, destination); err == nil { + t.Fatal("installModel() accepted an empty existing model") + } +} + +type archiveEntry struct { + name string + data []byte + mode int64 + typeflag byte +} + +func ocrArchive(t *testing.T, entries []archiveEntry) []byte { + t.Helper() + var buffer bytes.Buffer + compressed := gzip.NewWriter(&buffer) + archive := tar.NewWriter(compressed) + for _, entry := range entries { + typeflag := entry.typeflag + if typeflag == 0 { + typeflag = tar.TypeReg + } + if err := archive.WriteHeader(&tar.Header{Name: entry.name, Mode: entry.mode, Size: int64(len(entry.data)), Typeflag: typeflag}); err != nil { + t.Fatalf("write tar header: %v", err) + } + if _, err := archive.Write(entry.data); err != nil { + t.Fatalf("write tar entry: %v", err) + } + } + if err := archive.Close(); err != nil { + t.Fatalf("close tar writer: %v", err) + } + if err := compressed.Close(); err != nil { + t.Fatalf("close gzip writer: %v", err) + } + return buffer.Bytes() +} diff --git a/internal/settings/settings_test.go b/internal/settings/settings_test.go new file mode 100644 index 0000000..298edf6 --- /dev/null +++ b/internal/settings/settings_test.go @@ -0,0 +1,106 @@ +package settings + +import ( + "context" + "fmt" + "log/slog" + "strings" + "testing" +) + +type memorySettings struct{ values map[string]string } + +func (m *memorySettings) GetSetting(_ context.Context, key string) (string, error) { + return m.values[key], nil +} + +func (m *memorySettings) SetSetting(_ context.Context, key, value string) error { + m.values[key] = value + return nil +} + +func TestDefaults(t *testing.T) { + got := Defaults() + if got.LogLevel != slog.LevelInfo || !got.Recursive || !got.InspectContent || !got.OCR || !got.GoogleBooks || !got.OpenLibrary || !got.InternetArchive { + t.Fatalf("unexpected default preferences: %+v", got) + } + if got.MaxCandidates != 12 || got.CacheDays != 7 || got.BackgroundDelay != 10 { + t.Fatalf("unexpected default limits: %+v", got) + } +} + +func TestPreferencesValidateBoundaries(t *testing.T) { + valid := Defaults() + valid.LastFolder = strings.Repeat("a", 4096) + valid.GoogleBooksAPIKey = strings.Repeat("k", 512) + if err := valid.Validate(); err != nil { + t.Fatalf("valid boundary preferences rejected: %v", err) + } + for _, test := range []struct { + name string + edit func(*Preferences) + }{ + {"unknown log level", func(p *Preferences) { p.LogLevel = slog.Level(5) }}, + {"overlong folder", func(p *Preferences) { p.LastFolder += "x" }}, + {"NUL folder", func(p *Preferences) { p.LastFolder = "bad\x00path" }}, + {"zero candidate limit", func(p *Preferences) { p.MaxCandidates = 0 }}, + {"candidate limit too large", func(p *Preferences) { p.MaxCandidates = 51 }}, + {"negative cache age", func(p *Preferences) { p.CacheDays = -1 }}, + {"cache age too large", func(p *Preferences) { p.CacheDays = 366 }}, + {"zero background delay", func(p *Preferences) { p.BackgroundDelay = 0 }}, + {"background delay too large", func(p *Preferences) { p.BackgroundDelay = 301 }}, + {"API key too long", func(p *Preferences) { p.GoogleBooksAPIKey += "x" }}, + {"API key contains newline", func(p *Preferences) { p.GoogleBooksAPIKey = "bad\nkey" }}, + } { + t.Run(test.name, func(t *testing.T) { + prefs := valid + test.edit(&prefs) + if err := prefs.Validate(); err == nil { + t.Fatal("Validate() unexpectedly succeeded") + } + }) + } +} + +func TestLoadDefaultsAndLegacySettings(t *testing.T) { + for _, test := range []struct { + name string + values map[string]string + check func(Preferences) error + }{ + {name: "first launch", values: map[string]string{}, check: func(p Preferences) error { + if p != Defaults() { return fmt.Errorf("got %+v, want defaults %+v", p, Defaults()) } + return nil + }}, + {name: "legacy values", values: map[string]string{"last_folder": "/books", "google_books_api_key": "secret"}, check: func(p Preferences) error { + if p.LastFolder != "/books" || p.GoogleBooksAPIKey != "secret" { return fmt.Errorf("legacy fields not loaded: %+v", p) } + return nil + }}, + } { + t.Run(test.name, func(t *testing.T) { + got, err := Load(context.Background(), &memorySettings{values: test.values}) + if err != nil { t.Fatalf("Load() error = %v", err) } + if err := test.check(got); err != nil { t.Fatal(err) } + }) + } +} + +func TestLoadRejectsInvalidStoredPreferences(t *testing.T) { + db := &memorySettings{values: map[string]string{"preferences": `{"max_candidates":999}`}} + if _, err := Load(context.Background(), db); err == nil { + t.Fatal("Load() accepted invalid stored settings") + } +} + +func TestSaveValidatesAndPersistsPreferences(t *testing.T) { + db := &memorySettings{values: map[string]string{}} + prefs := Defaults() + prefs.LastFolder = "/books" + prefs.GoogleBooksAPIKey = "secret" + if err := Save(context.Background(), db, prefs); err != nil { t.Fatalf("Save() error = %v", err) } + got, err := Load(context.Background(), db) + if err != nil { t.Fatalf("Load() error = %v", err) } + if got != prefs { t.Fatalf("loaded preferences = %+v, want %+v", got, prefs) } + prefs.MaxCandidates = 0 + if err := Save(context.Background(), db, prefs); err == nil { t.Fatal("Save() accepted invalid preferences") } +} diff --git a/internal/store/store_test.go b/internal/store/store_test.go new file mode 100644 index 0000000..2938b08 --- /dev/null +++ b/internal/store/store_test.go @@ -0,0 +1,229 @@ +package store + +import ( + "context" + "database/sql" + "encoding/json" + "errors" + "path/filepath" + "testing" + "time" + + "shelfmark/internal/library" +) + +func openTestStore(t *testing.T) *Store { + t.Helper() + db, err := Open(context.Background(), filepath.Join(t.TempDir(), "shelfmark.sqlite3")) + if err != nil { + t.Fatalf("Open() error = %v", err) + } + t.Cleanup(func() { + if err := db.Close(); err != nil { + t.Errorf("Close() error = %v", err) + } + }) + return db +} + +func TestReadBookOverlaysAndReconcilesDraft(t *testing.T) { + ctx := context.Background() + db := openTestStore(t) + path := "/books/title.epub" + current := map[string]any{"title": "Original", "authors": []string{"Author"}} + working, staged, err := db.ReadBook(ctx, path, "fingerprint-one", current) + if err != nil { + t.Fatalf("initial ReadBook() error = %v", err) + } + if staged || working["title"] != "Original" { + t.Fatalf("initial ReadBook() = %#v, staged %t", working, staged) + } + + draft := library.Working(current, map[string]any{"title": "Edited"}) + if err := db.Stage(ctx, path, "fingerprint-one", current, draft); err != nil { + t.Fatalf("Stage() error = %v", err) + } + working, staged, err = db.ReadBook(ctx, path, "fingerprint-two", current) + if err != nil { + t.Fatalf("ReadBook() with draft error = %v", err) + } + if !staged || working["title"] != "Edited" { + t.Fatalf("ReadBook() failed to overlay draft: %#v, staged %t", working, staged) + } + + current["title"] = "Edited" + working, staged, err = db.ReadBook(ctx, path, "fingerprint-three", current) + if err != nil { + t.Fatalf("ReadBook() reconciliation error = %v", err) + } + if staged || working["title"] != "Edited" { + t.Fatalf("already-applied draft was not reconciled: %#v, staged %t", working, staged) + } +} + +func TestFinishSaveRetainsOnlyUnselectedChanges(t *testing.T) { + ctx := context.Background() + db := openTestStore(t) + file := map[string]any{"title": "File", "authors": []string{"Author"}} + draft := library.Working(file, map[string]any{"title": "Saved", "publisher": "Publisher"}) + if err := db.Stage(ctx, "/books/book.epub", "before", file, draft); err != nil { + t.Fatalf("Stage() error = %v", err) + } + written := library.Working(file, map[string]any{"title": "Saved"}) + remaining := map[string]any{"publisher": "Publisher"} + if err := db.FinishSave(ctx, "/books/book.epub", "after", written, remaining); err != nil { + t.Fatalf("FinishSave() error = %v", err) + } + state, err := db.State(ctx, "/books/book.epub") + if err != nil { + t.Fatalf("State() error = %v", err) + } + if state.Fingerprint != "after" || state.Metadata["title"] != "Saved" || state.Staged["publisher"] != "Publisher" { + t.Fatalf("saved state = %+v", state) + } +} + +func TestDiscardDraftsClearsSelectedPatches(t *testing.T) { + ctx := context.Background() + db := openTestStore(t) + for _, path := range []string{"/books/a.epub", "/books/b.epub"} { + metadata := map[string]any{"title": "Original"} + draft := map[string]any{"title": "Edited"} + if err := db.Stage(ctx, path, "fingerprint", metadata, draft); err != nil { + t.Fatalf("Stage(%q) error = %v", path, err) + } + } + if err := db.DiscardDrafts(ctx, []string{"/books/a.epub"}); err != nil { + t.Fatalf("DiscardDrafts() error = %v", err) + } + staged, err := db.Staged(ctx) + if err != nil { + t.Fatalf("Staged() error = %v", err) + } + if len(staged) != 1 || staged[0].Path != "/books/b.epub" { + t.Fatalf("remaining staged books = %+v", staged) + } +} + +func TestCacheExpirationAndBound(t *testing.T) { + ctx := context.Background() + db := openTestStore(t) + type result struct{ Value string } + if err := db.CachePut(ctx, "fresh", result{Value: "fresh"}); err != nil { + t.Fatalf("CachePut() error = %v", err) + } + var got result + ok, err := db.CacheGet(ctx, "fresh", time.Hour, &got) + if err != nil || !ok || got.Value != "fresh" { + t.Fatalf("fresh cache get = (%+v, %t, %v)", got, ok, err) + } + ok, err = db.CacheGet(ctx, "fresh", 0, &got) + if err != nil || ok { + t.Fatalf("disabled cache get = (%+v, %t, %v), want miss", got, ok, err) + } + for i := 0; i < 505; i++ { + key, value, err := cacheEntry(i) + if err != nil { + t.Fatal(err) + } + if err := db.CachePut(ctx, key, value); err != nil { + t.Fatalf("CachePut(%s): %v", key, err) + } + } + var count int + if err := db.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM search_matches").Scan(&count); err != nil { + t.Fatalf("count cache rows: %v", err) + } + if count > 500 { + t.Fatalf("cache has %d rows, want at most 500", count) + } +} + +func cacheEntry(i int) (string, map[string]int, error) { + encoded, err := json.Marshal(map[string]int{"value": i}) + return string(encoded), map[string]int{"value": i}, err +} + +func TestMigrationConvertsLegacyFullDraftsToPatch(t *testing.T) { + filename := filepath.Join(t.TempDir(), "legacy.sqlite3") + raw, err := sql.Open("sqlite", filename) + if err != nil { + t.Fatalf("open legacy sqlite: %v", err) + } + _, err = raw.Exec(` + CREATE TABLE settings(key TEXT PRIMARY KEY,value TEXT NOT NULL); + CREATE TABLE search_matches(cache_key TEXT PRIMARY KEY,result_json TEXT NOT NULL,updated_at INTEGER NOT NULL); + CREATE TABLE book_state(path TEXT PRIMARY KEY,fingerprint TEXT NOT NULL,metadata_json TEXT NOT NULL,staged_json TEXT,updated_at INTEGER NOT NULL); + PRAGMA user_version = 1; + INSERT INTO book_state(path,fingerprint,metadata_json,staged_json,updated_at) + VALUES('/books/book.epub','old','{"title":"Old","authors":["A"]}','{"title":"New","authors":["A"],"subtitle":"Sub"}',1); + `) + if err != nil { + t.Fatalf("create legacy database: %v", err) + } + if err := raw.Close(); err != nil { + t.Fatalf("close legacy database: %v", err) + } + + db, err := Open(context.Background(), filename) + if err != nil { + t.Fatalf("Open() migration error = %v", err) + } + defer func() { _ = db.Close() }() + state, err := db.State(context.Background(), "/books/book.epub") + if err != nil { + t.Fatalf("State() error = %v", err) + } + if len(state.Staged) != 2 || state.Staged["title"] != "New" || state.Staged["subtitle"] != "Sub" { + t.Fatalf("migrated patch = %#v", state.Staged) + } + var version int + if err := db.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil { + t.Fatalf("read schema version: %v", err) + } + if version != schemaVersion { + t.Fatalf("schema version = %d, want %d", version, schemaVersion) + } +} + +func TestMigrationFailureLeavesLegacyVersion(t *testing.T) { + filename := filepath.Join(t.TempDir(), "broken.sqlite3") + raw, err := sql.Open("sqlite", filename) + if err != nil { + t.Fatal(err) + } + _, err = raw.Exec(` + CREATE TABLE settings(key TEXT PRIMARY KEY,value TEXT NOT NULL); + CREATE TABLE search_matches(cache_key TEXT PRIMARY KEY,result_json TEXT NOT NULL,updated_at INTEGER NOT NULL); + CREATE TABLE book_state(path TEXT PRIMARY KEY,fingerprint TEXT NOT NULL,metadata_json TEXT NOT NULL,staged_json TEXT,updated_at INTEGER NOT NULL); + PRAGMA user_version = 1; + INSERT INTO book_state(path,fingerprint,metadata_json,staged_json,updated_at) + VALUES('/books/broken.epub','old','not json','{}',1); + `) + if err != nil { + t.Fatal(err) + } + _ = raw.Close() + if _, err := Open(context.Background(), filename); err == nil { + t.Fatal("Open() unexpectedly migrated malformed row") + } + check, err := sql.Open("sqlite", filename) + if err != nil { + t.Fatal(err) + } + defer func() { _ = check.Close() }() + var version int + if err := check.QueryRow("PRAGMA user_version").Scan(&version); err != nil { + t.Fatal(err) + } + if version != 1 { + t.Fatalf("failed migration set schema version %d, want 1", version) + } +} + +func TestStateMissingBook(t *testing.T) { + _, err := openTestStore(t).State(context.Background(), "/missing") + if !errors.Is(err, sql.ErrNoRows) { + t.Fatalf("State() error = %v, want sql.ErrNoRows", err) + } +} diff --git a/internal/webui/js/app.js b/internal/webui/js/app.js index bd15477..d14128e 100644 --- a/internal/webui/js/app.js +++ b/internal/webui/js/app.js @@ -182,6 +182,8 @@ queryElement('#cancelPicker').onclick = () => closeDialog(queryElement('#picker' async function select(b) { const generation = ++selectionGeneration; searchController?.abort(); + searchController = new AbortController(); + const signal = searchController.signal; active = b; candidates = []; chosen = -1; @@ -191,6 +193,7 @@ async function select(b) { queryElement('#search').onclick = search; queryElement('#query').addEventListener('keydown', event => { if (event.key === 'Enter') search(); }); void renderCompare(); + void loadCachedMatches(b, generation, signal); try { const d = await api('/api/cover', {id: b.id}); if (generation !== selectionGeneration) return; @@ -199,6 +202,24 @@ async function select(b) { } } +async function loadCachedMatches(book, generation, signal) { + try { + const result = await api('/api/search/cached', {id: book.id}, signal); + // A later selection or explicit search owns the UI once it aborts this + // request. Also leave edited search terms alone while the cache loads. + if (signal.aborted || generation !== selectionGeneration || + queryElement('#query')?.value !== book.search_query || queryElement('#ignoreCache')?.checked) return; + candidates = result.candidates || []; + if (candidates.length) { + renderMatches(); + if (result.warnings?.length) queryElement('#matches').insertAdjacentHTML('afterbegin', `

Previous search: ${esc(result.warnings.join('; '))}

`); + } + } catch (error) { + if (signal.aborted || generation !== selectionGeneration) return; + queryElement('#matches').innerHTML = `

Could not load cached matches: ${esc(error.message)}. Use Find matches to search again.

`; + } +} + async function search() { if (queryElement('#search').disabled) return; const b = active, query = queryElement('#query').value, btn = queryElement('#search'), generation = selectionGeneration;