diff --git a/CHANGELOG.md b/CHANGELOG.md index ae2cec1cef4..6298eff1ae1 100755 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ ## [Windows] Unreleased +- Claude: cost scans keep the parsed lines of each Claude Code transcript in memory and read only what was appended since the last scan. A Usage & Spend refresh walks the transcripts several times (7 days, 30 days, the selected period, token activity and the trend), and each walk used to re-read every transcript in its window. Totals, de-duplication across files, incomplete-request markers and pricing are still worked out on every scan, so the numbers do not change. A transcript that shrinks or is replaced is read again from the start. - Tray panel: **Settings > Menu > Panel look** switches between Modern and Classic. Classic is the layout from before the redesign (icon-only provider switcher, percent after the title, the full cost dashboard, shortcut hints and icons on every menu row). Drag either vertical edge of the panel to change its width (280 to 560 px at 100 % Panel scale); the width is saved, arrow keys on the focused edge step it, and a double-click or Home restores 310 px. At a high Panel scale the panel never grows wider than the screen's work area. - Tray panel: a new look. The panel sits on a soft tinted surface with larger type. Provider tabs show an icon and a name over a quota underline, wrap to extra rows when many providers are on, and keep the dense icon grid beyond 18. Each quota row is a title, a thicker bar, then the percent used on the left and the reset on the right. Sections are set off by hairlines, and the daily-cost dashboard becomes a collapsible **Cost** section that opens from a two-line summary. The menu is text-only: shortcut hints and the Settings, About and Quit icons are gone, and About no longer carries the version. - Claude: when Claude Code signs out but keeps its login entry with blank tokens, Settings and the tray still list the saved Claude Code accounts, and Switch and Add account work again. Before, every account action failed with "Claude login is missing accessToken" and "No Claude Code accounts found", even though the accounts were saved. diff --git a/rust/src/cost_scanner.rs b/rust/src/cost_scanner.rs index 3b53890375d..2972c60bf15 100755 --- a/rust/src/cost_scanner.rs +++ b/rust/src/cost_scanner.rs @@ -34,6 +34,7 @@ use crate::providers::claude::quota_history::ClaudeQuotaHistoryRecord; use crate::providers::opencodego::local as opencodego_local; use crate::settings::Settings; mod claude_events; +mod claude_file_cache; mod claude_incomplete; mod claude_pricing; mod claude_roots; @@ -257,7 +258,7 @@ fn file_len_i64(metadata: &fs::Metadata) -> i64 { i64::try_from(metadata.len()).unwrap_or(i64::MAX) } -#[derive(Debug)] +#[derive(Debug, Clone)] struct ClaudeUsageRecord { model: String, pricing_known: bool, diff --git a/rust/src/cost_scanner/claude_file_cache.rs b/rust/src/cost_scanner/claude_file_cache.rs new file mode 100644 index 00000000000..dd3d73f80af --- /dev/null +++ b/rust/src/cost_scanner/claude_file_cache.rs @@ -0,0 +1,416 @@ +//! In-process cache of parsed Claude transcript lines. +//! +//! Claude Code transcripts are append-only JSONL files, and one Usage & Spend +//! build walks them several times (7-day, 30-day and selected-period +//! summaries, the 365-day token activity grid and the trend history). Without +//! a cache every walk re-read and re-parsed every transcript in its window, +//! which on a large `~/.claude/projects` tree means gigabytes of JSON per +//! build. +//! +//! This cache keeps, per transcript path, the lines that matter to cost +//! scanning in the order they appear, and remembers how far the file was +//! parsed. A later scan reuses an unchanged file as is and parses only the +//! bytes appended since. Everything that depends on the walk rather than on the +//! file (the cutoff, cross-file de-duplication, incomplete-request +//! reconciliation and pricing) is still applied on every scan when the lines +//! are replayed, so results match a cold scan. +//! +//! Claude Code writes one line per content block of an assistant message, and +//! each of those lines repeats the message's usage under the same dedup key. +//! A later line whose usage equals the first line with its key is stored as a +//! small [`ClaudeParsedLine::Repeat`] (an index and its own timestamp) instead +//! of a second copy of the record, which keeps the cache close to one full +//! entry per request without changing any scan result. +//! +//! A file is re-parsed from the start when it shrank or when the bytes just +//! before the cached offset no longer match (the file was replaced). A final +//! line without a trailing newline is parsed on every scan and never +//! committed, because the writer may still be appending to it. + +use std::collections::HashMap; +use std::fs::File; +use std::hash::{DefaultHasher, Hasher}; +use std::io::{BufRead, BufReader, Read, Seek, SeekFrom}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::AtomicBool; +use std::sync::{Arc, LazyLock, Mutex, PoisonError}; +use std::time::{Duration, Instant, SystemTime}; + +use chrono::{DateTime, Utc}; + +use super::claude_pricing::ClaudeScanPricingResolver; +use super::claude_scan::claude_parsed_line; +use super::claude_usage::ClaudeUsageDedupKey; +use super::{ClaudeUsageRecord, is_cancelled}; + +/// Upper bound on cached lines across all transcripts. Past it, new files are +/// still scanned but not retained, so memory stays bounded on huge trees. +const MAX_CACHED_LINES: usize = 4_000_000; + +/// Bytes before the committed offset that must still match to resume. +const FINGERPRINT_BYTES: u64 = 4096; + +/// How often entries for deleted transcripts are dropped. +const PRUNE_INTERVAL: Duration = Duration::from_secs(30 * 60); + +/// One transcript line that cost scanning acts on. +#[derive(Debug, Clone)] +pub(super) enum ClaudeParsedLine { + /// A line that was not valid UTF-8 or not valid JSON. + Malformed, + /// A preliminary (incomplete) usage row. `row` is absent when the event + /// carried no message, which still counts as an incomplete request. + Preliminary { + row: Option<(Option, String, Option>)>, + }, + /// A usage row. Pricing fields hold the values from when it was parsed; + /// they are recomputed on replay. + Usage { + record: ClaudeUsageRecord, + cache_create_1h: u64, + }, + /// A usage line equal to the earlier `Usage` line at index `of` (in + /// [`ClaudeFileLines::get`] order) except for its timestamp. + Repeat { + of: usize, + timestamp: Option>, + }, +} + +/// Parsed lines of one transcript. +#[derive(Debug, Default)] +pub(super) struct ClaudeFileLines { + /// Lines that ended with a newline, up to `committed_len`. + pub(super) committed: Vec, + /// A trailing line without a newline, if any. + pub(super) tail: Vec, + committed_len: u64, + file_len: u64, + modified: Option, + fingerprint: Option, +} + +impl ClaudeFileLines { + pub(super) fn lines(&self) -> impl Iterator { + self.committed.iter().chain(self.tail.iter()) + } + + /// The line at `index`, counting committed lines first. + pub(super) fn get(&self, index: usize) -> Option<&ClaudeParsedLine> { + match index.checked_sub(self.committed.len()) { + None => self.committed.get(index), + Some(tail_index) => self.tail.get(tail_index), + } + } +} + +/// Lines for one scan, plus the read failures seen while producing them. +pub(super) struct ClaudeFileSnapshot { + pub(super) lines: Arc, + pub(super) read_failures: u32, +} + +#[derive(Default)] +struct ClaudeFileCache { + entries: HashMap>, + cached_lines: usize, + last_prune: Option, +} + +impl ClaudeFileCache { + fn insert(&mut self, path: PathBuf, lines: Arc) { + let added = lines.committed.len() + lines.tail.len(); + let removed = self + .entries + .get(&path) + .map_or(0, |old| old.committed.len() + old.tail.len()); + if self.cached_lines - removed + added > MAX_CACHED_LINES { + if let Some(old) = self.entries.remove(&path) { + self.cached_lines -= old.committed.len() + old.tail.len(); + } + return; + } + self.cached_lines = self.cached_lines - removed + added; + self.entries.insert(path, lines); + } + + fn remove(&mut self, path: &Path) { + if let Some(old) = self.entries.remove(path) { + self.cached_lines -= old.committed.len() + old.tail.len(); + } + } +} + +static CACHE: LazyLock> = + LazyLock::new(|| Mutex::new(ClaudeFileCache::default())); + +fn cache() -> std::sync::MutexGuard<'static, ClaudeFileCache> { + CACHE.lock().unwrap_or_else(PoisonError::into_inner) +} + +/// Lines of `path`, parsing only what the cache does not already hold. +/// +/// `file` must be an open handle to `path`. +pub(super) fn claude_file_lines( + path: &Path, + mut file: File, + cancel: Option<&AtomicBool>, + pricing: &mut ClaudeScanPricingResolver, +) -> ClaudeFileSnapshot { + prune_deleted_entries(); + let (file_len, modified) = match file.metadata() { + Ok(metadata) => (metadata.len(), metadata.modified().ok()), + Err(_) => (0, None), + }; + let cached = cache().entries.get(path).cloned(); + + if let Some(cached) = &cached + && modified.is_some() + && cached.modified == modified + && cached.file_len == file_len + { + return ClaudeFileSnapshot { + lines: Arc::clone(cached), + read_failures: 0, + }; + } + + let base = cached.filter(|cached| { + cached.committed_len <= file_len + && cached.fingerprint.is_some() + && fingerprint(&mut file, cached.committed_len) == cached.fingerprint + }); + let start = base.as_ref().map_or(0, |base| base.committed_len); + let segment = parse_segment(&mut file, start, cancel, pricing); + + let mut committed = base + .as_ref() + .map(|base| base.committed.clone()) + .unwrap_or_default(); + let mut anchors = RepeatAnchors::default(); + for (index, line) in committed.iter().enumerate() { + anchors.note(index, line); + } + for line in segment.committed { + let line = anchors.compact(&committed, &[], line); + anchors.note(committed.len(), &line); + committed.push(line); + } + let mut tail = Vec::with_capacity(segment.tail.len()); + for line in segment.tail { + let line = anchors.compact(&committed, &tail, line); + anchors.note(committed.len() + tail.len(), &line); + tail.push(line); + } + let committed_len = start + segment.committed_bytes; + let lines = Arc::new(ClaudeFileLines { + committed, + tail, + committed_len, + file_len, + modified, + fingerprint: fingerprint(&mut file, committed_len), + }); + + // A partial walk (read error or cancellation) or a file without a usable + // modification time is never cached, so the next scan reads it again. + let complete = !segment.read_failed && !segment.cancelled && modified.is_some(); + let mut cache = cache(); + if complete { + cache.insert(path.to_path_buf(), Arc::clone(&lines)); + } else { + cache.remove(path); + } + drop(cache); + + ClaudeFileSnapshot { + lines, + read_failures: u32::from(segment.read_failed), + } +} + +struct ParsedSegment { + committed: Vec, + committed_bytes: u64, + tail: Vec, + read_failed: bool, + cancelled: bool, +} + +/// Parse `file` from `start` to its end. Lines ending with a newline are +/// committed; a final unterminated line goes to `tail`. +fn parse_segment( + file: &mut File, + start: u64, + cancel: Option<&AtomicBool>, + pricing: &mut ClaudeScanPricingResolver, +) -> ParsedSegment { + let mut segment = ParsedSegment { + committed: Vec::new(), + committed_bytes: 0, + tail: Vec::new(), + read_failed: false, + cancelled: false, + }; + if file.seek(SeekFrom::Start(start)).is_err() { + segment.read_failed = true; + return segment; + } + let mut reader = BufReader::new(file); + let mut consumed = 0u64; + let mut buf = Vec::new(); + loop { + buf.clear(); + let read = match reader.read_until(b'\n', &mut buf) { + Ok(0) => break, + Ok(read) => read, + Err(_) => { + segment.read_failed = true; + break; + } + }; + if is_cancelled(cancel) { + segment.cancelled = true; + break; + } + consumed += read as u64; + let terminated = buf.last() == Some(&b'\n'); + while matches!(buf.last(), Some(b'\n' | b'\r')) { + buf.pop(); + } + let parsed = match std::str::from_utf8(&buf) { + Ok(line) => claude_parsed_line(line, pricing), + Err(_) => Some(ClaudeParsedLine::Malformed), + }; + if terminated { + segment.committed.extend(parsed); + segment.committed_bytes = consumed; + } else { + segment.tail.extend(parsed); + } + } + segment +} + +/// Index of the first `Usage` line for each dedup key in one file. +#[derive(Default)] +struct RepeatAnchors { + first: HashMap, +} + +impl RepeatAnchors { + /// Remember `line` (at `index`) as the anchor for its key if it is the + /// first `Usage` line with that key. + fn note(&mut self, index: usize, line: &ClaudeParsedLine) { + if let ClaudeParsedLine::Usage { record, .. } = line + && let Some(key) = &record.dedup_key + && !self.first.contains_key(key) + { + self.first.insert(key.clone(), index); + } + } + + /// `line`, or a `Repeat` of its anchor when only the timestamp differs. + /// `committed` and `tail` are the lines before it. + fn compact( + &self, + committed: &[ClaudeParsedLine], + tail: &[ClaudeParsedLine], + line: ClaudeParsedLine, + ) -> ClaudeParsedLine { + let ClaudeParsedLine::Usage { + record, + cache_create_1h, + } = &line + else { + return line; + }; + let Some(&of) = record + .dedup_key + .as_ref() + .and_then(|key| self.first.get(key)) + else { + return line; + }; + let anchor = match of.checked_sub(committed.len()) { + None => committed.get(of), + Some(tail_index) => tail.get(tail_index), + }; + let Some(ClaudeParsedLine::Usage { + record: anchor, + cache_create_1h: anchor_1h, + }) = anchor + else { + return line; + }; + let same_usage = anchor.model == record.model + && anchor.dedup_key == record.dedup_key + && anchor.input == record.input + && anchor.output == record.output + && anchor.cache_create == record.cache_create + && anchor.cache_read == record.cache_read + && anchor_1h == cache_create_1h; + if !same_usage { + return line; + } + ClaudeParsedLine::Repeat { + of, + timestamp: record.timestamp, + } + } +} + +/// Hash of up to [`FINGERPRINT_BYTES`] ending at `end`, or `None` when the +/// bytes cannot be read. An empty prefix has a fixed fingerprint. +fn fingerprint(file: &mut File, end: u64) -> Option { + let start = end.saturating_sub(FINGERPRINT_BYTES); + file.seek(SeekFrom::Start(start)).ok()?; + let len = usize::try_from(end - start).ok()?; + let mut bytes = vec![0u8; len]; + file.read_exact(&mut bytes).ok()?; + let mut hasher = DefaultHasher::new(); + hasher.write_u64(end); + hasher.write(&bytes); + Some(hasher.finish()) +} + +/// Drop entries whose transcript no longer exists, at most once per +/// [`PRUNE_INTERVAL`]. File checks run without holding the cache lock. +fn prune_deleted_entries() { + let paths: Vec = { + let mut cache = cache(); + let now = Instant::now(); + if cache + .last_prune + .is_some_and(|last| now.duration_since(last) < PRUNE_INTERVAL) + { + return; + } + cache.last_prune = Some(now); + cache.entries.keys().cloned().collect() + }; + let missing: Vec = paths.into_iter().filter(|path| !path.exists()).collect(); + if missing.is_empty() { + return; + } + let mut cache = cache(); + for path in &missing { + cache.remove(path); + } +} + +#[cfg(test)] +pub(super) fn cached_line_count(path: &Path) -> Option { + cache().entries.get(path).map(|lines| lines.committed.len()) +} + +#[cfg(test)] +pub(super) fn cached_repeat_count(path: &Path) -> Option { + cache().entries.get(path).map(|lines| { + lines + .committed + .iter() + .filter(|line| matches!(line, ClaudeParsedLine::Repeat { .. })) + .count() + }) +} diff --git a/rust/src/cost_scanner/claude_scan.rs b/rust/src/cost_scanner/claude_scan.rs index 39caa81dfcf..7cdbbced834 100644 --- a/rust/src/cost_scanner/claude_scan.rs +++ b/rust/src/cost_scanner/claude_scan.rs @@ -1,14 +1,15 @@ //! Reading one Claude transcript file into usage records, and folding those //! records into summaries, daily buckets and quota history. +use std::borrow::Cow; use std::collections::{HashMap, HashSet}; use std::fs::File; -use std::io::{BufRead, BufReader}; use std::path::Path; use std::sync::atomic::AtomicBool; use chrono::{DateTime, Utc}; +use super::claude_file_cache::{ClaudeParsedLine, claude_file_lines}; use super::claude_incomplete::ClaudeIncompleteTracker; use super::claude_pricing::ClaudeScanPricingResolver; use super::claude_usage::{ @@ -42,92 +43,147 @@ where }; }; - let mut result = ClaudeFileScanResult::default(); - // Use read_until so a final incomplete line (no trailing newline) is still - // processed when it is valid UTF-8 JSON, and so a single bad line does not - // stop the walk the way `lines().map_while(Result::ok)` would. - let line_result = for_each_jsonl_text_line(BufReader::new(file), |line| { + // Parsing is cached per file (only appended bytes are read again); the + // cutoff, de-duplication, incomplete tracking and pricing below run on + // every scan, so the result matches a cold read of the file. + let snapshot = claude_file_lines(path, file, cancel, pricing); + let mut result = ClaudeFileScanResult { + read_failures: snapshot.read_failures, + ..ClaudeFileScanResult::default() + }; + for line in snapshot.lines.lines() { if is_cancelled(cancel) { - return false; - } - if line.trim().is_empty() { - return true; - } - let Ok(event) = serde_json::from_str::(line) else { - result.malformed_lines = result.malformed_lines.saturating_add(1); - return true; - }; - if event.is_vertex_ai_usage_entry() { - return true; + break; } - if is_preliminary_claude_usage(&event) { - result.incomplete_requests = result.incomplete_requests.saturating_add(1); - if let Some(message) = event.message.as_ref() { - incomplete.record( - claude_usage_dedup_key( - message.id.as_deref(), - event.request_id.as_deref(), - event.session_id(), - ), - message.model.as_deref().unwrap_or("claude-3-5-sonnet"), - event.parsed_timestamp(), - cutoff, - ); + match line { + ClaudeParsedLine::Malformed => { + result.malformed_lines = result.malformed_lines.saturating_add(1); + } + ClaudeParsedLine::Preliminary { row } => { + result.incomplete_requests = result.incomplete_requests.saturating_add(1); + if let Some((key, model, timestamp)) = row { + incomplete.record(key.clone(), model, *timestamp, cutoff); + } + } + ClaudeParsedLine::Usage { + record, + cache_create_1h, + } => { + let record = repriced_claude_record(record, *cache_create_1h, pricing); + if should_count_claude_record(&record, cutoff, seen) { + result.counted += 1; + on_record(&record); + } + } + ClaudeParsedLine::Repeat { of, timestamp } => { + let Some(ClaudeParsedLine::Usage { + record: anchor, + cache_create_1h, + }) = snapshot.lines.get(*of) + else { + continue; + }; + // Same checks as `should_count_claude_record`, made before the + // record is materialized: almost always the anchor was counted + // and this repeat is a duplicate. + if timestamp.is_some_and(|timestamp| timestamp < *cutoff) + || anchor + .dedup_key + .as_ref() + .is_some_and(|key| seen.contains(key)) + { + continue; + } + let repeat = ClaudeUsageRecord { + timestamp: *timestamp, + ..anchor.clone() + }; + let record = repriced_claude_record(&repeat, *cache_create_1h, pricing); + if should_count_claude_record(&record, cutoff, seen) { + result.counted += 1; + on_record(&record); + } } - return true; - } - if let Some(record) = claude_usage_record_from_event_with_pricing(&event, pricing) - && should_count_claude_record(&record, cutoff, seen) - { - result.counted += 1; - on_record(&record); } - true - }); - result.malformed_lines = result - .malformed_lines - .saturating_add(line_result.malformed_lines); - result.read_failures = line_result.read_failures; + } result } -/// Walk JSONL text lines from `reader`, including a final incomplete line at EOF. -/// Continues past invalid UTF-8 segments. `on_line` returns `false` to stop early. -pub(super) fn for_each_jsonl_text_line(mut reader: R, mut on_line: F) -> ClaudeJsonlLineResult -where - R: BufRead, - F: FnMut(&str) -> bool, -{ - let mut result = ClaudeJsonlLineResult::default(); - let mut buf = Vec::new(); - loop { - buf.clear(); - match reader.read_until(b'\n', &mut buf) { - Ok(0) => break, - Ok(_) => {} - Err(_) => { - result.read_failures = result.read_failures.saturating_add(1); - break; - } - } - while matches!(buf.last(), Some(b'\n' | b'\r')) { - buf.pop(); - } - let Ok(line) = std::str::from_utf8(&buf) else { - result.malformed_lines = result.malformed_lines.saturating_add(1); - continue; - }; - if !on_line(line) { - break; - } +/// Classify one transcript line for cost scanning. Lines that never affect a +/// scan (blank lines, Vertex AI rows, non-usage events) yield `None`. +pub(super) fn claude_parsed_line( + line: &str, + pricing: &mut ClaudeScanPricingResolver, +) -> Option { + if line.trim().is_empty() { + return None; } - result + let Ok(event) = serde_json::from_str::(line) else { + return Some(ClaudeParsedLine::Malformed); + }; + if event.is_vertex_ai_usage_entry() { + return None; + } + if is_preliminary_claude_usage(&event) { + let row = event.message.as_ref().map(|message| { + ( + claude_usage_dedup_key( + message.id.as_deref(), + event.request_id.as_deref(), + event.session_id(), + ), + message + .model + .as_deref() + .unwrap_or("claude-3-5-sonnet") + .to_string(), + event.parsed_timestamp(), + ) + }); + return Some(ClaudeParsedLine::Preliminary { row }); + } + let cache_create_1h = event + .message + .as_ref() + .and_then(|message| message.usage.as_ref()) + .map(|usage| { + usage.one_hour_cache_creation_tokens(usage.cache_creation_input_tokens.unwrap_or(0)) + }) + .unwrap_or(0); + claude_usage_record_from_event_with_pricing(&event, pricing).map(|record| { + ClaudeParsedLine::Usage { + record, + cache_create_1h, + } + }) } -#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] -pub(super) struct ClaudeJsonlLineResult { - pub(super) malformed_lines: u32, - pub(super) read_failures: u32, +/// `record` with pricing recomputed by `pricing`, borrowed when unchanged. +fn repriced_claude_record<'a>( + record: &'a ClaudeUsageRecord, + cache_create_1h: u64, + pricing: &mut ClaudeScanPricingResolver, +) -> Cow<'a, ClaudeUsageRecord> { + let pricing_known = pricing.is_known(&record.model); + let computed = pricing.cost_usd_with_cache_ttl( + &record.model, + record.timestamp, + record.input, + record.cache_create, + cache_create_1h, + record.cache_read, + record.output, + ); + let cost = computed.is_finite().then_some(computed); + let same_cost = cost.map(f64::to_bits) == record.cost.map(f64::to_bits); + if pricing_known == record.pricing_known && same_cost { + return Cow::Borrowed(record); + } + Cow::Owned(ClaudeUsageRecord { + pricing_known, + cost, + ..record.clone() + }) } pub(super) fn finalize_claude_summary( diff --git a/rust/src/cost_scanner/tests.rs b/rust/src/cost_scanner/tests.rs index eab803c9f8b..b3263943ac4 100644 --- a/rust/src/cost_scanner/tests.rs +++ b/rust/src/cost_scanner/tests.rs @@ -72,6 +72,9 @@ mod codex_usage; #[path = "tests/claude_records.rs"] mod claude_records; +#[path = "tests/claude_file_cache.rs"] +mod claude_file_cache_tests; + #[path = "tests/claude_daily.rs"] mod claude_daily; diff --git a/rust/src/cost_scanner/tests/claude_file_cache.rs b/rust/src/cost_scanner/tests/claude_file_cache.rs new file mode 100644 index 00000000000..703970ef6de --- /dev/null +++ b/rust/src/cost_scanner/tests/claude_file_cache.rs @@ -0,0 +1,229 @@ +//! Per-file Claude transcript cache: appended bytes are parsed once, and a +//! cached replay matches a cold scan. + +use super::*; +use crate::cost_scanner::claude_file_cache::{cached_line_count, cached_repeat_count}; +use std::fs::OpenOptions; + +fn usage_row(request: &str, input: u64) -> String { + usage_row_at(request, input, Utc::now() - Duration::hours(1)) +} + +fn usage_row_at(request: &str, input: u64, at: DateTime) -> String { + let ts = at.format("%Y-%m-%dT%H:%M:%S%.3fZ").to_string(); + format!( + r#"{{"type":"assistant","timestamp":"{ts}","requestId":"{request}","message":{{"id":"msg_{request}","model":"claude-sonnet-4-6","stop_reason":"end_turn","usage":{{"input_tokens":{input},"output_tokens":10}}}}}}"# + ) +} + +/// Scan one file with fresh walk state, returning (counted, input tokens, +/// malformed lines). +fn scan(path: &Path) -> (usize, u64, u32) { + let cutoff = Utc::now() - Duration::days(1); + let mut seen = HashSet::new(); + let mut pricing = ClaudeScanPricingResolver::default(); + let mut incomplete = ClaudeIncompleteTracker::default(); + let mut input = 0; + let result = scan_claude_file_with_pricing( + path, + &cutoff, + &mut seen, + None, + &mut pricing, + &mut incomplete, + |record| input += record.input, + ); + (result.counted, input, result.malformed_lines) +} + +fn append(path: &Path, text: &str) { + let mut file = OpenOptions::new().append(true).open(path).unwrap(); + file.write_all(text.as_bytes()).unwrap(); +} + +#[test] +fn appended_rows_are_added_to_the_cached_rows() { + let root = tempfile::tempdir().unwrap(); + let path = root.path().join("transcript.jsonl"); + std::fs::write( + &path, + format!("{}\n{}\n", usage_row("a", 100), usage_row("b", 200)), + ) + .unwrap(); + + assert_eq!(scan(&path), (2, 300, 0)); + assert_eq!(cached_line_count(&path), Some(2)); + + append(&path, &format!("{}\n", usage_row("c", 400))); + assert_eq!(scan(&path), (3, 700, 0)); + assert_eq!(cached_line_count(&path), Some(3)); + + // Unchanged file: replayed from the cache with the same result. + assert_eq!(scan(&path), (3, 700, 0)); +} + +#[test] +fn unterminated_last_line_is_counted_but_not_committed() { + let root = tempfile::tempdir().unwrap(); + let path = root.path().join("transcript.jsonl"); + std::fs::write( + &path, + format!("{}\n{}", usage_row("a", 1), usage_row("b", 2)), + ) + .unwrap(); + + assert_eq!(scan(&path), (2, 3, 0)); + assert_eq!(cached_line_count(&path), Some(1)); + + // The writer finishes the line and adds another: nothing counts twice. + append(&path, &format!("\n{}\n", usage_row("c", 4))); + assert_eq!(scan(&path), (3, 7, 0)); + assert_eq!(cached_line_count(&path), Some(3)); +} + +#[test] +fn rewritten_file_is_parsed_again() { + let root = tempfile::tempdir().unwrap(); + let path = root.path().join("transcript.jsonl"); + std::fs::write( + &path, + format!("{}\n{}\n", usage_row("a", 100), usage_row("b", 200)), + ) + .unwrap(); + assert_eq!(scan(&path), (2, 300, 0)); + + // Shorter replacement. + std::fs::write(&path, format!("{}\n", usage_row("x", 5))).unwrap(); + assert_eq!(scan(&path), (1, 5, 0)); + + // Longer replacement whose earlier bytes differ from the cached prefix. + std::fs::write( + &path, + format!( + "{}\n{}\n{}\n", + usage_row("y", 7), + usage_row("z", 11), + usage_row("w", 13) + ), + ) + .unwrap(); + assert_eq!(scan(&path), (3, 31, 0)); +} + +#[test] +fn malformed_lines_and_cross_scan_dedup_survive_the_cache() { + let root = tempfile::tempdir().unwrap(); + let path = root.path().join("transcript.jsonl"); + std::fs::write( + &path, + format!( + "{}\n{{broken\n{}\n", + usage_row("a", 100), + usage_row("a", 100) + ), + ) + .unwrap(); + + // The duplicate request counts once and the broken line stays malformed, + // on the cold scan and on the cached replay alike. + assert_eq!(scan(&path), (1, 100, 1)); + assert_eq!(scan(&path), (1, 100, 1)); + + // A shared `seen` set still de-duplicates across files on replay. + let other = root.path().join("copy.jsonl"); + std::fs::write(&other, format!("{}\n", usage_row("a", 100))).unwrap(); + let cutoff = Utc::now() - Duration::days(1); + let mut seen = HashSet::new(); + let mut pricing = ClaudeScanPricingResolver::default(); + let mut incomplete = ClaudeIncompleteTracker::default(); + let mut counted = 0; + for file in [&path, &other] { + counted += scan_claude_file_with_pricing( + file, + &cutoff, + &mut seen, + None, + &mut pricing, + &mut incomplete, + |_| {}, + ) + .counted; + } + assert_eq!(counted, 1); +} + +#[test] +fn repeated_content_block_lines_are_stored_compactly() { + let root = tempfile::tempdir().unwrap(); + let path = root.path().join("transcript.jsonl"); + let at = Utc::now() - Duration::hours(2); + // One assistant message split over three lines, as Claude Code writes it. + std::fs::write( + &path, + format!( + "{}\n{}\n{}\n", + usage_row_at("a", 100, at), + usage_row_at("a", 100, at + Duration::milliseconds(5)), + usage_row_at("a", 100, at + Duration::milliseconds(9)), + ), + ) + .unwrap(); + + assert_eq!(scan(&path), (1, 100, 0)); + assert_eq!(cached_line_count(&path), Some(3)); + assert_eq!(cached_repeat_count(&path), Some(2)); + assert_eq!(scan(&path), (1, 100, 0)); +} + +#[test] +fn repeat_after_the_cutoff_counts_when_its_anchor_is_before_it() { + let root = tempfile::tempdir().unwrap(); + let path = root.path().join("transcript.jsonl"); + let before = Utc::now() - Duration::days(2); + let after = Utc::now() - Duration::hours(1); + std::fs::write( + &path, + format!( + "{}\n{}\n", + usage_row_at("a", 100, before), + usage_row_at("a", 100, after), + ), + ) + .unwrap(); + + // The cutoff (one day ago) falls between the two lines, so the second one + // is the row a cold scan counts, with its own timestamp. + for _ in 0..2 { + let cutoff = Utc::now() - Duration::days(1); + let mut timestamps = Vec::new(); + let result = scan_claude_file_with_pricing( + &path, + &cutoff, + &mut HashSet::new(), + None, + &mut ClaudeScanPricingResolver::default(), + &mut ClaudeIncompleteTracker::default(), + |record| timestamps.push(record.timestamp), + ); + assert_eq!(result.counted, 1); + assert_eq!(timestamps.len(), 1); + let counted = timestamps[0].expect("timestamped row"); + assert!(counted > cutoff, "the in-window line is the one counted"); + } + assert_eq!(cached_repeat_count(&path), Some(1)); +} + +#[test] +fn repeat_with_different_usage_is_kept_as_its_own_row() { + let root = tempfile::tempdir().unwrap(); + let path = root.path().join("transcript.jsonl"); + std::fs::write( + &path, + format!("{}\n{}\n", usage_row("a", 100), usage_row("a", 150)), + ) + .unwrap(); + + // Same key, different usage: not compacted, and the first one still wins. + assert_eq!(scan(&path), (1, 100, 0)); + assert_eq!(cached_repeat_count(&path), Some(0)); +}