od: preserve non-UTF-8 file operands by ozenyunus35 · Pull Request #15242 · uutils/coreutils

ozenyunus35 · 2026-10-11T09:40:54Z

Summary

od currently drops command-line arguments that are not valid UTF-8. When such an argument is a filename, od behaves as if no file was provided and falls back to reading standard input.

This can cause od to wait for input unexpectedly or read unrelated data instead of reporting a missing file.

Changes

Preserve file operands as OsString throughout argument parsing and file handling.
Pass the original arguments to clap instead of silently discarding non-UTF-8 values.
Keep filenames intact when passing them to File::open.
Only interpret operands as offsets or labels when they contain valid UTF-8.
Preserve existing behavior for normal filenames and standard input.

Testing

Added two regression tests:

A missing non-UTF-8 filename must produce a file error instead of reading stdin.
On Linux, an existing file with a non-UTF-8 filename must be read correctly.

The first test fails before the fix and passes afterward.

Validation performed on macOS:

test_od: 75 passed.
uu_od unit tests: 55 passed.
cargo fmt --check: passed.
Clippy with -D warnings: passed.
CSpell: passed.

The Linux-specific filesystem test could not be executed on macOS and will be exercised by Linux CI.

Compatibility

The change is limited to od argument and file handling. Existing behavior for valid UTF-8 filenames, standard input, and offsets is preserved.

AI Assistance

Claude Code was used to assist with implementation and testing.

od collected its arguments with `collect_ignore()`, which silently drops every argument that is not valid UTF-8. Such a file operand was lost, so `od $'\xff'` read standard input (and hung on a terminal) instead of dumping the file or reporting that it does not exist. Pass the raw arguments to clap and keep file operands as `OsString` all the way to `File::open`. Offsets and labels are still parsed from operands that are valid UTF-8. Fixes uutils#12794

RenjiSann · 2026-10-11T11:47:40Z

+
+    let (at, mut ucmd) = at_and_ucmd!();
+    let name = OsStr::from_bytes(b"\xff");
+    std::fs::write(at.plus(name), "abcdefghijklmnopqrstuvwxyz\n").unwrap();


Better to user at.write(file, content) here.

Unless I'm missing something that doesn't work because name is an OsStr whereas at.write expects a &str.

Ah yes indeed. Maybe we could update at.write to take and AsRef<Path> ?

Created #15287 to address this

Thanks for opening #15287! Once it lands I'm happy to switch these tests over to at.write, either here or in a small follow-up.

RenjiSann · 2026-10-11T11:53:26Z

The tests you added are broken in web-assembly, hence the CI fails.

I suggest setting the tests' cfgs to

#[cfg(not(any(target_os = "wasi", windows)))]

wasmtime cannot pass an argument that is not valid UTF-8 to the utility, so these tests cannot run there. Mark them like the other tests that depend on non-UTF-8 arguments.

ozenyunus35 · 2026-10-11T13:04:45Z

Thanks for the review! I updated the tests in 36237d8.

I used the repository's existing wasi_runner convention because these tests are compiled for the host and execute the utility through Wasmtime. The WASI jobs are now passing.

Regarding at.write, the filename is an OsStr and may contain invalid UTF-8, whereas at.write expects a &str, as cakebaker pointed out.

sylvestre · 2026-10-11T21:03:35Z

                                show_error!(
                                    "{}: {}",
-                                    self.file_name.unwrap_or("I/O"),
+                                    self.file_name.map_or("I/O".into(), OsStr::to_string_lossy),


could use maybe_quote() here like the open error above, no?

sylvestre · 2026-10-11T21:03:44Z

+        .pipe_in("not read")
+        .succeeds()
+        .stdout_only(unindent(ALPHA_OUT));
+}


could you please also add a test with an offset, e.g. od --traditional <non-utf8> 10?

… offset

ozenyunus35 · 2026-10-12T13:32:40Z

@sylvestre done in 4482a3a: the read error now uses maybe_quote() like the open error, and I added a test with an offset (od --traditional -c <non-utf8> 10). It's Linux-only like the other non-UTF-8 test.

codspeed · 2026-10-12T16:33:42Z

Merging this PR will degrade performance by 3.83%

⚠️

121 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

sort_reverse_utf8_locale

du_wide_tree[(5000, 500)]

cksum_default

cksum_multiple_files

od_chars

od_hex_bytes

factor_64bit_semiprime

od_default

cksum_raw_output

cksum_crc

…and 111 more

ℹ️ 1 benchmark has no flame graph

The flamegraph could not be generated because too much time was spent in runtime internals, making the visualization not useful for performance analysis.

sort_dictionary_order[500000]

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 398 untouched benchmarks
⏩ 54 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`five_38_bit_primes`	1.7 s	2.4 s	-31.25%
❌ ⚠️	Simulation	`sort_reverse_utf8_locale`	79.2 ms	81.9 ms	-3.28%
⚡	Simulation	`three_39_bit_primes`	490.4 ms	366.6 ms	+33.79%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ozenyunus35:fix/od-non-utf8-operands (4482a3a) with main (efca981)²}

54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on main (d43f2b8) during the generation of this report, so efca981 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

RenjiSann reviewed Oct 11, 2026

View reviewed changes

tests/od: skip the non-UTF-8 operand tests under the WASI runner

36237d8

wasmtime cannot pass an argument that is not valid UTF-8 to the utility, so these tests cannot run there. Mark them like the other tests that depend on non-UTF-8 arguments.

sylvestre reviewed Oct 11, 2026

View reviewed changes

od: quote the file name in read errors, test a non-UTF-8 file with an…

4482a3a

… offset

RenjiSann mentioned this pull request Oct 12, 2026

MakeAtPath::write to take an AsRef<Path> #15287

Open

Navigation Menu

Repository navigation

od: preserve non-UTF-8 file operands - #15242

od: preserve non-UTF-8 file operands#15242
ozenyunus35 wants to merge 3 commits into
uutils:mainfrom
ozenyunus35:fix/od-non-utf8-operands

ozenyunus35 commented Oct 11, 2026 •

edited

Loading

Uh oh!

RenjiSann Oct 11, 2026

Uh oh!

cakebaker Oct 11, 2026

Uh oh!

RenjiSann Oct 12, 2026

Uh oh!

RenjiSann Oct 12, 2026

Uh oh!

ozenyunus35 Oct 13, 2026

Uh oh!

RenjiSann commented Oct 11, 2026 •

edited

Loading

Uh oh!

ozenyunus35 commented Oct 11, 2026

Uh oh!

sylvestre Oct 11, 2026

Uh oh!

sylvestre Oct 11, 2026

Uh oh!

ozenyunus35 commented Oct 12, 2026

Uh oh!

codspeed Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ozenyunus35 commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

Compatibility

AI Assistance

Uh oh!

RenjiSann Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

cakebaker Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

RenjiSann Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

RenjiSann Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

ozenyunus35 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

RenjiSann commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ozenyunus35 commented Oct 11, 2026

Uh oh!

sylvestre Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

sylvestre Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

ozenyunus35 commented Oct 12, 2026

Uh oh!

codspeed Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 3.83%

Performance Changes

Footnotes

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ozenyunus35 commented Oct 11, 2026 •

edited

Loading

RenjiSann commented Oct 11, 2026 •

edited

Loading

codspeed Bot commented Oct 12, 2026 •

edited

Loading