Fix left-padded BOS removal along the last tensor axis

[AaravMehta-07]

Description

Use the final tensor dimension when removing the BOS token from left-padded sequences. get_tokens_with_bos_removed documents sequence length as the last dimension and already uses that dimension to locate BOS and shape the result, but its scatter operation is hard-coded to dimension 1. Rank-1 inputs raise a dimension error; rank-3 inputs scatter along the wrong axis or fail for valid leading padding.

The fix changes only the scatter dimension to -1. Eight synthetic value-based cases cover rank-1/2/3 inputs, BOS equal to PAD, a BOS-only sequence, right-padding and no-BOS controls. Existing tokenizer tests remain unchanged. No dependencies or public signatures change.

Type of Change

• Bug fix

Validation

• Before the fix, the new parameterized test reports 5 failed, 3 passed with Jaxtyping runtime checks enabled.
• After the fix, the full affected test file reports 10 passed, independently rerun by the orchestrating reviewer. Existing GPT-2 tokenizer cases use cached tokenizer assets only, with Hub/network access disabled; no model weights are downloaded.
• make format was executed through isolated uv run environments using the lockfile formatter versions. The isolated tools do not synchronize the complete project environment.
• An independent make check-format run with those versions passes (Pycln, isort, Black). As with the normal lockfile, this tool environment does not include Black's optional notebook-formatting extras. An additional check with those extras exposed unrelated notebook formatting/parser problems; none of those generated edits are included here.
• Full mypy . passes with 391 source files checked, using mypy 1.17.0 and the isolated locked Transformers 5.13.0 / Tokenizers 0.22.1 / Hub 1.15.0 / Datasets 4.8.5 prefix. The affected file's ten tests also pass with that prefix in an independent root run. No shared dependency installation was changed.
• Full project tests and model/GPU validation were not run. The affected-file test command overrides unavailable doctest/notebook plugin options while retaining Jaxtyping checks.

Checklist

• Added tests that reproduce the defect and verify the fix.
• New and existing tests in the affected module pass locally.
• Existing interface tests were not rewritten.
• Documentation already specifies the last-dimension contract; no documentation change is needed.

Assistance

AI-assisted implementation, tests, and description, with independent orchestrator code review and test execution.