A demo for POWSM's four tasks

[sw005320]

Phone recognition has a command (espnet phonemize), an MCP tool and — with espnet#6792 — a Space. This is its notebook, and the one place where all four of POWSM's tasks sit side by side on the same six seconds of speech:

task	you give	it answers
<asr>	the audio	the words
<pr>	the audio	the phones, in IPA
<g2p>	the audio and the words	the phones
<p2g>	the audio and the phones	the words

The last two are why the notebook is worth having: they are the part of POWSM that no one-argument command can show, because they take something written alongside the audio. The notebook hands <g2p> the words that were said and gets phones for those words, then hands <p2g> those phones back and gets words again.

<asr> is in there for contrast, and it is visibly worse than a model built for text — which the notebook says, rather than leaving a reader to conclude the model is broken.

Two things it reads off the checkpoint rather than hard-coding, since a reader will copy this into their own script: the 20-second window (OWSM's is 30) and s2t.no_language(), which is <unk> here and <nolang> for OWSM.

Name. s2t_pr_demo.ipynb, per the rule in Demos/README.md that tools/check_layout.py enforces: <task>_<variant>_demo.ipynb, and phone recognition is a variant of s2t rather than a task of its own.

Verified with tools/run_notebook.py against espnet master: RESULT ok, 16 cells.

CI skips until the release. The notebook pins espnet==202610.post2, which carries the page and the decode_window that passes a prompt on; the job says which release it is waiting for, as #61 taught it to.

🤖 Generated with Claude Code

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[sw005320]

Made switchable, per the CLI/MCP-on-CTC, notebook/Space-switchable split:

MODEL = "espnet/powsm_ctc"  # or "espnet/powsm", the encoder-decoder

and nothing else to change. What makes the swap actually work is that the notebook now decodes through decode_window rather than best_path: a CTC-only checkpoint is read off its CTC head, one with a decoder is searched, and the notebook does not have to know which it is holding. With best_path a reader who swapped the tag would have got the encoder-decoder's CTC branch, which answers what that branch was trained on — phones, whatever task you ask for.

The note beside it says what the trade is, measured on this file rather than remembered: the encoder-decoder's phones are finer (pʰ, tʰ, oʊ where the CTC writes p, t, o) and it is about four times slower on a CPU. The fuller table, including what happens to its text tasks and the prompt experiment, is in espnet#6792.

Re-ran after the change: RESULT ok, 16 cells.

[chinjouli]

the phone vocab are the same for both models, not "finer". And the decoder also writes "o" and "ʊ" separately too, the model is designed to split diphthongs.

[sw005320]

Corrected, and thank you — I had inferred an inventory difference from an output difference, which was wrong.

The section now says: "Both write the same phone set — a diphthong is two symbols in either, by design — so what differs between them is which phones they choose, not what they can say."

I had made the same claim in the POWSM Space card (espnet#6792); that is fixed too, and the row now reports what I actually observed on that file — ð ə s e ɪ l ʌ v from the decoder against d ə s e ɪ l ɔ v from the CTC head — with no explanation attached to it.

[chinjouli]

Worse compared to high resource languages like English, but competitive for languages that are less well presented in general datasets.

[sw005320]

Taken — "worse at transcription" without saying where is the kind of sentence that sends a reader away with the wrong impression. The cell now reads:

A phonetic model will transcribe. On English — where a model trained for text has seen far more — it is the weaker choice, and what comes back below shows it. On a language that general speech corpora barely cover, the comparison is a different one: POWSM is competitive there, because it was trained on a phone-annotated collection rather than on whatever happens to be plentiful.

If that last clause overstates the reason, tell me and I will cut it back to the claim alone — I am describing why from the outside.

[chinjouli]

"whatever it thought it heard" is too much. The text input here grounds the phonemic output, but it still listens to the audio.

[sw005320]

Fixed. "Whatever it thought it heard" made the audio sound discarded, which is not what the task does. Now:

Now the audio is not the only input. The words you hand it ground the answer — the phones come out for those words — and it is still listening to the audio, which is where the pronunciation comes from.

That also makes the demo's own result read correctly: the phones that come back for a given text are that text as this speaker said it, which is the interesting part and what I had talked the reader out of.