Skip to content

Let the ASR and ST demos run on either OWSM v4 #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sw005320 merged 1 commit into from
Sep 22, 2026
Merged

Let the ASR and ST demos run on either OWSM v4 #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 16 additions & 5 deletions Demos/asr_demo.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,14 @@
"source": [
"## Transcribe it\n",
"\n",
"There are two OWSM v4 checkpoints, and this notebook runs on either: change\n",
"`MODEL` and nothing else.\n",
"[`owsm_ctc_v4_1B`](https://huggingface.co/espnet/owsm_ctc_v4_1B) is\n",
"encoder-only: `decode_long` reads a recording of any length, in overlapping\n",
"windows, and decodes each on the CTC head with no beam search.\n"
"encoder-only — `decode_long` reads a recording of any length, in overlapping\n",
"windows, and decodes each on the CTC head with no beam search — and\n",
"[`owsm_v4_medium_1B`](https://huggingface.co/espnet/owsm_v4_medium_1B) is the\n",
"encoder-decoder, which searches, is slower, and is the one that can be given\n",
"a text prompt."
]
},
{
Expand All @@ -72,10 +77,11 @@
"source": [
"from espnet2.bin.s2t_inference import Speech2Text\n",
"\n",
"s2t = Speech2Text.from_pretrained(\"espnet/owsm_ctc_v4_1B\", device=\"cpu\")\n",
"MODEL = \"espnet/owsm_ctc_v4_1B\" # or \"espnet/owsm_v4_medium_1B\"\n",
"s2t = Speech2Text.from_pretrained(MODEL, device=\"cpu\")\n",
"\n",
"segments = s2t.decode_long(\"sample.wav\", lang_sym=\"<eng>\", task_sym=\"<asr>\")\n",
"print(\" \".join(text for _, _, text in segments))\n"
"print(\" \".join(text for _, _, text in segments))"
]
},
{
Expand All @@ -85,7 +91,12 @@
"## Let it work out the language\n",
"\n",
"`<nolang>` asks the model to identify the language instead of being told it.\n",
"The answer comes back as the first symbol of the decoded text.\n"
"The answer comes back as the first symbol of the decoded text.\n",
"\n",
"`best_path` is CTC decoding with no search. On the encoder-only checkpoint\n",
"that is the whole model; on the encoder-decoder it reads the CTC *branch*,\n",
"which answers what that branch was trained on rather than what the task\n",
"symbol asks for — call `s2t(...)` there instead, and pay for the search."
]
},
{
Expand Down
11 changes: 8 additions & 3 deletions Demos/st_demo.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,11 @@
"\n",
"The task symbol carries the target: `<asr>` transcribes, `<st_deu>`\n",
"translates to German, and so on. The model's token list is where the\n",
"available pairs are written, so ask it rather than a table.\n"
"available pairs are written, so ask it rather than a table.\n",
"\n",
"Either OWSM v4 checkpoint does this, and `MODEL` is the only line to change:\n",
"the encoder-only one below, or `espnet/owsm_v4_medium_1B`, which searches and\n",
"is slower."
]
},
{
Expand All @@ -70,10 +74,11 @@
"source": [
"from espnet2.bin.s2t_inference import Speech2Text\n",
"\n",
"s2t = Speech2Text.from_pretrained(\"espnet/owsm_ctc_v4_1B\", device=\"cpu\")\n",
"MODEL = \"espnet/owsm_ctc_v4_1B\" # or \"espnet/owsm_v4_medium_1B\"\n",
"s2t = Speech2Text.from_pretrained(MODEL, device=\"cpu\")\n",
"\n",
"targets = [t for t in s2t.s2t_model.token_list if t.startswith(\"<st_\")]\n",
"print(f\"{len(targets)} translation targets, e.g. {targets[:6]}\")\n"
"print(f\"{len(targets)} translation targets, e.g. {targets[:6]}\")"
]
},
{
Expand Down
Loading