Skip to content

EVAL.md

Latest commit

 

History

History
62 lines (49 loc) · 2.75 KB

EVAL.md

File metadata and controls

62 lines (49 loc) · 2.75 KB
Raw

Hinglish / Indian evaluation set (v0.1 plan)

This is a manually curated evaluation protocol for measuring OpenEar quality on Indian English, Hindi, and Hindi-English code switching. Phrases below are meant to be recorded by consenting volunteers during benchmarking; none of them are baked into app behavior.

Method

For each phrase: record in both a quiet room and a moderately noisy room, using at least two speakers with different English/Hindi accents and speaking rates. Measure word error rate (WER) against the reference transcripts using a standard WER script against the whisper.cpp transcription output, and additionally manually review:

  1. Code-switching correctness (English words inside Hindi sentences, and vice versa).
  2. Script choice: Roman Hinglish vs Devanagari. Document which the model picks per sentence; do not promise either.
  3. Proper nouns ("WhatsApp", "office", names) — high-error categories.
  4. Punctuation and meaning preservation (no paraphrase).

The same protocol compares tiny, base, small, and — separately — any quantized variants, noting transcription latency (seconds of audio → finish time), peak RSS, model load time, and thermal effects after 5 continuous minutes of dictation.

Roman Hinglish phrases

  1. "Kal meeting kitne baje start hogi?"
  2. "Please mujhe woh document WhatsApp par bhej dena."
  3. "Aaj office thoda late pahunchunga."
  4. "Is file ko download karke email kar do."
  5. "Mera phone charge khatam ho gaya hai, charger de dena."
  6. "Team ne aaj demo clear kar diya, badhiya kaam kiya."
  7. "Chai chahiye? Sirf do minute mein aa raha hoon."
  8. "Uska naam spelling kaise hota hai? A-A-R-A-V."
  9. "Payment pending hai kya? Kal process kara deta hoon."
  10. "Weekend pe ghar jaa raha hoon, Monday tak wapas."

Devanagari Hindi phrases (with English words embedded)

  1. "कल मीटिंग कितने बजे है?"
  2. "मुझे वह document WhatsApp पर भेज देना।"
  3. "आज office थोड़ा late पहुँचूँगा।"
  4. "यह file download करके email कर दो।"
  5. "सुबह दस बजे मिलते हैं, ठीक है?"

Indian English phrases

  1. "Kindly do the needful and revert by this evening."
  2. "My meeting got preponed to eleven o'clock."
  3. "Where is the charger? My phone has died."
  4. "The cab is waiting downstairs, please come fast."
  5. "I will join the call from home tomorrow."

Acceptance for choosing the default model

The shipped default model (base) stays the default unless benchmarked results on two low-end and one mid-range Android device show tiny matches accuracy within measurement error, in which case tiny becomes default for low-RAM devices via the "low resource mode" setting.