Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions docs/remote-input-recording-recovery.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# 远程输入:息屏与录音恢复

手机录音页新增「录音时保持亮屏」,默认开启,并在当前浏览器记住选择。只在前台录音期间申请屏幕唤醒锁,结束、取消、断线或离开页面时释放。浏览器不支持,或系统因省电等原因拒绝时,页面会提示,录音仍可使用。

```text
[ 点击开始 / 点击结束 ]
录音状态

录音时保持亮屏 [ 开 ]
息屏会结束本段录音,电脑继续处理已收到的部分。

电脑落字 [ 开 ]
```

## 息屏后会发生什么

- 自动息屏、手动锁屏、切到后台或麦克风被系统中断时,手机尽可能发送结束录音,电脑继续识别已经收到的音频。
- 如果手机来不及通知,电脑在断线时收尾;连接仍在但连续 15 秒没有有效音频时,也会结束采集并开始识别。
- 识别已经开始后,手机断线不会取消电脑上的识别与历史记录写入。主动取消、关闭远程输入、重置 PIN 仍会撤销会话。
- 回到页面后重新认证并查询上次会话,找回结果或提示到电脑历史中重试。不会自动重新打开麦克风。

这不能让浏览器在锁屏后持续采集,也不能补回尚未传到电脑的音频。电脑退出、崩溃或磁盘不可写时,不能保证自动完成识别。

## 录音与结果保留

已收到的远程音频使用现有录音目录和保留设置写入 WAV。识别失败时,音频可供现有自动重试及桌面历史重新转录使用;识别成功后,除非开启保留成功录音,否则按现有策略删除音频。文字仍按历史记录设置保留。

手机只保存上次会话 ID 和单独生成的随机恢复凭据。恢复查询要求有效配对,不提供历史列表;仅凭公开会话 ID 无法读取结果。服务内最多保留最近 64 个会话的恢复权限,最长 24 小时,取消会话、重启服务或重置 PIN 后失效。此后请在电脑历史记录中查看。

## 真机验收(待补)

记录手机型号、系统与浏览器版本、电脑平台、识别模式及测试 commit;分别检查 iPhone Safari 和 Android Chrome。

- 开关默认开启;关闭、刷新后仍关闭;录音中切换立即生效,结束后恢复系统自动息屏。
- 点按和按住模式各录音 1–2 分钟:允许自动息屏、手动锁屏、切后台后,电脑保留已收到的部分并完成识别;回到手机能看到结果。
- 识别过程中断开网络再重连;检查电脑历史以及手机找回的文字。
- 模拟识别失败,检查 WAV 保留与历史重新转录;检查保留天数和录音数量上限。
- 低电量或浏览器拒绝保持亮屏时,提示明确且能开始录音。
- 识别中主动取消或重置 PIN,确认没有迟到落字,旧恢复凭据失效。

自动化测试覆盖模拟浏览器事件、两分钟 PCM、归档与失败重试、恢复鉴权和权限撤销;这些检查不能代替手机锁屏与省电行为的真机验收。
52 changes: 52 additions & 0 deletions openless-all/app/crates/openless-core/src/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9346,6 +9346,58 @@ mod tests {
let _ = std::fs::remove_dir_all(data_dir);
}

#[tokio::test]
async fn external_audio_saves_failed_recordings_for_history_retry_and_prunes_successful_audio()
{
for fail in [false, true] {
let data_dir = std::env::temp_dir()
.join(format!("openless-remote-history-{}", uuid::Uuid::new_v4()));
let transcription = if fail {
crate::testing::FixtureTranscriptionEngine::failing(BackendError::new(
BackendErrorCode::Provider,
"fixture ASR failure",
))
} else {
crate::testing::FixtureTranscriptionEngine::successful("received speech", 120_000)
};
let engine = crate::PipelineDictationEngine::new(
Arc::new(crate::ExternalAudioRecorder::with_recordings_directory(
data_dir.join("recordings"),
)),
Arc::new(transcription.clone()),
Arc::new(crate::testing::FixtureTextPolisher::successful(
"complete transcription",
)),
);
let backend = backend_with_dictation_engine(data_dir.clone(), Arc::new(engine));
backend.start().await.unwrap();
let session = backend.start_external_dictation().await.unwrap();
let path = data_dir.join("recordings").join(format!("{session}.wav"));
for second in 0..120 {
backend
.feed_external_pcm(session, &vec![second; 32_000])
.unwrap();
}
let expected = transcription.pcm();
assert_eq!(&std::fs::read(&path).unwrap()[44..], expected);
let result = backend.stop_dictation_session(session).await;
assert_eq!(result.is_err(), fail);
let history = backend.list_history().unwrap();
assert_eq!(history.len(), 1);
assert_eq!(history[0].id, session.to_string());
assert_eq!(history[0].has_audio_recording, Some(fail));
if fail {
assert_eq!(history[0].error_code.as_deref(), Some("transcribeFailed"));
assert_eq!(&std::fs::read(path).unwrap()[44..], expected);
} else {
assert_eq!(history[0].final_text, "complete transcription");
assert!(!path.exists());
}
backend.shutdown().await.unwrap();
let _ = std::fs::remove_dir_all(data_dir);
}
}

#[tokio::test]
async fn dictation_freezes_channel_identity_protocol_and_model_for_the_session() {
let data_dir = std::env::temp_dir().join(format!(
Expand Down
39 changes: 39 additions & 0 deletions openless-all/app/crates/openless-core/src/domains.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1059,6 +1059,27 @@ pub trait RemoteInputRuntimeAdapter: Send + Sync {
&self,
session_id: SessionId,
) -> BoxFuture<'static, Result<(), BackendError>>;
/// 只读取指定会话;由 Core 校验手机持有的恢复凭据。
fn read_audio_history(
&self,
_session_id: SessionId,
) -> BoxFuture<'static, Result<Option<crate::types::DictationSession>, BackendError>> {
Box::pin(async { Ok(None) })
}
}

#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "camelCase")]
pub enum RemoteInputRecovery {
Pending,
Completed {
text: String,
},
Failed {
#[serde(rename = "hasAudioRecording")]
has_audio_recording: bool,
},
Unavailable,
}

pub trait RemoteInputApi: Send + Sync {
Expand Down Expand Up @@ -1094,6 +1115,24 @@ pub trait RemoteInputApi: Send + Sync {
) -> BoxFuture<'static, Result<(), BackendError>> {
unsupported("remote input")
}
fn recover_stream(
&self,
_connection_id: SessionId,
_session_id: SessionId,
_recovery_key: crate::credentials::SecretValue,
) -> BoxFuture<'static, Result<RemoteInputRecovery, BackendError>> {
unsupported("remote input")
}
fn recovery_key(
&self,
_connection_id: SessionId,
_session_id: SessionId,
) -> Result<crate::credentials::SecretValue, BackendError> {
Err(BackendError::new(
BackendErrorCode::Unsupported,
"remote recovery is unavailable",
))
}
fn start_stream(
&self,
_connection_id: SessionId,
Expand Down
Loading
Loading