Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
df90191
collaboration
alan16742 Nov 9, 2026
be88b03
history
alan16742 Nov 10, 2026
7fb1b6c
feat(shared): extract shared crypto, path, secret, and device utilities
alan16742 Nov 11, 2026
4418ff2
refactor(dropbox): use shared sha256 digest utility
alan16742 Nov 11, 2026
0e0eb17
refactor(s3): use shared sha256 text and digest utilities
alan16742 Nov 11, 2026
2f037c0
refactor(encryption): use shared sha256 digest utility
alan16742 Nov 11, 2026
3388763
feat(collaboration): extract shared helpers and unify status badge co…
alan16742 Nov 11, 2026
3436aa7
feat(history): batch FastCDC chunk hashing, add inline diff and recov…
alan16742 Nov 11, 2026
1b7a19e
feat(core): implement phased task executor, byte budget pool, and asy…
alan16742 Nov 11, 2026
e5c3b0e
feat(shared): add local device id utility and mock local storage on App
alan16742 Nov 11, 2026
24638c3
feat(history): decouple deviceId from data.json into local device sto…
alan16742 Nov 11, 2026
7b54ba0
feat(collaboration): support fastMode fallback and honor confirmDelet…
alan16742 Nov 11, 2026
26bf923
feat(core): register collaboration trigger with fastMode and confirmD…
alan16742 Nov 11, 2026
58a9521
refactor(history): move deviceid and gctime from data.json to indexedDB
alan16742 Nov 12, 2026
14717e2
fix(history,plugin): handle Android OkHttp stream errors and reduce r…
alan16742 Nov 12, 2026
ab4bac2
feat(history): add binary attachment history protection
alan16742 Nov 12, 2026
3302aae
refactor(history): move history storage outside vault directory
alan16742 Nov 13, 2026
cd3e91f
feat(history): tiered thinning Retention
alan16742 Nov 14, 2026
902c581
refactor: extract timing and configuration constants into dedicated m…
alan16742 Nov 15, 2026
032985e
feat(collaboration, history): negotiate lease upload, takeover timeou…
alan16742 Nov 15, 2026
5588d56
fix(history): use remote UID to skip unchanged head downloads
alan16742 Nov 15, 2026
bab741a
feat(collaboration): reconcile committed peer files and fix settings …
alan16742 Nov 15, 2026
d8551ed
feat: enforce strict checks for unused imports, local variables, and …
alan16742 Nov 15, 2026
a450853
fix(collaboration): download peer files instead of recording UID with…
alan16742 Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .oxlintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,11 @@
],
"unicorn/prefer-global-this": "off",
"eslint/no-new": "off",
"eslint/no-unused-vars": [
"error",
{ "argsIgnorePattern": "^_", "varsIgnorePattern": "^_", "caughtErrorsIgnorePattern": "^_" }
],
"eslint/no-unused-private-class-members": "error",
"eslint/sort-imports": "off",
"eslint/no-duplicate-imports": ["warn", { "allowSeparateTypeImports": true }],
"eslint/new-cap": "off",
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ Sync Engine is a vault syncing plugin to **synchronize vault files in multiple d
**Features**:

- [Client-side encryption](https://sync.consensia.cc/deep-dive/modules/encryption)
- [Realtime collaboration](https://sync.consensia.cc/deep-dive/modules/collaboration) over encrypted MQTT notifications
- Granular sync strategy configuration (bidirectional / mirror remote / mirror local / don't sync) using Glob rules.
- Startup / periodic / sync-on-focus / save-on-change syncing.
- Conflict resolve strategies ([smart merge](https://sync.consensia.cc/deep-dive/modules/smart-merge) / keep both / latest survive / keep remote / keep local / skip).
Expand Down
1 change: 1 addition & 0 deletions README.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ Sync Engine 是一款用于**多设备间同步 Vault 笔记文件**的插件,
**核心特性**:

- [客户端端到端加密](https://sync.consensia.cc/deep-dive/modules/encryption)
- [实时协作](https://sync.consensia.cc/deep-dive/modules/collaboration),通过加密的 MQTT 通知同步
- 使用 Glob 规则进行精细化的同步策略配置(双向同步 / 镜像远程 / 镜像本地 / 不同步)。
- 支持启动时同步、定时同步、焦点同步以及变更时同步。
- 灵活的冲突解决方案([智能合并](https://sync.consensia.cc/deep-dive/modules/smart-merge) / 保留两者 / 保留最新版 / 覆盖为远端 / 覆盖为本地 / 直接跳过)。
Expand Down
132 changes: 114 additions & 18 deletions bun.lock

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,11 @@ const localeConfig = configGenerator<ThemeConfig>((t) => {
link: `${deepDive}/modules/rclone-s3-timestamp`,
text: t('rcloneS3Timestamp'),
},
{
link: `${deepDive}/modules/collaboration`,
text: t('collaboration'),
},
{ link: `${deepDive}/modules/history`, text: t('history') },
],
text: t('modules'),
},
Expand Down
2 changes: 2 additions & 0 deletions docs/.vitepress/i18n.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ const en = {
benchmarking: 'Benchmarking',
claims: 'Claims',
code: 'en-US',
collaboration: 'Collaboration',
contributing: 'Contributing',
copyright: 'Copyright',
debugAndTesting: 'Debug & Testing',
Expand All @@ -27,6 +28,7 @@ const en = {
fileTree: 'File Tree',
folder: '',
gdrive: 'Google Drive',
history: 'History Versions',
home: 'Home',
licenseMessage:
'All content licensed under the <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a> License.',
Expand Down
73 changes: 73 additions & 0 deletions docs/src/pages/en/deep-dive/modules/collaboration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Collaboration Module

Collaboration is an optional realtime notification layer. It does not replace the configured remote backend: after a sync changes remote state, a device publishes a small encrypted _pulse_ over MQTT. The pulse lists only the file keys that changed in that sync, so peers can refresh just those keys against the backend instead of walking the whole vault on every notification.

The module registers a remote file system wrapper, a sync trigger, settings, and a conflict resolver that routes concurrently overwritten files into the configured merge strategy. The pulse carries no file content, UIDs, sizes, or rules — only the changed keys, inside the encrypted payload. The vault's full structure never travels over the broker.

## Room Code and Keys

A room code is generated once and stored in Obsidian's secret storage. It has the form `sec1.<room-id>.<secret>`, where the room ID is 16 random bytes and the secret is 32 random bytes, both encoded as base64url.

Every client derives the same keys from the room code:

- `PBKDF2-SHA-256` with 210,000 iterations derives a master secret from the room secret.
- The salt is `SHA-256("sync-engine-collaboration/v1:<room-id>")`.
- `HKDF-SHA-256` derives a 32-byte encryption key and a 32-byte topic key from the master secret.
- The MQTT topic is `sync-engine/collaboration/v1/<base64url(HMAC-SHA-256(topic-key, room-id))>`.

The room ID is therefore public inside the room code, while the topic key and encryption key are not. A client that knows the topic cannot decrypt messages without the room secret.

## Message Flow

Each client tracks a `dirty` flag and a set of changed keys per remote backend, both set whenever a local sync writes, moves, deletes, or creates something on the remote backend. A publish happens only when that flag is set, the transport is connected, and a remote UID is known. The flag and the key set are cleared after the pulse is sent, guarded by a revision counter so a write made during publishing is not lost.

The encrypted payload contains a timestamp and, when it fits, the changed keys. The key set is bounded by both a maximum count and a byte budget (the ciphertext must stay under 4096 bytes). A batch that exceeds either bound degrades to a keyless pulse, which asks peers to do a full remote walk instead. All framing metadata lives in the MQTT 5 PUBLISH variable header as User Properties — `v`, `client_id`, `batch_id`, and `sent_at` — so the PUBLISH payload carries only `nonce` and `ciphertext`, and the changed keys stay inside the encrypted bytes. Brokers can identify the sender without touching the encrypted bytes. Credentials for authenticated brokers are read from Obsidian's secret storage at connect time.

A pulse is a single small message sealed with `AES-GCM-256`. The additional authenticated data binds the topic, batch ID, client ID, and timestamp; because the User Properties feed that AAD, tampering with them makes the message fail to decrypt. There is no compression, chunking, or reassembly.

Messages outside a two-minute clock window are rejected. Duplicate `batchId` values are ignored, and the receiver drops pulses whose `client_id` matches its own. Message size depends on the size of the change batch, not on the total number of files in the vault.

## Applying a Pulse

When a pulse arrives, the sync bridge compares its timestamp and batch ID with any pending pulse and keeps the newer one. It requests a collaboration-triggered sync.

The sync core needs a _complete_ remote listing: a key missing from it is treated as a remote deletion and would remove the local file. So the trigger supplies a `remoteLister` built from the cached remote snapshot that the core's context wrapper already maintains in memory. Starting from that complete snapshot, it re-stats only the keys named in the pulse against the real backend and patches the snapshot, then returns the full array. This keeps the network cost proportional to the number of changed keys while the deciders still see the whole remote state.

A full remote `list('/')` is still used when the local snapshot has never been populated (for example just after startup) or when the receiver gets a keyless pulse. That walk also initializes file system wrappers such as [asymmetric storage](../asymmetric-storage) before any download and refreshes the snapshot. Scheduled and startup syncs do the same, so a peer that was offline and missed several pulses heals its snapshot through the next full walk rather than relying on a continuous pulse sequence. Because both peers read the same backend, no stat list needs to be shipped.

## Race Detection

A concurrent upload can overwrite a file this device just uploaded. The module does not try to prevent that race; it detects it and lets the configured conflict resolver merge the divergent versions. Every edit is already captured durably by the [history versions](./history) module, so no in-memory ledger or replay is needed.

The module journals recent uploads. During a collaboration-triggered sync, the wrapped decider checks the real remote listing: a changed UID for a recently uploaded key within the race window forces a conflict. That forced record routes the key into the configured conflict resolver, which merges both sides instead of silently dropping one. Rules whose strategy is `none` or already wrapped by the module are left unchanged.

## Settings

| Setting | Behavior |
| --- | --- |
| Enable collaboration | Connects to the broker and starts publishing and accepting pulses. |
| Broker URL | MQTT WebSocket URL. The default is the public EMQX broker. |
| Broker username | Optional username for authenticated brokers, stored in plain text. |
| Broker password | Optional. Selects a secret from Obsidian's secret storage for brokers that require authentication. |
| Client ID prefix | Prefilled with the Obsidian device type on first run and editable. A random suffix is added for each session so copied settings do not collide. |
| Room code | Selects a secret from Obsidian's secret storage. Generate and Copy actions are provided. |
| Topic | Copies the room topic to the clipboard with the button; no topic text is shown. |

The status row reports the current transport state and a broker error when one is available. Reconnect applies the current settings immediately.

## Security and Privacy

The broker sees the topic, packet timing, encrypted packet sizes, and the plaintext User Properties (`client_id`, `batch_id`, `sent_at`, `v`). It cannot read the changed-key names: they sit inside the `AES-GCM-256` ciphertext, not in the User Properties, and it sees no UIDs, sizes, rules, or file content. A holder of the room secret can decrypt a captured pulse and learn the names of the files changed in that batch, but never the vault's full structure or any file content. This is the deliberate trade-off against a keyless design: the broker itself learns nothing about files, while a leaked room code exposes the changed names of the pulses an attacker managed to capture, rather than only packet timing. The topic is derived from the room secret, so an observer cannot discover it from the room ID alone.

The room secret is the only credential. Anyone with the room code can join, decrypt pulses, and publish them. The module does not provide member management, key rotation, revocation, forward secrecy, or protection against a member replaying an old pulse within the freshness window.

The public default broker is suitable for testing and for metadata that users accept exposing to the broker. Users who need a private relay should configure their own MQTT WebSocket endpoint.

## Limits and Compatibility

- MQTT 5 over WebSocket is required.
- A pulse is a single small message; its size depends on the change batch, not on the vault size. An oversized batch degrades to a keyless pulse and a full remote walk on the receiver. There is no chunking or reassembly.
- Incremental sync relies on the in-memory remote snapshot. A receiver with an unpopulated snapshot (cold start) or a keyless pulse falls back to a full remote walk, which refreshes the snapshot. Missed pulses while offline are healed by such a walk, not by a replayed sequence.
- Collaboration assumes every device uses the same remote backend and comparable sync rules. Different backends can produce different UIDs and are not a supported sharing setup.
- The module uses the browser build of `mqtt`; no Node.js APIs are used.
- This is file-level race detection, not character-level collaborative editing. Concurrent edits go through the configured conflict resolver.
Loading