Fix unbounded service logs and HTTP/MQTT traces

config.example.toml

@@ -30,6 +30,13 @@ enable_topic_bridge = true
 [storage]
 data_dir = "/data"
 
+[logging]
+# Applies to each service .log and HTTP/MQTT .jsonl trace in runtime/.
+# MiB per file (1-1024), plus this many numbered backups (0-20).
+# 0 backups keeps only the active file; rotation remains enabled.
+max_size_mb = 10
+backup_count = 3
+
 [tls]
 # Cloudflare DNS-01 wildcard issuance. No public inbound access is required.
 mode = "cloudflare_acme"

docs/logging.md

@@ -0,0 +1,64 @@
+# Log retention
+
+The server automatically rotates the six service logs (`api_server.log`,
+`iot_server.log`, `wood_server.log`, `https_server.log`, `mqtt_server.log`, and
+`unknown_server.log`) and both protocol traces (`decompiled_http.jsonl` and
+`decompiled_mqtt.jsonl`) in the data directory's `runtime/` folder.
+
+The defaults are **10 MiB per file and three backups per file**. The active file
+keeps its usual name; `.1` is the newest backup and `.3` is the oldest. Older
+backups expire automatically. At these defaults, the eight log families normally
+use no more than about **320 MiB combined**. The threshold is per file, not a
+quota for the entire data directory. A single larger record is kept intact and
+can exceed the threshold; the next record causes rotation.
+
+Console output remains available through Home Assistant's Logs tab or
+`docker logs`. Container console-log retention is managed separately by the
+container host. Changing console verbosity does not disable the JSONL files.
+
+## Home Assistant
+
+Set these options in the add-on configuration, then restart it:
+
+```yaml
+log_max_size_mb: 10
+log_backup_count: 3
+```
+
+`log_max_size_mb` accepts an integer from 1 to 1024, measured in MiB
+(1,048,576 bytes). `log_backup_count` accepts an integer from 0 to 20.
+Setting the backup count to `0` keeps only the active file; it still rolls over
+when full. There is no unlimited-size setting.
+
+## Docker and standalone installs
+
+Add this optional section to `config.toml`, then restart:
+
+```toml
+[logging]
+max_size_mb = 10
+backup_count = 3
+```
+
+The same ranges apply. Existing configurations that omit this section receive
+the defaults automatically.
+
+## Existing large logs and changes to retention
+
+At startup, the server first recovers missing MQTT credentials and seeds missing
+zone-range state from the existing logs, including numbered backups. It then
+trims oversized logs to their most recent complete lines and removes backups
+beyond the configured count. This also applies when reducing either limit and
+to logs that are no longer receiving traffic. The trimming operation reads only
+a tail window and replaces each file atomically; it does not load a multi-GB
+log into memory or retain it as a multi-GB backup.
+
+Older diagnostic history is discarded. Copy any logs you need for an ongoing
+investigation before upgrading or lowering retention. Persistent credentials,
+inventory, zone ranges, device keys, and other state files are not subject to
+log rotation. Do not delete the entire `runtime/` directory to clear logs: it
+also contains persistent state.
+
+Rotation coordinates writers within one server process. Run only one server
+against a given data directory, and let the server manage these files rather
+than running a second external rotation job on them.

docs/updating.md

@@ -16,6 +16,11 @@ Use this page when moving an existing install to a newer stable release.
 
 ## Updating From A Stable Or RC Release
 
+Builds with automatic [log retention](logging.md) apply a default of 10 MiB per
+file with three backups, including to existing installations. On first startup,
+oversized logs are trimmed after state recovery; copy any historical logs you
+need before upgrading.
+
 If you were already on a stable release such as `0.0.5`, `0.0.6`, `1.0.0`, or a late release candidate, updating should usually not require manual migration.
 
 After updating:

mkdocs.yml

@@ -58,6 +58,7 @@ nav:
     - Mobile App Options: roborock_app.md
     - Camera (Live Video): camera.md
   - Reference:
+    - Log Retention: logging.md
     - Known Limitations: known_limitations.md
     - Scene Validity Reports: scene_validity.md
     - Tested Vacuums: tested_vacuums.md

roborock_local_server_addon/DOCS.md

@@ -45,6 +45,13 @@ For existing cloud integrations, Home Assistant does not support changing the se
 
 ## Notes
 
+Service logs and HTTP/MQTT JSONL traces rotate automatically at 10 MiB per file,
+with three backups each. Set `log_max_size_mb` (1–1024 MiB) and
+`log_backup_count` (0–20), then restart to change retention. Zero backups keeps
+only the active file while still limiting growth. Existing oversized logs are
+trimmed at startup after state recovery. See [Log retention](../docs/logging.md)
+for upgrade behavior and storage estimates.
+
 - Local-only access is still the preferred setup. If you expose it for remote access, the server handles auth and can disable new devices from connecting, but any publicly accessible self-hosted service has risk.
 - If you change `https_port` or `mqtt_tls_port`, update your DNS/clients to use those ports.
 - The current server advertises the same hostname for HTTPS and MQTT/TLS, so Home Assistant's Roborock entry should normally use `ssl://api-roborock.example.com:8881`, not a separate `mqtt-...` hostname.

roborock_local_server_addon/config.yaml

@@ -70,6 +70,8 @@ options:
   advertised_https_port: 0
   advertised_mqtt_tls_port: 0
   region: "us"
+  log_max_size_mb: 10
+  log_backup_count: 3
   tls_mode: "provided"
   tls_base_domain: ""
   tls_email: ""
@@ -96,6 +98,8 @@ schema:
   advertised_https_port: int
   advertised_mqtt_tls_port: int
   region: list(us|eu|cn|ru)
+  log_max_size_mb: int(1,1024)?
+  log_backup_count: int(0,20)?
   tls_mode: list(provided|cloudflare_acme)
   tls_base_domain: str
   tls_email: str

roborock_local_server_beta_addon/config.yaml

@@ -29,6 +29,8 @@ options:
   advertised_https_port: 0
   advertised_mqtt_tls_port: 0
   region: "us"
+  log_max_size_mb: 10
+  log_backup_count: 3
   tls_mode: "provided"
   tls_base_domain: ""
   tls_email: ""
@@ -48,6 +50,8 @@ schema:
   advertised_https_port: int
   advertised_mqtt_tls_port: int
   region: list(us|eu|cn|ru)
+  log_max_size_mb: int(1,1024)?
+  log_backup_count: int(0,20)?
   tls_mode: list(provided|cloudflare_acme)
   tls_base_domain: str
   tls_email: str

src/roborock_local_server/bundled_backend/https_server/routes/user/scene/service.py

@@ -6,6 +6,8 @@
 import logging
 from typing import Any, Callable, Sequence
 
+from roborock_local_server.log_retention import iter_log_lines
+
 from shared.context import ServerContext
 from shared.data_helpers import as_bool, as_int, default_home_id, get_value, stable_int
 from shared.inventory_io import WEB_API_INVENTORY_FILE, inventory_mutation, load_inventory, write_inventory
@@ -166,45 +168,36 @@ def _scene_zone_ranges_from_mqtt(
     *,
     tids_filter: set[str] | None = None,
 ) -> dict[tuple[str, int], list[int]]:
-    path = ctx.mqtt_jsonl
-    if not path.exists():
-        return {}
     zone_ranges: dict[tuple[str, int], list[int]] = {}
-    try:
-        with path.open("r", encoding="utf-8") as handle:
-            lines = handle.readlines()
-        for line in reversed(lines):
-            try:
-                entry = json.loads(line)
-            except json.JSONDecodeError:
-                continue
-            decoded_messages = entry.get("decoded_messages")
-            if not isinstance(decoded_messages, list):
+    # Oldest first so the latest coordinates win, including across rotations.
+    for line in iter_log_lines(ctx.mqtt_jsonl):
+        try:
+            entry = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if not isinstance(entry, dict):
+            continue
+        decoded_messages = entry.get("decoded_messages")
+        if not isinstance(decoded_messages, list):
+            continue
+        for decoded in decoded_messages:
+            if not isinstance(decoded, dict):
                 continue
-            for decoded in decoded_messages:
-                if not isinstance(decoded, dict):
-                    continue
-                rpc = decoded.get("rpc")
-                if isinstance(rpc, dict) and str(rpc.get("method") or "").strip() == "set_scenes_zones":
-                    _merge_scene_zone_ranges_from_request(
-                        zone_ranges,
-                        params=rpc.get("params"),
-                        tids_filter=tids_filter,
-                    )
-                response_to = decoded.get("response_to")
-                if isinstance(response_to, dict) and str(response_to.get("request_method") or "").strip() == "set_scenes_zones":
-                    _merge_scene_zone_ranges_from_response(
-                        zone_ranges,
-                        request_params=response_to.get("request_params"),
-                        result=response_to.get("result"),
-                        tids_filter=tids_filter,
-                    )
-            if tids_filter and all(
-                any(key[0] == tid for key in zone_ranges) for tid in tids_filter
-            ):
-                break
-    except OSError:
-        return {}
+            rpc = decoded.get("rpc")
+            if isinstance(rpc, dict) and str(rpc.get("method") or "").strip() == "set_scenes_zones":
+                _merge_scene_zone_ranges_from_request(
+                    zone_ranges,
+                    params=rpc.get("params"),
+                    tids_filter=tids_filter,
+                )
+            response_to = decoded.get("response_to")
+            if isinstance(response_to, dict) and str(response_to.get("request_method") or "").strip() == "set_scenes_zones":
+                _merge_scene_zone_ranges_from_response(
+                    zone_ranges,
+                    request_params=response_to.get("request_params"),
+                    result=response_to.get("result"),
+                    tids_filter=tids_filter,
+                )
     return zone_ranges
 
 

src/roborock_local_server/bundled_backend/shared/io_utils.py

@@ -7,6 +7,12 @@
 from pathlib import Path
 from typing import Any
 
+from roborock_local_server.config import LoggingConfig
+from roborock_local_server.log_retention import (
+    BoundedRotatingFileHandler,
+    append_jsonl_text,
+)
+
 _FILE_LOG_FORMAT = "%(asctime)s [%(levelname)s] %(message)s"
 _STREAM_LOG_FORMAT = "%(asctime)s [%(levelname)s] [%(name)s] %(message)s"
 _JSONL_STREAM_LOGGER_NAME = "real_stack.jsonl"
@@ -30,18 +36,31 @@ def _jsonl_stream_logger() -> logging.Logger:
 
 def append_jsonl(path: Path, entry: dict[str, Any]) -> None:
     encoded = json.dumps(entry, ensure_ascii=True, separators=(",", ":"))
-    path.parent.mkdir(parents=True, exist_ok=True)
-    with path.open("a", encoding="utf-8") as handle:
-        handle.write(encoded + "\n")
+    append_jsonl_text(path, encoded)
     _jsonl_stream_logger().debug("[%s] %s", path.name, encoded)
 
 
-def setup_file_logger(name: str, path: Path) -> logging.Logger:
+def setup_file_logger(
+    name: str,
+    path: Path,
+    *,
+    max_bytes: int | None = None,
+    backup_count: int | None = None,
+) -> logging.Logger:
+    defaults = LoggingConfig()
+    effective_max_bytes = defaults.max_bytes if max_bytes is None else max_bytes
+    effective_backup_count = defaults.backup_count if backup_count is None else backup_count
     logger = logging.getLogger(f"real_stack.{name}")
     logger.setLevel(logging.INFO)
     logger.propagate = False
-    logger.handlers.clear()
-    handler = logging.FileHandler(path, encoding="utf-8")
+    for previous in logger.handlers[:]:
+        logger.removeHandler(previous)
+        previous.close()
+    handler = BoundedRotatingFileHandler(
+        path,
+        max_bytes=effective_max_bytes,
+        backup_count=effective_backup_count,
+    )
     handler.setFormatter(logging.Formatter(_FILE_LOG_FORMAT))
     logger.addHandler(handler)
     logger.addHandler(_make_stream_handler())

src/roborock_local_server/bundled_backend/shared/runtime_credentials.py

@@ -11,6 +11,8 @@
 from typing import Any
 from urllib.parse import parse_qs
 
+from roborock_local_server.log_retention import iter_log_lines
+
 from .data_helpers import utcnow_iso
 
 
@@ -702,8 +704,6 @@ def recovery_pending_devices(self) -> list[dict[str, str]]:
 
     def backfill_device_mqtt_passwords(self, log_path: str | Path) -> int:
         path = Path(log_path)
-        if not path.exists():
-            return 0
         changed = 0
         with self._lock:
             pending_usernames = {
@@ -715,28 +715,24 @@ def backfill_device_mqtt_passwords(self, log_path: str | Path) -> int:
             return 0
 
         recovered_by_username: dict[str, str] = {}
-        try:
-            with path.open("r", encoding="utf-8", errors="replace") as handle:
-                for raw_line in handle:
-                    line = str(raw_line or "")
-                    if " CONNECT " not in line or " hex=" not in line:
-                        continue
-                    hex_value = line.rsplit(" hex=", 1)[-1].strip()
-                    if not hex_value:
-                        continue
-                    try:
-                        packet = bytes.fromhex(hex_value)
-                    except ValueError:
-                        continue
-                    parsed = parse_mqtt_connect_packet(packet)
-                    if parsed is None:
-                        continue
-                    username = _clean_str(parsed.get("username"))
-                    password = _clean_str(parsed.get("password"))
-                    if username in pending_usernames and password:
-                        recovered_by_username[username] = password
-        except OSError:
-            return 0
+        for raw_line in iter_log_lines(path):
+            line = str(raw_line or "")
+            if " CONNECT " not in line or " hex=" not in line:
+                continue
+            hex_value = line.rsplit(" hex=", 1)[-1].strip()
+            if not hex_value:
+                continue
+            try:
+                packet = bytes.fromhex(hex_value)
+            except ValueError:
+                continue
+            parsed = parse_mqtt_connect_packet(packet)
+            if parsed is None:
+                continue
+            username = _clean_str(parsed.get("username"))
+            password = _clean_str(parsed.get("password"))
+            if username in pending_usernames and password:
+                recovered_by_username[username] = password
 
         with self._lock:
             for username, password in recovered_by_username.items():