Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,28 @@
# Changelog


## Unreleased

- Accept IEC capacity units (GiB/MiB and related sizes) from newer dmidecode
output, restoring module/installed capacity and memory-array limits.
- Bound Bluetooth controller discovery to five seconds with probe telemetry;
failed/truncated discovery leaves the controller count unknown instead of zero.
- Stop and reap collector process trees before private staging cleanup on INT/TERM
or early failure, including producers in GNU timeout's separate process groups.
- Add hanging/failed/empty/successful Bluetooth and INT/TERM cleanup regressions.
- Add standard-profile typed SMBIOS RAM arrays/modules with capacity, DDR type,
manufacturer/part number, slot/bank, rank, ECC, rated/configured speeds and
explicit missing-tool/access/firmware/truncation coverage. Keep serial/asset
identifiers out of acquisition output.
- Add typed CPU topology/cache/selected ISA features, firmware models, GPU PCI
name/driver/available VRAM, disk models/transport/capacity, system
battery full/design capacity health, available RAM and swap counters.
- Reduce duplicate hardware text in automatic snapshots; retain focused evidence
under named hardware targets and `all`, without changing the schema or budgets.
- Add a lossless dictionary-coded `context.ai.json` AI view with round-trip
validation, residual scanning and manifest/archive coverage.
- Add hardware parser/sysfs/privacy fixtures and memory-collector integration tests.

## 1.0.0

- Fixed the Docker local-socket regression fixture so it validates the security invariant (explicit local Unix-socket pinning) without assuming a rootless socket wins over a real `/run/docker.sock` on developer workstations.
Expand Down
45 changes: 44 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,7 @@ Typical output:
```text
linux-context-HOST-YYYYMMDDTHHMMSSZ/
├── context.json
├── context.ai.json # lossless dictionary-coded AI view
├── manifest.sha256
├── REDACTION-REPORT.md
├── meta/
Expand Down Expand Up @@ -287,6 +288,25 @@ This is the file an AI should receive/read first. It contains:

It is intentionally minified and optimized for machine consumption, not human aesthetics.

### `context.ai.json` — lossless compact AI view

For richer hardware inventories, this alternate entrypoint reduces repeated
attribute names, device types and relation predicates using small dictionaries.
It keeps every canonical fact, value, relationship, provenance row, coverage
state and evidence route. Its embedded `encoding` explains how to read it.
Send **one** entrypoint to an AI; there is no benefit to sending both.

To reconstruct the original document exactly:

```bash
python3 -B -S lib/ai_view.py decode BUNDLE/context.ai.json reconstructed.json
```

Size savings depend on the inventory. If the dictionary header outweighs the
savings, this file uses the original canonical encoding, so it is never larger
in bytes than `context.json`. Actual token savings also depend on the model tokenizer.
`context.json` remains the canonical v5 format for existing integrations.

### `meta/graph.json` — overflow graph

Only present when the complete typed graph would exceed the profile's hard `context.json` ceiling. The full graph is preserved here rather than deleted; `context.json` becomes a bounded routing/summary layer.
Expand Down Expand Up @@ -348,7 +368,7 @@ Current domains include:

- host identity, distribution and capabilities;
- kernel/modules/taint/high-value sysctls and source configuration;
- CPU/memory/PCI/USB/block hardware;
- CPU topology/cache/features, RAM module/slot/type/speed/manufacturer, GPU model/driver/VRAM, and PCI/USB/block hardware;
- filesystem/mount/swap/LVM/RAID/ZFS/Btrfs topology;
- SMART/NVMe/filesystem health where safely available;
- interfaces, addresses, routes, DNS, sockets and firewall topology;
Expand All @@ -371,6 +391,29 @@ Current domains include:

The core does not assume all of these technologies exist. Feature detection is explicit; missing tools/subsystems are represented as unavailable rather than guessed.

### Identifying installed RAM and hardware

From `standard` onward, hardware identity is included as typed facts/entities in
`context.json`; it does not require `--target all` or reading raw DMI evidence.

```bash
sudo ./bin/linux-context --profile max
# Keep extra hardware evidence for a focused follow-up:
sudo ./bin/linux-context --profile max --target hardware
```

With optional `dmidecode` installed and firmware access available, the model
reports memory arrays and modules: capacity, DDR type, manufacturer/part number,
slot/bank, form factor, rank, rated/configured speed and reported ECC policy.
Firmware-installed capacity is distinct from RAM visible to Linux. Unknown
identity, missing tooling and inaccessible firmware are explicit; the exporter
never guesses DDR type from “16 GB” or installs dependencies automatically.

CPU topology/cache/features, motherboard/BIOS models, GPU PCI name/driver/VRAM,
disk model/transport and battery full/design capacity are also modeled
when exposed by the host. Automatic runs omit redundant hardware command dumps;
focused targets preserve extra evidence. See [hardware model details](docs/COLLECTORS.md#hardware-model-standard-and-above).

---

## Security and privacy model
Expand Down
19 changes: 19 additions & 0 deletions bin/linux-context
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ Recommended:
Exhaustive target-specific diagnostic acquisition, still bounded/read-only/redacted.

AI contract:
context.ai.json Lossless dictionary-coded AI view.
context.json Canonical compact typed host graph/index.
meta/evidence.json Evidence routing catalog; open only on demand.
meta/collection.json Collector/probe telemetry; not host context.
Expand Down Expand Up @@ -145,6 +146,18 @@ init_bundle
on_exit() {
local rc=$?
trap - EXIT
trap '' INT TERM
# Background shells can ignore terminal SIGINT and continue writing into
# staging. Stop and reap their full trees before removing private material.
local -a workers=()
mapfile -t workers < <(jobs -pr)
if (( ${#workers[@]} )); then
if ! lctx_python "$LCTX_PROJECT_ROOT/lib/stop_workers.py" "${workers[@]}"; then
log ERROR "Unable to stop collectors; private staging retained at $LCTX_TMP_DIR"
exit "$rc"
fi
wait || true
fi
cleanup_tmp || true
exit "$rc"
}
Expand Down Expand Up @@ -263,6 +276,11 @@ log INFO 'compiling compact canonical AI context'
compile_context || fatal "canonical context compilation/validation failed: $LCTX_OUTPUT_DIR/meta/validation.json"
# Defense in depth: sanitize compiler-generated JSON/metadata too, then rescan.
sanitize_bundle_text_in_place || fatal 'final bundle redaction pass failed'
# Derive a lossless AI view from sanitized canonical data, before the final
# residual scan and manifest. It receives the same privacy/publication contract.
lctx_python "$LCTX_PROJECT_ROOT/lib/ai_view.py" encode \
"$LCTX_OUTPUT_DIR/context.json" "$LCTX_OUTPUT_DIR/context.ai.json" \
|| fatal 'lossless AI view generation failed'
final_scan || fatal "residual secret-risk scan failed: $LCTX_OUTPUT_DIR/meta/redaction.json"
# Redaction rewrites first-party JSON structurally, so validate once more afterward.
compile_validation_only() {
Expand All @@ -282,5 +300,6 @@ cleanup_tmp
if (( LCTX_ARCHIVE && ! LCTX_KEEP_DIR )); then remove_published_bundle; fi

printf 'Bundle: %s\n' "$LCTX_OUTPUT_DIR"
if (( LCTX_KEEP_DIR )); then printf 'AI context: %s/context.ai.json\n' "$LCTX_OUTPUT_DIR"; fi
if [[ -n "$archive_path" ]]; then printf 'Archive: %s\n' "$archive_path"; fi
exit 0
14 changes: 7 additions & 7 deletions collectors/hardware/hardware.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
set -euo pipefail
COLLECTOR_ID='hardware.inventory'
COLLECTOR_MIN_PROFILE='standard'
COLLECTOR_TARGETS='system hardware'
COLLECTOR_TARGETS='system hardware cpu gpu memory'
COLLECTOR_PRIVILEGE='optional-root'
COLLECTOR_BASELINE=1
COLLECTOR_DESCRIPTION='CPU, memory, PCI/USB devices, and safe block-device hardware inventory.'
Expand All @@ -21,14 +21,14 @@ collector_collect() {
[[ "$value" =~ ^[0-9]+$ ]] && emit_fact hardware.memory.total_bytes "$value" /proc/meminfo observed 1.0 number
fi

command_exists lscpu && run_capture cpu "$LCTX_COMMAND_TIMEOUT" "$LCTX_COMMAND_MAX_BYTES" -- lscpu || true
command_exists free && run_capture memory "$LCTX_COMMAND_TIMEOUT" "$LCTX_COMMAND_MAX_BYTES" -- free -h || true
command_exists lsblk && run_capture block_hardware "$LCTX_COMMAND_TIMEOUT" "$LCTX_COMMAND_MAX_BYTES" -- lsblk -e 7 -o NAME,KNAME,TYPE,SIZE,FSTYPE,FSVER,MOUNTPOINTS,RO,RM,ROTA,TRAN,MODEL || true
{ target_requested hardware || target_requested cpu; } && command_exists lscpu && run_capture cpu "$LCTX_COMMAND_TIMEOUT" "$LCTX_COMMAND_MAX_BYTES" -- lscpu || true
{ target_requested hardware || target_requested memory; } && command_exists free && run_capture memory "$LCTX_COMMAND_TIMEOUT" "$LCTX_COMMAND_MAX_BYTES" -- free -h || true
{ target_requested hardware || target_requested storage; } && command_exists lsblk && run_capture block_hardware "$LCTX_COMMAND_TIMEOUT" "$LCTX_COMMAND_MAX_BYTES" -- lsblk -e 7 -o NAME,KNAME,TYPE,SIZE,FSTYPE,FSVER,MOUNTPOINTS,RO,RM,ROTA,TRAN,MODEL || true
command_exists lspci && run_capture pci "$LCTX_COMMAND_TIMEOUT" "$LCTX_COMMAND_MAX_BYTES" -- lspci -nnk || true
command_exists lsusb && run_capture usb "$LCTX_COMMAND_TIMEOUT" "$LCTX_COMMAND_MAX_BYTES" -- lsusb || true

# Sysfs DMI is available on many systems even when dmidecode is not installed.
run_shell_capture firmware_identity 5 262144 '
target_requested hardware && run_shell_capture firmware_identity 5 262144 '
for f in sys_vendor product_name product_version board_vendor board_name board_version bios_vendor bios_version bios_date; do
p="/sys/class/dmi/id/$f"; [ -r "$p" ] || continue; printf "%s=" "$f"; cat "$p"; done
[ -r /sys/devices/system/cpu/cpu0/microcode/version ] && { printf "microcode="; cat /sys/devices/system/cpu/cpu0/microcode/version; } || true' || true
Expand All @@ -37,12 +37,12 @@ collector_collect() {
command_exists nvidia-smi && run_capture nvidia_gpu 10 524288 -- nvidia-smi --query-gpu=name,driver_version,pstate,memory.total,memory.used,temperature.gpu,utilization.gpu --format=csv,noheader,nounits || true
command_exists sensors && run_capture sensors 10 524288 -- sensors || true
fi
if (( EUID == 0 )) && profile_at_least deep && command_exists dmidecode; then
if target_requested hardware && (( EUID == 0 )) && profile_at_least deep && command_exists dmidecode; then
# dmidecode includes unique Serial Number/UUID/Asset Tag fields by default.
# Those add negligible diagnostic value and are intentionally excluded at
# collection time. Keep model/topology/memory characteristics only.
run_shell_capture dmi 15 "$LCTX_COMMAND_MAX_BYTES" '
dmidecode -t system -t baseboard -t memory 2>/dev/null |
dmidecode -t system -t baseboard 2>/dev/null |
awk "
/^(System Information|Base Board Information|Memory Device)$/ {print; next}
/^[[:space:]]+(Manufacturer|Product Name|Version|Family|Type|Size|Form Factor|Locator|Bank Locator|Type Detail|Speed|Configured Memory Speed|Part Number|Rank|Configured Voltage|Minimum Voltage|Maximum Voltage):/ {print}
Expand Down
34 changes: 34 additions & 0 deletions collectors/hardware/memory.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
set -euo pipefail
COLLECTOR_ID='hardware.memory'
COLLECTOR_MIN_PROFILE='standard'
COLLECTOR_TARGETS='system hardware memory'
COLLECTOR_PRIVILEGE='optional-root'
COLLECTOR_BASELINE=1
COLLECTOR_DESCRIPTION='Typed RAM modules, firmware-reported slots/capacity/ECC, and explicit discovery gaps.'
source "$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../../lib" && pwd)/collector_api.sh"

collector_collect() {
local status='tool_missing' input=''
if command_exists dmidecode; then
status='access_or_firmware_unavailable'
# Filter identifiers BEFORE persistence. Headers retain local DMI handles
# for array/module links; serial numbers, UUIDs and asset tags are omitted.
if run_shell_capture memory_dmi 15 "$LCTX_COMMAND_MAX_BYTES" '
dmidecode --type 16,17 |
awk '\''
/^Handle 0x[0-9A-Fa-f]+, DMI type (16|17),/ { print; next }
/^(Physical Memory Array|Memory Device)$/ { print; next }
/^[[:space:]]+(Location|Use|Error Correction Type|Maximum Capacity|Number Of Devices|Array Handle|Size|Form Factor|Locator|Bank Locator|Type|Type Detail|Speed|Configured Memory Speed|Configured Clock Speed|Manufacturer|Part Number|Rank|Total Width|Data Width|Configured Voltage|Minimum Voltage|Maximum Voltage|Memory Technology):/ { print }
'\'''; then
if (( LCTX_CAPTURE_TRUNCATED == 0 )); then
input="$LCTX_SECTION_DIR/memory_dmi.txt"
status='no_records'
else
status='truncated'
fi
fi
fi
lctx_python "$COLLECTOR_LIB_DIR/hardware_model.py" memory "$input" "$status"
}
collector_main "$@"
46 changes: 46 additions & 0 deletions collectors/hardware/platform.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
set -euo pipefail
COLLECTOR_ID='hardware.platform'
COLLECTOR_MIN_PROFILE='standard'
COLLECTOR_TARGETS='system hardware cpu gpu display power'
COLLECTOR_PRIVILEGE='user'
COLLECTOR_BASELINE=1
COLLECTOR_DESCRIPTION='Typed CPU topology, firmware models, GPU drivers/VRAM, and battery capacity health.'
source "$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../../lib" && pwd)/collector_api.sh"

collector_collect() {
local cpu_file='' pci_file='' block_file=''
if command_exists lscpu; then
if probe_capture cpu_json 5 262144 -- lscpu --json; then
(( LCTX_CAPTURE_TRUNCATED == 0 )) && cpu_file=$LCTX_PROBE_FILE
fi
fi
if command_exists lspci; then
if probe_capture pci_models 5 262144 -- lspci -Dmm -nn; then
(( LCTX_CAPTURE_TRUNCATED == 0 )) && pci_file=$LCTX_PROBE_FILE
fi
fi
if command_exists lsblk; then
if probe_capture block_models 5 262144 -- lsblk --json --bytes -e 1,7 -o NAME,KNAME,TYPE,SIZE,RO,RM,ROTA,TRAN,MODEL; then
(( LCTX_CAPTURE_TRUNCATED == 0 )) && block_file=$LCTX_PROBE_FILE
fi
fi
# The helper reads only bounded allowlisted sysfs fields; no EDID, serials,
# arbitrary device properties, or writes to power-management controls.
local snapshot_file
if probe_capture platform_snapshot 10 "$LCTX_COMMAND_MAX_BYTES" -- \
python3 -B -S "$COLLECTOR_LIB_DIR/hardware_model.py" snapshot "$cpu_file" "$pci_file" "$block_file"; then
snapshot_file=$LCTX_PROBE_FILE
if (( LCTX_CAPTURE_TRUNCATED == 0 )); then
lctx_python "$COLLECTOR_LIB_DIR/hardware_model.py" platform "$snapshot_file"
else
record_collector_note 'platform-snapshot-truncated'
fi
fi
release_probe
local file
for file in "$cpu_file" "$pci_file" "$block_file"; do
[[ -z "$file" ]] || rm -f -- "$file"
done
}
collector_main "$@"
22 changes: 17 additions & 5 deletions collectors/workstation/devices.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,25 @@ collector_collect() {
fi

if command_exists bluetoothctl; then
local bt_list bt_count=0
local bt_list line bt_count=0
emit_fact workstation.bluetooth.client_present true 'command -v bluetoothctl' observed 1.0 boolean
bt_list=$(LC_ALL=C bluetoothctl list 2>/dev/null || true)
if [[ -n "$bt_list" ]]; then
while IFS= read -r _; do bt_count=$((bt_count+1)); done <<< "$bt_list"
# BlueZ can wait indefinitely when the daemon/bus is unavailable. A
# failed probe means unknown, not zero controllers; never parse partial
# output or persist personal controller names from this discovery step.
if probe_capture bluetooth_list 5 262144 -- bluetoothctl list; then
if (( ! LCTX_CAPTURE_TRUNCATED )); then
bt_list=$(<"$LCTX_PROBE_FILE")
while IFS= read -r line; do
[[ "$line" == Controller\ * ]] && bt_count=$((bt_count+1))
done <<< "$bt_list"
emit_fact workstation.bluetooth.controller_count "$bt_count" 'bluetoothctl list' observed 1.0 number
else
record_collector_note 'truncated:bluetooth_list (controller count unknown)'
fi
else
record_collector_note "bluetooth discovery unavailable:bluetooth_list (exit $LCTX_CAPTURE_RC; controller count unknown)"
fi
emit_fact workstation.bluetooth.controller_count "$bt_count" 'bluetoothctl list' observed 1.0 number
release_probe
# No controller is a normal state (for example bluez tools installed on a
# desktop with Bluetooth disabled/absent), not an evidence failure.
if (( bluetooth_focused && bt_count > 0 )); then
Expand Down
Loading