Skip to content

fix: paginate device scan, correct BSO timestamp units, track smithy build inputs - #575

Open
layertwo wants to merge 1 commit into
mainlinefrom
fix/device-scan-pagination
Open

layertwo wants to merge 1 commit into
mainlinefrom
fix/device-scan-pagination

Conversation

@layertwo

Copy link
Copy Markdown
Owner

Three issues found while fact-checking AGENTS.md against the code (#574). Each is small and independent; the third was only discovered because it prevented the second from taking effect.

1. DeviceManager.get_devices silently truncated device lists

It did a single unpaginated table.scan, so once the auth table exceeded 1 MB scanned, devices past that point were dropped with no error. Now follows LastEvaluatedKey, matching the existing StorageManager.list_collections pattern.

The loop deliberately does not stop on an empty page. DynamoDB caps a page at 1 MB scanned — before FilterExpression is applied — so a page full of non-device items returns zero Items while more devices remain. Stopping on an empty page would reintroduce the bug in a subtler form. The new test covers exactly that: page 2 is empty on purpose, with devices on pages 1 and 3.

The scan itself stays: UserCollectionsIndex is the only GSI in the stack, and the auth table has none.

2. BSO modified was documented in the wrong units

smithy/models/storage/bso.smithy said "Last modified timestamp (milliseconds since epoch)". The code uses secondsget_current_timestamp() is round(time.time(), 2) — and the wrong description propagated into the generated pydantic model.

Left account.smithy's keyRotationTimestamp alone: that one really is milliseconds (accounts store int(time.time() * 1000)), so a blanket fix of every "milliseconds" mention would have introduced a new error.

3. ./gradlew smithyBuild was incrementally blind to the IDL

Fixing (2) surfaced why it had gone unnoticed for so long: after editing the .smithy file, ./gradlew smithyBuild reported :smithyBuild UP-TO-DATE and emitted a stale spec. My "fix" appeared to do nothing.

Root cause: the models live in models/ (declared via "sources": ["models"] in smithy-build.json), but the Gradle plugin's default source dir is model/ — singular, and nonexistent here. So the IDL files were never task inputs. The Smithy CLI still reads them at execution time, which is why the build produced correct output while being incrementally wrong: any IDL edit was invisible to the up-to-date check.

Consequences were real. A stale spec feeds stale pydantic models into lambda/scripts/codegen.sh and a stale OpenAPI document into SpecRestApi. CI never noticed because it builds from a clean checkout with no prior outputs — so this could only ever be wrong locally, which is the environment nobody verifies.

Declaring models/ and smithy-build.json as inputs fixes it. Verified:

Scenario Before After
Edit a .smithy file UP-TO-DATE, stale spec rebuilds, change reaches the spec
Re-run with no changes UP-TO-DATE UP-TO-DATE

Verification

959 passed at 100% coverage (958 + the new pagination test); mypy, black, isort, flake8 all clean. The regenerated models were confirmed to carry the corrected description.

…inputs

Three issues found while fact-checking AGENTS.md against the code.

1. DeviceManager.get_devices did a single unpaginated table.scan, so device
   lists silently truncated once the auth table exceeded 1 MB scanned. Now
   follows LastEvaluatedKey, matching StorageManager.list_collections.

   The loop deliberately does not stop on an empty page: DynamoDB caps a page
   at 1 MB *scanned*, before FilterExpression is applied, so a page of
   non-device items returns zero Items while more devices remain. The new test
   covers exactly that case (page 2 is empty on purpose).

   The scan itself stays — UserCollectionsIndex is the only GSI in the stack
   and the auth table has none.

2. smithy/models/storage/bso.smithy documented BSO `modified` as "milliseconds
   since epoch"; the code uses seconds (get_current_timestamp() is
   round(time.time(), 2)). The wrong description propagated into the generated
   pydantic model. Left account.smithy's keyRotationTimestamp alone — that one
   really is milliseconds (accounts store int(time.time() * 1000)).

3. Fixing (2) surfaced why it had gone unnoticed: `./gradlew smithyBuild`
   reported UP-TO-DATE after the IDL edit and emitted a stale spec. The models
   live in `models/` (declared via `sources` in smithy-build.json) but the
   Gradle plugin defaults to `model/` (singular), so the IDL files were never
   task inputs. The Smithy CLI still read them at execution time, which is why
   the build worked while being incrementally wrong.

   Declaring models/ and smithy-build.json as inputs fixes it. Verified: an IDL
   edit now rebuilds and the change reaches the spec; a no-op run is still
   UP-TO-DATE. CI never saw this because it builds from a clean checkout.

959 tests pass at 100% coverage; mypy, black, isort and flake8 clean.
@github-actions

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  lambda/src/services
  device_manager.py
Project Total  

This report was generated by python-coverage-comment-action

@github-actions

Copy link
Copy Markdown

Diff for stage: DefaultStage

Warning

4 Destructive Changes

No Changes for stack: GitHubOidcStack ✅

Diff for stack: Service-prod - 4 to add, 5 to update, 4 to destroy

Details

[!WARNING]
Destructive Changes ‼️
Stack: Service-prod - Resource: AuthApiDeploymentB62B2E46bda7a3ee526845c6f831f98a8023195f - Impact: WILL_DESTROY

Stack: Service-prod - Resource: TokenApiDeploymentB896C219ee4433fd7a430b8eb183e22b78e32990 - Impact: WILL_DESTROY

Stack: Service-prod - Resource: ProfileApiDeployment84A54415750bd55753532dd2ebecd42716cf4155 - Impact: WILL_DESTROY

Stack: Service-prod - Resource: StorageApiDeployment3B4929C55affeb30ee76fabc67e717b56801d14f - Impact: WILL_DESTROY

Resources
[-] AWS::ApiGateway::Deployment AuthApiDeploymentB62B2E46bda7a3ee526845c6f831f98a8023195f destroy
[-] AWS::ApiGateway::Deployment TokenApiDeploymentB896C219ee4433fd7a430b8eb183e22b78e32990 destroy
[-] AWS::ApiGateway::Deployment ProfileApiDeployment84A54415750bd55753532dd2ebecd42716cf4155 destroy
[-] AWS::ApiGateway::Deployment StorageApiDeployment3B4929C55affeb30ee76fabc67e717b56801d14f destroy
[+] AWS::ApiGateway::Deployment AuthApiDeploymentB62B2E46ac99c3a69dd45f19958cb37bb317091a
[+] AWS::ApiGateway::Deployment TokenApiDeploymentB896C21940d30031daf269dad58a2f6f3cd50e46
[+] AWS::ApiGateway::Deployment ProfileApiDeployment84A54415225de1b126ba8d19e1f58d99b83b22e1
[+] AWS::ApiGateway::Deployment StorageApiDeployment3B4929C5d19069ae0722651b83088eb0f3753260
[~] AWS::Lambda::Function AuthApiHandlerED50ACFA
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 0591ee89c558a8ed5fa8f88e4083c55df9860b74fe3dd7ce90ee792698d3a55f.zip
 │       └─ [+] 689208ab57a8688fff4b5ec88020862f2f318731385815b8572b1a3e83679c68.zip
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.0591ee89c558a8ed5fa8f88e4083c55df9860b74fe3dd7ce90ee792698d3a55f
         └─ [+] asset.689208ab57a8688fff4b5ec88020862f2f318731385815b8572b1a3e83679c68
[~] AWS::Lambda::Function TokenApiHandler2E66DB25
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 0591ee89c558a8ed5fa8f88e4083c55df9860b74fe3dd7ce90ee792698d3a55f.zip
 │       └─ [+] 689208ab57a8688fff4b5ec88020862f2f318731385815b8572b1a3e83679c68.zip
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.0591ee89c558a8ed5fa8f88e4083c55df9860b74fe3dd7ce90ee792698d3a55f
         └─ [+] asset.689208ab57a8688fff4b5ec88020862f2f318731385815b8572b1a3e83679c68
[~] AWS::Lambda::Function ProfileApiHandler9B65A298
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 0591ee89c558a8ed5fa8f88e4083c55df9860b74fe3dd7ce90ee792698d3a55f.zip
 │       └─ [+] 689208ab57a8688fff4b5ec88020862f2f318731385815b8572b1a3e83679c68.zip
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.0591ee89c558a8ed5fa8f88e4083c55df9860b74fe3dd7ce90ee792698d3a55f
         └─ [+] asset.689208ab57a8688fff4b5ec88020862f2f318731385815b8572b1a3e83679c68
[~] AWS::Lambda::Function ApiHandler5E7490E8
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 0591ee89c558a8ed5fa8f88e4083c55df9860b74fe3dd7ce90ee792698d3a55f.zip
 │       └─ [+] 689208ab57a8688fff4b5ec88020862f2f318731385815b8572b1a3e83679c68.zip
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.0591ee89c558a8ed5fa8f88e4083c55df9860b74fe3dd7ce90ee792698d3a55f
         └─ [+] asset.689208ab57a8688fff4b5ec88020862f2f318731385815b8572b1a3e83679c68
[~] AWS::ApiGateway::Stage AuthApiDeploymentStageprodB0E4172A
 └─ [~] DeploymentId
     └─ [~] .Ref:
         ├─ [-] AuthApiDeploymentB62B2E46bda7a3ee526845c6f831f98a8023195f
         └─ [+] AuthApiDeploymentB62B2E46ac99c3a69dd45f19958cb37bb317091a
[~] AWS::ApiGateway::Stage TokenApiDeploymentStageprod11035AE4
 └─ [~] DeploymentId
     └─ [~] .Ref:
         ├─ [-] TokenApiDeploymentB896C219ee4433fd7a430b8eb183e22b78e32990
         └─ [+] TokenApiDeploymentB896C21940d30031daf269dad58a2f6f3cd50e46
[~] AWS::ApiGateway::Stage ProfileApiDeploymentStageprodF609D968
 └─ [~] DeploymentId
     └─ [~] .Ref:
         ├─ [-] ProfileApiDeployment84A54415750bd55753532dd2ebecd42716cf4155
         └─ [+] ProfileApiDeployment84A54415225de1b126ba8d19e1f58d99b83b22e1
[~] AWS::ApiGateway::RestApi StorageApiAD520CAE
 └─ [~] Body
     └─ [~] .components:
         └─ [~] .schemas:
             └─ [~] .BasicStorageObject:
                 └─ [~] .properties:
                     └─ [~] .modified:
                         └─ [~] .description:
                             ├─ [-] Last modified timestamp (milliseconds since epoch)
                             └─ [+] Last modified timestamp (seconds since epoch, 2 decimal places)
[~] AWS::ApiGateway::Stage StorageApiDeploymentStageprodF84F218F
 └─ [~] DeploymentId
     └─ [~] .Ref:
         ├─ [-] StorageApiDeployment3B4929C55affeb30ee76fabc67e717b56801d14f
         └─ [+] StorageApiDeployment3B4929C5d19069ae0722651b83088eb0f3753260
[~] AWS::Lambda::Function ChannelApiHandler02759D57
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 0591ee89c558a8ed5fa8f88e4083c55df9860b74fe3dd7ce90ee792698d3a55f.zip
 │       └─ [+] 689208ab57a8688fff4b5ec88020862f2f318731385815b8572b1a3e83679c68.zip
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.0591ee89c558a8ed5fa8f88e4083c55df9860b74fe3dd7ce90ee792698d3a55f
         └─ [+] asset.689208ab57a8688fff4b5ec88020862f2f318731385815b8572b1a3e83679c68

No Changes for stack: Frontend-prod ✅
No Changes for stack: Monitoring-prod ✅

Generated for commit c134c59 at 2026-09-18T21:43:54.312Z

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant