Harden v3 election management and data integrity

.github/workflows/type-tests.yml

@@ -36,3 +36,6 @@ jobs:
     - name: Type testing with mypy
       working-directory: v3  # Ensure mypy runs in the correct directory
       run: uv run mypy --cache-dir /tmp/ --ignore-missing-imports .
+    - name: Run regression tests
+      working-directory: v3
+      run: uv run python -m unittest discover -s tests -p 'test_*.py'

v3/ARCHITECTURE.md

@@ -43,7 +43,7 @@ v3/
 - **Authentication**: Relies on external ASF authentication via `asfquart.auth`. Requires committer-level access for most operations.
 - **Templates**: Uses EZT (Easy Template) for server-side rendering. Templates are stored in `templates/` and use `[variable]` syntax for substitution. Includes includes for headers/footers.
 - **Static Assets**: Served from `static/` directory, including Bootstrap CSS/JS, custom CSS, and JavaScript utilities.
-- **Middleware**: Includes session management, CSRF protection (placeholder), and flash messages for user feedback.
+- **Middleware**: Includes session management, session-token CSRF protection for mutations, and flash messages for user feedback. Election management is restricted to the election owner; `authz` delegation is not implemented.
 
 ### Business Logic Layer (`steve/`)
 

v3/README.md

@@ -209,6 +209,40 @@ A basic example of using the API is available via the
 
 See [tests/README.md](tests/README.md) for details on testing the codebase.
 
+## Management and Data Integrity
+
+Election creation requires an authenticated ASF PMC member whose account is
+present in the person database. Management of an existing election is restricted
+to its `owner_pid`. The `authz` field is reserved for a future delegation policy
+and currently grants no additional management access.
+
+All `/do-*` mutations require POST and a session CSRF token. HTML forms send
+`csrf_token`; JSON requests send `X-CSRFToken`. Opening and closing elections
+via GET is not supported. State conflicts return HTTP 409, invalid ballots
+return HTTP 400, and ineligible voters receive HTTP 403. A submitted ballot is
+atomic across all of its issues.
+
+Advisory dates entered in the web UI are stored as midnight UTC and displayed in
+UTC. They do not schedule opening or closing. Once opened, the opening date is
+fixed; once closed, both dates are fixed. Manual opening and closing record the
+actual operation time.
+
+The YAML creation script validates voters without rewriting their IDs and
+creates the election, issues, and eligibility records in one transaction.
+Missing voters and database failures leave no partially created election.
+Timezone-free YAML dates and datetimes are interpreted as UTC; integer
+timestamps remain seconds since the Unix epoch.
+
+Application database connections enforce SQLite foreign keys. Before deploying
+this behavior against an existing database, run `PRAGMA foreign_key_check` on a
+consistent backup and resolve any reported orphan records. Do not delete
+historical voting data to silence integrity errors.
+
+Use `Election` and `PersonDB.open()` as context managers when their connections
+are scoped to an operation. `Election.create(..., db=db)` and `PersonDB(db)`
+borrow an existing database connection; the caller owns its transaction and
+connection lifetime.
+
 ## Threat Model
 
 There are two primary threat vectors that can compromise the cryptographic

v3/TODO.md

@@ -1,52 +1,21 @@
-# TODO: Review and Fixes for v3/server/pages.py
+# Remaining v3 Work
 
-Based on a review of `v3/server/pages.py` (and related templates like `voter.ezt`), here are the identified issues, errors, and areas for improvement. These focus on correctness, logic, security, and completeness, while adhering to project conventions (minimal changes, no unsolicited refactors). Prioritized by impact.
+## Community Self-Service
 
-## 1. Missing Endpoints for Date Saving (Critical) - RESOLVED
-- **Issue**: The `manage.ezt` template includes JavaScript that makes POST requests to `/do-set-open_at/<eid>` and `/do-set-close_at/<eid>` for auto-saving open/close dates. These endpoints were not defined in `pages.py`, causing the auto-save functionality to fail (likely with 404 errors).
-- **Impact**: Users won't be able to save dates via the UI, breaking the intended workflow.
-- **Resolution**: Added the two endpoints with a refactored helper function `_set_election_date` to handle common logic (auth, JSON parsing, validation, setting dates, logging, and response). Endpoints now require authentication, validate dates, and log actions. CSRF handling remains a TODO (placeholder token in use). Test for proper date-setting and error handling. Added supporting methods `set_open_at` and `set_close_at` to the Election class in `election.py`, and corresponding cursors in `queries.yaml`.
+- Add voter roster management to the web UI, including ASF ID validation and a clear policy for issues added after a roster is configured.
+- Add STV creation and editing for candidate labels, names, ASF IDs, and seats. The current web issue creation flow supports YNA only.
+- Define delegated management semantics for `authz` before granting access beyond the election owner.
+- Expose closed-election tallying through an owner-authorized workflow with execution status, failure reporting, stored results, and controlled downloads. Separate summary results from participation lists and anonymized raw ballots.
+- Populate past elections in the voting dashboard and define who can view their results.
 
-## 2. Upcoming Elections Not Populated in `voter_page()` - RESOLVED
-- **Issue**: The `voter.ezt` template checks for `[if-any upcoming]` and loops over `upcoming` elections, but `voter_page()` only sets `result.election` (for open elections). `result.upcoming` is never defined, so the "Upcoming Elections" section will always be empty.
-- **Impact**: Upcoming elections (e.g., those in 'editable' state with a future open date) won't display, confusing users.
-- **Resolution**: Added a new Election class method `upcoming_to_pid` to return editable elections for a given PID with voting eligibility. Added corresponding query `q_upcoming_to_me` in `queries.yaml`. Updated `voter_page()` to fetch and post-process upcoming elections into `result.upcoming`. Updated `voter.ezt` to include a dedicated "Upcoming Elections" section with similar card layout, a "Preview Ballot" link, and subtle visual distinction (lighter background, "Upcoming" badge). No filtering on `open_at` (all editable elections included).
+## Operations and Performance
 
-## 3. Hardcoded `vtype` in `do_add_issue_endpoint()`
-- **Issue**: `vtype` is hardcoded to `'yna'`, and `kv = None`. The comment mentions handling SEATS for STV, but it's not implemented. If users try to add STV issues, it will fail or behave incorrectly.
-- **Impact**: Limits issue types; STV issues can't be added properly.
-- **Suggested Fix**: Make `vtype` dynamic (e.g., from form data). For STV, parse `seats` from the form and set `kv = {'seats': int(form.seats)}` or similar. Update the template's form to include a vtype selector and STV-specific fields.
+- Define production person-directory refresh and how identity changes interact with frozen election data.
+- Measure synchronous SQLite and cryptographic work in HTTP handlers before choosing an offloading strategy; keep database connections in their owning thread.
+- Resolve the advertised open-election tally flag: the underlying tally currently requires a closed election. Do not bypass that state restriction as a tooling cleanup.
+- Document the complete service deployment, database backup and restore procedure, and released-version pinning with Infra.
 
-## 4. Debug Prints in Endpoints
-- **Issue**: `print('FORM:', form)` in `do_add_issue_endpoint()` and `do_edit_issue_endpoint()` is leftover debug code.
-- **Impact**: Clutters logs/output; not production-ready.
-- **Suggested Fix**: Remove these `print` statements.
+## Validation
 
-## 5. Temporary `issue_count` Hack in `postprocess_election()`
-- **Issue**: `if 'issue_count' not in e: e.issue_count = 5` is a hardcoded placeholder.
-- **Impact**: Incorrect counts; should be removed once the query is fixed.
-- **Suggested Fix**: Ensure the Election query/class includes `issue_count` properly, then remove this.
-
-## 6. CSRF Token Placeholder
-- **Issue**: `basic.csrf_token = 'placeholder'` is not secure.
-- **Impact**: CSRF protection is bypassed.
-- **Suggested Fix**: Implement real CSRF tokens (e.g., generate a random token per session and verify in POST endpoints).
-
-## 7. Other Minor Notes
-- **Authz Checks**: Many endpoints have `### check authz` comments but no implementation. If authorization beyond basic auth is needed, add it (e.g., ensure the user owns the election).
-- **Error Handling**: Endpoints like `do_open_endpoint()` assume success; add try/except for potential Election class errors.
-- **Logging**: Consistent and useful; no issues.
-- **Imports/Constants**: All seem correct and follow conventions.
-- **No Obvious Syntax/Logic Errors**: The code parses and runs logically, but the above are functional gaps.
-
-If you'd like to implement any of these fixes (e.g., add the date endpoints or fix upcoming elections), provide confirmation and details. Let me know if you have more context or want to check specific sections!
-
-## 8. CSRF Checking
-- **Issue**: CSRF tokens are placeholders and not validated in POST endpoints.
-- **Impact**: Vulnerable to CSRF attacks.
-- **Suggested Fix**: Implement a decorator to check CSRF tokens on POST endpoints (e.g., compare form/session token). Generate real tokens per session.
-
-## 9. Error Handling in `submitFormWithLoading`
-- **Issue**: If the server returns an error during form submission via `submitFormWithLoading`, the page doesn't reload, and the button stays disabled, potentially confusing users.
-- **Impact**: Poor UX on submission failures.
-- **Suggested Fix**: Investigate and add client-side error handling (e.g., re-enable button on failure, show error message).
+- Run historical STV comparisons when the private meeting fixtures are available.
+- Extend boundary tests alongside new self-service features; avoid replacing behavior checks with coverage targets.

v3/docs/schema.md

@@ -21,7 +21,7 @@ Stores metadata for each election.
   - `eid` (TEXT, PRIMARY KEY): A 10-character hexadecimal election ID (e.g., "1a2b3c4d5e"). Not auto-incremented to prevent URL deduction. Must match the pattern `[0-9a-f]{10}`.
   - `title` (TEXT, NOT NULL): The election’s title.
   - `owner_pid` (TEXT, NOT NULL): The ID of the person who created/owns the election. References `person(pid)`.
-  - `authz` (TEXT): Specifies an LDAP group (e.g., group name or identifier) allowed to edit the election. If `NULL`, only `owner_pid` can edit. The exact format is TBD but will likely map to LDAP-based persons.
+  - `authz` (TEXT): Reserved for delegated management authorization. The format and enforcement policy are not implemented; currently only `owner_pid` can manage an election, regardless of this field's value.
   - `salt` (BLOB): A 16-byte salt generated using `secrets.token_bytes()` in Python, set when the election is opened. `NULL` if not opened.
   - `opened_key` (BLOB): A 32-byte hash (using Argon2) derived from election data to detect tampering, set when the election is opened. `NULL` if not opened.
   - `closed` (INTEGER): Indicates election status: `NULL` or `0` for not closed, `1` for closed (implies opened). Must be `NULL`, `0`, or `1`.
@@ -108,6 +108,6 @@ Records votes cast for issues, supporting re-voting.
   - The owner (`owner_pid`) can monitor and edit the election at all stages, including viewing voter eligibility and vote counts.
 
 ## Notes
-- **Authorization**: The `authz` field’s format is TBD but will likely use LDAP group identifiers to define editing permissions. Validation occurs outside the schema (e.g., via LDAP integration).
+- **Authorization**: Management currently checks `owner_pid` in the web application. The `authz` field does not grant access until a delegation policy is defined and implemented.
 - **Query Optimization**: The schema references a `queries.yaml` file, which maps query names to SQL statements for all database interactions. This file provides insight into common operations (e.g., vote tallying, eligibility checks). Specific queries are not included here but can be analyzed for index optimization.
 - **Future Changes**: The `ciphertext` encryption will transition from Fernet to XChaCha20-Poly1305. Additional `issue` types may be added beyond `yna` and `stv`.

v3/queries.yaml

@@ -132,7 +132,8 @@ election:
     # Which elections were created by PID?
     q_owned: |
         SELECT eid, title, authz, closed, open_at, close_at,
-               owner_pid, (opened_key IS NOT NULL) AS is_opened
+               owner_pid, (opened_key IS NOT NULL) AS is_opened,
+               (SELECT COUNT(*) FROM issue i WHERE i.eid = election.eid) AS issue_count
         FROM election WHERE owner_pid = ?
 
     # Returns all editable elections for voting by the specified person (PID).

v3/server/bin/create-election.py

@@ -26,10 +26,10 @@
 import datetime
 import pathlib
 import logging
+from contextlib import closing
 import yaml
 
 import steve.election
-import steve.persondb
 import steve.vtypes.stv
 
 _LOGGER = logging.getLogger(__name__)
@@ -42,36 +42,44 @@
 
 
 def parse_datetime(dt):
-    """Parse ISO datetime string to Unix timestamp, or return if already int."""
-    if not dt:
+    """Parse timestamps or YAML/ISO dates, interpreting timezone-free values as UTC."""
+    if dt is None:
         return None
-    if isinstance(dt, int):
+    if type(dt) is int:
         return dt
     if isinstance(dt, str):
-        dt_obj = datetime.datetime.fromisoformat(dt)
-        return int(dt_obj.timestamp())
+        dt = datetime.datetime.fromisoformat(dt)
+    if isinstance(dt, datetime.date) and not isinstance(dt, datetime.datetime):
+        dt = datetime.datetime.combine(dt, datetime.time())
+    if isinstance(dt, datetime.datetime):
+        if dt.tzinfo is None:
+            dt = dt.replace(tzinfo=datetime.timezone.utc)
+        return int(dt.timestamp())
     raise ValueError(f'Invalid datetime format: {dt}')
 
 
 def validate_issue(issue):
     """Validate an issue dict from YAML."""
-    if 'vtype' not in issue or issue['vtype'] not in VALID_VTYPES:
+    if not isinstance(issue, dict):
+        raise ValueError('Each issue must be a mapping')
+    if not isinstance(issue.get('vtype'), str) or issue['vtype'] not in VALID_VTYPES:
         raise ValueError(f'Invalid vtype: {issue.get("vtype")}')
+    if not isinstance(issue.get('title'), str) or not issue['title'].strip():
+        raise ValueError('Each issue must have a title')
+    if issue.get('description') is not None and not isinstance(
+        issue['description'], str
+    ):
+        raise ValueError('Issue description must be a string')
     if issue['vtype'] == 'stv':
-        kv = issue.get('kv', {})
-        if not all(k in kv for k in ['version', 'labelmap', 'seats']):
-            raise ValueError(
-                'STV issue missing required kv fields: version, labelmap, seats'
-            )
-        if not isinstance(kv['seats'], int) or kv['seats'] <= 0:
-            raise ValueError('STV seats must be a positive integer')
-        steve.vtypes.stv.get_candidates(kv)
+        steve.vtypes.stv.validate_configuration(issue.get('kv'))
     return issue
 
 
 def main(yaml_file):
-    with open(yaml_file, 'r') as f:
+    with open(yaml_file, encoding='utf-8') as f:
         data = yaml.safe_load(f)
+    if not isinstance(data, dict) or not isinstance(data.get('election'), dict):
+        raise ValueError('YAML must contain an election mapping')
 
     # Extract election data
     election_data = data.get('election', {})
@@ -81,74 +89,54 @@ def main(yaml_file):
     open_at = parse_datetime(election_data.get('open_at'))
     close_at = parse_datetime(election_data.get('close_at'))
 
-    if not title or not owner_pid:
+    if (
+        not isinstance(title, str)
+        or not title.strip()
+        or not isinstance(owner_pid, str)
+        or not owner_pid.strip()
+    ):
         raise ValueError('Election must have title and owner_pid')
+    if authz is not None and not isinstance(authz, str):
+        raise ValueError('authz must be a string')
 
     # Extract issues
     issues = data.get('issues', [])
+    if not isinstance(issues, list):
+        raise ValueError('issues must be a list')
     for issue in issues:
         validate_issue(issue)
 
     # Extract record: list of pid values for eligible voters
     record = data.get('record', [])
-    if not isinstance(record, list):
+    if not isinstance(record, list) or any(not isinstance(pid, str) for pid in record):
         raise ValueError('record must be a list of pid values')
-
-    # TODO: Re-enable transactions for safety once database setup allows.
-    # pdb = steve.persondb.PersonDB(DB_FNAME)
-    # pdb.db.conn.execute('BEGIN TRANSACTION')
-
-    try:
-        # Create election
+    if len(set(record)) != len(record):
+        raise ValueError('record contains duplicate pid values')
+
+    db = steve.election.Election.open_database(DB_FNAME)
+    with closing(db.conn), db.conn:
+        db.conn.execute('BEGIN IMMEDIATE')
+        for pid in {owner_pid, *record}:
+            if not db.conn.execute(
+                'SELECT 1 FROM person WHERE pid = ?', (pid,)
+            ).fetchone():
+                raise ValueError(f'PID {pid} not found in person database')
         election = steve.election.Election.create(
-            DB_FNAME, title, owner_pid, authz, open_at, close_at
-        )
-        _LOGGER.info(
-            f'Created election[E:{election.eid}]: "{title}" by owner "{owner_pid}"'
+            DB_FNAME, title, owner_pid, authz, open_at, close_at, db=db
         )
-
-        # Add issues
         for issue_data in issues:
-            iid = election.add_issue(
+            election.add_issue(
                 issue_data['title'],
                 issue_data.get('description'),
                 issue_data['vtype'],
                 issue_data.get('kv') if issue_data['vtype'] == 'stv' else None,
             )
-            _LOGGER.info(f'Added issue[I:{iid}] to election[E:{election.eid}]')
-
-        # Open a PersonDB using the existing DB from the Election
-        pdb = steve.persondb.PersonDB(election.db)
-
-        # HACK: Opened PDB using existing DB; lacks PersonDB cursors.
-        # q_person: SELECT * FROM person ORDER BY pid
-        pdb.q_person = pdb.db.cursor_for('SELECT * FROM person ORDER BY pid')
-
-        # Get all persons for validation
-        all_persons = pdb.list_persons()
-        all_pids = {person.pid for person in all_persons}
-
-        # Temporary hack: Map old PIDs to newer equivalents for testing.
-        # TODO: Remove once PID data is fully migrated.
-        _REMAP = {
-            'iroh': 'wells',
-        }
-
-        # Validate and add voters from record
         for pid in record:
-            pid = _REMAP.get(pid, pid)  # Apply remapping if needed
-            if pid not in all_pids:
-                raise ValueError(f'PID {pid} from record not found in person database')
             election.add_voter(pid)
-        _LOGGER.info(f'Added {len(record)} voters to election[E:{election.eid}]')
-
-        # pdb.db.conn.execute('COMMIT')
-        _LOGGER.info(f'Election[E:{election.eid}] fully created from {yaml_file}')
-
-    except Exception as e:
-        # pdb.db.conn.execute('ROLLBACK')
-        _LOGGER.error(f'Failed to create election from {yaml_file}: {e}')
-        raise
+    _LOGGER.info(
+        f'Created election[E:{election.eid}] with {len(record)} eligible voters'
+    )
+    return election.eid
 
 
 if __name__ == '__main__':