Skip to content

HTMX rewrite (IVS-1016) - #341

Open
aothms wants to merge 15 commits into
developmentfrom
htmx-rewrite
Open

aothms wants to merge 15 commits into
developmentfrom
htmx-rewrite

Conversation

@aothms

@aothms aothms commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

As promised hereby the AI directed rewrite to HTMX

The rewrite is the result of several quite lengthy sessions, first trying to come to a light weight datamodel to operate on, then the transformations from outcomes in the DB to this datamodel and then the HTMX templates to render out in the two ways (a) HTML report of single task (b) aggregated full report printed with weasyprint. Then I had AI build a system that traverses the UI reports (pagination, expand sections) to dump to a indented textual shape to do another structural comparison of the reports on a set of minimal but selected for wide coverage (incl. digital signatures, allowlisting etc.) of behavior.

Note that it's not only the report, the full UI (landing, dashboard, dropzone, etc.) is rewritten, I did a manual review of the CSS and personally think it's good enough, but some material ui aspects are not fully replicated.

Shoehorning things into a structured approach also made some inconsistencies surface like how we sort with a wrong comparator in JS, that we sometimes sort on severity before/after whitelisting, etc. Where it did not make sense to retain, these inconsistencies are addressed. So sometimes the reports are no longer identical.

I did not yet adapt the deployment nginx setup neither is the react ui deleted - so that we can do our own comparisons. Currently go to /v2/ for the HTMX based frontend.

I set USE_THOUSAND_SEPARATOR=False - I don't think users were previously seeing django-formatted values so this mostly affects us in the admin. If one of you is attached to it we can unlocalize in the templates instead.

The ui/ folder should still move some logic to the templates instead of building out page structures in dataclasses.

TOC

  • backend/apps/ifc_validation_bff/datamodel/ report/UI dataclasses and input loading, with shared validation statuses, severity handling, grouping, and pagination.
  • backend/apps/ifc_validation_bff/report/ Centralized report transformations, Gherkin lookups, value formatting, HTML endpoints, and PDF generation/storage.
  • backend/apps/ifc_validation_bff/{ui,templates,static,templatetags}/ Full HTMX frontend under /v2/ and shared web/PDF report templates, including navigation, uploads, dashboard interactions, icons, and styling.
  • backend/apps/ifc_validation/{tasks,management}/ Queues report generation after validation; preserves PDFs when retention removes original IFC files.

@aothms
aothms requested a review from Ghesselink November 10, 2026 13:49

@Ghesselink Ghesselink left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice :) I ran the branch locally, uploaded a couple of files and clicked through the UI; everything seems to work as expected. Regarding the UI, I don't see a lot of difference with react (but maybe I'm not the right person to ask about visual/ui appeal).

I set USE_THOUSAND_SEPARATOR=False - I don't think users were previously seeing django-formatted values so this mostly affects us in the admin. If one of you is attached to it we can unlocalize in the templates instead.

Not a lot of attachment here ..

So sometimes the reports are no longer identical.

I notice now, but it's quite a positive change imo.

  • Real errors come before whitelisted errors
  • The "errors and warnings only" filter is applied before the cap of 10, so the error will be shown before the passed outcomes

I took the liberty to commit a couple of things:

  • CI: Some checks were not running yet.
  • Retention test: Was not updated yet with your last commit (still expected pdf to be deleted).
  • PDF task: it retried three times on the time limit, and after an out-of-memory kill it was redelivered without end (180 times in 40 s in a test). It now does neither.

One point: what is our policy for PDFs?

  • Retention: When the IFC file goes, the PDF stays. Do we keep them forever? For reference; it's a roughly 14GB of disk space per year (not a reason for making a point imo).
  • Download after retention: Once the retention job has moved the IFC file itself (after 180 days), the PDF is still on disk but the user can no longer get it. The dashboard button is grey and the download link returns "PDF report not available". Should we change this? If the question on the point above is 'yes', that is. I've already prepared a commit and ran some tests, but wanted to discuss it beforehand.
  • Delete: the /v2/ delete removes the PDF. Deleting it through the React UI (although we'll delete this I suppose), the API or admin doesn't seem to work yet.

Comment on lines +81 to +85
outcomes = list(
ValidationOutcome.objects.filter(validation_task_id__in=latest.values())
.with_effective_severity().select_related("instance", "validation_task")
.order_by("validation_task_id", "id")
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loads all the outcomes of a task without the 10 per rule cap. I checked it with 60K schema outcomes (which is not even an outlier; there are tasks with 400K+ outcomes). It uses 150 MB in that case for one report page. My proposal would be to add a cap to it. Got a tested change locally (tested with 4 files, speed goes back from 2s to 0.6s and mem usage is 2MB), is it ok with you if I push it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes please. But should we from now on also limit the schema outcomes, maybe add a --max-results to ifcopenshell/validate.py similar to what we have in gherkin processing?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've implemented a query for the cap

Maybe add a --max-results to ifcopenshell/validate.py

Sounds good, I made a start and will open a PR on IfcOpenShell soon (added a note to Jira so I won't forget). Ran into an error with python -m ifcopenshell.validate --json on 0.9. Think the fix is doable, but not tested and deployed before Wednesday, and I'd really like to include this UI rewrite in the next release :-)

Comment thread docker/backend/Dockerfile Resolved
Comment on lines +25 to +29
def task_summary(task):
reason = task.reason
if task.status == 'Completed' and reason:
reason = None if reason.startswith('agg_status =') else reason.splitlines()[0]
return replace(task, reason=reason)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def task_summary(task):
reason = task.reason
if task.status == 'Completed' and reason:
reason = None if reason.startswith('agg_status =') else reason.splitlines()[0]
return replace(task, reason=reason)
def task_summary(task):
reason = None
if task.status == 'Skipped' and task.reason:
reason = task.reason.splitlines()[0]
elif task.status == 'Failed':
reason = 'This check could not be completed.'
return replace(task, reason=reason)

status_reason is written for admin/logs. Think we should only display a message with skip/fail.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. I just removed the entire column though. 28af011

<td>{{ check.warning_outcomes }}</td>
<td>{% if check.title == 'STEP Syntax' or check.title == 'IFC Schema' %}{% if check.status == 'v' %}✓{% else %}✗{% endif %}{% else %}{{ check.passed_rules }}{% endif %}</td>
<td>{% if check.title == 'STEP Syntax' or check.title == 'IFC Schema' %}-{% else %}{{ check.not_applicable_rules }}{% endif %}</td>
<td>{{ check.not_applicable_rules }}</td>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<td>{{ check.not_applicable_rules }}</td>

WeasyPrint throws this away (the line above fills in N/A)

@evandroAlfieri

Copy link
Copy Markdown
Collaborator

When is the pdf first generated? When the validation completes or when the user click to get it? If the latter, why not deleting it after 180 with its file? And if users want they go back in the validation table and re-download it (for us, re-generate)

@Ghesselink

Copy link
Copy Markdown
Contributor

When is the pdf first generated?

When the validation completes, for every upload.
After 180 days the IFC file is retained, but everything that is included in the pdf (outcomes, file name, header data) stays in the database. So we don't win a lot by removing it (other than disk space)

@civilx64 civilx64 changed the title HTMX rewrite HTMX rewrite (IVS-1016) Nov 14, 2026
@aothms

aothms commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for having a look @Ghesselink ! I did a couple more things so that we can toggle the frontend with an env var and port some of the tests. Let's see if we can make it by Wed... :)

@aothms

aothms commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Merge branch 'development' into htmx-rewrite

Indeed... do you have an overview of whether there are things that need to be ported to HTMX?

@Ghesselink

Copy link
Copy Markdown
Contributor

Indeed... do you have an overview of whether there are things that need to be ported to HTMX?

Yes. The only thing that needed porting was the header policy details.(merged this afternoon) I did it myself since it was still fresh in my head. The report PDF is now also a bit more informative for header errors.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants