Repository navigation
Conversation
Ghesselink
left a comment
There was a problem hiding this comment.
Nice :) I ran the branch locally, uploaded a couple of files and clicked through the UI; everything seems to work as expected. Regarding the UI, I don't see a lot of difference with react (but maybe I'm not the right person to ask about visual/ui appeal).
I set USE_THOUSAND_SEPARATOR=False - I don't think users were previously seeing django-formatted values so this mostly affects us in the admin. If one of you is attached to it we can unlocalize in the templates instead.
Not a lot of attachment here ..
So sometimes the reports are no longer identical.
I notice now, but it's quite a positive change imo.
- Real errors come before whitelisted errors
- The "errors and warnings only" filter is applied before the cap of 10, so the error will be shown before the passed outcomes
I took the liberty to commit a couple of things:
- CI: Some checks were not running yet.
- Retention test: Was not updated yet with your last commit (still expected pdf to be deleted).
- PDF task: it retried three times on the time limit, and after an out-of-memory kill it was redelivered without end (180 times in 40 s in a test). It now does neither.
One point: what is our policy for PDFs?
- Retention: When the IFC file goes, the PDF stays. Do we keep them forever? For reference; it's a roughly 14GB of disk space per year (not a reason for making a point imo).
- Download after retention: Once the retention job has moved the IFC file itself (after 180 days), the PDF is still on disk but the user can no longer get it. The dashboard button is grey and the download link returns "PDF report not available". Should we change this? If the question on the point above is 'yes', that is. I've already prepared a commit and ran some tests, but wanted to discuss it beforehand.
- Delete: the
/v2/delete removes the PDF. Deleting it through the React UI (although we'll delete this I suppose), the API or admin doesn't seem to work yet.
| outcomes = list( | ||
| ValidationOutcome.objects.filter(validation_task_id__in=latest.values()) | ||
| .with_effective_severity().select_related("instance", "validation_task") | ||
| .order_by("validation_task_id", "id") | ||
| ) |
There was a problem hiding this comment.
This loads all the outcomes of a task without the 10 per rule cap. I checked it with 60K schema outcomes (which is not even an outlier; there are tasks with 400K+ outcomes). It uses 150 MB in that case for one report page. My proposal would be to add a cap to it. Got a tested change locally (tested with 4 files, speed goes back from 2s to 0.6s and mem usage is 2MB), is it ok with you if I push it?
There was a problem hiding this comment.
Yes please. But should we from now on also limit the schema outcomes, maybe add a --max-results to ifcopenshell/validate.py similar to what we have in gherkin processing?
There was a problem hiding this comment.
I've implemented a query for the cap
Maybe add a --max-results to ifcopenshell/validate.py
Sounds good, I made a start and will open a PR on IfcOpenShell soon (added a note to Jira so I won't forget). Ran into an error with python -m ifcopenshell.validate --json on 0.9. Think the fix is doable, but not tested and deployed before Wednesday, and I'd really like to include this UI rewrite in the next release :-)
| def task_summary(task): | ||
| reason = task.reason | ||
| if task.status == 'Completed' and reason: | ||
| reason = None if reason.startswith('agg_status =') else reason.splitlines()[0] | ||
| return replace(task, reason=reason) |
There was a problem hiding this comment.
| def task_summary(task): | |
| reason = task.reason | |
| if task.status == 'Completed' and reason: | |
| reason = None if reason.startswith('agg_status =') else reason.splitlines()[0] | |
| return replace(task, reason=reason) | |
| def task_summary(task): | |
| reason = None | |
| if task.status == 'Skipped' and task.reason: | |
| reason = task.reason.splitlines()[0] | |
| elif task.status == 'Failed': | |
| reason = 'This check could not be completed.' | |
| return replace(task, reason=reason) |
status_reason is written for admin/logs. Think we should only display a message with skip/fail.
There was a problem hiding this comment.
You're right. I just removed the entire column though. 28af011
| <td>{{ check.warning_outcomes }}</td> | ||
| <td>{% if check.title == 'STEP Syntax' or check.title == 'IFC Schema' %}{% if check.status == 'v' %}✓{% else %}✗{% endif %}{% else %}{{ check.passed_rules }}{% endif %}</td> | ||
| <td>{% if check.title == 'STEP Syntax' or check.title == 'IFC Schema' %}-{% else %}{{ check.not_applicable_rules }}{% endif %}</td> | ||
| <td>{{ check.not_applicable_rules }}</td> |
There was a problem hiding this comment.
| <td>{{ check.not_applicable_rules }}</td> |
WeasyPrint throws this away (the line above fills in N/A)
|
When is the pdf first generated? When the validation completes or when the user click to get it? If the latter, why not deleting it after 180 with its file? And if users want they go back in the validation table and re-download it (for us, re-generate) |
When the validation completes, for every upload. |
Co-authored-by: Geert Hesselink <54070862+Ghesselink@users.noreply.github.com>
|
Thanks for having a look @Ghesselink ! I did a couple more things so that we can toggle the frontend with an env var and port some of the tests. Let's see if we can make it by Wed... :) |
|
Indeed... do you have an overview of whether there are things that need to be ported to HTMX? |
Yes. The only thing that needed porting was the header policy details.(merged this afternoon) I did it myself since it was still fresh in my head. The report PDF is now also a bit more informative for header errors. |
As promised hereby the AI directed rewrite to HTMX
The rewrite is the result of several quite lengthy sessions, first trying to come to a light weight datamodel to operate on, then the transformations from outcomes in the DB to this datamodel and then the HTMX templates to render out in the two ways (a) HTML report of single task (b) aggregated full report printed with weasyprint. Then I had AI build a system that traverses the UI reports (pagination, expand sections) to dump to a indented textual shape to do another structural comparison of the reports on a set of minimal but selected for wide coverage (incl. digital signatures, allowlisting etc.) of behavior.
Note that it's not only the report, the full UI (landing, dashboard, dropzone, etc.) is rewritten, I did a manual review of the CSS and personally think it's good enough, but some material ui aspects are not fully replicated.
Shoehorning things into a structured approach also made some inconsistencies surface like how we sort with a wrong comparator in JS, that we sometimes sort on severity before/after whitelisting, etc. Where it did not make sense to retain, these inconsistencies are addressed. So sometimes the reports are no longer identical.
I did not yet adapt the deployment nginx setup neither is the react ui deleted - so that we can do our own comparisons. Currently go to /v2/ for the HTMX based frontend.
I set USE_THOUSAND_SEPARATOR=False - I don't think users were previously seeing django-formatted values so this mostly affects us in the admin. If one of you is attached to it we can unlocalize in the templates instead.
The ui/ folder should still move some logic to the templates instead of building out page structures in dataclasses.
TOC
backend/apps/ifc_validation_bff/datamodel/report/UI dataclasses and input loading, with shared validation statuses, severity handling, grouping, and pagination.backend/apps/ifc_validation_bff/report/Centralized report transformations, Gherkin lookups, value formatting, HTML endpoints, and PDF generation/storage.backend/apps/ifc_validation_bff/{ui,templates,static,templatetags}/Full HTMX frontend under /v2/ and shared web/PDF report templates, including navigation, uploads, dashboard interactions, icons, and styling.backend/apps/ifc_validation/{tasks,management}/Queues report generation after validation; preserves PDFs when retention removes original IFC files.