Skip to content

[benchmarker] Only inspect current step's operations when checking criteria - #1778

Closed
barroco wants to merge 1 commit into
interuss:mainfrom
Orbitalize:benchmarker-optimize-inspection
Closed

barroco wants to merge 1 commit into
interuss:mainfrom
Orbitalize:benchmarker-optimize-inspection

Conversation

@barroco

@barroco barroco commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

This PR improves the criteria evaluation process for large number of operations by only considering operations of interest for the current evaluated step.

@barroco
barroco force-pushed the benchmarker-optimize-inspection branch from 4b7d81f to 723d153 Compare October 7, 2026 11:50
@barroco
barroco marked this pull request as ready for review October 7, 2026 12:09
PERIODIC_STATUS_PERIOD_S = 30.0

OPERATION_ORDER_TOLERANCE = timedelta(seconds=60)
"""Operations are recorded as they complete, so they are ordered by completion time to within this tolerance."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why the operations are ordered within a 60 seconds tolerance because they are recorded as they complete?
Also: I think making this assumption here of logic defined elsewhere is not good. If that assumption changes (and I don't think it is guaranteed by anything?), this will break this logic here.
Maybe replace operations with a data structure that has some guarantees on the sorting?

)


def first_index_completed_since(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you considered instead doing something like summarize_and_report_step?

    step_ops = [
        op
        for op in operations
        if op.completed_at.datetime >= step_start_time
        and op.completed_at.datetime <= step_end_time
    ]

@barroco
barroco marked this pull request as draft October 7, 2026 20:51
@BenjaminPelletier

Copy link
Copy Markdown
Member

Is the purpose of this PR to reduce the processing time required by check_stability_criteria by reducing the number of operations to iterate over when checking various criteria? If so, I would probably expect a search for the cutoff index to use a more performant algorithm (like binary search) and then just pass the cutoff/start index as an additional parameter to check_stability_criteria rather than creating an entirely new list to pass to check_stability_criteria. Though I am surprised that the check_stability_criteria processing time would be appreciable, even with a pretty large volume of operations.

@barroco

barroco commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Thank you @BenjaminPelletier and @mickmis, let me re-open this issue with more evidence so we can better address root cause.

@barroco barroco closed this Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants