Report per operation timings in the multi editor and perspective tests - #4232
Open
vogella wants to merge 1 commit into
Open
Report per operation timings in the multi editor and perspective tests#4232vogella wants to merge 1 commit into
vogella wants to merge 1 commit into
Conversation
Migrates OpenMultipleEditorTest and PerspectiveSwitchTest the same way as OpenCloseEditorTest and EditorSwitchTest: drop org.eclipse.test.performance, warm up before measuring, time the individual operations and print the distribution through UIPerformanceTestUtil.reportTimings. The performance database these reported to has not been configured for years, so commitMeasurements and assertPerformance were no-ops and the tests could neither fail nor report anything. OpenMultipleEditorTest additionally reports the first and the last ten editors separately. Cost per editor grows with the number already open, and that is only visible from the two ends of the same run, which also makes the comparison immune to machine load. It also opens 90 instead of 100 editors: the workbench recycles the oldest editor once REUSE_EDITORS are open, which defaults to 99, so the hundredth open silently reused the first editor and the test measured reuse rather than opening. An assertion now pins that down. PerspectiveSwitchTest reports each switch direction separately and skips through an assumption when a perspective is missing, instead of printing to stdout and returning, which reported a pass for something never measured. Three of its four cases reference JDT perspectives and are skipped in this target platform, which is now visible in the test results.
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues #4230 for the two tests that cover the operations users feel most: opening many editors and switching perspectives. Both drop org.eclipse.test.performance, warm up before measuring, time the individual operations and print the distribution, so a change in one operation can be attributed instead of disappearing into a single number that nothing recorded.
OpenMultipleEditorTest also reports the first and the last ten editors separately, which turns it into the only test we have that shows how cost scales with the number of open editors. Opening the 90th editor currently costs 20 to 67 percent more than the 10th, and closing an editor while 90 are open costs about three times as much as when 10 remain. Because both ends come from the same run, that comparison is unaffected by machine load, which the absolute numbers very much are.
While adding assertions I found that the test never measured what it claimed: the workbench recycles the oldest editor once REUSE_EDITORS are open, defaulting to 99, so the hundredth open silently reused the first editor. The count is now 90 with an assertion explaining the threshold. PerspectiveSwitchTest had a related problem, silently returning and printing to stdout when a perspective is missing, which reported a pass for something never measured; it now skips through an assumption, so the three JDT cases show up as skipped rather than green.
Sample output of a local run: