Repository navigation
[Label] Allow specifying full directories/files as order targets #51
Description
Hello,
I've been fiddling with pytest-order for a personal project.
I wanted to order tests based on files and/or subdirectories, and I believe that it is not feasible at the moment with pytest-order.
Looking a bit into it, I've observed (correct me if I'm misunderstanding something) that the labels should mostly hint at discrete tests, and never at a bunch of other tests (all tests from a directory, file).
Is it possible to provide a feature such as full directory/file-based ordering ? I felt it should be, since the module already supports specifying multiple tests as before/after ordering labels.
I believe it may relate to #50, as the subject matter might end up cross paths.
Note: I've fiddled a bit with the code to obtain something that looks like it might work, but I might also be unknowingly passing-by some major problematics in doing so. If you're interested nonetheless, I can provide an initial PR to serve as a discussion basis, and hopefully contribute to obtaining said feature.
Cheers !
Activity
ltrujello commentedon Mar 1, 2023
@Joacchim would you mind sharing your stab at implementing this feature? I am also interested.
In my opinion, this feature would be much cooler than what pytest-order currently offers.
It is actually much more useful to specify the order your test files run rather than the individual tests. E.g., most projects that are complicated enough to have tests usually have hundreds of tests. Are the developers really going to label the order of every singe test function? Probably not, that's just too complicated. It'd be much more useful to order the files that the hundreds of tests are split up between.
For context my use case: I would like my very slow integration test file to run last. Because of alphabetical ordering, they run first. Thus, every time I make a new, fast test, I have to wait for my slow integration tests to run first. I still want that test to run of course, but it'd be more useful for my newly added fast test to fail first (failing fast helps you iterate faster).
mrbean-bremen commentedon Mar 1, 2023
@ltrujello - have a look at this question for your problem. I know it's not ideal, but at least it will work.
For the work @Joacchim has started - see his PR #52. If you want to have a shot, you can take that as a base. It is unlikely that I will get to it myself very soon, as I am busy with other projects at the moment.
jenstroeger commentedon May 2, 2025
I’d like to see this feature too: in my case I’d like to test an API and first test the signup endpoint (i.e. run the
test_signup.pymodule); next test the login endpoint (i.e. run thetest_login.pymodule) because it requires signup to work; next test the profile endpoint (i.e. run thetest_profile.pymodule) because it requires login to work; and so forth…mrbean-bremen commentedon May 2, 2025
@jenstroeger: have a look at #124. If this is all you want to do, you can add
in
test_signup.py,in
test_login.pyand so on.Thinking about this, you actually may not have to use
--order-group-scope=moduleas mentioned in the discussion issue, though I'm not quite sure about this, and it wouldn't hurt.jenstroeger commentedon May 2, 2025
@mrbean-bremen I think you’re right: the pytestmark global worked as you suggested, even without the cmd line arg 👍🏼
mrbean-bremen commentedon May 2, 2025
I'll probably add this to the documentation - this is a valid use case that is not obvious.
Document ordering of test modules
Document ordering of test modules