Refactor rstests as tests

[ymcx]

As (shortly) discussed in #1078 and first introduced in #704 (comment).

Just like the title implies, this PR pretty much replaces all rstest unit tests along with their test cases with regular tests by manually iterating over all of the individual test cases one at a time.

Even though the refactor itself is quite simple and could easily have been and probably should have been automated, I did it completely by hand. I tried to keep all ignored test cases intact along with the comments if any, but I can't guarantee I didn't miss any. Other than that it should functionally match 1:1 to the previous behaviour.

I'm sorry for whoever has to go through the changelog as it's quite unreadable due to the sheer size of it.

[kronberger-droid]

I will give it a shot if i find some time.

[sholderbach]

That is certainly a tedious change to roll by hand. Going ahead with it should certainly current benefits into account instead of just the opinion of someone who doesn't spend their time anymore on this repo.
But my two cents regarding rstest are, that there is a significant benefit during debugging if you have those case matrices separated into their own test function to run. Thus they can run in parallel and fail and succeed individually, compared to loops of tests that fail at first assertion. With that you can share the same test logic and quickly drill down which invariant or property is broken instead of having to recompile and run until all tests in the chain pass. Nevertheless there are the additional proc-macro funs, so worth weighing what is important now.

On obvious downside is that the syntax for the case macros isn't the prettiest (but IMHO better than some of the pytest idiosyncracies that came from the python inspiration)

[kronberger-droid]

I have been investigating the root for the rtests change: it traces back to @stormasm in #704, for two reasons (1) it seemed to be blocking nextest and (2) he couldn't figure out how to #[ignore] individual cases.

Both look moot now. The nextest problem was traced to tests touching the terminal , not rtest and per-case ignore works fine now.

So i would agree with @sholderbach's objections, no parallel running of tests, no ignored in test output, no per-case pass/fail and cargo test name::case filtering.

I am very sorry @ymcx since you definitely put a lot of work into this PR, but I would lean towards keeping rtest.
Please correct me if I have understood something wrong and there was another reason to remove it. (#1078 just for rev)

[ymcx]

So i would agree with @sholderbach's objections, no parallel running of tests, no ignored in test output, no per-case pass/fail and cargo test name::case filtering.

Agreed. Honestly I should've opened an issue or something and first dialed down whether this is even needed or not. While writing #1078 and reading the commits/PRs related to it, I came across the comment you mentioned and figured it was still something worth taking into account when deciding between an rstest and a normal unit test for the one test I wrote for that PR.

Then, as a continuation of that PR, I figured replacing all rstests alltogether would be worthwhile, which was more so accelerated by fdncred's comment on that PR. Looking back I should've first verified whethever this refactoring was even needed at all instead of jumping straight back to my IDE.

I am very sorry @ymcx since you definitely put a lot of work into this PR

It's all good. At least the discussion we've had here can be used as a reference for any future decision on whether to use rstest or not.

I'll close this PR, as it seems quite clear now that using rstests makes more sense than having normal tests that naively iterate over all test cases in a loop.

[sholderbach]

Nevertheless thanks for being proactive and being willing to dive into the less glamorous parts :)

[stormasm]

@ymcx thank you for your effort !

Honestly I should've opened an issue or something and first dialed down whether this is even needed or not

yes, for future reference do this first 😄

stop by discord and just check in with some folks and get the feedback first

especially since when I filed my issue it was over two years ago and A LOT
has changed since then...

In any case, glad to have you participate on the Nushell projects...

And even though the PR was closed --- it got you more up to speed
on how all of the code is wired up...