Skip to content

[WIP] Do not rely on vitest for memory tests - #1791

Open
peaBerberian wants to merge 1 commit into
devfrom
simple-test-lib
Open

[WIP] Do not rely on vitest for memory tests#1791
peaBerberian wants to merge 1 commit into
devfrom
simple-test-lib

Conversation

@peaBerberian

@peaBerberian peaBerberian commented Jan 23, 2026

Copy link
Copy Markdown
Collaborator

This is a low priority proposal.
Even if not merged, I plan to use it at least on my side to be able to run memory tests on my computer.


Until now, we relied on vitest + webdriverio to run our "memory tests", which check for memory leaks.

Those two are huge and complex libraries, a complexity that we don't need for those tests (or even for integration tests): vitest can readily patch native browser APIs, mock files, hoist logic blocks before tests - none of that is needed here.

More importantly: I always struggled to run them on my computer.
After some headaches, I had a container-based setup that worked (without a container it ended up picking the wrong network interface, or not finding the browser, or timeouting ...) but now it is broken again since very recently (the browser does not run, not sure what changed, but looking up why it broke each time is too time consuming).


Yet what we want to do in our memory tests is extremely simple by comparison to what those libs allow: we want to run a test page on a specific browser (with specific flags), then we want to log the results and finally exit either with success or failure depending on that result.

On that point we already have our own testing library for our "performance tests", because the paradigm of those tests is very different from the ones offered by the testing libs we know of (those tests compare the timings of some operations of two builds of the RxPlayer, by running them hundreds of time on each, and then check if there's a sensible diff with a null hypothesis test + some heuristics).

So vitest doesn't work for me, is very complex for reasons we don't need, and we already have a simpler setup which works for our need.

Adapting that setup for our memory tests was very straightforward. For now I copy-pasted the performance tests's run.js file (the test server + browser runner) and adapted it to memory tests just because I did not spend any time factorizing yet.

I also wrote here a small vitest/jest-compatible mininal testing library (providing describe / it / beforeEach etc. - and re-exporting vitest's assertions).
I use esbuild to bundle the test file and then run a browser on the output, exactly like for our performance tests.

@peaBerberian
peaBerberian force-pushed the simple-test-lib branch 22 times, most recently from 70a05b4 to 5f71d29 Compare January 26, 2026 12:48
@canalplus canalplus deleted a comment from github-actions Bot Jan 26, 2026
@canalplus canalplus deleted a comment from github-actions Bot Jan 26, 2026
@peaBerberian
peaBerberian force-pushed the dev branch 6 times, most recently from 374c14e to 7bf9f96 Compare January 26, 2026 17:50
@peaBerberian
peaBerberian force-pushed the dev branch 3 times, most recently from 0142e34 to 1fd9df3 Compare January 27, 2026 11:59
@canalplus canalplus deleted a comment from github-actions Bot Feb 27, 2026
@peaBerberian peaBerberian added the Priority: 4 (Very low) This issue or PR has a very low priority. label Apr 2, 2026
@peaBerberian
peaBerberian force-pushed the dev branch 7 times, most recently from 470c3f8 to 3dde189 Compare June 4, 2026 16:11
@peaBerberian
peaBerberian force-pushed the simple-test-lib branch 2 times, most recently from 1fbcb17 to 6e22e5e Compare August 7, 2026 08:54
@peaBerberian
peaBerberian force-pushed the dev branch 2 times, most recently from 01b00b9 to 3a8968e Compare August 7, 2026 15:08
@canalplus canalplus deleted a comment from github-actions Bot Aug 10, 2026
@canalplus canalplus deleted a comment from github-actions Bot Aug 10, 2026
@canalplus canalplus deleted a comment from github-actions Bot Aug 24, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

✅ Automated performance checks have passed on commit 8a1da0b781a2bc07e738bf1fba0cb63cc46e8935 with the base branch dev.

Details

Performance tests 1st run output

No significative change in performance for tests:

Name Mean Median
loading 34.15ms -> 33.92ms (corrected: 2.965ms, A/A bias: -2.737ms, z: 0.37528) 28.30ms -> 28.30ms (corrected: 0.050ms, A/A bias: -0.025ms, z: 0.02887)
seeking 619.04ms -> 638.80ms (corrected: -74.694ms, A/A bias: 54.931ms, z: 1.87639) 1007.00ms -> 1006.80ms (corrected: -0.800ms, A/A bias: 0.525ms, z: 1.12583)
audio-track-reload 30.28ms -> 30.11ms (corrected: 0.552ms, A/A bias: -0.381ms, z: 1.28942) 29.40ms -> 29.30ms (corrected: 0.300ms, A/A bias: -0.175ms, z: 0.89489)
loading over active content 20.53ms -> 20.31ms (corrected: 0.191ms, A/A bias: 0.036ms, z: 0.18764) 18.75ms -> 18.35ms (corrected: 0.050ms, A/A bias: 0.700ms, z: 0.18764)
large multi-period manifest 87.00ms -> 86.83ms (corrected: -0.377ms, A/A bias: 0.545ms, z: 0.38490) 81.75ms -> 81.15ms (corrected: -0.175ms, A/A bias: 2.025ms, z: 0.38490)
cold loading multithread 51.10ms -> 49.69ms (corrected: 1.653ms, A/A bias: -0.244ms, z: 4.53701) 49.50ms -> 48.40ms (corrected: 1.225ms, A/A bias: 0.050ms, z: 4.67654)
seeking multithread 208.92ms -> 231.22ms (corrected: -32.658ms, A/A bias: 10.352ms, z: 0.72169) 11.30ms -> 11.30ms (corrected: -0.400ms, A/A bias: 0.250ms, z: 0.69763)
audio-track-reload multithread 29.05ms -> 29.06ms (corrected: -0.255ms, A/A bias: 0.245ms, z: 0.54848) 28.10ms -> 28.45ms (corrected: -0.150ms, A/A bias: 0.050ms, z: 1.27498)
hot loading multithread 18.45ms -> 18.21ms (corrected: 0.088ms, A/A bias: 0.149ms, z: 1.36640) 17.70ms -> 17.40ms (corrected: 0.300ms, A/A bias: 0.000ms, z: 1.79941)

@canalplus canalplus deleted a comment from github-actions Bot Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Priority: 4 (Very low) This issue or PR has a very low priority. status-unknown

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant