AMD GPU test is, for some reason, looking into a completely different homedir for snap/rocm-validation-suite/common

[bladernr]

Bug Description

While looking into #2800 and trying to recreate the failure on a MI350P system AMD is hosting for us, I ran into this completely different issue.

Running test-gpgpu and all the AMD tests fail because the scripts are, for some reason, trying to run things from the wrong home directory. I don't know yet if this is something broken in the snap, or the scripts, or is an environmental issue on the host machine, but filing the bug for now to track it while I debug further.

Cert-blocker Test Case

• cert-blocker

To Reproduce

connect to the Chalupa system in the AMD lab
run test-gpgpu
tests fail immediately due to the noted traceback

Expected Result

rvs suite tests execute correctly

Actual Result

AMD gpu test cases all fail because they cannot find the scripts in the snap common dir.

Environment

AMD Chalupa (Turin)
AMD MI350P GPU
Ubuntu 26.04
Checkbox from edge:

canonical@chalapu-49b0-os:~$ dpkg -l |grep checkbox
ii  checkbox-ng                            7.5.0~dev20~ubuntu24.04.1                        all          CheckBoxNG test runner
ii  checkbox-provider-base                 7.5.0~dev20~ubuntu24.04.1                        amd64        Checkbox base provider
ii  checkbox-provider-certification-server 7.5.0~dev20~ubuntu24.04.1                        all          Server Certification provider for Checkbox
ii  checkbox-provider-gpgpu                7.5.0~dev20~ubuntu24.04.1                        amd64        Checkbox provider for GPGPU testing
ii  checkbox-provider-resource             7.5.0~dev20~ubuntu24.04.1                        amd64        Checkbox resource jobs provider
ii  python3-checkbox-ng                    7.5.0~dev20~ubuntu24.04.1                        all          CheckBoxNG test runner (Python 3 library)
ii  python3-checkbox-support               7.5.0~dev20~ubuntu24.04.1                        all          collection of Python modules used by PlainBox providers
canonical@chalapu-49b0-os:~$ snap info rocm-validation-suite 
name:      rocm-validation-suite
summary:   A system validation and diagnostics tool for AMD GPUs
publisher: Pedro Avalos
store-url: https://snapcraft.io/rocm-validation-suite
contact:   https://github.com/pedro-avalos/rocm-validation-suite-snap/issues
license:   MIT
description: |
  NOTE: This is a wrapper of the official package, but it is not verified,
  affiliated with, or supported by AMD.
  
  NOTE: Currently, the snap only works in `--devmode`. This is not recommended
  unless you know what you are doing. To learn more about the confinement
  levels, read https://snapcraft.io/docs/snap-confinement.
  
  The ROCm Validation Suite (RVS) is a system validation and diagnostics tool
  for monitoring, stress testing, detecting and troubleshooting issues that
  affects the functionality and performance of AMD GPU(s) operating in a
  high-performance/AI/ML computing environment. RVS is enabled using the ROCm
  software stack on a compatible software and hardware platform.
  
  RVS is a collection of tests, benchmarks and qualification tools each
  targeting a specific sub-system of the ROCm platform. All of the tools are
  implemented in software and share a common command line interface. Each set
  of tests are implemented in a "module" which is a library encapsulating the
  functionality specific to the tool. The CLI can specify the directory
  containing modules to use when searching for libraries to load. Each module
  may have a set of options that it defines and a configuration file that
  supports its execution.
commands:
  - rocm-validation-suite
snap-id:      2WcqXYYqyyxtL4F762BrFF6vWnPRpmOg
tracking:     latest/stable
refresh-date: 30 days ago, at 17:45 UTC
channels:
  latest/stable:    6.3.3 2025-03-04 (16) 2.25GB -
  latest/candidate: ↑                            
  latest/beta:      ↑                            
  latest/edge:      6.3.3 2025-03-01 (16) 2.25GB -
installed:          6.3.3            (16) 2.25GB -

Relevant log output

Traceback (most recent call last):
  File "/tmp/nest-mp8m0l42.b5a3e546969996e60afd0b2fbcab77a4f5ee4a0825a0d4846f8a384b7616e136/rvs.py", line 214, in <module>
    main()
  File "/tmp/nest-mp8m0l42.b5a3e546969996e60afd0b2fbcab77a4f5ee4a0825a0d4846f8a384b7616e136/rvs.py", line 207, in main
    runner = RVS_MODULES[args.module](args.rvs, args.config)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/nest-mp8m0l42.b5a3e546969996e60afd0b2fbcab77a4f5ee4a0825a0d4846f8a384b7616e136/rvs.py", line 54, in __init__
    shutil.copy(config, RVS_SNAP_COMMON)
  File "/usr/lib/python3.12/shutil.py", line 435, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.12/shutil.py", line 262, in copyfile
    with open(dst, 'wb') as fdst:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/amd/snap/rocm-validation-suite/common'

I am NOT running this using the amd user account, I am using a completely different account. Also of note, this test worked well the last time I was in there. the only difference is that I connected this time, did a dist-upgrade, and rebooted to get the updated kernel/driver loaded.

UPDATE: I found the problem:

Press enter to start Checkbox.
WARNING:checkbox_ng.user_utils:Using `amd` user

so user_utils I think is the culprit.

Additional context

No response

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-2321.

This message was autogenerated

[bladernr]

Adding this section to the launcher works around user_utils guess...

[agent]
normal_user = <username>

[bladernr]

For closure sake the ultimate issue is that when AMD set this up they renamed the Ubuntu account to amd, and then added two or three other accounts, so user_utils was piicking the wrong one sometimes.

Specifying it in the launcher by hacking the launcher works around it. I supposed specifying it via checkbox-cli remote will also work.