Use testcontainers to start acceptance tests fixtures

[mathieu-lemay]

Instead of having to manually start the docker containers for the tests, and then exporting FORGEJO_API_TOKEN as an environment variable, we use testcontainers. This makes running the tests literally as simple as make testacc. The containers will be started automatically, and the host and api token will be injected in the test's env automatically.

By default, testcontainers will delete containers after a period of inactivity. Keeping the containers can be desirable for quicker tests. This behaviour can be configured.

[acch]

Hi @mathieu-lemay - thanks much for this PR! I had not looked into testcontainers in detail, yet - appreciate that you bring this up!

I'm not entirely sure I fully get the benefit of your suggestion, though. I understand that you're removing complexity from the GitHub workflow definition; but on the other hand you're adding quite a bit of complexity in other areas (fixtures package).

Am I missing the point?

[mathieu-lemay]

@acch To me, the main benefit is for the developers working on this provider. It completely removes the need to start the docker compose, then manually executing commands to create an admin user and an API key, and finally exposing that API key as an environment variable. Running tests is now as trivial as make testacc or simply running one in your favorite IDE. It does simplify the GitHub workflow, but that's just a side effect.

I fully agree that it does add some complexity, but it's worth it in my opinion. For what it's worth, we're using the same concept at my workplace and I've only got positive feedback from my peers since running tests requires no setup whatsoever.

[acch]

Understood, thanks for the explanation! I'll look into this in greater detail... Certainly sounds interesting - I'll see if I can adopt this in my local dev setup.

[acch]

Hi @mathieu-lemay - sorry for the delay. I finally found some time to dig into test containers in more detail... again, many thanks for bringing this up!

I think I now understand the advantages of having a Go-native way of spinning up a temporary container for testing. Just to make sure I'm getting it right: your proposed changes would mean that every time I run make testacc, it would spin up a temporary container, run the tests, and then shut it down again - correct?

Is there a way to make this optional? I'm used to starting a permanent test instance at the beginning of my coding session (docker-compose start), then doing all kinds of experiments against it, which involves logging into the UI and trying things out, and then shutting it down at the end of my coding session (docker-compose stop). Your changes would mean that I'm not running acceptance tests against my permanent instance, but I'm spinning up a fresh, temporary container each and every time - correct?

(please excuse my beginner's question!)

[acch]

Let's add a constant for the version - I'd like to keep this up to date with the current LTS version...

[acch]

Hmm - after a bit more experimentation: I can't get it to work on my local setup...

My primary dev system is running Fedora Linux, which defaults to Podman. I usually code inside a Dev Container. Running make testacc from inside the Dev Container gives me:

provider_test.go:35: Error getting test containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I guess we would need to make the docker/podman socket available inside the container (.devcontainer/devcontainer.json?)...

Next up, running make testacc from outside - on the Fedora host - gives me:

ERROR run error="new reaper: ping: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head \"http://%2Fvar%2Frun%2Fdocker.sock/_ping\": dial unix /var/run/docker.sock: connect: permission denied"

(note that $DOCKER_HOST is set correctly, as per the documentation)

So while this might be an option for our CI setup, it will require more work to replace the local dev setup...

[mathieu-lemay]

Thanks for taking a look!

For your first question, I'm actually very glad you asked, because I had the exact same concern when I implemented this whole thing at work. By default, it behaves as you described, ie. the containers are cleaned after the test run, but it is definitely possible to keep the containers and re-use them.

The easiest way to do this is by exporting TESTCONTAINERS_RYUK_DISABLED=true. This will disable ryuk, the resource reaper, and your containers will stay up and running until you manually remove them. This is also my personal preference. More details here.

With this in place, the containers would start on the first test run and stay up until you decide to delete them. The only real difference right now would be that the port assigned to the forgejo API is dynamic, docker will assign it a random port. I've done this to avoid port clashes, but this is trivial to change, we can hardcode the ports to replicate what is done in the docker-compose file.

For your second question, this is indeed a problem. I don't use podman, nor devcontainers, so I hadn't thought of that. I'll setup a Fedora VM to experiment with this. This is a real blocker, because the point of this whole thing it to make life easier for devs. Re-using it in the CI is really just a matter of being consistent. Otherwise, I really don't see the point in doing this only for CI, your current solution is good enough. I'll get back to you!

[mathieu-lemay]

I've had the chance to check out podman a little bit and indeed I encountered the same issues as you did. Let's start with the issues I've found and how to fix them.

The first issue I had was due to the podman service not being started. As I understand it, unlike docker, podman works without a background service. However, due to testcontainers assuming docker, the service needs to be running. This emulates docker, and testcontainers is happy. I think you have that covered already.

The second issue is the same one you had on the host: new reaper: ping: permission denied while trying to connect to the Docker daemon socket at .... There are two ways around this: either set TESTCONTAINERS_RYUK_CONTAINER_PRIVILEGED=true or TESTCONTAINERS_RYUK_DISABLED=true. In your case, the 2nd option would be preferable because, like me, you want to keep the containers running between tests.

The third issue I had was related to SELinux permissions. The mounted app.ini file would be completely inaccessible from inside the docker. I could fix it by mounting with the :z flag, but that will change SELinux permissions of the file on the host, and I didn't like that. I've fixed that by copying the file inside the container instead (see here).

With the latest commit, you should be able to run the tests from this branch, at least on the host, by setting either of the RYUK env vars. If that works, I can see how to integrate that with devcontainers.