diff --git a/README.md b/README.md index 3bc8026d..c8bfa4ec 100644 --- a/README.md +++ b/README.md @@ -101,3 +101,204 @@ jobs: - name: Run tests run: opam exec -- make test ``` + +## Development on xs-opam + +xs-opam's purpose is to provide packages for the ocaml xapi-project packages. +It's used also as what would now would be called a 'lock file', so most of the time only one version of a package is present in the repository to minimize the size of the tarballs produced from it and easily tell which version is used. +The exception to this rule is to allow the project to work for different versions of the OCaml compiler at the same time. + +### Setting up a working environment to develop on xs-opam + +Clone this repository, then use the file location to use it as the repository for a new switch, and install all the packages, like so: +```bash +opam switch create 9.0-xcp --repos=xs-opam-local=git+file://${PWD} ocaml.4.14.2 +opam install xs-toolstack dev-tools -y +``` + +This allows us to test upgrades and see that all packages are installable. + +### Finding upgradable packages + +This needs adding the default repository, but with lower priority that the local repository, as well as explicitly request to upgrade all packages, while locking some packages to a version. +Locking packages to a version is useful when newer versions of a package breaks its interface. Locking can be done by using the already-installed version, or using a specific version that's known to not contain breaking changes. + +This can be done like so: +```bash +opam repo add default && opam repo priority default 2 && OPAMNO=true opam upgrade --all tar.2.6.0 cohttp.5.3.1 +``` + +Output: +``` +<><> Updating package repositories ><><><><><><><><><><><><><><><><><><><><><><> +[xs-opam-local] no changes from git+file:///home/pau/code/xcp/ocaml/xs-opam +[default] no changes from https://opam.ocaml.org +The following actions will be performed: +=== recompile 174 packages + +[...] + +=== upgrade 75 packages + ↗ alcotest 1.9.0 to 1.9.1 [uses dune] + ↗ alcotest-lwt 1.9.0 to 1.9.1 [uses dune] + ↗ ambient-context 0.1.0 to 0.2 [uses dune] + ↗ base v0.16.3 to v0.16.5 [uses dune] + ↗ base64 3.5.1 to 3.5.2 [uses dune] + ↗ bos 0.2.1 to 0.3.0 [uses fmt] + ↗ ca-certs 1.0.1 to 1.0.3 [uses dune] + ↗ checkseum 0.5.2 to 0.5.3 [uses dune] + ↗ chrome-trace 3.20.2 to 3.23.1 [uses dune] + ↗ cohttp 5.3.1 to 6.1.1 [uses dune] + ↗ cohttp-lwt 5.3.0 to 6.1.1 [uses dune] + ↗ cohttp-lwt-unix 5.3.0 to 6.1.1 [uses dune] + ↗ conf-gmp 4 to 5 [required by conf-gmp-powm-sec] + ↗ conf-gmp-powm-sec 3 to 4 [required by mirage-crypto-pk] + ↗ conf-libev 4-12 to 4-13 [required by lwt] + ↗ conf-pkg-config 4 to 5 [required by dlm] + ↗ crowbar 0.2.1 to 0.2.2 [uses dune] + ↗ crunch 3.3.1 to 4.0.0 [uses dune] + ↗ ctypes 0.23.0 to 0.24.0 [uses dune] + ↗ ctypes-foreign 0.23.0 to 0.24.0 [uses dune] + ↗ domain-name 0.4.1 to 0.5.0 [uses dune] + ↗ dune 3.20.2 to 3.23.1 [required by tar] + ↗ dune-build-info 3.20.2 to 3.23.1 [uses dune] + ↗ dune-configurator 3.20.2 to 3.23.1 [uses dune] + ↗ dune-private-libs 3.20.2 to 3.23.1 [uses dune] + ↗ dune-rpc 3.20.2 to 3.23.1 [uses dune] + ↗ dune-site 3.20.2 to 3.23.1 [uses dune] + ↗ duration 0.2.1 to 0.3.1 [uses dune] + ↗ dyn 3.20.2 to 3.23.1 [uses dune] + ↗ eqaf 0.9 to 0.10 [uses dune] + ↗ ezcurl 0.2.4 to 0.3 [uses dune] + ↗ fix 20230505 to 20250919 [uses dune] + ↗ fmt 0.10.0 to 0.11.0 [required by alcotest] + ↗ integers 0.7.0 to 0.8.0 [uses dune] + ↗ ipaddr 5.6.0 to 5.6.2 [uses dune] + ↗ ipaddr-sexp 5.6.0 to 5.6.2 [uses dune] + ↗ lambda-term 3.3.2 to 3.4.0 [uses dune] + ↗ logs 0.8.0 to 0.10.0 [uses fmt] + ↗ lwt 5.9.1 to 5.10.0 [uses dune] + ↗ macaddr 5.6.0 to 5.6.2 [uses dune] + ↗ menhir 20240715 to 20260209 [uses dune] + ↗ menhirCST 20240715 to 20260209 [uses dune] + ↗ menhirLib 20240715 to 20260209 [uses dune] + ↗ menhirSdk 20240715 to 20260209 [uses dune] + ↗ mirage-crypto 2.0.1 to 2.1.0 [uses dune] + ↗ mirage-crypto-ec 2.0.1 to 2.1.0 [uses dune] + ↗ mirage-crypto-pk 2.0.1 to 2.1.0 [uses dune] + ↗ mirage-crypto-rng 2.0.1 to 2.1.0 [uses dune] + ↗ ocaml-version 4.0.0 to 4.1.2 [uses dune] + ↗ ocamlc-loc 3.20.2 to 3.23.1 [uses dune] + ↗ ocamlformat 0.28.1 to 0.29.0 [uses dune] + ↗ ocamlformat-lib 0.28.1 to 0.29.0 [uses dune] + ↗ ocamlformat-rpc-lib 0.28.1 to 0.29.0 [uses dune] + ↗ ocp-indent 1.8.1 to 1.9.0 [uses dune] + ↗ odoc 3.1.0 to 3.2.1 [uses dune] + ↗ odoc-parser 3.1.0 to 3.2.1 [uses dune] + ↗ opentelemetry 0.10 to 0.90 [uses dune] + ↗ opentelemetry-client-ocurl 0.10 to 0.90 [uses dune] + ↗ ordering 3.20.2 to 3.23.1 [uses dune] + ↗ pbrt 3.1.1 to 4.1 [uses dune] + ↗ polly 0.4.1 to 0.5.0 [uses dune] + ↗ ppx_deriving_rpc 9.0.0+2 to 10.2.0 [uses dune] + ↗ prometheus 1.2 to 1.3 [uses dune] + ↗ qcheck-alcotest 0.24 to 0.91 [uses dune] + ↗ qcheck-core 0.24 to 0.91 [uses dune] + ↗ re 1.12.0 to 1.14.0 [uses dune] + ↗ rpclib 9.0.0+2 to 10.2.0 [uses dune] + ↗ rpclib-lwt 9.0.0+2 to 10.2.0 [uses dune] + ↗ stdune 3.20.2 to 3.23.1 [uses dune] + ↗ topkg 1.0.8 to 1.1.1 [required by fmt] + ↗ tyre 0.5 to 1.0 [uses dune] + ↗ utop 2.16.0 to 2.17.0 [uses dune] + ↗ uucp 16.0.0 to 17.0.0 [uses topkg] + ↗ uuseg 16.0.0 to 17.0.0 [uses topkg] + ↗ xdg 3.20.2 to 3.23.1 [uses dune] +=== install 7 packages + ∗ fs-io 3.23.1 [required by stdune] + ∗ http 6.1.1 [required by cohttp] + ∗ menhirGLR 20260209 [required by menhir] + ∗ opentelemetry-client 0.90 [required by opentelemetry-client-ocurl] + ∗ pbrt_yojson 4.1 [required by opentelemetry] + ∗ thread-local-storage 0.2 [required by ambient-context] + ∗ top-closure 3.23.1 [required by stdune] + +Proceed with ↻ 174 recompilations, ↗ 75 upgrades and ∗ 7 installations? [Y/n] n +``` + +### Upgrade a new package version + +Now we're ready to start updating packages, giving special attention to the package that come from the same source repository which need to be updated at the same time. + +To update the metadata, a helper is recommended, like this one: +```fish +function opam-update + if test (count $argv) -ne 3 + echo "opam_update: expected 3 args" + echo "Usage: oswitch " + return -1 + end + set name "$argv[1]" + set old "$argv[2]" + set new "$argv[3]" + git mv packages/$name/$name.$old packages/$name/$name.$new + opam show --raw $name.$new > packages/$name/$name.$new/opam +end +``` + +This helper moves the opam metadata file for a package to the new version, and replace its contents with metadata from the default repository. + +For example, to update all the dune packages at the same time, such a command can be used: +```fish +for pkg in chrome-trace dune dune-build-info dune-configurator dune-rpc dyn ocamlc-loc ordering stdune xdg; and opam-update $pkg 3.18.2 3.20.2; end +``` + +Now the change should be committed before testing it, or opam will ignore the changes. + +When that is done, the change can be tested by removing the default repository and upgrading opam: +```bash +opam repo remove default && opam update -u +``` + +To repeat the step re-add the default repository so the metadata can be fetched by the helper again. + + +### Add a new package +Similar to upgrading a package, a helper is useful for this, which works when the switch has enabled the default repository: +```fish +function opam-new + if test (count $argv) -lt 2 + echo "opam_new: expected a least 2 args" + echo "Usage: oswitch " + return -1 + end + set name "$argv[1]" + set ver "$argv[2]" + set package "$name.$ver" + mkdir -p "packages/$name/$package" + opam show --raw "$package" > "packages/$name/$package/opam" +end +``` + +One thing to note is that if it meant to be used, it needs to be added as a dependency to the metapackage of xs-toolstack or dev-tools, if it's meant to be packaged; or to the ignored packages, if it's only meant to be used for testing a feature that's not package yet. + + +### Updating stale metadata + +To update the stale metadata a (surprise, surprise) helper can be used yet again. +It also needs to have the default repository in the loaded switch: + +```fish +function opam-update-upstream + opam repo add default + opam repo priority default 1 + for dir in packages/*/* + set -l versioned (path basename $dir) + if string match --regex -v '.master' $versioned + opam show --raw "$versioned" > "$dir/opam" + end + end +end +``` + +The helper only works on the master branch and not LCM branch, but the versions in the LCM are already outdated with respect of the upstream versions available, so they usually don't require updating.