Skip to content

docs: cover macOS libssh and stale operator key - #1072

Open
jklare wants to merge 2 commits into
mainfrom
manager-upgrade-guide-macos-and-operator-key
Open

docs: cover macOS libssh and stale operator key#1072
jklare wants to merge 2 commits into
mainfrom
manager-upgrade-guide-macos-and-operator-key

Conversation

@jklare

@jklare jklare commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Two problems that came up while actually walking through the manager
upgrade guide. Both of them are in the local venv path of
environments/manager/run.sh.

macOS: the venv build fails on a missing libssh

The venv installs ansible-pylibssh, and that package publishes wheels
only for Linux and for x86_64 macOS — there is no arm64 wheel. On
macOS with Apple Silicon uv therefore falls back to the source
distribution and compiles it, which fails on a missing libssh. The
seed chapter listed only the Ubuntu packages (libssh-dev), so the
Homebrew equivalent is added there, together with the CFLAGS /
LDFLAGS exports for the Homebrew prefix, and the upgrade guide points
at that section.

I ran into this myself and worked out the fix by hand. Installing the
two Homebrew packages was not enough on its own — the build only
started once CFLAGS and LDFLAGS were exported as well:

brew install libssh pkg-config
export CFLAGS="-I$(brew --prefix libssh)/include"
export LDFLAGS="-L$(brew --prefix libssh)/lib"

Worth deciding during review: the seed chapter currently presents the two
exports as a fallback ("if the build still does not find them"), but in
the run above they were required, so it may be better to promote them to
a normal part of the macOS steps.

Running the playbooks in the seed container avoids the build entirely, so
the docs now name that as the easier route on macOS.

A stale id_rsa.operator survives an operator key rotation

run.sh authenticates against the manager with
environments/manager/id_rsa.operator. It writes that file from
operator_private_key in environments/secrets.yml, but
only when the file does not exist yet.
Since id_rsa.operator is listed in .gitignore, a copy dropped there
by an earlier run survives every git pull unnoticed, and run.sh keeps
offering the revoked key after the operator key has been rotated. I hit
this after rotating my own key — the local checkout still had the old one
lying around.

This is now noted in the upgrade guide and added as an explicit step to
the operator key rotation guide.

Alternative: fix the key generation instead

Documenting rm -f environments/manager/id_rsa.operator is a workaround
for what is arguably a bug in osism/generics
(environments/manager/run.sh). If run.sh regenerated the file
unconditionally, or compared it against operator_private_key and
rewrote it on a mismatch, the stale key could never be used in the first
place — and the note added here could be dropped from the docs again.
Happy to open an issue or a patch there if we prefer that route.

Checks

  • MegaLinter documentation flavor: markdownlint and codespell clean
    on all three files
  • yarn build succeeds, so the new cross-guide links and the
    #option-2-manual-installation anchor resolve
  • The two commits were verified to be independently consistent: the
    combined diff is identical to the original squashed change, and
    markdownlint is clean against the intermediate tree

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown

⚠️MegaLinter analysis: Success with warnings

Descriptor Linter Files Fixed Errors Max errors Warnings Elapsed time
✅ ACTION actionlint 5 0 0 0.04s
✅ JSON jsonlint 4 0 0 0.08s
✅ JSON prettier 4 0 0 0.34s
✅ JSON v8r 4 0 0 9.21s
✅ MARKDOWN markdownlint 168 0 0 2.61s
✅ MARKDOWN markdown-table-formatter 168 0 0 0.43s
✅ REPOSITORY betterleaks yes no no 0.72s
✅ REPOSITORY checkov yes no no 19.29s
✅ REPOSITORY git_diff yes no no 0.05s
✅ REPOSITORY secretlint yes no no 3.45s
✅ REPOSITORY trufflehog yes no no 4.69s
✅ SPELL codespell 178 0 0 0.55s
⚠️ SPELL lychee 178 1 0 23.43s
✅ YAML prettier 6 0 0 0.41s
✅ YAML v8r 6 0 0 7.57s
✅ YAML yamllint 6 0 0 0.45s

Detailed Issues

⚠️ SPELL / lychee - 1 error
📝 Summary
---------------------
🔍 Total.........1045
🔗 Unique.........802
✅ Successful.....983
⏳ Timeouts.........2
🔀 Redirected.......5
👻 Excluded........60
❓ Unknown..........0
🚫 Errors...........0
⛔ Unsupported......0

Errors in docs/release-notes/osism-10.md
[TIMEOUT] https://gateway-api.sigs.k8s.io/ (at 831:3) | Request timed out
[TIMEOUT] https://headlamp.dev/ (at 834:70) | Request timed out

Hint: Followed 5 redirects. You might want to consider replacing redirecting URLs with the resolved URLs. Use verbose mode (`-v`/`-vv`) to see redirection details.

See detailed reports in MegaLinter artifacts

Your project could benefit from a custom flavor, which would allow you to run only the linters you need, and thus improve runtime performances. (Skip this info by defining FLAVOR_SUGGESTIONS: false)

  • Documentation: Custom Flavors
  • Command: npx mega-linter-runner@10.0.0 --custom-flavor-setup --custom-flavor-linters ACTION_ACTIONLINT,JSON_JSONLINT,JSON_V8R,JSON_PRETTIER,MARKDOWN_MARKDOWNLINT,MARKDOWN_MARKDOWN_TABLE_FORMATTER,REPOSITORY_CHECKOV,REPOSITORY_GIT_DIFF,REPOSITORY_BETTERLEAKS,REPOSITORY_SECRETLINT,REPOSITORY_TRUFFLEHOG,SPELL_LYCHEE,SPELL_CODESPELL,YAML_PRETTIER,YAML_YAMLLINT,YAML_V8R

MegaLinter is provided by OX Security
Show us your support by starring ⭐ the repository

jklare added 2 commits August 19, 2026 08:18
The local venv path of environments/manager/run.sh installs
ansible-pylibssh, and that package publishes wheels only for Linux and
for x86_64 macOS. On macOS with Apple Silicon there is no matching
wheel, so uv falls back to the source distribution and compiles it,
which fails on a missing libssh.

The seed chapter listed only the Ubuntu packages, so the Homebrew
equivalent is added there, together with the CFLAGS/LDFLAGS fallback for
the Homebrew prefix in case the build still does not find the library.
The manager upgrade guide points at that section, because the same venv
is built when steps 2 and 3 run outside the seed container.

Running the playbooks in the seed container avoids the build entirely,
so that is named as the easier route on macOS.

The workaround was tested by hand on macOS with Apple Silicon.

Assisted-by: Claude:claude-opus-5[1m]
Signed-off-by: Jan Klare <klare@osism.tech>
run.sh authenticates against the manager with
environments/manager/id_rsa.operator. It writes that file from
operator_private_key in environments/secrets.yml, but only when the file
does not exist yet, and id_rsa.operator is listed in .gitignore. A copy
dropped there by an earlier run therefore survives every git pull
unnoticed, and after the operator key has been rotated run.sh keeps
offering the revoked key until the file is deleted by hand.

This is noted in the manager upgrade guide, where the failure shows up
as an SSH login error in steps 2 and 3, and added as an explicit step to
the operator key rotation guide, which is where the file has to be
removed from every checkout that is used with run.sh.

Fixing the generation in osism/generics instead, so that run.sh rewrites
id_rsa.operator whenever it does not match operator_private_key, would
make the stale key unusable in the first place and would allow both
notes to be dropped again.

Assisted-by: Claude:claude-opus-5[1m]
Signed-off-by: Jan Klare <klare@osism.tech>
@jklare
jklare force-pushed the manager-upgrade-guide-macos-and-operator-key branch from 5d3d204 to 33ecb72 Compare August 19, 2026 08:20
@osfrickler osfrickler moved this from New to In review in Human Board Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In review

Development

Successfully merging this pull request may close these issues.

3 participants