Skip to content

troubleshooting.md

Latest commit

 

History

History
374 lines (255 loc) · 8.08 KB

troubleshooting.md

File metadata and controls

374 lines (255 loc) · 8.08 KB
Raw

Troubleshooting Guide

This guide covers problems shared across multiple labs. Each lab README remains the source of truth for lab-specific prerequisites, commands, expected output, and cleanup.

Do not apply generic cleanup commands to every lab. Some labs intentionally create evidence files, Kubernetes resources, Docker Swarm state, Vault data, or local registries.

Before Troubleshooting

Capture the environment first:

docker version
docker info
docker compose version
uname -a

For Kubernetes-based labs:

kubectl version --client
kubectl cluster-info
kubectl get nodes -o wide

When opening an issue, include:

  • Operating system and architecture
  • Docker Engine or Docker Desktop version
  • Docker Compose version
  • Lab number and exact command
  • Full error output
  • Relevant container logs
  • Changes made from the documented lab instructions

Docker Daemon and Permissions

Permission Denied Connecting to Docker

Typical error

permission denied while trying to connect to the Docker daemon socket

Linux

sudo usermod -aG docker "$USER"
newgrp docker
docker info

Log out and back in if the new group membership is not applied.

Membership in the docker group grants daemon-level control and should be treated as root-equivalent access.

macOS and Windows

Confirm Docker Desktop is running, then retry:

docker info

Docker Daemon Not Running

Linux

sudo systemctl status docker
sudo systemctl start docker

Docker Desktop

Open Docker Desktop and wait until the engine reports that it is running.

Docker Compose Compatibility

Use Compose v2 syntax:

docker compose version
docker compose up
docker compose down

Some older examples may use docker-compose. Prefer docker compose unless a lab explicitly requires otherwise.

Port Conflicts

Typical error

port is already allocated

Find the process or container using the port:

docker ps --format 'table {{.Names}}\t{{.Ports}}'
lsof -i :8080

Stop the conflicting workload or update the lab's documented port mapping. Do not arbitrarily change internal service ports without reviewing dependent configurations.

Image Pull and Registry Authentication

Inspect the exact pull error:

docker pull IMAGE_NAME
docker login REGISTRY_HOST

Common causes:

  • Authentication required
  • Incorrect registry hostname
  • Unsupported architecture
  • Rate limits
  • Missing or mistyped image tag
  • Corporate proxy or TLS interception

For Docker Hardened Images, follow the authentication instructions in Lab 12.

Script Permission and Line-Endings

Permission Denied Running a Script

chmod +x script-name.sh
./script-name.sh

/bin/bash^M or bad interpreter

The file likely uses Windows CRLF line endings.

sed -i.bak 's/\r$//' script-name.sh
chmod +x script-name.sh

On Linux with dos2unix installed:

dos2unix script-name.sh

Memory and Resource Problems

Container Killed by OOM

Inspect the container:

docker inspect CONTAINER_NAME \
  --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.State.Error}}'

Review current usage:

docker stats --no-stream

For Docker Desktop, increase the VM memory allocation when the lab requires more memory.

For Compose workloads, inspect the lab's resource configuration before changing limits. Lab 11 intentionally tests OOM behavior, so an OOM event may be expected rather than an environment failure.

Disk Space Exhaustion

docker system df
df -h

Review objects before deleting them:

docker ps -a
docker image ls
docker volume ls
docker network ls

Avoid docker system prune --volumes unless you understand which lab evidence and persistent data will be removed.

Falco Installation Issues

Typical symptoms

  • Falco fails to start
  • Kernel module cannot load
  • Driver or probe mismatch

On Linux, verify kernel headers:

uname -r
sudo apt-get update
sudo apt-get install "linux-headers-$(uname -r)"

Review Falco's current driver-selection guidance before forcing a specific kernel module or eBPF mode:

The previous troubleshooting file included a privileged docker run command with latest. It is intentionally not carried forward as a default fix because driver setup is version- and platform-dependent and privileged execution should not be suggested without context.

Security Tool Installation

Verify tools independently:

trivy --version
syft version
grype version
cosign version

Use official installation documentation:

The repository's shared setup instructions are in setup-guide.md. Lab-specific version requirements remain in the lab README.

Certificate and TLS Problems

Confirm OpenSSL is available:

openssl version

For Lab 08:

cd labs/08-network-security/certs
chmod +x generate-certs.sh
./generate-certs.sh

If TLS still fails, inspect:

  • Certificate subject and validity
  • File permissions
  • Container mount paths
  • Nginx configuration
  • Hostname used by the client
  • Whether the lab uses a self-signed certificate

Docker Networks

List networks:

docker network ls

Inspect the relevant network:

docker network inspect NETWORK_NAME

Use the cleanup script provided by the selected lab rather than deleting all Docker networks.

For Lab 08:

cd labs/08-network-security
./cleanup.sh

Kubernetes-Based Labs

Cluster Not Reachable

kubectl config current-context
kubectl cluster-info
kubectl get nodes

For kind-based labs:

kind get clusters
docker ps --filter name=kind

Admission Policy Does Not Trigger

Check policy and webhook status:

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
kubectl get clusterpolicies,policies -A
kubectl get policyreports,clusterpolicyreports -A

For Kyverno-specific problems, follow the Lab 12 documentation and inspect the Kyverno pods:

kubectl get pods -n kyverno
kubectl logs -n kyverno -l app.kubernetes.io/part-of=kyverno --tail=200

Secrets and Credentials

Never post real credentials in issues or logs.

Before sharing output, review it for:

  • API keys
  • Registry tokens
  • Vault tokens
  • Cloud credentials
  • SSH keys
  • .env values
  • Docker authentication files

Lab 11 requires generated local keys and an OpenAI API key. Lab 13 may interact with agent configuration and sandbox environments. Follow their README instructions exactly and use test-only credentials.

Lab-Specific Troubleshooting

Use these documents before opening a general issue:

Cleanup and Recovery

Before cleanup, record what the lab created:

docker ps -a
docker image ls
docker volume ls
docker network ls

Run the lab's cleanup script where provided:

./cleanup.sh

Some labs use different commands such as stop.sh, cleanup-all.sh, or platform teardown scripts. Read the lab README before removal.

Getting Help

  1. Re-run the failing command and capture its complete output.
  2. Review the selected lab README.
  3. Review this shared troubleshooting guide.
  4. Search existing repository issues.
  5. Open a new issue with sanitized diagnostic information.

Repository issues: