diff --git a/README.md b/README.md index a52430bbf..47133028b 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ Starting with v1.0.0, Terratest follows [semantic versioning](https://semver.org only happen in major releases (e.g. v2.0.0). Symbols renamed or replaced in v1 are kept with `// Deprecated:` annotations pointing at the new name; removals happen -in v2. Migrating from v0.x: see the [v1 migration guide](https://terratest.gruntwork.io/docs/migrating-to-v1/overview/). +in v2. Migrating from v0.x: see the [v1 migration guide](https://terratest.gruntwork.io/docs/migrating-to-v1/overview/). Migrating from v1 to v2: see the [v2 migration guide](https://terratest.gruntwork.io/docs/migrating-to-v2/overview/). **v1 maintenance.** With v2 in development, the v1 line has entered maintenance: it receives security fixes only, delivered on the `v1` branch, until 12 months after v2.0.0 reaches general availability. v2 ships under new `/v2` diff --git a/docs/README.md b/docs/README.md index 12ddbf74a..8caf3617d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -89,7 +89,7 @@ nav_title_link: /docs/ # X Cannot be changed * `layout` - do not change! (Layout sets components like a navigation sidebar, page header, footer, etc.) * `title` - document title -* `categories` - the document's category. Four categories are in use for now: "getting-started", "testing-best-practices", "alternative-testing-tools", and "community". +* `category` - the document's category. The categories in use are: 'getting-started', 'testing-best-practices', 'migrating-to-v1', 'migrating-to-v2', and 'community'. * `excerpt` - description. Try to keep it short. * `tags` - check other posts to see common tags, but you can set a new as well. * `order` - it is used to sort the documents within collection. @@ -186,7 +186,7 @@ nav_title_link: /docs/ # OPTIONAL * categories - set one category. Use downcase with dashes, e.g. `getting-started`. * excerpt - the doc description. * tags - doc tags. -* order - it is use to list documents in the right order. "Getting Started" starts from 100, "Features" starts from 200, and "Community" starts from 300. +* order - it is used to list documents in the right order. "Getting Started" starts from 100, "Testing Best Practices" from 200, "Migrating to v1" from 300, "Migrating to v2" from 400, and "Community" from 500. * nav_title - the title above navigation. It's optional. It's a link if `nav_title_link` is set. * nav_title_link - it is a URL. If it is set, `nav_link` is transformed to the link. diff --git a/docs/_docs/01_getting-started/version-pinning.md b/docs/_docs/01_getting-started/version-pinning.md index 1da74509c..4579adb8a 100644 --- a/docs/_docs/01_getting-started/version-pinning.md +++ b/docs/_docs/01_getting-started/version-pinning.md @@ -25,3 +25,5 @@ Replace `v0.56.0` with the tag you want from the [Releases page](https://github. ## Use `go get`, not subpath requires `go get github.com/gruntwork-io/terratest@` is enough; the pin applies to every subpackage your tests import. Avoid writing subpath requires by hand (e.g. `go mod edit -require github.com/gruntwork-io/terratest/modules/terraform@`); Go searches for a `modules/terraform/` tag and fails with `unknown revision`. + +This applies to v1. In v2 each module is published separately and does carry its own `modules//` tag, so you require them individually. See [migrating to v2]({{site.baseurl}}/docs/migrating-to-v2/overview/). diff --git a/docs/_docs/03_migrating-to-v1/overview.md b/docs/_docs/03_migrating-to-v1/overview.md index c3ebcb72e..e46f26627 100644 --- a/docs/_docs/03_migrating-to-v1/overview.md +++ b/docs/_docs/03_migrating-to-v1/overview.md @@ -15,6 +15,9 @@ changes to the public API only happen in major releases (e.g. v2.0.0), per [semver](https://semver.org/). Renamed or replaced symbols stay around as deprecated aliases inside v1; full removal is deferred to v2. +Already on v1 and looking for v2? See +[migrating to v2]({{site.baseurl}}/docs/migrating-to-v2/overview/). + This page is the orientation map for v0.x to v1.0.0. It tells you what shape the changes take and where to look; the per-service guides hold the mechanical details. diff --git a/docs/_docs/04_migrating-to-v2/behavior-changes.md b/docs/_docs/04_migrating-to-v2/behavior-changes.md new file mode 100644 index 000000000..aa6c7335f --- /dev/null +++ b/docs/_docs/04_migrating-to-v2/behavior-changes.md @@ -0,0 +1,87 @@ +--- +layout: collection-browser-doc +title: Behavior changes +category: migrating-to-v2 +excerpt: >- + The two v2 changes the compiler will not find for you. Both are in the + k8s module. +tags: ["migration", "v2"] +order: 403 +nav_title: Documentation +nav_title_link: /docs/ +--- + +Everything else in the v2 migration is a compile error. These two are not: +your code builds and behaves differently. Both are in `k8s`, so skip this +page if you do not use it, and see the [v2 +overview]({{site.baseurl}}/docs/migrating-to-v2/overview/) for the rest of +the migration. + +## Node addresses prefer `ExternalIP` + +`k8s.FindNodeHostnameContextE`, and `GetServiceEndpointContextE` for a +NodePort service, now return the `ExternalIP` recorded on the Node object +when one is present. They fall back to the internal hostname exactly as +before when it is not. + +Cloud controller managers record an instance's public IP as an +`ExternalIP`. Terratest previously ignored that field and, on AWS, called +`ec2:DescribeInstances` to find the public IP instead. Reading the Node +object is the same answer without the API call, so: + +- `ec2:DescribeInstances` is no longer needed for this path +- `k8s` no longer depends on the `aws` module, which removes 22 AWS service + clients from its dependency graph + +**What to check.** If your cluster advertises an `ExternalIP` and your test +previously received an internal hostname, it now receives the external +address. That is the documented behavior and almost certainly what you +wanted, but it is a different string. Tests that assert on the endpoint +value, or that rely on reaching the node over its internal address, are the +ones to look at. + +Signatures are unchanged. The `ExternalIP` preference applies to every +provider. For AWS-backed nodes that advertise no `ExternalIP`, a new pair of +functions takes `*KubectlOptions` and consults +a lookup you provide: + +```go +options := k8s.NewKubectlOptions("", kubeconfig, "default") +options.NodePublicIPLookup = aws.GetPublicIpsOfEc2InstancesContextE + +hostname, err := k8s.FindNodeHostnameWithOptionsContextE(t, ctx, options, node) +``` + +`NodePublicIPLookup` is only consulted after the Node's own `ExternalIP` +has been checked, so most callers can leave it nil. + +## `KubectlOptions` carrying a `RestConfig` cannot be saved + +`json.Marshal` of a `KubectlOptions` built by +`NewKubectlOptionsWithRestConfig` now fails with an error wrapping +`k8s.ErrRestConfigNotSerializable`. Match it with `errors.Is`, not `==`: +`encoding/json` returns a `*json.MarshalerError` around it. This affects +`k8s.SaveKubectlOptions`, `teststructure.SaveTestData`, and any code of +your own that marshals options. + +This already failed in v1, with an opaque `json: unsupported type: +transport.WrapperFunc`, because `rest.Config` holds func-typed fields that +`encoding/json` rejects. What changed is that the error now says what is +wrong and what to do instead. + +It is deliberately an error rather than silently dropping the config. +Dropped, the reloaded options would carry no cluster identity at all, fall +back to the ambient kubeconfig, and run your test against a different +cluster. + +**What to do.** For staged tests, build options from a kubeconfig path or +from in-cluster auth. Both round trip: + +```go +options := k8s.NewKubectlOptions(contextName, configPath, namespace) +// or +options := k8s.NewKubectlOptionsWithInClusterAuth() +``` + +If you need a `rest.Config` at runtime, keep building one, but rebuild it +in each stage rather than saving it. diff --git a/docs/_docs/04_migrating-to-v2/import-map.md b/docs/_docs/04_migrating-to-v2/import-map.md new file mode 100644 index 000000000..d491098f0 --- /dev/null +++ b/docs/_docs/04_migrating-to-v2/import-map.md @@ -0,0 +1,80 @@ +--- +layout: collection-browser-doc +title: Import map +category: migrating-to-v2 +excerpt: >- + Every v1 import path and what it becomes in v2. +tags: ["migration", "v2"] +order: 402 +nav_title: Documentation +nav_title_link: /docs/ +--- + +The complete v1 to v2 path mapping. See [rewriting +imports]({{site.baseurl}}/docs/migrating-to-v2/rewriting-imports/) for how +to apply it in bulk. + +## Collapsed into `core` + +Six utility packages became subpackages of one module. The package +identifier at call sites is unchanged. + +| v1 | v2 | +|---|---| +| `modules/random` | `modules/core/v2/random` | +| `modules/files` | `modules/core/v2/files` | +| `modules/logger` | `modules/core/v2/logger` | +| `modules/shell` | `modules/core/v2/shell` | +| `modules/retry` | `modules/core/v2/retry` | +| `modules/testing` | `modules/core/v2/testing` | + +`modules/logger/parser` becomes `modules/core/v2/logger/parser`. + +`modules/core/v2/formatting` is new in v2 and has no v1 equivalent to +migrate: it was `internal/lib/formatting`, which was never importable. + +## Renamed + +Path *and* package identifier change. + +| v1 | v2 | +|---|---| +| `modules/http-helper`, `http_helper.X` | `modules/httphelper/v2`, `httphelper.X` | +| `modules/dns-helper`, `dns_helper.X` | `modules/dnshelper/v2`, `dnshelper.X` | +| `modules/test-structure`, `test_structure.X` | `modules/teststructure/v2`, `teststructure.X` | + +## Suffix only + +Path gains `/v2`; package identifier unchanged. + +| v1 | v2 | +|---|---| +| `modules/aws` | `modules/aws/v2` | +| `modules/azure` | `modules/azure/v2` | +| `modules/gcp` | `modules/gcp/v2` | +| `modules/k8s` | `modules/k8s/v2` | +| `modules/helm` | `modules/helm/v2` | +| `modules/ssh` | `modules/ssh/v2` | +| `modules/docker` | `modules/docker/v2` | +| `modules/packer` | `modules/packer/v2` | +| `modules/database` | `modules/database/v2` | +| `modules/opa` | `modules/opa/v2` | +| `modules/terraform` | `modules/terraform/v2` | +| `modules/terragrunt` | `modules/terragrunt/v2` | + +## Removed + +| v1 | Replacement | +|---|---| +| `modules/collections` | stdlib `slices` | +| `modules/environment` | stdlib `os.Getenv` | +| `modules/git` | stdlib `os/exec` | +| `modules/slack` | none; vendor from v1 if needed | +| `modules/version-checker` | none; shell out | +| `modules/oci` | none; Oracle Cloud is not carried forward to v2 | +| `cmd/pick-instance-type` | none | +| `cmd/terratest_log_parser` | none as a binary; the library survives at `modules/core/v2/logger/parser` | + +These were deprecated in v1 first and deleted at the v2 cutover. If you +depend on `slack`, `version-checker` or `oci`, v1 stays available and is +the place to stay. diff --git a/docs/_docs/04_migrating-to-v2/overview.md b/docs/_docs/04_migrating-to-v2/overview.md new file mode 100644 index 000000000..b764f8d07 --- /dev/null +++ b/docs/_docs/04_migrating-to-v2/overview.md @@ -0,0 +1,159 @@ +--- +layout: collection-browser-doc +title: v2 overview +category: migrating-to-v2 +excerpt: >- + What changes when you move from Terratest v1 to v2, and the order to do + it in. +tags: ["migration", "v2"] +order: 400 +nav_title: Documentation +nav_title_link: /docs/ +--- + +Terratest v2 splits the single `github.com/gruntwork-io/terratest` module +into 16 independent modules, so you depend only on the parts you use. A +test that imports `terraform` no longer pulls in the AWS SDK, client-go, +and every other provider's dependencies. + +The cost is that every import path changes. This page tells you what the +changes are and the order to apply them; the per-topic pages hold the +details. + +## Should you migrate yet + +v2 is in beta. v1 is in maintenance and receives security fixes only, until +12 months after v2.0.0 reaches general availability. Migrate now if you want +the smaller dependency graph or are starting fresh; wait for v2.0.0 if you +would rather not track beta releases. + +You can migrate incrementally. v1 and v2 import paths differ, so both can +coexist in one module while you convert package by package. The change +touches only `.go` files, `go.mod` and `go.sum`, so `git checkout` undoes +it. + +The minimum Go version is unchanged. + +## The five changes + +**1. Everything v1 deprecated is gone.** v1 kept deprecated aliases +alongside their replacements; v2 deletes them. This is the largest edit in +the migration, and it is not only the `Context` variants: + +- non-`Context` wrappers: `terraform.Apply` is now only `ApplyContext` +- initialism renames: `random.UniqueId` is now `UniqueID`, + `aws.GetAccountIdE` is now `GetAccountIDContextE` +- reshaped helpers: `packer.BuildAmi` is now `BuildArtifactContextE` + +The reliable way to find all of it is to run staticcheck against your +existing v1 code and clear every SA1019 (deprecated symbol) warning before +you touch imports. Once v1 is warning-free, the rest of this guide applies. + +```go +// v1 +out := terraform.Apply(t, options) +// v2 +out := terraform.ApplyContext(t, t.Context(), options) +``` + +The `Context` variants always take `(t, ctx, ...originalArgs)`. +`t.Context()` is the best default; `context.Background()` also works. The +[v1 guide]({{site.baseurl}}/docs/migrating-to-v1/overview/) covers this +migration in detail, and doing it on v1 first, where both forms still +compile, is easier than doing it at the same time as the import rewrite. + +**2. Import paths gain a `/v2` suffix.** The `/v2` goes after the module +root, not at the end of the path: + +```go +// v1 +"github.com/gruntwork-io/terratest/modules/terraform" +// v2 +"github.com/gruntwork-io/terratest/modules/terraform/v2" +``` + +**3. Six utility packages collapse into `core`.** `random`, `files`, +`logger`, `shell`, `retry` and `testing` are no longer separate packages: + +```go +// v1 +"github.com/gruntwork-io/terratest/modules/random" +// v2 +"github.com/gruntwork-io/terratest/modules/core/v2/random" +``` + +**4. Three packages are renamed to drop the hyphen.** This changes the +package identifier at call sites, not just the import path: + +`http-helper` becomes `httphelper`, `dns-helper` becomes `dnshelper`, and +`test-structure` becomes `teststructure`. The full table is in the [import +map]({{site.baseurl}}/docs/migrating-to-v2/import-map/#renamed). + +**5. Each module needs its own `require`.** Where v1 was one line in +`go.mod`, v2 needs one per module you import. Add them with `go get` +rather than by hand: + +```bash +go get github.com/gruntwork-io/terratest/modules/terraform/v2@v2.0.0-beta.2 +go get github.com/gruntwork-io/terratest/modules/aws/v2@v2.0.0-beta.2 +``` + +Each module is tagged `modules//vX.Y.Z`, so the tag for the command +above is `modules/terraform/v2.0.0-beta.2`. The current version is on the +[releases page](https://github.com/gruntwork-io/terratest/releases). All 16 +modules are released together and their cross-module requires are pinned to +the release version, so keep them on the same version. + +The full path mapping is in [the import +map]({{site.baseurl}}/docs/migrating-to-v2/import-map/). + +## Order to do it in + +1. Move to the `Context` variants, ideally while still on v1 so both forms + compile. +2. Rewrite import paths and package identifiers. This is mechanical and the + compiler finds everything. +3. Fix the symbol relocations. Also compile errors, also mechanical. They + are listed under [rewriting + imports]({{site.baseurl}}/docs/migrating-to-v2/rewriting-imports/). +4. Add a `require` per module, then `go mod tidy`. +5. Review the [behavior + changes]({{site.baseurl}}/docs/migrating-to-v2/behavior-changes/), which + the compiler will *not* find for you. There are two, both in `k8s`. + +Steps 1 to 4 are compiler-detectable, so the build tells you when they are +complete. Step 5 is not: the code compiles either way, so it needs reading +before you call the migration done. + +## Removed packages + +Six packages and two binaries are not carried forward. Three have a +standard library replacement: + +| v1 | Replacement | +|---|---| +| `modules/collections` | stdlib `slices` | +| `modules/environment` | stdlib `os.Getenv` | +| `modules/git` | stdlib `os/exec` | +| `modules/slack` | none; vendor from v1 | +| `modules/version-checker` | none; shell out | +| `modules/oci` | none; Oracle Cloud is not carried forward, stay on v1 | + +`cmd/pick-instance-type` and `cmd/terratest_log_parser` are gone as +binaries. The log parser's library survives at +`modules/core/v2/logger/parser`. + +## What did not change + +The `Foo` / `FooE` convention is unchanged: `FooContext` fails the test, +`FooContextE` returns an error. Beyond dropping the non-`Context` wrappers +and the relocations above, function arguments and return types are the +same. Test data written by v1 loads in v2, since filenames and JSON layout +are unchanged. + +## Need help + +Open an issue on the [Terratest +repo](https://github.com/gruntwork-io/terratest/issues) with the version +you are coming from and the error you hit. If you spot a gap in this +guide, send a PR against `docs/_docs/04_migrating-to-v2/`. diff --git a/docs/_docs/04_migrating-to-v2/rewriting-imports.md b/docs/_docs/04_migrating-to-v2/rewriting-imports.md new file mode 100644 index 000000000..4f0804bff --- /dev/null +++ b/docs/_docs/04_migrating-to-v2/rewriting-imports.md @@ -0,0 +1,124 @@ +--- +layout: collection-browser-doc +title: Rewriting imports +category: migrating-to-v2 +excerpt: >- + The mechanical part of the v2 migration, and the places a blind + find-and-replace gets it wrong. +tags: ["migration", "v2"] +order: 401 +nav_title: Documentation +nav_title_link: /docs/ +--- + +Every v2 import path changes. This page covers doing that in bulk, then the +symbol relocations, then the cases a scripted rewrite cannot finish on its +own. + +The commands below are written for BSD `sed`, which is what macOS ships. +On Linux, drop the `''` after `-i`. They use `#` as the delimiter rather +than `|`, because `|` collides with regex alternation, and they avoid `\b`, +which BSD `sed` does not support and silently ignores. + +## 1. Collapse the utility packages into `core` + +```bash +find . -name '*.go' -exec sed -i '' -E \ + 's#gruntwork-io/terratest/modules/(random|files|logger|shell|retry|testing)#gruntwork-io/terratest/modules/core/v2/\1#g' {} + +``` + +## 2. Rename the three hyphenated packages + +Path, package identifier, and any stale alias: + +```bash +find . -name '*.go' -exec sed -i '' -E \ + -e 's#gruntwork-io/terratest/modules/http-helper#gruntwork-io/terratest/modules/httphelper/v2#g' \ + -e 's#gruntwork-io/terratest/modules/dns-helper#gruntwork-io/terratest/modules/dnshelper/v2#g' \ + -e 's#gruntwork-io/terratest/modules/test-structure#gruntwork-io/terratest/modules/teststructure/v2#g' \ + -e 's#(^|[^A-Za-z0-9_])http_helper\.#\1httphelper.#g' \ + -e 's#(^|[^A-Za-z0-9_])dns_helper\.#\1dnshelper.#g' \ + -e 's#(^|[^A-Za-z0-9_])test_structure\.#\1teststructure.#g' \ + -e 's#^([[:space:]]*)(http_helper|dns_helper|test_structure) "#\1"#' {} + +``` + +That last expression matters. These packages were commonly imported under +an explicit alias: + +```go +test_structure "github.com/gruntwork-io/terratest/modules/test-structure" +``` + +Rewriting only the path leaves the old alias bound to the new package, so +every rewritten call site fails with `undefined: teststructure`. The +expression drops the alias so the package name applies. + +## 3. Add the `/v2` suffix to the rest + +```bash +find . -name '*.go' -exec sed -i '' -E \ + 's#gruntwork-io/terratest/modules/(aws|azure|gcp|k8s|helm|ssh|docker|packer|database|opa|terraform|terragrunt)([^/a-z]|$)#gruntwork-io/terratest/modules/\1/v2\2#g' {} + +``` + +## 4. Reformat + +The rewrite reorders import paths alphabetically, so the blocks are no +longer sorted: + +```bash +gofmt -w . +``` + +## Symbol relocations + +Eight functions moved to the module that owns the type they operate on, so +`teststructure` no longer requires `aws`, `k8s`, `packer` and `ssh`. +Signatures and on-disk filenames are unchanged, so this is a qualifier +swap. + +| v1 | v2 | +|---|---| +| `test_structure.{Save,Load}Ec2KeyPair` | `aws.{Save,Load}Ec2KeyPair` | +| `test_structure.{Save,Load}KubectlOptions` | `k8s.{Save,Load}KubectlOptions` | +| `test_structure.{Save,Load}PackerOptions` | `packer.{Save,Load}PackerOptions` | +| `test_structure.{Save,Load}SSHKeyPair` | `ssh.{Save,Load}SSHKeyPair` | + +The target module is usually already imported, because the value being +saved came from it. Where it is not, add the import, and re-run `go mod +tidy` afterwards since this can pull in a module you did not previously +require. + +Everything else stays in `teststructure`: `RunTestStage`, +`CopyTerraformFolderToTemp`, the Terraform option helpers, +`SaveString`/`LoadString`, `SaveInt`/`LoadInt`, +`SaveArtifactID`/`LoadArtifactID`, and the generic +`SaveTestData`/`LoadTestData`. + +## Files that alias Terratest's `aws` + +A file importing both the AWS SDK and Terratest's `aws` usually binds plain +`aws` to the SDK: + +```go +import ( + "github.com/aws/aws-sdk-go-v2/aws" + terraAws "github.com/gruntwork-io/terratest/modules/aws/v2" +) +``` + +Here the relocation above resolves to the SDK and fails to compile. Use +that file's alias: `terraAws.LoadEc2KeyPair`. This is the one place the +scripted rewrite needs a human, and the compiler will point at it. + +## Verify + +No v1 paths left, and everything still builds: + +```bash +grep -rn 'gruntwork-io/terratest/modules/' --include='*.go' . | grep -v '/v2' +go test -run '^$' ./... +``` + +The `grep` should print nothing. Use `go test -run '^$'` rather than +`go build`: it compiles `_test.go` files without running anything, and for +a Terratest suite that is where all of your code lives. diff --git a/docs/_docs/04_community/contributing.md b/docs/_docs/05_community/contributing.md similarity index 98% rename from docs/_docs/04_community/contributing.md rename to docs/_docs/05_community/contributing.md index c9423063f..97119f793 100644 --- a/docs/_docs/04_community/contributing.md +++ b/docs/_docs/05_community/contributing.md @@ -1,373 +1,373 @@ ---- -layout: collection-browser-doc -title: Contributing -category: community -excerpt: >- - Terratest is an open source project, and contributions from the community are very welcome! -tags: ["contributing", "community"] -order: 400 -nav_title: Documentation -nav_title_link: /docs/ -custom_js: - - examples - - prism - - collection-browser_scroll - - collection-browser_search - - collection-browser_toc ---- - -Terratest is an open source project, and contributions from the community are very welcome\! Please check out the -[Contribution Guidelines](#contribution-guidelines) and [Developing Terratest](#developing-terratest) for -instructions. - -## Contribution Guidelines - -Contributions to this repo are very welcome! We follow a fairly standard [pull request -process](https://help.github.com/articles/about-pull-requests/) for contributions, subject to the following guidelines: - -1. [Types of contributions](#types-of-contributions) -1. [File a GitHub issue](#file-a-github-issue) -1. [Update the documentation](#update-the-documentation) -1. [Update the tests](#update-the-tests) -1. [Update the code](#update-the-code) -1. [Create a pull request](#create-a-pull-request) -1. [Merge and release](#merge-and-release) - -### Types of contributions - -Broadly speaking, Terratest contains two types of helper functions: - -1. Integrations with external tools -1. Infrastructure and validation helpers - -We accept different types of contributions for each of these two types of helper functions, as described next. - -#### Integrations with external tools - -These are helper functions that integrate with various DevOps tools—e.g., Terraform, Docker, Packer, and -Kubernetes—that you can use to deploy infrastructure in your automated tests. Examples: - -* `terraform.InitAndApplyContext`: run `terraform init` and `terraform apply`. -* `packer.BuildArtifactsContext`: run `packer build`. -* `shell.RunCommandContextAndGetOutput`: run an arbitrary shell command and return `stdout` and `stderr` as a string. - -Here are the guidelines for contributions with external tools: - -1. **Fixes and improvements to existing integrations**: All bug fixes and new features for existing tool integrations - are very welcome! - -1. **New integrations**: Before contributing an integration with a totally new tool, please file a GitHub issue to - discuss with us if it's something we are interested in supporting and maintaining. For example, we may be open to - new integrations with Docker and Kubernetes tools, but we may not be open to integrations with Chef or Puppet, as - there are already testing tools available for them. - -#### Infrastructure and validation helpers - -These are helper functions for creating, destroying, and validating infrastructure directly via API calls or SDKs. -Examples: - -* `httphelper.HTTPGetWithRetryContext`: make an HTTP request, retrying until you get a certain expected response. -* `ssh.CheckSSHCommandContext`: SSH to a server and execute a command. -* `aws.CreateS3BucketContext`: create an S3 bucket. -* `aws.GetPrivateIpsOfEc2InstancesContext`: use the AWS APIs to fetch IPs of some EC2 instances. - -The number of possible such helpers is nearly infinite, so to avoid Terratest becoming a gigantic, sprawling library -we ask that contributions for new infrastructure helpers are limited to: - -1. **Platforms**: we currently only support three major public clouds (AWS, GCP, Azure) and Kubernetes. There is some - code contributed earlier for other platforms (e.g., OCI), but until we have the time/resources to support those - platforms fully, we will only accept contributions for the major public clouds and Kubernetes. - -1. **Complexity**: we ask that you only contribute infrastructure and validation helpers for code that is relatively - complex to do from scratch. For example, a helper that merely wraps an existing function in the AWS or GCP SDK is - not a great choice, as the wrapper isn't contributing much value, but is bloating the Terratest API. On the other - hand, helpers that expose simple APIs for complex logic are great contributions: `ssh.CheckSSHCommandContext` is a - great example of this, as it provides a simple one-line interface for dozens of lines of complicated SSH logic. - -1. **Popularity**: Terratest should only contain helpers for common use cases that come up again and again in the - course of testing. We don't want to bloat the library with lots of esoteric helpers for rarely used tools, so - here's a quick litmus test: (a) Is this helper something you've used once or twice in your own tests, or is it - something you're using over and over again? (b) Does this helper only apply to some use case specific to your - company or is it likely that many other Terratest users are hitting this use case over and over again too? - -1. **Creating infrastructure**: we try to keep helper functions that create infrastructure (e.g., use the AWS SDK to - create an S3 bucket or EC2 instance) to a minimum, as those functions typically require maintaining state (so that - they are idempotent and can clean up that infrastructure at the end of the test) and dealing with asynchronous and - eventually consistent cloud APIs. This can be surprisingly complicated, so we typically recommend using a tool like - Terraform, which already handles all that complexity, to create any infrastructure you need at test time, and - running Terratest's built-in `terraform` helpers as necessary. If you're considering contributing a function that - creates infrastructure directly (e.g., using a cloud provider's APIs), please file a GitHub issue to explain why - such a function would be a better choice than using a tool like Terraform. - -### File a GitHub issue - -Before starting any work, we recommend filing a GitHub issue in this repo. This is your chance to ask questions and -get feedback from the maintainers and the community before you sink a lot of time into writing (possibly the wrong) -code. If there is anything you're unsure about, just ask! - -### Update the documentation - -We recommend updating the documentation *before* updating any code (see [Readme Driven -Development](http://tom.preston-werner.com/2010/08/23/readme-driven-development.html)). This ensures the documentation -stays up to date and allows you to think through the problem at a high level before you get lost in the weeds of -coding. - -The documentation is built with Jekyll and hosted on the Github Pages from `docs` folder on `main` branch. Check out [Terratest website](https://github.com/gruntwork-io/terratest/tree/main/docs#working-with-the-documentation) to learn more about working with the documentation. - -### Update the tests - -We also recommend updating the automated tests *before* updating any code (see [Test Driven -Development](https://en.wikipedia.org/wiki/Test-driven_development)). That means you add or update a test case, -verify that it's failing with a clear error message, and *then* make the code changes to get that test to pass. This -ensures the tests stay up to date and verify all the functionality in this Module, including whatever new -functionality you're adding in your contribution. The instructions for running the automated tests can be -found [here](https://terratest.gruntwork.io/docs/community/contributing/#developing-terratest). - -### Update the code - -At this point, make your code changes and use your new test case to verify that everything is working. As you work, -please make every effort to avoid unnecessary backwards incompatible changes. This generally means that you should -not delete or rename anything in a public API. - -If a backwards incompatible change cannot be avoided, please make sure to call that out when you submit a pull request, -explaining why the change is absolutely necessary. - -Note that we use pre-commit hooks with this project. To ensure they run: - -1. Install [pre-commit](https://pre-commit.com/). -1. Run `pre-commit install`. - -One of the pre-commit hooks we run is [goimports](https://godoc.org/golang.org/x/tools/cmd/goimports). To prevent the -hook from failing, make sure to : - -1. Install [goimports](https://godoc.org/golang.org/x/tools/cmd/goimports) -1. Run `goimports -w .`. - -We have a [style guide](https://gruntwork.io/guides/style%20guides/golang-style-guide/) for the Go programming language, -in which we documented some best practices for writing Go code. Please ensure your code adheres to the guidelines -outlined in the guide. - -### Create a pull request - -[Create a pull request](https://help.github.com/articles/creating-a-pull-request/) with your changes. Please make sure -to include the following: - -1. A description of the change, including a link to your GitHub issue. -1. The output of your automated test run, preferably in a [GitHub Gist](https://gist.github.com/). We cannot run - automated tests for pull requests automatically due to [security - concerns](https://circleci.com/docs/2.0/oss/#security), so we need you to manually provide this - test output so we can verify that everything is working. -1. Any notes on backwards incompatibility or downtime. - -#### Validate the Pull Request for Azure Platform - -If you're contributing code for the [Azure Platform](https://azure.com) and if you have an active _Azure subscription_, it's recommended to follow the below guidelines after [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). If you're contributing code for any other platform (e.g., AWS, GCP, etc), you can skip these steps. - -> Once the Terratest maintainers add `Azure` tag and _Approve_ the PR, following pipeline will run automatically to perform a full validation of the Azure contribution. You also can run the pipeline manually on your forked repo by following the below guideline. - - -We have a separate CI pipeline for _Azure_ code. To run it on a forked repo: - -1. Run the following [Azure Cli](https://docs.microsoft.com/cli/azure/) command on your preferred Terminal to create Azure credentials and copy the output: - - ```bash - az ad sp create-for-rbac --name "terratest-az-cli" --role contributor --sdk-auth - ``` - -1. Go to Secrets settings page under `Settings` tab in your forked project, `https://github.com//terratest/settings`, on GitHub. - -1. Create a new `Secret` named `AZURE_CREDENTIALS` and paste the Azure credentials you copied from the 1st step as the value - - > `AZURE_CREDENTIALS` will be stored in _your_ GitHub account; neither the Terratest maintainers nor anyone else will have any access to it. Under the hood, GitHub stores your secrets in a secure, encrypted format (see: [GitHub Actions Secrets Reference](https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets) for more information). Once the secret is created, it's only possible to update or delete it; the value of the secret can't be viewed. GitHub uses a [libsodium sealed box](https://libsodium.gitbook.io/doc/public-key_cryptography/sealed_boxes) to help ensure that secrets are encrypted before they reach GitHub. - -1. Create a [new Personal Access Token (PAT)](https://github.com/settings/tokens/new) page under [Settings](https://github.com/settings/profile) / [Developer Settings](https://github.com/settings/apps), making sure `write:discussion` and `public_repo` scopes are checked. Click the _Generate token_ button and copy the generated PAT. - -1. Go back to settings/secrets in your fork and [Create a new Secret](https://docs.github.com/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository) named `PAT`. Paste the output from the 4th step as the value - - > `PAT` will be stored in _your_ GitHub account; neither the Terratest maintainers nor anyone else will have any access to it. Under the hood, GitHub stores your secrets in a secure, encrypted format (see: [GitHub Actions Secrets Reference](https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets) for more information). Once the secret is created, it's only possible to update or delete it; the value of the secret can't be viewed. GitHub uses a [libsodium sealed box](https://libsodium.gitbook.io/doc/public-key_cryptography/sealed_boxes) to help ensure that secrets are encrypted before they reach GitHub. - -1. Go to Actions tab on GitHub (https://github.com//terratest/actions) - -1. Click `ci-workflow` workflow - -1. Click `Run workflow` button and fill the fields in the drop down - * _Repository Info_ : name of the forked repo (_e.g. xyz/terratest_) - * _Name of the branch_ : branch name on the forked repo (_e.g. feature/adding-some-important-module_) - * _Name of the official terratest repo_ : home of the target pr (_gruntwork-io/terratest_) - * PR number on the official terratest repo : pr number on the official terratest repo (_e.g. 14, 25, etc._). Setting this value will leave a success/failure comment in the PR once CI completes execution. - - * Skip provider registration : set true if you want to skip terraform provider registration for debug purposes (_false_ or _true_) - -1. Wait for the `ci-workflow` to be finished - - > The pipeline will use the given Azure subscription and deploy real resources in your Azure account as part of running the test. When the tests finish, they will tear down the resources they created. Of course, if there is a bug or glitch that prevents the clean up code from running, some resources may be left behind, but this is rare. Note that these resources may cost you money! You are responsible for all charges in your Azure subscription. - -1. PR with the given _PR Number_ will have the result of the `ci-workflow` as a comment - -### Merge and release - -The maintainers for this repo will review your code and provide feedback. Once the PR is accepted, they will merge the -code and release a new version, which you'll be able to find in the [releases page](https://github.com/gruntwork-io/terratest/releases). - -## Developing Terratest - -1. [Running tests](#running-tests) -1. [Versioning](#versioning) -1. [Developing For Azure](#developing-for-azure) - -### Running tests - -Terratest itself includes a number of automated tests. - -**Note #1**: Some of these tests create real resources in an AWS account. That means they cost money to run, especially -if you don't clean up after yourself. Please be considerate of the resources you create and take extra care to clean -everything up when you're done! - -**Note #2**: In order to run tests that access your AWS account, you will need to configure your [AWS CLI -credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html). For example, you could -set the credentials as the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. - -**Note #3**: Never hit `CTRL + C` or cancel a build once tests are running or the cleanup tasks won't run! - -**Prerequisite**: The tests expect Terraform, Terragrunt, Packer, and/or Docker to already be installed and in your `PATH`. - -To run all the tests: - -```bash -go test -v -timeout 30m -p 1 ./... -``` - -To run the tests in a specific folder: - -```bash -cd "" -go test -timeout 30m -``` - -To run a specific test in a specific folder: - -```bash -cd "" -go test -timeout 30m -run "" -``` - -### Versioning - -This repo follows the principles of [Semantic Versioning](http://semver.org/). You can find each new release, -along with the changelog, in the [Releases Page](https://github.com/gruntwork-io/terratest/releases). - -Starting with `1.0.0`, breaking changes to the public API only happen in major releases. Symbols renamed or replaced -inside the v1 line are kept as `// Deprecated:` aliases so test code that compiled against an earlier v1.x.y release -will keep compiling against later ones; full removal is deferred to v2. - -### Developing For Azure - -Azure supports multiple cloud environments. In order to properly register the correct environment for you test code, you need to use the Azure SDK Client Factory. - -#### Azure SDK Client Factory - -This documentation provides and overview of the `client_factory.go` module, targeted use cases, and behaviors. This module is intended to provide support for and simplify working with Azure's multiple cloud environments (Azure Public, Azure Government, Azure China, Azure Germany and Azure Stack). Developers looking to contribute to additional support for Azure to Terratest should leverage client_factory and use the patterns below to add a resource REST client from Azure Go SDK. By doing so, it provides a consistent means for developers using Terratest to test their Azure Infrastructure to connect to the correct cloud and its associated REST apis. - -##### Background - -The Azure REST APIs support both Public and sovereign cloud environments (at the moment this includes Public, US Government, Germany, China, and Azure Stack environments). If you are interacting with an environment other than public cloud, you need to set the base URI for the Azure REST API you are interacting with. - -###### Base URI - -You must use the correct base URI's for the Azure REST API's (either directly or via Azure SDK for GO) to communicate with a cloud environment other than Azure Public. The Azure Go SDK supports this by using the `WithBaseURI` suffixed calls when creating service clients. For example, when using the `VirtualMachinesClient` with the public cloud, a developer would normally write code for the public cloud like so: - -```go -import ( - "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-07-01/compute" -) - -func SomeVMHelperMethod() { - subscriptionID := "your subscription ID" - - // Create a VM client and return - vmClient, err := compute.NewVirtualMachinesClient(subscriptionID) - - // Use client / etc -} -``` - -However, this code will not work in non-Public cloud environments as the REST endpoints have different URIs depending on environment. Instead, you need to use an alternative method (provided in the Azure REST SDK for Go) to get a properly configured client (*all REST API clients should support this alternate method*): - -```go -import ( - "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-07-01/compute" -) - -func SomeVMHelperMethod() { - subscriptionID := "your subscription ID" - baseURI := "management.azure.com" - - // Create a VM client and return - vmClient, err := compute.NewVirtualMachinesClientWithBaseURI(baseURI, subscriptionID) - - // Use client / etc -} -``` - -Using code similar to above, you can communicate with any Azure cloud environment just by changing the base URI that is passed to the clients (Azure Public shown in above example). - -##### Lookup Environment Metadata - -Developers MUST avoid hardcoding these base URI's. Instead, they should be looked up from an authoritative source. The AutoRest-GO library (used by the Go SDK) provides such functionality. The `client_factory` module makes use of the AutoRest `EnvironmentFromName(envName string)` function to return the appropriate structure. This method and Environment structure is documented on GoDoc [here](https://godoc.org/github.com/Azure/go-autorest/autorest/azure#EnvironmentFromName). - -To configure different cloud environments, we will use the same `AZURE_ENVIRONMENT` environment variable that the Go SDK uses. This can currently be set to one of the following values: - -|Value |Cloud Environment | -|---------------------------|-------------------| -|"AzureChinaCloud" |ChinaCloud | -|"AzureGermanCloud" |GermanCloud | -|"AzurePublicCloud" |PublicCloud | -|"AzureUSGovernmentCloud" |USGovernmentCloud | -|"AzureStackCloud" |Azure stack | - -When using the "AzureStackCloud" setting, you MUST also set the `AZURE_ENVIRONMENT_FILEPATH` variable to point to a JSON file containing your Azure Stack URI details. - -##### Putting it all together - - `client_factory` implements this pattern described above in order to instantiate and return properly configured *REST SDK for GO* clients so that test implementers don't have to consider REST API client implementation as long as they have the correct `AZURE_ENVIRONMENT` env setting. If this environment variable is not set, the client will assume public cloud as the cloud environment to communicate with. We strongly recommend developers creating Terratest helper methods for Azure use this pattern with client factory to create REST API clients. This will reduce effort for Terratest users creating test for Azure resources. - -Note the following: - -* TERRAFORM uses [ARM_ENVIRONMENT](https://www.terraform.io/docs/backends/types/azurerm.html#environment) environment variable to set the correct cloud environment. -* The default behavior of the `client_factory` is to use the AzurePublicCloud environment. This requires no work from the developer to configure, and ensures consistent behavior with the current SDK code. - -###### Wait, I don't see the client in client factory for the rest api I want to interact with - - If you require a client that is not already implemented in client factory for your helper method, you will need to create a corresponding method that instantiates the client and accepts base URI following the patterns discussed. Below is a walkthrough for adding a client to client factory. - -##### Walkthrough, adding a client to client_factory - -###### Add your client namespace to client factory - -In the Azure SDK for GO, each service should have a module that implements that services client. You can find the correct module [here](https://godoc.org/github.com/Azure/azure-sdk-for-go). Add that module to the client factory imports. Below is an example for client imports that shows clients for compute, container service and subscriptions. - -{% include examples/explorer.html example_id='client-factory' file_id='client_factory_code' class='wide quick-start-examples' skip_learn_more=true skip_view_on_github=true skip_tags=true snippet_id='client_factory_example.imports' %} - -###### Add your client method to instantiate the client - -The next step is to add your method to instantiate the client. Below is an example of adding the method to create a client for Virtual Machines, note that we lookup the environment using `getEnvironmentEndpointE` and then pass that base URI to the actual method on the Virtual Machines Module to create the client `NewVirtualMachinesClientWithBaseURI`. - -{% include examples/explorer.html example_id='client-factory' file_id='client_factory_code' class='wide quick-start-examples' skip_learn_more=true skip_view_on_github=true skip_tags=true snippet_id='client_factory_example.CreateClient' %} - -###### Add a unit test to client_factory_test.go - -In order to ensure that your CreateClient method works properly, add a unit test to `client_factory_test.go`. The unit test MUST assert that the base URI is correctly set for your client. Some key points for writing your unit test are: - -- Use table-driven testing to test the various combinations of cloud environments -- Give the test case a descriptive name so it is easy to identify which test failed. -- PRs will be rejected if a client is added without a corresponding unit test. - -Below is an example of the Virtual Machines client unit test: - -{% include examples/explorer.html example_id='client-factory' file_id='client_factory_test' class='wide quick-start-examples' skip_learn_more=true skip_view_on_github=true skip_tags=true snippet_id='client_factory_example.UnitTest' %} - -###### Use your CreateClient method in your helper - -We now can use this client creation method in our helpers to create a Virtual Machines client. Below is an example for how to call into this create method from `client_factory`: - -{% include examples/explorer.html example_id='client-factory' file_id='client_factory_helper' class='wide quick-start-examples' skip_learn_more=true skip_view_on_github=true skip_tags=true snippet_id='client_factory_example.helper' %} +--- +layout: collection-browser-doc +title: Contributing +category: community +excerpt: >- + Terratest is an open source project, and contributions from the community are very welcome! +tags: ["contributing", "community"] +order: 500 +nav_title: Documentation +nav_title_link: /docs/ +custom_js: + - examples + - prism + - collection-browser_scroll + - collection-browser_search + - collection-browser_toc +--- + +Terratest is an open source project, and contributions from the community are very welcome\! Please check out the +[Contribution Guidelines](#contribution-guidelines) and [Developing Terratest](#developing-terratest) for +instructions. + +## Contribution Guidelines + +Contributions to this repo are very welcome! We follow a fairly standard [pull request +process](https://help.github.com/articles/about-pull-requests/) for contributions, subject to the following guidelines: + +1. [Types of contributions](#types-of-contributions) +1. [File a GitHub issue](#file-a-github-issue) +1. [Update the documentation](#update-the-documentation) +1. [Update the tests](#update-the-tests) +1. [Update the code](#update-the-code) +1. [Create a pull request](#create-a-pull-request) +1. [Merge and release](#merge-and-release) + +### Types of contributions + +Broadly speaking, Terratest contains two types of helper functions: + +1. Integrations with external tools +1. Infrastructure and validation helpers + +We accept different types of contributions for each of these two types of helper functions, as described next. + +#### Integrations with external tools + +These are helper functions that integrate with various DevOps tools—e.g., Terraform, Docker, Packer, and +Kubernetes—that you can use to deploy infrastructure in your automated tests. Examples: + +* `terraform.InitAndApplyContext`: run `terraform init` and `terraform apply`. +* `packer.BuildArtifactsContext`: run `packer build`. +* `shell.RunCommandContextAndGetOutput`: run an arbitrary shell command and return `stdout` and `stderr` as a string. + +Here are the guidelines for contributions with external tools: + +1. **Fixes and improvements to existing integrations**: All bug fixes and new features for existing tool integrations + are very welcome! + +1. **New integrations**: Before contributing an integration with a totally new tool, please file a GitHub issue to + discuss with us if it's something we are interested in supporting and maintaining. For example, we may be open to + new integrations with Docker and Kubernetes tools, but we may not be open to integrations with Chef or Puppet, as + there are already testing tools available for them. + +#### Infrastructure and validation helpers + +These are helper functions for creating, destroying, and validating infrastructure directly via API calls or SDKs. +Examples: + +* `httphelper.HTTPGetWithRetryContext`: make an HTTP request, retrying until you get a certain expected response. +* `ssh.CheckSSHCommandContext`: SSH to a server and execute a command. +* `aws.CreateS3BucketContext`: create an S3 bucket. +* `aws.GetPrivateIpsOfEc2InstancesContext`: use the AWS APIs to fetch IPs of some EC2 instances. + +The number of possible such helpers is nearly infinite, so to avoid Terratest becoming a gigantic, sprawling library +we ask that contributions for new infrastructure helpers are limited to: + +1. **Platforms**: we currently only support three major public clouds (AWS, GCP, Azure) and Kubernetes. There is some + code contributed earlier for other platforms (e.g., OCI), but until we have the time/resources to support those + platforms fully, we will only accept contributions for the major public clouds and Kubernetes. + +1. **Complexity**: we ask that you only contribute infrastructure and validation helpers for code that is relatively + complex to do from scratch. For example, a helper that merely wraps an existing function in the AWS or GCP SDK is + not a great choice, as the wrapper isn't contributing much value, but is bloating the Terratest API. On the other + hand, helpers that expose simple APIs for complex logic are great contributions: `ssh.CheckSSHCommandContext` is a + great example of this, as it provides a simple one-line interface for dozens of lines of complicated SSH logic. + +1. **Popularity**: Terratest should only contain helpers for common use cases that come up again and again in the + course of testing. We don't want to bloat the library with lots of esoteric helpers for rarely used tools, so + here's a quick litmus test: (a) Is this helper something you've used once or twice in your own tests, or is it + something you're using over and over again? (b) Does this helper only apply to some use case specific to your + company or is it likely that many other Terratest users are hitting this use case over and over again too? + +1. **Creating infrastructure**: we try to keep helper functions that create infrastructure (e.g., use the AWS SDK to + create an S3 bucket or EC2 instance) to a minimum, as those functions typically require maintaining state (so that + they are idempotent and can clean up that infrastructure at the end of the test) and dealing with asynchronous and + eventually consistent cloud APIs. This can be surprisingly complicated, so we typically recommend using a tool like + Terraform, which already handles all that complexity, to create any infrastructure you need at test time, and + running Terratest's built-in `terraform` helpers as necessary. If you're considering contributing a function that + creates infrastructure directly (e.g., using a cloud provider's APIs), please file a GitHub issue to explain why + such a function would be a better choice than using a tool like Terraform. + +### File a GitHub issue + +Before starting any work, we recommend filing a GitHub issue in this repo. This is your chance to ask questions and +get feedback from the maintainers and the community before you sink a lot of time into writing (possibly the wrong) +code. If there is anything you're unsure about, just ask! + +### Update the documentation + +We recommend updating the documentation *before* updating any code (see [Readme Driven +Development](http://tom.preston-werner.com/2010/08/23/readme-driven-development.html)). This ensures the documentation +stays up to date and allows you to think through the problem at a high level before you get lost in the weeds of +coding. + +The documentation is built with Jekyll and hosted on the Github Pages from `docs` folder on `main` branch. Check out [Terratest website](https://github.com/gruntwork-io/terratest/tree/main/docs#working-with-the-documentation) to learn more about working with the documentation. + +### Update the tests + +We also recommend updating the automated tests *before* updating any code (see [Test Driven +Development](https://en.wikipedia.org/wiki/Test-driven_development)). That means you add or update a test case, +verify that it's failing with a clear error message, and *then* make the code changes to get that test to pass. This +ensures the tests stay up to date and verify all the functionality in this Module, including whatever new +functionality you're adding in your contribution. The instructions for running the automated tests can be +found [here](https://terratest.gruntwork.io/docs/community/contributing/#developing-terratest). + +### Update the code + +At this point, make your code changes and use your new test case to verify that everything is working. As you work, +please make every effort to avoid unnecessary backwards incompatible changes. This generally means that you should +not delete or rename anything in a public API. + +If a backwards incompatible change cannot be avoided, please make sure to call that out when you submit a pull request, +explaining why the change is absolutely necessary. + +Note that we use pre-commit hooks with this project. To ensure they run: + +1. Install [pre-commit](https://pre-commit.com/). +1. Run `pre-commit install`. + +One of the pre-commit hooks we run is [goimports](https://godoc.org/golang.org/x/tools/cmd/goimports). To prevent the +hook from failing, make sure to : + +1. Install [goimports](https://godoc.org/golang.org/x/tools/cmd/goimports) +1. Run `goimports -w .`. + +We have a [style guide](https://gruntwork.io/guides/style%20guides/golang-style-guide/) for the Go programming language, +in which we documented some best practices for writing Go code. Please ensure your code adheres to the guidelines +outlined in the guide. + +### Create a pull request + +[Create a pull request](https://help.github.com/articles/creating-a-pull-request/) with your changes. Please make sure +to include the following: + +1. A description of the change, including a link to your GitHub issue. +1. The output of your automated test run, preferably in a [GitHub Gist](https://gist.github.com/). We cannot run + automated tests for pull requests automatically due to [security + concerns](https://circleci.com/docs/2.0/oss/#security), so we need you to manually provide this + test output so we can verify that everything is working. +1. Any notes on backwards incompatibility or downtime. + +#### Validate the Pull Request for Azure Platform + +If you're contributing code for the [Azure Platform](https://azure.com) and if you have an active _Azure subscription_, it's recommended to follow the below guidelines after [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). If you're contributing code for any other platform (e.g., AWS, GCP, etc), you can skip these steps. + +> Once the Terratest maintainers add `Azure` tag and _Approve_ the PR, following pipeline will run automatically to perform a full validation of the Azure contribution. You also can run the pipeline manually on your forked repo by following the below guideline. + + +We have a separate CI pipeline for _Azure_ code. To run it on a forked repo: + +1. Run the following [Azure Cli](https://docs.microsoft.com/cli/azure/) command on your preferred Terminal to create Azure credentials and copy the output: + + ```bash + az ad sp create-for-rbac --name "terratest-az-cli" --role contributor --sdk-auth + ``` + +1. Go to Secrets settings page under `Settings` tab in your forked project, `https://github.com//terratest/settings`, on GitHub. + +1. Create a new `Secret` named `AZURE_CREDENTIALS` and paste the Azure credentials you copied from the 1st step as the value + + > `AZURE_CREDENTIALS` will be stored in _your_ GitHub account; neither the Terratest maintainers nor anyone else will have any access to it. Under the hood, GitHub stores your secrets in a secure, encrypted format (see: [GitHub Actions Secrets Reference](https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets) for more information). Once the secret is created, it's only possible to update or delete it; the value of the secret can't be viewed. GitHub uses a [libsodium sealed box](https://libsodium.gitbook.io/doc/public-key_cryptography/sealed_boxes) to help ensure that secrets are encrypted before they reach GitHub. + +1. Create a [new Personal Access Token (PAT)](https://github.com/settings/tokens/new) page under [Settings](https://github.com/settings/profile) / [Developer Settings](https://github.com/settings/apps), making sure `write:discussion` and `public_repo` scopes are checked. Click the _Generate token_ button and copy the generated PAT. + +1. Go back to settings/secrets in your fork and [Create a new Secret](https://docs.github.com/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository) named `PAT`. Paste the output from the 4th step as the value + + > `PAT` will be stored in _your_ GitHub account; neither the Terratest maintainers nor anyone else will have any access to it. Under the hood, GitHub stores your secrets in a secure, encrypted format (see: [GitHub Actions Secrets Reference](https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets) for more information). Once the secret is created, it's only possible to update or delete it; the value of the secret can't be viewed. GitHub uses a [libsodium sealed box](https://libsodium.gitbook.io/doc/public-key_cryptography/sealed_boxes) to help ensure that secrets are encrypted before they reach GitHub. + +1. Go to Actions tab on GitHub (https://github.com//terratest/actions) + +1. Click `ci-workflow` workflow + +1. Click `Run workflow` button and fill the fields in the drop down + * _Repository Info_ : name of the forked repo (_e.g. xyz/terratest_) + * _Name of the branch_ : branch name on the forked repo (_e.g. feature/adding-some-important-module_) + * _Name of the official terratest repo_ : home of the target pr (_gruntwork-io/terratest_) + * PR number on the official terratest repo : pr number on the official terratest repo (_e.g. 14, 25, etc._). Setting this value will leave a success/failure comment in the PR once CI completes execution. + + * Skip provider registration : set true if you want to skip terraform provider registration for debug purposes (_false_ or _true_) + +1. Wait for the `ci-workflow` to be finished + + > The pipeline will use the given Azure subscription and deploy real resources in your Azure account as part of running the test. When the tests finish, they will tear down the resources they created. Of course, if there is a bug or glitch that prevents the clean up code from running, some resources may be left behind, but this is rare. Note that these resources may cost you money! You are responsible for all charges in your Azure subscription. + +1. PR with the given _PR Number_ will have the result of the `ci-workflow` as a comment + +### Merge and release + +The maintainers for this repo will review your code and provide feedback. Once the PR is accepted, they will merge the +code and release a new version, which you'll be able to find in the [releases page](https://github.com/gruntwork-io/terratest/releases). + +## Developing Terratest + +1. [Running tests](#running-tests) +1. [Versioning](#versioning) +1. [Developing For Azure](#developing-for-azure) + +### Running tests + +Terratest itself includes a number of automated tests. + +**Note #1**: Some of these tests create real resources in an AWS account. That means they cost money to run, especially +if you don't clean up after yourself. Please be considerate of the resources you create and take extra care to clean +everything up when you're done! + +**Note #2**: In order to run tests that access your AWS account, you will need to configure your [AWS CLI +credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html). For example, you could +set the credentials as the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. + +**Note #3**: Never hit `CTRL + C` or cancel a build once tests are running or the cleanup tasks won't run! + +**Prerequisite**: The tests expect Terraform, Terragrunt, Packer, and/or Docker to already be installed and in your `PATH`. + +To run all the tests: + +```bash +go test -v -timeout 30m -p 1 ./... +``` + +To run the tests in a specific folder: + +```bash +cd "" +go test -timeout 30m +``` + +To run a specific test in a specific folder: + +```bash +cd "" +go test -timeout 30m -run "" +``` + +### Versioning + +This repo follows the principles of [Semantic Versioning](http://semver.org/). You can find each new release, +along with the changelog, in the [Releases Page](https://github.com/gruntwork-io/terratest/releases). + +Starting with `1.0.0`, breaking changes to the public API only happen in major releases. Symbols renamed or replaced +inside the v1 line are kept as `// Deprecated:` aliases so test code that compiled against an earlier v1.x.y release +will keep compiling against later ones; full removal is deferred to v2. + +### Developing For Azure + +Azure supports multiple cloud environments. In order to properly register the correct environment for you test code, you need to use the Azure SDK Client Factory. + +#### Azure SDK Client Factory + +This documentation provides and overview of the `client_factory.go` module, targeted use cases, and behaviors. This module is intended to provide support for and simplify working with Azure's multiple cloud environments (Azure Public, Azure Government, Azure China, Azure Germany and Azure Stack). Developers looking to contribute to additional support for Azure to Terratest should leverage client_factory and use the patterns below to add a resource REST client from Azure Go SDK. By doing so, it provides a consistent means for developers using Terratest to test their Azure Infrastructure to connect to the correct cloud and its associated REST apis. + +##### Background + +The Azure REST APIs support both Public and sovereign cloud environments (at the moment this includes Public, US Government, Germany, China, and Azure Stack environments). If you are interacting with an environment other than public cloud, you need to set the base URI for the Azure REST API you are interacting with. + +###### Base URI + +You must use the correct base URI's for the Azure REST API's (either directly or via Azure SDK for GO) to communicate with a cloud environment other than Azure Public. The Azure Go SDK supports this by using the `WithBaseURI` suffixed calls when creating service clients. For example, when using the `VirtualMachinesClient` with the public cloud, a developer would normally write code for the public cloud like so: + +```go +import ( + "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-07-01/compute" +) + +func SomeVMHelperMethod() { + subscriptionID := "your subscription ID" + + // Create a VM client and return + vmClient, err := compute.NewVirtualMachinesClient(subscriptionID) + + // Use client / etc +} +``` + +However, this code will not work in non-Public cloud environments as the REST endpoints have different URIs depending on environment. Instead, you need to use an alternative method (provided in the Azure REST SDK for Go) to get a properly configured client (*all REST API clients should support this alternate method*): + +```go +import ( + "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-07-01/compute" +) + +func SomeVMHelperMethod() { + subscriptionID := "your subscription ID" + baseURI := "management.azure.com" + + // Create a VM client and return + vmClient, err := compute.NewVirtualMachinesClientWithBaseURI(baseURI, subscriptionID) + + // Use client / etc +} +``` + +Using code similar to above, you can communicate with any Azure cloud environment just by changing the base URI that is passed to the clients (Azure Public shown in above example). + +##### Lookup Environment Metadata + +Developers MUST avoid hardcoding these base URI's. Instead, they should be looked up from an authoritative source. The AutoRest-GO library (used by the Go SDK) provides such functionality. The `client_factory` module makes use of the AutoRest `EnvironmentFromName(envName string)` function to return the appropriate structure. This method and Environment structure is documented on GoDoc [here](https://godoc.org/github.com/Azure/go-autorest/autorest/azure#EnvironmentFromName). + +To configure different cloud environments, we will use the same `AZURE_ENVIRONMENT` environment variable that the Go SDK uses. This can currently be set to one of the following values: + +|Value |Cloud Environment | +|---------------------------|-------------------| +|"AzureChinaCloud" |ChinaCloud | +|"AzureGermanCloud" |GermanCloud | +|"AzurePublicCloud" |PublicCloud | +|"AzureUSGovernmentCloud" |USGovernmentCloud | +|"AzureStackCloud" |Azure stack | + +When using the "AzureStackCloud" setting, you MUST also set the `AZURE_ENVIRONMENT_FILEPATH` variable to point to a JSON file containing your Azure Stack URI details. + +##### Putting it all together + + `client_factory` implements this pattern described above in order to instantiate and return properly configured *REST SDK for GO* clients so that test implementers don't have to consider REST API client implementation as long as they have the correct `AZURE_ENVIRONMENT` env setting. If this environment variable is not set, the client will assume public cloud as the cloud environment to communicate with. We strongly recommend developers creating Terratest helper methods for Azure use this pattern with client factory to create REST API clients. This will reduce effort for Terratest users creating test for Azure resources. + +Note the following: + +* TERRAFORM uses [ARM_ENVIRONMENT](https://www.terraform.io/docs/backends/types/azurerm.html#environment) environment variable to set the correct cloud environment. +* The default behavior of the `client_factory` is to use the AzurePublicCloud environment. This requires no work from the developer to configure, and ensures consistent behavior with the current SDK code. + +###### Wait, I don't see the client in client factory for the rest api I want to interact with + + If you require a client that is not already implemented in client factory for your helper method, you will need to create a corresponding method that instantiates the client and accepts base URI following the patterns discussed. Below is a walkthrough for adding a client to client factory. + +##### Walkthrough, adding a client to client_factory + +###### Add your client namespace to client factory + +In the Azure SDK for GO, each service should have a module that implements that services client. You can find the correct module [here](https://godoc.org/github.com/Azure/azure-sdk-for-go). Add that module to the client factory imports. Below is an example for client imports that shows clients for compute, container service and subscriptions. + +{% include examples/explorer.html example_id='client-factory' file_id='client_factory_code' class='wide quick-start-examples' skip_learn_more=true skip_view_on_github=true skip_tags=true snippet_id='client_factory_example.imports' %} + +###### Add your client method to instantiate the client + +The next step is to add your method to instantiate the client. Below is an example of adding the method to create a client for Virtual Machines, note that we lookup the environment using `getEnvironmentEndpointE` and then pass that base URI to the actual method on the Virtual Machines Module to create the client `NewVirtualMachinesClientWithBaseURI`. + +{% include examples/explorer.html example_id='client-factory' file_id='client_factory_code' class='wide quick-start-examples' skip_learn_more=true skip_view_on_github=true skip_tags=true snippet_id='client_factory_example.CreateClient' %} + +###### Add a unit test to client_factory_test.go + +In order to ensure that your CreateClient method works properly, add a unit test to `client_factory_test.go`. The unit test MUST assert that the base URI is correctly set for your client. Some key points for writing your unit test are: + +- Use table-driven testing to test the various combinations of cloud environments +- Give the test case a descriptive name so it is easy to identify which test failed. +- PRs will be rejected if a client is added without a corresponding unit test. + +Below is an example of the Virtual Machines client unit test: + +{% include examples/explorer.html example_id='client-factory' file_id='client_factory_test' class='wide quick-start-examples' skip_learn_more=true skip_view_on_github=true skip_tags=true snippet_id='client_factory_example.UnitTest' %} + +###### Use your CreateClient method in your helper + +We now can use this client creation method in our helpers to create a Virtual Machines client. Below is an example for how to call into this create method from `client_factory`: + +{% include examples/explorer.html example_id='client-factory' file_id='client_factory_helper' class='wide quick-start-examples' skip_learn_more=true skip_view_on_github=true skip_tags=true snippet_id='client_factory_example.helper' %} diff --git a/docs/_docs/04_community/license.md b/docs/_docs/05_community/license.md similarity index 94% rename from docs/_docs/04_community/license.md rename to docs/_docs/05_community/license.md index 72a139af9..e8fac295e 100644 --- a/docs/_docs/04_community/license.md +++ b/docs/_docs/05_community/license.md @@ -1,15 +1,15 @@ ---- -layout: collection-browser-doc -title: License -category: community -excerpt: >- - This code is released under the Apache 2.0 License. Read more here. -tags: ["license"] -order: 402 -nav_title: Documentation -nav_title_link: /docs/ ---- - -## License - -This code is released under the Apache 2.0 License. See [LICENSE](https://github.com/gruntwork-io/terratest/blob/main/LICENSE){:target="_blank"} and [NOTICE](https://github.com/gruntwork-io/terratest/blob/main/NOTICE){:target="_blank"} for more details. +--- +layout: collection-browser-doc +title: License +category: community +excerpt: >- + This code is released under the Apache 2.0 License. Read more here. +tags: ["license"] +order: 502 +nav_title: Documentation +nav_title_link: /docs/ +--- + +## License + +This code is released under the Apache 2.0 License. See [LICENSE](https://github.com/gruntwork-io/terratest/blob/main/LICENSE){:target="_blank"} and [NOTICE](https://github.com/gruntwork-io/terratest/blob/main/NOTICE){:target="_blank"} for more details. diff --git a/docs/_docs/04_community/support.md b/docs/_docs/05_community/support.md similarity index 95% rename from docs/_docs/04_community/support.md rename to docs/_docs/05_community/support.md index 182698b3e..936e866ae 100644 --- a/docs/_docs/04_community/support.md +++ b/docs/_docs/05_community/support.md @@ -1,31 +1,31 @@ ---- -layout: collection-browser-doc -title: Support -category: community -excerpt: >- - Need help? -tags: ["support", "community"] -order: 401 -nav_title: Documentation -nav_title_link: /docs/ ---- - -## Github Discussions - -Search our [Knowledge Base](https://github.com/gruntwork-io/knowledge-base/discussions) to find existing questions or ask your own. Github Discussions is a good place for general discussions and questions. - -## Github Issues - -Read through [existing issues](https://github.com/gruntwork-io/terratest/issues) or post a new one. Github issues is a good place to: - -- report a bug, - -- ask for a help, - -- ask for improvements, - -- to start contributing by solving simple issues. - -## Commercial support - -Does your company rely on Terratest in production? If so, you can get commercial support directly from Gruntwork, the creators of Terratest! Check out the [Gruntwork Support Page](https://gruntwork.io/support) for more details. +--- +layout: collection-browser-doc +title: Support +category: community +excerpt: >- + Need help? +tags: ["support", "community"] +order: 501 +nav_title: Documentation +nav_title_link: /docs/ +--- + +## Github Discussions + +Search our [Knowledge Base](https://github.com/gruntwork-io/knowledge-base/discussions) to find existing questions or ask your own. Github Discussions is a good place for general discussions and questions. + +## Github Issues + +Read through [existing issues](https://github.com/gruntwork-io/terratest/issues) or post a new one. Github issues is a good place to: + +- report a bug, + +- ask for a help, + +- ask for improvements, + +- to start contributing by solving simple issues. + +## Commercial support + +Does your company rely on Terratest in production? If so, you can get commercial support directly from Gruntwork, the creators of Terratest! Check out the [Gruntwork Support Page](https://gruntwork.io/support) for more details. diff --git a/docs/v2-import-map.md b/docs/v2-import-map.md index 830fdebde..6b697e743 100644 --- a/docs/v2-import-map.md +++ b/docs/v2-import-map.md @@ -82,7 +82,7 @@ Signatures and on-disk filenames are unchanged, and every call site is a compile Nothing else moved. Watch for files that alias Terratest's `aws` because plain `aws` is the AWS SDK. -## Behaviour changes during the v2 beta +## Behavior changes during the v2 beta **Node addresses prefer `ExternalIP`** (#1878). `k8s.FindNodeHostnameContextE` and `GetServiceEndpoint` (NodePort) now return the Node's `ExternalIP` when present, falling back to the internal hostname as before. On EKS that is the