Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
164 changes: 85 additions & 79 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,29 +14,29 @@ This is an npm workspace monorepo (workspaces defined in the root `package.json`

### Core Packages

| Package | Description |
|---------|-------------|
| `@stela/api` | Main Express API server — the primary package where most development happens |
| `@stela/logger` | Centralized logging via Winston |
| `@stela/permanent_models` | Shared TypeScript models and data types |
| `@stela/event_utils` | Event validation utilities |
| `@stela/file-utils` | File manipulation utilities |
| `@stela/s3-utils` | AWS S3 integration utilities |
| `@stela/archivematica-utils` | Archivematica integration utilities |
| Package | Description |
| ---------------------------- | ---------------------------------------------------------------------------- |
| `@stela/api` | Main Express API server — the primary package where most development happens |
| `@stela/logger` | Centralized logging via Winston |
| `@stela/permanent_models` | Shared TypeScript models and data types |
| `@stela/event_utils` | Event validation utilities |
| `@stela/file-utils` | File manipulation utilities |
| `@stela/s3-utils` | AWS S3 integration utilities |
| `@stela/archivematica-utils` | Archivematica integration utilities |

### Lambda / Cron Packages

| Package | Type | Description |
|---------|------|-------------|
| `@stela/record_thumbnail_attacher` | Lambda | Attaches thumbnails to records via CloudFront signed URLs |
| `@stela/account_space_updater` | Lambda | Updates account storage ledger |
| `@stela/access_copy_attacher` | Lambda | Attaches access copies generated by Archivematica to records |
| `@stela/metadata_attacher` | Lambda | Processes and attaches metadata to records |
| `@stela/trigger_archivematica` | Lambda | Triggers archivematica preservation workflows |
| `@stela/thumbnail_refresh` | Cron | Refreshes expired CDN thumbnail URLs |
| `@stela/file_url_refresh` | Cron | Refreshes expired file URLs |
| `@stela/archivematica_cleanup` | Cron | Cleans up archivematica processes |
| `@stela/event_send` | Cron | Sends unsent events from the event table to SNS |
| Package | Type | Description |
| ---------------------------------- | ------ | ------------------------------------------------------------ |
| `@stela/record_thumbnail_attacher` | Lambda | Attaches thumbnails to records via CloudFront signed URLs |
| `@stela/account_space_updater` | Lambda | Updates account storage ledger |
| `@stela/access_copy_attacher` | Lambda | Attaches access copies generated by Archivematica to records |
| `@stela/metadata_attacher` | Lambda | Processes and attaches metadata to records |
| `@stela/trigger_archivematica` | Lambda | Triggers archivematica preservation workflows |
| `@stela/thumbnail_refresh` | Cron | Refreshes expired CDN thumbnail URLs |
| `@stela/file_url_refresh` | Cron | Refreshes expired file URLs |
| `@stela/archivematica_cleanup` | Cron | Cleans up archivematica processes |
| `@stela/event_send` | Cron | Sends unsent events from the event table to SNS |

## Technology Stack

Expand All @@ -45,7 +45,7 @@ This is an npm workspace monorepo (workspaces defined in the root `package.json`
- **Framework:** Express
- **Database:** PostgreSQL, accessed via TinyPg (SQL file-based query library)
- **Authentication:** FusionAuth (external identity provider)
- **Testing:** Jest with ts-jest, Supertest, jest-when, nock, jest-mock-extended
- **Testing:** Vitest, Supertest, vitest-when, nock, vitest-mock-extended
- **Linting:** ESLint (eslint-config-love + prettier), SQLFluff (PostgreSQL dialect)
- **API Docs:** OpenAPI, linted with Redocly CLI
- **Error Tracking:** Sentry
Expand Down Expand Up @@ -89,9 +89,10 @@ The `@stela/api` lint script runs these checks sequentially:
### Testing

Tests for `@stela/api` require a running PostgreSQL instance. The test setup:

1. Starts Docker containers (`docker compose up`)
2. Creates a fresh `test_permanent` database from the main database schema
3. Runs Jest inside the Docker container
3. Runs Vitest inside the Docker container

Other workspace tests can generally run independently.

Expand Down Expand Up @@ -153,6 +154,7 @@ exampleController.get(
```

Key conventions:

- Authentication middleware injects `emailFromAuthToken` and `userSubjectFromAuthToken` into `req.body`
- Validation uses Joi with TypeScript `asserts` type guards
- Validation errors produce 400 responses; other errors are passed to `next()`
Expand All @@ -165,17 +167,18 @@ Services contain business logic and database access:

```typescript
export const getThingById = async (id: string): Promise<Thing> => {
const result = await db
.sql<ThingRow>("domain.queries.query_name", { id })
.catch((err: unknown) => {
logger.error(err);
throw new createError.InternalServerError("Failed to get thing");
});
return result.rows[0];
const result = await db
.sql<ThingRow>("domain.queries.query_name", { id })
.catch((err: unknown) => {
logger.error(err);
throw new createError.InternalServerError("Failed to get thing");
});
return result.rows[0];
};
```

Key conventions:

- Use `db.sql<RowType>("domain.queries.file_name", params)` for parameterized queries
- The SQL file path is dot-notation: `domain.queries.file_name` maps to `packages/api/src/domain/queries/file_name.sql`
- Errors are logged then re-thrown as HTTP errors
Expand All @@ -186,21 +189,22 @@ Key conventions:
Validators use Joi with TypeScript assertion functions:

```typescript
export const validateSomething: (
data: unknown,
) => asserts data is SomeType = (
data: unknown,
export const validateSomething: (data: unknown) => asserts data is SomeType = (
data: unknown,
): asserts data is SomeType => {
const validation = Joi.object()
.keys({ /* schema */ })
.validate(data);
if (validation.error !== undefined) {
throw validation.error;
}
const validation = Joi.object()
.keys({
/* schema */
})
.validate(data);
if (validation.error !== undefined) {
throw validation.error;
}
};
```

Shared authentication validation fields are in `packages/api/src/validators/shared.ts`:

- `fieldsFromUserAuthentication` — requires `emailFromAuthToken` + `userSubjectFromAuthToken`
- `fieldsFromAdminAuthentication` — requires `emailFromAuthToken` + `adminSubjectFromAuthToken`

Expand Down Expand Up @@ -240,6 +244,7 @@ SQL files are linted with SQLFluff (PostgreSQL dialect, colon-style placeholder
### Schema

The base database schema lives in `database/base.sql` (dumped from production). Key tables include:

- `account`, `account_archive` — user accounts and archive memberships
- `archive` — archives (collections)
- `folder`, `folder_link` — folder hierarchy
Expand All @@ -263,48 +268,49 @@ Test files are co-located with source: `controller.test.ts` next to `controller.
### Integration Test Pattern (API)

```typescript
jest.mock("../database");
jest.mock("../middleware");
jest.mock("@stela/logger");
vi.mock("../database");
vi.mock("../middleware");
vi.mock("@stela/logger");

const setupDatabase = async (): Promise<void> => {
await db.sql("domain.fixtures.create_test_accounts");
await db.sql("domain.fixtures.create_test_data");
// ... load all required fixtures
await db.sql("domain.fixtures.create_test_accounts");
await db.sql("domain.fixtures.create_test_data");
// ... load all required fixtures
};

const clearDatabase = async (): Promise<void> => {
await db.query(`TRUNCATE account, archive, ... CASCADE`);
await db.query(`TRUNCATE account, archive, ... CASCADE`);
};

describe("GET /endpoint", () => {
beforeEach(async () => {
mockVerifyUserAuthentication("test@permanent.org", "uuid-here");
await clearDatabase();
await setupDatabase();
});

afterEach(async () => {
await clearDatabase();
jest.restoreAllMocks();
jest.clearAllMocks();
});

const agent = request(app);

test("description", async () => {
const response = await agent.get("/api/v2/endpoint").expect(200);
expect(response.body).toEqual(/* ... */);
});
beforeEach(async () => {
mockVerifyUserAuthentication("test@permanent.org", "uuid-here");
await clearDatabase();
await setupDatabase();
});

afterEach(async () => {
await clearDatabase();
vi.restoreAllMocks();
vi.clearAllMocks();
});

const agent = request(app);

test("description", async () => {
const response = await agent.get("/api/v2/endpoint").expect(200);
expect(response.body).toEqual(/* ... */);
});
});
```

Key patterns:

- The database module, middleware, and logger are always mocked at the module level
- `setupDatabase` loads SQL fixtures; `clearDatabase` truncates tables
- Middleware mocks are in `packages/api/test/middleware_mocks.ts`
- Each test block clears and rebuilds the database in `beforeEach`/`afterEach`
- Use `jest-when` for conditional mock behavior based on arguments
- Use `vitest-when` for conditional mock behavior based on arguments
- Supertest `agent` is created from the Express `app`

## Authentication & Authorization
Expand Down Expand Up @@ -369,25 +375,25 @@ API docs are at `packages/api/docs/src/api.yaml` using modular YAML references.

GitHub Actions workflows in `.github/workflows/`:

| Workflow | Trigger | Purpose |
|----------|---------|---------|
| `lint.yml` | Every push | Builds and lints all workspaces |
| `test.yml` | Push to non-main branches | Runs all workspace tests with Codecov |
| `build.yml` | Called by deploy workflows | Builds Docker images, pushes to AWS ECR |
| `dev_deploy.yml` | Merge to main | Auto-deploys to dev environment |
| `staging_deploy.yml` | Manual | Deploys to staging |
| `prod_deploy.yml` | Manual (requires approval) | Deploys to production |
| `deploy_docs.yml` | Docs changes | Publishes API docs to GitHub Pages |
| Workflow | Trigger | Purpose |
| -------------------- | -------------------------- | --------------------------------------- |
| `lint.yml` | Every push | Builds and lints all workspaces |
| `test.yml` | Push to non-main branches | Runs all workspace tests with Codecov |
| `build.yml` | Called by deploy workflows | Builds Docker images, pushes to AWS ECR |
| `dev_deploy.yml` | Merge to main | Auto-deploys to dev environment |
| `staging_deploy.yml` | Manual | Deploys to staging |
| `prod_deploy.yml` | Manual (requires approval) | Deploys to production |
| `deploy_docs.yml` | Docs changes | Publishes API docs to GitHub Pages |

## Lambda Handler Pattern

```typescript
export const handler: SQSHandler = async (event: SQSEvent) => {
for (const record of event.Records) {
// Validate and parse SQS message
// Execute business logic
// Handle errors with Sentry + logger
}
for (const record of event.Records) {
// Validate and parse SQS message
// Execute business logic
// Handle errors with Sentry + logger
}
};
```

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ Note that the database tests run against is dropped and recreated at the beginni
For running tests on a single file in `stela`, you can make an adjustment to the existing `test` command in **packages/api/package.json** on your machine. See the example below (replace **src/middleware/authentication.test.ts** with a pattern to match your file):

```json
"test:file": "npm run start-containers && npm run clear-database && npm run create-database && npm run set-up-database && (cd ../..; docker compose run stela node --experimental-vm-modules ../../node_modules/jest/bin/jest.js -i --silent=false -- src/middleware/authentication.test.ts)",
"test:file": "npm run start-containers && npm run clear-database && npm run create-database && npm run set-up-database && (cd ../..; docker compose run vitest run -- src/middleware/authentication.test.ts)",
```

## Running the API Server Locally
Expand Down
15 changes: 10 additions & 5 deletions eslint.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import typescriptEslint from "typescript-eslint";
import prettier from "eslint-config-prettier";
import love from "eslint-config-love";
import globals from "globals";
import jest from "eslint-plugin-jest";
import vitest from "@vitest/eslint-plugin";
import js from "@eslint/js";

export default defineConfig([
Expand All @@ -17,7 +17,7 @@ export default defineConfig([
languageOptions: {
globals: {
...globals.node,
...globals.jest,
...vitest.environments.env.globals,
},
},

Expand Down Expand Up @@ -48,13 +48,13 @@ export default defineConfig([
files: ["**/*.test.ts"],

plugins: {
jest,
vitest,
},

rules: {
"@typescript-eslint/unbound-method": "off",
"jest/unbound-method": "error",
"jest/no-focused-tests": "error",
"vitest/unbound-method": "error",
"vitest/no-focused-tests": "error",
// Test files are allowed to be long because they need to be able to comprehensively test
// the relevant code, however many tests that takes. Their natural structure also makes
// them more navigable than other lengthy files might be.
Expand All @@ -65,5 +65,10 @@ export default defineConfig([
// This interferes with some mocking patterns that we use widely
"@typescript-eslint/strict-void-return": "off",
},
languageOptions: {
globals: {
...vitest.environments.env.globals,
},
},
},
]);
5 changes: 0 additions & 5 deletions jest.config.js

This file was deleted.

Loading