Yhl/llm example

[Yhlong00]

Description

Brief description of what this PR adds or fixes.

Type of Change

• New example
• Bug fix
• Enhancement to existing example
• Documentation update
• Other (please describe):

Example Category

If adding a new example, which category does it belong to?

• 01 - Getting Started
• 02 - ML Inference
• 03 - Advanced Workers
• 04 - Scaling & Performance
• 05 - Data Workflows
• 06 - Real World Applications
• misc - Miscellaneous

Checklist

Functionality

• Example runs successfully with flash run
• All endpoints return correct responses
• Tested locally
• Error handling implemented

Code Quality

• Code follows project style guidelines
• Type hints added for function signatures
• Async functions used where appropriate
• No hardcoded credentials
• Proper logging (not print statements)

Documentation

• README.md included with all required sections
• Code comments added for complex logic
• API endpoints documented
• Deployment instructions included
• .env.example file provided

Dependencies

• All dependencies listed in requirements.txt
• Dependencies are pinned to specific versions
• pyproject.toml included with project metadata

Testing

• Manually tested all endpoints
• Unit tests added (if applicable)
• No syntax errors (python -m py_compile)
• Linting passes (if configured)

Security

• No secrets or API keys committed
• Input validation implemented
• Security best practices followed

What This Example Demonstrates

List the key concepts or patterns this example demonstrates:

Testing Instructions

How should reviewers test this example?

# Steps to test
cd path/to/example
pip install -r requirements.txt
# Add your test steps here

Screenshots/Output (if applicable)

Add screenshots or example output if relevant.

Additional Context

Any additional information reviewers should know about this PR.

Related Issues

Closes #(issue number)

[capy-ai]

Added 2 comments

[capy-ai]

[🟠 High] [🔵 Bug]

The worker is explicitly schedulable on any GPU, but the pipeline forces torch.bfloat16; requests can fail at model load time on GPU types that do not support BF16, causing intermittent endpoint failures depending on where the job lands. Use an adaptive dtype so the worker remains portable across heterogeneous GPU assignments.

# 02_ml_inference/01_text_generation/gpu_worker.py
gpu_config = LiveServerless(
    gpus=[GpuGroup.ANY],  # Run on any GPU
)
...
torch_dtype=torch.bfloat16,

Suggested change

	torch_dtype=torch.bfloat16,
	torch_dtype="auto",

[capy-ai]

[🟠 High] [🟡 Investigate]

This new example imports tetra_rp directly, but the repository’s documented/base install contract is centered on runpod-flash (uv sync from the repo root installs that dependency set), so a clean environment may hit ModuleNotFoundError when flash run imports this module. Verify by running the repo quick-start flow in a fresh venv; if it fails, migrate this example to the supported runpod_flash API or add/document project-level dependency installation for tetra_rp.

# 02_ml_inference/01_text_generation/gpu_worker.py
from tetra_rp import (
    GpuGroup,
    LiveServerless,
    remote,
)