Add model-agnostic Langfuse observability

[ganievs]

Description

• Add provider-agnostic Langfuse tracing that works with all LLM backends (OpenAI, Ollama, Gemini, Bedrock, etc.)
• Replace the old langfuse.openai drop-in approach with the @observe decorator pattern that creates proper hierarchical trace trees
• Bump Langfuse SDK requirement to >=4.0.6
• Add Lanfuse dependency by default in base Docker image [Feature Request]: Please Install langfuse in the official docker image for the Langfuse to work out of the box. #2581

Related Issues

• [Feature Request]: Please Install langfuse in the official docker image for the Langfuse to work out of the box. #2581
• [Feature Request]: Add model-agnostic Langfuse observability #2936

Changes Made

New file: lightrag/tracing.py — thin utility layer

• is_tracing_enabled() - checks env vars + package availability
• create_traced_llm_wrapper() - wraps any LLM function as a Langfuse generation using observe() functional form
• report_token_usage() - forwards token counts to current Langfuse generation
• propagate_trace_attributes() - sets user_id, session_id, tags on traces
• flush() / shutdown() - lifecycle management

lightrag/lightrag.py

• @observe on core methods

lightrag/utils.py

• embedding + cache tracing

lightrag/llm/openai.py

• token usage reporting
• Removed old langfuse.openai drop-in import (was creating independent traces)

Dockerfile — installs observability extra by default
pyproject.toml — bumped langfuse>=4.0.6

Checklist

• Changes tested locally
• Code reviewed
• Documentation updated (if necessary)
• Unit tests added (if applicable)

Additional Notes

[Add any additional notes or context for the reviewer(s).]

[ganievs]

I've checked via OpenAI and Bedrock both works well.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d98fe4b606

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ganievs]

Hi @danielaskdd - friendly ping on this one.
It's been about a month since the enhancement label went on and I wanted to check in on the merge timeline.
A few teams (including ours) are actively looking to roll out model-agnostic Langfuse tracing across LightRAG deployments - right now we're running off a fork, and we'd much rather consume this from upstream so we stay aligned with future changes.

Is there anything blocking on your side, or a rough timeframe you can share? Thanks!

[ganievs]

Up