Unify on agent_memory_toolkit_v2 and add the Agent Analytics & Optimization plane - #74
Open
Mark Brown (markjbrown) wants to merge 260 commits into
Open
Unify on agent_memory_toolkit_v2 and add the Agent Analytics & Optimization plane#74Mark Brown (markjbrown) wants to merge 260 commits into
Mark Brown (markjbrown) wants to merge 260 commits into
Conversation
…dataUpdates Updating checkpoint to use the new azure-langchain package
Branch mjbrown/unify-v2 is based on upstream/agent_memory_toolkit_v2. This commit layers on the durable, architecture-agnostic assets: - Carry forward .github/copilot-instructions.md from main (v2 lacked it). - Bring the full analytics/ initiative (Open Analytics Schema, ADRs 0001-0005, data generator/enricher, acceptance scenarios, Power BI, vision/charter). - ADR-0006: adopt v2 as the unified baseline and land v2 + dependency modernization + analytics in one PR back to main (maintainer-approved). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
v2 shipped store_debug_log/store_debug_log_from_response orphaned (never called) and OTel commented out, so the token/agent/cost analytics pillars produced nothing. Capture token usage, model metadata, tool calls, and supervisor->sub-agent delegations from v2's astream_events stream in chat_event_generator and persist one Debug log per turn. - _extract_msg_usage: native usage_metadata + response_metadata.token_usage fallback (langchain-core 1.x). - _persist_turn_debug_log: derive agent_path/handoff_count from sub-agent TOOL calls (find_places, create_or_update_itinerary) since v2 invokes sub-agents as tools, not graph nodes. - store_debug_log: additive optional agent_path/handoff_count propertyBag. - thread debug_log_id from the completion so debugLogId matches the message. Validated live on TravelAssistantV2: total_tokens 23025 (in 22211/out 814/ cached 15488), agent_path supervisor,find_places,create_or_update_itinerary, handoff_count 2. ADR-0007 records the Debug-first, OTel-later decision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Allow resuming an interrupted generation run from the Nth persona (1-based) without duplicating earlier personas' sessions/memories. run() takes a start_index so log numbering stays aligned to the full persona list. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Trips were never saved for the requesting user (Trips=0 in analytics):
the itinerary ReAct sub-agent filled user_id/tenant_id on the trip MCP
tools itself and hallucinated placeholders (e.g. user_id='user'), so
trips landed under the wrong partition; and the prompt's two-step
create_new_trip -> update_trip flow made the model echo a bad trip id
into update_trip ('Trip tripId not found').
- Inject the request (user_id, tenant_id) into create_new_trip/update_trip/
get_trip_details via a _current_identity ContextVar + _wrap_trip_tool,
mirroring the existing preference-vector injection. The sub-agent LLM can
no longer set identity.
- itinerary_agent.prompty: create a new trip in a SINGLE create_new_trip call
with the full days array (no fragile follow-up update_trip); reserve
update_trip for a real existing trip_id, used verbatim.
Validated live: trip persisted as trip_trip_user3_barcelona_... (3 days)
under the correct user; MCP log shows 'Creating trip for user: trip_user3';
no ToolException, single create call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
v2's entry points log emoji heavily but did not reconfigure stdout, so on a default Windows console (cp1252) the first emoji log raises UnicodeEncodeError. Port the PR AzureCosmosDB#71 reconfigure block to the v2 entry points (mcp_http_server, travel_agents_api, seed_data). Verified on this machine without PYTHONUTF8: default stdout=cp1252 -> reconfigured to utf-8 -> emoji print succeeds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The classic routing eval asserts which specialist NODE ran (orchestrator->hotel/dining/activity/itinerary_generator); v2 has no such nodes. Add a v2 routing eval that tracks which sub-agent TOOL the supervisor delegated to (find_places / create_or_update_itinerary) via on_tool_start events, with membership-based correctness (itinerary fans out to find_places first). Runs fully local (no LangSmith / LANGCHAIN_API_KEY) against the 02_completed reference solution, matching the analytics initiative's Cosmos-first philosophy (ADR-0003). Validated 7/7 delegations; README documents the model and the tool-usage/e2e follow-ups. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
_mcp_recall_memories_tool was assigned filter_tools_by_prefix(...) LIST, but recall_memories_tool calls it as a single tool (.ainvoke), so every agent memory recall failed with a list-has-no-ainvoke error (the is-None guard does not catch a list). Memory creation was unaffected (SDK auto-trigger), but the agent never read stored memories to personalize. Take the first matched tool (or None). Validated: after the fix, asking maya_chen about dietary preferences recalls her stored memories (pescatarian, vegan options, peanut allergy, budget near Khao San Road); 0 recall errors in logs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…fix) Module-02 taught the itinerary sub-agent tools without identity injection, so a student's saved trips would land under a hallucinated user_id (the same bug fixed in the reference solution, commit 0603b54). Propagate the fix into both the teaching walkthrough and the Module Solution: - add `from contextvars import ContextVar` - add `_current_identity` ContextVar + `_wrap_trip_tool` / `_with_identity_injection` helpers next to the shared sub-agent helpers - wrap `_mcp_itinerary_tools` with `_with_identity_injection` - set/reset `_current_identity` around the itinerary sub-agent call in `create_or_update_itinerary_tool` - explanatory prose in Activities 1, 3, and 5 Module Solution code block verified with py_compile. LangSmith content in Modules 05/06 is untouched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ics Schema" Universal rename for clarity - "Open Analytics Schema" was too general. Updates all 20 prose occurrences across ADRs, charter, and vision; renames adr-0002 file + slug and fixes the README index link. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Establish the lab's optimization-scenario framework, anchored to the vision: - the two tiers of business questions (analytical + action) - the eight optimization dimensions (agent quality, workflow efficiency, memory effectiveness, routing, tool utilization, model selection, cost efficiency, business outcomes) - signal inventory (what TravelAssistantV2 already captures) with Trips.status as the shared outcome anchor - fix-seam safety tiers (prompt/config = safe one-click; code = human-review) - three discovery methods (data-first mining, behavioral probes, naive UI use) - catalog of 8 scenarios spanning all dimensions SCEN-001 (supervisor re-asks for a city it could infer from the active trip) is fully worked: symptom, root-cause evidence, detection metric + SQL sketch, dashboard card, prompt fix, and before/after. Behavior intentionally left unfixed as the canonical lab example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… risk domains Align the optimization-scenarios catalog with the vision's Optimization Maturity Model (L1 Visibility -> L5 Adaptive) and its human-governed risk model: prompt/workflow/code changes are HIGHER-risk (human-governed, ceiling L3 Assisted), while memory salience/retention, retrieval weighting, routing thresholds, tool/model-selection policies are LOWER-risk (autonomous-eligible, L4/L5). Corrects the earlier prompt=one-click framing; retags SCEN-001 as a prompt fix capped at L3, and marks the policy/threshold scenarios as the ones that reach self-adapting L4/L5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…004/007 First data-first mining pass over the v2_analytics baseline (291 Debug turns, 760 memories, 11 trips), persisted as baseline-findings.md. Real numbers validate the candidates and honestly PARK SCEN-006 (token/turn is flat - no drift in short sessions). Promote the two strongest L4/L5 (autonomous) examples to full worked scenarios, complementing SCEN-001's L3 prompt example so the lab spans the maturity ladder: - SCEN-007 model selection: 48% of turns are trivial (no delegation, <60 output tokens) yet run on the full model; 86% cache already. Teaches the two-step path to autonomy (build the model-router seam once = L3 code, then tune the policy autonomously = L4/L5). - SCEN-004 stale-memory retention: 67% of memories superseded; salience populated (mean 0.82). Knobs already exist (TTL/salience/cadence) so it is autonomous-ready today - the cleanest L4/L5 example. Update the catalog: SCEN-003/005/008 marked validated; SCEN-006 parked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The data-first discovery tool behind baseline-findings.md. Reads only signal the app already captures (Debug/Messages/Trips/memories) and prints the Tier-1 metrics per candidate scenario. Verified to reproduce the documented numbers (48% trivial turns, 67% stale memories, 90% place-intent no-delegation, 219k tokens/outcome). Parameterized by --tenant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tunity, not current behavior) Make explicit that the app builds one shared model used by the supervisor and all sub-agents (100% gpt-4.1-mini in the baseline). Model selection is an opportunity dimension here - SCEN-007 introduces per-turn/per-task routing rather than tuning existing variation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… utilization) SCEN-003: the north-star cost-per-successful-outcome KPI (219k tokens/outcome, 44% of spend produced no booking) - framed as the scoreboard the other optimizations move, not a single fix. SCEN-008: supervisor under-uses find_places (90% no-delegation on place intent) and occasionally double-calls it (find_places,find_places) - tool-utilization grounding + de-dup, L3 prompt / L4 policy. Worked scenarios now span all dimensions + the maturity ladder: SCEN-001 (L3 prompt), 004/007 (L4/L5 policy), 003 (KPI), 008 (L3-L4 tool/routing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…y effectiveness) SCEN-005: cost concentrates in a fat tail - the itinerary path costs ~8x a supervisor turn (22% of tokens from 4% of turns). Group-by on Debug.agent_path; feeds SCEN-003. L3 workflow reshape / L4 cost-aware routing. SCEN-002: memory-effectiveness gap - which memories improve outcomes. Honest caveat: recall volume is measurable (18% of turns) but per-memory outcome lift is NOT in current data; needs a small capture add (retrieved memory IDs per Debug turn) before it becomes a data-validated L4/L5 scenario. Catalog now has 7 worked scenarios spanning all 8 dimensions + the full maturity ladder; remaining candidate SCEN-006 (drift) stays parked (data flat). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d model selection First end-to-end optimization apply-loop (detect -> recommend -> apply -> verify), proven live against TravelAssistantV2. - optimization_policy.py: Cosmos-backed, versioned, reversible policy store (self-provisioning OptimizationPolicies container; proposed/active/reverted; short-TTL cache). Applying/reverting is a status flip + audit, never a code edit. - azure_open_ai.get_chat_model(): cached per-deployment model factory; reasoning models (gpt-5*/o-series) omit temperature and use api 2025-04-01-preview. - travel_agents.py: heuristic turn classifier (trivial/routine/complex) + per-tier prebuilt supervisor (shared tools+checkpointer). Default deployment when no active policy, so behavior is unchanged out of the box. - travel_agents_api.py: selects the tiered supervisor per turn; records model_tier/model_deployment on the Debug turn log. - optimization_recommendations.py + optimization_api.py: /optimizations REST surface (recommend card, propose/apply/revert). Prices are labeled ESTIMATES; the measured verify is authoritative. - optimization_mining.py --verify: per-tier token+cost report from Debug signal. Verified live: trivial->gpt-5-nano, routine->gpt-4.1-mini, complex->gpt-5.1; Debug records the actual serving model; revert returns to default. gpt-5-nano is a reasoning model (493 out-tokens on "hi") but input-token savings still make the trivial turn ~4x cheaper than mini. Deployed gpt-5-nano + gpt-5.1 to openai-kfpokdh52vbec (gpt-5.4 quota was fully allocated subscription-wide; later-gen models had free GlobalStandard quota). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ction) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Documents the core-app changes (02_completed), the learner target under the chosen pedagogy (pre-built plumbing + learner-built decision layer), updated/new learning objectives for an Analytics & Optimization module, the 01_exercises sync plan, and a self-contained maintainer change summary. Links it from the docs index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…imization) Resolve the 4 open decisions: new Module 07 (Lessons -> 08); supervisor-turn tiering core + worker sub-agent tiering as stretch; REST/CLI now + Angular card fast-follow; manual verify v1 + automated eval quality-gate as the autonomous capstone. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ons to 08 - New Module-07.md (Analytics & Optimization) teaching the apply-loop: detect -> build classify_turn_tier -> apply -> verify, plus stretch (worker tiering) and capstone (eval quality gate for autonomous apply). Carries a prerequisite banner for the v2 analytics baseline. - Renumber Lessons Learned 07 -> 08; update Home.md learning path and Module-06 nav. - workshop-integration.md: record verified finding that 01_exercises is behind 02_completed (no ADR-0007 Debug re-wire in the scaffold), making the v2 analytics baseline a prerequisite before the apply-loop code can be ported to 01. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Deliver the apply-loop into the exercise track WITHOUT touching Modules 01-05, via an additive layer that brings its own instrumentation. - infra (Bicep, provisioned by azd up): add OptimizationPolicies (/scenario) and OptimizationTurns ([/tenantId,/userId,/sessionId]) containers + gpt-5-nano and gpt-5.1 GlobalStandard deployments. az bicep build passes. - services/optimization.py: self-contained engine (policy store, model factory, select/get_supervisor_for_turn, record_optimization_turn -> OptimizationTurns, recommendations). classify_turn_tier ships as the learner STUB. No self- provisioning (Bicep owns containers); imports only existing 01 symbols. - optimization_api.py: /optimizations REST (recommend/propose/apply/revert). - Module-07.md rewritten for the additive flow: confirm Bicep tiers -> tour layer -> wire 4 hooks -> detect -> implement classifier -> apply -> verify -> stretch (worker tiering) -> capstone (eval quality gate). - optimization_mining.py --verify --container OptimizationTurns (flat schema; still supports --container Debug for 02_completed). - workshop-integration.md: record the additive-layer decision + implementation status; per workshop convention, all Bicep-deployed infra is pre-provisioned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scope for Module 07 (Analytics: instrument/detect/measure, L1-L2) and Module 08 (Optimization: apply/verify/autonomous, L3-L5), Lessons -> 09. Provided surfaces (Power BI + Optimization Console web app + Fabric/reverse-ETL); learners write loop logic, not UX; infra via Bicep. Includes per-module activity outlines, build backlog, coverage matrix, and open decisions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on; Lessons -> 09 Two-module Analytics & Optimization lab per the approved scope: - Module 07 Agent Analytics (Visibility & Insight, L1-L2): vision/dimensions/ maturity/risk, instrument hook, data pipeline, detect, cost-per-outcome. - Module 08 Agent Optimization (Apply & Autonomy, L3-L5): classify_turn_tier, apply/verify model selection, human-governed prompt contrast (SCEN-001), worker-tiering stretch, autonomous quality-gate capstone. - Renumber Lessons 08 -> 09; update Home.md learning path + Module-06 nav. Modules are complete but lighter (~14KB) than reference modules 05/06 (~27KB); depth expansion to match workshop pacing follows next. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… expected outputs, troubleshooting) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…endpoint
- optimization.py: build_turn_metrics (turns, cost, model distribution, trivial %,
per-tier breakdown, cost-per-outcome via Trips); validated against seeded data.
- optimization_api.py: GET /optimizations/{tenant}/metrics.
- console/index.html: self-contained dashboard (own port 8050) - KPI tiles with
'why this matters' talking points, model usage, cost-by-tier, recommendation
cards with one-click Apply/Revert via REST. Didactic, not bare-bones.
- console/README.md + Module-07 launch command (python -m http.server 8050).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ge optimization
Contrasts the model-selection policy (runtime apply) with a prompt fix that must
NOT be auto-applied:
- optimization.py: build_city_context_recommendation detects the 'agent re-asks a
known city' pattern from Messages+Trips; apply_mode=staged_change, maturity L3.
stage_prompt_change records status 'staged' (never active) + the proposed prompt diff.
- optimization_api.py: POST /{scenario}/stage; /apply now returns 400 for the
human-governed scenario, directing to /stage.
- console: renders staged-change cards distinctly (Stage-for-review + proposed
change, no runtime Apply/Revert).
- Module-08 Activity 5 updated with the real detect->apply(blocked)->stage flow.
Validated end-to-end: both cards surface; apply->400; stage->status=staged w/ diff.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… read) - traffic_simulator.py: continuous realistic optimization-turn generator (direct Cosmos writes or app mode) for the real-time demo. Validated: 83 turns -> Cosmos -> Fabric mirror within ~60s. - fabric/TravelAssistantOptimizationInsights.ipynb: reverse-ETL notebook (compute KPIs from mirror, write OptimizationInsights to Cosmos via Cosmos Spark connector + Fabric AAD). Write path job-safe (no pip). Read path being finalized (mirrored Delta uses deletion vectors; needs SQL-endpoint/Lakehouse-shortcut read). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…erified live Add engine/quality/: a reference-free LLM judge (model injected as an invoke callable so the engine stays import-clean), per-agent rubrics (supervisor / find_places / itinerary), the pluggable EvaluationResult primitive, and calibrate() reporting agreement/precision/recall vs human labels. Self-test calibrates a deterministic baseline within tolerance. Verified live (data/verify_quality_live.py): the real Azure OpenAI reference-free judge agreed with a per-agent labeled set at agreement 1.0 / precision 1.0 / recall 1.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rediscovery acceptance B5: detectors/statistical.cost_regression fires only on a statistically significant (z>=3), practically material (>=20%), and stable/consistent per-agent token regression, with a min-sample gate. Self-test proves: suppressed before N, silent on stationary baseline, not tripped by a single outlier, fires on a real regression; stays silent on the stationary simulator. B14: pipeline.rediscovered_scenarios maps discovered opportunities to catalogued SCEN ids; self-test acceptance shows the full pipeline rediscovers SCEN-007 end-to-end from telemetry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…1-C5) Add optimization_agent_api.py (separate small router, not bloating the legacy turn-centric file): agent scorecard, discovered opportunities (SLO-gated via a governance policy the engine consumes), staged-diff review (C2), approve/reject (C4), deploy-attest/confirm-revert (C1), SLO policy (C3), and learner schema declaration bound through the engine SDK (C5). Backed by a new self-provisioning OptimizationGovernance audit store and a node-exec query helper. Verified live against cosmos: scorecard, opportunities, and every C1-C5 action. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…C1-C5) Evolve the standalone console to the agents x dimensions model, split into small files (index.html shell + console.css + api.js client + console.js view). Renders the agent scorecard, engine-discovered opportunities (with SLO gate + governed state), and the C1-C5 governed actions: Review diff, Approve/Reject, Attest deploy/Confirm revert, the SLO policy form, and the declare-a-schema form. Verified end-to-end: api.js exercised live against the API, the view render functions produce correct DOM from live data (headless), and all files serve over http. Ledger C1-C5 marked Grounded (verified live). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add analytics/fabric/verify_engine_on_fabric.py: generates a verification
notebook, runs it as a job on the live F2 capacity, and reads results back
from Cosmos. Verified live on fabf2tx5x7js4bwi:
- B20: Fabric built-in gpt-5-mini runs on F2 via SynapseML OpenAIPrompt
(keyless, capacity-billed) and returns a real completion.
- B22: external Azure OpenAI (gpt-5.1) called KEYLESS via
notebookutils.credentials.getToken('https://cognitiveservices.azure.com')
-> 200 OK (retires the previously-open Entra-only auth risk).
- B21: a burst of 20 built-in calls completed 20/20 with no throttling
(SynapseML internal exponential backoff); F2 handles demo scale.
Infra finding: %pip fails in non-interactive RunNotebook jobs; use
pre-installed SynapseML + requests (no openai package).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…le fallback shipped The Power BI Option-A surface has two external gates that cannot be closed in this environment: building report visuals requires Power BI Desktop, and the translytical button write-path is a transient product bug (owner: fix ~mid-Aug 2026). The deployed UDF target exists. The agent-centric Console (now built and live-verified, C1-C5 Grounded) is the working fallback delivering the same loop. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A polluted PORT (e.g. a stray PowerShell job object stringified into the env)
caused int(os.getenv('PORT', 8080)) to raise ValueError and abort MCP startup.
Parse defensively and fall back to 8080 with a warning.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ixes Replace the Optimization Console with a single-page web Analytics Portal (analytics/dashboard) served at /analytics/ from the frontend container, and make it the primary optimization surface across the solution and workshop. - Analytics Portal: 7 tabs (Overview, Optimizations, Model Selection, Memory, Agents, Business, Governance) reading the Travel API /optimizations/* live, with Live (recompute) and Reverse-ETL (notebook) sources plus Apply/Revert and governance actions. Baked into the frontend image via azure.yaml hooks. - Docs: rewrite USER_GUIDE and workshop Modules 06-10 to be portal-first and demote Power BI to an optional secondary report; add the Module 10 analytics recap and the Module 06 -> Module 10 exit ramp; add a root README analytics section. - Module 09: complete 7-tab portal rundown mapped to notebook sections, an edit-then-Run-all notebook flow, a self-contained LLM-analyst hands-on, and clarified that the mirror SQL endpoint is injected at provision time. - Optimization engine: LLM analyst (propose) + deterministic guardrails (dispose), tool-dedup projection, model-selection wiring; the Fabric reverse-ETL notebook now produces every dashboard metric. - Frontend: fix production build (typed catch in explore.component) and add 429 rate-limit handling. - Housekeeping: remove console/ folders; relocate workshop screenshots into 01_exercises/workshop/media; drop .DS_Store from tracking and gitignore OS junk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
Mark Brown (markjbrown)
changed the title
…reset tooling - Remove the A/B demo seeder (ab_demo_seed.py) and retire the LLM data generators (data_generator.py, data_enricher.py, trivial_seed.py); inline the minimal Travel API client + Conversation model into marvel_seed.py so it stays self-contained. - Reframe demo-script.md to drive the model-selection before/after with the policy-aware traffic_simulator; trim the portal dataset dropdown to analytics/marvel and update the dashboard/analytics READMEs. - Remove demo_live_turns.py (superseded by traffic_simulator --mode app). - Consolidate/retire point-in-time docs (powerbi build-notes/report-delta, workshop lab-scope/integration, cosmos-throughput-analysis, verification/, optimization-scenarios/, MemoryIntelligence_Page_Spec) into the durable solution-architecture-guide; repoint the ADRs accordingly. - Move charter.md into docs/vision/. - Add reset_optimization_state.py and the portal "Freshen times" button/handler for demo data hygiene. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
- New analytics/powerbi/ (build guide, console theme, committed .pbix, and the legacy/ report templates moved under it); update the .gitignore pbix exception and provision_fabric.py's report-import default path. - New analytics/scripts/ (funnel_seed, marvel_seed, traffic_simulator, optimization_mining, Run-TrafficSimulator.ps1); fix each script's repo-root resolution (parents[1]->[2]) and the ps1's repo-root, and update the azure.yaml seed hooks plus every doc reference. - Remove the redundant rbac-mirror.ps1/.sh — provision_fabric.py already creates and assigns the custom FabricMirroringRole (readMetadata + readAnalytics) to the deploying user, workspace identity, and app identity automatically. - Rewrite analytics/README.md's file map; repoint USER_GUIDE, Modules 00/07/09, demo-script, the fabric READMEs, and ADRs 0008/0009 to the new paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
Add a hero shot of the Cosmos Voyager web app and an Analytics Portal Overview-tab screenshot to break up the text-heavy top-level README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
Mark Brown (markjbrown)
marked this pull request as ready for review
August 8, 2026 00:16
Mark Brown (markjbrown)
requested review from
Theo van Kraay (TheovanKraay) and
Aayush Kataria (aayush3011)
August 8, 2026 00:16
…rtal Demo tools Make the Analytics Portal fully functional without Fabric, and add discreet demo-maintenance controls that stay invisible in the workshop. - Relocate the OptimizationInsights builders from analytics/fabric/compute_insights.py into a new app service src/app/services/optimization_insights.py (single source of truth); slim compute_insights.py to a thin CLI wrapper that delegates to it. - Add POST /optimizations/insights — recompute the OptimizationInsights snapshot in-process (Fabric-independent), so the Business / Memory / Governance views light up with no notebook or mirror run. - Add POST /optimizations/reset — clear the runtime optimization state (governance + insights), reusing reset_optimization_state.py's logic via a demo_data service function. - Expose a `capabilities` flag on GET /optimizations/policies for client feature-detection. - Portal: replace the prominent Freshen button with a discreet gear (Demo tools) menu — Recompute insights, Freshen turn times, Reset optimization state. Recompute/Reset are feature-detected and hidden unless the API advertises them, so they never appear when the shared portal is served against the 01_exercises workshop API. Scope: 02_completed only (the workshop's optimization API lacks the recommendation engine). No deploy auto-run, preserving the Module-09 "run the notebook -> tabs light up" reveal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
…lds bake the current portal azd's ACR remote build excludes gitignored files from the upload context, so the frontend image kept baking a stale cached COPY layer (the portal was frozen before the Freshen button + Demo tools menu). Track the staged copy (kept in sync by the prepackage hook) so it is always in the build context. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
…he live turns-by-minute chart moves build_turns_timeline and build_turn_metrics bucket Debug.timeStamp, but refresh_turn_times only re-stamped OptimizationTurns, so the live turns-by-minute chart stayed stale after Freshen. Re-stamp both containers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
…rator; model dedup Make the portal DRIVE the apply -> generate -> re-measure loop (not just observe it), so a reviewer can explore and reset with no CLI and no Fabric. - POST /optimizations/reset now also normalizes captured turns back to the single-premium baseline (in addition to clearing governance + insights), so the model donut shows one model and "apply model-selection -> tier" reads as a clean before/after. Tokens, the funnel signal, and app data are untouched; only non-baseline turns are rewritten. - POST /optimizations/traffic generates policy-aware synthetic turns (baseline single-model until model-selection is applied, capability-tiered once active), dual-writing Debug + OptimizationTurns so every live view reflects it — the in-process equivalent of traffic_simulator.py --mode direct. - Fix the model-distribution duplicate: canonicalize versioned model names (gpt-5.1-2025-11-13 -> gpt-5.1) so the same model isn't split across two donut slices. - Portal: the discreet gear "Demo tools" menu gains Generate traffic; Reset becomes Reset-to-baseline; both are feature-detected via the /policies capabilities flag so they stay hidden against the 01_exercises workshop API. - Docs: add a "Demo tools (gear)" section + reviewer quick loop to USER_GUIDE.md (with zero-setup notes on Acts 4/5); refresh the demo-script.md gear-menu note. 02_completed only — the workshop's simpler optimization service has no recommendation engine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
… tag (AgentMemoryToolkit#36) Adopt the toolkit owner's guidance from AzureCosmosDB/AgentMemoryToolkit#36: exclude soft-pruned memories INSIDE the vector query instead of over-fetching and post-filtering in Python. No toolkit change is required. 02_completed (reference solution): - Soft-prune by adding a reversible `sys:retention-pruned` lifecycle tag to the memory's /tags (partial PATCH; never rewrites the embedding vector). - recall_memories passes exclude_tags=[RETENTION_PRUNED_TAG] to search_cosmos, which emits `NOT ARRAY_CONTAINS(c.tags, @tag)` — cutting retrieval cost, not just context tokens. Guarded so older toolkit builds fall back cleanly. - prune_and_measure_recall stays as a defensive fallback + measurement hook. - _is_pruned recognizes both the tag and the legacy retention_status field; revert strips both. 01_exercises/Module-08.md: - Clarify the exercise deliberately keeps the measurable post-filter, and add a "Going further" note describing the reference solution's in-query approach. Verified: offline logic tests (apply/revert/idempotency/fallback/exclude_tags wiring) pass against azure-cosmos-agent-memory 0.2.0b3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da629ea2-d4ce-4322-93e9-31bd0e76a962
…ises (AgentMemoryToolkit#36) Mirror the reference-solution storage model into the workshop exercise while preserving Module-08's live "measured saving" demo. optimization.py (provided service): - Add RETENTION_PRUNED_TAG (`sys:retention-pruned`) + `_is_pruned` (tag or legacy retention_status). - apply/revert now tag/untag via a partial PATCH of /tags (never rewrites the embedding); revert also strips any legacy retention_status. - _superseded_memory_rows selects c.tags; prune_and_measure_recall and the recommendation count use `_is_pruned`. The measured recall hook is unchanged (still the primary, observable path in the exercise). mcp_server/recall_memories: - Shipped inline post-filter now drops both the reserved tag and the legacy field. Module-08.md: - Update the prune narrative + shown snippet + comment-out block to the tag model. - Convert the "Going further" prose into a concrete OPTIONAL hands-on step that adds `exclude_tags=["sys:retention-pruned"]` to recall, with an explicit trade-off note (in-query exclusion stops the live avoided-token measurement — why the main exercise keeps the post-filter). Verified: byte-compile + adapted offline logic suite (apply/revert/idempotency, tag+legacy detection, measured hook, inline-filter recall) pass against azure-cosmos-agent-memory 0.2.0b3. Safe on read: search_cosmos returns raw dicts (no pydantic tag validation), so `sys:`-tagged memories are neither rejected nor stripped; writes use a direct Cosmos PATCH that bypasses the validator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da629ea2-d4ce-4322-93e9-31bd0e76a962
Record the deliberate decision not to build a context-window detector/recommendation card in the optimization plane. Rationale (evidence-cited): - The recommendation catalog is exactly five cards (model-selection, memory-retention, tool-call-dedup, cost-per-outcome, agent-path-cost); none analyze prompt/context size, and a grep for context_window/max_tokens/truncate/prompt_size finds no such logic. - Context reduction is already delivered and MEASURED by existing governed levers (memory-retention's avoided-input-token telemetry) plus the runtime summarizer, recall top_k, and capability-tiered model selection. - Per-call context/token visibility is trace-grain telemetry that LangSmith already surfaces — and it's already wired here via @langsmith.traceable — so a visibility card would duplicate observability and blur the plane's govern→apply→measure differentiator. - Per ADR-0010, "a lens points; the policies act": a context-window card would be a redundant lens aimed at agents the existing policies already cover. Deferred (Option C): a guardrail extending existing levers (cap recall top_k / summarize earlier) only if hard context-limit failures appear. Index updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da629ea2-d4ce-4322-93e9-31bd0e76a962
Wire measured analytics telemetry, make Fabric setup and report validation fail truthfully, preserve participant policy state, add notebook run checkpoints, and align workshop guidance with verified behavior. (cherry picked from commit 00db72f)
Replace the legacy PBIX with source-controlled PBIR/TMDL deployment, a seven-page data-driven report, and state-aware Fabric UDF policy actions. Align the reverse-ETL producers and notebooks with the report schema, and rewrite Module 09 plus supporting documentation around the web and Power BI workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b83ec7bc-2897-489f-bd7c-e4bfae9401b7
Contributor
Author
Integration updateTheo's PR #75 was reviewed and validated across its 14 changed files: Python compiled, PowerShell parsed, generated notebooks reproduced byte-for-byte, and the integration signatures matched the v2 architecture. It was approved and merged into The final Power BI/workshop work is now included in commit |
Contributor
Author
Power BI and Fabric implementationThe Power BI/Fabric portion is now complete:
The canonical report and semantic model were updated successfully in the live Fabric workspace. |
Contributor
Author
Reverse-ETL, workshop, and validation updateThe workshop now teaches the completed two-surface analytical loop:
Additional completion work:
|
Contributor
Author
Upstream integration branch synchronizedThe completed Power BI/workshop change set is now also on the upstream |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Two intertwined efforts, one unifying PR to
main— grounded inanalytics/docs/vision/agent-analytics-and-optimization-vision.mdandanalytics/docs/vision/charter.md, and superseding the first attempt (#73):agent_memory_toolkit_v2— replace the classic orchestrator + specialist-nodes architecture with the v2 model: supervisor + sub-agents-as-tools, theazure-cosmos-agent-memorySDK, and thelangchain-azure-cosmosdbcheckpointer.maincurrently reverted the memory-toolkit work, so this PR carries the full classic → v2 transformation. (ADR-0006.)The Agent Analytics & Optimization plane
Built across all six pillars from the vision — Agent Performance, Agent Collaboration, Cost Intelligence, Memory Intelligence (flagship), Evaluation Intelligence, Workflow Intelligence — at Levels 1–3 across the pillars, one bounded, audited-and-reversible Level 4 slice (memory-retention / model-selection), with Level 5 as conceptual framing (per the charter).
analytics/dashboard, served at/analytics/) — the operational optimization surface: pillar views reading the Travel API/optimizations/*live, with Live (recompute) and Reverse-ETL (notebook) sources, one-click Apply / Revert, and a governance lifecycle.OptimizationInsightsrows back to Cosmos; an LLM analyst proposes recommendation cards and deterministic guardrails dispose (the engine computes the saving); optional translytical Apply/Revert via a Fabric User Data Function.v2 adoption details (ADR-0006, carried from #73)
find_places,create_or_update_itinerary) replace the orchestrator + five specialist nodes.azure-cosmos-agent-memorySDK owns extraction / dedup / summaries / supersession (replaces the hand-rolled memory subsystem).langchain-azure-cosmosdbcheckpointer.cosmosdb-gsi.bicep,seed_gsi_trips.py) is carried forward from v2 but gated behindDEPLOY_GSIand not used in the default deployment.Bug fixes (supersedes #69–#72)
Delivered on v2:
recall_memories(a tool list was invoked as one tool, so memories were never read); saved trips landing under a hallucinateduser_id(fixed via identity injection); Windows UTF-8/emoji logging crash; real token + agent/hand-off capture via the re-wired CosmosDebugpath; frontend production-build fix (typedcatch) + 429 rate-limit handling.Related
search_cosmos(exclude_tags=[...])(emitted asNOT ARRAY_CONTAINS(c.tags, @tag)), so no toolkit change or follow-up PR is required. Adopting that guidance, the reference solution (02_completed) now soft-prunes superseded memories by adding a reversiblesys:retention-prunedtag (a partial PATCH that never rewrites the embedding) and passesexclude_tags=["sys:retention-pruned"]at recall — dropping pruned memories in-query and cutting retrieval cost. Module-08 in01_exercisesteaches the same approach as an optional step, while keeping the measured post-filter as the default so the live "avoided-token" saving stays observable end-to-end.Notes for reviewers
02_completed/; the build-along workshop in01_exercises/workshop/; the knowledge base (vision, charter, ADRs, deep docs) inanalytics/docs/.Why this PR is large (377 files) — and how to review it efficiently
This is a first-time unification of three long-lived lines of work into
main, which is still ata70a8e1(PR #67). None of the prerequisites were ever merged tomain— they landed on integration branches, so they all show up here:agent_memory_toolkit_v2(PRs Perf improvements: supervisor + @tool sub-agents #57, update readme #59, Updating Module 01-06 #64, Updating checkpoint to use the new azure-langchain package #65 — already reviewed) → ~95 files.analytics(PRs Add Fabric Analytics #48, update #49, Performance Improvements for Travel Multi-Agent Workshop #51, Sync analytics branch with latest main + fixes #68) → ~40 files.The count is also inflated ~2× by the mirrored trees:
01_exercises/(130 files) and02_completed/(136 files) carry largely the same edits — the teaching scaffold vs. the completed solution.Efficient review path — to see the net-new work without the mirror duplication or the already-reviewed toolkit commits:
analytics/(107 files) — the new Analytics & Optimization plane (portal, Fabric reverse-ETL, engine, docs).02_completed/— the completed solution; you can skip the01_exercises/mirror, which tracks the same changes as the teaching scaffold.Power BI and Module 09 completion
TravelAssistantAnalyticsReportreport and semantic model.OptimizationInsights, with a native master-detail selection pattern rather than fixed scenario cards.optimization-apply-loopFabric User Data Function and the liveOptimizationPoliciessource of truth.agent_opportunity,recommendation_card, andslo_metricrows.Validation completed
OptimizationInsightsrows for theanalyticstenant.