feat: collections v2 aggregate Tinybird pipes and toggle column (IN-1193) by gaspergrom · Pull Request #4336 · linuxfoundation/crowd.dev

gaspergrom · 2026-07-14T08:13:33Z

Summary

Backend support for the LFX Insights Collections v2 UI epic (IN-1191, subtickets IN-1193 and IN-1195):

New migration: showAggregateTabs boolean column on collections (default false, backfilled true for curated/LF Foundation collections where ssoUserId IS NULL) — per-collection toggle for the new in-depth aggregate tabs.
collection_insights_aggregate.pipe — resolves a collection's member projects via insights_projects_populated_ds.collectionsSlugs (has() filter, same pattern as insightsProjects_filtered.pipe) and aggregates Project count / unique Contributors (deduplicated across all projects, filtered to code contributions or collaborations) / Avg. Health Score for a given collectionSlug (IN-1193's header metrics row).
CDP admin UI: a new "Show aggregate tabs" toggle in the collection add/edit form, wired through the data-access-layer, mirroring the existing starred field's pattern.

Note: collection_contributors_leaderboard_aggregate.pipe, collection_popularity_aggregate.pipe, and collection_development_aggregate.pipe were removed from this PR — those tabs (leaderboard, popularity, development) are being reworked separately and will ship in a follow-up PR. This PR only covers the header metrics row aggregate (IN-1193).

Deploy order

This PR must merge and collection_insights_aggregate.pipe must be deployed to Tinybird before the companion insights PR's aggregate features will show real data — until then, the insights UI degrades gracefully to empty/fallback states (verified, does not crash).

Adds the showAggregateTabs toggle column on collections, and new Tinybird pipes aggregating project_insights and activity-relations data across an arbitrary list of projects for the LFX Insights Collections v2 UI (IN-1193, IN-1195). Signed-off-by: Gašper Grom <gasper.grom@gmail.com>

github-actions

Conventional Commits FTW!

Copilot

Pull request overview

Adds Collections v2 backend support through a database toggle and collection-wide Tinybird metrics.

Changes:

Adds aggregate insights, popularity, development, and contributor leaderboard pipes.
Adds the showAggregateTabs collection column.
Deduplicates collection-wide contributor and organization counts.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`collection_popularity_aggregate.pipe`	Aggregates stars and forks.
`collection_insights_aggregate.pipe`	Computes collection header metrics.
`collection_development_aggregate.pipe`	Counts active contributors and organizations.
`collection_contributors_leaderboard_aggregate.pipe`	Produces a collection-wide contributor leaderboard.
`V1782500000__addShowAggregateTabsColumnToCollectionsTable.sql`	Adds the aggregate-tabs toggle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…project ids - Takes collectionSlug directly instead of a caller-supplied project-id array, resolving collection membership via insights_projects_populated_ds.collectionsSlugs (has() filter, same pattern as insightsProjects_filtered.pipe). - Fixes segmentId resolution: project_insights_copy_ds has no segmentId column at all (confirmed by the Tinybird deploy error); insights_projects_populated_ds has both segmentId and healthScore. Verified live against Tinybird - returns correct projectCount, uniqueContributorCount, and avgHealthScore for a real collection. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>

Contributors/Popularity/Development collection tabs are hidden in the insights UI while IN-1195 is being reworked - removing the corresponding Tinybird pipes so the PR doesn't ship dead backend code. The showAggregateTabs migration and collection_insights_aggregate.pipe (IN-1193's header metrics, already deployed and working) stay. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>

…ll curated - Replaces the earlier hand-written migration with one scaffolded via ./scripts/cli scaffold create-migration, per repo convention. - showAggregateTabs now defaults to false for all collections, then backfills to true for curated (LF Foundation) collections only (ssoUserId IS NULL) - the tab-gating logic lives entirely in this column now, not split between it and a CURATED type check on the insights frontend. - Adds an editable "Show aggregate tabs" toggle to the CDP admin collection add/edit form, defaulting to true for new collections (admin-created collections are always curated), following the exact same DAL/model/form pattern already used for the sibling starred field. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Frontend dev only needs api running; the packages_worker workers (npm/pypi/maven/osv/dockerhub/cargo/go/nuget/rubygems/security-contacts, bq-dataset-ingest, github-repos-enricher) plus automatic-projects-discovery-worker, pcc-sync-worker, and projects-evaluation-worker were being built/started unnecessarily on every clean-start-fe-dev / service-restart-fe-dev. Claude-Session: https://claude.ai/code/session_01J17dWfDoruiB5xn5iNM89V Signed-off-by: Gašper Grom <gasper.grom@gmail.com>

Pairs with V1784026542 - DROP COLUMN reverses the ADD COLUMN + backfill, following this repo's occasional U<version> undo-migration convention. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>

cursor · 2026-07-14T12:35:00Z

PR Summary

Low Risk
Changes are a new read-only analytics pipe and local dev CLI service lists; no auth, migrations, or write paths appear in this diff.

Overview
Adds collection_insights_aggregate.pipe, a Tinybird endpoint that takes a required collectionSlug and returns one row of collection-wide metrics: project count, deduplicated unique contributors (code contribution or collaboration activity across member project segments), and rounded average health score. Membership is resolved with the same has(collectionsSlugs, …) pattern as insightsProjects_filtered.pipe on enabled projects in insights_projects_populated_ds.

Updates service-restart-fe-dev and clean-start-fe-dev in scripts/cli so additional Insights-related workers (e.g. registry sync, package workers, project discovery/evaluation) stay skipped when restarting only the frontend in dev, matching the existing “ignore heavy workers” behavior.

^{Reviewed by Cursor Bugbot for commit cc3b759. Bugbot is set up for automated code reviews on this repo. Configure here.}

…IN-1195

- Filter uniqueContributorCount in collection_insights_aggregate.pipe to code contributions/collaborations only, matching the existing segmentId_aggregates_snapshot.pipe convention (was inflated by stars, joins, and other non-contribution activity types). - Default showAggregateTabs based on collection type in createCollection (community collections with ssoUserId now default to false, matching the migration's backfill policy, instead of always defaulting true). - Fix accessible name association for the "Show aggregate tabs" switch by moving its description text into LfSwitch's default slot instead of a sibling span outside the label. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

collection.page.vue's featured-star toggle builds a CollectionRequest object literal that omits showAggregateTabs (and other optional fields), relying on the backend to preserve omitted values on partial update. The field was typed as required, which breaks that existing call site. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

+    SELECT
+        (SELECT count(distinct id) FROM collection_insights_aggregate_projects) AS projectCount,
+        uniq(ar.memberId) AS uniqueContributorCount,
+        (SELECT round(avg(healthScore)) FROM collection_insights_aggregate_projects) AS avgHealthScore
+    FROM activityRelations_deduplicated_cleaned_bucket_union AS ar
+    WHERE
+        ar.memberId != ''
+        AND ar.segmentId IN (SELECT segmentId FROM collection_insights_aggregate_projects)
+        AND (ar.type, ar.platform) IN (
+            SELECT activityType, platform
+            FROM activityTypes
+            WHERE isCodeContribution = 1 OR isCollaboration = 1
+        )


        ;;
    service-restart-fe-dev)
-        IGNORED_SERVICES=("frontend" "python-worker" "job-generator" "webhook-api" "profiles-worker" "organizations-enrichment-worker" "merge-suggestions-worker" "members-enrichment-worker" "exports-worker" "entity-merging-worker")
+        IGNORED_SERVICES=("frontend" "python-worker" "job-generator" "webhook-api" "profiles-worker" "organizations-enrichment-worker" "merge-suggestions-worker" "members-enrichment-worker" "exports-worker" "entity-merging-worker" "automatic-projects-discovery-worker" "pcc-sync-worker" "projects-evaluation-worker" "bq-dataset-ingest" "cargo-worker" "dockerhub-sync" "github-repos-enricher" "go-worker" "maven-worker" "npm-worker" "nuget-worker" "osv-worker" "pypi-worker" "rubygems-worker" "security-contacts-worker")


        ;;
    clean-start-fe-dev)
-        IGNORED_SERVICES=("frontend" "python-worker" "job-generator" "webhook-api" "profiles-worker" "organizations-enrichment-worker" "merge-suggestions-worker" "members-enrichment-worker" "exports-worker" "entity-merging-worker" "cache-worker" "categorization-worker" "cron-service" "data-sink-worker" "git-integration" "integration-run-worker" "integration-stream-worker" "nango-webhook-api" "nango-worker" "script-executor-worker" "search-sync-api" "search-sync-worker" "security-best-practices-worker" "snowflake-connectors-worker")
+        IGNORED_SERVICES=("frontend" "python-worker" "job-generator" "webhook-api" "profiles-worker" "organizations-enrichment-worker" "merge-suggestions-worker" "members-enrichment-worker" "exports-worker" "entity-merging-worker" "cache-worker" "categorization-worker" "cron-service" "data-sink-worker" "git-integration" "integration-run-worker" "integration-stream-worker" "nango-webhook-api" "nango-worker" "script-executor-worker" "search-sync-api" "search-sync-worker" "security-best-practices-worker" "snowflake-connectors-worker" "automatic-projects-discovery-worker" "pcc-sync-worker" "projects-evaluation-worker" "bq-dataset-ingest" "cargo-worker" "dockerhub-sync" "github-repos-enricher" "go-worker" "maven-worker" "npm-worker" "nuget-worker" "osv-worker" "pypi-worker" "rubygems-worker" "security-contacts-worker")


joanagmaia · 2026-07-14T12:51:58Z

+-- ssoUserId is only set for community/user-curated collections (see V1772438175), so
+-- ssoUserId IS NULL identifies curated collections.
+UPDATE collections
+SET "showAggregateTabs" = true


Why do we need this column? Can't we simply infer this from wether or not ssoUserId is null?

joanagmaia · 2026-07-14T12:52:23Z

+              <!-- Show aggregate tabs -->
+              <article class="mb-6">
+                <lf-field label-text="Show aggregate tabs">
+                  <div class="flex items-center gap-2">
+                    <lf-switch v-model="form.showAggregateTabs" size="small">
+                      <span class="text-2xs text-gray-400">Show in-depth aggregate metric tabs on the public collection page</span>
+                    </lf-switch>
+                  </div>
+                </lf-field>
+              </article>


Again, I don't think this is needed.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

+    // Only LF Foundation (curated) collections get aggregate tabs by default.
+    // ssoUserId is only set for community/user-curated collections, so
+    // ssoUserId being unset identifies curated collections (see V1784026542).
+    showAggregateTabs: !collection.ssoUserId,


+ALTER TABLE collections
+    ADD COLUMN "showAggregateTabs" boolean DEFAULT false NOT NULL;


        ;;
    service-restart-fe-dev)
-        IGNORED_SERVICES=("frontend" "python-worker" "job-generator" "webhook-api" "profiles-worker" "organizations-enrichment-worker" "merge-suggestions-worker" "members-enrichment-worker" "exports-worker" "entity-merging-worker")
+        IGNORED_SERVICES=("frontend" "python-worker" "job-generator" "webhook-api" "profiles-worker" "organizations-enrichment-worker" "merge-suggestions-worker" "members-enrichment-worker" "exports-worker" "entity-merging-worker" "automatic-projects-discovery-worker" "pcc-sync-worker" "projects-evaluation-worker" "bq-dataset-ingest" "cargo-worker" "dockerhub-sync" "github-repos-enricher" "go-worker" "maven-worker" "npm-worker" "nuget-worker" "osv-worker" "pypi-worker" "rubygems-worker" "security-contacts-worker")


…ement Per product decision: which collections show the in-depth aggregate tabs doesn't need to be independently manageable per foundation - the insights frontend can derive this directly from collection type (curated vs community), which it already knows. Removes the migration, the DAL/admin UI wiring, and the per-foundation CDP toggle entirely. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit cc3b759. Configure here.}

cursor · 2026-07-14T13:21:11Z

+        AND has(
+            collectionsSlugs,
+            {{ String(collectionSlug, description="Filter by collection slug", required=True) }}
+        )


Missing empty segmentId filter

Medium Severity

The projects node does not exclude rows where segmentId is an empty string before those ids feed the activity IN filter. Elsewhere (e.g. leaderboards_members, health_score_copy) empty segmentId values are dropped. Matching on '' can pull unrelated activity rows and inflate uniqueContributorCount.

^{Reviewed by Cursor Bugbot for commit cc3b759. Configure here.}

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

+    SELECT
+        (SELECT count(distinct id) FROM collection_insights_aggregate_projects) AS projectCount,
+        uniq(ar.memberId) AS uniqueContributorCount,
+        (SELECT round(avg(healthScore)) FROM collection_insights_aggregate_projects) AS avgHealthScore
+    FROM activityRelations_deduplicated_cleaned_bucket_union AS ar
+    WHERE
+        ar.memberId != ''
+        AND ar.segmentId IN (SELECT segmentId FROM collection_insights_aggregate_projects)
+        AND (ar.type, ar.platform) IN (
+            SELECT activityType, platform
+            FROM activityTypes
+            WHERE isCodeContribution = 1 OR isCollaboration = 1
+        )


        ;;
    service-restart-fe-dev)
-        IGNORED_SERVICES=("frontend" "python-worker" "job-generator" "webhook-api" "profiles-worker" "organizations-enrichment-worker" "merge-suggestions-worker" "members-enrichment-worker" "exports-worker" "entity-merging-worker")
+        IGNORED_SERVICES=("frontend" "python-worker" "job-generator" "webhook-api" "profiles-worker" "organizations-enrichment-worker" "merge-suggestions-worker" "members-enrichment-worker" "exports-worker" "entity-merging-worker" "automatic-projects-discovery-worker" "pcc-sync-worker" "projects-evaluation-worker" "bq-dataset-ingest" "cargo-worker" "dockerhub-sync" "github-repos-enricher" "go-worker" "maven-worker" "npm-worker" "nuget-worker" "osv-worker" "pypi-worker" "rubygems-worker" "security-contacts-worker")


        ;;
    clean-start-fe-dev)
-        IGNORED_SERVICES=("frontend" "python-worker" "job-generator" "webhook-api" "profiles-worker" "organizations-enrichment-worker" "merge-suggestions-worker" "members-enrichment-worker" "exports-worker" "entity-merging-worker" "cache-worker" "categorization-worker" "cron-service" "data-sink-worker" "git-integration" "integration-run-worker" "integration-stream-worker" "nango-webhook-api" "nango-worker" "script-executor-worker" "search-sync-api" "search-sync-worker" "security-best-practices-worker" "snowflake-connectors-worker")
+        IGNORED_SERVICES=("frontend" "python-worker" "job-generator" "webhook-api" "profiles-worker" "organizations-enrichment-worker" "merge-suggestions-worker" "members-enrichment-worker" "exports-worker" "entity-merging-worker" "cache-worker" "categorization-worker" "cron-service" "data-sink-worker" "git-integration" "integration-run-worker" "integration-stream-worker" "nango-webhook-api" "nango-worker" "script-executor-worker" "search-sync-api" "search-sync-worker" "security-best-practices-worker" "snowflake-connectors-worker" "automatic-projects-discovery-worker" "pcc-sync-worker" "projects-evaluation-worker" "bq-dataset-ingest" "cargo-worker" "dockerhub-sync" "github-repos-enricher" "go-worker" "maven-worker" "npm-worker" "nuget-worker" "osv-worker" "pypi-worker" "rubygems-worker" "security-contacts-worker")


Copilot AI review requested due to automatic review settings July 14, 2026 08:13

github-actions Bot reviewed Jul 14, 2026

View reviewed changes

Copilot started reviewing on behalf of gaspergrom July 14, 2026 08:13 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

gaspergrom added 3 commits July 14, 2026 10:38

Copilot AI review requested due to automatic review settings July 14, 2026 12:09

Copilot started reviewing on behalf of gaspergrom July 14, 2026 12:09 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

gaspergrom added 2 commits July 14, 2026 13:22

chore: add undo migration for showAggregateTabs column

bd5c2b1

Pairs with V1784026542 - DROP COLUMN reverses the ADD COLUMN + backfill, following this repo's occasional U<version> undo-migration convention. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>

gaspergrom requested a review from joanagmaia July 14, 2026 12:34

gaspergrom self-assigned this Jul 14, 2026

gaspergrom marked this pull request as ready for review July 14, 2026 12:34

cursor Bot reviewed Jul 14, 2026

View reviewed changes

Comment thread services/libs/tinybird/pipes/collection_insights_aggregate.pipe

Comment thread services/libs/data-access-layer/src/collections/index.ts

Merge branch 'main' into feat/collections-v2-aggregate-pipes-IN-1193-…

b2c18e1

…IN-1195

Copilot AI review requested due to automatic review settings July 14, 2026 12:37

Copilot started reviewing on behalf of gaspergrom July 14, 2026 12:38 View session

cursor Bot reviewed Jul 14, 2026

View reviewed changes

Comment thread services/libs/tinybird/pipes/collection_insights_aggregate.pipe

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 14, 2026 12:43

Copilot started reviewing on behalf of gaspergrom July 14, 2026 12:44 View session

cursor Bot reviewed Jul 14, 2026

View reviewed changes

Comment thread services/libs/tinybird/pipes/collection_insights_aggregate.pipe

gaspergrom changed the title ~~feat: Collections v2 aggregate Tinybird pipes and toggle column~~ feat: Collections v2 aggregate Tinybird pipes and toggle column (IN-1193) Jul 14, 2026

gaspergrom changed the title ~~feat: Collections v2 aggregate Tinybird pipes and toggle column (IN-1193)~~ feat: collections v2 aggregate Tinybird pipes and toggle column (IN-1193) Jul 14, 2026

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 14, 2026 12:50

Copilot started reviewing on behalf of gaspergrom July 14, 2026 12:50 View session

joanagmaia reviewed Jul 14, 2026

View reviewed changes

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 14, 2026 13:19

Copilot started reviewing on behalf of gaspergrom July 14, 2026 13:20 View session

cursor Bot reviewed Jul 14, 2026

View reviewed changes

gaspergrom requested a review from joanagmaia July 14, 2026 13:23

Copilot AI reviewed Jul 14, 2026

View reviewed changes

joanagmaia approved these changes Jul 14, 2026

View reviewed changes

gaspergrom merged commit 7a486b3 into main Jul 14, 2026
15 checks passed

gaspergrom deleted the feat/collections-v2-aggregate-pipes-IN-1193-IN-1195 branch July 14, 2026 13:27

		ALTER TABLE collections
		ADD COLUMN "showAggregateTabs" boolean DEFAULT false NOT NULL;

Uh oh!

Conversation

gaspergrom commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Deploy order

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

joanagmaia Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jul 14, 2026

Choose a reason for hiding this comment

Missing empty segmentId filter

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gaspergrom commented Jul 14, 2026 •

edited

Loading

github-actions Bot left a comment •

edited

Loading

cursor Bot commented Jul 14, 2026 •

edited

Loading