Add LargeVis manifold embedding - #345
Open
RicardoSantos-99 wants to merge 4 commits into
Open
Conversation
Adds Scholar.Manifold.LargeVis, the optimization half of LargeVis: takes
the approximate k-NN graph that Scholar.Neighbors.LargeVis already builds
and produces a low-dimensional embedding from it, the same relationship
Scholar.Manifold.TSNE has to an exact affinity matrix.
The k-NN graph is directed and each point has a fixed k neighbors, which
fits a dense {n, k} tensor fine, but symmetrizing it the way t-SNE does
(p + p^T) needs the reverse graph, whose in-degree per node is unbounded
and does not fit a fixed shape. This treats the graph as a flat edge list
of (source, target, weight) triples instead: the reverse graph is that
same list with the two index columns swapped, still fixed size, and
concatenating forward and reverse edges is exactly equivalent to summing
the dense p + p^T without ever forming it. Verified this identity exactly
on small n by building both the dense and edge-list versions and diffing
to 0.0.
The embedding itself is optimized with stochastic gradient descent,
sampling positive edges from that list proportional to weight and
negative pairs uniformly at random, using the same Cauchy low-dimensional
kernel t-SNE uses. No canonical or deduplicated edge set is needed since
duplicate edges just get sampled more often in proportion to their
combined weight.
scikit-learn has no LargeVis equivalent to validate against, so the test
checks neighborhood purity instead: on synthetic clusters, how many of
each point's nearest neighbors in the fitted embedding share its true
cluster. Consistently at or near 1.0 across several seeds, well above a
PCA baseline on the same data (which drops as clusters get closer
together), and well above the 1/num_clusters a meaningless embedding
would score.
Related to elixir-nx#272.
Scholar.Neighbors.LargeVis returns each point as one of its own neighbors with distance 0. Left in, that entry dominates the softmax over the neighborhood (it was absorbing about half of each row's probability mass), distorting the perplexity search and wasting the positive samples drawn on self edges, whose gradient is zero. Masking its distance to infinity zeroes both its affinity and its sampling probability. On the synthetic blobs benchmark this roughly doubles convergence speed: neighborhood purity at 30 iterations goes from 0.43 to 0.69 and at 60 iterations from 0.66 to 0.93. Since one neighbor slot is taken by the point itself, perplexity now must be smaller than num_neighbors - 1. Also adds tests for f32 and integer inputs, determinism under a fixed key, the squared_euclidean metric and degenerate data with identical points.
The affinity kernel was applied to whatever distances the chosen metric returned, so the default :euclidean produced exp(-d / 2 sigma^2) instead of the Gaussian exp(-d^2 / 2 sigma^2) the LargeVis paper (and t-SNE) use. Unlike constant factors, the kernel shape is not absorbed by the perplexity search. Squaring is monotone, so both metrics find the same neighbors and now produce the same embedding; the metric option only affects the graph construction. Measured quality is unchanged.
CI runs mix test on the interpreted binary backend, where the quality test at 400 iterations over 100 points exceeded the 60s ExUnit timeout. Shrinks it to 3 clusters of 15 points and 150 iterations, which still reaches a neighborhood purity above 0.99 across seeds while running in a few seconds.
josevalim
approved these changes
Jul 23, 2026
Member
|
@RicardoSantos-99 I saw the pull request, it's just that I am extremely busy these days (mostly interviewing) to review it in detail. Will have a look next week. @josevalim I need more time for this one as it might be an important algorithm. I hope that is fine. |
Contributor
|
No worries, take your time! If anyone needs the algorithm in the meanwhile, they can try the PR too! |
Contributor
Author
|
Sounds good, thanks. No rush, @krstopro do it whenever you have the time. Let me know if anything needs explanation or changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #272.
Adds
Scholar.Manifold.LargeVis, the dimensionality-reduction half of LargeVis, on top of the existingScholar.Neighbors.LargeVisapproximate k-NN graph. The API mirrorsScholar.Manifold.TSNE:fit/2returns the embedding tensor.The blocker discussed in the issue was symmetrizing the k-NN graph, which normally needs the reverse graph, whose in-degree per node is unbounded and does not fit a fixed-shape tensor. This avoids materializing the reverse graph at all: the k-NN graph is kept as a fixed-size edge list of
(source, target, weight)triples, and its reverse is the same list with the two index columns swapped, still fixed size. Concatenating the two is exactlyP + P^T, and the optimizer only ever samples individual edges, so the per-node grouping that the variable in-degree would require never comes up.The embedding is optimized with the paper's noise-contrastive objective: edges sampled proportional to weight are attracted, uniformly sampled pairs are repelled, under the t-SNE Cauchy kernel
1 / (1 + ||y_i - y_j||^2).Tests cover shape and options, JIT, f32/f64, determinism, degenerate input, and embedding quality (neighborhood purity on well-separated blobs, since there is no scikit-learn reference for LargeVis).
cc @krstopro