Vibe Spaces for Creatively Connecting and Expressing Visual Concepts

Intuition

We illustrate our method using a 2D latent space example. Our approach curves around the non-linear manifold.

(1) Example Data Manifold

(2) Path Finding Process

(3) How Gradient of Ncut Vectors Guides Path Finding

We construct a graph diffusion space that captures the manifold's intrinsic geometry. Path finding proceeds in two steps: (1) linearly interpolate Ncut vectors \(\mathbf{\Psi}_t(\mathbf{x}_\alpha) = (1-\alpha)\mathbf{\Psi}_t(\mathbf{x}_A) + \alpha\mathbf{\Psi}_t(\mathbf{x}_B)\), then (2) use inverse mapping to recover the corresponding point in the original space: $$ \gamma(\alpha) = \arg\min_{\mathbf{x^*}} \big\| \mathbf{\Psi}_t(\mathbf{x^*}) - \mathbf{\Psi}_t(\mathbf{x}_\alpha) \big\|_2^2 $$

Ncut vectors (eigenvectors \(\mathbf{\Psi}\)) are computed before encoder training. In this 2D example, step (2) uses gradient descent (visualized in (3)) because the gradient \(\nabla_{\mathbf{x}^*}\mathbf{\Psi}_t(\mathbf{x}^*)\) is directly computable. For real images, we instead use the learned Vibe Space encoder-decoder to obtain the optimal point.

Overview

Forward Mapping:

• (a.) Compute affinity graph W and graph Laplacian L
• (b.) Compute Ncut eigenvectors Ψ(x) of L as manifold coordinates

Inverse Mapping:

• (c.) Interpolate in manifold space and invert to recover the path in feature space

Application to Real Images:

• (d.) Extract DINO patch tokens as graph nodes, compute affinity W
• (e.) Top m eigenvectors yield co-salient segments and manifold coordinates
• (f.) Apply inverse mapping to CLIP space, generate images via Stable Diffusion IP-Adapter

Why Flag Space Multi-Scale Loss

Graph Laplacian eigenvectors capture geometry at different scales: leading eigenvectors describe global structure, while higher-order eigenvectors encode local variations. Truncating to a fixed number \(m\) of eigenvectors selects one scale, leading to paths that focus on too many or too few attributes.

We use a flag space, a hierarchy of nested embeddings \(\mathbf{\Psi}^{1:m_1} \subset \mathbf{\Psi}^{1:m_2} \dots \subset \mathbf{\Psi}^{1:m_M}\), to encapsulate both coarse and fine manifold structures. The multi-scale path is recovered by minimizing the average reconstruction error across scales: $$ \gamma(\alpha) = \arg\min_{\mathbf{x^*}} \frac{1}{|\mathcal{M}|} \sum_{m_k \in \mathcal{M}} \big\| \mathbf{\Psi}_t^{1:m_k}(\mathbf{x^*}) - \mathbf{\Psi}_t^{1:m_k}(\mathbf{x}_\alpha) \big\|_2^2 $$ This enforces consistency across global and local geometry when finding paths between points.

Why Path Finding by Graph Diffusion

Unlike discrete shortest path algorithms like Dijkstra's algorithm, which operate on a fixed graph and find paths through existing nodes, our approach using graph diffusion is robust to leaks and outliers in the data manifold.

Dijkstra's algorithm fails when data contains noise and leaks because it relies on direct graph connectivity. A single noisy edge or outlier can create spurious shortcuts that lead the algorithm astray. In contrast, graph diffusion captures the global structure of the manifold through spectral decomposition, effectively filtering out local noise and outliers.

How Vibe Space is Trained to Approximate Inverse Mapping

For real images, gradient descent on the full graph is computationally expensive. Instead, we train two lightweight MLP networks to approximate the forward and inverse mappings:

• Encoder: Maps input features \(\mathbf{x}\) to diffusion coordinates \(\mathbf{\Psi}_t(\mathbf{x})\), simulating the forward mapping
• Decoder: Maps diffusion coordinates back to feature space, approximating the inverse mapping

Training takes under 30 seconds and ensures that Euclidean distances and linear paths in Vibe Space correspond to geodesic distances and semantic paths on the underlying manifold. This enables efficient path finding without gradient descent at inference time.

Why Computing Latent Space from Few Examples Enables Vibe Identification

We compute the latent space from only 2-4 image examples rather than the entire training dataset. This context-specific manifold captures the dominant shared attributes—the "vibe"—between the specific images being blended.

Our approach: By constructing the graph Laplacian from only the input images, the leading eigenvectors naturally capture the most relevant attributes for that specific pair.

Alternative approaches (e.g., Yu et al.): Methods that compute the latent space from the entire training dataset create a global manifold encoding all possible relationships. This global perspective cannot identify which attributes are most relevant for blending a specific pair, as it mixes irrelevant relationships from the full dataset.

Left: Local Vibe Space (Few Examples) — focused path capturing dominant attributes
Right: Global Latent Space (All Data) — dense manifold with multiple paths obscuring specific relationships