Some Vector Algebra for Model Steering

Toward an Interface for Generative Model Output Manipulation via High-Dimensional Space Navigation

Like a painter, like a palette by their side,
Reaching for a hue that no tube can provide.

Interpolating high dimensional vectors can work like mixing paint on a palette: each color stands for a vector in latent space, and blending them at the center gives an interpolated vector. Each paint contributes in proportion to how close it sits to the mixing point. This type of interface can steer what a generative model produces. Drag the mixing point z, or the paints z_1 \ldots z_7 themselves. Under "wander," z sets the mixing point drifting and z_i the paints.

Generative models encode data and behavior in high-dimensional representation spaces, where surprisingly simple vector operations can often produce meaningful changes in model outputs. This article develops an intuitive geometric view of model steering through three basic ingredients: a base point, a direction, and a step size. Using a deterministic generative model (StyleGAN2) trained on a dataset of human faces (FFHQ) as a running example, we show how semantic directions such as age, pose, gender, and eyeglasses can be constructed from contrastive examples and transferred across different faces. These examples also expose the limits of global directions, from entangled attributes to changes that vary across the latent space, which motivates an interface built from a global map, contrastive directions and local interpolation.

King − Man + Woman ≈ Queen

Many people have encountered the famous example of king − man + woman ≈ queen observed in word embedding models . Although surprisingly simple and intuitive, this example demonstrates an important idea: relationships become directions in a high-dimensional space. Arithmetic on these vectors does not merely manipulate numbers—it manipulates meaning. This observation provides an intuitive starting point for understanding how similar vector operations can steer modern generative AI systems.

Steering Generative Models

Generative AI is now foundational across domains: language models generate text and code, diffusion models synthesize images and video, speech models produce audio, and scientific models assist with protein design and materials discovery. However different, these systems learn rich internal representations that can be navigated and controlled. Activation steering uses this to influence a model without retraining it: injecting chosen directions into its internal representations can encourage desired behaviors, suppress undesirable ones, improve safety or guide generation toward specific concepts, changing where the computation moves rather than modifying the model itself.

The representations differ: StyleGAN2 maps a random vector to an intermediate style code, diffusion models steer a denoising process, and language models carry information through sequences of activations. Yet in each, generation can be read as navigation in a high-dimensional space: we choose where to begin, find a direction and decide how far to travel, and much of the apparent magic rests on a little vector addition and subtraction.

Latent Spaces: Internal Representations of Data

To understand how steering works, we first need to understand latent spaces. In image generator models such as StyleGAN2, every generated image is represented internally by a latent code, a high-dimensional vector encoding the characteristics of the image. Nearby vectors generally produce similar outputs, while moving along particular directions changes interpretable attributes such as age, hairstyle, facial expression, or pose. Instead of editing pixels directly, we manipulate these latent vectors, allowing modifications that remain consistent with the distribution learned by the model.

Generative model is composed of a seuqence of computation blocks, transforming input latent code (e.g., samples from a gaussian distribution) into intermediate/latent representations. By carefully choosing which latent space we work on for manipulation/steering, we are able to plot a sensible map of the entire latent landscape. As an example, here we use StyleGAN2's 32x32 convolutional block output, average downsampled to 5x5 feature maps, and use UMAP to reduce the 5x5x512 dimensional vectors to 2D. below we show a couple of semantically meaningful regions we found in this generative landscape:

The latent landscape of StyleGAN2/FFHQ, laid out flat: every point is one latent code sample projected onto 2D by UMAP. Distance on this map represents similarity in the latent space, so neighbourhoods read as kinds of face — babies in the middle, eyeglasses in their own island on the right.

Three Ingredients for Navigation
Where to Begin, Which Direction to Take, and How Far to Travel

The math behind many steering techniques is remarkably simple: Starting from the latent vector of one example, we add or subtract another vector representing a direction. These operations often produce smooth, meaningful transformations despite involving nothing more than elementary linear algebra. In this session we explain the contextural meaning behind adding and subtracting vectors in the latent space. And show how it behaves in real-world datasets and generative models.

The base: where do we start?

As with any journey across a map, we first have to decide where it begins. And in everyday life we rarely name that starting point by its coordinates. Latitude and longitude are precise and useful, but they are not how people actually navigate a place; we reach instead for landmarks — familiar checkpoints, and positions described relative to them.

Latent space works the same way. Its landmarks are the sample points the dataset already gives us, and each one also has an exact address: the high-dimensional coordinates that play the role of latitude and longitude. So a journey begins at some sample $z_i$, written as the vector z_i = (z_{i1}, z_{i2}, \ldots z_{iM}). Taking a weighted average of several such points is the latent space version of placing somewhere new by its position among the landmarks around it — and because the average moves continuously as we reweight, it lets us navigate the space continously rather than jumping between known samples.

The direction \vec{d}: which way do we go next?

The direction determines which way the base point moves in the latent space. Given two groups of examples of opposite concepts (e.g., short and long hair), the difference between their average latent representations captures what distinguishes them and steers a base point toward either concept. In the simplest form, each group is a single example: \vec{d} = z_{pos} - z_{neg}

The step size \alpha: how far should we go?

Once we know which direction to move, the next question is how far to go. The step size sets the strength of the change: a small step gives a subtle change, a larger one pushes the result farther along the direction. With \alpha measured relative to the distance between the positive and negative examples, the steered point is z_{new} = z_0 + \alpha (z_{pos} - z_{neg})

Here, \alpha is the step size. Its magnitude determines how strongly the attribute is changed, while its sign determines whether we move along the direction or in the opposite direction.

Where do we start? A map of the global latent space

Given a map of the latent space, we can browse all of it by sweeping a lens across the map, continuously blending the samples under the lens into a single latent vector (a weighted interpolation we detail in the next section). This lifts movement on the 2D map back into the high-dimensional space: as the lens moves, it traces a continuous 2D surface of interpolated latent vectors. Unlike a linear projection, which is confined to a single linear subspace, this surface can bend through the space, so simple two-dimensional navigation reaches a richer variety of the latent landscape.

The map of samples from a latent space of StyleGAN2 model, with one viewing lens on it. Dragging the lens takes weighted average across the 10,000 StyleGAN2 samples (FFHQ faces, or AFHQ cats in the second tab). The picture inside the lens (also enlarged and shown on the right) is decoded from a weighted mean of style vectors through the generator. Press wander to let the lens walk the map on its own.

Where exactly do we start? A fine-tunable control pad for local navigation

A landmark is rarely exactly where we want to be. Rather than snapping to the nearest sample, we can place a point among several of them and let each one pull on it according to how close it is. One way to think about this interaction is as a color-mixing palette: moving reference examples, like different paints, closer to the central interpolation point gives them greater weight in the resulting mixture. Another analogy is an octopus pulling on a collection of marbles: examples closer to the control point exert a stronger influence, while more distant examples contribute less.

awwwwwwww! Screenshot from the Terraria wiki: Plantera.

awwwwwwww! Screenshot from the Terraria wiki: Plantera.

The mixing interface for three dimensions, where every sample z_i is a 3D point. A layout of the z_i in the interface (left) embeds a 2D mesh in 3D (right). Drag z on either side and watch it slides over the mesh; drag a z_i and watch the parametrization or the mesh itself changes. Drag empty space to in 3D view to rotate, shift-drag to pan, scroll to zoom, double-click to reset. In the color mixing example, double-click a paint to pick its color. This color palette image is generated by AI.

Specifically, we combine the latent codes of the reference samples, $z_i$, using a weighted linear combination: z = \sum_{i=1}^{K} w_i z_i, where each weight $w_i$ is determined by the distance $d_i$ between reference point $i$ and the central control point in the 2D interface. The actual choice of the weight function as a function is very flexible. For example, consider the two weighting schemes: w_i = \frac{1/d_i}{\sum_{j=1}^{K} 1/d_j} \;\text{ or }\; w_i = \frac{1-smoothstep(d_i / R)}{\sum_{j=1}^{K} 1-smoothstep(d_j / R)} The first uses inverse-distance weighting, while the second uses a smooth, compactly supported weighting function. Here, $R$ denotes the cutoff radius: reference points beyond this radius receive zero weight, while points closer to the control point contribute progressively more to the interpolation.

The interface therefore behaves partly like PromptPaint : pulling the control point toward an example raises its share of the mixture, much as adding more of one paint makes its color more prominent. Moving $z$ in 2D continuously changes the weights for high dimensional mixing, and so moves the mixture through a 2D subspace of the latent space, whose shape depends on both the weighting function and the arrangement of the points around $z$. In the octopus figure, drag $z$ and the $z_i$ to get a feel for how they shape this 2D surface of linear combinations.

Unlike a slider, which moves along one predefined direction, one 2D gesture reweights many examples at once, so the layout of the neighborhood around the mixing point matters as much as the weighting function: it decides which subspace the interface can reach. The mesh on the right of the figure above is that subspace, and rearranging the samples on the left reshapes it; we return to what makes a good layout in the discussion.

Which way do we go next? A direction field defined by contrastive pairs

As we have seen, a steering direction can be defined by contrasting two sets of examples: a positive group and a negative group. The difference between their aggregated latent representations determines a semantic direction in the latent space. The global browsing interface can therefore serve not only as a tool for exploration, but also as an interactive mechanism for selecting and constructing these contrastive groups directly from the data. By browsing the latent landscape and identifying representative regions, users can define steering directions visually rather than specifying them through individual examples or predefined labels.

As a simple example that we have explained, a direction can be defined by subtracting two endpoints: an example representing the desired destination minus an example representing the opposite concept. In this case, the direction is specified using only two examples.

\vec{d} = z_{pos} - z_{neg}

More generally, a direction can also be defined as the difference between the averages of two groups, or as the average of the differences between matched positive–negative pairs.

\vec{d} = \text{average}(\{\cdots z_{pos} \cdots\}) - \text{average}(\cdots z_{neg} \cdots\}) or \vec{d} = \text{set\_difference}(\{\cdots z_{pos} \cdots\}, \{\cdots z_{neg} \cdots\})

Contrastive pairs are only one way to get a direction. Any vector field over the latent space can serve, and a direction can also be derived from the model's local geometry, for example from a basis given by its Jacobian matrix. Contrastive pairs are the simpler and computationally cheaper choice.

Example: Adjusting Facial Features

We use faces to demonstrate the basic idea of vector arithmetic in generative models:

One direction, applied everywhere, on StyleGAN2 faces (FFHQ) or cats (AFHQ). Each tab is a contrastive pair, drawn as its neg and pos rings and the arrow between them, and the map is tiled with decodes of base + α (pos − neg), the lens mix at each tile steered by that pair. Drag the slider or press play to change α; hover a tile to preview it in the corner card, and click to keep it. Scroll or pinch to zoom into finer grids.

On the same StyleGAN2/FFHQ map as above, laid out by UMAP , we apply the base + direction algebra in the style space of StyleGAN2, trained on the FFHQ dataset . Analogous to the classic king–queen example, we compute semantic directions corresponding to attributes such as pose, age, gender, and eyeglasses. Applying these directions to different faces reveals somewhat consistent transformations across identities. Rather than overfitting to individual images, the latent space captures reusable semantic concepts that can be transferred through vector addition.

A direction defined by the difference between two sets of examples works well in style space, but it is only an approximation and easily picks up correlated attributes: in the example above, the gender direction also changes hair length. The fewer the examples in each group, the worse this gets, since their difference captures several correlated variations at once. Ideally the two ends would differ in a single feature, but such matched pairs are rare in the data. This is what the local mixing of the previous section is for: the base, positive and negative points need not be individual examples, and each can be a weighted mix of nearby ones, adjusted until the unwanted variations cancel.

Beyond Contrastive Pairs

Contrastive examples can also come from elsewhere other than a map-based interface, such as manual annotation or crowdsourcing. And, as noted above, the single global direction used here is the simplest case of a vector field, whose direction can depend on where it is applied. This part of the interface is meant for exploration, to get an initial feel for the data landscape and experiment with steering, rather than for precise control or for defining contrastive pairs at scale.

How far should we go? A slider control

How far one steer travels, on the map. Each tab is a contrastive pair, drawn as its neg and pos rings and the arrow between them. Hover the map to pick a base point: the solid line is where z_0 + \alpha (z_{pos} - z_{neg}) lands as \alpha grows, each step placed by the parametric UMAP and shown as its decoded picture. The path bends away from the arrow because UMAP is not linear. The tiles are the unsteered grid (\alpha = 0), and the corner card shows the hovered point at the slider's step; play sweeps \alpha. Scroll or pinch to zoom into finer grids, and drag to pan.

Setting the step size means deciding how far to follow the direction arrow. Read as a vector field \vec{v}(z) over the latent space, steering is a finite integration along it: from z_0, each step moves a little along the arrow the field gives at the current point,

z_{t+1} = z_t + \Delta\alpha\, \vec{v}(z_t),

and the step size \alpha is how far the integration runs. In the simplest case the field is the same everywhere, a single global steering direction \vec{v} = z_{pos} - z_{neg}, and following it uniformly collapses the integration to one straight step, z_0 + \alpha (z_{pos} - z_{neg}): the path in the figure above. How far that path can usefully go is then a question of where it leaves the region of valid outputs, which we return to in the discussion.

Together, the examples show how a map-based interface supports each stage of steering: browsing the latent space, picking examples to define a direction, and refining it locally through interpolation.

Towards a Unified Model Steering Interface

Dimensionality reduction lays a latent space out as a 2D map to explore with the generative model, but no 2D projection preserves all of a high-dimensional space's structure. How, then, should an interface support navigation within such a space? Part of the answer is that the three ingredients need not share an interface, or even a space. In our prototype they already part: the map projects the generator's internal activations, the mixing and steering happen in its style space, and how far a step may go can be judged on the decoded outputs. A unified interface has to coordinate these, letting the user place a base on one map, define a direction in another and set the step size and its boundary against a third, while keeping the views consistent.

Discussions

Desired Properties for Inverse Design

The steering mechanism in this article was designed by hand, but an interface like the octopus or the palette is, in general, a mapping from 2D interaction to motion in a high-dimensional latent space, and the properties we want of that mapping can guide its design toward one that both matches our intuition and respects the latent geometry. We outline several for the 2D surface the control point sweeps out over a layout of reference points; the weight functions' own properties are in the technical details.

Coverage: Because the reference points can be moved freely, the interface can reach any convex combination of them, that is, any point of the simplex they span. Reaching one may take several moves, but like a set of sliders it loses nothing: the 2D interface offers a lower-dimensional way through the full set of mixtures.

Choice of Anchor Points: The anchors decide which simplex that is. Anchors close together span a small region whose mixtures stay near the data; anchors far apart reach further, but their mixtures can leave the valid region (in a diffusion model's latents, an average of distant samples decodes to little more than a blur). Which anchors, and how many, trades reach against simplicity, a choice that could be guided by the data's density or the diversity of the selection.

Less Twisted Surfaces: The layout decides how the surface sits in the latent space. If neighbors in the layout are far apart in latent space, the surface folds over itself and a small move produces a large, erratic change. A layout whose neighbors are also latent neighbors avoids this, for instance by ordering the anchors along a shortest Hamiltonian path through their latent positions (NP-hard exactly, though a heuristic may do for a handful of anchors). In the same spirit, codes that co-occur in the model's output would ideally sit close together, so they are activated together in the interface.

What Interactions Imply About the Subspace: Dragging the control point z moves within the current surface; dragging a reference point z_i moves or reshapes the surface itself, like turning a projection plane to see another slice. Navigation thus works within a subspace and between subspaces, and showing users how the surface sits in the latent space, as the 3D view does, helps them predict what a gesture will do.

Boundary Conditions and Constraints

So far we have treated every direction as available and every point as reachable. In practice, not every location in a generative model's representation space yields a meaningful, valid or desirable output, so a steering interface has to consider not only where a user can move but also where they should. The reasons vary: in safety-critical applications some regions are harmful or prohibited; in engineering and design, physical laws, fabrication limits or interlocking geometry rule out combinations that can still be represented numerically; in science, domain constraints decide whether a generated material, molecule or structure is plausible.

One view separates the ambient space from its valid subset. Under the manifold assumption, meaningful samples lie on a lower-dimensional region and navigation should stay close to it; alternatively, the model's density gives a soft boundary, keeping exploration where the likelihood exceeds a threshold. Constraints can also become part of the interaction itself: an interface could project a proposed move back onto the valid region, bend the available directions near a boundary, or warn as a trajectory approaches infeasible space. Building such constraints in is beyond the scope of this article, but it is what would turn latent-space navigation from an explanatory metaphor into a practical interface.

Beyond Latent Space

The same intuition extends beyond latent representations, to methods that modify the model itself. A Low-Rank Adaptation (LoRA) is a compact parameter update that can be scaled, combined or contrasted with others, and systems such as SliderSpace expose such updates as continuous controls, much like the step size \alpha along a latent direction. The framework of base + direction + step size thus applies more broadly than latent-space editing: the base may be a latent code, an activation state, a model input or the model's parameters; the direction a semantic vector, an activation difference or a low-rank update; and the step size the strength of the intervention.

Conclusion

The equation king − man + woman ≈ queen carries a simple language for model steering: choose a base, a direction and a step size. That picture is an approximation, since directions entangle correlated attributes, vary across the latent space and resist any single 2D projection, which is why we pair a global map with contrastive directions and local interpolation. Vector arithmetic thus serves not only as a mathematical tool, but as an abstraction for designing more interpretable and controllable interfaces to generative models.

Technical Details

Example: Projection of a shape with known topology: Klein Bottle

To see what interpolating over a 2D map does to a shape whose structure we know, we use the Klein bottle . Topologically it is a flat square (\theta, v) \in [0, 2\pi)^2 with one coordinate, v, wrapping around directly and the other, \theta, wrapping with a flip: (\theta + 2\pi, v) \sim (\theta, \pi - v). The flip makes it a closed, non-orientable surface that cannot sit in three dimensions without passing through itself; it embeds without self-intersection only in four dimensions, for instance by

\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \end{bmatrix} = \begin{bmatrix} a\,(\cos\frac{\theta}{2}\cos v - \sin\frac{\theta}{2}\sin 2v) \\ a\,(\sin\frac{\theta}{2}\cos v + \cos\frac{\theta}{2}\sin 2v) \\ b\cos\theta\,(1 + \epsilon\sin v) \\ b\sin\theta\,(1 + \epsilon\sin v) \end{bmatrix},

with constants a, b and \epsilon. Our samples come from this embedding, and UMAP projects their four coordinates to a plane. Since no 2D embedding is free of self-intersection, the projection has to tear the surface into pieces: for about four in five samples, the nearest neighbor on the map is not the nearest neighbor on the surface. The figure triangulates the samples on the flat square itself, so edges stretch across the map, showing where it tore. Interpolation of a point z in the map uses the same weight functions as in the previous sections. Beside the weights, it shows where the mixture lands on the square and its four coordinates, so a smooth path on the map can be compared with where it actually goes on the surface.

Drag z, or let it wander.

Choice of Weight Functions

The weight a reference point receives at distance d from the control point, before normalizing. Toggle each weight function and hover the plot to read every visible curve at one distance; drag R to move the cutoff all four share.

In principle the weighting function can be any decreasing function of distance: nearby examples contribute more, distant ones less. The figures offer four, each reaching zero at a cutoff radius R: 1/d - 1/R, inverse distance shifted down to vanish at the cutoff; the straight ramp 1 - d/R; a smoothstep that flattens into zero at R; and the logarithm w_i = \frac{-\log(d_i / R)}{\sum_{j=1}^{K} -\log(d_j / R)} \;\;\text{ for }\;\; d_i < R, with w_i = 0 beyond. The logarithm diverges as d_i \to 0, like inverse distance, so a control point placed on a sample is dominated by it, and it meets the cutoff with a corner, behaving like (R - d_i)/R, so a sample leaving range stops counting abruptly rather than fading out. Its base is immaterial: normalization divides any constant factor back out.

Because all four have a cutoff, all four give a local mixture: only samples within R contribute, and a control point far enough from every sample leaves the mixture empty. The radius sets the trade-off. A larger R gives a better-estimated local mean that changes gradually as the control point moves; a smaller one commits to a few samples and rewrites the mixture faster.

Within that radius the functions differ in where they discriminate, which is how much of the neighborhood one control point speaks for. The smoothstep is nearly flat close in and spends its discrimination near the rim, giving a strictly local mixture with a nearly uniform interior, much like a smooth k-nearest-neighbor average. Inverse distance responds most sharply close in, so the nearest samples dominate, and the logarithm spreads its falloff more evenly between the two; both stay anchored to the nearest sample while borrowing from the rest, the case where the intended contrast is carried by one example and refined by its neighbors.

Three properties of the navigation surface follow from the weight function. All four are continuous in distance, so the mixture moves continuously as z does, and a smooth one gives a smooth surface with no abrupt change in the output. The two that diverge at zero, 1/d - 1/R and -\log(d/R), pass through every reference point: as z nears one, its normalized weight tends to one and all others to zero. The ramp and the smoothstep need not, so a control point placed on a reference point can still give its neighbors weight. Finally, the cutoff makes every mixture sparse, drawing only on nearby samples, which limits its effective dimensionality and suits models that favor sparse codes themselves, such as sparse autoencoders.