Standard PCA is a workhorse of dimensionality reduction, but it operates strictly on linear combinations of input features. When your data lies on a curved or otherwise nonlinear manifold, PCA will miss the structure entirely. Kernel PCA (KPCA) addresses this limitation, and understanding how it works illuminates why the choice of kernel function is one of the most consequential decisions you make when using it.

The Technical Explanation

The core idea behind KPCA is elegant: rather than performing eigen-decomposition on the raw data covariance matrix, you first implicitly map the data into a much higher-dimensional feature space through a nonlinear function \(φ(·)\), and then run ordinary PCA there. As Schölkopf et al. (1998) describe, you are performing linear PCA in a high-dimensional feature space that is non-linearly related to input space, such that principal eigenvector projections form nonlinear contours in the original space. The genius of the kernel trick is that you never have to compute \(φ(·)\) explicitly: you only ever evaluate inner products in that space through a kernel function \(k(xᵢ, xⱼ) = ⟨φ(xᵢ), φ(xⱼ)⟩\). This yields an n × n Gram matrix K, and the eigen-decomposition is performed on that matrix instead.

Or put simply: the algorithm can capture nonlinear patterns that are completely invisible to standard PCA. A recent survey of dimensionality reduction methods explains the mechanism concisely: KPCA maps data into a higher-dimensional feature space and uses the kernel trick to compute inner products there without ever explicitly constructing the feature vectors¹. Common kernel choices include the RBF (Gaussian), polynomial, and sigmoid kernels, and the right choice depends entirely on the geometry of your data.

Uh, ELI5 Please.

Imagine you have a bunch of points shaped like a circle on paper. Regular PCA can only draw straight lines through data, so it tries to find the "longest straight direction." But a circle has no long direction: everything is equally far around. Kernel PCA fixes this by doing something clever:

• Do some fancy math to lift the circle into 3D, like turning it into a bowl.
• Now the data does have a direction (top and bottom of the bowl).
• Do normal PCA in that pretend 3D world.
• Then project the results back down to 2D.

The trick: It never actually goes to 3D. Instead, it uses a kernel function to compute what the distances would be in that imaginary space.

Another analogy that helped me: PCA is like trying to figure out what an object is just by looking at its shadow. Kernel PCA is technically the same thing, but it moves the light around to extrapolate information about the object from its shadow. A slinky's shadow might just look like a zig-zag, while kernel PCA would reveal to you its 3D spiral shape.

What does this mean for me?

Standard PCA is good for data that more or less goes in a straight line when you plot it on a graph. If your data curves or wiggles, kernel PCA is the better choice.

Here's a quick comparison of kernel behaviors:

Understanding the Math

The core problem with PCA is that it can only find linear structure. If the data forms a curve in 2D, PCA will miss it, so the fix is to map data into a higher dimension where those nonlinear patters become linear, then run PCA there. However, this introduces a new problem: higher-dimensional space (3D, 4D, etc., which I shall refer to hereafter as XD) can be huge, even infinite, so computing coordinates there is expensive or impossible.

PCA in any space needs only one thing: dot products between data points (i.e. how similar each pair of points is). This means we don't really need to calculate points in XD space, we just need to compute the dot products between the points.

So how do we calculate point similarity if we don't know the points? First, let's look at the dot product equation:

\[ φ(x) · φ(z)\]

Given points \(x\) and \(z\), their dot product after mapping to XD space is \(φ(x) · φ(z)\). \(φ\) is the mapping function. Literally interpreted, remapped point \(x\) times remapped point \(z\) equals the dot product of points \(x\) and \(z\).

The trick is that for certain mapping functions \(φ\), this dot product has a cheap closed-form equivalent, a kernel function \(k(x, z)\), that you can compute directly in the original 2D space:

\[ φ(x) · φ(z) = k(x, z)\]

Say your data is 2D: \(x = (x₁, x₂)\). ...Actually, let's make it even more concrete and give these points real numbers: \(x = (2, 3)\). We'll keep it simple and say our second point, \(z\), also equals \((2, 3)\), so \(x = z = (2, 3)\).

If we were to approach this the hard, non-kernel function way, we'd have to extrapolate both x and z into 3D space. The formula for mapping to 3D is:

\[ φ(**x**) = (x₁², √2 x₁x₂, x₂²)\]

• \(x₁²\) is how much the first dimension "matters" on its own, squared.
• \(√2 · x₁x₂\) is how much the two dimensions interact with each other (the \(√2\) is just a balancing weight to make the algebra work out cleanly).
• \(x₂²\) is how much the second dimension matters on its own, squared.

So we plug in our \((2, 3)\) numbers:

\[ φ(2, 3) = (2², √2 · 2 · 3, 3²)\]

Which simplifies to:

\[ φ(2, 3) = (4, 6√2, 9)\]

Your 2D point \((2, 3)\) just became the 3D point \((4, 6√2, 9)\). You have three coordinates now instead of two, so you've moved into 3D space. Then we plug those coordinates into the dot product formula (remember that in our example, both \(x\)and \(z\) are \((2, 3)\):

\[ φ(x) · φ(z) = 4·4 + 6√2·6√2 + 9·9 = 16 + 72 + 81 = 169\]

Sweet, using the calculated 3D space, we were able to get the dot product.

Now let's put that on the back-burner for a second and look at the dot product formula again. The dot product \(φ(x) · φ(z)\) expands out to:

\[ x₁²z₁² + 2x₁x₂z₁z₂ + x₂²z₂²\]

...which equals \((x · z)²\). Huh, that's just squaring the original dot product. Which means, the formula for dot product \(φ(x) · φ(z)\) and our kernel function \(k(x · z)²\) should get the same result! Let's test it. First we calculate the original dot product:

\[ (x · z)² = (2·2 + 3·3) = (4 + 9) = 13\]

Now we apply the kernel by squaring the dot product:

\[ k(13) = 13² = 169\]

That's it, one multiplication with the same result as the full 3D transformation. This is great because finding square roots is much, much more computationally expensive than multiplication (in fact, it's so expensive that programmers on underpowered hardware find it more efficient to use a massive lookup table instead of actually running the square root calcuation). So thanks to this cool math trick, all KPCA really has to do is store the n×n similarity table, never storing or computing a single XD coordinate.

References