Proof that the math renders

Before writing anything worth reading, it seems prudent to check that the site can display it. If inline math typeset by KaTeX, a centred display equation, a syntax-highlighted Python snippet, and a captioned figure all render cleanly below, the plumbing is in order and future posts can concern themselves with their contents rather than their appearance.

Inline math inside prose

#

Scaled dot-product attention computes a soft lookup from queries \(Q\), keys \(K\), and values \(V\). Written inline, it is \(\text{Attention}(Q, K, V) = \text{softmax}\!\left(\tfrac{QK^{\top}}{\sqrt{d_k}}\right) V\), where the \(\sqrt{d_k}\) divisor keeps the pre-softmax logits from growing too large as the key dimension \(d_k\) increases.

A display equation

#

Language models are trained to minimize the token-level cross-entropy loss. For a sequence of tokens \(x_1, \dots, x_T\), this is

Each term \(\log p_\theta(x_t \mid x_{\lt t})\) is the log-probability the model assigns to the true next token given everything that came before.

A code block

#

Here is a compact PyTorch-flavoured implementation of the attention scores from the equation above. Nothing fancy, just enough to see syntax highlighting work.

import math
import torch
import torch.nn.functional as F

def attention(Q, K, V, mask=None):
    """Scaled dot-product attention."""
    d_k = Q.size(-1)
    scores = Q @ K.transpose(-2, -1) / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float("-inf"))
    weights = F.softmax(scores, dim=-1)
    return weights @ V, weights

A figure with caption

#

Below is a 4-token causal attention pattern after softmax. The lower-triangular structure reflects the mask: token \(t\) can only attend to tokens \(\le t\).

That’s it. If everything above reads cleanly in both light and dark mode, the site is calibrated correctly.