Practical guide to keyword clustering with LLMs and embeddings

Keyword clustering used to mean a spreadsheet, a rainy afternoon, and a lot of judgment calls. In 2026 you can hand a list of ten thousand terms to an embedding model and get topic groups back in minutes. This is the run-it-today version of that workflow: which models to use, which algorithm fits your keyword volume, the prompts that turn raw clusters into editor-ready topics, and the one validation step almost every tutorial skips.

Here is the thesis up front. Embeddings are brilliant at grouping keywords by meaning, but meaning is not the same thing as a shared search results page, and the search results page is what actually decides whether two keywords can live on one URL. Cluster with embeddings for speed, then validate against the SERP before you commit a keyword to a page. Get that order right and you ship a topical map that ranks; get it wrong and you build a pile of pages that cannibalize each other.

Why embeddings and LLMs beat manual keyword clustering in 2026

An embedding is just a list of numbers that captures what a phrase means. Phrases that mean similar things end up close together in that number space, so "cheap running shoes" and "budget trainers" land next to each other even though they share no words. That is the whole trick, and it is why embedding-based grouping crushes the old approach.

The manual method matched keywords on shared words or matched them by hand. It missed synonyms, it drowned on lists over a few hundred terms, and it turned one analyst into a bottleneck. Embedding-based grouping reads meaning instead of characters, so it catches paraphrases and scales to lists you could never sort by hand. Open-source models like Sentence-Transformers run on your own machine, and the hosted options keep getting better on public leaderboards like the MTEB benchmark.

Swap in a modern model and the quality jumps: OpenAI's text-embedding-3-small scores about 62.3% on MTEB and costs almost nothing to run. If you want the no-code, five-minute version of clustering for planning, we already published a keyword clustering for content planning walkthrough built on a single ChatGPT prompt. This guide is the engineering companion to it: the same idea, but reproducible at scale and tuned for teams that cluster tens of thousands of terms.

The end-to-end workflow, from raw keywords to publishable clusters

Every reliable pipeline runs the same seven stages. Skip a stage and you pay for it later in messy clusters or a blown API bill.

1. Collect and clean. Pull your keyword universe, strip duplicates, drop the obvious junk, and lowercase everything. Garbage in, garbage clusters.
1. Embed. Turn each keyword into a vector with one model. Never mix models in the same run; their number spaces are not comparable.
1. Reduce dimensions (optional). For large sets, UMAP compresses vectors before clustering so density-based methods behave.
1. Cluster. Group the vectors with a scikit-learn algorithm chosen for your volume (next section).
1. Label. Hand each cluster to an LLM to name it in the searcher's language and tag its intent.
1. Validate and QA. Spot-check clusters against live search results, then have a human read the edge cases.
1. Export. Write each cluster to a row: topic, member keywords, intent, priority. That row is a brief waiting to happen.

The defaults that work for most teams: one embedding model, cosine distance, and a clustering step that outputs somewhere between 8 and a few hundred groups depending on list size. Do not overthink stages 1 through 4; the money is made in stages 5 and 6, where the LLM and the SERP check turn numbers into a plan.

Choosing an algorithm and presets by keyword volume

Three algorithm families cover almost every job. Pick by keyword volume and by how much noise you are willing to tolerate.

• KMeans is fast and predictable, but you must tell it how many clusters to make and it forces every keyword into a group. Best for tidy, mid-size lists where you already know roughly how many topics you want.
• HDBSCAN finds clusters by density and leaves genuine outliers unassigned, so you get cleaner groups and an honest "noise" bucket. Best when your list is messy and you do not know the topic count in advance.
• Agglomerative clustering builds a tree you can cut at any height, which is perfect when you want pillar topics and sub-topics from the same run. Slower, but the hierarchy maps neatly onto a content structure.

Presets by size. Under 5,000 keywords: KMeans or agglomerative, no dimensionality reduction needed, tune the cluster count by hand until groups read cleanly. Between 5,000 and 50,000: UMAP down to a few dozen dimensions, then HDBSCAN with a min-cluster-size around the smallest page you would actually publish. Over 50,000: batch your embeddings, store them in an approximate nearest-neighbor index like FAISS, and cluster in chunks so memory does not blow up.

On cost, the embedding step is the cheap part. Hosted small-embedding models run around $0.02 per 1M tokens on current API pricing, so even a large list costs less than lunch. The expensive resource is your attention during labeling and validation, which is exactly where you should spend it.

Prompting an LLM to label and validate clusters

Clustering gives you numbered groups; it does not tell you what they are. That is the LLM's job, and it is good at it. Feed the model the keywords in one cluster and ask for three things in strict JSON: a short topic name in the searcher's language, the dominant search intent, and any keyword that does not belong. Keep the temperature low so the output is stable, give it two or three worked examples, and never paste more keywords than the context window can hold without truncating.

A labeling prompt that holds up: "You are an SEO editor. Here are keywords that were grouped by meaning. Return JSON with topic (5 words max), intent (informational, commercial, transactional, or navigational), and outliers (any keyword that belongs in a different topic)." That last field is the quiet workhorse, because it turns the LLM into a second pair of eyes on the clustering itself.

Here is the step almost every tutorial skips, and it is the whole reason clusters fail in the wild. Embedding similarity is not the same as SERP overlap. Two keywords can sit millimeters apart in vector space and still return completely different top-10 results, which means Google sees them as different jobs that need different pages. The only ground truth for "can these share a URL" is whether their live search results actually overlap. So run a second pass: for each cluster, check whether the members share enough ranking pages, and split the ones that do not. Our topic-cluster research process walks through that SERP-overlap test in detail, and Google's own helpful content guidance is built on the same one-page-per-intent logic.

Do not let the LLM do the clustering itself. Asking a model to sort a raw keyword list into groups is non-deterministic, does not scale past a few hundred terms, and hallucinates structure that is not there. Embeddings handle recall, the LLM handles precision and language, and the SERP handles the final ruling. If you are wiring LLMs into any part of this, our notes on AI SEO tools and human checks cover where these models earn their keep and where they quietly lie.

Evaluating cluster quality before you commit

You need one number and one human check. The number is a silhouette score, which measures how tight and well-separated your clusters are; sweep the cluster count and keep the setting that scores highest. The human check is the acceptance test that matters more than any metric: one cluster should map to exactly one publishable page with one dominant intent. If a cluster needs two pages to serve it, split it. If two clusters would fight over the same SERP, merge them.

Score intent explicitly rather than eyeballing it. Reading intent from the live results page is faster and more honest than guessing from the words, and it is the same signal Google rewards. Our search-intent vector framework turns that read into a repeatable score, so a cluster that mixes "how to" and "buy now" queries gets flagged before it becomes a page that ranks for neither. Sample a handful of clusters by hand every run; the failures cluster together, and you will spot the pattern fast.

Turning clusters into a content funnel

A validated cluster is not a to-do item yet; it is a candidate. Rank candidates by two things: how close the intent sits to a purchase, and how winnable the SERP looks. Commercial and transactional clusters become money pages and comparison pages. Informational clusters become the pillars and supporting posts that feed them internal links. The tree from agglomerative clustering hands you that hierarchy for free: the cut near the top is your pillar topics, the branches below are the supporting articles.

Wire the internal links along the cluster structure, not at random. Every supporting post links up to its pillar with descriptive anchor text, and the pillar links back down to its strongest children. That gives search engines an unambiguous map of which page owns which topic, and it is how a clustered plan compounds instead of sprawling. Turn each cluster row into a brief, staff the highest-value clusters first, and let the informational tier build the authority that lifts the money pages.

How VarynForge fits in

VarynForge runs this whole pipeline for you. It pulls your keyword universe, clusters it into topic groups, validates each cluster against live search results, and returns a prioritized content plan you can brief against the same day. If you would rather skip the notebook and the embedding bill, compare what the clustering and content-mapping tiers include on the pricing page.

Key takeaways

Keyword clustering in 2026 is a two-tool job: embeddings for recall, an LLM for labeling and precision. Pick your clustering algorithm by volume, keep the embedding step cheap, and spend your real effort on labeling and validation. The one habit that separates a plan that ranks from a pile of cannibalizing pages is the SERP-overlap check: never let semantic similarity alone decide that two keywords share a page. Cluster fast, validate against the live results, ship one page per intent, and wire the internal links along the cluster tree. Do that and every run turns a flat keyword export into a topical map your team can actually publish.

Further reading

• Keyword Clustering for Content Planning: From CSV to Calendar
• How to Do Keyword Research for Topic Clusters in 2026
• Types of Search Intent: A Vector Framework for Content Teams
• Sentence-Transformers documentation

Sources

• OpenAI - Embeddings guide and MTEB scores
• OpenAI - API pricing
• scikit-learn - Clustering algorithms
• HDBSCAN - Density-based clustering documentation
• UMAP - Dimensionality reduction documentation
• FAISS - Approximate nearest neighbor library
• Google Search Central - Creating helpful content