Automate SEO Keyword Clustering by Search Intent Using Python

Automate SEO Keyword Clustering by Search Intent with Python

Understanding search intent is vital for modern SEO success. As search engines evolve, especially with the rise of AI-driven search, aligning your keywords with user intent matters more than ever. Clustering keywords by search intent helps you organize content, improve website structure, and boost your visibility. In this guide, you’ll learn how to automate keyword clustering using Python, making your SEO strategy smarter and more efficient.

If you want to take your search engine optimization to the next level, Cyberset offers expert solutions, including content marketing, local internet marketing, and pay per click marketing to help you grow.

Why Search Intent Matters for SEO

Search intent reveals what users want when they type a query. Google and other search engines now focus on delivering results that match this intent, not just the keywords. If your content aligns with user intent, you’ll rank higher and attract more qualified visitors.

With AI search, the focus is on delivering the best answer quickly and efficiently. This means your SEO strategy should group keywords by intent, not just by topic or volume. Clustering keywords by search intent helps you create content that truly meets user needs.

How SERPs Reveal User Intent

The search engine results pages (SERPs) hold valuable clues about user intent. By analyzing which URLs rank for different keywords, you can see which queries share the same intent. If two keywords show similar results on page one, they likely have the same intent.

Traditionally, SEO professionals compared SERPs manually. Now, automation lets you scale this process and get more accurate clusters in less time. This approach leverages Google’s own AI ranking to do the heavy lifting for you.

Step-by-Step: Automate Keyword Clustering with Python

Ready to automate keyword clustering by search intent? Here’s a simple Python workflow you can use. You’ll need a CSV file with SERP data for your target keywords.

1. Import Your SERP Data

Start by importing your SERP results into a Python notebook using Pandas:

import pandas as pd
serps_input = pd.read_csv('data/serps_input.csv')
del serps_input['Unnamed: 0']
serps_input

Now your data is ready for filtering.

2. Filter for Page 1 Results

Focus on the top results for each keyword. These are most relevant for intent clustering.

serps_grpby_keyword = serps_input.groupby("keyword")
k_urls = 15

def filter_k_urls(group_df):
    filtered_df = group_df.loc[group_df['url'].notnull()]
    filtered_df = filtered_df.loc[filtered_df['rank'] <= k_urls]
    return filtered_df

filtered_serps = serps_grpby_keyword.apply(filter_k_urls)
filtered_serps_df = pd.concat([filtered_serps], axis=0)
del filtered_serps_df['keyword']
filtered_serps_df = filtered_serps_df.reset_index()
del filtered_serps_df['level_1']
filtered_serps_df

3. Compress URLs into a String

Combine the URLs for each keyword into a single string. This makes comparison easier.

filtserps_grpby_keyword = filtered_serps_df.groupby("keyword")

def string_serps(df):
    df['serp_string'] = ''.join(df['url'])
    return df

strung_serps = filtserps_grpby_keyword.apply(string_serps)
strung_serps = pd.concat([strung_serps], axis=0)
strung_serps = strung_serps[['keyword', 'serp_string']].drop_duplicates()
strung_serps

4. Compare SERP Similarity

Now, compare each keyword’s SERP string with every other keyword. This helps you measure how similar their search results are.

def serps_align(k, df):
    prime_df = df.loc[df.keyword == k].rename(columns={"serp_string": "serp_string_a", 'keyword': 'keyword_a'})
    comp_df = df.loc[df.keyword != k].reset_index(drop=True)
    prime_df = prime_df.loc[prime_df.index.repeat(len(comp_df.index))].reset_index(drop=True)
    prime_df = pd.concat([prime_df, comp_df], axis=1)
    prime_df = prime_df.rename(columns={"serp_string": "serp_string_b", 'keyword': 'keyword_b', "serp_string_a": "serp_string", 'keyword_a': 'keyword'})
    return prime_df

columns = ['keyword', 'serp_string', 'keyword_b', 'serp_string_b']
matched_serps = pd.DataFrame(columns=columns)
queries = strung_serps.keyword.to_list()
for q in queries:
    temp_df = serps_align(q, strung_serps)
    matched_serps = matched_serps.append(temp_df)
matched_serps

You now have all keyword pairs ready for similarity scoring.

5. Score SERP Similarity

Use a custom function to score how similar two SERPs are. This function measures both overlap and order of URLs.

import py_stringmatching as sm
ws_tok = sm.WhitespaceTokenizer()

def serps_similarity(serps_str1, serps_str2, k=15):
    denom = k+1
    norm = sum([2*(1/i - 1.0/(denom)) for i in range(1, denom)])
    serps_1 = ws_tok.tokenize(serps_str1)[:k]
    serps_2 = ws_tok.tokenize(serps_str2)[:k]
    match = lambda a, b: [b.index(x)+1 if x in b else None for x in a]
    pos_intersections = [(i+1,j) for i,j in enumerate(match(serps_1, serps_2)) if j is not None]
    pos_in1_not_in2 = [i+1 for i,j in enumerate(match(serps_1, serps_2)) if j is None]
    pos_in2_not_in1 = [i+1 for i,j in enumerate(match(serps_2, serps_1)) if j is None]
    a_sum = sum([abs(1/i -1/j) for i,j in pos_intersections])
    b_sum = sum([abs(1/i -1/denom) for i in pos_in1_not_in2])
    c_sum = sum([abs(1/i -1/denom) for i in pos_in2_not_in1])
    intent_prime = a_sum + b_sum + c_sum
    intent_dist = 1 - (intent_prime/norm)
    return intent_dist

matched_serps['si_simi'] = matched_serps.apply(lambda x: serps_similarity(x.serp_string, x.serp_string_b), axis=1)
matched_serps[['keyword', 'keyword_b', 'si_simi']]

Now you have a similarity score for every keyword pair.

6. Cluster Keywords by Intent

Set a similarity threshold (like 0.4) to group keywords with similar SERPs. These clusters represent shared search intent.

simi_lim = 0.4
keysv_df = serps_input[['keyword', 'search_volume']].drop_duplicates()
keywords_crossed_vols = matched_serps.merge(keysv_df, on='keyword', how='left')
keywords_crossed_vols = keywords_crossed_vols.rename(columns={'keyword': 'topic', 'keyword_b': 'keyword', 'search_volume': 'topic_volume'})
keywords_filtered_nonnan = keywords_crossed_vols.dropna()

Next, use a dictionary to group keywords by their similarity scores.

topic_groups = {}
# (Add your logic to populate topic_groups based on similarity threshold)

Convert your clusters into a DataFrame for easy analysis:

topic_groups_lst = []
for k, l in topic_groups.items():
    for v in l:
        topic_groups_lst.append([k, v])
topic_groups_dictdf = pd.DataFrame(topic_groups_lst, columns=['topic_group_no', 'keyword'])
topic_groups_dictdf

How to Use Keyword Clusters for SEO

Once you have your keyword clusters, you can:

Organize your site structure by search intent, not just product or service.
Create targeted content for each cluster to boost relevance and rankings.
Improve your content marketing strategy with focused topics.
Streamline your pay per click marketing by grouping keywords for higher Quality Scores.
Eliminate redundant pages and merge similar content for better user experience.

You can also use these clusters to enhance your website design or professional custom website development projects, ensuring your site matches user expectations from the start.

Scale Your SEO with Automation

Automating keyword clustering by search intent saves you hours of manual work. It also ensures your SEO is data-driven and scalable. Whether you’re managing a small business or a large ecommerce site, this approach helps you stay ahead of the competition.

For even more impact, combine automated clustering with Cyberset’s social media marketing and email marketing services to reach your audience across all channels.

Get Started with Automated Keyword Clustering

Automating keyword clustering by search intent is a game changer for SEO. With Python, you can quickly group thousands of keywords, create better content, and improve your site structure. This process helps you deliver what users want and rank higher in search results.

Ready to make your SEO more effective? Try this workflow or contact Cyberset for expert help with search engine optimization, website design, and more.

With the right tools and strategy, you can turn keyword data into actionable insights and real business growth.