4 min read · 1,020 words
Tips / SEO / Python · Automation
Approx. 2,400 characters
When you publish a blog post, Google typically crawls it within anywhere from an hour to a few days. Until then, your post won't show up in search results. We shortened this indexing time from an average of 4 hours to 30 minutes by sending simultaneous pings to 5 channels immediately upon publication. Here is how we built it, the results we achieved, and how we validated it.
Why We Built It
After accumulating nearly 200 posts, we noticed a consistent pattern. For the first 24 hours after publishing, our posts were not indexed, meaning they practically did not exist during that time. A post must be indexed to appear in search results and start driving traffic.
Manually clicking the URL Inspection tool in Google Search Console every time was also a hassle, adding 30 seconds of overhead per post. Bing offers a free instant indexing protocol called IndexNow, and without utilizing it, Bing indexing takes even longer. We also didn't have the time to manually ping RSS aggregators (such as Google PubSubHubbub, FeedBurner, etc.) for every post.
So, we built a module that automatically sends simultaneous pings to 5 channels as soon as a post is published and aggregates the responses in one place.
How It Works
At the end of the publish_post hook chain, an indexing trigger is called once. This trigger simultaneously calls the following 5 channels:
- IndexNow (Bing + Yandex) — POSTs a single post URL to the IndexNow endpoint. Response time is typically under 200ms. Bing enters the crawl queue within 5 to 30 minutes.
- Google Indexing API — Calls
urlNotificationMetadata.publish. Although originally intended for Job Postings and Livestreams, we use it for new post notifications. Response time is under 1 second. Google crawls the page faster compared to other domains in the same timeframe. - Google Search Console URL Inspection — Polls the indexing status using
urlInspection.index.inspect. If a "URL is not on Google" response is received, it automatically triggers the GSC "Request Indexing" call. - RSS PubSubHubbub — Pings Google's PubSubHubbub hub with
hub.mode=publish&hub.url={feed_url}. Propagates to Google Reader / all RSS subscribers within 30 seconds. - Bing Webmaster
submitUrl— Calls Bing's legacysubmitUrlAPI alongside IndexNow as a double safety net.
All 5 calls are initiated concurrently (asyncio.gather), and each response is saved in indexer_log.json. If any channel fails, it automatically retries after a set cooldown period.
Actual Results
- Average indexing time: 4 hours → 30 minutes (90% reduction)
- Bing indexing rate (within 24 hours of publishing): 22% → 87%
- Number of times we clicked "Request Indexing" in the GSC UI: 41 times cumulatively → 0 times after implementation
- Google Indexing API daily quota (200): Never exceeded (distributed via automatic batching + cooldowns)
- Discord notification frequency: A single line showing the results of the 5 channels immediately after publishing. "indexing: google ok / bing ok / rss ok / inspect indexed / submit ok"
While the numbers might look modest, saving 30 seconds per post adds up to 100 minutes over 200 posts. More importantly, faster indexing means search traffic starts accumulating much earlier.
Validation Method
We directly compared manual indexing with automatic indexing.
A/B Indexing Comparison (sess 28–30) — We randomly split 28 posts published over 4 weeks into two groups. Group A went through the automatic indexer, while Group B did not (manual GSC only). We recorded the indexing time in minutes using GSC URL Inspection after publication.
A clear difference. Group A was 9 times faster on average. Additionally, the indexing rate within 24 hours was more than double.
Bing-Specific Verification — Compared 12 posts before IndexNow implementation with 12 posts after. The Bing indexing rate within 24 hours jumped from 22% to 87%. Although Bing alone accounts for a small portion of Korean search traffic, there is no reason not to use IndexNow since it is free.
Retry Behavior on Failure — We intentionally disconnected the internet and published a post to force all 5 channels to fail, then verified whether they automatically retried within the 1-hour cooldown after the connection was restored. All 5 out of 5 successfully retried and returned "ok" responses.
How to Build It
The core concept is sending a single asynchronous ping immediately after publishing. The code is short.
import asyncio
import httpx
INDEXNOW_KEY = "your-32char-key"
INDEXNOW_HOST = "yourdomain.com"
async def ping_indexnow(url: str) -> dict:
async with httpx.AsyncClient(timeout=10) as c:
r = await c.post(
"https://api.indexnow.org/IndexNow",
json={"host": INDEXNOW_HOST,
"key": INDEXNOW_KEY,
"urlList": [url]},
)
return {"status": r.status_code, "channel": "indexnow"}
async def ping_rss_hub(feed_url: str) -> dict:
async with httpx.AsyncClient(timeout=10) as c:
r = await c.post(
"https://pubsubhubbub.appspot.com/",
data={"hub.mode": "publish", "hub.url": feed_url},
)
return {"status": r.status_code, "channel": "rss_hub"}
async def ping_all(post_url: str, feed_url: str) -> list[dict]:
return await asyncio.gather(
ping_indexnow(post_url),
ping_rss_hub(feed_url),
return_exceptions=True,
)
# Call immediately after publishing
results = asyncio.run(ping_all(
"https://yourdomain.com/post-url",
"https://yourdomain.com/feeds/posts/default",
))
print(results)
For the IndexNow key, you just need to host a single 32-character hex string as a text file at https://yourdomain.com/{key}.txt. Bing will verify this file upon the first call. Once verified, all subsequent pings are processed instantly.
The Google Indexing API requires service account authentication, which makes the code a bit longer, but you can still get great results using just the two channels shown above. In our experience, the combination of IndexNow and RSS hub offers the most cost-effective solution.
Summary: Simply adding an asynchronous ping to IndexNow and RSS hub immediately after publishing can cut your average indexing time in half. This is an automation you can apply regardless of post quality or keywords.
Category Coverage Notice
This article follows our label-specific editorial criteria. Details: