An agent that watches Google indexing for you, every day
Checking in Search Console why pages are not indexed, and clicking "request indexing" for a dozen sites, is daily tedious work. The agent does it itself, every day, for all sites.
Industry: Site owners, SEO agencies
In short
- 16 sites checked every day
- 13 not-indexed reasons reviewed automatically
- Requesting indexing, which Google offers no API for
- Blocklist and sitemap cleaned on their own
About the project
For a site to show in Google, it must be indexed. Search Console tells you which pages dropped and why, but checking that and requesting re-indexing, page by page, is daily clicking.
The challenge
Part of Search Console work has no API. The full list of excluded URLs and the request-indexing button are visible only in the browser, so an agent had to click like a human.
Goals
- Check indexing daily, without a human
- Request indexing for URLs worth re-crawling
- Clean the sitemap of dead URLs
- Leave a daily report of what happened
Technical challenges
Not everything has an API
The full list of excluded URLs and request-indexing cannot be done via API. The agent does it in the browser, like a human, within Google's daily limit.
Cleaning up after itself
Dead URLs, redirects and duplicates go to a blocklist that cleans the sitemap so junk is not submitted.
Solution and implementation
A daily round without you
Every day the agent goes through all sites, records the not-indexed reasons and requests indexing for what is worth it.
The sitemap cleans itself
URLs that will not be indexed anyway go to the blocklist, and the sitemap stays free of junk.
Results
Pages in Google faster
URLs worth crawling are submitted daily, not whenever someone finally checks Search Console.
No more daily clicking
The work that ate time on every site is done by the agent, and you get a report.
Takeaways
Not everything has to be clicked by hand
If you repeat something daily in a browser, you can usually put an agent on it.
Scope
- GSC round
- Requesting indexing
- Sitemap cleanup
- Daily report
Technologies
- Browser agent
- Search Console
- Node.js
Frequently asked questions
What does this agent do?
Every day it checks in Search Console which pages did not get indexed and why, and requests indexing for those worth re-crawling.
Why an agent, not an API?
Google offers no API for the full list of excluded URLs or the request-indexing button. It has to be done in the browser.
How many sites can it handle?
In this deployment it goes through sixteen sites daily, within a request limit Google enforces.
What about dead URLs?
URLs like 404s, redirects and duplicates go to a blocklist that cleans the sitemap.
Who is it for?
For site owners and agencies who watch indexing across many sites and do not want to do it by hand.