Aaaaaaaggghhh! Google!! in Site Updates
- Oct. 17, 2013, 6:27 p.m.
- |
- Public
I've noticed some recent concern about Google indexing public site content.
(I, for one, welcome our not-so-new search overlords)
To address this concern, I've created a robots.txt file. My understanding is that Google and reputable search engines will honor this, and not index URLs that begin with:
- /box/
- /book/
- /entry/
- /search/
Basically, any URLs I could think of that may provide user-specific data in search results. So hopefully that solves it and puts people at ease. Obviously the effects won't be instantaneous; Google will have to crawl the site again and cycle out their current content. So we shall see.
Loading comments...