Update: So the IT director here at the Chronicle just stopped by my desk and told me something interesting. It seems that we learned just yesterday that, immediately after our website upgrade, our robots.txt file was accidentally replaced with one that disallowed all search spiders. Likely this was put in place while the new site was in the development phase to keep the search engines from crawling it before it was ready and somebody just forgot to fix it.
At any rate, sometime between 5 p.m. and now, someone fixed the robots.txt file to allow crawlers. This could be the real solution for our inclusion in Google News; however, the timing is extraordinary. If this robots.txt file is indeed the cause of all this, then perhaps we will find ourselves back on the Yahoo News listing soon.
Original Post: At the beginning of March, the Chronicle redesigned its website. Consequently, our RSS feeds and URL structure changed. Within a few days, it became clear that this meant that our stories were no longer appearing in Yahoo Local News and Google News. Eventually, both news sites forgot about us.
Admittedly, we noticed our absence from Yahoo first. Yahoo News was one of our top traffic sources online, and it’s hard not to notice when one of those goes missing.
(Google is, of course, our top, non-internal source of traffic, but traffic from Google News appears to be mixed in with all the other traffic in our analytics, so any change was not as noticeable.)
I sent message after message through Yahoo News’ system for “submitting a news source” to no avail. I made a few hopeful postings on Twitter, asking people if they knew how to get re-listed after a major site change. I managed to get in touch with one guy who said he worked at Yahoo Local News, but I never heard back from him.
I thought all was lost, he wrote, over-dramatically.
Then this morning, ever optimistic, I posted the above tweet asking people again for some help getting re-listed. Then something magical happened.
On a whim, I decided to do a site search for bozemandailychronicle.com in Google News, just to make sure that none of our stories were in the system. What I found is illustrated in the screenshot displayed here.
Lots of our stories. All with timestamps only a few minutes old.
I presume that the tweep behind @google saw my tweet and did something about it, but I can’t be sure. I did, after all, submit a request to Google News a couple weeks ago to have our site re-listed. Perhaps it’s just coincidence.
I’m not convinced.
Now if only Yahoo would re-list us…