Years ago, when the search engine category was a lot more competitive, I did a lot of comparing between contenders. For awhile HotBot was ahead, then AltaVista, then AllTheWeb/FAST… Not necessarily in that order, but you get the point.
Then Google won. Huge. They were just bigger and better than everybody at finding nearly everything.
But lately Google has frustrated me. When I do lookups for subjects, it gives results that include misspellings and other approximations, even when I use the “advanced” settings. Worse, it’s been useless at something it used to do perfectly: find old blog entries of mine.
For example, when I was writing this post about cameras and lenses, I wanted to see what I’d written on my blog back when I first picked up my Canon 30D. I kinda remember that it was in early ’06, but beyond that I wasn’t sure how to go hunting, since Google seems to have lost track of that blog, even though its archives are online with inbound links. See, my old blog was moved off of weblogs.com and mothballed (with welcome help from Dave Winer) at the domain doc-weblogs.com. Some links got 404’d, but Google re-made enough connections so that people could at least search for specific words and strings and find stuff. But somewhere along the way, that ended. When I looked for doc searls tamron zoom canon lenses on Google, I found page after page of nothing.
So I went where I haven’t gone for a long time: Yahoo, and did the same search. Voila! The blog post I wanted to find was right up top. That rocked.
I just repeated the experiment with the same two searches, and found that Google has caught up, and now lists what I was looking for in the first page of the 17 results it brings up, with yesterday’s post pointing to it as the top result. Obviously Google followed the link and indexed the old page.
So I decided to do a search for another day in that same time-frame, for another bunch of words that only occur on that day’s blog postings. The day I chose was Thursday, May 18, 2006. The words I searched were doc searls rob cottingham profitable tragedy. Google found nothing, but gave me three pages of results, including misspellings of Searls. Yahoo delivered four results, including exactly the page I was looking for. In fact, every test I do, for every day of my old blog, brings up a Yahoo result. (One sample.) To Google, my old blog pretty much doesn’t exist unless there’s a new inbound link to one of its pages.
Obviously, Yahoo is doing a better job of following links here.
And that means it’s doing a better job — to me at least — of competing with Google than most folks give it credit for.
My hat’s off to them.
-
Oh my goodness. I didn’t believe you – but then something twigged. On the weekend I was looking for some wallpapers I’d done and posted on my blog a while back. I couldn’t remember much about the post, but I googled “Michael Lucas-Smith smalltalk backgrounds”.
I failed to find it with google – so I use the on-site search that the blog website provides and eventually found it.
After reading your post – I went to yahoo and did the same search I did on the weekend. The very first hit was the correct page.
I’m shocked, to say the least – I kind of assumed Google would just.. work.
-
Ah New England! I was on a New Hampshire cruise yesterday and decided to come back down from my wooded seclusion. Funny how you don’t think about the trip being over until you walk into your place and turn on the lights and you think…”I guess I’m home.” Speaking of Boston when will Doc come back to engage his local fans?
-
And people just don’t get it when I say that Clusty returns as good or better results.
http://clusty.com/search?query=doc+searls+tamron+zoom+canon+lenses
http://clusty.com/search?query=doc+searls+rob+cottingham+profitable+tragedy
-
No. It’s simply not a better search engine than Google. I think their stock price is pretty evident of their success. But if you’re doing a non-scientific study such as this, I would say your problem is specifically SEO, for which variables you’ve decided to leave out are obviously of some importance.
-
I have to agree with you. I picked a random blog posting of mine from a few years back and typed in a google-compatible search. Google didn’t find it at all; it was item 3 on clusty when I used exactly the same search terms, but #1 on Yahoo.
-
actually yahoo is much better than google. google results are not to the mark, one can use it to find hacks etc. but for real searches use yahoo.
-
Yeah. Hey look, Yahoo is better than Google at finding my own sites from 2 years ago, and it takes Google all of 3 seconds to catch up. The question in the title is not at all loaded.
-
I noticed the same: Yahoo search is getting better, especially with some of their new initatives. See also http://poorbuthappy.com/ease/archives/2008/08/01/4245/yahoo-search-getting-better
-
Patrick, a search engine should find the information somebody looks for. It should especially find anything that’s reasonably durable and that exists on the Web. My old blog has been sitting still in plain sight of search engines at its current location for more than a year. The fact that there are plenty of inbound links, and links within the blog to every other day in the blog — meaning that all can be found by a link-following crawler — should be SEO enough.
If I have to do SEO tricks with the content I put online — tricks meant to “increase traffic” and otherwise game a search engine’s ranking algorithms — just to get that content indexed at all by a search engine — then I would say that search engine is corrupted by the external systems built to game it. Or worse, by its own advertising business model.
If that’s actually what’s happening at Google (and I won’t believe it is until somebody proves it), I would say that Google has failed its mission.
The fact that Google fails to find pages that have been sitting in plain sight of its crawlers for long periods of time, while Yahoo finds those same pages, is rather interesting data.
-
Ian, if Google only catches up when one makes a new post pointing to old posts that Yahoo has seen all along, which is the better search engine?
-
Doc
Busy so short post
Serendipity … the site I clicked after yours (adjoining tab) is Jeffhttp://jeffjonas.typepad.com/jeff_jonas/2008/08/foo-camp-2008-how-to-beat-google-at-search.html
Somewhat relevant?
Ciao
Chip -
Doc,
The better search engine is the one that delivers the most relevant results to a wide and varied range of searches. You cannot use one example to make a judgment, and I’m afraid you can’t even say that it works better for you – you just happened across an instance where the Yahoo index had a page Google’s didn’t.
-
I know some of the commentors above are bashing you for the non-scientific nature of your test of google v. yahoo but that’s not the point of the article IMHO. Google, for whatever reason, is presumed by the populace to be far and away the best search engine. Nobody even questions it anymore, google gets a free pass. But we searchies have all had problems with google in recent years. It’s time to take a fresh look at yahoo as an alternative. Thanks for the article.
-
@Doc — This may be a subject specific anomaly, then again it may not be. Here’s why I feel it may be the later. Recently I have embarked upon running queries of hundreds of searches on both Yahoo! and Google (A, B). At first impression Yahoo! has many more relevant results per search phrase and Google has many more results. What you have found in your query isn’t all that surprising to me, in fact it’s what i am seeing for quite a larger sample size. In terms of “Better” that is a little too subjective to me if we could qualify “better” assign a number value….and I digress.
@Ian — Your right the the margin of error in the data but this is what I’m working on (work in search marketing) and again with a larger sample it seems to be very close in line with what Doc experienced.
Anyone think this may be part of the reason why Google launched Knol? Is Google starving for “Pro-Content” that it knows will lead to greater time on page and increase in click through rate? In addition to the fact tat they want more content to advertise along?
-
I beg to disagree doc.. try searching my blog.. yahoo doesnt even throw it up.. google does… search for whenpathscollide
-
Google was returning great results in the days where SEO and domain squawks haven’t taken over the web. Nowadays, all google returns are a bunch of worthless adsense cluttered drone sites that has SEO targeted contents. It getting to a point where google is becoming useless and worst every single day.
-
I’ve said it many times, Yahoo is the strongest competitor Google has. Google is still better for most searches but there are areas where a Yahoo search will give you better quality results than Google at least in the first page. If you can’t find something with Google, Yahoo is the place to go search for it.
-
I tend to agree with Ian. Some of it is what you’re searching for, and some of it is how you search.
Personally, I’ve never had the problem you encountered when Google searching my own blog. Here’s why…
I take advantage of Google’s advanced searching abilities. For your original search, I would have searched for:
tamron zoom canon lenses site:doc-weblogs.com
Notice the “site:”? That tells Google to only search that site.
Here’s a couple more examples to try on your site:
Adobe PostScript site:doc-weblogs.comPeanut Gallery site:doc-weblogs.com
Learn more about Google’s special features on this page (and be sure to click the Basic Search and Advanced Search links in the left navigation too):
http://www.google.com/intl/en/help/features.htmlIn fairness, Yahoo! has its own special features (many of them the same as Google offers) for those times you’re using Yahoo! and need to narrow your results.
http://tools.search.yahoo.com/newsearch/resourcesHope that helps some comment readers!
-
Angela,
good mention, I agree with the use of ‘site:’ because really with as specific a task as he was wanting, why not search smart, and specify both what you want to search for and where. The internet is too big a place to think 6 equally weighted keywords will point you to what you thought it would.
-
Great thread, everybody.
Joe, thanks for getting the basically simple point of the post.
Angela, thanks for the tips. I had forgotten about the site feature in Advanced. That said, there’s no reason Google shouldn’t include those results in a general keyword search. Why should any site be deep-sixed so its contents can only be found with “advanced” searches of just those sites? This is a bug, not a feature.
The good thing is, Yahoo, Clusty and perhaps others are truly competitive with Google. This is good.
-
It seems to me that the problem was with the age of your post. I seem to remember that Google require new incoming links to keep your site high in ranking. That your old posts were un-indexed seems to imply they will remove it from there index if no one cares about it enough to post.
I would say Google is serving one definite purpose: it will give you a time-relevant result. If you’re looking for something that has been talked about recently, they’ll take you straight to a good answer about it. Yahoo, on the other hand, is probably better for research. It would seem they keep there index, no matter how old it is. Meaning old attempts to game SEO result will still be there, and may clog up the results for any kind of keyword searches, but if you know what you’re looking for, and willing to spend the time investigating a number of false positives, it will probably have more useful results.
-
There is one fact that people forget about Google.
When they first started, all search engines made an OR search.
doc OR searls
That returned many results and many irrelevant ones.
Remember you and to specifically ask for +doc +searls ?
Then Google decided that what people want is relevant results so they switched to AND results.
doc AND searls
All other search engines saw the light and also switched from OR to AND.
Lately, I see that Google sometimes decides on OR which return many results, but less relevant ones.
And this is exactly what you, Doc, experienced.
-
That’s true, Hanan, and I hate it. Yet Google is doing some other stuff that’s cute, I guess, for “consumers” and other ordinary folk; but infuriating for somebody wanting to do serious search without having to go to “advanced” every time. The worst for me is including misspellings in results. If somebody goes to the trouble of looking for Searls and not Searles, dammit don’t include Searles. (I only use that example because of my confirmed experience with it. And because I don’t have the time to look for misspellings in results for Swartz, Ts’o, Tridin, Chapell or Moser. But I’ll bet they’re in there.)
Yahoo gets too cute too, and transparently cross-promotional. http://images.yahoo.com might as well be a search front end for Flickr, because that’s basically what it is. I love Flickr, but I don’t want nothing but Flickr shots in image search results.
-
Skolor, there is no excuse for removing something from an index just because nobody has linked to it recently, unless Google explicitly wishes to be a search engine for current material. I don’t think that’s the case.
But clearly there is a value system here that says that old Web material doesn’t matter.
It would be fine if the desired results were pushed down in the rankings somewhere. But what I wanted was absent from the results, not just demoted by a ranking algorithm. It was purposefully forgotten.
Meanwhile, as Angela showed above, Google does have the searched-for post in its index; but it’s accessible only with a site-specific search. That’s just wrong.
Meanwhile, Yahoo’s respect for old material (to which there in fact are links, because crawlers do find it) is a big advantage — to me, at least; and I suspect to many others who want to do serious, enduring work on the Web, and not just game the web for advertising and SEO.
-
A quick search on Google Blog Search for “doc searls” shows five different blogs (the related blogs section). http://www.google.com/blogsearch?hl=en&q=doc+searls&btnG=Search+Blogs
The multiple blogs/locations/duplicate content might be affecting your results somehow in Google.
You still have a strong pagerank on your original blog. Did you setup a 301 permanent redirect to this blog? I’d recommend setting this up for the main blog page – as well as for individual posts (if you moved those too). When I setup our CMS for clients and the urls to individual pages change, I make sure to set permanent redirects for each one. On a recent site we launched just a few weeks ago, Google has already removed most of the old pages from their index completely and replaced them with the updated pages. And from a usability perspective – if someone tries to access the old link, they get redirected to the new page rather than getting a 404 page not found (or in your case browsing the old blog).
The misspellings and approximations seem to be a fairly recent feature. I had mixed feelings at first, but I don’t mind Google doing that when the search would have otherwise returned zero results.
-
Yahoo search is complete crap. I search for “russ” and don’t find myself in the first five pages.
-
I had noticed the same problem with Google NOT locating some old blog posts of mine.
I used to use it for the same reason as Doc, to look at what I wrote aways back, and maybe link to it. But for the past few months, no dice.
I hadn’t thought about trying yahoo, now I will.
As Doc says, “The fact that Google fails to find pages that have been sitting in plain sight of its crawlers for long periods of time, while Yahoo finds those same pages, is rather interesting data.”
I wonder what other bits of the world’s knowledge have dropped off the back of the Google truck.
-
Compass, the five blogs are in fact two:
http://doc-weblogs.com (my old blog, now inactive)
http://blogs.law.harvard.edu/doc/ (my new blog)Two of the other three are the same URLs as the two above, with different title text, used briefly while the blogs were being set up. Don’t know why Google Blogsearch remembers those.
The third is http://doc.searls.com, which used to be where I pointed to some podcasts I’d done, but which now redirects here.
The redirects for both doc.weblogs.com and doc.searls.com are 301s, or “friendly.”
As for misspellings in search results, I agree that this is a recent “feature”; but I don’t find it acceptable. I have done many searches where the only results were many pages of misspellings. It would be better if Google came back with “Did you mean: _______”, as it does with single word searches.
In the early days Google succeeded not by having the largest index (it didn’t), but by returning accurate and relevant results for both word strings and keywords. Or, as Google sahere, “understands exactly what you mean and gives you back exactly what you want”. For reasons unknown, Google has made two decisions that compromise those original virtues: 1) not returning keyword results for some kinds of old material in its index; and 2) including in results misspellings of keywords.
That’s what I’ve learned through comments on this thread.
-
I’ve been on Blogger since ’04 and Google Blog Search has yet to turn up a single post of mine.
And backlinks! Last time I looked Yahoo was in the thousands and Google a few hundred. What’s up with that. Google is *lazy* …
-
I’m shocked at this ‘either/or’ conversation. In the late 90s the answer was always ‘all of the above’ via consolidator Copernic. The ‘integrated’ (complexity) answer is always ‘both’.
And in the end, it depends on what you’re looking for, thus the beauty of the architectural design approch taken by Viewzi to focus the search on what is being sought in a ‘customized’ format.
-
I’ve found Yahoo good for ‘topic’ searches where they put some value add into the classification of sites. The judicious use of their ‘my web’ facility also helps with classification by people. There are some really good ideas here. So why am I drifting to google? I need my other parts of he site presence to synchronise. Yahoo task list is better (well, all it really has to do is exist to be better), but do tasks, to do, my address list, my calendar synch with iPhone, iCal? Yahoo desktop Search used to access Network and USB drives. That stopped, and now I use Google desktop search. Nowhere near as good – except for that vital factor – it works.
Is it unfair to judge a search engine by the peripheral ‘stuff’ – sure, but that is how I’m doing it as a consumer, as a person who decides where to allocate my presence. Every new item needs Firefox support on day 1, Mac and Windows support on Day 1. Sorry, but that’s how it is.
I use Stumbleupon to record interesting pages now rather than ‘My web’ because it’s faster, because Firefox support for Yahoo Toolbar was late one version. All these little things eat away at market share. Plenty of people are waiting to invite me. All of these things affect ‘presence’ and therefore the ability to monetise visitors. If we measure by market share or by capitalisation, we are measuring these other aspects of an entry point, not just the search. I’ve always liked Yahoo search, I just use Google more now.
-
I’ve argued for a long time that the first page results from Yahoo are at the very least as good as Google, if not a lot better in some cases. The thing is though that after the first page Yahoo seems to go a bit funny, but then again even with all the web pages out there how many REALLY match what you are looking for?
My rule is this, if you’re looking to buy something go Google (they really favour people with products), if it’s for information use Yahoo.
-
Doc, if you want to search for an exact term without pulling in synonyms, possible spell corrections, etc. then one way to do it is to put a + in front of the word. If you do the search [football coaches] you might see a word like “coaching” highlighted in bold in our snippets, but if you search for [football +coaches] then you will only see coaches highlighted. If you prefer you could also put a word or phrase in double-quotes to search for that exact word. So the search [football “coaches”] would also search for the exact word coaches.
I’ll point some people to your other feedback, but I wanted to mention that tip.
-
Thanks, Matt. Good advice (that I should have known, I guess). And I hope the feedback helps. This has been a useful thread.
-
I have been very frustrated for over a year that much of my web page never comes up in a Google keyword or exact phrase search. Some phrases would be found by Google for a long time, then for months Google wouldn’t find them anymore. I always figured that I didn’t create the web page properly. A few weeks ago I searched some terms that are unique to my site, that Google can’t find, on Yahoo. Yahoo found them all. In fact Yahoo found every phrase from my site that I tried. I couldn’t believe it. Then I got curious and tried HotBot. That was my favorite search way back when. I was shocked that HotBot also found everything I threw at it. I didn’t even know it was still in business. Then I remembered old AltaVista, and tried that. Holy crap, even AltaVista found everything. What the hell is going on with Google?
-
Okay, I’m editing a Wikipedia page (a first for me), and looking for something I once wrote about James Normoyle.
So I looked up searls normoyle on Google. First time, what I wanted (an old blog post) came up at the bottom of the first page of results. Second time, I added + in front of both Searls and Normoyle, and the post I wanted came up at the top of the second page of results. Then I tried a few more times, a few minutes apart (without the +s), and by then the result had moved up to #7 on the first page of results.
When I looked for searls normoyle on Yahoo, what I wanted was the lucky top result, first time.
Matt, you might want to bring this up with the algorithm folks.
-
I commented, above, on Google not indexing all of my small web page. I can’t count the number of times over the last few years that I’ve filled in their web form request to crawl my site. Here are a few examples of searches that will demonstrate the problem. These phrases have been on my web page for a long time. Search for the exact phrase “Alaska photo research” in Google, Yahoo, and AltaVista. Google is the only one that can’t find it. Now try “very rare sled dog title”. Again, Google is the only search engine that can’t find it. Now try “Josephine Smith Wilson”. Again, Google doesn’t find my site. Yahoo and AltaVista do find all these phrases in my page.
Another problem with Google is when searchers are looking for something to purchase. The searcher types in the title of a rare book they wish to buy. They get a lot of hits of libraries that have the book, but my site, which is the only place in the world that has that title for sale, isn’t included in the search results, because it doesn’t have as much traffic as a library web site.
-
Pingback from Engwar » Blog Archive » Blue-pencil syndrome on August 24, 2008 at 6:15 am
-
“Matt, you might want to bring this up with the algorithm folks.”
Hi Doc, I did that this afternoon. Our most senior ranking person and I sat down and read through this entire column and the comments, and I believe that we identified the issue (it should be fixed now).
If you’re curious, the short answer is that doc-weblogs.com has thousands of spammy pages under doc-weblogs.com . Visit http://bit.ly/Rfdzp as a safe url to see 1000+ of those spammy pages in Google. But if you try to view one of those spammy pages, the spammy page will redirect you away to really nasty stuff.
In short, there were so many spammy pages on doc-weblogs.com that it affected the reputation of all of doc-weblogs.com. Normally our ranking system would be able to separate the spammy/bad pages from your posts with a fine level of granularity. The fact that we didn’t was the root cause (heck, let’s go ahead and call it a bug) of why your specific searches didn’t work well. That issue should be fixed now. For the query [searls normoyle] I see this post first and then a post from doc-weblogs.com at #2. I also see you showing up for the queries [doc searls tamron zoom canon lenses] and [doc searls rob cottingham profitable tragedy] just fine.
So I think that in this case it was an isolated issue related to spammy pages on doc-weblogs.com and it’s fixed now. I’ll also be checking how the issue occurred in the first place to make sure that it doesn’t affect any other sites. If you’ve got any other questions or see other poor searches, definitely feel free to email me at the address that I left in my comment. I’m sorry that this issue affected doc-weblogs.com for a while; I use Google to find old blog posts of mine all the time, so I know it’s annoying when a blog post doesn’t show up when you expect it.
-
My sites rank better in Yahoo – and when I check and compare inbound links – Yahoo shows hundreds, even thousands on some and Google only shows about 5% of the links Yahoo shows.
Wish more people Yahooed instead of Googled a web search.
-
Matt, Google search is still not finding any of these three exact phrases:
“Alaska photo research”
“very rare sled dog title”
“Josephine Smith Wilson”on
Is it because I have some duplicate pages that I’m slowly getting to as I fill in information? When I created this web site I made a page for each category that I needed. You have to put something on the page in order to be able to save the page. Some pages have a message that items in this subject will be added when time allows, so there is some duplication. If that is the problem, it doesn’t seem to have been a problem for Yahoo search and AltaVista search. If that’s not the problem, what is keeping Google from completely indexing my web site?
-
Oops, the URL to the site that these 3 phrases are are on was stripped out. Click on my name, on the right, to get that URL.
-
Thanks, Matt. That’s great, and much appreciated (on behalf of others as well as myself). More when I’m not on a plane…
-
Hi Promo Items, you know that Google shows only a sample of links for the link: command, right? If you want to see all the backlinks for your site, you can verify your site at http://www.google.com/webmasters/ and that will let you download a CSV file with practically all the links we saw to your site.
Dick, I’m happy to ask someone to check into that. I passed it on to someone.
-
Pingback from Doc Searls Weblog · How about a find engine? on September 3, 2008 at 5:41 pm
-
Pingback from Google fyller tio år | KATTKORGEN on September 8, 2008 at 5:54 am
-
Pingback from Rob Cottingham » Rob Cottingham, go-to guy on September 18, 2008 at 1:47 am
Comments are now closed.
59 comments