First a tiny bit (I promise) of history.
In the early days of the World Wide Web, there were millions of new web pages going online every day, but the content was difficult to find. websites that served as directories popped up like mushrooms, the most prominent being Yahoo!—but these were all human curated. Dedicated workers, mostly hobbyists and volunteers, surfed the Web and gathered links to interesting content, organized into categories. But human curation couldn’t (and can’t) keep up with the explosion of content sprouting up by the hour.
To solve the problem, along came automatic indexing, where robotic spiders “crawl” overall those millions of pages, pulling out keywords and creating an index of locations for fast later retrieval, exactly the way the index works in a book. There were many search engines back in those days—Alta Vista, Ask Jeeves, Lycos and Excite, to name a few. But in 1998, along came the king of them all. Google quickly became the default search engine and thus the most powerful and important resource on the Internet. Today, its name is synonymous with Search, and Google is so powerful that there actually is no Internet without it.
Shortly after this, in the late 90s, websites came inside the corporate firewall in the form of the private intranet. Especially with larger organizations, each with a large body of content, Search, too, had to come in from the cold, so employees could find information like HR forms, benefits packages, and information about the company’s clients, for example. Numerous software companies stepped up and, unable to compete with the 800-pound gorilla Google on the public Internet, started offering Search tools tailored to private intranets (Google moved into this space too, with offerings like the Google Search Appliance). The category became known as Enterprise Search. And then, within a few short years, a common cry arose, which is still heard strongly to this day:
Why is intranet search so universally reviled by employees? It’s simple—we’ve all been spoiled by the astoundingly relevant results of the Google Page Rank algorithm. Google ascended to the throne of Internet search for a reason—the accuracy of its results. The thing that many corporate users don’t appreciate when trying to apply accurate indexing to a private body of content is that accuracy like Google’s comes at a high price. An incredibly high price.
It starts with engineering power. It takes a lot of ingenuity to create and continuously maintain Google’s Page Rank algorithm (which, BTW, is actually named after Google co-founder Larry Page and not, er, page ranking). Page Rank is the most jealously guarded corporate secret since the Colonel’s Secret Recipe of 11 Herbs and Spices. And as I just mentioned, the algorithm that is able to rank the most relevant results in the world is so good because it is constantly, constantly being adjusted and cultivated based on information more valuable than the content itself — namely, the hundreds of millions of real-world, human searches, and the 'winning' results, being conducted twenty-four hours a day.
With all of those hundreds of millions of pages on the public Internet, and the statistics generated by searches and clickthroughs, corporate search can’t really compete. Most private intranets have only hundreds of pages, perhaps thousands. And virtually no corporate IT departments perform the important work of TUNING the results—a task with which Google is obsessed. For a corporate intranet’s search engine to get anywhere near end-user satisfaction with the result relevance, an organization needs at least a year, in my estimation, of query/result data to analyze and apply to tuning the results.
Tuning includes such esoteric activities as Best Bets, well-tended thesauri, entity identification (e.g., training the indexer to recognize, say, the name of a client as the name of a client), and, most importantly, identifying and prioritizing the authority of the sources of content.
And of course, as with any kind of organizational change, search-engine tuning falls under Louis Gerstner’s often-quoted maxim: you can’t manage what you don’t measure. To know if their efforts are actually worthwhile, an organization must formally poll the users before and after (and after that), and analyze the results to determine if they actually are getting better. A vague and anecdotal idea that “everybody hates the search” is not very scientific.
I know what you’re thinking—all of this effort to get better search results sounds expensive. One response to that complaint: how much money are you losing from the proven fact that the average desk worker spends about 2.5 hours per day looking for information (source: IDC)?
The other, messier, answer is that perhaps, after almost twenty years of unfulfilled potential, perhaps it’s time to take a totally new approach to reducing the billions of dollars wasted searching for information. What is that new approach? It is: your organization’s own custom ChatBot.
Read the rest of the story in Part 2 of this blog series.
Does your enterprise search suck? Contact Slater Hill today to start discussing how to cut that 2.5 hours per day per employee.
