Knowledge – Interview

Google is the de facto standard for Internet search engines. Pictures of the Future asked Craig Silverstein, 31, Google’s Director of Technology, how his market leading company sees the future of intelligent search technologies on the Internet.

The Internet is an unstructured collection of information in which Google searches for the proverbial needle in the haystack. What will be the major development trends for the Internet in the next decades

Silverstein: There will be a quantitative change. Much more information will be online. But the real difference will be qualitative. When everything you want to know is on the Web—not only movie schedules or heart-attack symptoms, but also where to find the items you need at your local grocery store—then the way you find and use information will be very different. This is particularly true when you can access all this data even when on the move.

How many websites are indexed by Google’s spiders?

Silverstein: We have the world’s largest index, with 4.3 billion documents. As the Internet—and our capabilities—expand, this number will grow. One of our key challenges is to broaden our search to incorporate other sources of public information. Recent efforts that go beyond Web pages include providing access to printed catalogues (Google Catalogue Search), newsgroup postings (Google Groups) and Google News.

What’s the difference between Google and other search engines?

Silverstein: Our technology is our primary differentiator. We have a scalable architecture designed to cull the plethora of Web pages, newsgroup messages, images, news items and shopping-related information and to deliver the most relevant results quickly and easily. We do this on a large scale, handling more than 200 million inquiries a day.

According to Google’s search algorithm, popularity counts. Doesn’t this support average search behavior?

Silverstein: Popularity is not the only criterion for our ranking system. The secret is that the system takes into account not merely popularity, but also reputation. If, on my homepage, I provide a link to a Web page saying, "This is the best Web page ever," this has little effect on Google, unless I can convince other Webmasters to share my opinion and they then link their Web pages to mine. So, far from having average results, Google ends up with the cream of the crop—websites that are so good that Webmasters the world over are willing to stake their reputations on them.

But the technology and the ranking system alone aren’t enough to equip Google for the future. One problem plaguing today’s users is that they often can’t find what they want, and this problem is likely to worsen. How do you propose to deal with it

Silverstein: Search results are already closely tailored to users’ needs, as indicated by what they enter in a search box. In my experience, an accurate query is all that’s needed for a search engine like Google to get good results.

This means the user has to be well-versed enough to search correctly, and the search engine offers no assistance. Will this change

Silverstein: Yes. We want to make the search engine smart enough—or at least enable it to pretend to be smart enough—to find what users want even when their inquiry is vaguely expressed. We won’t ever be as good as a human being at figuring out what somebody is looking for, but we hope to do an even better job in the future.

Will intelligent search software such as software agents be able to help users?

Silverstein: Not very much in the next ten to 20 years, I’m afraid. It will still be a long time before machines approach human intelligence. But luckily, machines don’t actually have to be intelligent; they just have to fake it. Access to a wealth of information, combined with a rudimentary decision-making capacity, can often be almost as useful. Of course, the results are better yet when coupled with intelligence. A reference librarian with access to a good search engine is a formidable tool.

What are the limits of a technical search? How intelligent can it be?

Silverstein: Eventually, very intelligent indeed. Why shouldn’t computers be able to reason in much the way humans do? But this presents an extraordinary challenge which, I believe, will take centuries to master—even though others are more optimistic.

To make searches more efficient, Web pioneers like Tim Berners-Lee propagate the use of meta-tags to structure the information on the Internet in what they call the Semantic Web. What’s your opinion here?

Silverstein: The goal of all searches is to give users the information they want while suppressing information they don’t need. Structure can be useful here. But structure also implies some amount of compartmentalization, which can cause the user to miss serendipitous connections. Also, depending on the number of mistakes inherent in the structure, information can be inadvertently lost. All large libraries certainly have a number of books that are mis-shelved, making them practically inaccessible. I think the people working on structuring the Web are well aware of these dangers. Structure is valuable as a factor in helping people find information—and since it’s created by intelligent humans rather than dumb machines, it can be a very useful factor. But it can also be a dangerous crutch.

Will the Semantic Web prevail?

Silverstein: Some parts of the Web will be structured, but I doubt whether the Web as a whole will be. The amount of information is so enormous and unwieldy that structuring it—if at all possible—would take at least ten or 20 years.

Another issue that is gaining in importance is making money on the Internet. Many companies, especially content providers, are increasingly in favor of paid content. How will Google deal with such an Internet?

Silverstein: There’s room for both paid content and unpaid content on the Web. I think both will be very healthy Web sectors for a long time to come. For years, we’ve been hearing concerns that—because of advertising—we shouldn’t trust the content on the Web any more because it is deteriorating. This hasn’t proven true yet, and I don’t believe it will be true in the future.

Will Google offer a paid search one of these days?

Silverstein: Google’s goal is to make all of the world’s information available, and we don’t have any preconceived notions about how to do it. Undoubtedly different answers will make sense for different types of information. The Google Answers program, at answers.google.com, already has a payment component. Users pay to have researchers provide in-depth answers to their questions. The questions are often highly complex, and users are more than willing to pay for this service.

In the early days, Google founders Sergey Brin and Larry Page weren’t really into making money. Google is sometimes described as creative chaos. How can you run a company with 1,000 employees with an approach like that?

Silverstein: In the beginning, when people asked us how we planned to make money, we’d say, "The key thing is to provide a service that people like to use. Once we have the users, the money will follow." And it’s worked. If we have 1,000 employees who take this approach, it may be pretty chaotic but the end result is valuable and useful. I really believe that we make a positive difference in people’s lives by making information more widely available and easily accessible than ever before. Whether it’s helping someone buy the best digital camera, or helping people with rare diseases find support groups, search engines have had a profound impact on the way people live their lives. I’m truly grateful that I’ve been able to make a contribution.

Interview conducted by Werner Pluta