Search continues to present many interesting problems, some focusing on new parameters, some others rewiring existing parameters, and a few others shielding these parameters from adversaries (e.g. spam). Through a blog post, Scott Huffman shares how Google evaluates improvements.

..but we are constantly evaluating everything, which can include:
- proposed improvements to segmentation of Chinese queries
- new approaches to fight spam
- techniques for improving how we handle compound Swedish words
- changes to how we handle links and anchortext
- and everything in between

Though spam and the web graph feature prominently, it is interesting to note how "internationalization" features in many of these evaluation examples, reflecting Google’s overall push in this direction. Evaluation is through click-through improvements, and statistically sound relevance metrics.

Evaluation metrics, I think, is one of those areas where academia can greatly influence and impact search. This is one of the more theory centric problems in search, not limited by the lack of large information retrieval data sets. Indeed, the recently concluded SIGIR conference featured many papers in this direction.

Score Standardization for Inter-Collection Comparison of Retrieval Systems [PDF]
W. Webber, A. Moffat and J. Zobel  (University of Melbourne)

The Good and the Bad System: Does the Test Collection Predict Users’ Effectiveness? [PDF]
A. Al-Maskari, M. Sanderson and P. Clough  (University of Sheffield)

Retrieval Sensitivity Under Training Using Different Measures [blog]
B. He, C. Macdonald and I. Ounis  (University of Glasgow)

Evaluation Over Thousands of Queries [PDF]
B. Carterette, V. Pavlu, E. Kanoulas, J. Allan, and J. A. Aslam  (University of Massachusetts Amherst/Northeastern University)

Novelty and Diversity in Information Retrieval Evaluation [PDF]
C. Clarke, M. Kolla, G. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon  (University of Waterloo)

Relevance Assessment: Are Judges Exchangeable and Does it Matter [PDF]
P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. de Vries and E. Yilmaz  (NIST/Northeastern University/Microsoft/CWI/CSIRO ICT Centre)

Intuition-Supporting Visualization of User’s Performance Based on Explicit Negative Higher-Order Relevance [link]
H. Keskustalo, K. Jarvelin, A. Pirkola and J. Kekalainen  (University of Tampere)

Elsewhere, comments on the article  –

Seo Dialect:

The rest of the points are things we’ve been hearing from Google for a long time. We know they’re progressing on universal and personalization search efforts, all in their famous intent to create the best user experience.

Webtribution:

Anyone remotely involved in SEO or digital marketing should always take advantage of any information / insight Google opens to the public.

SearchEngineCaffe:

One of my biggest issues with TREC and similar environments is the single focus on relevance … for example, a spam post that is relevant to a topic would be acceptable, even if you would never want to read it in real life. It’s time we move beyond the basics and find ways to tackle the more challenging retrieval quality aspects…

Also, at webmasterworld.

For readers interested in the overall problem of IR evaluation, a paper by Kalervo & Jaana on "IR evaluation methods for retrieving highly relevant documents" offers an excellent introduction.

Kai-Fu Lee, from Google China presented the keynote at SIGIR. The theme was centered around — “Can a multinational Internet company succeed in China?”. Many of the presented ideas are applicable to the Indian market as well.

Internet in China, now:

• Largest Internet market, with over 250 million users (200 million in the US)
• Broadband penetration is 86%, mainly through ADSL
• Internet cafes quite popular even in small townships

Google’s strategy over the last three years:

• Starting 2006, learn about Chinese users
• 2007 and 2008, build the best Chinese search engine
• Launch products designed by Chinese engineers

Hopefully, market share will follow. To show that this isn’t that simple, Kai-Fu presented a detailed analysis, as summarized through this table:

Attribute	US	China	Implications
Alternatives to Internet	Plenty	None	High dependence and trust on Internet information
New Users	3%	35%	Embrace new users to gain market share
Internet-Cafe Users	<1%	33%	First time user sees Internet at IM/gaming/music
Average Age	45	25	Entertainment & Communities
English skills	High	Low	Highest quality content is still in English; its phenomenal in the medical domain
Internet Music	iTunes, iPod	Baidu, PC	Monetizing difficult
Piracy	21%	96%	Ready for high quality free-software as service
Credit Cards	2/person	0.02/person	Local e-commerce more applicable
Cell Phones	190 million	600 million	Mobile Internet users have never touched a PC before (India in a similar situation)

Using this background, Kai-Fu shared a few additional characteristics of Chinese users.

On new users:

• Internet is more an entertainment source, less an informational source
• Since users pay by the hour in Internet-Cafes, they love the concept of directories. In one user study, he mentioned a user questioning Google’s design with the question — “Did you forget to design rest of the page?”
• More of an emphasis to Universal Search
• Winning on metrics did not reflect on increase in market share

On users in general:

• Time spent on a result page is 10 seconds in the US. Kai-fu suggested it’s much higher in China, about 30-60 seconds. Most of these users are supposedly exploratory and curious, and click on more search results. Many of the queries are also informational, rather than navigational. To support this behavior Google used window pop-ups on search result click.
• To support informational needs of users, Google has turned their zeitgeist into a directory product in China.
• To alleviate the problem of misspellings, Google is branding “g.cn” as their official domain. Chinese users, apparently have problems spelling even “Google”. Query suggestions are also turned on by default.
• Much of the useful content on the Web is in English, mostly in the health verticals. To make such content available to Chinese users, Google is aggressively pursuing the content translation problem. Kai-Fu hinted that as the Web matures, this could be an important differentiation across search engines.
• To be more relevant to Chinese users, Google has developed new applications. One such application is “SMS Greeting” search. Kai-Fu revealed that in the Chinese culture SMS messages are replacing “greeting cards”. China supposedly has one of the highest SMS traffic around the New Year. Such niche applications can be useful for overall branding.

Kai-Fu finally ended by sharing that many of these directions helped Google increase market share from 14% in 2006 to about 25% in 2008 within China. We will have to wait to see if multinationals make further inroads in China. Time will tell.

Elsewhere on the keynote:

• Google in China and Africa
• Greg Linden: Kai-Fu Lee Keynote
• Name-not-available

The organizers of SIGIR 2008 shared numbers at the start of the conference. Highlights follow.

Healthy registration numbers, similar to those seen in 2007.

Event	2008	2007
Conference	574	599
Tutorial	464	363
Workshops	292	287

Slightly lesser overall numbers understandable, given that Singapore is 10-20 hrs away from most of Europe and Americas. Tutorial numbers high, thanks to a “package” registration initiated this year — an idea that could be adopted by other conferences as well. A quarter of the attendees are from the US. 19 are from India, not bad, but not high enough either.

In total, 85 full papers were accepted for an overall acceptance rate of 17%. The conference also features 99 posters (acceptance rate 50%), and 11 demonstrations (acceptance rate 58%).

The Program Committee shared some interesting numbers on acceptance rate by geography.

Geo	Submitted	Accepted
North America	159	41
EU	104	15
Pacific Rim (except US)	186	24
India	31	1

North America has a very high hit-rate. Acceptance rate for submissions from India needs much improvement, but its good to see 31 papers submitted, a good sign.

Lastly, the organization committee shared that they convinced the Singapore Government to host the National food festival
during SIGIR. Imagine something similar in other parts of the world!

I am now at the SIGIR conference in Singapore. Haven’t seen much user generated content around it though. Here’s a short (to be updated) list:

• Collaborative Blog, hosted by NUS
• Flickr Group
• Map, on activities

• Greg Linden
• Jacob
• Raman Chandrashekar
• Mark Sanderson

If you are around, feel free to ping.