Web Excursions 2022-01-12
Is Google Search Deteriorating? Measuring Google's Search Quality in 2022
There's been a lot of discussion on Hacker News recently about the quality of Google Search, and whether it’s deteriorated.
A couple plausible reasons:
Google has been prioritizing short-term ad revenue over search quality.
Information is moving beyond traditional webpages.
Historically, Google Search contained little ML.
Crucially, measuring search quality is a very difficult problem.
it's very difficult to measure search quality using traditional metrics.
Clicks aren't necessarily something you want to optimize for
Neither is time spent searching: is a short session a good thing (perhaps you found your answer immediately) or a bad thing (the search results were so bad you quickly gave up)?
Reformulations: if your initial search query failed, you may rewrite your query and try again, so an increase in reformulations could be viewed as a bad thing.
But many people will give up instead of reformulating,
and how do you tell whether a query is a reformulation anyways?
Maybe long-term metrics are the solution.
So in order to measure the quality of Google Search, here was my process:
I leveraged a set of human raters
I asked 250 raters to look up a recent search in their browser history, and to use that as the search query to rate.
Each rater then explained their original intent, rated how well the Google SERP satisfied that intent on a 1-5 scale, and explained their judgment.
an examples of a search result Google performed poorly on.
"tim lee vlogger age"
"Only some of the results were about the right person, and I couldn’t find his age from the results at all."
Example: Google vs. Bing
"Natural ways to heal cats who have allergies"
The Google results page was not focused on my topic.
The results were confused in that half of the content pertained to allergies in cats, and
the rest were for general pet allergies or allergies to cats in humans.
Bing's results were spot on.
They provided products to buy in the advertisement section,
then they offered a specifically targeted article that suggested how to treat and prevent allergies in cats utilizing home remedies.
"what is message blocking on iphone"
GOOG (passable): The first and third search results misunderstood my query and thought I was asking how to block other people. The second result, however, was helpful.
BING (good): The website search results all understood what I was looking for, and were related to the message I received.
"Indianapolis free COVID PCR tests"
GOOG (bad): a page full of ads.
BING (good): The second search result was a map with a listing of testing resources, their locations, and when they were open.
The Good, the Bad, and the Bimyou
In Japan, you will regularly be given offers you can’t refuse.
With a straight “no,” after all, you might invite confrontation or offense.
Instead, say bimyou (微妙), a word as indistinct as a wisp of cloud, as nonbinding as a weather report.
Bimyou is a kind of negative space,
a vehicle for doubt or uncertainty or anything counter to the expected flow of an interaction.
Bimyou has roots in the Buddhist concept mimyo, referring to something of an indescribable wonder.
Its widely used meaning is “subtle,” which has a pleasant literary quality.
By 2000, it had morphed into something more colloquially, dismissively bland and was the most widely recognized piece of slang in a 2015 survey by the Japanese Agency for Cultural Affairs.
The Nobel Prize-winning novelist Kazuo Ishiguro characterizes the relationship between the main characters of his book Never Let Me Go as “fragile and bimyou.”
The avant-garde artist Taro Okumoto once described a woman’s kiss as the moment when her “mind and body are subtly”—bimyouni—“intertwined,”
In speech, it can be used to describe anything from a sensitive issue (bimyou na mondai) to a tricky political relationship (bimyou na kankei).
More recently, young people began using it as a way to express negativity or apathy, a kind of verbal shrug.
It basically means ‘iffy’ or ‘questionable,’ but it’s often used to avoid saying something bad
“If I asked you how a restaurant was and you replied with ‘bimyou,’ I’m not going there.
If I gave you a gift and you called it ‘bimyou,’ I’d think: ‘Great, she hates this gift.’”
Bimyou also describes the unwillingness of young people to engage with politics.
It would be fatal, though, not to recognize something steely in being bimyou.
There’s a determination not to be drawn into something that would make one vulnerable.
If you receive a sense of bimyou in Japan, in politics or otherwise, move on.
It is unlikely that you’ll get the answer you’re looking for.
There is that kind of clarity to the word, after all.
This past summer we narrowly avoided a major user interface regression on Apple devices.
Why did this happen in the first place? - My answer is something I call “consistency sin”
You can get into trouble when this consistency starts to affect the user experience.
Design is not how it looks, it’s how it works. —Steve Jobs said a lot of smart things, but I use this advice most often.
The roots of consistency sin take hold when folks disregard the inherent differences between platforms.
The consistency sin in Safari was to come up with a good design for iOS and assume that it would also work well on iPadOS and macOS.
It practice, these new tabs were difficult to use in a different work environment.
Placement is also an issue: on iOS controls tend toward the bottom of the screen (for reachability).
The opposite is true on macOS where they tend toward the top of the screen so they’re closer to the menu bar and window controls.
Consistency sin says that notifications should always be at the top of the screen.
With iOS, there’s a nice visual and functional separation between app interactions in the lower half of the screen and notifications in the upper half.
On the Mac, notifications are just another thing fighting for real estate at the top of the screen.
If you’re a designer or developer, it’s your job to push back on the notion of consistency when it begins to affect a user’s experience.
Remember design is how it works, and work is not the same on every device.