My Data Says You Like Beautiful Things

I admit I already wrote about Doug Bowman's story of how Google prefers data over designer judgement. But something stuck with me about this story. I didn't feel like I had really gotten to the heart of it, and I couldn't put my finger on why that was. Until now.

The idea, I assume, behind using large data sets to make decisions is to remove the human ego from the equation and let the machine be purely objective. The thing that's so vexing about this approach is that it's nearly impossible to argue with because it provides "empirical evidence," and yet there's a lingering "it just doesn't feel right" scent to it. The subtext is that the machine is more perfect than its flawed human counterparts.

Yes, egos can block progress, but sometimes you need a borderline crazy person to push through all of the doubts and get somewhere new. Sometimes you need to ship the wrong thing to get to the right thing later. The designers I trust and respect understand this, but it's much harder to explain to everyone else.

One piece of software I'm working on right now has gone through many iterations of user interface, implementation, and even overall strategy. You could argue that all of the prototypes we didn't use were failures, and we would have saved time using a large dataset and an algorithm to eliminate the non-perfect options immediately. Or we could have simply declared it "good enough" and checked it off our list. But every time we went down the "wrong" path, we learned something concrete — if not about software than at least about people.

It also helps the designers develop their intuition for what a good idea looks like. No matter how good the machine's algorithm is, you need someone to dream up the seed idea in the first place. Listening to people reason out what they like and don't like helps you come up with bigger and better ideas, but you don't get any of that from a pie chart.

So my issue with reducing everything to a simple yes/no question is that we miss out on all of the peripheral value of process, which prevents us from making something really spectacular — and that "something" may be a team of talented engineers. It's like trying to "solve" night by putting giant mirrors in space. Yes, it appears to fix one problem in the very short term, but it throws everything off balance.

Putting trust in the aggregate is useful when you're actually looking for statistical data, but could a computer have ever dreamed up an iPhone? I doubt it.
Design Element
My Data Says You Like Beautiful Things
Posted Oct 18, 2009 — 17 comments below


Alex — Oct 18, 09 6945

Nice piece. I've always felt that Google was not evil, but was becoming slightly more non-human...

Darcy Murphy — Oct 18, 09 6946

Likewise, Henry Ford proclaimed that if he had asked people what they wanted they would have told him they wanted a faster horse.

While reading this the idea of a mob mentality crept into my mind. Rarely in those situations do good outcomes occur, yet I feel that's what happens when you pour thousands of people down such a limited set of paths. You create mobs of relatively weak values because they can only react to whats directly in front of them, and can't extract themselves from the situation and look at the bigger picture. Nor can a single person's voice be heard if they do. The overwhelming majority drowns out the sensical voice.

Alan — Oct 18, 09 6947

My despair about human factors began many years ago upon going through volumes and volumes of the journal, Human Factors, where evidence was reported using statistical graphics of wretched quality, with thinner data and worse designs than even in corporate annual reports. Also the methodological quality of the research was poor, and so nothing was credible. The findings seemed entirely context-dependent, univariate (design and seeing are profoundly multivariate), and without scope: what did it matter if some students in freshman psychology in Iowa preferred one clunky font compared to another clunky font in an experiment conducted by a teaching assistant? Later, while consulting, I saw this naive dust-bowl empiricism fail again and again for nearly a decade in trying design a competent PC OS interface. (And with the Mac interface sitting there, smiling, all the time. Apple's superb interface guidelines seemed to me to be a retrospective account of the beautiful hands-on craft of a few brilliant designers, not a reason to have experimental psychologists attempt to design OS/2 and Windows.)

At any rate, if this was the scientific practice and the design craft of applied psychology, I concluded the field did not have much to contribute to my own work on analytical design.

I happily fled to the classics of science, art, and architecture.

-- Edward Tufte, November 27, 2002

Marc Edwards — Oct 18, 09 6948

Agree completely with everything you've said here.

I didn't understand it at all while studying design. It took a very vague, but insanely talented art director (my mentor and definitely a "borderline crazy person") several years to teach me this. The journey will consist of many failures, but when you reach the destination, you'll know exactly why. That means you'll have a better idea of what you should and shouldn't do in the future with the project.

If you were given the final design on a silver platter, you wouldn't treat it with the same respect.

Keith Lang — Oct 18, 09 6949

Google's design approach is a little like evolution. Well, if there's some randomness thrown in there, a LOT like evolution.

The question is, are all design solutions possible through evolution? That is, could there be a great design solution which is not possible to get to via a pathway of data-driven evolved design?

I don't know the answer.

Jonathan Stark — Oct 18, 09 6950

Great post!

Hmmm... a series of failures that lead to perfection? We had a word for that back in music school:



Jacob Rus — Oct 18, 09 6951

I took a user interface design course at college a couple of years ago, and all the way through I felt vaguely uneasy about its approach, but I think this post (and the Tufte quote from Alan) really nails it. Our assignments glorified process for the sake of process, and formality for the sake of formality. On the surface, this seems like it would be “better” preparation for the “real world” in which people do formal usability studies and carefully test tiny bits of options in a rigorous way, but the focus was in my opinion exactly backward.

Usability studies, informal or formal, are about gathering data and questioning our intuitions and assumptions, but they are helpful tools to be used, rather than ends in themselves. The real goal is understanding – understanding the problem domain, understanding the users’ goals and mental models, understanding the capabilities and constraints of the input devices and processing power, etc. – followed by communication. And so far, we haven’t invented the machine that can synthesize and analyze these data in any way comparable to the sensitive human observer/designer.

I’m somewhat reminded of Abelson and Sussman’s oft-quoted aphorism, “programs must be written for people to read, and only incidentally for machines to execute.” Some generalization of this applies to science, art, design, etc. Whenever we privilege process and systematization, we run the risk of forgetting that communication of hard-won insight is the ultimate goal.

... sorry if the above is incoherent. I’ll think about this some more when I’ve had enough sleep. :-)

Steve Marmon — Oct 18, 09 6952


I think you're spot on about Google's design process being like evolution. They pick a design, then let their computers tweak that design until they've maximized their ad clicks. It's like so many bacteria mutating in a petri dish.

The fundamental problem with this approach is that it is equivalent to what is known as hill climbing in the world of algorithms. If you haven't heard of hill climbing, imagine a mountainous terrain. Each point in that terrain corresponds to a different point in the design space, and the height corresponds to the "goodness" of that design.

What Google does is pick a point in that design space, throw it in front of a bunch of users, then assign it a score. Next, they tweak a parametersay, the shade of blueand throw it out there again. If the change results in a lower score, they must be going downhill. If it results in a higher score, then they must be going uphill. Lather, rinse, repeat.

The great thing about this is that once they come up with in initial design, the designers at Google can sit back and relax as their computers tweak that design until they can't get a higher score. This might seem brilliant, but there is a flaw. Remember how they just picked a point in the terrain and started climbing from there? Well, chances are, the mountain they started climbing probably wasn't the highest one in the range. And unless they are willing to go back to the drawing board to try a radically different design, there is no way that their automated parameter-tweaking technique is ever going to get them there.

This is not to say that Google only ever tries just one design. Maybe they do come up with several very different designs, then empirically test them with users. That's great. But the fact remains that there's a very big design space out there, and computers just don't have the imagination to get you to a truly spectacular height.

Jacob Rus — Oct 18, 09 6953

One more link that belongs here, and some selected quoted chunks:

Alan Kay’s talk at Creative Think seminar, July 20, 1982:

The best way to predict the future is to invent it.

Turn up your nose at good ideas. You must work on great ideas, not good ones.

The computer shouldn't act like it knows everything.

Relative judgements have no place in art

Point of view is worth 80 IQ points

Final advice: content over form, go for fun.

Jacob Rus — Oct 18, 09 6954

Bah, my link got eaten.

David — Oct 19, 09 6955

A great post! When I was in college I took some courses in marketing. I was taught about "marketing savvy" which is similar to what you describe above. People who understand and act at an intuitive level. The problem, my professors said, was that you can't count on this marketing savvy. It can bring great success but also great failure and you can't always find someone who has this magical quality. So, these ingenious academicians tried to define a new way, which involved turning marketing from an art into a science. Marketing research, focus groups, rigid methodologies to follow. The result is often mediocrity. When you apply science to human endeavors, you get a flatter curve ... less failures but also less spectacular awesomeness.

I'll take the crazy road any day. Think outside of the box? What box?

Lukas Mathis — Oct 19, 09 6956

"Putting trust in the aggregate is useful when you're actually looking for statistical data, but could a computer have ever dreamed up an iPhone?"

I don't really understand the dichotomy here. Google doesn't use computers to "dream up" user interfaces; they use statistical data to evaluate the "effectiveness" of user interfaces dreamt up by humans. I think the main difference between Google and most other companies (and the reason why Google is more successful than most other companies) is that Google puts a design's actual "effectiveness" above its designer's ego.

So this:

"One piece of software I'm working on right now has gone through many iterations of user interface, implementation, and even overall strategy."

Is probably pretty much what Google does, too; but Google uses statistical data to decide whether a design needs to be abandoned or improved, and to decide which one of a number of possible designs works best.

Martin Pilkington — Oct 19, 09 6957

Ego is needed in design. You need to be able to argue for the merits of your design. If you're just choosing what is the best UI for the average user based on data then that is equivalent to asking a focus group. How do you define what is best? You can't. What is best for me is not best for someone else. All google is doing is making bland UIs that appeal to the masses, but do little to truly innovate.

Lukas Mathis — Oct 19, 09 6958

> Ego is needed in design.

I don't deny that. But ego shouldn't be put above results. Software designers are not designing for an art gallery; their products need to provide a useful service to their customers.

> If you're just choosing what is the best UI for the average user based on data then that is equivalent to asking a focus group.

I guess I see where we disagree. You associate using data to improve design with focus groups. Focus groups don't provide data, they provide anecdotes; subjective opinions. That may in some cases be useful, but it has nothing to do with statistical data showing how people actually use a product.

When Google makes a design change and presents it to a focus groups, that may or may not result in useful feedback (probably not).

When Google makes a design change, runs an A/B test, a 5% fewer people manage to successfully sign up for GMail with the new design, then the new design needs to be improved regardless of how pretty it is, and regardless of the designer's ego.

Statistical data is a *design tool* designers should use to improve their designs (just like usability tests and accessibility tests). It's not in any way the antipode of design. It doesn't in any way negate the requirement to have great designers.

Chris Johnston — Oct 19, 09 6960

Good design is 100% usability and doing the proper research, testing, and modifications to ensure that your application is aesthetically pleasing and users can actually use it.

Great design is 80% good design and 20% telling your users what they want and how they should use it. Contrary to popular belief, users do not always know what they want and they can't always recognize what they want when they see it. Great design and great interfaces are about going that extra 20% and figuring out what your users really want and providing it for them.

Lukas Mathis — Oct 20, 09 6961

Contrary to popular belief, users do not always know what they want and they can't always recognize what they want when they see it.

Contrary to popular belief, usability testing and statistical analysis of user behavior have nothing to do with giving people what they ask for, and everything to do with giving people what they actually really want (even if they don't know it) ;-)

Usability testing and A/B testing are not about asking people. They are about observing people, about looking at what they actually do. There is absolutely no contradiction between telling people what they really want, and doing usability research. In fact, if you're going to do something crazy that has never been done before, you better do the research and make sure people can actually use it.

mmw — Jan 09, 10 7088

- math are god's expectation, physic and bio are real, cannot solve anything with math but we can expect.

- machines, computers are stupid but they can compute faster than mankind brain and it's useful to get expectation, machine-learning is a joke never worked and won't, if you calculate the result (time/cost) since 40 years, is inferior of 10 000 000 to cancer/aid researches (the worst fields regarding discovers) in 20 years threshold (that's applied math)

- can a machine ever understand the concept of madness? Vincent van Gogh

- can a machine ever understand the concept of music?
IRCAM? a joke

- can a machine ever understand something ...

- machine state won't be never creative, machines are a good tool that's it


Comments Temporarily Disabled

I had to temporarily disable comments due to spam. I'll re-enable them soon.

Copyright © Scott Stevenson 2004-2015