My Data Says You Like Beautiful Things
I admit I already wrote about Doug Bowman's story of how Google prefers data over designer judgement. But something stuck with me about this story. I didn't feel like I had really gotten to the heart of it, and I couldn't put my finger on why that was. Until now.The idea, I assume, behind using large data sets to make decisions is to remove the human ego from the equation and let the machine be purely objective. The thing that's so vexing about this approach is that it's nearly impossible to argue with because it provides "empirical evidence," and yet there's a lingering "it just doesn't feel right" scent to it. The subtext is that the machine is more perfect than its flawed human counterparts.
Yes, egos can block progress, but sometimes you need a borderline crazy person to push through all of the doubts and get somewhere new. Sometimes you need to ship the wrong thing to get to the right thing later. The designers I trust and respect understand this, but it's much harder to explain to everyone else.
One piece of software I'm working on right now has gone through many iterations of user interface, implementation, and even overall strategy. You could argue that all of the prototypes we didn't use were failures, and we would have saved time using a large dataset and an algorithm to eliminate the non-perfect options immediately. Or we could have simply declared it "good enough" and checked it off our list. But every time we went down the "wrong" path, we learned something concrete — if not about software than at least about people.
It also helps the designers develop their intuition for what a good idea looks like. No matter how good the machine's algorithm is, you need someone to dream up the seed idea in the first place. Listening to people reason out what they like and don't like helps you come up with bigger and better ideas, but you don't get any of that from a pie chart.
So my issue with reducing everything to a simple yes/no question is that we miss out on all of the peripheral value of process, which prevents us from making something really spectacular — and that "something" may be a team of talented engineers. It's like trying to "solve" night by putting giant mirrors in space. Yes, it appears to fix one problem in the very short term, but it throws everything off balance.
Putting trust in the aggregate is useful when you're actually looking for statistical data, but could a computer have ever dreamed up an iPhone? I doubt it.
My Data Says You Like Beautiful Things
Posted Oct 18, 2009 — 17 comments below
Posted Oct 18, 2009 — 17 comments below
Alex — Oct 18, 09 6945
Darcy Murphy — Oct 18, 09 6946
While reading this the idea of a mob mentality crept into my mind. Rarely in those situations do good outcomes occur, yet I feel that's what happens when you pour thousands of people down such a limited set of paths. You create mobs of relatively weak values because they can only react to whats directly in front of them, and can't extract themselves from the situation and look at the bigger picture. Nor can a single person's voice be heard if they do. The overwhelming majority drowns out the sensical voice.
Alan — Oct 18, 09 6947
At any rate, if this was the scientific practice and the design craft of applied psychology, I concluded the field did not have much to contribute to my own work on analytical design.
I happily fled to the classics of science, art, and architecture.
-- Edward Tufte, November 27, 2002
Marc Edwards — Oct 18, 09 6948
I didn't understand it at all while studying design. It took a very vague, but insanely talented art director (my mentor and definitely a "borderline crazy person") several years to teach me this. The journey will consist of many failures, but when you reach the destination, you'll know exactly why. That means you'll have a better idea of what you should and shouldn't do in the future with the project.
If you were given the final design on a silver platter, you wouldn't treat it with the same respect.
Keith Lang — Oct 18, 09 6949
The question is, are all design solutions possible through evolution? That is, could there be a great design solution which is not possible to get to via a pathway of data-driven evolved design?
I don't know the answer.
Jonathan Stark — Oct 18, 09 6950
Hmmm... a series of failures that lead to perfection? We had a word for that back in music school:
Practice
;-)
Jacob Rus — Oct 18, 09 6951
Usability studies, informal or formal, are about gathering data and questioning our intuitions and assumptions, but they are helpful tools to be used, rather than ends in themselves. The real goal is understanding – understanding the problem domain, understanding the users’ goals and mental models, understanding the capabilities and constraints of the input devices and processing power, etc. – followed by communication. And so far, we haven’t invented the machine that can synthesize and analyze these data in any way comparable to the sensitive human observer/designer.
I’m somewhat reminded of Abelson and Sussman’s oft-quoted aphorism, “programs must be written for people to read, and only incidentally for machines to execute.” Some generalization of this applies to science, art, design, etc. Whenever we privilege process and systematization, we run the risk of forgetting that communication of hard-won insight is the ultimate goal.
... sorry if the above is incoherent. I’ll think about this some more when I’ve had enough sleep. :-)
Steve Marmon — Oct 18, 09 6952
I think you're spot on about Google's design process being like evolution. They pick a design, then let their computers tweak that design until they've maximized their ad clicks. It's like so many bacteria mutating in a petri dish.
The fundamental problem with this approach is that it is equivalent to what is known as hill climbing in the world of algorithms. If you haven't heard of hill climbing, imagine a mountainous terrain. Each point in that terrain corresponds to a different point in the design space, and the height corresponds to the "goodness" of that design.
What Google does is pick a point in that design space, throw it in front of a bunch of users, then assign it a score. Next, they tweak a parametersay, the shade of blueand throw it out there again. If the change results in a lower score, they must be going downhill. If it results in a higher score, then they must be going uphill. Lather, rinse, repeat.
The great thing about this is that once they come up with in initial design, the designers at Google can sit back and relax as their computers tweak that design until they can't get a higher score. This might seem brilliant, but there is a flaw. Remember how they just picked a point in the terrain and started climbing from there? Well, chances are, the mountain they started climbing probably wasn't the highest one in the range. And unless they are willing to go back to the drawing board to try a radically different design, there is no way that their automated parameter-tweaking technique is ever going to get them there.
This is not to say that Google only ever tries just one design. Maybe they do come up with several very different designs, then empirically test them with users. That's great. But the fact remains that there's a very big design space out there, and computers just don't have the imagination to get you to a truly spectacular height.
Jacob Rus — Oct 18, 09 6953
One more link that belongs here, and some selected quoted chunks:
Alan Kay’s talk at Creative Think seminar, July 20, 1982:
Jacob Rus — Oct 18, 09 6954
http://www.folklore.org/StoryView.py?story=Creative_Think.txt
David — Oct 19, 09 6955
I'll take the crazy road any day. Think outside of the box? What box?
Lukas Mathis — Oct 19, 09 6956
I don't really understand the dichotomy here. Google doesn't use computers to "dream up" user interfaces; they use statistical data to evaluate the "effectiveness" of user interfaces dreamt up by humans. I think the main difference between Google and most other companies (and the reason why Google is more successful than most other companies) is that Google puts a design's actual "effectiveness" above its designer's ego.
So this:
Is probably pretty much what Google does, too; but Google uses statistical data to decide whether a design needs to be abandoned or improved, and to decide which one of a number of possible designs works best.
Martin Pilkington — Oct 19, 09 6957
Lukas Mathis — Oct 19, 09 6958
I don't deny that. But ego shouldn't be put above results. Software designers are not designing for an art gallery; their products need to provide a useful service to their customers.
> If you're just choosing what is the best UI for the average user based on data then that is equivalent to asking a focus group.
I guess I see where we disagree. You associate using data to improve design with focus groups. Focus groups don't provide data, they provide anecdotes; subjective opinions. That may in some cases be useful, but it has nothing to do with statistical data showing how people actually use a product.
When Google makes a design change and presents it to a focus groups, that may or may not result in useful feedback (probably not).
When Google makes a design change, runs an A/B test, a 5% fewer people manage to successfully sign up for GMail with the new design, then the new design needs to be improved regardless of how pretty it is, and regardless of the designer's ego.
Statistical data is a *design tool* designers should use to improve their designs (just like usability tests and accessibility tests). It's not in any way the antipode of design. It doesn't in any way negate the requirement to have great designers.
Chris Johnston — Oct 19, 09 6960
Great design is 80% good design and 20% telling your users what they want and how they should use it. Contrary to popular belief, users do not always know what they want and they can't always recognize what they want when they see it. Great design and great interfaces are about going that extra 20% and figuring out what your users really want and providing it for them.
Lukas Mathis — Oct 20, 09 6961
Contrary to popular belief, usability testing and statistical analysis of user behavior have nothing to do with giving people what they ask for, and everything to do with giving people what they actually really want (even if they don't know it) ;-)
Usability testing and A/B testing are not about asking people. They are about observing people, about looking at what they actually do. There is absolutely no contradiction between telling people what they really want, and doing usability research. In fact, if you're going to do something crazy that has never been done before, you better do the research and make sure people can actually use it.
mmw — Jan 09, 10 7088
- machines, computers are stupid but they can compute faster than mankind brain and it's useful to get expectation, machine-learning is a joke never worked and won't, if you calculate the result (time/cost) since 40 years, is inferior of 10 000 000 to cancer/aid researches (the worst fields regarding discovers) in 20 years threshold (that's applied math)
- can a machine ever understand the concept of madness? Vincent van Gogh
- can a machine ever understand the concept of music?
IRCAM? a joke
- can a machine ever understand something ...
- machine state won't be never creative, machines are a good tool that's it