To measure contentiousness, we looked at the variance of the ratings for each work. If most participants gave a work roughly the same rating, then it’s safe to say that work is not contentious. However, if participants disagree, if there’s a large amount of variance in the ratings, then that work might be contentious. (I say “might” for a good reason: while high variance of ratings may indicate disagreement, it could also simply indicate confusion. I’ll come back to this later.)

In Split Second Stats #4: Engagement we found that certain tasks in the experiment had a strong effect on the variance of ratings. This is important because it indicates that the context of presentation and the way participants engage with a work can change the variance. Here, however, we’ll take a look at how variance and contentiousness were related to specific properties of the works themselves. All of the analyses below apply to the unlimited time experimental tasks only.

As in many of the analyses described in previous blog posts, complexity played a big role here. We found that as paintings got more complex, they became less contentious. That is, we found a negative correlation between complexity and variance (cor = -.35, p = 0.03). This is not too surprising: we found previously that when time was unlimited, people tend to rate complex paintings very well, a finding which already implies inter-participant agreement. A more puzzling finding concerned color: The higher the overall saturation of the colors in a work, the higher the variance (cor = .42, p < 0.01). One possible, but entirely speculative, explanation for this effect is that one large group of our participants reacted very positively to highly saturated color palettes, which another large group reacted very negatively. Similarly, we found that the larger the frame of the painting, the more variance in ratings. This again might suggest (speculatively!) a division of the participant population into two groups: those that found large frames interesting, and those that found them to get in the way of the work.

Some of the strongest effects concerning variance were not clearly related to quantifiable properties of the works themselves. One very strong, reliable finding was that as the average amount of time participants spend looking at a work increased, the variance of the ratings of that work decreased (cor = -.47, p = 0.002). That is, the more time was spent looking at a work, the more our participants tended to agree about how to rate it. Though this finding seems to push against the gist of the thin slicing theory, it also seems like an encouraging experimental result: in order to get people to agree about art, you just need to get them to hold still and look at it for a long time. However, it’s a little bit more complicated than that. People decide for themselves whether or not they want to spend a long time looking at an artwork. This finding lets us know that when our participants spent that time, they tended to agree, but it doesn’t tell us why they decided to spend their time in the first place. There is also a cause-and-effect problem: it could be that the decreasing variance and the increasing time are themselves caused by a third factor we didn’t measure. (Though complexity looks like it may account for some of this effect, it certainly doesn’t account for all of it.)

Indian. Utka Nayika, late 18th century. Opaque watercolor on paper, sheet: 9 13/16 x 7 9/16 in. (24.9 x 19.2 cm). Brooklyn Museum, Gift of Dr. Ananda K. Coomaraswamy, 36.241

Finally, we found that some of the works in the experiment were simply contentious on their own terms. The most contentious object, Utka Nayika (pictured above), is unfinished. Though we have no quantifiable measure that points toward it being an unfinished work, it seems like a safe bet that this peculiarity accounts for the high variance in participants’ ratings. As I mentioned before, it’s important to differentiate between contentiousness and confusion. We can identify this work as being truly contentious, and not simply confusing, by looking at a histogram showing how it was rated.

In the case of a work which was simply confusing, we would expect a uniform distribution of ratings, where any one rating was as likely to occur as any other. Instead, what we see here are distinct peaks and valleys. There are small peaks around 25 and 100, and larger peaks around 50 and 75. This indicates participants’ opinions about the work split them into at least three groups: those who did not like it (the peak at 25), those who were decidedly indifferent (the peak at 50), and those who liked it a lot (the peaks at 75 and 100). A similar situation can be seen in the rankings histogram for the second most contentious object, The Bismillah, a work which is distinguished by its calligraphic, non-representational nature:

Indian. The Bismillah, 1875-1900. Opaque watercolor and gold on paper, sheet: 19 5/8 x 11 13/16 in. (49.8 x 30.0 cm). Brooklyn Museum, Gift of Philip P. Weisberg, 59.206.8

In both of these cases, symbolic factors not accounted for by our experimental model had an extremely strong effect on the results, strongly suggesting a direction for further research. As interesting as it is to see the symbolic world bursting out of our tightly constrained experimental framework, it’s not surprising: we are, after all, looking at art.

In the time-limited, Split Second task, we found various attributes of the frame of a painting had strong effects on how that painting was rated. The strongest effect was caused by the frame size, where bigger frames resulted in lower ratings. However, we also found that the surface complexity of the frame had a positive effect on ratings (cor = 0.19, p = 0.014). This effect was smaller, but definitely significant.

Indian. Krishna and Balarama on Their way to Mathura, Folio from a Dispersed Bhagavata Purana Series, ca. 1725. Opaque watercolor and gold on paper, sheet: 9 1/2 x 12 in. (24.1 x 30.5 cm). Brooklyn Museum, Gift of Mr. and Mrs. Paul E. Manheim, 69.125.4

A major goal of this experiment was coming up with some preliminary answers to the question of what, exactly, is factored into a split-second judgment. When we make judgments in time-limited contexts, we’re not able to make a thorough survey of the thing we’re judging. Instead we produce a judgment based on a number of subconscious processes which may be affected by more than the thing itself. In this particular case, we were interested in knowing whether the complexity of the frame was affecting conscious, systematic judgments, or was operating on a subconscious level.

To answer this question, we looked at how the complexity of the frame affected ratings in all of the other tasks. In the time-unlimited control task, where participants were given as much time as they liked to rate a work without being asked to do anything else, the frame complexity effect disappeared completely. That is, when people were allowed to take a thorough look at a work, the complexity of its frame did not affect their judgment. This was also true for all of the engagement tasks, which makes sense because those tasks require participants to take a systematic approach to evaluating each work’s surface.

In the time-unlimited Think tasks, where participants read information about the work, the frame complexity effect returned. That is, when participants paid attention to information about the painting, their judgment was again affected by the complexity of the frame. This suggests that attention paid to curatorial labels was also attention shifted from the work itself, and that this shift allowed certain aspects of the work to have a subconscious effect which would not occur in other circumstances. This effect was strongest when the full curatorial label was added (cor = 0.4, p = 0.01).

This finding is important from an exhibition design perspective. Curatorial interventions in the gallery space are always engaged in a kind of struggle with the art itself for spectator attention. Depending on how the attention of the spectator is focused, certain properties of artworks may be activated or suppressed. Some of these properties, such as the complexity of the frame, may only be activated when viewer attention is diverted or split in some way. A key aspect of the role of the curator is awareness of and sensitivity to the complex interdependencies between in-gallery interventions and various properties of the works. This experiment suggests an analysis of these interdependencies in terms of attention management: for any given curatorial intervention, how is attention diverted or split, and how does that activate or suppress properties of the work itself?

There are lots of ways a work of art could be complex. Complexity could be a function of the visual surface of a work, as in the arrangement of contrasting elements within it, or of things outside of it, as in the network of references pointed to by its content. Complexity could also come from a connection between the work and the viewer, as in the use of multiple perspective or other perceptual effects, or the viewer’s specific, personal relationship to a given historical context. Further complicating the situation, when people talk about a work being “complex” they usually don’t refer to only one of these possibilities—a complex work of art is rarely complex in just one way.

Indian. King Solomon and His Court, 1875-1900. Opaque watercolor and gold on paper, sheet: 19 11/16 x 11 7/8 in. (50.0 x 30.2 cm). Brooklyn Museum, Gift of James S. Hays, 59.205.16

When approaching the study of complexity (or any subjective idea) from a scientific perspective, its necessary to pick a one of two approaches. The first approach is to pick a specific way of quantifying the idea. This is always painful, because it means the implicit rejection of other approaches which may be really important to the feel of the idea. The second approach is to ask lots of people what they think—rather than trying to quantify the idea itself, you quantify peoples’ judgments about the idea. The problem with this approach is that it often requires an extra experiment in order to quantify those judgments before you can even get started working on the question you’re really interested in.

We chose the first approach. Rather than studying the complex, subjective idea of complexity, we decided to focus on one specific, measurable type of complexity, which could be called “surface complexity.” We were interested in how much was going on in a work without considering its content. For example, a work with a busier visual surface (with more dots, lines, curves, scratches, brush strokes, marks, etc.) would have greater surface complexity than a work with just a few lines, a couple of repeating patterns, and lots of open space.

Indian. Portrait of Rao Chattar Sal of Bundi, ca. 1675. Opaque watercolor and gold on paper, sheet: 7 5/16 x 4 11/16 in. (18.6 x 11.9 cm). Brooklyn Museum, Gift of Amy and Robert L. Poster, 82.227.1

This can be confusing, because it doesn’t necessarily match up with a normal idea of what complexity means. An image which is just a big field of scratches might have a much higher surface complexity than an intricate portrait—it just depends on how they were both painted, i.e. what’s happening on the surface of the work, ignoring its content.

Indian. Nanda Requests a Horoscope for Krishna, Page from a Bhagavata Purana series, ca. 1725. Opaque watercolor and gold on paper, sheet: 9 1/8 x 10 1/2 in. (23.2 x 26.7 cm). Brooklyn Museum, Gift of Mr. and Mrs. Robert L. Poster, 78.260.5

We quantified the surface complexity of an image in terms of the amount of data a computer needs to store in order to recreate it. That is, if one could describe the form of an image in one sentence, it would be less complex than an image requiring ten sentences. Because we were working with digital files, this approach was quite convenient: the surface complexity of an image corresponds exactly to that image’s file size after compression. The bigger the compressed file size, the more complex the image. Specifically, we used the ratio of the file size of the image to the number of pixels, allowing us to compare images with different dimensions.

One of the most interesting things we found out about surface complexity was that it variably affects participants’ reactions to artworks depending on what task they were completing. This means that what people were doing and paying attention to affects their reaction to complexity. Based on our results, we speculate that during certain tasks, complexity has a profound subconscious effect on participants’ reactions. I’ll discuss this in more detail in the next post!

In this task, participants were split up in groups. Each group was asked to perform a specific task which required them to engage with the content of the work they were looking at. The tasks were as follows:

• Counting: Type in the number of figures in the work.
• Description: Describe the work in your own words.
• Color: Name the dominant color in the work.
• Free association: Type the first thing that comes to mind when looking at the work.
• Tagging: Type a single word which describes the subject or mood of the work.
• No task: our control group, as described in our first stats blog post.

I expected that after completing any of these tasks participants would have a stronger emotional connection to the work, so the average rating would go up. Surprisingly, this was not the case. None of the engagement tasks had a statistically significant effect on average rating. Our curator Joan Cummins was not surprised by this, saying that curatorial interventions such as engagement tasks were not intended to make people enjoy the work more, but to get them to learn about it.

However, though the engagement tasks did not affect the average rating, they did affect the way ratings were distributed, i.e. how all of the participants’ ratings were spread out around the scale. We found that when participants completed an engagement task, their ratings clustered much more tightly together. In statistical terms, engagement tasks reduced the variance of the ratings. This means that, though engagement tasks don’t make people like things more, they make people’s ratings more consistent, or increase agreement about a work across the whole population of participants.

Indian. Episode Surrounding the Birth of Krishna, Page from a Dispersed Bhagavata Purana Series, late 17th-early 18th century. Opaque watercolor on paper, sheet: 10 1/8 x 15 15/16 in. (25.7 x 40.5 cm). Brooklyn Museum, Gift of Emily Manheim Goldman, 1991.180.10

Also surprising to me was which task reduced the variance the most. I expected that the description or tagging tasks would create the most agreement across participants, because they require people to evaluate what’s being portrayed in the work in linguistic terms. However, the counting task reduced variance the most, followed by the color and free-response tags (a tie for second place), then tagging, with the description task coming in dead last. We’ve speculated that this may be because of how the various tasks manipulated conscious attention—the description task focuses conscious attention on the content of the painting, whereas the counting task focuses your conscious attention on a more-or-less objective formal property (the number of figures).

Why, exactly, this would reduce variance is unclear. It may be because focusing on form instead of content means people don’t pay attention to things that might otherwise affect their rating of the painting, e.g. controversial subject matter. It may also create a situation where evaluation of the quality of the painting (as opposed to evaluate of its form) is passed along to the subconscious, and that (to extend Gladwell’s thin-slicing hypothesis) subconscious judgments may naturally tend to have less variance. This suggests yet another interesting direction for further research.

While ratings increased for both men and women when information was added, women’s ratings increased more. Men’s ratings go up by an average of 7.58 points, while women’s ratings go up by an average of 10.4 points. This indicates women react more enthusiastically to the addition of information than men. Now, this finding is an average calculated across all 40 objects included in the information section of the experiment. When we look at individual objects, the story gets more complicated. For certain objects, one gender increased their ratings substantially more than the other, and sometimes men and women would change their ratings in the opposite direction.

Indian. The Bismillah, 1875-1900. Opaque watercolor and gold on paper, sheet: 19 5/8 x 11 13/16 in. (49.8 x 30.0 cm). Brooklyn Museum, Gift of Philip P. Weisberg, 59.206.8

The largest difference between men and women when information was added was for The Bismillah, pictured above. When information was added to this painting, men’s ratings increased by 2.36 points, while women’s ratings increased by 20.37 points—an order of magnitude larger. Because these rating changes are a function of the form of both the painting and the additional information, it’s very hard to say why the changes are they way they are. The Bismillah is a non-representational painting, but we didn’t have enough similar paintings in our sample to determine whether that was the determining factor. Similarly, without conducting more controlled research, we can’t determine whether there was a specific element or topic of the additional information which caused the change, or whether there was some interaction between the non-representational nature of the painting and the description of its religious function.

Karam and Mahata Chandji. Double-sided Leaf from a Chandana Malayaqiri Varta series, 1745. Opaque watercolor and gold on paper, sheet: 11 3/8 x 7 7/8 in. (28.9 x 20.0 cm). Brooklyn Museum, Gift of Mr. and Mrs. Paul E. Manheim, 69.125.5

The gender-switched counterpart to The Bismillah is the Double-sided Leaf from a Chandana Malayaqiri Varta series. While women increased their ratings dramatically more than men for The Bismillah, men increased their ratings dramatically more than women for the Double Sided Leaf. Women’s ratings increased by an average of 2.25, while men’s increased by an average of 12.86. Again, this result is perplexing. The content of the painting doesn’t include any of the familiar tropes of gender-difference discussions like sexuality or violence. Although livestock are pictured, and there are some prominently featured mustaches, it’s hard to say whether these factors were decisive.

Mughal (style of). Lady with a Yo-yo, ca. 1770. Opaque watercolor and gold on paper, sheet: 9 1/4 x 6 3/16 in. (23.5 x 15.7 cm). Brooklyn Museum, Gift of Alan Kirschbaum, 80.268.1

For a few paintings, men’s and women’s ratings moved in opposite directions. The most dramatic example is Lady with a Yo-yo, pictured above. Women’s ratings increased by an average of 8.4, while men’s ratings dropped by an average of 5.6.

We did find a relationship between these ratings changes and the complexity of an image. For women, as paintings got more complex, their ratings increases got lower (cor = -.34, p = 0.028). That is, women were less affected by additional information as the complexity of the image increased. (Viewed a different way, women were more affected by additional information when paintings were simpler.) Men showed the same pattern, but it was much weaker—so weak, in fact, that we can’t be completely sure it was a statistically significant effect (cor = -.27, p > 0.08). Unfortunately, this finding does not explain the dramatic differences described above. To decide that question decisively, we’d need to design an experiment which would allow us to analyze these ratings changes in terms of the type of content included in a work.

Stay tuned for the next post!

In the Split Second section of the online experiment, participants were asked to pick one of two paintings in less than 4 seconds. We compared the Split Second results with results from a control task, where participants rated images one by one on a linear scale and had unlimited time to make their decisions. Now I’m going to talk about a third task: the “info” task. In the info task, participants were asked to read some information about the painting they were looking at, and then to rate the painting on a linear scale. In the info task there was unlimited time to make decisions. Participants in the info task were split into three groups: one group read the painting’s caption, another read a list of “tags” (single word descriptions), and the final group read a full curatorial label.

Whether or not labels and other additional information help or hurt our experience of art is a point of contention. The cases for and against labels both seem intuitively strong: labels enhance our understanding of the art and its history, but at the same time they can cloud our intuitive reaction to the work, interfering with the purely visual aspects of the “thin-slicing” process as described in Blink. The relevant question, from the perspective of a museum (as opposed to an art gallery) is whether education is at odds with enjoyment. Do educational labels just get in the way of the work itself?

Our data suggest the answer is a decisive “no.” For every painting ratings improved decisively with the addition of information. Adding captions improved ratings by an average of 5 points, tags an average of 6 points, and full labels a remarkable 10 points. Not only do the labels not get in the way of the art, they seem to make people enjoy it even more.

Now, we have a few more findings which complicate this picture:

Captions and tags seem to cause about the same boost in score. At first this was confusing—captions seem to be more information rich, so why weren’t they causing bigger boosts than tags? We think this slightly puzzling result has to do with our choice of Indian art. Unlike much contemporary art, where the title often provides crucial information for interpreting a work, the titles of our Indian paintings tend to be very tag-like, simply describing the work’s content. So, in this case, our assumption that captions were more information-rich than tags may have been incorrect.

Portrait of an Old Man is a simple painting whose score improved dramatically (20 points) when participants read its full curatorial label.

We found that some works improved much more with the addition of information than others. Images which got low scores in the Split Second and control tasks made the biggest improvements. That is, Split Second scores were negatively correlated with score improvement (cor = -.39, p = .013), as were scores in the control task (cor = -.62, p < .0001). Additionally, as with last week’s story, visual complexity is a significant factor. We found that simpler images reliably got bigger score boosts than complex images (cor = -.42, p = .006). Further, once information was added, the visual complexity of paintings was no longer correlated with their score. That is, the addition of information doesn’t just mute the positive effect of visual complexity, but completely removes it from the picture. This finding suggests that the addition of information about a work changes the process of judgement in a deep, fundamental way, activating an entirely different set of evaluative criteria.

Finally, at the Split Second event at the museum last Thursday, an audience member asked if we found any relationship between between the length of the label and the score. We did find such a relationship, but it wasn’t strong enough that we’re sure that it’s meaningful. We found that the scores given by participants who read the full label were correlated with the length of the label (cor = .28), but that we weren’t absolutely sure that this finding wasn’t due to random chance (CI: -.04 to .54, p = 0.08). The concern is that it may not be the content of the label which people are responding to when they give higher scores, but simply its length. We can’t yet answer this question, but our finding is strong enough (and the question tantalizing enough!) that this certainly suggests a direction for further research.

In this, the first in a series of posts about the Split Second experiment and our findings, I’m going to describe the first part of the experiment, and then say something about what some of the results might tell us about thin-slicing.

In the first part of the experiment, participants were presented with a series of pairs of Indian paintings, making snap decisions about which of each pair they liked best. We called this the “Split Second” task. Decisions made during the Split Second task were “thin” in two ways: First, each decision had a time limit of 4 seconds. Second, participants had no extra information about the painting, and had to “go from their gut.” The results from the Split Second task told us which paintings did better in thin-sliced conditions.

But looking at thin-slicing alone wasn’t quite enough for us. In order to really learn about how thin-slicing works, we needed to compare thin-sliced decisions to other kinds of decisions. To do this, we split off a number of participants into a “control group.” Rather than completing the second section of the experiment like the rest of the participants, the control group completed a neutral, unlimited time task with which we could compare all of the other tasks (like the Split Second task). The control group was presented with a series of individual paintings, with no additional information, and given unlimited time to rank each painting on a linear scale from “Meh…” to “Amazing!”

The result of each of these tasks came in the form of a ranking. The first ranking was based on thin-sliced decisions, and the second was based on decisions made with unlimited time. When we analyzed the two rankings, this is what we found:

• If a painting did well (or poorly) in the Split Second task, it probably did well (or poorly) in the control task, too. In science-speak: thin-sliced judgments were very strongly correlated with unlimited time judgments (cor = .8, p < .00000001). Good news for thin-slicing!

• However, when the time restriction was lifted, there were some huge rankings upsets! You can look at a comparison of the rankings on the Split Second exhibition page. Some objects saw big drops in ranking; e.g. Utka Nayika. More dramatically, we see King Solomon and His Court jump from somewhere near the middle of the pack all the way up to number one! This is an extremely dense, complex painting, which leads us to the next finding…

Indian. King Solomon and His Court, 1875-1900. Opaque watercolor and gold on paper, sheet: 19 11/16 x 11 7/8 in. (50.0 x 30.2 cm). Brooklyn Museum, Gift of James S. Hays, 59.205.16

• Visually complex paintings did pretty well in the Split Second task, but did extremely well in the control task. That is, paintings that had a lot of different stuff in them (patterns, many different colors, lots of changes throughout the painting, intricate detail) had higher ratings than other paintings in both tasks, but when time was unlimited, those paintings were ranked a lot higher. This suggests the 4-second time limit muted some of the positive effect complexity can have on judgment of a work. In science-speak, we found that complexity was correlated with ranking in both tasks, but that the correlation was much stronger in the control task (Split Second task: cor = .25, p = .001; control task: cor = .49, p = .001).
• Paintings whose central four colors were very different from each other did worse in the unlimited time task (cor = -.38, p = .013).
• Finally, we found that paintings with big frames did much more poorly in the Split Second task than in the control task. This might not sound like an impressive finding at first, but it’s quite strong: the negative hit paintings with large frames took in the Split Second task (cor = -.48, p < .000001) was just as big as the boost complex paintings got in the unlimited time task (cor = .49, p = .001). This suggests thin-sliced judgments were strongly biased against paintings with big frames.

These results paint a complicated picture of thin-slicing. I think one of the big issues with Gladwell’s Blink is that it doesn’t really give a good idea of when thin-slicing makes sense and when it doesn’t. Thin-slicing is clearly a powerful, effective tool, but it privileges certain qualities over others. Hopefully, by studying these qualities, we can help figure out in what circumstances thin-slicing works best, and when thicker slices might do a better job.

The results summarized above suggest thin-slicing privileges images which are vibrant and clear. On a computer monitor, a large frame means a smaller central image, and high complexity makes paintings harder to understand quickly—but it seems wrong to suggest that the complexity or frame style of King Solomon and His Court are flaws which should cause its ranking to drop. Indeed, when participants were able to take their time, it rose to the top of the list. This suggests that, despite its overall effectiveness, thin-slicing doesn’t reliably engage with complexity, and this can cause us to overlook some gems.

We wanted to provide a way of reading Wikipedia that could be passed from person to person without anybody getting really lost. A big problem with mouse- and keyboard-based interactive kiosks is that sitting down at a computer can create a situation where one person is in charge of what happens and everybody else is just along for the ride. This is a serious problem when it comes to engaging groups of users; one can’t just pass a mouse and keyboard around from person to person. Hand-held touch devices like the iPad do a lot to get around this problem. They can move from person to person, and they make being a backseat driver a lot more fun. We settled on the idea of a sliding frame with buttons for each artist which, when tapped, would load the Wikipedia article for that artist in a content frame above.

To minimize distraction and maximize fun, we also decided we needed preserve the feeling of using a native iPad application. To this end, we built our first open source software release: BklynFlow. BklynFlow is a MooTools class for creating Coverflow-like user interfaces for the web. It’s easy to use (check out BklynFlow on GitHub for an example), and has has several features that we hope make it particularly appealing: thumbnails can have captions, it supports both touch and mouse interaction, and click/tap behavior isn’t prescribed ahead of time — a click or tap can call any JavaScript function.

BklynFlow makes use of hardware accelerated 3D transforms, so right now it only works in Safari and Mobile Safari. It was in large part inspired by Zflow. Please let us know what you think!

—

This post is part of a three-part series on Wikipop.