Metrics can tell a story, but they are often not enough.

One of our windows into visitor behavior is metrics. In the case of our ipad kiosks, up until now at least, this is a pretty small window. The metrics we were able to gather were limited to Google Analytic tracking and YouTube’s own analytics package. Taking last season’s Ai Weiwei and Swoon kiosks as an example, we know how many questions were asked and how many times someone pressed play on the video prompt or the response videos. What we don’t know is if a person was pressing play in the gallery or online because the videos were hosted on YouTube and then embedded into the ASK interface. In addition to being shown in the gallery on the iPad, the video prompt and responses are also available on the exhibition’s webpage. YouTube statistics tell us how many times a video was watched, but we don’t know if this is an online on in-gallery viewing. And of course metrics also can’t tell us how many videos one person may have watched or if the people posing questions are the same as those reviewing answers.

Observation showed us that visitors often use the kiosks in social groups.

To begin to get answers to these questions, our Digital Community Liaison, Brooke Baldeschwiler, spent time in the galleries observing and speaking with visitors in the Ai Weiwei and Swoon exhibitions. From this, we were able to determine that there are two fairly distinct user groups—the question posers and the response reviewers. In fact, observation was showing us that most visitors used the kiosks to review answers. Reasons for this varied and included not having time, not having a question, or seeing the answer to their question already posted. Brooke also determined that multiple visitors use the kiosks at once. This falls right in line with visitor behavior in general; we have a social bunch here (most of our visitors come in pairs and small groups) and they like to experience the museum together. Looking at these stats and their limitations and armed with a better understanding of use through observation and interviews, we have updated our approach for the ASK kiosks this exhibition season in a number of key ways.

The iPads remain the place to pose your question, but we’ve added a companion touchscreen that will show the answered questions.

First, we divided the experience into ask and answers. The iPads remain the place to pose your question, but we’ve added a companion touchscreen that will show the answered questions. This not only caters to the two user groups and social use we’ve identified through observation, but also allows us to get more specific metrics because the new design allows us the ability to track when an answer is read.  We can now get a greater sense of the overall participation from askers to readers. Additionally, putting the answers on a larger screen is a very preliminary form of prototyping for future digital signage, which we plan to eventually deploy as part of this project. We’re not really sure what kinds of content will encourage people to approach and touch and this will give us some early insight into those motivations. We’ll be showing answers in both video and text format and can begin to see what visuals and what content draw people in.

In our new setup the ipad is where you ask your question, but we’ve paired with a larger touchscreen which displays the answers.

This season we’re also tracking the play button so we’ll be able to see how many people watch the videos and if that play button was pressed in-gallery or online. We’ll be able to see how much of each video they watch, which will help us get a feel for good video length. Of course, we’ll still be tracking how many people leave questions and be able to compare that to the number of people watching the video, so we’ll know how many answered our call to action.

Observation has shown us that those who ask and those who read are different audiences; the larger screen of answers helps facilitate the readers.

However, metrics still won’t tell us who is pressing play. We’ll be doing more observation to learn that. Observation will also give us an idea of the stay rate at the kiosks, how much content users delve into, and what general user experience is like. Is the function of each component (iPad and touchscreen) clear? Easy to navigate? Is the content compelling? We’ll also do some interviews to help answer those questions. That’s really one of my personal key learnings from this project—to get the real story, you have to pair metrics with observation and audience evaluation. We’ll continue to use both as the project unfolds.

For our spring 2014 exhibitions, ipad kiosks started with a video call to action under the heading, “In Conversation.”

One example of this is our comment book kiosks, which you may remember started as generic electronic versions of comment books and then were tweaked into “In Conversation” kiosks—all running on ipads throughout our exhibitions.  When we switched to the newer conversational format, the big change was a video call to action on the start screen.  Nicely, the video format meant we could do anything in that initial call, so we started our experiment with content. As part of our spring 2014 exhibitions, we shifted the focus of those videos to “ask us a question” and both of our featured artists, Swoon and Ai Weiwei, answered visitor questions that were left on the ipads.

In our latest pilot, the same ipads have a re-branded start screen to match our ASK program for Bloomberg Connects.

For fall, we are continuing the “ask us a question” format, but we are extending it into certain areas of the permanent collection and, additionally, changing the design of the start screen to reflect the greater ASK program.  The bonus?  If this works, it can provide us with a way to make ASK functionality available to those who do not have (or do not want to use) a mobile device. Given these kiosks may become part of the greater Bloomberg Connects program we are thinking about a lot of issues as we move through this pilot.

At first glance, this seems easy, right?  Change the start screen and the call to action and done(!), but there’s a lot more nuance that goes into a change like this one. Unlike our future mobile setup, these kiosks don’t operate in real time; the visitor gets their response via email after their visit, so we need to think about messaging and visitor expectations.  Additionally, not everyone who leaves a question on one of these kiosks receives an answer—not the case in mobile.

There’s a big difference in responses given depending on if the device is fixed vs. mobile.

We are also thinking a lot about differences in contributed content and how different an “ask” can be from a fixed device to a mobile one. These ipads are in a fixed position in the galleries, so the questions that we get through them are slightly different. So far, we’ve seen questions from the kiosks either more general in nature—about the overall themes in the show and/or about works of art installed very near where the kiosk is mounted. Testing on mobile has, so far, seen questions that tend to be much more specific and granular—about single works of art and about any work of art in the gallery because the device goes with the user.  Does the resulting content from two slightly different iterations mesh together well enough?

From an administrative perspective—vetting questions, providing responses, managing users—we wonder if this system needs to be integrated into the ASK management and, if so, what that would look like.  There are benefits to integration because it could help a single dedicated staff manage all ask/answer content in the building using a single administrative setup, but there’s a large technical overhead in merging the two systems.

So, there’s a lot of complexity and this pilot is an important one because it allows us to tweak an existing setup and learn from it before we build and/or combine programs. Sara is up next to talk about some additional changes we are making to these kiosks based on visitor observation in those spring exhibitions.

For the most part these worked to serve the purpose and we didn’t think much about changing them, but this year we started to wonder if there would be value in upping the game a bit. As they stood, they were the electronic version of a one way comment drop box.  Should they become a more personal experience and one that leans toward a two way dialog? Could we elevate the discussion from a simple “leave us your thoughts” to more directed questions that might provoke deeper engagement?  Could that same format help give visitors access to someone—a curator, an educator, an artist, a conservator—to help get their questions answered?

An iPad in LaToya Ruby Frazier uses a video prompt of the artist asking visitors for their questions.

Starting with a recent crop of exhibitions—GO, El Anatsui, Sargent, LaToya Ruby Frazier—Sara and I worked with curators and educators to begin experimenting with the format and comment kiosks were transformed into a more dynamic experience by using video to ask visitors directed questions and put a more personal spin on these devices.  LaToya Ruby Frazier says “no question is too big or too small,” while Radiah Harper asks how El Anatsui has “left a charge on you.”  Comments are threaded, have a popularity ranking (“I have this question, too”), and email a visitor when a response is posted to their query.  Just as before, appropriate staff are emailed digests and comments and conversations sit on the appropriate exhibition pages of our website.

Pause Play Play Prev | Next

The responses from visitors have been thoughtful and insightful; the questions we are seeing asked when we give access to an artist or conservator have been inspiring.  Sara and I estimate that we’ve gone from a ratio of 30% of insightful comments on the older kiosks to a 70% rate on the new ones; there’s been a lot of dramatic change as a result of just thinking (and designing) these a little differently.  Comments now go very far beyond the  “awesome exhibition” that we were seeing in the older stations.

Right now, you have the unique opportunity to see both the older kiosks and the new in place throughout the building because we are just starting to work with curators to figure out how this new format might serve our permanent collections as we evaluate these current changes.  Sara has implemented a visitor experience study around them and we’ve got a lot of metrics to eventually share, so look for a future post from her about even more specifics.

This is a first step in re-thinking about responsiveness and our visitors will likely be seeing a lot of tweaks and improvements to these new comment stations as we move forward.

In the new realism that typified American art of the decade, liberated modern bodies resonate with classical ideals, the teeming modern city is rendered empty and silent, and still life is pared to an essentialized clarity.

In creating an in-gallery interactive, the challenge was finding an activity that would highlight the disparity between what we’ve come to associate with decade and the idealized vision created by its artists.

What did the Jazz Age look like? Interactive asks visitors to make their own selection from an array of popular photographs to see how it compares to the imagery created by the American artists featured in Youth and Beauty.

The resulting interactive uses the Google Images API as a way to show what’s in the popular imagination of four themes related to the show. A visitor searches for imagery on a theme and is asked to select an image from Google’s results; the selected image is displayed along side a related work from the exhibition and the interactive explores how the popular imagery delivered via Google differs from the artists’ depiction.

Youth and Beauty interactive utilizes the Google Images API and runs on iPads embedded into a popular culture timeline.

Given this is a live search, the results are not always perfectly accurate to the time period, but they are pretty close.  We’ve also tweaked it a bit to help the results gain a little more accuracy; turning on Google’s “safe search” and displaying only black and white imagery. The interactive runs on four iPads in the gallery where the devices are embedded into a popular culture timeline in the exhibition. You can also play with it on the web.

In The Moving Garden, Lee Mingwei asks visitors who take a flower to give it to a stranger.

One of the great things about working with living artists is the chance to work with them when they bring projects into the building. When I first heard about this piece, I was struck by what could happen between strangers in the exchange, so Eugenie and I asked the artist if he would let us create something that would allow visitors to document their gift giving.  He felt that the mystery of giving the gift was central to the piece, but he was also curious about exchanges and thought we could try it as long as we made it clear that the documentation was an optional step in the process, not a requirement to take part.

I gave my flower to a stranger at the corner of Washington Ave and Lincoln Place. It was an experience I'm not likely to forget.

With that, #mygardengift was born.  It’s a simple interactive that we hope extends the life of the project outside our walls.  Visitors are invited to document their exchanges by tagging on Flickr, Twitter and Instagr.am.  In addition, for the first time, we are using SMS text messaging in an interactive.  Visitors can text us about the exchange and we use the Twilio API to map their responses and bring them into the interactive. There’s a page on the website that shows all the responses and we also use an iPad to display the exchanges in the gallery.

This is really the kind of project that we want to be using social media for—working directly with an artist to show a community’s experience around a work. Given the four platforms that we are using, I’m curious to see which ones get used the most and how the information coming to us may differ on each.  Mostly, though, I’m excited to see our community participate and to watch the mystery unfold in some of the exchanges and I can’t wait to talk to the artist to see his own response to this part of the project as it grows.

If you come to Lee Mingwei: “The Moving Garden”, take a flower and then use #mygardengift to document your exchange.

Visitors using the iPads in the exhibition.

Overall, use of the iPads among visitors went pretty well.  We know from our statistics that 8,629 avatars were issued to visitors via the quiz kiosks, which means roughly 28% of the people coming to the show took part in the activity. Of those surveyed by Roslyn, 100% found the iPads inviting and easy to use; 82% felt the iPads provided the right kind of information, kept their attention and didn’t distract from the artwork; 64% felt the iPads provided the right amount of information; 73% felt the interactive made them think more about the artwork and was helpful to their experience of the art; 91% felt use of the iPads was a positive experience. Of those surveyed, visitors were slightly more likely to use the iPad kiosk if they were familiar with the device, but the data does not indicate a strong relationship between prior experience with the device and actual use or non-use of the iPad in this setting.  Interestingly, users of the iPad kiosks were likely to have also used other interpretive materials, indicating that this was a supplementary experience to our labels and didactics.

Even though those metrics give an overall picture that is pretty good, there are other findings we should talk about.

Visitors take a quiz to determine which avatar will follow them through the exhibition. Kiosks are located outside the exhibition space just off the elevators.

By design, we created a linear experience in which, ideally, visitors would take a quiz to determine which avatar should accompany them throughout the exhibition and then check in at other kiosks to either find their avatar in works of art or vote for their favorite depictions of their avatar in the show.

In early discussions, we decided to try something new and experiment by placing the quiz kiosks right outside the show so visitors would see them as they got off the elevator prior to entering the exhibition space.  The reasons for this were two-fold.  First, we wanted to see if this location would capture more people before they went in.  Second, we were conscious of the cultural issues with the subject matter and wanted to ensure this was seen as an optional path through the show, not a required one.

11% of visitors using the introductory "quiz" kiosk did so on the way out of the exhibition instead of the way in.

The reality is more than a few visitors missed the quiz kiosks in this experimental location because they were seen as outside the exhibition space and because of this, many visitors encountered the interactive at a later point having not taken the initial quiz.  While you could still take part without having taken this first step, the subsequent sets of kiosks were designed in a linear fashion assuming you had taken the quiz and you had a specific avatar you were trying to find.  We’ve learned a couple of things for the next time around:  1) Those first kiosks really need to be in the gallery, not outside of it.  2)  Even if we move the kiosks to a better location, we are never going to ensure that every visitor funnels through space the same way, so we need to design the interactive in a much less linear fashion or provide clearer instructions for visitors who missed the first set of kiosks.

Switching to a clear bezel helped expose the hardware. Compare to the images above where it's difficult to tell an iPad is contained in the case.

Another issue cropped up when we mounted the iPads.  Unlike our Wikipop iPad interactive for Seductive Subversion where the iPads were freestanding and could be picked up, this time we used LaunchPad mounts from Sprocket. Sprocket produces very elegant and sturdy mounts, but we didn’t expect that they would turn the iPad into a very normal-looking small touch screen because the iPad was so well hidden within the beautiful case.  Issues cropped up almost immediately. As visitors started interacting with the devices, they were treating them like a standard touch screen—we were seeing a lot of hard tapping, jabbing or trying to use fingernails to navigate—the iPads, which are used to a much more low-key touch experience, became unresponsive.  We worked with Sprocket to resolve this and they provided a clear bezel for the LaunchPad mounts, so visitors could see the hardware.  We’ll never forget going down to switch out the bezels and one visitor standing nearby said, “Oh, that’s an iPad!”  The switch to the clear bezel helped and now whenever we are using mounted iPads in the gallery, we’ll be using these to expose the hardware a bit better.

I’m not going to go too far into the actual findings around the content on the devices because without having taken part it’s a little hard to explain all the specifics.  That said, there’s an important statistic worth noting: only 64% felt the iPads provided the right amount of information.  These were activity based kiosks focused on helping visitors find avatars in the works of art and while visitors did report the iPad activity made them look closer at the works in question (yay, limited screen suck!), visitors still wanted more information. Interview participants showed a strong preference for content that would provide additional information and indicated they expected the iPads would contain didactic content and looked to the devices for it.  This is especially interesting given the findings on the Wikipop project—more, may indeed, be more.

There’s a lot of research that I can’t possibly fit into this post, but if you have questions ask in the comments and I’ll be happy to give you more.

More often than not, visitors were browsing the iPads in groups.

I’ll mention there is a lot of possible error with those numbers. For starters, we were seeing that the iPads were overwhelmingly social devices with often more than one person using them at a time, so while we can get a rough idea of sessions, it’s not indicative of how many visitors actually used them or how many of the same visitors may have used more than one device throughout their visit.  In addition, there were some anomalies in the stats that had us questioning how we were capturing some of the metrics.  That said, it seems better to release what we know and ask you to take these metrics with a grain of salt as we dive deeper into this post.

Map of the gallery showing iPad placement.

In the exhibition layout, we had three iPads installed throughout the gallery near the works of art to be used while standing in the space.  There were also two iPads installed in a seating area at the end of the exhibition.  We were curious about possible differences in seating vs. standing metrics, so we were tracking stats accordingly.  Units that were placed in the galleries to be utilized standing were overall more popular than the units placed in a seating area at the end of the exhibition.  On average, wall units were used for ten minutes with visitors viewing 11.18 wiki articles, while units near seating were used for eight minutes with visitors taking a look at 9.55 wiki articles—again, that’s for the average session with the possibility of multiple visitors per session.  We had expected more use at the seated units, so these figures surprised us a bit.  I don’t want to jump to too many conclusions remembering that the very end of an exhibition is less trafficked than beginning/middle and it’s possible these numbers wash out in the end, but it may point to visitors wanting the resource near the works of art.

The coverflow app that Beau developed worked very well.  Stats indicated that visitors were traversing the entire length of the 26 artists.  The artist names were presented alphabetically in the coverflow, but visitors didn’t just click on names earlier in the alphabet, they swiped and clicked on names all over the menu.  Interestingly, visitors looked at almost the same artists no matter if they were seated or standing.  When you look at the top ten in each category, the lists are almost the identical with the order changing slightly:

top 10 at end of exhibition with access to seating—

Dorothy_Grebenak, Dorothy_Iannone, Chryssa, Jann_Haworth, Vija_Celmins, Marjorie_Strider, Pauline_Boty, Idelle_Weber, Letty_Eisenhauer, Kay_Kurt

top 10 in gallery standing—

Dorothy_Grebenak*, Marjorie_Strider, Jann_Haworth, Dorothy_Iannone, Idelle_Weber*, Pauline_Boty, Chryssa*, Vija_Celmins, Rosalyn_Drexler*, Marisol_Escobar*

In the standing top 10, I’ve included asterisks that indicate artists with work in the show physically near an iPad, possibly changing the stats.  Devices were also near objects by Lee Lozano and Kay Kurt, but those two artists did not make it into the standing top 10 list.  I’ll note at this point that choice of thumbnail easily skewed stats and, it seems, sex and money still sell in the end—take a look at some of the images used in the coverflow and then revisit the top 10 lists:

On average visitors were looking at 10.66 wiki articles, but given we allowed access to the entire English Wikipedia what were visitors looking at? Across the board, the top 26 articles were exclusively our featured artists; people stayed within our exhibition framework for most of the time. Beyond that, articles on pop art, psychedelic art and Sgt. Pepper’s Lonely Hearts Club Band were universally popular no matter if visitors were seated or standing. (Jann Haworth, one of the artists featured in the exhibition, designed the album cover along with Peter Blake and the link to Sgt. Pepper is in the first sentence of her wiki article, so this makes sense especially given she was in the top 10 on both lists.) What’s more fascinating is to keep looking….visitors who were standing seemed intrigued by articles about other museums. More often that not, a standing visitor would click on articles about Met, Whitney, MoMA, Gugg, New Museum and the list goes on and on to institutions far and wide. (Seriously, other museums should have paid us for in-gallery advertising—it was just that noticeable.) At the seated units, visitors were more likely to browse deeper in subject matter: sexual revolution, pop artist, painting, happenings, WWII, sculptor, surrealism, artist names, etc. Even though our 26 artists were universally popular, we saw lots and lots of statistics pointing to visitors going down the wiki rabbit hole given the opportunity to do so.

Psst...It's okay to pick up the iPad.

After opening, it became clear that visitors were apprehensive about picking up the device.  Even though we copied the Apple store’s display which allows people to pick them up, we found most visitors were using it on the stand instead of cradling the device. After adding some signage guards reported more visitors picking them up, but it’s interesting to remember that in a museum setting old habits die hard and even with iPads people were cautious to touch too much.  At this point, I’ll also note that we were using the same alarm system Apple uses in their stores and we never had a visitor attempt to walk away with an iPad.  We did have the alarm go off once and we all got a good laugh when we discovered it happened during the press preview as one of the reporters got a little over curious about the setup.

Overall, this project worked well on many levels.  A high percentage of visitors utilized the devices for long periods of time going pretty deep into the wiki catalog, but also staying focused on exhibition content. Given most people come to museums with other people, the iPads turned out to be a social device which engaged people in a way that seemed natural to their visit.   The information in the wiki articles on these 26 artists is now out in the world via Wikipedia and will contribute to information sharing beyond our exhibition.  This leaves us likely to do it again at the next opportunity.

We wanted to provide a way of reading Wikipedia that could be passed from person to person without anybody getting really lost. A big problem with mouse- and keyboard-based interactive kiosks is that sitting down at a computer can create a situation where one person is in charge of what happens and everybody else is just along for the ride. This is a serious problem when it comes to engaging groups of users; one can’t just pass a mouse and keyboard around from person to person. Hand-held touch devices like the iPad do a lot to get around this problem. They can move from person to person, and they make being a backseat driver a lot more fun. We settled on the idea of a sliding frame with buttons for each artist which, when tapped, would load the Wikipedia article for that artist in a content frame above.

To minimize distraction and maximize fun, we also decided we needed preserve the feeling of using a native iPad application. To this end, we built our first open source software release: BklynFlow. BklynFlow is a MooTools class for creating Coverflow-like user interfaces for the web. It’s easy to use (check out BklynFlow on GitHub for an example), and has has several features that we hope make it particularly appealing: thumbnails can have captions, it supports both touch and mouse interaction, and click/tap behavior isn’t prescribed ahead of time — a click or tap can call any JavaScript function.

BklynFlow makes use of hardware accelerated 3D transforms, so right now it only works in Safari and Mobile Safari. It was in large part inspired by Zflow. Please let us know what you think!

—

This post is part of a three-part series on Wikipop.

An initial search on Wikipedia revealed that only 14 of the 25 artists featured in Seductive Subversion had existing Wikipedia pages. Of those pages, at least three qualified as “stubs” (short article in need of expansion).  The remaining 11 artists had no Wikipedia presence at all, except Barbro Östlihn, about whom there is a small paragraph featured on Swedish Wikipedia.

I certainly wasn’t expecting to find Wikipedia entries for all the artists in Seductive Subversion. After all, a good number of them, such as Mara McAfee, Dorothy Grebenak, and Kay Kurt, have been virtually forgotten over the years.  But I simply couldn’t believe that many celebrated artists, including May Stevens, Dorothy Iannone, and Lee Lozano, had no Wikipedia presence whatsoever, while Pauline Boty, Britain’s reigning “Queen of Pop,” had one paltry paragraph dedicated to her brief but stellar life.

So I knew I had my work cut out for me. Over the summer and early fall I created and expanded pages for the artists who needed them most.  In so doing, I learned a great deal about their lives. Who would have guessed, looking at Evelyne Axell’s psychedelic nudes, that she had learned to paint by taking private lessons with René Magritte?  Or that Rosalyn Drexler, in addition to being a Pop artist, was also an award-winning playwright and one-time Mexican wrestler? The more I learned—of Letty Eisenhauer’s rousing performances at early Happenings, Boty’s friendship with Bob Dylan, McAfee’s hilarious illustrations for National Lampoon—the happier I was to know that the biographies of these remarkable women would soon be widely available.

Of course, getting all these great anecdotes to appear on Wikipedia presented somewhat of a challenge.  After a few meetings with Shelley, and with the patient help of several Wikipedians over live chat and page discussions, I mastered the basics of WikiMedia editing.  I learned how to create sections within articles, make bulleted lists, insert block quotes, and, most fun of all, hyperlink to other Wikipedia articles.

Creating hyperlinks led to a fair amount of insight into the Pop Art landscape on Wikipedia.  It was interesting to see which personalities of the 1960s art world were well represented, and which were not.  I was hard pressed to find a male Pop artist who didn’t have a Wikipedia page.  Even the gallerists who represented them, men like Leo Castelli, Sidney Janis, and Arne Glimcher, merited their own articles. Meanwhile Jill Kornblee, a New York City art dealer who represented women artists like Drexler and Kurt in the early 1960s when male gallerists simply would not, remains without a page. Even the Wikipedia entry for Pop Art, which traces the style’s evolution in five different countries and tells of myriad male artists’ accomplishments, makes only passing reference to two women artists—Niki de Saint Phalle and Marisol.  On this page, I added several more women to the list of “Notable artists” included towards the end, a small and admittedly insufficient remedy for the glaring omissions in the text above.

The artists featured in Seductive Subversion deserve to be better integrated into the narrative of Pop Art, in text books, on museum walls, and, yes, even on Wikipedia.  What I’ve done is simply lay the groundwork for their presence on this popular site, in the hopes of generating deeper interest in their lives in work amongst visitors to our exhibition and the general public alike. The pages featured on the iPads in our galleries, like all Wikipedia pages, are continually being updated.  Already Wikipedians have begun contributing to the pages I created just a few weeks ago.

I encourage you all visit these articles, but more than that, I hope you will join us in the project of revising Wikipedia to be ever-more inclusive and mindful of its lacunae.

—

This post is part of a three-part series on Wikipop.