For our testing of the first response, we wanted to learn if an information-based or inquiry-based first response was most effective at engaging users with art and the app. From the standpoint of the user, what type of response would help the user understand the nature of the app experience, and look more closely at art or engage more deeply with the artwork? From the standpoint of the Team, what type of response was best at providing an immediate response, and what type would be most compelling to the user?

In our first round of tests we experimented with having the first response from the Team be strictly information.  The user received the opening prompt, “What work of art are you looking at?” and the Team responded with information only. Information-based responses varied, some provided information about a specific detail that the user would be able to observe on an object; other answers included broader information that provided geographic, historical, or other types of contextual information about the object. For example, here is an exchange the draws the user to look at specific details, and provides contextual information:

In testing information-based responses, we learned users appreciated information—it provided context and a broader view of the object they are looking at, and had the potential to support closer looking. We also learned that the right type of information helps to establish the authority of the Team. Users trusted the app experience more when they believed that the individuals responding to them were knowledgeable, and could offer information that they would not have access to just by looking at the work of art or reading the label.

Our second round of tests experimented using inquiry-based response, the question was simple, and we used the same wording consistently, “What drew you to that object?” A majority of users told us that they liked being posed a question, “I liked being asked the question, it made me look at it [the object] again,” and “I liked that I had to think.” Additionally, using this simple and proscribed first response had the advantage of providing an immediate response to the user, and also helped the Team member have additional time to gather information about the object.  While the user was crafting a response, the Team was able to collect information about the object using the Dashboard, and our collection of wikis.

In addition to providing time, using inquiry had the advantage requiring action on the part of the user, and functioned as tool to immediately engage the user with the work.  It addressed one of our big picture goals for the ASK app experience — provide visitors with an experience that has them engaged with works of art.  By asking, “What drew you to this object?” the Team was able to quickly gather some information about the user’s interest, and helped us engage the user on a more personal level, and generally led to a deeper discussion about the work of art.  By answering our question, users found themselves looking back at works of art more closely, and thinking more critically about the objects in the Museum.

With this in mind we will use inquiry as our first response moving forward, and also integrate it into the next version of the opening prompt.  In my next blog post I will discuss our process of crafting a new prompt, and what we learned from our second round of testing using inquiry.

The ASK app functions like a text message conversation between users and the Audience Engagement Team.  Users can send a text message or a photo. In our first few testing sessions we learned, very quickly, some basic rules which have remained constant in our two months of testing—in retrospect these basics are obvious—users wanted the experience to be similar to how they use text messages in their daily life, and they wanted the experience to feel personal:

• Users wanted to receive an immediate response after they sent their first message.
• Users preferred short messages in response, rather than a large blurb of text.  We could send the same information, we just needed to do it in bite size bits.
• Users enjoyed when the conversation had an informal tone to it as it helped establish that there was indeed a real person responding.
• Users appreciated receiving new information that they didn’t know, and they also appreciated when we revealed that we didn’t have an immediate answer to their questions—it actually helped to create more trust from the users—as per above, it helped to establish a sense of familiarity and a personal conversation.

Our original prompt.

Using this basic information as a starting point we set out to deconstruct our text message conversations, focusing specifically on the first message within the text message exchange.  We wanted to learn how users would respond to our “opening prompt” (the first message that the user receives when opening the ASK app).

The opening prompt that the app presents to the user has a huge responsibility. We learned from early testing that users did not want to read lengthy directions or go through a multistep “onboarding” process.  With this in mind we knew that the prompt needed to be short, and needed to get the user actually using the app immediately.  We created a prompt that was short, directed, and began with art: “What work of art are you looking at right now?”

Through our testing sessions, we wanted to know if the opening prompt was effective in quickly generating a conversation between the visitor and ASK team. From the user’s standpoint, what will get people interested and using the app quickly? From the Team’s standpoint, what will provide the best starting point for conversation?

Prompt was changed to elicit more deliberate action on the part of the user, a prompt that would require to the user to not just immediately engage with the app, but also immediately engage with the art in the Museum in a thoughtful manner.

Data from post-testing feedback sessions (group conversations with testers), and information gathered from surveys brought us to the conclusion that the opening prompt was successful in getting users to use the app because it was easy to respond to and testers began using the app immediately. However, while the prompt was easy to respond to, testers were confused as to what would happen next. Additionally, we’d see users arbitrarily choose an artwork to send and that was frequently the first work of art they saw, and not necessarily an object they were interested in.

Based on this information we knew that we needed a prompt that, like this one, motivated testers to begin using the app immediately.  The prompt needed to be equally directed, but somehow provide the user with an idea of what the app experience would be, and have the user motivated to want to continue the conversation.  We decided that the prompt needed to elicit more deliberate action on the part of the user, a prompt that would require to the user to not just immediately engage with the app, but also immediately engage with the art in the Museum in a thoughtful manner.  This led us to our new prompt, “Find a work of art that intrigues you.  Send us a photo.”

It immediately proved to be a positive change.  As with the previous prompt, users engaged with the app immediately, and in addition, they remarked on how the prompt initiated them to start looking at the art more closely, to really consider what work of piqued their curiosity and interested them.  Some users continued to note some confusion as to what the full app experience was “supposed” to be.  However, we received at least half of the number of these types of comments as compared to when users tested with the first prompt.

We will continue to use this new prompt, and experiment with ways in which the Team is following up to users first message.  I will discuss the process of finding the best type of first response in my next blog post.

Ed Bleiberg, curator of Egyptian Art, describing one of our signature objects, the Statuette of Queen Ankhnes-meyre II and her Son, Pepy II.

The tour was amazing. We had the opportunity to look closely with Ed at ten key objects on display. One of these objects was, “Statuette of Queen Ankhnes-meyre II and her Son, Pepy II.” He described the significance of the iconography and its influence in the history of art and later religious iconography. The six year-old King is seated on his mother’s lap, an iconography we can see in later works depicting the Virgin and Christ child in Christian objects. By looking closely with Ed, and having the opportunity to ask questions, we left with insight into the symbolic significance of the statue, its material and where it may have come from, an understanding of kings in ancient Egypt, and stylistic choices in ancient Egyptian art—for example, we asked about the elongated toes and fingers. The answer provided us with the skills to look more closely at other objects in the collection—great attention to detail was often employed when rendering hands and toes in Egyptian art, in some works you can even see the details of the figure’s cuticles; this same attention to detail was not placed, for example on the figure’s facial features. While this may seem like a minor detail, it really does change with way that you look at other objects on your own.

The Audience Engagement team working in the Egyptian galleries to write wiki content.

Following our talk with Ed, each team member chose one of the ten objects that Ed shared with us to research more in depth, and write object wikis. The wikis, as Marina mentioned in an earlier post, will provide the team with necessary information when they are engaging with the public through ASK. Over the course of training, and during ASK’s soft launch, the team will continue to write wikis for objects throughout the collection. To be sure that the wikis are resources that will provide the information that we need when manning the dashboard we also practiced using the wikis in an ASK practice session.

Alisa Besher using the app to ask questions during a team training session.

Practicing with ASK with the team in the galleries for the first time, did feel a lot like playing, in the best possible way. The feeling of excitement, nerves, fun, and fear that I had when I played capture-the-flag, and ghost-in-the-graveyard when I was a kid, were the same feeling that I and the team felt when we tried had our first ASK practice session. Two people from the team manned the dashboard and answered questions about the objects for which they prepared wikis, and the rest of the team went through the galleries and asked about those and related objects.

Megan Mastrobattista (left) and Katie Apsey (right) taking the controls of the dashboard during audience engagement team training sessions.

As was the case in a real testing environment, when manning the dashboard we felt the pressure of responding instantly and comprehensively, in an engaging and friendly way. It gave us a sense of the type of pressure that we’ll be under when ASK is live, but, the experience also gave us a glimpse of how much fun it will be. Because the team was being posed with inquiries about objects that they had just studied, they had information and ways in which to provide immediate responses, and different ways to engage those asking about the objects. Those answering questions found it exciting to have to think—at lightning speed—about the best way to share information; and those asking the questions were excited to get responses and gain new understanding.

This isn’t to say that everything was perfect. Each of us on the ASK team feel a deep sense of responsibility to the objects, the curators, and scholarship around the collections. There were plenty of inquiries that we didn’t have responses to, and even those that we did, in the back of our heads was the nagging thought voiced during the session, “Everything I write makes me nervous, there is no room to give wrong information, everything must be absolutely correct.” Fortunately, we are only into week one of training. Over the next two months the team will be working together with each other, and colleagues across the Museum to learn and study.  Including asking all the questions we have to be sure that we get it right.

I delineate here the type of information that the ASK team will have because it is this type of information that makes this app more than just a “human Google.”  Anyone can Google a question, and look up information—what ASK is allowing our visitor to do is to connect with a person who has a nuanced understanding of the works of art, AND an understanding of the different ways in which people interact with art.

As part of training, our Audience Engagement team is walking through the galleries with each collection curator. Here they are getting a tour through American Art with Terry Carbone.

With all of this in mind, how do these six individual humans engage museum visitors with 5,000 years of art? How can the team prepare to be at-the-ready to answer questions and engage in dialogue thoughtfully about any object in the collection at any given moment? It is a daunting task indeed!

To best address this challenge, we have decided that each individual team member will have a “major” and “minor” collection area of focus, and of course, each will have an understanding the many different ways in which museum goers engage with art.

Nancy Rosoff, Andrew W. Mellon Curator of the Arts of the Americas, works with the team to take a closer look at our Life-Death Figure.

To begin our work together we’ve started learning about the full collection in tandem with experimenting with the app. Although everyone will have two collection areas on which they are focusing, it is important that everyone has a broad understanding and familiarity with the full collection so that we can make connections across collection areas (and if we’re overloaded with a high volume of inquiries, we’ll be prepared to respond to some queries that are outside our focus areas). Over the course of training and our soft launch the full team will meet with all of the curators, write one comprehensive wiki for each collection area, write 7-9 object wikis in their respective “major” collection areas of focus, and practice manning ASK’s dashboard as much as possible.

Our Audience Engagement team from left to right: Stephanie Cunningham, Alisa Besher, Megan Mastrobattista, Sara Garzon, Jessica Murphy, Katie Apsey

Katie Apsey is returning to the Brooklyn Museum after completing coursework for a Ph.D. Katie’s previous positions in the Museum included working as a Curatorial Assistant in the Arts of Asia, Africa, the Pacific Islands, and the Islamic World, and as an Intern Educator in the Education Division.

Alisa Besher is returning to the Brooklyn Museum after years of working as an educator, visual and performing arts programmer, and administrator, at the Museum of the Moving Image, CEC ArtsLink, The Center for the Humanities at the CUNY Graduate Center; and worked 3 Operas by Robert Ashley at the Whitney Biennial as a performer, and Fundraising and Event Coordinator.  Alisa’s prior position at the Brooklyn Museum was an Intern Educator in the Education Division.

Stephanie Cunningham is returning to the Brooklyn Museum after completing her M.A. in Art History and years of experience as an adjunct professor, and working as an educator and intern within education departments at museums including the New-York Historical Society, the Studio Museum in Harlem, and the Weeksville Heritage Center.  Stephanie’s previous positions here at the Museum included working as intern in the Education Division, and as a curatorial intern in Asian Art.

Sara Garzon comes to us having recently completed a full-time curatorial internship in the Modern and Contemporary Art Department at the Metropolitan Museum of Art where she worked with curatorial staff, and taught the Museum’s public.  Sara is currently completing her M.A. in Art History, and will be pursuing doctoral studies in the fall.

Megan Mastrobattista comes to us with experience working as an intern and administrator at the Metropolitan Museum of Art in their Education and Asian Art Departments, and at the David Zwirner Gallery. Megan comes with experience teaching in Encyclopedic collections.

Jessica Murphy comes to us with a Ph.D. in Art History and years of experience as a Research Associate and contractual Educator at the Metropolitan Museum of Art working in the American Wing and the Department of Modern and Contemporary Art, and as a Curatorial Assistant at the Philadelphia Museum of Art.

One of the great things about working with a team of people is the collective knowledge, energy, and support that everyone brings together. Obviously we are excited about the individual strengths that everyone is bringing, but we’re also excited by the collaborative nature of the team, and everyone’s true commitment to working together, and taking this project on as a challenge that we will work to collectively address throughout training, and once we launch in June.

Learning to use the dashboard prior to user testing.

One of my first experiences in the position was to participate in a round of user testing—the largest thus far.  It was intense, to say the least.  We had thirty-five individuals (from a range of backgrounds) over the course of three hours in our American Identities exhibition asking questions, and sharing their thoughts on objects.  Our Chief Curator, Kevin Stayton, was there to answer questions, and Marina Kliger (our new Curatorial Liaison for the project), and I were there to run the dashboard—typing Kevin’s responses to the users, and providing reinforcement by responding to some of the user’s inquiries, and on-the-fly research when necessary.  At the end of the three hours our collective heart-rates must have been alarming.

Earlier rounds of user testing used this prompt which felt too automated to users and proved a barrier to their participation.

The first hour was especially stressful.  In the first fifteen minutes we received around one question every 30 – 60 seconds.  It started with users sending snapshots or titles to artworks in response to the app’s initial prompt, “What work of art are you looking at right now?” The intent of this prompt was to immediately engage the user with the Museum’s collection (and the app)  and for us (Kevin, Marina, and I) to follow with a question or information about the object to instigate close looking, or further inquiry about the object on behalf of the user.

As mentioned throughout the blog, the development of this app is being executed through an iterative process.  The team had learned during previous evaluations that users found this prompt to be automated, and felt that they needed to craft a smart question, which limited the amount of engagement on behalf of the users.  Considering that the prompt created a barrier for the users, we wanted to present the user with an initial prompt that would invite immediate participation.  As one of the intended outcomes of this project is to foster dialogue about the Museum’s collection, we decided to begin with the collection.

Shifting to this prompt in the latest round immediately engaged users and gives us flexibility in how we respond.

Changing the prompt proved effective in getting the users to use the app immediately.  We had twenty individuals registered for the first hour (6-7pm) of testing, and everyone showed up right on time.  As we hoped, the prompt instigated immediate engagement with the app once the users entered the galleries.  We hadn’t anticipated, however, the stressful situation of receiving a deluge of inquiries at once.  Fortunately, we were able to temper the deluge by staggering entry into the galleries for the next two rounds of registrants.  We know that we won’t have control of how many visitors will be using the app once it’s live, so we will have to continue to refine the prompt over the coming months to encourage participation, but in a way that’s manageable on the backend.

Our second intent for the new prompt was to encourage further inquiry on behalf of the user by sharing information that could hopefully spark curiosity about the objects and collection.  We found this to be true in some of our conversations.  For example, the first snapshots that we received was of “Winter Scene in Brooklyn.”  We received it and asked each other, “What can we say about this that will get them (the visitor) curious, or have them look more closely?”  The object is rich with details—the groups of men in conversation, the man carrying the bundle of wood,  the various storefronts—each provide us with a glimpse into the daily life and labor force of early 19th century Brooklyn.

Francis Guy (American, 1760-1820). Winter Scene in Brooklyn, ca. 1819-1820. Oil on canvas, 58 3/8 x 74 9/16 in. (148.2 x 189.4 cm). Brooklyn Museum, Transferred from the Brooklyn Institute of Arts and Sciences to the Brooklyn Museum, 97.13

We had to decide—in a flash—how we were going to engage the visitor with the painting.  As we were deciding on the response,  a flood of other snapshots and object titles inundated the dashboard—we had to get our first response out so we could attend to other visitor’s who were already waiting.  As time was a constraint, we responded first with a general background, “The picture is one of the richest one’s for content and stories,” hoping that this would serve as a teaser for the visitor to look for some of the stories, and content and follow with questions.  Which they did (!), their next question was,  “I’m curious which portion of Downtown Brooklyn depicts.” Kevin knew the answer immediately, and we responded to the visitor, “This is the area near the Fulton Ferry, low on the horizon, rather than on the hills of the Heights,” and then a few moments later we added, “but none of the buildings in this picture survive.”  The visitor again responded with a “thanks,” and “That’s development for you.”

This snippet of one our first conversations from our night of user testing reflects what we’re hoping the Audience Engagement Team Members will be able to accomplish: provide accurate information at a rapid fire pace, framed in a way that instigates closer looking, in a manner that is conversational and hopefully opens further dialogue.

A tester during our last testing session. Engagement through the app encourages closer looking.

We are now in the process of hiring the six individuals who will make up that team.  Having the user testing just before the hiring process has provided us with a great insight into what we’re looking for in the Team Members.  As I mentioned above, they will need to provide information on the fly, which means that we are looking for individuals with a breadth of art historical knowledge, as well as the ability to do background research under the pressure of time constraints (within minutes!).  The level of pressure that we felt with such an incredibly tight time constraint was not something I had anticipated before the user testing—which is great to know when hiring.  The ability to stay calm, and personable in a stressful situation will be essential for individuals on the team.  In addition to having grace under pressure, and a breadth of knowledge and curiosity to learn more, we’re looking for individuals who are also curious about people, and engaging them with art objects through thoughtful conversation, and the sharing of information.

I envision working with the team as a cohort of individuals who are learning and experimenting together to finding the best ways engage our Museum visitors to the collection using the ASK app. If you know anyone who would like to join the team, send them our way!