The ASK team’s new home. A dedicated info host is stationed with the team at all times to help visitors.

Since our team location testing was completed last summer, they have been working behind the scenes in a temporary space. We were a bit curious as to what this change would do to our traffic. After all, no matter where we put the team during our crazy summer of roaming desks, the app traffic didn’t fluctuate much. While Shelley and I always conceived of the project with a visible team, was it truly necessary? Well, we’ve had about 6 months to find out the answer: yes! Since we pulled the team off the floor, our app traffic has slowly and steadily declined.

As I explained in a previous post, the single biggest challenge with ASK is helping people understand what the app actually is. The fact that you can chat in real-time with real people during your visit just doesn’t compute. This concept is particularly challenging to grasp from just a sign or a palm card—even if we say it it in person (though an in-person explanation helps). We’re constantly fighting people’s expectations of what a museum app is, and the single clearest way to communicate the concept of ASK is by having the team visible.

We know from testing several team locations in the building last summer, that a space just inside the Great Hall is the best location. This puts the team just beyond the entry experience hub-bub and closer to the art. It also places them right in a main thoroughfare of the building (our internal nickname for this space is “42nd street,” if that tells you anything). Since their presence seems vital to helping people understand the app, we want people to walk past them. However, there are several ways to access galleries from the lobby, so funneling traffic-flow will be key.

Another lesson learned from last summer was the importance of having other staff available to “run interference” so the team can focus on answering the questions coming in via the app (as opposed to questions asked in person). We’ve stationed information hosts with the team and, much like our visitor liaisons last summer, their role is tell people about the app, provide assistance with downloading when needed, and speak with visitors so the team is able to focus on their chats without interruption. The host  also hands out palm cards as visitors go by. I’m already seeing this work: more than once, I’ve watched people read the card as they’re walking away, stop, and turn to look at the team. I can see comprehension dawn on their faces. The concept clicks for them because the team is right there. It’s early yet, but I’m hopeful because we’ve already seen our app traffic climb. 

Staff hand visitors a palm card advertising the app.

While we’re pretty confident that the team’s current location is good, we’ll be continuing to test marketing materials to determine what language and key phrases work the best. We’ll also be working with the info hosts to determine the best ways to approach visitors and explain the app, building on our learnings from last summer. We haven’t found solutions to all our challenges just yet, but we’re getting there. In the meantime, next time you’re at the Museum, come say “hi” to the ASK team!

A very typical day in the lobby. One of our Visitor Liaisons tries to help stem the tide of questions, but once one person is there asking…more follow.

Testing in the lobby proved to be an incredibly tough spot. In this location, the team was highly visible, but this visibility was confusing because visitors saw them as general information points. And the kind of information visitors were looking for included everything from, “Isn’t there a zoo around here?” (referring to the Prospect Park Zoo) to “I need to sign up for the Bernie Sander’s campaign.” There was so much of this questioning going on, in fact, that it became difficult for the team to actually work and, in some cases, there were delays answering questions coming in via the app because interactions were proving to be too distracting. It should also be said that this working environment also included plenty of noise.

Simply put, this location proved to be too early in a visitor’s trajectory for visitor to be aware that there is an app and who the ASK team is in relation to it. They need to hear about the app from the ticketing transaction and see the team as a second (or even third) point of contact for everything to really gel.

The sheer amount of traffic of pre-visit questions coming to the team necessitated a hack of our “staff workspace” signage. Normally, these signs are used only when desks are not occupied, but here the use has been adapted to help visitors identify what’s going on here.

These findings do not necessarily mean the ASK team won’t eventually end up in the lobby, but they do help us figure out what that presence would need to be more like in order to be more successful. A full marketing plan at the entry could help the awareness factor, so the team becomes a second point of contact even at this early stage. Also, a “glass box” with planned interaction time a la Southbank Center could also work in this location helping allow the team to get their work done. The planned interaction time would become key, though, in keeping with the project’s engagement goals (something Southbank did well through meetups and other scheduled interventions).

One big thing the lobby testing has taught us? Even with traffic patterns that now have much better clarity, the human presence is still something people really crave. We need to do some thinking here about the greeting process especially in light of how to work with our new information desk, which is part of the Situ Studio designed furniture set; our visitors services area is on this one.

We also tested team location in the galleries and some of the findings here have proven interesting. How close should the team be to works of art? How best to handle directional questions? When in the visit is the public most responsive to the team’s presence?—all of these questions are things we’ve been evaluating in this series of moves.

Testing in Connecting Cultures where the team was more embedded in and among the works of art helped show that proximity helped drive conversations about art.

The team was placed in our Connecting Cultures exhibition located on our first floor; this location is post-ticketing, but fairly early in a visit because this is considered an introduction to the Museum’s collection where some visitors begin their visit. Testing here was a little complicated due to construction in the area, which created a considerable amount of noise (the team requested ear plugs at one point). Construction also didn’t help us much because it closed off exits, so many visitors would get in the space and some of the questions they had for the team were directional along the lines of, “Now how do I get out of here?” Interestingly, we don’t get many of these directional queries when people are using the app itself and that’s great, but we ideally want the team in a location that can foster in-person conversations about art. This space proved interesting because once the team was embedded in the exhibition, the conversations about art were on the rise. In the data collected the construction seemed to cause an imbalance of directional questions, but this tide would likely be stemmed once the space was fully restored to its normal state.

Testing in our forth floor elevator lobby where the team presence is a more cohesive unit, there’s proximity to works of art, but the space is also transitional which has its own set of pluses and minuses.

Our next testing (going on now) has involved our elevator lobbies on the fourth and fifth floors. These are small spaces, so the team has a concentrated visible presence. These spaces are used for small exhibitions and/or have works installed, but they are also transitional in that most people passing through them are on their way somewhere. Both spaces are in a direct traffic line to special exhibitions. The fifth floor is unique in that most people start their visit on the fifth floor and start to work their way down the building, so the team in the fifth floor elevator lobby is earlier in a visit. The fourth floor elevator lobby is still in the traffic line, but more of a mid-way point in someone’s visit.

Fourth floor testing showed us that being in the middle of a visit pattern may be very beneficial. In this location, people seem more ready to talk about art and the team’s presence is more recognized because in-building marketing prior to this point helps with the connection. In one recent interaction, I watched as someone stepped off the elevator quickly making her way through the space. She spotted the team and you could see the lightbulb go off—”Oh, you’re the one answering questions in the app? The answers are so great. Thank you so much.” This is exactly the kind of thing we hope to see with the team being so accessible.

We’re still testing these areas more fully, but there are some things we know already that will help us in our quest to find an appropriate home for this team:

• Proximity to art helps drives art-related conversations.
• Discovery of the team mid-visit helps recognition.
• Transition spaces might be a good fit if the team is not overwhelmed with directional questions.
• Directional questions are an inevitable part of being on the floor, so being in a space where it’s easy to give instructions—Bathroom? ….Take the elevator down one flight. Basquiat exhibition? …Right down this hall.—helps put us in a position where we can at least quickly answer with minimal distraction.

During all of this testing, one thing has remained a constant. While the visibility of the ASK Team is important for the engagement goals of the program, their very presence does not seem to change our app’s usage numbers, so seeing the team at work does not necessarily help advertise the program.

As summer closes we’ve got a lot more to work with and we’ll begin some internal discussions about where this team might eventually land. This will, of course, involve many more factors because we have to take the learnings and align them with the most important thing of all—institutional goals.

ASK is a little bit more complicated than an average app launch; we reconfigured the whole entry experience because the ASK team—who work in full view of the public—need to be a part of it. Early testing has shown us that visitors are interested seeing the ASK team work and the knowledge there are real people answering questions has been very compelling. The ASK team working in full view of visitors seem to be vital to the project’s success and this means we’re launching two related, but separate components: an app and a visible human presence within the experience.

There are some days when Sara and I think movable furniture was both the best and worst idea we ever had. Even our director has been seen helping configure our furniture setup at times.

In our case, the moving parts are quite literal. The lobby now consists of movable furniture which will let us configure and reconfigure how the ASK team becomes part of this space. As we continue through the summer, you’ll find us testing the ASK team setup in various locations. When thinking about how to place the ASK team, we are considering both the visibility and working process of the team—are they visible, is the space too distracting to get work done, are visitors able to approach the team while also being mindful of the work they are doing? The physical space is a consideration—are there enough power outlets for their desks, does traffic flow work in our favor? There are also practical considerations, too, like how much we have to move furniture to make a location feasible.

Our original setup with the ASK team working in the glass pavilion didn’t work so well with visitors more interested in getting to ticketing than engaging with the team.

We spent two weeks testing in our glass pavilion. The team was located just before the brick piers and they were on view as people entered the building; power benches were also located in the same area. Visitors were more interested in getting inside the main lobby to get to ticketing and they would breeze by the team quickly often not reading signs or even noticing their presence.

Visitors loved the benches in the pavilion setup, but they felt disjointed from the ASK team.

In this location, visitors would sit on the benches to charge their devices, but often they were at the end of their visit past the point at which the app would be useful. Limited availability of power outlets in the pavilion also meant that the benches and the hubs—the ASK team desks—could not work together as visual components and the setup felt disjointed. The pavilion also proved to be further complicated by the sheer number of events in the space that required moving the furniture almost daily. In the end, this location proved to have little traction and the result was our lowest use rates since launch.

Our second location test began when we moved the team to the area just off the lobby beyond the “art” doors. As part of the ticketing process, admissions staff would ask visitors if they had an iphone and, if so, they were given a special admission tag. Our visitor liaisons would see the special tags as people entered the “art” doors and could begin the greeting process, introducing the team, and helping visitors get started with their download.

Our second round of testing moved the ASK team to an area just beyond the “art” doors. This worked well when traffic patterns were in our favor.

This setup worked exceedingly well for a couple of weeks because the natural traffic flow of the building worked in our favor. After most visitors received their ticket, they proceeded through the “art” doors to begin their visit where they would see the team and be greeted by our liaisons. However, once our special exhibitions—Sneakers and Faile—opened on July 10, many visitors began using the elevator on the opposite side of lobby, bypassing the team totally.

The setup just beyond the “art” doors allowed the ASK team a room of their own, which had its benefits.

Unlike the pavilion, the area off the lobby provided ample room for signage and integrated seating. Our visitor liaisons had a natural place to greet visitors as they came through the “art” doors. Even with our temporary set up, the area felt unified—it was clear this was a space dedicated to ASK where our staff were working and visitors were lounging. However, once the traffic flow changed, it was clear the benefits of the space couldn’t outweigh the lack of visibility.

Our next steps will be to move the team out into the lobby and into various galleries where their presence is feasible. On a logistical level, these locations are tough—that careful balance of team in the space, but not in the way…combined with endless moving of furniture controlled by the location of the nearest power outlet—makes for complicated puzzle. The trick is knowing just because the furniture is movable, doesn’t always mean it should move. During testing, we’ll need be disciplined to set things up and try a location fully before moving the parts around, so we can get an accurate read on what’s working (or not).

For our testing of the first response, we wanted to learn if an information-based or inquiry-based first response was most effective at engaging users with art and the app. From the standpoint of the user, what type of response would help the user understand the nature of the app experience, and look more closely at art or engage more deeply with the artwork? From the standpoint of the Team, what type of response was best at providing an immediate response, and what type would be most compelling to the user?

In our first round of tests we experimented with having the first response from the Team be strictly information.  The user received the opening prompt, “What work of art are you looking at?” and the Team responded with information only. Information-based responses varied, some provided information about a specific detail that the user would be able to observe on an object; other answers included broader information that provided geographic, historical, or other types of contextual information about the object. For example, here is an exchange the draws the user to look at specific details, and provides contextual information:

In testing information-based responses, we learned users appreciated information—it provided context and a broader view of the object they are looking at, and had the potential to support closer looking. We also learned that the right type of information helps to establish the authority of the Team. Users trusted the app experience more when they believed that the individuals responding to them were knowledgeable, and could offer information that they would not have access to just by looking at the work of art or reading the label.

Our second round of tests experimented using inquiry-based response, the question was simple, and we used the same wording consistently, “What drew you to that object?” A majority of users told us that they liked being posed a question, “I liked being asked the question, it made me look at it [the object] again,” and “I liked that I had to think.” Additionally, using this simple and proscribed first response had the advantage of providing an immediate response to the user, and also helped the Team member have additional time to gather information about the object.  While the user was crafting a response, the Team was able to collect information about the object using the Dashboard, and our collection of wikis.

In addition to providing time, using inquiry had the advantage requiring action on the part of the user, and functioned as tool to immediately engage the user with the work.  It addressed one of our big picture goals for the ASK app experience — provide visitors with an experience that has them engaged with works of art.  By asking, “What drew you to this object?” the Team was able to quickly gather some information about the user’s interest, and helped us engage the user on a more personal level, and generally led to a deeper discussion about the work of art.  By answering our question, users found themselves looking back at works of art more closely, and thinking more critically about the objects in the Museum.

With this in mind we will use inquiry as our first response moving forward, and also integrate it into the next version of the opening prompt.  In my next blog post I will discuss our process of crafting a new prompt, and what we learned from our second round of testing using inquiry.

The ASK app functions like a text message conversation between users and the Audience Engagement Team.  Users can send a text message or a photo. In our first few testing sessions we learned, very quickly, some basic rules which have remained constant in our two months of testing—in retrospect these basics are obvious—users wanted the experience to be similar to how they use text messages in their daily life, and they wanted the experience to feel personal:

• Users wanted to receive an immediate response after they sent their first message.
• Users preferred short messages in response, rather than a large blurb of text.  We could send the same information, we just needed to do it in bite size bits.
• Users enjoyed when the conversation had an informal tone to it as it helped establish that there was indeed a real person responding.
• Users appreciated receiving new information that they didn’t know, and they also appreciated when we revealed that we didn’t have an immediate answer to their questions—it actually helped to create more trust from the users—as per above, it helped to establish a sense of familiarity and a personal conversation.

Our original prompt.

Using this basic information as a starting point we set out to deconstruct our text message conversations, focusing specifically on the first message within the text message exchange.  We wanted to learn how users would respond to our “opening prompt” (the first message that the user receives when opening the ASK app).

The opening prompt that the app presents to the user has a huge responsibility. We learned from early testing that users did not want to read lengthy directions or go through a multistep “onboarding” process.  With this in mind we knew that the prompt needed to be short, and needed to get the user actually using the app immediately.  We created a prompt that was short, directed, and began with art: “What work of art are you looking at right now?”

Through our testing sessions, we wanted to know if the opening prompt was effective in quickly generating a conversation between the visitor and ASK team. From the user’s standpoint, what will get people interested and using the app quickly? From the Team’s standpoint, what will provide the best starting point for conversation?

Prompt was changed to elicit more deliberate action on the part of the user, a prompt that would require to the user to not just immediately engage with the app, but also immediately engage with the art in the Museum in a thoughtful manner.

Data from post-testing feedback sessions (group conversations with testers), and information gathered from surveys brought us to the conclusion that the opening prompt was successful in getting users to use the app because it was easy to respond to and testers began using the app immediately. However, while the prompt was easy to respond to, testers were confused as to what would happen next. Additionally, we’d see users arbitrarily choose an artwork to send and that was frequently the first work of art they saw, and not necessarily an object they were interested in.

Based on this information we knew that we needed a prompt that, like this one, motivated testers to begin using the app immediately.  The prompt needed to be equally directed, but somehow provide the user with an idea of what the app experience would be, and have the user motivated to want to continue the conversation.  We decided that the prompt needed to elicit more deliberate action on the part of the user, a prompt that would require to the user to not just immediately engage with the app, but also immediately engage with the art in the Museum in a thoughtful manner.  This led us to our new prompt, “Find a work of art that intrigues you.  Send us a photo.”

It immediately proved to be a positive change.  As with the previous prompt, users engaged with the app immediately, and in addition, they remarked on how the prompt initiated them to start looking at the art more closely, to really consider what work of piqued their curiosity and interested them.  Some users continued to note some confusion as to what the full app experience was “supposed” to be.  However, we received at least half of the number of these types of comments as compared to when users tested with the first prompt.

We will continue to use this new prompt, and experiment with ways in which the Team is following up to users first message.  I will discuss the process of finding the best type of first response in my next blog post.

Learning to use the dashboard prior to user testing.

One of my first experiences in the position was to participate in a round of user testing—the largest thus far.  It was intense, to say the least.  We had thirty-five individuals (from a range of backgrounds) over the course of three hours in our American Identities exhibition asking questions, and sharing their thoughts on objects.  Our Chief Curator, Kevin Stayton, was there to answer questions, and Marina Kliger (our new Curatorial Liaison for the project), and I were there to run the dashboard—typing Kevin’s responses to the users, and providing reinforcement by responding to some of the user’s inquiries, and on-the-fly research when necessary.  At the end of the three hours our collective heart-rates must have been alarming.

Earlier rounds of user testing used this prompt which felt too automated to users and proved a barrier to their participation.

The first hour was especially stressful.  In the first fifteen minutes we received around one question every 30 – 60 seconds.  It started with users sending snapshots or titles to artworks in response to the app’s initial prompt, “What work of art are you looking at right now?” The intent of this prompt was to immediately engage the user with the Museum’s collection (and the app)  and for us (Kevin, Marina, and I) to follow with a question or information about the object to instigate close looking, or further inquiry about the object on behalf of the user.

As mentioned throughout the blog, the development of this app is being executed through an iterative process.  The team had learned during previous evaluations that users found this prompt to be automated, and felt that they needed to craft a smart question, which limited the amount of engagement on behalf of the users.  Considering that the prompt created a barrier for the users, we wanted to present the user with an initial prompt that would invite immediate participation.  As one of the intended outcomes of this project is to foster dialogue about the Museum’s collection, we decided to begin with the collection.

Shifting to this prompt in the latest round immediately engaged users and gives us flexibility in how we respond.

Changing the prompt proved effective in getting the users to use the app immediately.  We had twenty individuals registered for the first hour (6-7pm) of testing, and everyone showed up right on time.  As we hoped, the prompt instigated immediate engagement with the app once the users entered the galleries.  We hadn’t anticipated, however, the stressful situation of receiving a deluge of inquiries at once.  Fortunately, we were able to temper the deluge by staggering entry into the galleries for the next two rounds of registrants.  We know that we won’t have control of how many visitors will be using the app once it’s live, so we will have to continue to refine the prompt over the coming months to encourage participation, but in a way that’s manageable on the backend.

Our second intent for the new prompt was to encourage further inquiry on behalf of the user by sharing information that could hopefully spark curiosity about the objects and collection.  We found this to be true in some of our conversations.  For example, the first snapshots that we received was of “Winter Scene in Brooklyn.”  We received it and asked each other, “What can we say about this that will get them (the visitor) curious, or have them look more closely?”  The object is rich with details—the groups of men in conversation, the man carrying the bundle of wood,  the various storefronts—each provide us with a glimpse into the daily life and labor force of early 19th century Brooklyn.

Francis Guy (American, 1760-1820). Winter Scene in Brooklyn, ca. 1819-1820. Oil on canvas, 58 3/8 x 74 9/16 in. (148.2 x 189.4 cm). Brooklyn Museum, Transferred from the Brooklyn Institute of Arts and Sciences to the Brooklyn Museum, 97.13

We had to decide—in a flash—how we were going to engage the visitor with the painting.  As we were deciding on the response,  a flood of other snapshots and object titles inundated the dashboard—we had to get our first response out so we could attend to other visitor’s who were already waiting.  As time was a constraint, we responded first with a general background, “The picture is one of the richest one’s for content and stories,” hoping that this would serve as a teaser for the visitor to look for some of the stories, and content and follow with questions.  Which they did (!), their next question was,  “I’m curious which portion of Downtown Brooklyn depicts.” Kevin knew the answer immediately, and we responded to the visitor, “This is the area near the Fulton Ferry, low on the horizon, rather than on the hills of the Heights,” and then a few moments later we added, “but none of the buildings in this picture survive.”  The visitor again responded with a “thanks,” and “That’s development for you.”

This snippet of one our first conversations from our night of user testing reflects what we’re hoping the Audience Engagement Team Members will be able to accomplish: provide accurate information at a rapid fire pace, framed in a way that instigates closer looking, in a manner that is conversational and hopefully opens further dialogue.

A tester during our last testing session. Engagement through the app encourages closer looking.

We are now in the process of hiring the six individuals who will make up that team.  Having the user testing just before the hiring process has provided us with a great insight into what we’re looking for in the Team Members.  As I mentioned above, they will need to provide information on the fly, which means that we are looking for individuals with a breadth of art historical knowledge, as well as the ability to do background research under the pressure of time constraints (within minutes!).  The level of pressure that we felt with such an incredibly tight time constraint was not something I had anticipated before the user testing—which is great to know when hiring.  The ability to stay calm, and personable in a stressful situation will be essential for individuals on the team.  In addition to having grace under pressure, and a breadth of knowledge and curiosity to learn more, we’re looking for individuals who are also curious about people, and engaging them with art objects through thoughtful conversation, and the sharing of information.

I envision working with the team as a cohort of individuals who are learning and experimenting together to finding the best ways engage our Museum visitors to the collection using the ASK app. If you know anyone who would like to join the team, send them our way!

And because people are at the heart of ASK, we knew it was essential for us to find the right ones to help us make our project a success. Shelley and I are basically focused on the proverbial forest that is this initiative and we needed to add team members to tend the trees. We needed two (we eventually determined) staff members who would focus on the day-to-day operations of building and executing the experience—hire and train the team of people to answer visitor questions, get the database of content going, help us figure out what the visitor/staff interaction should be like, etc.

Monica Marino, left, and Marina K, right, running the dashboard during a recent user testing session. Our Chief Curator, Kevin Stayton, helps answer questions.

The first position we hired for is the Audience Engagement Team Lead, whose main role is to hire, train, and manage the team of people who answer incoming visitor queries. For this position, we needed an experienced museum professional with a strong art history background who was comfortable working with our encyclopedic collection, had experience training others, and had both passion and experience facilitating visitor engagement around works of art. In addition to hiring, training, and managing the team, the Team Lead will be the eyes and ears on the floor as we roll out the project, helping to determine what our visitors need to make this program a success and how we might meet those needs including reporting on the success of the ASK app user experience and make recommendations for potential updates after launch. We were extremely lucky to have Monica Marino, former Senior Museum Educator and Intern Coordinator here, return to us from the Met where she was Assistant Museum Educator, Internships and Academic Programs for past four years. We are currently in the midst of the hiring process for her team (apply here) and she will share her approach to selecting that team in a future post.

The second position we hired is the Curatorial Liaison, whose main role is to act as a bridge between curatorial and the Audience Engagement Team. We needed one person to spearhead the monumental effort to gather all the knowledge in the building from curators, educators, and other staff and synthesize and share it in a format that the rest of the team can access. The Liaison will also be responsible for staff training and support related to the dashboard (the internal tool used to answer visitor questions that come in via the ASK app) and will help to determine what staff need to make this program a success and how we might meet those needs. Since the bulk of the work for this position is pre-launch and just after, we made this role temporary so that we are able to assess the needs once the app has been on the floor for about 6 months. The position may stay as-is, morph, or we may find we no longer need it. Happily, this temporality suits Marina Kliger. Marina is a PhD candidate at the Institute of Fine Arts focusing on early 19th-century French art and a former research associate at the Art Institute of Chicago. She and Monica are currently working feverishly to gather knowledge and determine the best platform for hosting it. Marina will post later about their selection.

Shelley and I also talked a bit about what department these new staff members should join. Should they be with Shelley in Tech or with me in Audience Engagement/Interpretive Materials? It’s an interesting question because really it could be either. They way we work here, tech and interpretation often go hand-in-hand and Shelley and I work together a lot. However, two major aspects of the project helped us decide to put the team with me, one: the fact that the project emphasizes people as opposed to tech (audience engagement as opposed to digital engagement, splitting hairs a little bit, but there it is) and two: the information we learn from ASK about what visitors want to know can greatly inform our interpretive efforts throughout the Museum. This is one of the aspects of this initiatives that excites me the most. We’ll be gathering metrics on what areas and works are the most talked about. Trends in that data can help us spot areas that might need more interpretive attention, for example. We’ll be able to learn more information about how visitors use (or not) our existing interpretation and what kinds of information they want than ever before.

So, how do you hire for a project that’s running on an iterative process? Perhaps not surprisingly, using a kind of iterative process. We ended up posting the positions several times with description updates as we determined the desired skill set and were better able to define the positions. The tight timeline of this project means we needed self-starters who can get the big picture quickly and help us dive into the details. And, as is the nature of an iterative project, there are still a lot of questions and we needed people who want to help us find answers. Ultimately, we built the job descriptions around the (mostly immediate) needs as we understood them and included a lot of flexibility. We also tried to be very clear about what we didn’t know and where we were hoping the selected candidate would help flesh out his or her own role. Thankfully both Monica and Marina were game.