We began with a very short list of options in a drop down menu that included the special exhibitions, a few specific collection areas, and the collection more generally. We quickly found the need to add a few more options. For example, the admissions team asked for a “just in the neighborhood” option as it’s a common response to the question (though the data shows it’s not as common as they likely felt it was).

The survey appears in a pop-up window and has a drop-down menu of options. Unfortunately, the option to skip or cancel is baked-in; we can’t make this a required question to complete the transaction.

Results for the first two months are interesting. In July, the permanent collection ranked highest in response rate, while for August it was our Pierre Cardin special exhibition. The initial lack of options is one of the reasons for the high “other” response rate in July, which dipped the second month as more options were added. Currently, we have 16 options plus skip/decline. This feels like a lot, but maybe it’s ok. In particular, I wonder about including Korean art and African art in the list at the moment since both are temporarily off-view, but it would help us track an uptick once those collections are on view once more. We also have to remember to update the list regularly as special exhibitions open and close. For example, both Liz Johnson Artur and One: Egúngún exhibitions closed mid-August, which explains the dip in responses.

A quick comparison of the total number of survey responses (which should be every transaction) to total number of visitors who were required to visit the admissions desk shows the transaction count is about 60-65% of the visitor count. Multiple tickets can be purchased through a single transaction—and we know most of our visitors come in pairs or groups—and that feels close to the right percentage. I think we are still getting more cancellations than we should and we’ll keep working on it. The admissions team is meant to pose the question in a casual and conversational manner so it doesn’t feel like a survey (or an interrogation!) and select a response in order to proceed with the sale, although it is possible to cancel and move on. To avoid cancellation, we included a skip/decline option. Unfortunately, not everyone is consistently asking the survey question, which we know because we can run reports on who is logging which responses. For example, we found one person mostly just cancelled the survey in the first week and were able to speak with them. While we don’t want survey completion rate to become punitive, we do want to encourage completion because the information is important for us as an institution. Finding that balance can be tricky.

After two months, we are still working out the kinks, mostly in terms of making this process habit for the admissions team. A next step is to work with our Tech team to create a report that would knit together the survey answer, ticket info, and zip code from each transaction so we can compare the data set as a whole. That would be a pretty powerful triumvirate. 

This was the message on palm cards that our ASK Ambassadors distributed to Museum visitors throughout June. As a special engagement activity for Pride Month, visitors could take an ASK-guided tour of our galleries and learn more about gender and queer identity in art. 

The card featured a detail of a work by Amaryllis DeJesus Moleski, on view in the exhibition “Nobody Promised You Tomorrow.”

This tour could be taken as a complementary activity to the special exhibition Nobody Promised You Tomorrow: Art 50 Years After Stonewall or as a standalone activity. And, as with all our ASK engagement offerings, we kept things responsive and personalized —every visitor could set their own pace and tone.

Visitors could begin their experience in the Museum lobby at a painting by Kehinde Wiley.

As we envisioned it, this app-guided tour would include a few very popular works from our collections (like Kehinde Wiley’s Napoleon Crossing the Alps) as well as some lesser-known works. They could be works by artists who identified as LGBTQ+, portraits of LGBTQ+ individuals, or works that touched on broader themes of gender identity.

The ASK Team collaborated to select ten works of art with a range of dates and media, from Donald Moffett’s Lot 043017 (Multiflora, Radiant Blue) to a coffin in the Ancient Egyptian collection, from Aaron Ben-Shmuel’s stone bust of Walt Whitman to Deborah Kass’s neon wall-piece After Louise Bourgeois. They compiled information about these works into a reference document and they strategized about giving directions to help the visitor navigate from stop to stop.

Elizabeth of the ASK Team tracked these tours (which accounted for about 22% of our app traffic) throughout June , and she noticed an interesting split. Visitors who began engaging with us on the Museum’s first floor were more likely to invest in the total tour experience, following our cues to visit works on the third, fourth, and fifth floors of the Museum. They often spent more than a half-hour with us for this itinerary.

Special labels with Pride flag icons were placed beside the “tour stops.”

Meanwhile, other visitors encountered individual works with our ASK Pride Month labels in the galleries and sent questions about them. These visitors were usually satisfied with learning about that particular work and might move one more stop nearby when we invited them to continue chatting. However, they were less interested in experiencing the complete tour.

The ASK Team also received a few requests from visitors who were ready to go even further. For example, when one visitor asked whether they could see anything by LGBTQ+ artists in the new special exhibition Rembrandt to Picasso: Five Centuries of European Works on Paper, we added a drawing by Rosa Bonheur to our list.

It’s been two years since we first tailored an ASK activity to a specific show or event, during Georgia O’Keeffe: Living Modern, and we continue to learn from each iteration. Next up? An engagement option related to the special exhibition Pierre Cardin: Future Fashion. More about that soon!

A visitor study of “Frida Kahlo” was a perfect project for some of my Pratt grad students.

In my last post, I posited that although we don’t have a CRM, we are gathering data in the ways we can to help inform our decisions. One recent example are the two evaluation projects completed by Pratt graduate students in the Museums and Digital Culture program last semester. I had eight students in the Audience Research & Evaluation course, so I split them into two groups of four for final projects. One team was responsible for completing a visitor study about ASK users and the other about visitors to Frida Kahlo. For this post, I’ll focus on the latter. The student project team for the Kahlo evaluation was Sydney Stewart, Megan Paqua, Grace Poole, and Caroline Baer. The results I’m sharing here are their interpretation of the data (reviewed by me) and their work (including the visuals).

The objective of the study, as defined by our institutional needs and finalized by project team, was to determine: who is coming to Frida Kahlo and will they return to the Museum? In addition, we wanted to know if the Spanish language materials were useful. The first part was really to help us check our assumption that Frida Kahlo might bring in a lot of first-time visitors. The second was to give us some insight on the ROI of the translation work we did for this exhibition, which was the most extensive we’ve ever done—ticketing pages, signs in the building, floor plan, and labels were all provided in both English and Spanish. While we’ve provided label copy in other languages in the past, we had yet to translate logistical information beyond the floor plan. We have a small team and none of our editors are native Spanish speakers. Best practice requires someone for translation as well as proofreading and layout review. Thankfully we have several fluent Spanish speakers (some of whom are native speakers) who helped in the review process and even translated our last minute signs. (Thank you Stephanie, Rachel, Jaime, and Allison!) Working with Eriksen Translations and our in-house powerhouse, we were able to get the translation completed, but it took a lot of extra time and resources.

We determined that the best approach to getting answers to our research objectives were interviews with Kahlo visitors. While we had visitor time and attention, we also asked about how they heard about the exhibition and how their experiences was. All eight students, myself, and Rachel Lewis (who conducted interviews in Spanish for us, thank you!) conducted interviews in the exhibition shop, which was exit point of the exhibition. We offered 10% shop coupons as an incentive to take the survey.

Results

We interviewed visitors during class time on four Thursday afternoons from about 3:30pm-5:30pm in March-April 2019. In total we completed 259 interviews. Below are the results, as written in the students’ final report (in quotes):

Who came to Frida Kahlo?

“Visitors came from a wide geographic area. Visitors to Frida Kahlo were almost evenly split between tourists and New York City residents. The majority of New York City residents were from Brooklyn, but the show also attracted visitors across the city. Overall, domestic visitors came from 26 states plus Puerto Rico, and international visitors came from 8 different countries.” Side note: the data is skewed since it only accounts for responses during a certain time window: Thursday afternoons. Responses would likely be different on different days. Weekends, for example, likely bring in more locals on their day off.

Visitorship was almost evenly split between locals and tourists, which may be accounted for by the fact that interviews took place Thursday afternoons only.

Did they use the Spanish language materials?

“Visitors responded positively to the inclusion of Spanish language materials, particularly the exhibition labels, whether or not they used them personally, and indicated that continued inclusion of bilingual materials would make them more likely to make a return visit to the museum.

Visitors frequently remarked on the relevance of the Spanish language materials for an exhibition about Frida Kahlo. Many visitors expressed an interest in expanding foreign language offerings in future exhibitions when appropriate, especially for exhibitions featuring artist from non-English speaking communities.”

The majority of visitors interviewed did not use the Spanish language materials.

How did they hear about the exhibition?

“35% of visitors reported learning about Frida Kahlo through word of mouth, although social media posts as well as reviews in major news publications were the next most effective at reaching visitors. On-site signage was also useful for attracting visitors who had not heard about the exhibition prior to their arrival at the museum.”

Word-of-mouth ranked highest among ways people learned about the exhibition.

How was their experience? Will they come back to the Museum?

“No matter where visitors are coming from, Frida Kahlo was the primary motivation for visitors to come to the Brooklyn Museum, whether it was out of personal interest, to visit with a friend or family member, or based on a recommendation or advertisement. The majority of visitors expressed having positive opinions about the exhibition and their visit overall, even when they had complaints about other aspects of their experience.

Visitors had a clear interest in returning to the museum for special exhibitions, especially if the focus is on diverse communities or artists of color. Many visitors stated that they associated the Brooklyn Museum with this type of programming and were more positively disposed towards the institution as a result.”

For 71% of repeat visitors, “Frida Kahlo” specifically and special exhibitions in general are the reason to return to the Museum.

What would visitors like to see more of at the Museum?

“When asked what they would like to see more of at the museum, 10.71% of visitors responded with requests for more activities designed for children. These visitors were overwhelmingly returning visitors, indicating that this is a current gap in the Museum’s programming within the exhibition space.”

Learnings

This evaluation confirmed some of what we already knew:

• Special exhibitions are a big draw for first time and repeat visitors alike.
• Word-of-mouth is important, though it’s sometimes tough to parse out the nuances of this answer. While some people genuinely couldn’t remember exactly where they heard about it, for others there might be more to the story. For example, their friend told them about it either through social media or because that friend saw it on social media. So does that count as word-of-mouth or social media? Or both? More to play with here to get better data to help our marketing team.

And gave us some new insights:

• The majority of visitors felt that the Spanish materials contributed positively to their experience whether they used them or not.
• There’s a desire for more activities for children specifically in the exhibition galleries. As reported: “Of the individuals who responded to what they would like to see more of at the museum, 10.71% indicated a desire for more kid-friendly activities. The kid-friendly advocating visitors are also primarily returning visitors (88.9% who desire kid friendly activities).”

Those interviewed felt that the Spanish materials were positive and useful whether they speak Spanish or not.

The results of this evaluation have been really useful as we think about planning our upcoming exhibitions. In particular the usefulness of languages has come up a few times as we plan for our Pierre Cardin: Future Fashion exhibition opening next month. Do we translate to French, the designer’s native tongue? Ultimately we decided not to go that route, but mostly because we don’t think we’ll have high attendance of Francophiles (since Cardin has his own museum in Paris).

In addition to providing some good information about visitors to Frida Kahlo, the evaluation also gave us good practice as crafting questions and things to think about in terms of how to tease out nuances of how people hear about the exhibitions. As always, it’s a work-in-progress.

Without a unified CRM, the data landscape can look a little bleak.
Stephen McMillan (American, born 1949). Zabriskie Point, 1976, 1976. Aquatint on paper, sheet: 22 1/8 x 29 3/4 in. (56.2 x 75.6 cm). Brooklyn Museum, Gift of ADI Gallery, 77.152.2. © artist or artist’s estate (Photo: Image courtesy of Stephen McMillan, CUR.77.152.2_StephenMcmillan_photograph.jpg)

Despite our lack of CRM, we’ve been working hard to gather data in ways we can. For example, we started a single question survey as part of our admissions process—What brought you to the Museum today?—that our front-of-house team is meant to ask as part of a casual and welcoming conversation. That, along with the zip codes we gather as part of every onsite transaction (a requirement of city funding is to report on zip codes), helps us begin to get some basic information about who is coming and why.

On top of regular focused data points like zip codes and single question surveys, we will be running an annual visitor study. We recently partnered with SightX for our research needs and have launched a baseline visitor study to get a better idea of visitor museum-going behavior and demographics. We haven’t run such a survey consistently for 3 years or more. I’m really looking forward to having that data ongoing. We’ve tried to be thoughtful in not only what questions we pose, but also how we phrase them.

In addition to our own visitor study efforts, one amazing resource we’ve been able to tap to build our data stores is our relationship with Pratt Institute School of Information and their Museums and Digital Culture graduate program in particular. In addition to our ongoing fellowship and Sydney’s recent work on ASK data, I’m able to work with the students I teach (and other professors’ classes) on projects that both give the students real-world experience and help the Museum—a true win-win. This past semester, I taught Audience Research & Evaluation. Through class work, students ran one evaluation related to the Frida Kahlo exhibition and another on our ASK app. I’ll report on the findings of these studies in a future post.

So data gathering is happening, albeit piecemeal sometimes, and we’re trying to smart about knitting it into processes and moments that make sense such as with the admissions process, or a short, three-question survey with our thank-you emails to online ticket holders. What does our data landscape really look like and what can we do with the data we do have?  I’m happy to say, that’s where a dedicated data analyst comes in. Thanks to Bloomberg Philanthropies’ ongoing support, we are able to contract a data analyst (job description forthcoming) to help us begin to make sense of all these data points, with ASK data as a pivotal element in the data landscape. I’m really looking forward to carving out time to get the big picture of all the ways we gather data and streamlining those moments. If you know of any good people, please send them our way. In the meantime, check back as we share the results of our recent visitor studies.

Next came standardizing the themes so I could code them and compare them across collections.

This chart serves as basic data viz for themes across collections. Note that these themes are my interpretation from the questions visitors asked and could vary based on what another researcher codes the questions as.

Several universal themes across collections came out of this analysis. I personally find ‘damages/missing parts’ one of the most fascinating findings. If a work of art has something that appears to be missing from it, intentional or not, visitors will likely ask about it. Other themes across most of the different collection areas include:

• Creation techniques/materials
• Symbolism
• Purpose/function
• Curatorial decisions
• Significance/meaning

The coding system of themes, definitions, and collection areas. Note ECANEA stands for Egyptian, Classical and Ancient Near Eastern Art.

While this thematic breakdown does provide interesting insight into what visitors want to know, I still wanted to delve deeper into user behavior with the app, especially in regards to the complete trajectory of a visit. These themes are based on only a snippet of entire conversation. What could we learn if we looked at that entire conversation? Unfortunately, the dashboard won’t export entire conversations yet, so I had to pause this line of inquiry. But I picked up another thread and began to follow it. More on that next week!

Don’t get me wrong, we still care about use rate and want as many visitors as possible to use ASK, but we are no longer consumed by that metric. We have kept the tactics that work best. For example, our ASK Ambassador program is going strong and still makes the greatest difference: there is a direct correlation between Ambassador staffing and use rate. We’ve seen this repeatedly since launching the Ambassador program in 2017. We’ve also continued to play with engagement via ASK, particularly in relation to major special exhibitions, which—let’s face it—is why a majority of visitors come to the Museum. For the David Bowie is exhibition, we created a special trivia activity that was so popular we could barely keep up. Jessica will share more about that in a future post. She’ll also share about what we are currently doing in relation to Frida Kahlo: Appearances Can Be Deceiving, on view until May 12.

The ASK Ambassadors wear branded t-shirts and hats (hats optional).

In addition to these initiatives, I’m delighted to say we’ve added a Pratt fellow to our team this year who has been focusing on the data. Sydney Stewart is a second year graduate student in the Museums and Digital Culture program in the School of Information at Pratt Institute. (Full-disclosure, I teach for that program and Sydney is one of my stellar students). It’s amazing what we have been able to learn by having one person focus on the data, and I’ve invited Sydney to share her research and results here in the coming weeks.  

When it comes to truly getting into the ASK data, I find we’re constantly bumping up against what happens when you build minimal viable product (MVP) as part of an agile process: short-sightedness. We purposefully weren’t thinking long-term when developing the dashboard (the interface the team uses to answer questions and process chats), only what we needed in the moment. Because we were building MVP,  we didn’t plan for or build ways to access the larger data set. We only created tools for our initial needs, which were basic metrics and ways to share conversations with curators for fact-checking purposes. Now that we are trying to get a better handle on the scope and possibilities of the data, we are having to look into building tools to access it. For now, the Tech team runs reports for us when we know what to ask for. I suppose you could say we’re having to get agile once again by using Sydney’s research path as a way to help us understand what we actually want and need to know from the data set. Unfortunately, that makes it a little difficult for her as there is a delay between her determination of needed data and our ability to give her that data. Fortunately, she’s been more than up to the task, and she’ll share some of her creative workarounds and what she’s been able to do with existing metrics.

Our period of radio silence is over, so stay tuned!

Five star app reviews fueled our increased use rate desire. With all these great reviews, surely more visitors would use the app.

Turns out, not that many more people are using it. After a great deal of testing and improved marketing efforts with insights from an outside evaluator, we have managed to double our use rate from 1% to just over 2% pretty consistently. That’s with a lot of effort on our part: incentives and contests, staff hired specifically to promote the app, and marketing materials including palm cards and object labels. When the stars align and our team is really on fire, we’ve seen over 3%, but that is likely the best we are going to get with what we’ve got. And you know what? Maybe that’s OK.

I’ve been doing a lot of soul searching about this project, about what success should look like, and about the original goals for ASK. And to me, the raw numbers are becoming less important. We created ASK in order to facilitate engagement with art. Do I wish more people used it? Sure. But those that do get something really special out of it. That might just be enough. Interestingly, when you compare ASK to other engagement strategies we’ve offered in the past, it fits within a norm. For example, our audio guides (offered until 2012) and our In Conversation kiosks also netted about 2% use rate.

So where do we go from here?

It’s time to shift our focus away from “more” and towards one of the most unique aspects of ASK: the data. We have over 11,000 chats just waiting to be examined. We have metrics not only on app use, but also most-asked-about artworks, information related to where and how many places people ask us questions, and more. We can determine the kinds of questions people have about works of art, how they think about or look at artworks, how they experience the museum, and more. All of this data would be of interest to researchers of all kinds: educators, art historians, and technologists. It’s certainly of interest to us.

The ASK dashboard tracks certain metrics already, but we have yet to delve into the 11,000+ chats we’ve had with users.

That is where we are headed. ASK is no longer about “more,” but instead about identifying the unique insights we can gain about how visitors view and understand works of art. As Shelley stated in an early post about measuring success: “Three components help us determine the health of the ASK: engagement goals, use rates, and (eventually) institutional knowledge gained from the incoming data.” We’ve nailed the engagement goals. Use rate is the best we can get with the tools at our disposal. It’s time to shift to the final measurement: institutional knowledge.

The next step is to reach out to our colleagues across various departments—curatorial, education, visitor services—and determine what questions they have. We’ll then add those to our list of questions and begin to narrow down a research focus. As we delve into the data, we will be sharing insights along with way. I can’t wait to see what we learn.

One question that came up again and again was: how much are we guiding our visitors, and how much are we letting them steer the tour themselves? We were thinking about logistics—e.g., how to get the visitors from one tour “stop” to the next—but also about engagement in our new “choose your own adventure” format.

For way finding and directions, the brochure that our Design team is developing for us plays an important part. Sara will be blogging next week about this brochure, so I’ll just give a few quick hints here: Thumbnail images of the works will help visitors spot them easily. A floor plan will give the “big picture.” And the designation of works as “highlights” versus “hidden gems” will allow visitors to craft their own experience. Meanwhile, the ASK team was working on an outline of factual content for each “highlight” and “gem.” This way, we would have access to consistent, prepared information and language to share with our visitors, no matter who was staffing the tour on any given day.  As things turned out, however, our shift from a tightly curated highlights list to a broader “menu” required some tweaks to our engagement strategy.

Colleagues who tested the tour concept with us, fill out a survey after their experience. A big take-away was the need to give visitors a certain amount of control over their experience (e.g. in selecting the works to visit), while also providing enough structure to the content with each work so visitors felt appropriately “guided.”

We tested our tour concept with several groups of colleagues from July through November and we gathered lots of helpful feedback. Some of it was related to the choice of works, but most of it revolved around engagement. Many of our testers felt a bit lost, even with their map prototypes. They would have liked more directives about how exactly to connect with each work of art and when to move on. They also expressed a desire to know more about our reasoning for choosing each work (i.e., what makes it a highlight?) as well as transitions and connections between stops. And they noticed when our replies seemed “scripted”; apparently we needed to think about the problem of our replies feeling packaged, rather than personalized.

In follow-up discussions, we agreed that we had already figured out how to make the most of our project’s unique qualities, like its self-directed, “go-at-your-own-pace” nature. Now we realized that we could also benefit from drawing on our collective experience as educators and “traditional” gallery guides.

After discussion, here are a few of the conclusions that we decided to put into practice:

• We could prompt the tour visitors with a “call to action” for each work, e.g., “Send me a photo of your next selection when you arrive there!
• We should establish our reasons for including each work of art at the outset of our conversation.
• We would guide their looking as if we were right there in the gallery with them: “If you look at the upper area of the canvas, you’ll spot an eagle flying the sky. Now let your eye move downwards…”
• We needed to use our own language as much as possible, relying on our outline for information but not for exact wording.

Once again, we were looking for the “sweet spot” that balanced structure and flexibility. Like many things we do with ASK, a blend of learned experience and new thinking seems to serve us best.

And, speaking of timing, the Highlights and Hidden Gems tour was just added to our website’s Group Tours page this week. We’re looking forward to booking our first groups!

The ASK team’s new home. A dedicated info host is stationed with the team at all times to help visitors.

Since our team location testing was completed last summer, they have been working behind the scenes in a temporary space. We were a bit curious as to what this change would do to our traffic. After all, no matter where we put the team during our crazy summer of roaming desks, the app traffic didn’t fluctuate much. While Shelley and I always conceived of the project with a visible team, was it truly necessary? Well, we’ve had about 6 months to find out the answer: yes! Since we pulled the team off the floor, our app traffic has slowly and steadily declined.

As I explained in a previous post, the single biggest challenge with ASK is helping people understand what the app actually is. The fact that you can chat in real-time with real people during your visit just doesn’t compute. This concept is particularly challenging to grasp from just a sign or a palm card—even if we say it it in person (though an in-person explanation helps). We’re constantly fighting people’s expectations of what a museum app is, and the single clearest way to communicate the concept of ASK is by having the team visible.

We know from testing several team locations in the building last summer, that a space just inside the Great Hall is the best location. This puts the team just beyond the entry experience hub-bub and closer to the art. It also places them right in a main thoroughfare of the building (our internal nickname for this space is “42nd street,” if that tells you anything). Since their presence seems vital to helping people understand the app, we want people to walk past them. However, there are several ways to access galleries from the lobby, so funneling traffic-flow will be key.

Another lesson learned from last summer was the importance of having other staff available to “run interference” so the team can focus on answering the questions coming in via the app (as opposed to questions asked in person). We’ve stationed information hosts with the team and, much like our visitor liaisons last summer, their role is tell people about the app, provide assistance with downloading when needed, and speak with visitors so the team is able to focus on their chats without interruption. The host  also hands out palm cards as visitors go by. I’m already seeing this work: more than once, I’ve watched people read the card as they’re walking away, stop, and turn to look at the team. I can see comprehension dawn on their faces. The concept clicks for them because the team is right there. It’s early yet, but I’m hopeful because we’ve already seen our app traffic climb. 

Staff hand visitors a palm card advertising the app.

While we’re pretty confident that the team’s current location is good, we’ll be continuing to test marketing materials to determine what language and key phrases work the best. We’ll also be working with the info hosts to determine the best ways to approach visitors and explain the app, building on our learnings from last summer. We haven’t found solutions to all our challenges just yet, but we’re getting there. In the meantime, next time you’re at the Museum, come say “hi” to the ASK team!