We began with a very short list of options in a drop down menu that included the special exhibitions, a few specific collection areas, and the collection more generally. We quickly found the need to add a few more options. For example, the admissions team asked for a “just in the neighborhood” option as it’s a common response to the question (though the data shows it’s not as common as they likely felt it was).

The survey appears in a pop-up window and has a drop-down menu of options. Unfortunately, the option to skip or cancel is baked-in; we can’t make this a required question to complete the transaction.

Results for the first two months are interesting. In July, the permanent collection ranked highest in response rate, while for August it was our Pierre Cardin special exhibition. The initial lack of options is one of the reasons for the high “other” response rate in July, which dipped the second month as more options were added. Currently, we have 16 options plus skip/decline. This feels like a lot, but maybe it’s ok. In particular, I wonder about including Korean art and African art in the list at the moment since both are temporarily off-view, but it would help us track an uptick once those collections are on view once more. We also have to remember to update the list regularly as special exhibitions open and close. For example, both Liz Johnson Artur and One: Egúngún exhibitions closed mid-August, which explains the dip in responses.

A quick comparison of the total number of survey responses (which should be every transaction) to total number of visitors who were required to visit the admissions desk shows the transaction count is about 60-65% of the visitor count. Multiple tickets can be purchased through a single transaction—and we know most of our visitors come in pairs or groups—and that feels close to the right percentage. I think we are still getting more cancellations than we should and we’ll keep working on it. The admissions team is meant to pose the question in a casual and conversational manner so it doesn’t feel like a survey (or an interrogation!) and select a response in order to proceed with the sale, although it is possible to cancel and move on. To avoid cancellation, we included a skip/decline option. Unfortunately, not everyone is consistently asking the survey question, which we know because we can run reports on who is logging which responses. For example, we found one person mostly just cancelled the survey in the first week and were able to speak with them. While we don’t want survey completion rate to become punitive, we do want to encourage completion because the information is important for us as an institution. Finding that balance can be tricky.

After two months, we are still working out the kinks, mostly in terms of making this process habit for the admissions team. A next step is to work with our Tech team to create a report that would knit together the survey answer, ticket info, and zip code from each transaction so we can compare the data set as a whole. That would be a pretty powerful triumvirate. 

1. Are ASK users first-time or returning museum visitors?
2. Are ASK users avid museum goers?
3. What kind of experience are users having?
4. Are the users participating in the Frida Kahlo ASK activities?
5. Are the ASK users using the app in Spanish?

The students determined a survey was best, which the ASK team sent via a link as part of the chat. This was more challenging than we thought because figuring out the timing for when to send the survey link proved difficult. You don’t want to share it first-thing, before anyone’s engaged, but finding the end of the conversation can be tough; some users just stop texting. The team had to play with timing a bit and I’m not sure we ever found the perfect moment. 

As with the Kahlo visitor study, we offered an incentive for completion: every 25th respondent got free tickets to Frida Kahlo. We ended up switching that incentive to free Kahlo Dance Party tickets since most users during this time had already seen the exhibition. In the end, we netted 115 responses over 6 weeks. As with the Kahlo evaluation, the results I’m sharing here include some of their interpretation of the data (reviewed by me) as well as some of my own analysis. 

Let’s break down the results. 

Are ASK users first-time or returning museum visitors?

Respondents were almost 50/50 split, with just slightly more repeat visitors. A caveat here though, is that we make this question time-bound, i.e. “within the last year,” so we don’t know if visitors meant life-time visits or something else.

Are ASK users avid art museum goers?

Yes! A majority (72%) visit art museums three plus times in a typical year (this question was timebound). Only 7% of respondents noted it was their first visit to any art museum, while almost 40% visit 6+ times a year.

What kind of experience are users having?

As indicated in the past through positive app store reviews, I’m happy to say we continue to provide a good experience for users. Over 98% of respondents would recommend the app to a friend. The most popular reasons given for this answer included descriptions that could be coded as “fun,” “informative,” and “helpful.” 

Are the users participating in the Frida Kahlo ASK activities?

Yes. Roughly a third (34%) of respondents checked the scavenger hunt as a way they used ASK, while 19% used the Kahlo themed cards.  Interestingly, although this question was a “check all,” a slight majority (57%) only checked on option. It’s possible this could be attributed to users have one main reason for using the app, though it could also be people simply filling out the survey quickly without stopping to think critically about all the ways they used it. Users tend to rely heavily on photos in chats, so the fact that only 22% check that use suggests it’s wise to take these responses with a grain of salt. 

Are the ASK users using the app in Spanish?

Unfortunately, the dashboard doesn’t track this for us automatically, so the way the students chose to get at this was to determine users’ preferred language. It doesn’t quite answer the question, but helps unpack user preferences. Over 80% of respondents (93) listed English and only 2 people listed Spanish. We do know, however, from the Kahlo visitor study that a little over a third of visitors interviewed used the Spanish language materials. Perhaps the lack of synergy between studies means Spanish-speakers aren’t using the app, despite invitations in Spanish to do so. More data would have been required to be really sure.

So, what does it all mean?

We learned a few things from the study. First, our assumption that more first-time visitors would use the app for the Kahlo activities—mostly because we felt like it was an easy way into exploring a new place—was correct. Those least likely to use the scavenger hunt in particular were the users who had visited us the most. 

Second, it’s nice to know most people have a good experience with us, no matter how they use ASK. A few comments include: 

The ask app totally enhanced my museum going experience. Typically I can find usage of phones in the galleries to be pretty distracting, but this app advances the experience. It’s great to have my questions answered in a friendly way- and by a person!

Due to ASK at the Brooklyn Museum, I rate this museum the top museum out of all I visited before!

Keep it up please! This is the best form of digital technology yet!

And third, as can be the case with visitor studies, this one leaves me wanting a bit more. Not through any fault of the students who worked on it, but due to the fact that it’s whetted my appetite for more info about our users. This study paints a small picture of what some of our users experienced, but what would happen if we ran this for longer? Switched out some questions? Got more responses? Something we may have to explore…

A visitor study of “Frida Kahlo” was a perfect project for some of my Pratt grad students.

In my last post, I posited that although we don’t have a CRM, we are gathering data in the ways we can to help inform our decisions. One recent example are the two evaluation projects completed by Pratt graduate students in the Museums and Digital Culture program last semester. I had eight students in the Audience Research & Evaluation course, so I split them into two groups of four for final projects. One team was responsible for completing a visitor study about ASK users and the other about visitors to Frida Kahlo. For this post, I’ll focus on the latter. The student project team for the Kahlo evaluation was Sydney Stewart, Megan Paqua, Grace Poole, and Caroline Baer. The results I’m sharing here are their interpretation of the data (reviewed by me) and their work (including the visuals).

The objective of the study, as defined by our institutional needs and finalized by project team, was to determine: who is coming to Frida Kahlo and will they return to the Museum? In addition, we wanted to know if the Spanish language materials were useful. The first part was really to help us check our assumption that Frida Kahlo might bring in a lot of first-time visitors. The second was to give us some insight on the ROI of the translation work we did for this exhibition, which was the most extensive we’ve ever done—ticketing pages, signs in the building, floor plan, and labels were all provided in both English and Spanish. While we’ve provided label copy in other languages in the past, we had yet to translate logistical information beyond the floor plan. We have a small team and none of our editors are native Spanish speakers. Best practice requires someone for translation as well as proofreading and layout review. Thankfully we have several fluent Spanish speakers (some of whom are native speakers) who helped in the review process and even translated our last minute signs. (Thank you Stephanie, Rachel, Jaime, and Allison!) Working with Eriksen Translations and our in-house powerhouse, we were able to get the translation completed, but it took a lot of extra time and resources.

We determined that the best approach to getting answers to our research objectives were interviews with Kahlo visitors. While we had visitor time and attention, we also asked about how they heard about the exhibition and how their experiences was. All eight students, myself, and Rachel Lewis (who conducted interviews in Spanish for us, thank you!) conducted interviews in the exhibition shop, which was exit point of the exhibition. We offered 10% shop coupons as an incentive to take the survey.

Results

We interviewed visitors during class time on four Thursday afternoons from about 3:30pm-5:30pm in March-April 2019. In total we completed 259 interviews. Below are the results, as written in the students’ final report (in quotes):

Who came to Frida Kahlo?

“Visitors came from a wide geographic area. Visitors to Frida Kahlo were almost evenly split between tourists and New York City residents. The majority of New York City residents were from Brooklyn, but the show also attracted visitors across the city. Overall, domestic visitors came from 26 states plus Puerto Rico, and international visitors came from 8 different countries.” Side note: the data is skewed since it only accounts for responses during a certain time window: Thursday afternoons. Responses would likely be different on different days. Weekends, for example, likely bring in more locals on their day off.

Visitorship was almost evenly split between locals and tourists, which may be accounted for by the fact that interviews took place Thursday afternoons only.

Did they use the Spanish language materials?

“Visitors responded positively to the inclusion of Spanish language materials, particularly the exhibition labels, whether or not they used them personally, and indicated that continued inclusion of bilingual materials would make them more likely to make a return visit to the museum.

Visitors frequently remarked on the relevance of the Spanish language materials for an exhibition about Frida Kahlo. Many visitors expressed an interest in expanding foreign language offerings in future exhibitions when appropriate, especially for exhibitions featuring artist from non-English speaking communities.”

The majority of visitors interviewed did not use the Spanish language materials.

How did they hear about the exhibition?

“35% of visitors reported learning about Frida Kahlo through word of mouth, although social media posts as well as reviews in major news publications were the next most effective at reaching visitors. On-site signage was also useful for attracting visitors who had not heard about the exhibition prior to their arrival at the museum.”

Word-of-mouth ranked highest among ways people learned about the exhibition.

How was their experience? Will they come back to the Museum?

“No matter where visitors are coming from, Frida Kahlo was the primary motivation for visitors to come to the Brooklyn Museum, whether it was out of personal interest, to visit with a friend or family member, or based on a recommendation or advertisement. The majority of visitors expressed having positive opinions about the exhibition and their visit overall, even when they had complaints about other aspects of their experience.

Visitors had a clear interest in returning to the museum for special exhibitions, especially if the focus is on diverse communities or artists of color. Many visitors stated that they associated the Brooklyn Museum with this type of programming and were more positively disposed towards the institution as a result.”

For 71% of repeat visitors, “Frida Kahlo” specifically and special exhibitions in general are the reason to return to the Museum.

What would visitors like to see more of at the Museum?

“When asked what they would like to see more of at the museum, 10.71% of visitors responded with requests for more activities designed for children. These visitors were overwhelmingly returning visitors, indicating that this is a current gap in the Museum’s programming within the exhibition space.”

Learnings

This evaluation confirmed some of what we already knew:

• Special exhibitions are a big draw for first time and repeat visitors alike.
• Word-of-mouth is important, though it’s sometimes tough to parse out the nuances of this answer. While some people genuinely couldn’t remember exactly where they heard about it, for others there might be more to the story. For example, their friend told them about it either through social media or because that friend saw it on social media. So does that count as word-of-mouth or social media? Or both? More to play with here to get better data to help our marketing team.

And gave us some new insights:

• The majority of visitors felt that the Spanish materials contributed positively to their experience whether they used them or not.
• There’s a desire for more activities for children specifically in the exhibition galleries. As reported: “Of the individuals who responded to what they would like to see more of at the museum, 10.71% indicated a desire for more kid-friendly activities. The kid-friendly advocating visitors are also primarily returning visitors (88.9% who desire kid friendly activities).”

Those interviewed felt that the Spanish materials were positive and useful whether they speak Spanish or not.

The results of this evaluation have been really useful as we think about planning our upcoming exhibitions. In particular the usefulness of languages has come up a few times as we plan for our Pierre Cardin: Future Fashion exhibition opening next month. Do we translate to French, the designer’s native tongue? Ultimately we decided not to go that route, but mostly because we don’t think we’ll have high attendance of Francophiles (since Cardin has his own museum in Paris).

In addition to providing some good information about visitors to Frida Kahlo, the evaluation also gave us good practice as crafting questions and things to think about in terms of how to tease out nuances of how people hear about the exhibitions. As always, it’s a work-in-progress.

Without a unified CRM, the data landscape can look a little bleak.
Stephen McMillan (American, born 1949). Zabriskie Point, 1976, 1976. Aquatint on paper, sheet: 22 1/8 x 29 3/4 in. (56.2 x 75.6 cm). Brooklyn Museum, Gift of ADI Gallery, 77.152.2. © artist or artist’s estate (Photo: Image courtesy of Stephen McMillan, CUR.77.152.2_StephenMcmillan_photograph.jpg)

Despite our lack of CRM, we’ve been working hard to gather data in ways we can. For example, we started a single question survey as part of our admissions process—What brought you to the Museum today?—that our front-of-house team is meant to ask as part of a casual and welcoming conversation. That, along with the zip codes we gather as part of every onsite transaction (a requirement of city funding is to report on zip codes), helps us begin to get some basic information about who is coming and why.

On top of regular focused data points like zip codes and single question surveys, we will be running an annual visitor study. We recently partnered with SightX for our research needs and have launched a baseline visitor study to get a better idea of visitor museum-going behavior and demographics. We haven’t run such a survey consistently for 3 years or more. I’m really looking forward to having that data ongoing. We’ve tried to be thoughtful in not only what questions we pose, but also how we phrase them.

In addition to our own visitor study efforts, one amazing resource we’ve been able to tap to build our data stores is our relationship with Pratt Institute School of Information and their Museums and Digital Culture graduate program in particular. In addition to our ongoing fellowship and Sydney’s recent work on ASK data, I’m able to work with the students I teach (and other professors’ classes) on projects that both give the students real-world experience and help the Museum—a true win-win. This past semester, I taught Audience Research & Evaluation. Through class work, students ran one evaluation related to the Frida Kahlo exhibition and another on our ASK app. I’ll report on the findings of these studies in a future post.

So data gathering is happening, albeit piecemeal sometimes, and we’re trying to smart about knitting it into processes and moments that make sense such as with the admissions process, or a short, three-question survey with our thank-you emails to online ticket holders. What does our data landscape really look like and what can we do with the data we do have?  I’m happy to say, that’s where a dedicated data analyst comes in. Thanks to Bloomberg Philanthropies’ ongoing support, we are able to contract a data analyst (job description forthcoming) to help us begin to make sense of all these data points, with ASK data as a pivotal element in the data landscape. I’m really looking forward to carving out time to get the big picture of all the ways we gather data and streamlining those moments. If you know of any good people, please send them our way. In the meantime, check back as we share the results of our recent visitor studies.

Let’s start with our status of engagement. Visitors have been using the app and are having great experiences as evidenced by five star reviews in the app store and feedback we are getting directly through the in-app conversations. Use has remained mostly consistent from our earlier findings—a soft launch without much marketing is yielding a 1-2% use rate and we’ve had a little over 2600 conversations thus far. Visitors are taking the app through multiple galleries and, on average, asking questions in at least two spaces. The power user average—defined as those who ask questions in three or more galleries—are asking questions, on average, in six galleries and these users represent 18.79% of our total. Conversations are fairly deep with chats consisting, on average, of 13 messages back and forth. One of the most compelling things we’ve seen is that people using the app can remember—sometimes days later—the names of the ASK team who helped them. Beyond any numerical statistic, this particular trend shows us that the exchanges are both personal and meaningful in a way that is similar to face to face communication. That’s a big win for this app and something I’m incredibly proud of; it also shows just how great the ASK team members are at their jobs of engaging the public.

Coming into the museum there’s not much telling visitors we have an app, but that marketing plan is something we’ve been working on.

Having said all that we’ll also tell you that if you’ve come to the museum lately you have not seen the ASK team interacting on the floor. From earlier posts, you know we had done a lot of prototyping work with the team in various locations and found that the best spot was likely going to be a somewhere mid-visit. Finding a place for the team in the heart of the museum’s spaces has been a challenge and with new institutional direction the galleries are changing considerably; essentially, we know what kind of home they need, but right now is not the most ideal time to be building it. This means the team is working behind the scenes and, to my own surprise, this does not seem to be affecting the actual user experience. If anything, the team is finding they are more equipped to handle incoming questions via the app because there are no additional distractions and they can communicate amongst themselves more effectively.

One issue in having the team off the floor is the lack of visibility in the museum. If you walked into the Brooklyn Museum today, you’d be hard pressed to know we have an app because we just don’t have much marketing and we are heavily relying on the front desk staff to tell people about it. That will change, however, in late spring when we do a more formal launch which will include a version for Android. This marketing plan is one of the most important things we are working on right now. Given what we are seeing with the engagement and the current institutional goals, the thought is we should let the marketing do the job of building awareness while the team thinks about other ways to engage on site. This might translate to meeting and greeting the public through public programming instead of a permanent presence on the floor.

What else have we been doing? This is a three year project—year one was about getting mobile into the hands of visitors, year two is figuring out what the interactions teach us, and year three is still very much a work in progress. The single most important thing we’ve been doing is data review because what we learn from the ASK interactions and how that could transform the visitor experience is at this initiative’s core.

Marina Klinger, the ASK Curatorial Liaison, has been organizing and leading data review meetings between the ASK team and our curators.

To this end, we’ve been meeting with curators in every single collection. Curators are given “snippet reports” which contain each exchange we’ve had with visitors on every object in their collection and these reports represent one of the deepest and richest data sets I’ve ever seen. We export exchanges into Google Docs (using their API) and share with curatorial teams who can comment on the conversations. We then have followup meetings to discuss the data. This process gets the interaction to curators, but it also serves our ongoing need to train the ASK team; curators can tell us where answers might need improvement and make sure we are aligned with curatorial vision. Already we are hearing the data is giving curatorial staff some key learnings which may help them think about visitor experience as they reinstall and/or make changes to galleries. We are also taking this opportunity to find out how often curators would like reports, how we can make reporting more efficient, and how reporting may need to differ from permanent collection galleries to special exhibitions.

So, while it may seem like we’ve been quiet over here, we’ve been steadily working away and making decent progress on this year two of learning from the interaction. It’s pretty critical to move through this year with great measure because what we learn at this stage helps us figure out what we want year three of ASK to be, so you may see us blogging less frequently, but you’ll find when we do we’ve got a lot of information to share. Speaking of, Sara will be up next to talk about similar meetings with education staff, engagement strategy, and the introduction of new staff members.