1. Are ASK users first-time or returning museum visitors?
2. Are ASK users avid museum goers?
3. What kind of experience are users having?
4. Are the users participating in the Frida Kahlo ASK activities?
5. Are the ASK users using the app in Spanish?

The students determined a survey was best, which the ASK team sent via a link as part of the chat. This was more challenging than we thought because figuring out the timing for when to send the survey link proved difficult. You don’t want to share it first-thing, before anyone’s engaged, but finding the end of the conversation can be tough; some users just stop texting. The team had to play with timing a bit and I’m not sure we ever found the perfect moment. 

As with the Kahlo visitor study, we offered an incentive for completion: every 25th respondent got free tickets to Frida Kahlo. We ended up switching that incentive to free Kahlo Dance Party tickets since most users during this time had already seen the exhibition. In the end, we netted 115 responses over 6 weeks. As with the Kahlo evaluation, the results I’m sharing here include some of their interpretation of the data (reviewed by me) as well as some of my own analysis. 

Let’s break down the results. 

Are ASK users first-time or returning museum visitors?

Respondents were almost 50/50 split, with just slightly more repeat visitors. A caveat here though, is that we make this question time-bound, i.e. “within the last year,” so we don’t know if visitors meant life-time visits or something else.

Are ASK users avid art museum goers?

Yes! A majority (72%) visit art museums three plus times in a typical year (this question was timebound). Only 7% of respondents noted it was their first visit to any art museum, while almost 40% visit 6+ times a year.

What kind of experience are users having?

As indicated in the past through positive app store reviews, I’m happy to say we continue to provide a good experience for users. Over 98% of respondents would recommend the app to a friend. The most popular reasons given for this answer included descriptions that could be coded as “fun,” “informative,” and “helpful.” 

Are the users participating in the Frida Kahlo ASK activities?

Yes. Roughly a third (34%) of respondents checked the scavenger hunt as a way they used ASK, while 19% used the Kahlo themed cards.  Interestingly, although this question was a “check all,” a slight majority (57%) only checked on option. It’s possible this could be attributed to users have one main reason for using the app, though it could also be people simply filling out the survey quickly without stopping to think critically about all the ways they used it. Users tend to rely heavily on photos in chats, so the fact that only 22% check that use suggests it’s wise to take these responses with a grain of salt. 

Are the ASK users using the app in Spanish?

Unfortunately, the dashboard doesn’t track this for us automatically, so the way the students chose to get at this was to determine users’ preferred language. It doesn’t quite answer the question, but helps unpack user preferences. Over 80% of respondents (93) listed English and only 2 people listed Spanish. We do know, however, from the Kahlo visitor study that a little over a third of visitors interviewed used the Spanish language materials. Perhaps the lack of synergy between studies means Spanish-speakers aren’t using the app, despite invitations in Spanish to do so. More data would have been required to be really sure.

So, what does it all mean?

We learned a few things from the study. First, our assumption that more first-time visitors would use the app for the Kahlo activities—mostly because we felt like it was an easy way into exploring a new place—was correct. Those least likely to use the scavenger hunt in particular were the users who had visited us the most. 

Second, it’s nice to know most people have a good experience with us, no matter how they use ASK. A few comments include: 

The ask app totally enhanced my museum going experience. Typically I can find usage of phones in the galleries to be pretty distracting, but this app advances the experience. It’s great to have my questions answered in a friendly way- and by a person!

Due to ASK at the Brooklyn Museum, I rate this museum the top museum out of all I visited before!

Keep it up please! This is the best form of digital technology yet!

And third, as can be the case with visitor studies, this one leaves me wanting a bit more. Not through any fault of the students who worked on it, but due to the fact that it’s whetted my appetite for more info about our users. This study paints a small picture of what some of our users experienced, but what would happen if we ran this for longer? Switched out some questions? Got more responses? Something we may have to explore…

A visitor study of “Frida Kahlo” was a perfect project for some of my Pratt grad students.

In my last post, I posited that although we don’t have a CRM, we are gathering data in the ways we can to help inform our decisions. One recent example are the two evaluation projects completed by Pratt graduate students in the Museums and Digital Culture program last semester. I had eight students in the Audience Research & Evaluation course, so I split them into two groups of four for final projects. One team was responsible for completing a visitor study about ASK users and the other about visitors to Frida Kahlo. For this post, I’ll focus on the latter. The student project team for the Kahlo evaluation was Sydney Stewart, Megan Paqua, Grace Poole, and Caroline Baer. The results I’m sharing here are their interpretation of the data (reviewed by me) and their work (including the visuals).

The objective of the study, as defined by our institutional needs and finalized by project team, was to determine: who is coming to Frida Kahlo and will they return to the Museum? In addition, we wanted to know if the Spanish language materials were useful. The first part was really to help us check our assumption that Frida Kahlo might bring in a lot of first-time visitors. The second was to give us some insight on the ROI of the translation work we did for this exhibition, which was the most extensive we’ve ever done—ticketing pages, signs in the building, floor plan, and labels were all provided in both English and Spanish. While we’ve provided label copy in other languages in the past, we had yet to translate logistical information beyond the floor plan. We have a small team and none of our editors are native Spanish speakers. Best practice requires someone for translation as well as proofreading and layout review. Thankfully we have several fluent Spanish speakers (some of whom are native speakers) who helped in the review process and even translated our last minute signs. (Thank you Stephanie, Rachel, Jaime, and Allison!) Working with Eriksen Translations and our in-house powerhouse, we were able to get the translation completed, but it took a lot of extra time and resources.

We determined that the best approach to getting answers to our research objectives were interviews with Kahlo visitors. While we had visitor time and attention, we also asked about how they heard about the exhibition and how their experiences was. All eight students, myself, and Rachel Lewis (who conducted interviews in Spanish for us, thank you!) conducted interviews in the exhibition shop, which was exit point of the exhibition. We offered 10% shop coupons as an incentive to take the survey.

Results

We interviewed visitors during class time on four Thursday afternoons from about 3:30pm-5:30pm in March-April 2019. In total we completed 259 interviews. Below are the results, as written in the students’ final report (in quotes):

Who came to Frida Kahlo?

“Visitors came from a wide geographic area. Visitors to Frida Kahlo were almost evenly split between tourists and New York City residents. The majority of New York City residents were from Brooklyn, but the show also attracted visitors across the city. Overall, domestic visitors came from 26 states plus Puerto Rico, and international visitors came from 8 different countries.” Side note: the data is skewed since it only accounts for responses during a certain time window: Thursday afternoons. Responses would likely be different on different days. Weekends, for example, likely bring in more locals on their day off.

Visitorship was almost evenly split between locals and tourists, which may be accounted for by the fact that interviews took place Thursday afternoons only.

Did they use the Spanish language materials?

“Visitors responded positively to the inclusion of Spanish language materials, particularly the exhibition labels, whether or not they used them personally, and indicated that continued inclusion of bilingual materials would make them more likely to make a return visit to the museum.

Visitors frequently remarked on the relevance of the Spanish language materials for an exhibition about Frida Kahlo. Many visitors expressed an interest in expanding foreign language offerings in future exhibitions when appropriate, especially for exhibitions featuring artist from non-English speaking communities.”

The majority of visitors interviewed did not use the Spanish language materials.

How did they hear about the exhibition?

“35% of visitors reported learning about Frida Kahlo through word of mouth, although social media posts as well as reviews in major news publications were the next most effective at reaching visitors. On-site signage was also useful for attracting visitors who had not heard about the exhibition prior to their arrival at the museum.”

Word-of-mouth ranked highest among ways people learned about the exhibition.

How was their experience? Will they come back to the Museum?

“No matter where visitors are coming from, Frida Kahlo was the primary motivation for visitors to come to the Brooklyn Museum, whether it was out of personal interest, to visit with a friend or family member, or based on a recommendation or advertisement. The majority of visitors expressed having positive opinions about the exhibition and their visit overall, even when they had complaints about other aspects of their experience.

Visitors had a clear interest in returning to the museum for special exhibitions, especially if the focus is on diverse communities or artists of color. Many visitors stated that they associated the Brooklyn Museum with this type of programming and were more positively disposed towards the institution as a result.”

For 71% of repeat visitors, “Frida Kahlo” specifically and special exhibitions in general are the reason to return to the Museum.

What would visitors like to see more of at the Museum?

“When asked what they would like to see more of at the museum, 10.71% of visitors responded with requests for more activities designed for children. These visitors were overwhelmingly returning visitors, indicating that this is a current gap in the Museum’s programming within the exhibition space.”

Learnings

This evaluation confirmed some of what we already knew:

• Special exhibitions are a big draw for first time and repeat visitors alike.
• Word-of-mouth is important, though it’s sometimes tough to parse out the nuances of this answer. While some people genuinely couldn’t remember exactly where they heard about it, for others there might be more to the story. For example, their friend told them about it either through social media or because that friend saw it on social media. So does that count as word-of-mouth or social media? Or both? More to play with here to get better data to help our marketing team.

And gave us some new insights:

• The majority of visitors felt that the Spanish materials contributed positively to their experience whether they used them or not.
• There’s a desire for more activities for children specifically in the exhibition galleries. As reported: “Of the individuals who responded to what they would like to see more of at the museum, 10.71% indicated a desire for more kid-friendly activities. The kid-friendly advocating visitors are also primarily returning visitors (88.9% who desire kid friendly activities).”

Those interviewed felt that the Spanish materials were positive and useful whether they speak Spanish or not.

The results of this evaluation have been really useful as we think about planning our upcoming exhibitions. In particular the usefulness of languages has come up a few times as we plan for our Pierre Cardin: Future Fashion exhibition opening next month. Do we translate to French, the designer’s native tongue? Ultimately we decided not to go that route, but mostly because we don’t think we’ll have high attendance of Francophiles (since Cardin has his own museum in Paris).

In addition to providing some good information about visitors to Frida Kahlo, the evaluation also gave us good practice as crafting questions and things to think about in terms of how to tease out nuances of how people hear about the exhibitions. As always, it’s a work-in-progress.

When we look at engagement goals, Sara and I are really going for a gold standard.  If someone gets a question asked and answered, is satisfied, and the conversation ends—that’s great, but we’ve already seen much deeper engagement with users and that’s what we’re shooting for. Our metrics set can show us if those deeper exchanges are happening. Our engagement goals include:

• Does the conversation encourage people to look more closely at works of art?
• Is the engagement personal and conversational?
• Does the conversation offer visitors a deeper understanding of the works on view?
• How thoroughly is the app used during someone’s visit?

We doing pretty well when it comes to engagement. We regularly audit chats to ensure that the conversation is leading people to look at art and that it has a conversational tone and feels personal. The ASK team are also constantly learning more about the collection and thinking about, experimenting with, and learning what kinds of information and conversation via the app open the door for deeper engagement and understanding of the works. In September, we’ll begin the process of curatorial review of the content, too, which will add another series of checks and balances ensuring we hit this mark of quality control.

Right now the metrics show us conversations are fairly deep; 13 messages on average through this soft launch period (starting June 10 to date of this post). The team is getting a feel for how much the app is used throughout a person’s visit; they’ve been having conversations throughout multiple exhibitions over the course of hours (likely an entire visit). Soon we’ll be adding a metric which will give us a use rate that also shows the average number of exhibitions, so we’ll be able to quantify this more fully. Of course, there are plenty of conversations that don’t go nearly as deep and don’t meet the goals of above (we’ll be reporting more about this as we go), but we are pretty confident in saying the engagement level is on the higher end of the “success” matrix. The key to this success has been the ASK team who’ve worked long and hard to study our collection and refine interaction with the public through the app.

Use rate is on the lower end of the matrix and this is where our focus is right now. We define our use rate by how many of our visitors are actually using the app to ask questions. From our mobile use survey results, we know that 89% of visitors have a smartphone and we know from web analytics that 83% of our mobile traffic comes from iOS devices. So, we’ve roughly determined that, overall, 74% of the visitors coming through the doors have iOS devices and are therefore potential users. To get our use rate, we take 74% of the attendance rate (eligible iOS device wielding users) and balance that with the number of conversations we see in the app giving us a percentage of overall use.

Use rate during soft launch has been bouncing around a bit from .90% to 1.96%, mostly averaging in the lower 1% area. All kinds of things affect this number from the placement of the team, how consistent the front desk staff is at pitching the app as first point of contact, the total number of visitors in the building, and the effectiveness of messaging. As we continue to test and refine, the numbers shift accordingly and we won’t really know our use rate until we “launch” in fall with messaging throughout the building, a home for our ASK team, and a fully tested process for the front desk pitch and greeting process.

Our actual download rate doesn’t mean much especially given the app only works in the building. Instead, the “use rate” is a key metric defined as actual conversations compared to iphone-wielding visitors. The one thing the download rate stats do show us is the pattern of downloads runs in direct parallel with our open hours. Mondays and Tuesdays are the valleys in this chart and that’s also when we are closed to the public; Saturdays and Sundays are the peaks when attendance is higher.

Still, even with these things in flux, our use rate is concerning because one trend we are seeing is a very low conversion on special exhibition traffic. As it stands, ASK is being used mostly by people who are in our permanent collection galleries. Don’t get me wrong—this is EXCELLENT—we’ve worked for years on various projects (comment kiosks, mobile tagging, QR codes, etc) that would activate our permanent collections; none have seen this kind of use rate and/or depth of interaction. However, the clear trend is ASK is not being taken advantage of in our special exhibitions and this is where our traffic resides. We are starting with getting effective messaging up more prominently in these areas. Once we get the visibility up, we’ll start testing assumptions about audience behavior. It may be that this special exhibition traffic is here to see exactly what they came for with little want of distraction; if ASK isn’t on the agenda it may be an uphill battle to convert this group of users. Working on this bit is tricky and it will likely be a few exhibition cycles before we can see trends, test, and (hopefully) better convert this traffic to ASK.

There’s a balance to be found between ensuring visibility is up so people know it’s available (something we don’t yet have) and respecting the audience’s decision about whether to use it. Another thing we are keeping in mind is the ASK team is in the galleries and answering questions in person—this may or may not convert into app use, but having this staff accessible is important and it’s an experience we can offer because of this project. Simply put, converting traffic directly may not be an end goal if the project is working in other ways.

The last bit of determining success—institutional knowledge gained from the incoming data—is something that we can’t quantify just yet. We do know that during the soft launch period the larger conversations have been broken down into 1,241 snippets of reusable content (in the form of questions and answers) all tagged with object identification. Snippets are integrated back into the dashboard so the ASK team has previous question/answer pairings at their fingertips when looking at objects. Snippets also tell us which objects are getting asked about, what people are asking, and will likely be used for content integration in later years of the project. The big step for us will come in September when we send snippet reports to curatorial so this content can be reviewed. We hope these reports and meetings help us continue to train the ASK team, work on quality control as a dynamic process, and learn from the incoming engagement we are seeing.  

Is ASK successful?  We’re leaving you with the picture that we have right now. We’re pretty happy with the overall depth of engagement, but we believe we need to increase use. It will be a while before we can quantify the institutional knowledge bit, so measuring the overall success of ASK is going to be an ongoing dialog. One thing we do know is the success of the project has nothing to do with the download rate.