Home screen of the upcoming Brooklyn Museum App. We use geofencing to shift the focus based on your proximity to our building when you are outside (left) and inside (right).

Our app will use the device’s geolocation to serve information accordingly. When a visitor is outside the building, the app will display what we believe visitors are commonly looking for in the form of pre-visit information—hours, directions, exhibitions and events information. When a visitor comes in the building the app will present the ASK activity front and center.

Opening the ASK feature from outside the building (left) and inside the building (right).

ASK, in fact, is only active when someone is in the building—we are specifically not providing this service outside the building. We don’t see our team as a human version of Google and we’ve put our focus on driving engagement toward the collection when someone is standing in our galleries in front of a work of art.  If you are outside the building and tap on ASK, you’ll see a splash screen advertising the feature and explaining that it becomes active when you visit us.

There were a few big questions for us, though. The first is the industry-wide trend we are seeing toward multiple apps for single uses. More often than not, people are releasing apps that try and do one thing well.  When they want to do something else well, they just release another app focused on that thing. The issue for us is we know there’s already a lot of friction in downloading an app.  We don’t think splitting the brand over two apps would be useful to the visitor. We also can’t realistically develop and maintain two apps; that’s not a sustainable path for us.

We use the sandwich menu as the consistent element. It’s present both inside and outside and always displays the same information.

So, our task was to figure out how to present both things (pre-visit information and ASK) in a way that allows both to take center stage at the right time, but otherwise keep the two functions as simple as possible. Internal discussion started to zero in on consistency and user expectations as the second big question we needed to answer. Do users expect the app to remain consistent? Will it be confusing to open the app outside the building and see one thing, but find something else when it’s opened at the Museum? We settled on a deploying a sandwich menu in the upper left, which will always stay consistent in both its placement and options.

So what happens if you try and tap on ASK from within the sandwich menu, but you are standing outside the building where the feature is not active? Well, you get an informational screen telling you it’s something you can use when you get here. Internal discussion started to shift toward ASK awareness. How much should we be advertising ASK functionality outside the Museum? We settled on including ASK within the menu, but we decided not to make a big advertisement for it on the home screen people see when they are outside the building. Why?

At first glance you’d think….don’t you want to advertise this ASK thing at all times???? Well, yes and no. If a user’s primary interest before they get here is to figure out when we’re open, what train to take, and what they want to see, then that has to be primary for the users’ sake. The last thing we want to do is make those things harder to find.  Also, while you might want to know this is a feature you could use when you get here, it’s pretty doubtful that you’ll remember when your primary focus is this awesome show you want to catch. Lastly, it’s a pretty big downer to be given an option that you can’t do anything with….yet.

So, ASK is in the menu from outside the building and if you poke around and get some awareness about it it great. Otherwise, it takes center stage when you get here and we are going to rely on our entry experience and our team to get visitors focused on the new opportunity that awaits them. As with everything, this is just the path we are taking right now.  We are going to learn a heck of a lot when we deploy and things may change as a result. This is one of those features you can’t easily evaluate in user testing until the app is actually out the door.

Our web team on the ASK dashboard while one of our test participants testing the app observes “Trade Sign (Boy Riding Bicycle)”

Our first round of visitor testing took place in late October with several small groups of Members. We invited testers into our American galleries on the 5th floor where we were set up. Sara Devine was typing at the dashboard for our chief curator, Kevin Stayton, who answered visitor questions. Each visitor was given an iPod touch pre-loaded with our app. We gave them brief verbal instructions which can be summed up as “use the app to ask any questions you have about our artwork as you wander.” We were careful not to hand-hold or walk them through the interface. The visitors were then free to roam the galleries with our iPods. One reason why we wanted to test in small groups is each visitor was shadowed by a staffer as they used the app and we encouraged them to verbalize what they were thinking/feeling/doing at any point as we logged observations. At the end of each session testers met with the whole team where we posed questions about the entire experience.

Interviewing our testers on the ASK experience after the test.

The unanimous feedback from visitors who used our app was that the experience is very personal and friendly. Visitors thought that the quality of responses was very high, such that it inspired them to ask better questions. Visitors expressed that they found themselves looking at the art closer so they could ask thoughtful questions. All of these things we had heard from our earliest pilot with iMessage, so we were hoping to see that again with the app that we built specifically for the program. Boom. Nailed it.

We noticed our visitors wandering in a “U” pattern; standing at a work they would ask a question then they would wander close by until an answer was given. Often, once they received the answer they would circle back to the artwork to take another look. We had seen this behavior in the iMessage pilot and were encouraged to see it again. Unlike the iMessage pilot, we found our visitors constantly checking their devices to see if a new answer came. Our test groups resoundingly asked for a vibration alert to notify them of an answer so they could put their device away while without the need to check. iMessage has this built-in, so that’s something that worked from the earlier pilot, but is needed in our own implementation to help encourage the behaviour we want—phones away and looking at art as much as possible.

Testers reported (and were observed) looking more closely at works of art often putting down screens in the process. The tester here is a good example – he’s looking at our Francis Guy painting while the mobile device is in his right hand at his side.

We were concerned about the wait time for visitors to get responses from our audience engagement team, in this case, Kevin Stayton. However, our visitors expressed that they didn’t mind the wait because they found the answers to be so high quality that they believed them to be worth waiting for and realized they were being connected to a real person, not an automated system. This is something we will need to revisit as we test with a larger audience because how we scale will be a challenge that we can tackle a number of ways. One idea we have right now is to warn visitors of slow response times during high traffic periods.

Kevin Stayton, chief curator answering visitor questions with Sara Devine typing them on the ASK dashboard.

Does it matter to the visitor who exactly (staff or curator) is responding to the questions if the answers are good? When we asked testers directly, the answers that we received were that it didn’t matter as long as the answers were perceived as having value often defined as “good,” “worthwhile,” and/or “interesting.” We attempted to A/B test this by introducing our chief curator Kevin Stayton as the question answerer to part of the group, while keeping this a secret from others. This failed when our members recognized Kevin from previous museum visits. We will definitely be looking to test this with larger groups in later phases. When asked, members felt it was not important to be connected to a single voice throughout their visit, but this also something we will need to test as we get a full audience engagement team in place and they trade off in answering questions coming into the queue. While it would be ideal to have the same person on the audience engagement team respond to a visitor, it is not feasible due to our limitations with staffing and/or areas of expertise among the team manning the dashboard.

One of the biggest lessons from our testing was the reveal that our onboarding process is completely broken. We find that visitors skip reading the text on our first screen which has a form to sign up. They repeatedly miss the fields that ask for their name and email, even if there are error messages. They keep clicking our “get started” button in the hope of advancing which prevents them unless they fill out their details. They certainly are not reading why we are asking for this information, which is obscured in the text wall they didn’t read. Additionally, our error messaging was not noticeable enough and they didn’t recognize when they were hitting problems.

Our onboarding process needs some work and changes to it are the next thing we’ll be testing in early December.

Knowing our onboarding needs a major revision, we are focusing on that for our next round of testing. We are testing new ideas using internal staffers outside of the tech department and will put the best of those in front of members in early December.

Throughout, our initial testing has been with small numbers of people and we are being careful to write everything down and look for trends that we hear across all participants. Tons of great features have been suggested, but until we get many more people using the app in later stages of testing it’s important to simply gather information and identify what is a bug fix and where included features could be improved. In one instance, we saw critical mass when almost every tester expressed a desire in taking their conversation home. From the get go we designed the ASK app to only work on-site, so visitors are asking questions as they are looking at the art. In doing so, we were completely locking the visitor out from seeing their conversation history. We now know that we need to open it up and even though we are seeing that feedback across the board, we may wait on development to do more focus group testing before specific implementation.

Onward! Stay tuned on revisions for our next round of testing.

I’m playing the role of a visitor testing the ASK mobile app.

As a part of our process of developing the ASK app, we knew user testing in the museum with visitors was absolutely essential. It allows us as a team to study what our visitors are doing when they use the app and how it might be different from our expectations. We can then use this knowledge, iterate on our product, and deliver an effective experience for the next release.

Prior to user testing with actual visitors, we focussed on testing with team members internally. The goal was to smooth out any kinks with our end-to-end experience and fix major bugs before visitors got to use the ASK app.

A big lesson that we learned in our internal testing is the importance of testing early and often. Our ASK app is composed of many pieces that speak to each other. These pieces are namely: the ASK mobile app, the ASK dashboard (a web interface for the audience engagement team to receive and answer questions), and an API that connects them both. The most effective way for us to test these pieces in sync are by role-playing the experience of visitor and question-answerer. This internal role-playing led us to quickly roll out fixes.

James, our back-end web developer role-playing as visitor and testing the ASK mobile app.

The developers working on the ASK app switched roles frequently during testing so all of us would be familiar with all the pieces. Any time we rolled out a new change, we found ourselves forming small groups to test. We loaded our app on iPod touches, a few people went to our American galleries on the 5th floor and asked questions. Another team member stayed back manning the dashboard answering questions. As more of us used the ASK app as we would in the real-world, we discovered problems very quickly.

The design mock-ups that we referenced to develop the dashboard had failed in filling some important blanks like transitions and button behaviors. Our web designer was able to quickly jump into the code and make edits that made sense in the browser in order to make the dashboard friendlier. Everything was lined up perfectly and clicking around made sense. These changes were absolutely essential for someone using the dashboard so they could quickly see and act on incoming questions.

We discovered inaccuracies with iBeacon in indicating visitor location. In order to figure out what was happening, we added debug and log messages everywhere to see if the problem was with iBeacon, the mobile app, the api, or the dashboard. Jennie, our iOS dev has written a post on how we addressed a lot of our iBeacon woes.

Weaker wifi signal in some galleries would cause messages or images to not successfully deliver to the dashboard. This was one of many bugs that were missed when we tested fixes at our desks.

Internal testing of the ASK app resulted in the team using the chat interface to log bugs or notes for discussion.

Users in the dashboard were not able to send messages to visitors if the message was not in response to an unanswered question. This was a limitation that we had built in without realizing and a quick test made it very clear that dashboard users should be able to send messages at any time after a conversation has been initiated with a visitor.

We started noticing latency issues in the dashboard and the api as more people were using the dashboard and mobile app at the same time. This resulted in everything being processed a little slower and inaccuracies at many points. We knew we had to introduce optimizations so things were faster. In some cases of latency, we were able to doctor our interfaces so things felt faster than they were in reality.

We found ourselves using the ASK app as a way to communicate bugs and make notes for testing to discuss later as a team. Most of our problems would have gone unnoticed without an end-to-end test.

After several rounds of internal testing we were ready for a test with real visitors. The first question was—who do we invite? We knew from past projects that our audience is diverse in their use of tech; some are tech savvy, some are not, many own smartphones, but others do not or choose not to use them at the Museum. In previous projects, we had tested with a more exclusively tech savvy audience and this created problems with end products because we were not testing directly with the entire range of our core audience. When testing ASK, we wanted to think about these challenges and test directly with visitors.

Monthly newsletter inviting members to participate in user testing the ASK app.

Because ASK is designed with our most local visitors in mind, we decided to test with  museum members first—those who have a deep relationship with the museum, have been here many times, and have a want to make the visitor experience the best that it can be and are willing to help us do so. Our membership department invited participants in their monthly newsletter and we scheduled our testing groups during the day and evening to attract  different audience segments.

We were surprised at how many members answered this call-to-action and that has helped us schedule people through various rounds of testing.  We are testing with small groups in our American Identities galleries on our 5th floor.  Up next, I’ll talk about what we learned.

iMac kiosks in The Latino List that record video reaction from visitors.

We are also super excited to see how visitors to The Latino List create their own “speaking portraits” at the exhibition’s community voice kiosk, an interactive that was such a successful part of The Black List exhibition that we knew we had to offer it again.  During The Black List visitors were invited to record on-the-spot videos of their response to the question: “How has race made an impact on your life and accomplishments?”  Videos were published to the museum’s YouTube channel and the best of them could also be viewed in the gallery during the course of the exhibition.  I was blown away by the candor, humor, pride, anger, and power in these videos.  One of the most fascinating things about the responses was their diversity and range.  Not only did each individual naturally have their own personal take on the question, but people reflected on how their own race is perceived and experienced as well as how they perceive and experience people of other races.

For The Latino List we wanted to elicit similarly inclusive and reciprocal responses, so the question we pose to visitors this time—in English and Spanish—is: “How has your culture shaped your life and accomplishments? (¿Qué impacto ha tenido su cultura en su vida y en sus logros?). The word “culture” conjures family and community traditions, and certainly one of the things that unite the stories shared by the Latino List participants is the impact and influences that family and tradition have had on their lives and identities.  The word evokes a range of concepts, from race to religion to heritage, without being  limiting or exclusionary: everyone comes from a culture of some kind, whether they abandon it or embrace it, and it shapes the way they experience the world and, to some extent, for better or worse, the way the world experiences them.

This time, we’re expanding the interactive to include not just visitors to the gallery, but anyone, anywhere, through a bilingual iPhone app.  You can record your video response directly on your iPhone, upload it to The Latino List YouTube channel, learn about the exhibition, and watch videos made by other people.

As always, we want to hear from you:  download the app, come to The Latino List, and make a video to share your thoughts about your culture and experiences.

Results from the App Store when searching "Brooklyn Museum." The two apps are not distinct enough and confusion ensues.

We are pretty thrilled that our API has been used to create such a public-facing use and hope that this innovation continues, but we also needed to make some changes to our API terms to help eliminate possible confusion in the App Store.  These changes are especially important now because we’ve seen other developers requesting keys to create their own apps with our content, so it’s likely that we’ll eventually have even more choice in the App Store for our visitors.

Interestingly, this tweet came up just as we were making the API changes.

Recent changes to our API terms now state our name, likeness and logo can’t be used without prior permission and there are a few other app-specific disclaimers that we require such as ensuring apps developed with our content are distributed free and without any in-app advertising.  It should be noted that even though this wasn’t explicitly stated in the terms prior, every developer that has worked with our API and wanted to use our logo has specifically asked our permission and, in the case of Brooklyn Museum Mobile Collection, Adam had conscientiously worked with us to both name and brand the app in a way that worked for all of us.  However, now that time has passed and there are more apps in the store and, likely, more apps on the way – we are making a formal change.

So, what now?  These changes will help us moving forward, but what do we do about the two existing apps in the store?  We’ve been talking with Adam and are privileged that he’s been so open to the problems we’ve been experiencing and open to making adjustments.  As it turns out, he’s been working on a new and improved version of the app that will contain both new features and fix crashing in iOS4.  In addition, he’s moved from Iconoclash to Caravan Interactive and was looking for a way to move the app from his old company account to the new one, which we hear from Apple can’t be done.   Given all of this, it seemed like the best answer was to pull the existing Brooklyn Museum Mobile Collection from the store under Iconoclash and release the new and improved version with different branding under Caravan in the coming months.