Up until very recently distributing test versions of an app had a limitation of up to 100 devices. This was great in theory but those 100 devices were over the course of a whole year and surprisingly we’d already used about 20 of those device tokens just on our own internal testing. Reaching that 100 mark wouldn’t take much.

Of course the other important factor was making the process of having a visitor test our app quick and easy. Distributing a test app to someone’s device was a several step process. We were using TestFlight internally for distribution of our app which requires a tester to create a TestFlight account and ask to join our app distribution list. This then registers their device UUID—used to uniquely identify an Apple device. But it doesn’t end there, each time a new device is added to a distribution list a new version of the app and various other Apple certificates need to generated…painful at best.

So to try and circumvent the process we made the decision to use our own devices and hand them out to visitors to test with. This seemed like the easiest approach. We’d no longer have to worry about hitting that 100 device limit and we’d totally bypass the pain of installing the app on a new device. This also gave us full control over our testing environment. We’d know what iOS version was being run and set any device settings accordingly—turning notifications on etc…

Our stock of iPod Touch devices ready to go for user testing.

We had seven iPod Touches (5th generation) left over from the very first test. We thought we’d created a perfect testing environment…

That’s when we starting learning about the limitations of an iPod Touch. In our first round of internal testing we had some of our team using their own devices (mainly iPhones) and various other team members using the iPod Touches. Interestingly, we noticed that the Bluetooth signal kept dropping out on the iPod Touches, but not once did that happen on the iPhones. This was a key feature of our app to detect visitor location using Bluetooth LE and iBeacon technology. It wasn’t acceptable to have the Bluetooth connection drop out intermittently during member testing. In addition we also saw that the wifi signal dropped out on occasion which we rarely saw with the iPhones. Again this was problematic—no wifi meant no notifications arriving to tell a visitor the question was answered. In fact questions couldn’t be asked in the first place as no means of network connectivity could be established without wifi on the iPod Touches.

We realized at this point that there was a fundamental problem testing our app with the iPod Touches. With an iPhone if the device auto locks (this is usually managed in settings with a default of five minutes) notifications will still arrive but with an iPod Touch if the device auto locks the wifi drops out and notifications will never arrive.

So controlling the environment needed a little more control. We had to set the iPod Touches to never auto lock which would keep a steady-ish wifi signal. We also had to add code to start and restart the iBeacon detection manager in an effort to keep Bluetooth enabled. We were working around the problems, but it was starting to feel a little too far from a real world scenario. We were forcing behaviors that didn’t mimic those used by real app users.

The iPod Touch doesn’t have a vibration mechanism making notifications in a museum a tricky thing.

One final interesting point about the iPod Touches was also around receiving a notification. After a couple of rounds of member testing we had feedback asking for a vibration when a response to their question arrived on the device, this way they wouldn’t keep looking at the device to see if a response had arrived. Sure no problem—and thats when we found out that an iPod Touch doesn’t have a vibration mechanism—who knew! Yes we could add a subtle sound (and for testing purposes we did) but long term that sort of interruption in a museum setting is never desirable and a vibration is definitely preferred.

Along comes September 2014—Apple Worldwide Developers Conference (WWDC). During that conference Apple announced that they were partnering with TestFlight to allow even easier distribution of test apps. We’d now have up to a 1000 device limit for testing and the process of installing the test version on someone’s device was greatly streamlined. At the end of October 2014 Apple had officially integrated with TestFlight. We quickly made a decision that to really be testing our app we needed to have our visitors install it on their device. In a couple of weeks we are going to invite visitors to do just that and are excited to see more real world results!

To achieve this we placed beacon hardware in the galleries so that when a visitor asks a question the app detects the closest beacon; the visitor’s location is then transmitted to our audience engagement team who will answer queries.  Each beacon has location information assigned to it so we know where the visitor is in the museum. We initially chose two beacon vendors to begin our beacon testing:  Estimote and Kontakt. Both are big players in the newly founded beacon vendor world. And so we set out ordering a handful of each and began our tests.

The first round of testing ended with disappointing results. We experienced very erratic results…very, very erratic. We could be standing right on top of a beacon and it wouldn’t be detected—although strangely, it might detect the beacon on the other side of the room.

With test one behind us we started to look a little deeper and do our research into what was going on. Beacon technology wasn’t going to be as plug and play as we’d hoped. We referred to endless blog posts of experience working with beacons.  Stephen Elliott founder of Llama Digital had written an insightful post on beacon positioning for best detection results.  The article suggests using brackets to mount beacons in such a way as to achieve optimal signal strength. Because of the gallery layouts and general architecture we aren’t able to follow this approach so onward we went. We spoke with Jess Elder and Scott Newman over at the National Geographic Society. They’d recently added beacons into one of their installations. They found mounting beacons on the ceiling gave them more accurate results. Although they also cited similar frustrations with inconsistent beacon detection. Placing beacons on the ceiling wasn’t an option for us either—in many cases our ceilings are incredibly high and getting lifts through the galleries for placement, testing, and battery changing was not going to be practical over the short or long haul.

We performed various other tests, including changing power settings on the beacon which would in theory make the signal weaker or stronger. We tried adding and removing beacons in a room. None of these tests were giving us the consistency we’d hoped for. One gleaming limitation is of course the technology used to transmit beacons—Bluetooth low energy technology. Bluetooth technology is based on radio waves. Radio waves are inherently flaky because of interference. In our case interference might mean any work of art, vitrine, installation partitions, or even other visitors in the gallery! Ultimately each gallery has its fair share of radio wave blockers.

Installation of beacons has been informed both by accuracy and detection for our app as well as minimizing intrusiveness in the gallery as much as we can allow.

In the end we found that basic detection was not enough and so the app works a little harder to determine which beacon is the truly closest one. This involves caching beacon results and time of detection as well as using the accuracy metric from the Apple Core Location API. This metric gives the number of meters the Apple device is from the beacon…in theory.  The accuracy measure is at times wildly inaccurate but tracking that measure and averaging results helps give a more ‘accurate’ result. Beacon placement was very important to make sure we were getting decent coverage in the space we were attempting to identify. We placed the beacons roughly 7-8 feet high and 15-20 feet apart in each gallery. Having the beacons sit higher on the walls works well to remove the beacon from the general range of vision.

One thing we were certain on was that Estimote and Kontakt were both decent beacon vendors. We ultimately chose Estimote because we’d found their developer community to be sizable and very active. The Estimote forums discuss issues which Estimote then fix in a relatively timely manner. The SDK has had various releases as has the firmware. These releases have addressed some big improvements in battery life and security around the beacon.

‘Icy Marshmallow’ is the best choice for us for now as we await more color choices.

Of course the one big difference between the two beacons is the look. At the museum the aesthetics of each gallery are very important. A great deal of time is spent designing the galleries and adding beacons throughout was a sticking point for some. The Estimote beacons come in a limited number of colors and have a distinctive look. As a starting point we’ve chosen the least intrusive color ‘Icy Marshmallow’ and we are eagerly awaiting more color options which we understand are coming by the first quarter of 2015.  The decision around placement of the beacons has been informed both by accuracy and detection for our app as well as minimizing intrusiveness in the gallery as much as we can allow.

Adhesive was also a concern as objects in galleries are often changed and so might the beacon placement. We had to be sure the adhesive wouldn’t damage the wall and would be easily transferable to the next location. In most cases Estimote beacon adhesive seemed to work ok with several exceptions where we’ve doubled up with Scotch Removable Mounting Squares.

So all in all, it’s been a fantastic learning experience for our team—not only working with new technology and evolving software and hardware SDKs, but also how they can best interact and co-inhabit a delicate environment like the traditional museum gallery space. This is only one of many ways we are trying to merge the physical and digital experience, and look for a simple, elegant application of technology that is invisible to the visitor, yet enhances the experience.

What’s next for us? Our Beta test of the ASK app will be happening soon. We are hopeful this will reveal more insights into iBeacon technology and its best application for us.

Our core experience principles for the ASK app are simplicity, usability and ease. A key consideration in the development process is a solid codebase that can be extended and ported across various platforms. A codebase that can be iterated upon, evolved depending on new requirements and usage data, and in the future—the addition of new features.

With the completion of our pilot visitors were given an iPod Touch and asked to use iMessage to pose questions to a team of on-hand experts in the museum. We used these findings to help guide the prototyping process. Knowing that 83% of our mobile traffic was coming from iOS devices, our efforts have been focused on developing the app for iOS Apple devices—specifically focusing on iPhones.

We looked to existing models that worked well rather than re-inventing the wheel.

The main functionality of the app is around messaging. It was important not to reinvent the wheel as iMessage works well—we wanted to build upon a proven messaging system. The first challenge was creating our own version of messaging which wouldn’t need a set of instructions on how to use it. So we stuck to the iMessage fundamentals and followed the usability and interaction pattern that by now is ingrained in so much of how we communicate these days.

Of course Apple don’t offer a ‘Messaging API’ to just plug and play but we were able to create a similar look and feel with our own design aesthetic. The app allows a user to ask a question with the option of sending a photograph. Capturing images is a key feature to help give context to the question being asked. The technology used for sending and receiving the message notifications is the Apple Push Notification Service. This is a server run by Apple and the same system that iMessage uses to send messages back and forth between devices without using a Telecommunications service provider.

Our start screen begins with an onboarding process designed to be conversational.

We realize it will be important for the visitor to know that they are speaking to an actual person and so to make sure this comes across while still aiming to keep the app simple, we gather two key pieces of information about the visitor—name and email address. The name helps to remove some of the formality while the email address is useful if the question can’t be answered by the expert and instead will be forwarded onto someone else who could respond via email at a later time.

We focused on making this screen feel like a conversation to keep it consistent with our theme. This screen helps give an introduction while gathering key information; and with literally a couple of clicks you’re ready to start asking questions. We were sensitive about making this process easy and quick without it feeling like a long drawn out registration sign up.

Another issue we addressed was the visitor’s location when viewing the app. When not in the Museum the ASK app isn’t available. We use GeoLocation to determine where the visitor is and display the app accordingly; if the visitor is outside of the Museum we want to offer information about their visit before they arrive. Then when the visitor arrives the ASK app becomes available. This also allows us to create a nice before/during/after Museum experience.

We’ll be beta testing the first version of the ASK app soon. More details on the app including our implementation of iBeacons will be forthcoming in future blog posts.