Selectively Flying Blind After Android User Testing

ASK Brooklyn Museum for Android is now available on Google Play. We had one early quandary, but this was a fairly straightforward development process. That is, until we got to user testing.

Android development is tricky. There are a lot of devices all running different system versions in various states of updates with hardware manufactured by different parties, distributed independently or by various carriers. By comparison, iOS is a fairly controlled environment; we knew we could develop the iOS version of the app in house, but it was clear to us that an external team would need to tackle Android and we contracted with HappyFunCorp.

In the beginning of our Android development process we looked at our Google Analytics to figure out which devices/systems were in our majority and this became our supported device list. Simply put, there are too many devices running too many systems to be able to support all of them, so you have to pick your battles. We settled on a combination of support for devices running at least Android 4.3 and with Samsung Galaxy 4S (and higher) and Nexus 5 (and higher) hardware.

As with our iOS release, we did a number of invited user testing sessions onsite at the Museum. Many of these sessions were attended with just a few users giving us their time. Each session helped us surface bugs, but it was difficult to get a critical mass. One thing we started to see, however, is that at each session we had users show up with hardware that was not on our supported list and, inevitably, we saw a lot of bugs on these devices. It was the very well attended session with Bloomberg employees that helped us identify a trend, come to terms, and make a critical decision that will affect all Android users (for the better).

Most of the bugs we found on unsupported devices came down to problems with beacon ranging. We could reliably require Bluetooth on supported devices, but on others we’d see two problems. First, if a device didn’t have bluetooth support the user couldn’t use the app at all. This requirement on iOS devices made sense because of the near ubiquity of Bluetooth on current hardware, but was more difficult on the plethora of Android hardware situations. Second, if users were on an unsupported device beacon ranging was hit or miss often causing problems like device sluggishness or outright device crashes.

We had three options. Option one would be to not allow the download on unsupported devices, but this would mean some users could find it in Google Play and other users wouldn’t see it at all. This presented a nightmare for messaging—”We have an Android app….sort of…”  Option two allowed the download, but many users would experience bugs and it would be difficult to communicate why.  Option three would be to turn off bluetooth/beacon ranging for all unsupported devices, but this would mean the ASK team would not see a user’s location.

In the end, we went with option three and decided to turn off beacon ranging for all unsupported devices. This means ASK will work on most Android devices, but on devices where we’ve disabled beacon ranging, the ASK team will be flying blind with “no location found.” They can chat with users, but the object information won’t be at their fingertips so readily in what we hope are users who represent the very edge case.