User testing sessions are a critical part of the process.

Android development is tricky. There are a lot of devices all running different system versions in various states of updates with hardware manufactured by different parties, distributed independently or by various carriers. By comparison, iOS is a fairly controlled environment; we knew we could develop the iOS version of the app in house, but it was clear to us that an external team would need to tackle Android and we contracted with HappyFunCorp.

In the beginning of our Android development process we looked at our Google Analytics to figure out which devices/systems were in our majority and this became our supported device list. Simply put, there are too many devices running too many systems to be able to support all of them, so you have to pick your battles. We settled on a combination of support for devices running at least Android 4.3 and with Samsung Galaxy 4S (and higher) and Nexus 5 (and higher) hardware.

As with our iOS release, we did a number of invited user testing sessions onsite at the Museum. Many of these sessions were attended with just a few users giving us their time. Each session helped us surface bugs, but it was difficult to get a critical mass. One thing we started to see, however, is that at each session we had users show up with hardware that was not on our supported list and, inevitably, we saw a lot of bugs on these devices. It was the very well attended session with Bloomberg employees that helped us identify a trend, come to terms, and make a critical decision that will affect all Android users (for the better).

Bloomberg employees helped us test both our iOS and Android app prior to launch.

Most of the bugs we found on unsupported devices came down to problems with beacon ranging. We could reliably require Bluetooth on supported devices, but on others we’d see two problems. First, if a device didn’t have bluetooth support the user couldn’t use the app at all. This requirement on iOS devices made sense because of the near ubiquity of Bluetooth on current hardware, but was more difficult on the plethora of Android hardware situations. Second, if users were on an unsupported device beacon ranging was hit or miss often causing problems like device sluggishness or outright device crashes.

It was during the Bloomberg testing session when we could see a number of users all having the same problems that issue became really clear.

We had three options. Option one would be to not allow the download on unsupported devices, but this would mean some users could find it in Google Play and other users wouldn’t see it at all. This presented a nightmare for messaging—”We have an Android app….sort of…”  Option two allowed the download, but many users would experience bugs and it would be difficult to communicate why.  Option three would be to turn off bluetooth/beacon ranging for all unsupported devices, but this would mean the ASK team would not see a user’s location.

When an unsupported device is in use, a “no bluetooth” icon appears on the ASK team dashboard alerting them to the situation.

In the end, we went with option three and decided to turn off beacon ranging for all unsupported devices. This means ASK will work on most Android devices, but on devices where we’ve disabled beacon ranging, the ASK team will be flying blind with “no location found.” They can chat with users, but the object information won’t be at their fingertips so readily in what we hope are users who represent the very edge case.

As I start to outline what’s going on here, keep in mind there seems to be a general fear in the world about the perception of asking stupid questions. In the early days of user testing, we heard this time and time again and it was clear from the onset that if we were going to raise the bar in the interaction we were going to have to give people a safe space to engage. From the get go, we made the decision that we wouldn’t onboard users asking for personal information, we wouldn’t collect a login, or even a name to get started. Essentially, we know if you return, but we don’t know anything about you because we don’t ask for any information up front.

In iOS, we use an Apple ID to recognize a user from multiple devices (if they own and iPad and an iPhone, for example) and, as long as they use the same Apple ID it carries a user through when the user upgrades their phone. All of this is pretty seamless on iOS because it happens out of view of the user and, bonus, we are not storing personally identifiable information, so we’re where we want to be on privacy.

Going with the Google ID to recognize users across devices may be problematic for ASK engagement.

Android operates a little differently. We can use a Google ID, but this action happens in view of the user and this is what creates a conundrum. On first use, a user is presented with Google IDs from which to choose. The good news is we still wouldn’t be storing personal information, but the really bad news is twofold. First, it’s impossible to tell users that we are not storing personal information and the natural assumption may be that by selecting an ID it feels like we’d know them more deeply than we do. Second, a known user ID associated with the app may significantly change user interaction because it runs counter to what we’ve heard from users. Namely, people like the anonymity of the app for fear of asking what might be perceived as a stupid question; the app feels like a safe space to explore.

The issue, for ASK, is a big one. A known user ID may change that behavior and in the interest of time, we’ve decided to go with Device ID (seamless to users) and then think about switching to Google ID post-launch when we have enough space to accomplish focus group testing around the change.

In the end, we’ve decided to use the device ID, but going this route only helps us identify the same user on that particular device; if a user upgrades or uses a different device they look like a new user to us. Using the device ID means we can’t effectively welcome someone back, see their conversation history, and make recommendations that build on that relationship.

We’re okay with this as a stopgap measure because it’s the most surefire way for us to retain the engagement that we know has been working. Post launch, however, this will be one of the first things we have to think about re-factoring because those long term goals of building relationships are key. As we rethink this, we’ll need to do a lot of focus groups paired with A/B testing to see if engagement changes with a Google ID and, if so, how much.

When thinking about Android expansion we had to re-evaluate how we were staffing mobile. In our first year it was invaluable to have someone on the internal team dedicated to iOS mobile development because the process at that time was more iterative. We were developing features and testing with users as we went along—having someone on staff to make changes as we discovered them was critical. Moving beyond that stage we had to reconsider the most efficient way of working and the best route forward would be to shift from staffing internally to hiring a firm. We contracted HappyFunCorp (HFC) to develop the ASK app for Android using our iOS app as a model. HFC is also handling our ongoing iOS maintenance allowing us to shift away from staffing internally in full.

The Android version of the app will function the same as iOS and in a future post I’ll talk about some of the changes that make ASK feel more appropriate for this platform and one of the bigger challenges we hit. Mostly, though, the transition to Android has been straightforward and, luckily for us, that meant we could concentrate on more vexing issues like how the app detects beacons and sends locations back to the ASK team. What follows is a lengthy post that details how our code works and the adjustments we’ve made. We are also taking this opportunity to release all of our code related to beacons in both iOS and Android regardless of the state it’s in—read on. 

In Android, permissions are granted in a one-step process at initial run. iOS, by contrast, stages permission actions as a user needs them. This delay in granting access to bluetooth may be causing “no location found” on start messages because we can’t range for beacons and build our array quickly enough.

So let’s take a look at the problem at hand. We’ve been seeing “no location found” on 15% of messages sent to the team with and high proportion of those on a user’s first message in a chat. We have a hunch this is likely because the beacon ranging starts too late. In iOS ranging only begins when a user turns bluetooth on and this prompt occurs very close to when a user would send that first message; turning on bluetooth is one of many things a user needs to enable and all of these prompts have been carefully staggered, so that users are not overwhelmed at the start. In Android, a user is asked for all permissions as a one step process up front and this means ranging for beacons starts right away. We think this change will help enormously, but we are still testing and this is to be determined. 

The other cause we’ve seen with “no location found” is attributed to the human error side of the equation. We have an admin tool that keeps track of our beacons and assigns them to museum locations. The beacon may be missing from that tool having been entered incorrectly (or not at all). To solve these issues the BKM web developer team enabled server-side logging; each time a beacon is sent to the dashboard that is not in the beacon database we’ve logged it in an admin tool so we can periodically use the data to chase down these problems.

Admin tool shows when we receive an invalid beacon ID likely the cause of a data entry error in our beacon tool.

The HFC team has also coded a debugger tool within the app which shows, in real time, all of the beacons in the application’s cache and the closest beacon the application would send with a message. This is helps us get visibility beyond the Estimote app because it shows what’s happening in our own application. Chris Wilson at HFC explains:

We now have a Chat/Beacon Log page that shows the list of messages sent since the list was reset. It has the beacons (with message optionally visible) showing the message timestamp, and the beacon’s major and minor ids. It uses live beacon data from the museum’s web api to determine if the beacons associated with these messages are valid, invalid, or if no beacon info was sent. The messages in the list are then color coded based on these designations. For easy visibility, messages with valid beacons are colored green, invalid designations are colored yellow, and messages sent with no beacon data are colored red. There are also total counts for each designation visible on the log screen.

Mobile side debugger tool developed by HFC to show beacons being ranged and which beacon would be sent with a message if a user were to hit send.

Our coding changes have not just been limited to the addition of debugging tools and as we discuss improvements it’s worth reviewing how the beacon code in our ASK app actually works. In a nut shell, as a user walks around the building the app ranges beacons as encountered by the device and builds an array with the associated beacon distance. When a user composes a message and hits send, we send the closest beacon to the user that is in the array. The following bulleted lists are coming direct from HFC:

Here’s the way the (newer) Android code works—

• On app start, the app begins ranging beacons using the Android Beacon Library.
• About every second the beacon ranging returns a list of beacons that have been seen.
• It cycles through each beacon and adds them to the cache, removing old copies of beacons that have been ranged with new distances.
• It removes beacons from the cache that have outlived the TTL (currently 2,500 ms – this is something we can try to tweak to improve accuracy).
• It then cycles through the list to determine which beacon is closest, replacing the closest beacon variable with this beacon. TTL on this is 3 minutes.
• The closest beacon variable is picked up and sent along to the chat server when the user hits the send button.

Here’s what we know about the way the (older) iOS code works—

• On app start, it the app starts the beacon ranging. However, the bluetooth check is only conducted when the user tries to send a message. Ranging requires bluetooth to be on, so this may be the source of “no location found” issues.
• When beacon ranging is run, an array of beacons, sorted by proximity is returned every second. If the proximity is unknown, the beacon is removed from the array. Only the first beacon (the one with closest proximity in the array) is used until the next ranging cycle. If the cache is empty, that beacon is added to the cache.
• If the cache is not empty, then the first beacon on this list (the one with closest proximity) is compared with the last beacon found with the last cache object. (1) If the major/minor ID of the beacon is the same AND the distance is less than the object, it adds the beacon to the list. If the major/minor ID of the beacon is the same and the distance is more, then it is not added to the cache. (2) If the major/minor ID of the beacon is different from the last object, it is added to the bottom of the cache.
• The last beacon in the cache array is grabbed along with the message when the user taps the “send” button in the chat message. If the beacon has been in the cache for more than 5 minutes, no beacon information will be sent.

So, what are the differences?

• Beacons aren’t removed from the cache in the iOS app, so duplicate beacons with different distances are added.
• Rather than comparing all new beacons found to all cache beacons and updating existing beacons and adding new ones as in the Android app, the iOS app compares only the last beacon found to see if it is closer than the last cache array beacon.
• There is a TTL of 5 minutes in the cache in the iOS app, whereas the TTL on beacons in the cache in the Android app is 2.5 seconds, and the TTL of the closest beacon if no new beacons have been ranged is 3 minutes.
• In the Android app, in addition to the short lived cache of beacons, there is also a closest beacon variable set in case there are no ranged beacons for a period of time. This beacons is then sent with messages if it has been more than 2.5 seconds but less than 3 minutes. In the iOS app there is no concept of a closest beacon variable.

We are now going to begin the process of testing Android with users to see if these changes have helped and, if so, we’ll start to port these lessons learned back into the iOS code after April. In the meantime, given how many people are working (and struggling) with beacon deploys, we’ve decided to release both sets of code in the state they are currently in along with the mobile side debugging tools. Having a fresh set of eyes from HFC looking at the code has help a bunch and we hope having many more eyes on this code will only help everyone.

Lastly, I’d be remiss if I didn’t take this opportunity to talk a bit about our funders as related to this post in particular. ASK Brooklyn Museum is supported by Bloomberg Philanthropies and one reason we are releasing this code today is the amount of encouragement and enthusiasm that has come from the Bloomberg team toward information sharing in all states of a project’s progress. This blog, our lessons, and our code are published in the largest part due to their support; we are honored to be as open as we are because of the standard they have set among their grantees.