There is a lot of research into Computer Vision right now but most of it focuses on image recognition (identifying a thing in an image like a dog, car, tree, etc.) but for this project we need to do image matching which is comparing the entirety of an image to see if it is the same or similar to another image. The most common use of image matching is a reverse image search. Google has a service like this where you can give it an image and it will search the web to see if the same photo is out there on the web somewhere even if it has been slightly cropped or edited. For example, I did a reverse image search to see if one of my photos is used anywhere else and found a tumblr blog and a few sites in Japan that are using it.

Original image on the left. Modified version found through a reverse image search on the right.

What we’re doing is quite different than just trying to find the same photo. We want to see if part of a photo matches the official image of an object in the museum collection. After some searching, we found an open source tool called Pastec. (Special thanks to John Resig for bringing Pastec to our attention.) From it’s GitHub repo homepage, “Pastec does not store the pixels of the images in its database. It stores a signature of each image thanks to the technique of visual words.” Because it uses parts of the image to make “visual words” and we are trying to match an object that would be in a photo, not the photo itself, it seemed like a promising option. (The visual words concept is very interesting and if you’d like to read more about it, check out this Wikipedia page.)

Visitor image on the left. Brooklyn Museum original image on the right.

To test out Pastec, we compared visitor images to the official image on a few objects in the collection. It matched flat 2D objects, like paintings and photographs, about 92% of the time but only matched 3D objects, like sculptures, about 2% of the time. This makes sense since recognizing 3D objects is a whole separate and even harder problem which Pastec isn’t designed to do. Again from the GitHub homepage, “It can recognize flat objects such as covers, packaged goods or artworks. It has, however, not been designed to recognize faces, 3D objects, barcodes, or QR codes.”  Still, recognizing a huge portion of the collection 92% of the time would be very useful so we decided to move forward.

The next step was to upload the main image for every object in the collection through Pastec so it can index it and find the visual words. The indexing took about 36 hours on an m1.medium AWS EC2 instance which has two ECUs (EC2 Compute Unit) with 3.75 GB of ram. One ECU is roughly equivalent to a 1.0-1.2 GHz Opteron processor. After indexing all objects in the collection we tested searching for an image match and were getting results back in under 1 second. We also tested search speed with only 2D objects in the index, since they were so much more likely to match than 3D objects, but the difference in search speed was so minimal we decided to index them all. With that information we were ready to implement image search results into the dashboard.

When chatting with visitors, the ASK app uses iBeacon data to show the team a thumbnail grid of what objects are near a visitors current location. This location data is the main research tool for ASK team because it helps them see the object’s data, past conversations about the objects, and objects nearby the visitor. Image matching is a supplement to it and the design should reflect that. So in the dashboard if there is a match, that object will be highlighted above the main grid of images, and if there isn’t a match, nothing extra will show.

If we can match an image, we display it above the other images, so the ASK team has quick and easy access without having to hunt for it.

In the end, the experiment of “Can we use image matching to supplement iBeacon data?” was well worth the effort because after about a week’s worth of work we’re able to shave a few seconds off of the time it takes to get back to visitor questions making that experience more seamless. There is a secondary benefit as well because now that the image index is created it has potential to help solve more problems down the road and could be a great research tool.

This decision was informed by the lessons Brian and I learned at General Assembly’s Mobile Prototyping Bootcamp. Though the workshop was centered on mobile apps, we picked up some cross-disciplinary techniques like creating prototypes out of paper and code. We learned that prototypes were a great way to test out an idea without investing too much time and resources to learn if we were correct in our assumptions and to answer any questions we might have about the project’s design. Also with the prototypes we made in the workshop, we had a common language with which we can talk about features and user flows. You would be surprised at how designers and web developers can describe and think of the same action in completely different ways. Prototypes can help bridge disciplines.

As I briefly mentioned in my last post, I created prototypes to evaluate each framework to learn what we were getting into. Two questions I wanted to answer were: “How easy it is use to this framework?” and “Does this framework have everything we need to build the dashboard?” I figured the best way to answer these questions was to use the frameworks to build the same web app and then to compare and contrast.

My approach to creating a prototype was to pick a critical user interface element and to quickly build it out. As I mentioned previously, I picked the chat window.

My goal was to quickly build it out; which meant that I can disregard best practices and settle for an ugly design. A prototype is not meant to be released publicly since the feature being prototyped could be immature or the code could be buggy. It is for demonstration purposes, for “drafting” a feature or for testing an idea. If the chat window was usable, it was a working prototype.

By making a prototype I learned that if we picked Backbone.js, we would have more code to maintain because unlike Angular, Backbone.js is a minimal framework that doesn’t have as many features that Angular has.

After we chose our code framework, I built another prototype, this time using our chosen framework, Angular. In addition to implementing the chat window, I also prototyped the queue and the “research pane,” where hosts could tag conversations with related art objects.

Prototype of our dashboard web application.

While visual designs could give us an idea of the look and feel of the dashboard, a working prototype could gives us an idea on how what it would be like to use the dashboard. Visual designs are static while prototypes are interactive; interface elements like buttons and text dialogs are clickable.

The dashboard app in action during a recent user testing session.

A side benefit of making prototypes is that they can serve as documentation that could help us plan and communicate how the dashboard worked. For example, Brian and I were discussing an animation that occurs when a user selects a name. After a user selects a name, a confirmation dialog should pop-up with an animation. I wasn’t too sure what direction he wanted to go in, so I quickly coded a few examples.

Another example is when we were designing the snippet creation flow. Brian had designed the following user experience: The user is presented with a conversation and they are able to select individual messages to create a snippet of that conversation. I wanted to confirm that Brian and I were on the same page in terms where the user was able to click and what would happen after the user clicked a message. You can view this prototype here: http://codepen.io/amycheng/full/cLIBw/.

Visual designs and code prototypes were being created concurrently, which led to one influencing the other. This also prevented designs and code prototypes from going too far in the wrong direction. Working this way helped us answer questions like: Does this user flow make sense? Is the screen too busy? Is this functionality too complicated?

At one point in the project, Brian was in the process of finalizing designs for the activity tab and I was in the middle of building a quick prototype of the activity tab. We discovered that the activity tab, which could potentially contain the conversation queue, a conversation window, and a research pane with any number of related art objects, could look cluttered on narrower computer monitors (Brian was designing on a much wider computer screen than mine). This discovery allowed us to come up with strategies to mitigate the differences between screen sizes.

It was a good idea to create prototypes because it helped us to clarify our thinking about the user experience and code architecture.

For the dashboard, we had to develop it in a timely manner and still be able to make changes to the codebase without breaking it (think Jenga). Also, we wanted the app to be easy to maintain over time, since it would be a web app that we would be using for years. The best case scenario is an app that was coded in such a way that it is quick to build, is easy to maintain and is easy to adapt to any unforeseen needs. The worse case scenario is unruly legacy code—codebase that degrades over time—and spaghetti code—a disorganized code base.

Legacy code degrades because people forget how to use the code. This could be due to the lack of documentation about the code base or obtuse application architecture. With spaghetti code, it becomes more of a chore to find the causes of bugs and implementing a new feature becomes more involved and troublesome, since introducing a new part into a disorganized system might cause problems with existing parts of that system. Spaghetti code inevitably leads to legacy code.

There are a few ways to mitigate legacy and spaghetti code. There could be a dedicated effort to write good code documentation and to establish a logical app architecture. We pledged to ourselves to do those things, but also thought it be a good (and a smart idea) to use a framework (in the case of the dashboard, a Javascript framework). Coding frameworks provide guidelines and conventions for how code is structured and organized.

Avoiding the mess of legacy and spaghetti code by leveraging frameworks in developing our dashboard—the web application our audience engagement team will use to answer incoming questions.

Using a framework is an easier way for a team to provide organization to their codebase. In addition to a kind of application blueprint, some frameworks also provide abstractions and functions for common development tasks (ie. communicating with an API, implementing animations, data modeling, rendering data to the user). Developers will not have to start from scratch and write their own, idiosyncratic functions for common tasks. Instead, team members would be using the same library of methods and variables. Some frameworks (especially free, open-source ones) also have large communities of developers supporting each other in using that framework. Using a framework, would allow us to draw upon that community as a resource.

It seems like every day there is a new JavaScript framework popping up, but I focused our search to two frameworks: Backbone.js and Angular.js. I chose to look at those two because both frameworks are being used in live web apps by large web companies. Both have good documentation (which makes the framework easy to learn) and large communities.

The way I judged Backbone and Angular was by building a quick and dirty prototype in each. I chose to build a feature that we knew would be in the final version of the dashboard app: a chat window. This chat window allows the user of dashboard to choose a visitor who is waiting in a queue. Then the user of the dashboard can continue the conversation with the visitor. After building the same feature in both frameworks, we decided that Angular was best suited to our needs.

Angular has been criticized for being “highly opinionated,” which means that the framework has a strict application blueprint. For us, Angular having opinions and conventions for application structure meant that we can build something really quickly because we did not have to spend too much time in the planning stages. Angular’s guidelines for building a web app were strict, but not too rigid. Angular strongly encourages developers to write modular code, which means pieces of code can be re-usuable and dropped in when its needed. Every kind of function and variable has its place, which helps with fighting spaghetti code.

Angular also includes a large library of functions for a lot of tasks that we hope to accomplish with the Dashboard app. Tasks like animating user interface elements and interacting with our custom API. With Backbone, we would have had to write a decent amount of boilerplate code (that will likely become legacy code). With Angular, developers won’t have to spend time re-inventing the wheel and they won’t have to spend time re-learning how and why their team mates wrote a certain function.

In addition, unit and integration testing are core features in Angular. Unit and integration testing ensures that any feature implemented in the future does not break existing features. Tests are actually bits of code that makes sure other bits of code are working properly. Angular makes writing tests easier, which makes it more likely for developers to write them (thus preventing legacy code). Tests can also serve as documentation on how the code base works.

Using a framework does not guarantee completely against legacy and spaghetti code, but they can provide a strong foundation for building large apps, like our dashboard.

Having experienced the software development cycle fall short of expectations a few times, we’ve decided to look at a more Agile software development process, perhaps with some skepticism since its rhetoric tends to sound a bit like Scientology or Crossfit, but with code. Much like any ideology, Agile is split into several camps including Scrum, Kanban, and eXtreme Programming. Kanban was inspired by manufacturing processes and being able to change the design of a thing if a problem is discovered, Scrum involves a full-time “Scrum Master” and eXtreme Programming is known for it’s focus on pair programming. After reading Jonathan Rasumusson’s The Agile Samurai, we decided to start adopting pieces of Agile methodologies into our latest web project.

Previously in this blog you we’ve introduced our mobile development and transforming kiosks into the greater ASK program. On the other side of the conversations happening on our mobile app is our audience engagement team, who will need to field questions from several people at once being outnumbered by curious visitors while bringing up relevant information on what artwork is near the user and more. This is software for a job which hasn’t existed in the past, exactly. Unlike building, say, a blog where its use cases are well established and understood, we needed to be able have a solid and clear understanding of what this dashboard would do.

We adopted Agile user stories so that our four-strong web developer team would ultimately land on the same page. Using a textbook Agile technique, we wrote stories down on physical sticky notes using a strong cryptographic cipher known as my handwriting. Bringing in our stakeholder, designer, and entire team into discussion around these stories helped build a coherent narrative which invoked more of the right questions and helped push decisions along quickly. This, along with standup meetings helped keep our vision consistent with our code. Unlike the waterfall model, the project was subject to change as we discovered new problems to make changes so that we would build a system that would survive reality.

Although sticky notes enforced a consistent narrative, the pieces needed to be built were still siloed in individual developers’ todo lists. This turned out to be problematic since it disjoined work from stories and scattered work into tasks across everything rather than tackling a story at a time. Our first in-house alpha launch fell short of its release date partially due to the underestimation of the amount of front-end work that needed to be done—this was also our first use of a JavaScript front-end framework. Since tasks were not attached to stories, it wasn’t easy to tell, from a project management perspective, which particular stories are affected or even whether tasks belonged in a story and should not have been worked on. Because of that, there was no new knowledge to gain by looking at the sticky notes and thus, everyone forgot about them. We also happened to be using free sticky notes gifted to us by a vendor we’ve worked with, so with the science of wall adhesive still not quite perfected a few stories literally fell through the cracks and weren’t rediscovered until we rearranged the furniture in the office.

After that first release, we migrated the stories into Trello, which is structured so that each “card” represents a story with checklists attached. With that structure, it’s a lot easier to see the time needed for a story grow from its original estimate as tasks are added. Once it reaches a rough “too many tasks” peak, it can then be split into multiple stories with a new time score added in, triggering a reality check on the overall project scope.

Another aspect of Agile we adopted was test-driven development. The dashboard is made up of several layers of code and data, with an SlimPHP API at the core, A basic FuelPHP site as its mantle, and a large and very busy AngularJS application at its surface. If a bug should be discovered, it’s easier to find in a test for just that part of the stack while it’s being built than to build everything and go on a time-consuming multi-layer bug hut. Writing tests hasn’t eliminated bugs, but it has reduced them to mostly display or protocol/connectivity-related issues. In our case, we used phpunit for the SlimPHP and FuelPHP layers, and Jasmine for the Angular layers. Since we’re using Jenkins as our continuous deployment system, we were able to make sure code was only pushed live when tests passed.

Even with unit and integration tests passing, there is no better bloodhound for bugs than a human operator. Before any story can be moved to “Done,” it must go through “Verification” where I and sometimes others recreate the question and answer action that the story satisfies, and only move it to done if nothing is missing. This catches a lot of “it works on my machine” type issues, which can then be tossed back to be revisited and verified again. After verification, we then simulate a test with other department staff in-gallery where the whole flow of the visitor experience is tested holistically.

We’re not only inventing a new type of application with our dashboard, but also a new type of workflow for a new type of job to cover it. The moving target of the scope as we tread into new territory is bound to change. Even though we’re not canon orthodox Agile (can’t always pair program!), we’re moving forward with much more agility than we’ve been historically.