One of the tools we’ve leveraged to help us in this goal is called Elasticsearch which is a full text search server.  While not strictly just a machine learning tool, it uses many NLP algorithms (which are machine learning based) in its backend to do more intelligent text searching to match similar phrases. This means instead of doing a ‘dumb’ search that has to match exact phrases, we can do more fuzzy style searching that can locate similar phrases. For example, if we have a block of text that has the phrase ‘Where are the noses?,’ if we did a search using the phrase ‘What happened to the noses?,’ the first block of text would be returned near the top of the results.

This particular use case is exactly what we were looking for when we needed to solve a problem with our snippets. We’ve talked about snippets in a previous post, but just to recap snippets are pieces of conversations between visitors and our audience engagement team about works of art in the museum. Due to their usefulness in not just highlighting great conversations, but also in their acting as a sort of knowledge base, snippets have become an integral part of ASK. This means we’d like to create snippets as much as we can to grow this knowledge base and help spread the wisdom gleaned from them. However, over the course of this process it’s easy to accidentally create snippets for the exact same question which clutters the system with duplicates.  This is problematic not just for search results but also because all snippets go through a curatorial approval process and seeing the same snippets creates unnecessary extra work for everyone involved.

The dashboard automatically queries our Elasticsearch server to look for similar or duplicate snippets during the snippet creation process.

In addition to solving the problem with duplicates, cleaning up the system means we can much more accurately track the most commonly asked questions.  All of this happens as part of a seamless process during snippet creation.  When an audience engagement team member begins the snippet creation process, the dashboard automatically queries our Elasticsearch server to look for similar or duplicate snippets.  These search results show up next to the snippet editor which makes it easy to quickly find if there are existing duplicates.  If a duplicate snippet does exist, the team member simply clicks a “+1” counter next to the snippet.  This increments a number attached to the snippet which we can then use for various metrics we track in the system.

If the ASK team finds a duplicate snippet does exist, they click the “+1” counter. This increments a number attached to the snippet which we can then use for various metrics we track in the system.

Just based on our short time using machine learning tools, it’s clear how powerful the advantages of using these tools are in the here and now.  We’ve already talked about how they’re already improving our response times, metrics, and knowledge base, but that may just be the tip of the iceberg.  The AI revolution is coming and as tools get more sophisticated yet simpler to use, the question isn’t if you’re going to use it, but when and how?

There is a lot of research into Computer Vision right now but most of it focuses on image recognition (identifying a thing in an image like a dog, car, tree, etc.) but for this project we need to do image matching which is comparing the entirety of an image to see if it is the same or similar to another image. The most common use of image matching is a reverse image search. Google has a service like this where you can give it an image and it will search the web to see if the same photo is out there on the web somewhere even if it has been slightly cropped or edited. For example, I did a reverse image search to see if one of my photos is used anywhere else and found a tumblr blog and a few sites in Japan that are using it.

Original image on the left. Modified version found through a reverse image search on the right.

What we’re doing is quite different than just trying to find the same photo. We want to see if part of a photo matches the official image of an object in the museum collection. After some searching, we found an open source tool called Pastec. (Special thanks to John Resig for bringing Pastec to our attention.) From it’s GitHub repo homepage, “Pastec does not store the pixels of the images in its database. It stores a signature of each image thanks to the technique of visual words.” Because it uses parts of the image to make “visual words” and we are trying to match an object that would be in a photo, not the photo itself, it seemed like a promising option. (The visual words concept is very interesting and if you’d like to read more about it, check out this Wikipedia page.)

Visitor image on the left. Brooklyn Museum original image on the right.

To test out Pastec, we compared visitor images to the official image on a few objects in the collection. It matched flat 2D objects, like paintings and photographs, about 92% of the time but only matched 3D objects, like sculptures, about 2% of the time. This makes sense since recognizing 3D objects is a whole separate and even harder problem which Pastec isn’t designed to do. Again from the GitHub homepage, “It can recognize flat objects such as covers, packaged goods or artworks. It has, however, not been designed to recognize faces, 3D objects, barcodes, or QR codes.”  Still, recognizing a huge portion of the collection 92% of the time would be very useful so we decided to move forward.

The next step was to upload the main image for every object in the collection through Pastec so it can index it and find the visual words. The indexing took about 36 hours on an m1.medium AWS EC2 instance which has two ECUs (EC2 Compute Unit) with 3.75 GB of ram. One ECU is roughly equivalent to a 1.0-1.2 GHz Opteron processor. After indexing all objects in the collection we tested searching for an image match and were getting results back in under 1 second. We also tested search speed with only 2D objects in the index, since they were so much more likely to match than 3D objects, but the difference in search speed was so minimal we decided to index them all. With that information we were ready to implement image search results into the dashboard.

When chatting with visitors, the ASK app uses iBeacon data to show the team a thumbnail grid of what objects are near a visitors current location. This location data is the main research tool for ASK team because it helps them see the object’s data, past conversations about the objects, and objects nearby the visitor. Image matching is a supplement to it and the design should reflect that. So in the dashboard if there is a match, that object will be highlighted above the main grid of images, and if there isn’t a match, nothing extra will show.

If we can match an image, we display it above the other images, so the ASK team has quick and easy access without having to hunt for it.

In the end, the experiment of “Can we use image matching to supplement iBeacon data?” was well worth the effort because after about a week’s worth of work we’re able to shave a few seconds off of the time it takes to get back to visitor questions making that experience more seamless. There is a secondary benefit as well because now that the image index is created it has potential to help solve more problems down the road and could be a great research tool.