One of the tools we’ve leveraged to help us in this goal is called Elasticsearch which is a full text search server.  While not strictly just a machine learning tool, it uses many NLP algorithms (which are machine learning based) in its backend to do more intelligent text searching to match similar phrases. This means instead of doing a ‘dumb’ search that has to match exact phrases, we can do more fuzzy style searching that can locate similar phrases. For example, if we have a block of text that has the phrase ‘Where are the noses?,’ if we did a search using the phrase ‘What happened to the noses?,’ the first block of text would be returned near the top of the results.

This particular use case is exactly what we were looking for when we needed to solve a problem with our snippets. We’ve talked about snippets in a previous post, but just to recap snippets are pieces of conversations between visitors and our audience engagement team about works of art in the museum. Due to their usefulness in not just highlighting great conversations, but also in their acting as a sort of knowledge base, snippets have become an integral part of ASK. This means we’d like to create snippets as much as we can to grow this knowledge base and help spread the wisdom gleaned from them. However, over the course of this process it’s easy to accidentally create snippets for the exact same question which clutters the system with duplicates.  This is problematic not just for search results but also because all snippets go through a curatorial approval process and seeing the same snippets creates unnecessary extra work for everyone involved.

The dashboard automatically queries our Elasticsearch server to look for similar or duplicate snippets during the snippet creation process.

In addition to solving the problem with duplicates, cleaning up the system means we can much more accurately track the most commonly asked questions.  All of this happens as part of a seamless process during snippet creation.  When an audience engagement team member begins the snippet creation process, the dashboard automatically queries our Elasticsearch server to look for similar or duplicate snippets.  These search results show up next to the snippet editor which makes it easy to quickly find if there are existing duplicates.  If a duplicate snippet does exist, the team member simply clicks a “+1” counter next to the snippet.  This increments a number attached to the snippet which we can then use for various metrics we track in the system.

If the ASK team finds a duplicate snippet does exist, they click the “+1” counter. This increments a number attached to the snippet which we can then use for various metrics we track in the system.

Just based on our short time using machine learning tools, it’s clear how powerful the advantages of using these tools are in the here and now.  We’ve already talked about how they’re already improving our response times, metrics, and knowledge base, but that may just be the tip of the iceberg.  The AI revolution is coming and as tools get more sophisticated yet simpler to use, the question isn’t if you’re going to use it, but when and how?

Nitrous makes cars go faster. Servers not so much.

All jokes aside, for most consumer/public facing apps, performance is a very serious consideration. Even if your app has the best design, bad performance can make your Ferrari feel like a Pinto. While performance can mean many things in a technical context, in this post I’ll just be talking about the performance of our back-end.

As I mentioned in an earlier post, we use an internal API to power both our mobile app and the dashboard our Visitor Engagement team uses to chat with visitors. This API has to be able to not just handle requests, but do it in a very performant way. This is especially true given the nature of ASK which revolves around a real-time conversation between visitors and our Visitor Engagement team.

When taking performance into consideration, it’s easy to fall into one of the deadly programming sins: premature optimization. Premature optimization is what happens when you try to optimize your code or architecture before you even know if, when, and where you have bottlenecks. To hedge against this, the solution is simple: benchmark your application. Since I’m just talking about the back-end in this post, application in this context means our API.

When we benchmark our API, we’re not just benchmarking the webserver the API is being served from; we’re benchmarking every component the API is comprised of. This includes (but not limited to) our webserver, API code, database, and networking stack. While our back-end is relatively simple by industry standards, you can see from this list that there are still many components in play that can each have an impact on our performance. With so many factors to account for, how do we narrow down where the bottlenecks are?

Similarly to power plants, back-end servers also need to be able to meet peak demand. photo credit: 2009-09-10 AKW Gundremmingen via photopin (license)

Well first we have to ask ourselves, “What is an acceptable level of performance?” This is a question you can’t answer fully unless you also add the variable of time to the equation. Similarly to the way power utility companies determine how much electricity they need to generate, we also look at the same thing: peak load (see also: Brown Outs). Peak load is simply how much load do you anticipate having during the busiest times? If we know our system can handle peak load, then nothing more needs to be done in terms of optimization.

In practice, our real bottlenecks are most likely to be the human element of the equation: our Visitor Engagement team. Since we only have a few working at any given point in time, and the fact that quality answers can sometimes take a little while to come up with, having too many people asking questions and not enough people answering can be our worst bottleneck at times. That being said, when we’re optimizing for a certain load average on our backend, we didn’t want to just aim for that number; we wanted to aim a bit higher to give ourselves some cushion.

So how do we actually figure out where our bottlenecks are? In essence, this is a basic troubleshooting problem. If something is broke, where is it broke? Often times the simplest way to figuring this out is by isolating each component from each other and benchmarking each by itself. Once you have a baseline for each, you can then figure out where the bottleneck lies. Depending on what the actual bottleneck is, the solution can vary wildly and can be a massive engineering effort depending on the scale at which your application operates at. I recommend reading engineering blog posts from Facebook, Netflix, and other companies dealing with extremely large scales to get a better sense of what goes into solving these type of technical problems.

At the end of the day our number one priority is providing a great experience for our visitors. Our back-end is just one piece of the overall effort that goes into make sure that happens, and when it’s running well, nobody should notice it at all. Kind of like a well-oiled machine running quietly in the background…so cool, so chrome….

Seated Buddha Shakyamuni, 965-1025. Gilt bronze, 8 1/2 x 7 1/4 x 4 3/4 in. (21.6 x 18.4 x 12.1 cm). Brooklyn Museum, Gift of the Asian Art Council in memory of Mahmood T. Diba; Mary Smith Dorward Fund, 1999.42. Creative Commons-BY
Image: overall, 1999.42_SL1.jpg. Brooklyn Museum photograph

As David has mentioned in a previous blog post, we follow a very Agile like process to guide us through our technical development process. One of the most important aspects of this process is writing tests for everything we can possibly write a test for.

What does a typical test look like? Let’s take a look at our earlier example of 1 + 1 = 2. Imagine we have a block of code (we’ll call this a function) that computes the sum of 1 + 1. We know that the sum of 1 + 1 should be 2. Knowing the definition/result of what we’re looking for, we can then construct a test to help us determine whether our function is performing up to standard. This test could simply check that result is two. If we wanted to get more in-depth, we could check that the result is less than 3 and greater than 1.

This is a pretty simple example but sometimes the things we need to test for encompass more than just one isolated block of code but rather how a group of disparate pieces of code work together. When we write tests for the former we call those ‘unit tests’. When we write tests for the latter we call those ‘integration tests’. With a combination of unit tests and integration tests, we can cover most of the different permutations of our use cases.

Jenkins displays a report that lets us know what
percentage of our code is covered by tests. For this particular project
it is close to 100% test coverage.

However, even with an extremely comprehensive test suite, this doesn’t prevent bugs from cropping up or other strange errors. Since writing tests is code too and all code needs to be maintained, there arrives a point of diminishing returns for a certain number of tests. After a certain point, a programmer will have a hard time coming up with different permutations to test against due to the inherent complexities in writing software. This isn’t an excuse but rather a reality of projects where the resources of time and money are always a constraint. Sometimes going for the ‘most bang for buck’ tests is the best course of action.

So if writing tests doesn’t ensure bug free code, what do we get from them?

One of the best advantages of writing tests is giving us some level of protection from regressions. As an example, imagine we found a bug in our code. Instead of just fixing the bug and being done with it we also wrote a test covering the newly discovered case the bug presented. Now if for some reason another programmer ever wrote code that reintroduced the same bug, our new test will now catch the error. While this sounds kind of stupid, this can be a fairly common problem due to the sometimes amorphous nature of code.

Another huge advantage of having code tests is that we can now run automated tests. As part of our development process, we use a continuous deployment system that only deploys new code if the new code passes tests. We can also run tests before attempting to push our code live as part of each developer’s own development process. This means we can often times catch issues before they show up in production environments, meaning the end user should typically only see very stable versions of our software being used.

While our current development process has been pretty good, we know that we can always do better. It’s part of our overarching theme of treating everything like an iterative process. As someone famous once said, “if you’re not getting better, you’re getting worse.”

As we talked about in earlier blog posts, we are hard at work on the Bloomberg Connects project which aims to cross the divide between each visitor’s curiosity, and the knowledge of our museum staff.  As part of bringing these two together, we’ve been building software to help us. In technical terms, we have a mobile app for visitors to ask questions, and a dashboard app for audience engagement team to talk to our visitors.  For both apps, we are using the same internal API communicate across the two apps.

Why?  Internal APIs allow us to develop and iterate faster by giving us a clean separation of concerns from our back end data logic from our front end logic. What this means is that for our developers focused on the front end (the people who build the interface which our users interact with), instead of having to know how our whole system works, they only have to know how to interact with the API. Since the backend of a web development project can involve a ton of different things not necessarily important to the average front end developer, such as databases, web servers, caching, CDN, etc. this means a front end developer can focus on what they do best which is building a fantastic user experience.

A very high overview describing the relationship between our API and our mobile and dashboard applications.

Having an internal API also give us the flexibility to serve more than one consumer at a time.  This means our mobile app and our dashboard app can use the same API to get things done.  This is vastly superior to the more common approach in the past where each application would have to write its own data interfacing logic. This means that if a web app and a mobile app have some functionality that needs to be implemented in both, the exact same logic would need to be written in two different code bases. In the development and agile world we call this a violation of the DRY principle, which stands for “Don’t Repeat Yourself.”  DRY is one of the fundamental truths that guide our development process as it helps to protect us from buggy, hard to manage code.

Essentially the goal we are striving for is a “loose coupling” of our code base. To illustrate how important this is, imagine a world where everyone has to write their own programs to use a computer. This is obviously untenable as not everyone has the time, knowledge, and expertise to create their own programs. Instead we just download programs others have created. A computer program decouples the implementation details (code) from the actual use of the program. Programs themselves are already abstractions which hide many details of how they work to let a user do work in a higher level of abstraction. The creator of the program could modify the implementation of the program but keep the way the user interacts with the program the same and the user would not know the difference. That is loose coupling in practice.

Here at our technology department we’ve adopted the agile spirit 100% and are always looking for ways to improve upon what we’ve done in the past. Building APIs to power not just our external services, but also our internal services has been a fantastic step in that direction. We’re excited for the journey ahead and can’t wait to show you more!