At Instacart, Hybrid AI Solves The Stubborn AI Reliability Problem

Instacart And The Art Of Managing Uncertainty

Instacart, which lets you order groceries and household goods for home delivery from most any retail chain, faces an inherent limitation: The person shopping on your behalf may sometimes find that the product you selected is out of stock.

Predictive AI is well suited to solve for this. By leveraging AI to predict which replacement item is most likely to satisfy the customer, Instacart can offer the best possible substitute when an item is out of stock.

This perfectly exemplifies AI’s pervasive conundrum. It’s one thing to solve the complex problem at hand – in this case, selecting the best possible replacement item from across a tremendous number of items in the catalog. Culturally, AI focuses on taking on complex problems like that. But it’s another thing for a system to gauge its own confidence and determine whether to then act autonomously. This critical step hasn’t yet gained its due cultural focus in the industry.

Enter Ahsaas Bajaj, Instacart’s engineering manager for machine learning. With a computer science master’s from UMass Amherst, and prior positions at Walmart, Goldman Sachs and Samsung, Ahsaas has developed systems that drive hundreds of millions of real-time operational decisions that preserve high customer satisfaction.

Ahsaas recognizes the inherent confidence-related challenge at hand. “What makes replacement product suggestion harder than many machine learning projects is the sheer size of the decision space: potentially thousands of candidate items per prediction,” he told me. To handle this, Ahsaas’ system acts on explicit confidence levels. These gauges on confidence drive runtime decisions. “Filter out potential replacement items too aggressively and you leave value on the table,” he told me. “But filter too loosely and you surface low-confidence suggestions that erode user trust.” Closing the gap between predictive model quality and driving clear business decisions is where AI systems are able to deliver extreme value.

So how does an AI system gauge its own confidence?

AI Systems That Gauge Their Own Confidence

Predictive AI provides a reliability guardrail for all kinds of AI systems. By predicting which cases warrant the precious time of a human in the loop, an otherwise unusable system will gain the trust needed to be broadly unleashed. This is one of the most important rapidly-emerging forms of hybrid AI.

For example, hospitals and provider networks use AI systems to automatically prepare medical insurance claims – but then use a separate predictive AI model to determine which cases are at a high risk of claim denial. This keeps humans in the loop judiciously, having them review the higher-risk claims before submission.

But sometimes you don’t need a separate predictive model to calculate per-case confidence. In 2011, IBM’s Watson – which defeated the all-time human champions of the TV quiz show Jeopardy!, succeeded not only by predicting the right answer to each question, but also by gauging its confidence in that answer. Confidence levels are key to the game, since you risk losing points if you “buzz in” to answer a question and get it wrong.

Watson uses a machine learning model to predict whether a candidate answer is the right answer. It applies the model repeatedly over a large number of potential answers, and then it goes with the candidate answer that it is predicting as most likely correct. Killing two birds, that same predictive score used to rank candidate answers also serves as a gauge of confidence in the top favored answer that it goes with.

For example, Jeopardy! once posed this question: “In cell division, mitosis splits the nucleus and cytokinesis splits this liquid cushioning the nucleus.” The system ranked the top candidate answers as follows:

Cytoplasm: 92% confidence

Organelle: 19% confidence

Vacuole: 10% confidence

Plasma: 3% confidence

Mitochondria: 2% confidence

Watson went with the top candidate answer, cytoplasm – and the confidence level of 92% that ranked it most highly also serves as the confidence level for that answer. For an overview of how Watson works, see Chapter 6 of Predictive Analytics.

Instacart’s replacement system works in a similar way. For example, let’s say you ordered a certain brand of honey-flavored cereal. Based on product features such as brand, department, cost and dietary attributes like organic, vegan, vegetarian or fat-free, its model scores candidate replacement items. This could include alternative brands of honey-flavored cereal, as well as other candidate replacement items such as granola clusters. The confidence level of each – the predicted probability that the customer would be satisfied with it as a replacement – serves to not only rank and select the best candidate item, but also to establish the confidence in that candidate.

Confidence levels drive decisions. Just as Watson uses the confidence level to decide – in real time – whether to buzz in and attempt to answer a question, Instacart’s product replacement system uses confidence to identify the best replacement option to suggest to a customer. As a machine learning practitioner, Ahsaas Bajaj’s job isn’t to unilaterally decide how Instacart should act on confidence levels. Rather, his job is to analyze how acting on confidence levels can translate to customer satisfaction and provide deployment options to his company so that a number of decision makers there can determine the best replacement to make.