“What papers talk about P?”
“Who argues that P?”

Those are the kinds of questions that David Bourget (Western) is hoping that PhilPapers, the comprehensive online philosophical index and bibliography, will be able to reliably provide direct, concise, and correct answers to.

To make that happen, Bourget, one of PhilPapers’ co-founders, is working on incorporating AI models into PhilPapers, and has developed a game called Beat AI: a contest using philosophical concepts to help train them.

In the following guest post, Bourget explains how he plans to make use of AI at PhilPapers, how Beat AI is played, and how playing it helps the AI improve.

He also provides a link to the game so you can go play it. Maybe you’ll get a high score.*

Play a Game to Improve a Philosophical Research Tool
by David Bourget

Today the PhilPapers team launched an online game called Beat AI: A contest using philosophical concepts. This post offers some information on the game’s technical background and its place in the broader PhilPapers ecosystem. It’s also an invitation to ask questions in the discussion thread.

In short, the objective of the game is to trick AI models using your superior mastery of philosophical concepts. You do this by submitting a triplet of expressions: an anchor, a substitute, and a decoy. The anchor and substitute are supposed to be close in meaning. The decoy is supposed to be an expression that AI models may think is closer in meaning to the anchor than the substitute is. You win when the AI models make mistakes.

In case it’s not obvious, our primary aim in making this game was to generate a large set of examples that we can use to train better AI models. Ultimately, we want to enable more powerful search and other features on PhilPapers. Even if you’re not fully taken in by the game, please consider giving it a good run to help us collect the data we need!

There are three AI models competing in the game: OpenAI’s ada3-large, BGE-large, and my own home-grown model, which I call philai-embeddings. These models are not “talking” AIs like ChatGPT (they aren’t “generative AI”). They are what are called embedding models. An embedding model is a neural network that’s designed to convert a piece of natural language to a large vector of numbers. In the case of BGE and philai-embeddings, the output is a vector of 1024 real numbers. OpenAI’s ada3 outputs vectors of 3072 numbers.

Embedding models are trained so that certain semantic relationships between their inputs are reflected in numerical relationships between corresponding outputs. In the case of phiai-embeddings, the chief aim was for closeness of topic between the inputs to correspond to the angle between the output vectors: the more the topic(s) of two inputs are similar, the smaller the angle between the output vectors (the “embeddings”). By “topics”, I mean what the input texts are about: the subjects, views, properties, relations, and other things mentioned in them. My model only cares about sameness of topic, and not about logical relations such as entailment or consistency, because it’s designed to be used for a first-pass retrieval of passages relevant to a query on PhilPapers, and in a first pass it’s simpler if we ignore logical relations.

This takes me to the broader purpose of this project. Embedding models are meant to allow efficient comparison of large numbers of passages for relevant semantic relationships. In my case, what I want to do is fully encode all works indexed on PhilPapers using the model, so that we can then use the embedding corresponding to a user’s search query (or something derived from a user’s search query) to retrieve likely relevant passages by comparing the angles between this embedding and the stored embeddings.

The advantage of this technique compared to keyword search is that embeddings, when they work well, allow us to cope with two major stumbling blocks for keyword search: synonymy and ambiguity. The problem of synonymy, as the name indicates, is that some distinct keywords can be used to convey the same or roughly the same meaning. But if you search for just one of the synonymous keywords, you will find only the texts that use that keyword. The problem of ambiguity is that some words may not always have the meaning that is relevant to your query. Embeddings, when well trained, can cut through both problems by giving synonymous expressions nearby vectors while not giving nearby vectors for inputs that contain the same keywords but with different meanings (when the disambiguating context is part of the input).

So, my aim is to convert all the text available on PhilPapers to embeddings in order to enable more powerful search. I plan to embed not just the abstracts of papers, but over 1.5M full texts to which we have access. For the texts, we will embed paragraph-sized chunks individually, which will allow very fine-grained searches. At the end of the day, I hope that we can reliably give direct, concise answers to queries such as “What papers talk about P?” or “Who argues that P?”

The home-grown model that BeatAI players are competing against, philai-embeddings, is a very first step. It’s a version of Google’s BERT that I further trained on a large number of sentences from PhilArchive. That made it much better with philosophical text than the base BERT. Surprisingly, it’s doing about as well as OpenAI’s Ada3 despite being much smaller and computationally efficient. But there’s still a long way to go, as players of Beat AI will probably find. Please go play and let our future models learn from you!

[BeatAI Individual Leaderboard as of late last night]