Game Theory, Blockchain Tech and Predictive Modeling at Numerai

Histograms of Numerai leaders

Numerai is a hedge fund that crowdsources its model-building through data science competitions. It does this by publishing a dataset with blinded and encrypted variables so that data scientists can build predictive models without having full access to Numerai’s proprietary data. Competitors can stake Numeraire, a token built on Ethereum, to signal confidence in their model and earn a payout if it performs well.

Here is a quick look at earnings in USD vs. Numeraire staked, which I visualized based off the leaderboard that Numerai publishes:

I first heard about Numerai in an AVC post from Fred Wilson, whose firm USV is an investor in the company. More recently, I listened to Patrick O’Shaughnessy interview the founder of Numerai and decided to learn more about it.

I am fascinated with and quite skeptical of Numerai’s model. I will start with the things I find most interesting.

This is a good application of blockchain technology

Numeraire’s structure seems like it gets the incentives right. Data scientists can earn money if they build a good model, which is simple enough. But on its own this would create an incentive to create lots of accounts and submit lots of models. Since data scientists must stake Numeraire in order to get paid, there is incentive to only submit high-quality models or risk Numeraire being destroyed.

There is also a built-in network effect where early competitors who own Numeraire get value from new competitors who increase the value of the token.

I expect my infatuation with this mechanism comes in part from the whitepaper, which made me feel like I was back in a college game theory class when life was simple and if you solved the payoff functions correctly you got an A:

This is a great way to access global data science talent

There are a few exciting aspects of the crowdsourcing that Numerai is doing:

  • Machine learning talent on Wall Street is incredibly expensive and this model could save millions of dollars a year in salaries
  • The Numerai model lends itself perfectly to a person anywhere in the world contributing a winning model. Because the variables are abstracted and there isn’t any American finance-specific domain expertise, someone in Seoul or Lagos can contribute just as easily as someone in San Francisco or New York.
  • There is a market need for data science candidates to signal their abilities to potential employers and Numerai competitions could serve this need, similarly to Kaggle competitions

Numerai’s incentive structure and the potential upsides of the model are really exciting and I can understand why the company has attracted top-tier investors.

Here are the things I’m skeptical about:

How much value is there in building blinded models?

I have heard from lots of people I respect that most of the value a data scientist can bring is in domain expertise and insight into how variables interact with one another.

The Numerai model separates these sklils entirely from the model-building process. At this point, how much value is the human creating? Why couldn’t you just automatically generate millions of models with all the possible permutations of features and parameters and keep the ones with the lowest log-loss?

Especially in a highly dynamic domain like investing, it seems to me that relying on blinded models is highly risky and that by blinding the models you are capturing a small percentage of the value that a data scientist with non-blinded access to the information would create.

This is a tough problem to solve because obviously for a hedge fund data is an edge and probably needs to be kept proprietary. But it seems like a big and potentially existential problem to me.

Stickiness for competitors: Won’t they get bored?

When I work with data, by far the most satisfying and exciting aspect is learning new and unexpected things about whatever domain the data represents. Because of this, Numerai competitions don’t appeal to me as a competitor at all. It feels more like a Sudoku where you are solving a puzzle but not really learning anything other than how to better solve a Sudoko.

I expect that there are people who derive satisfaction from the pure model-building process but I also expect there are lots of people like me who would quickly get bored modeling on blinded data like this. The data seem to support this. I analysed the top 1000 users from the Leaderboard that Numerai publishes and it looks like there are a few users who have staked a lot and win a lot and then a ton of users who have dabbled in the idea but aren’t consistent users.

This sort of power law is what I would expect in an area like this but the fact that this is on supposedly the top 1,000 users and there is still this stark a drop-off concerns me for the health of the platform.

A quick aside: I wanted to quickly see histograms for the numeric variables in this dataset and the R code I found to create the above plot was so beautifully concise that I wanted to share it here:

leaders %>%
  keep(is.numeric) %>% 
  gather() %>% 
  ggplot(aes(value)) +
    facet_wrap(~ key, scales = "free") +
    geom_histogram(bins = 40) +

The money has to be good for competitors to stay

Especially if the competition isn’t inherently super fun, people have to be making decent money in order to stick around. And the opportunity cost for the type of person who builds a winning model in this competition is high.

It helps that Numerai can access talent in markets around the world where a machine learning expert wouldn’t command $400,000+ a year but it seems safe to assume that for the best people, the opportunity cost is $100 an hour or more.

Presumably it takes 5+ hours to build a winning model and so people need to be earning $500+ on expectation for it to make sense to consistently stake entries in Numerai competitions. It doesn’t seem to me like enough people are earning this for there not to be significant churn in the talent pool. The below is the distribution of users on the leaderboard who have positive earnings:

Presumably as a way to rally competitors to the noble cause of helping a hedge fund generate alpha, Numerai has some rah-rah verbiage on their site that I found funny:

If you have a great model, it means you have discovered pernicious pricing errors that are harming the global economy. We can correct these prices together, and move capital to where it belongs…Our only true adversaries are the other hedge funds

I am pulling for Numerai because i think it is such an interesting model. But I think they have a lot to figure out. I wonder if this model could be applied to something like design, where you could crowdsource a creative brief and have designers compete to create a new logo for a company, for example.

This is a complex company and I am not an expert on blockchain tech (or anything I write about ever, really). Please reach out to me if you have any thoughts on the analysis or see any holes in my thinking.

Thanks for reading and if you enjoy this sort of piece, sign up below and I will email you when I write something new.