Numerai's True Contribution Signal Evaluation Method

Author: Numerai

Compiled by: ChainCatcher

Numerai has released a new signal evaluation method called True Contribution. True Contribution is calculated by treating Numerai as an end-to-end AI system. By calculating the gradient of optimized portfolio returns relative to the NMR staked on signals using a differentiable convex optimization layer, Numerai can now surface and incentivize signals that make the greatest intelligence contribution to our hedge fund. Numerai has performed well (with a Sharpe ratio* of 2.54 over the past 12 months), but True Contribution has the potential to make Numerai the first "Type IV" hedge fund.

The Kardashev scale, proposed by Soviet astrophysicist Nikolai Kardashev, is a method of classifying civilizations based on the energy they utilize for technological advancement. Type I civilizations can harness all the energy available on their planet. Type II civilizations can harness all the energy in their solar system, while Type III civilizations can harness all the energy in their galaxy.

The Kardashev scale does not contain the actual blueprints for creating these civilizations, but it does provide fertile ground for thought experiments regarding the technologies required for different types of civilizations. For example, to become a Type II civilization, we would need to invent something like a Dyson Sphere to capture all the energy from the sun. To become a Type III civilization, we would need self-replicating robotic technology to traverse the galaxy.

So, is there a Kardashev scale for hedge funds? What does it mean for different levels of technological advancement in hedge funds?

We can think of each hedge fund as trading some stock market prediction signals that should be related to the future returns of stocks. If the signal can predict future returns, the stocks that are best to buy will have the highest signal values, while the stocks that are best to short will have the lowest signal values. For quantitative hedge funds, their signals come directly from their mathematical models.

I would like to propose the following Kardashev scale for hedge funds, rating them based on the quality of their signals:

• Type I hedge funds have a signal that can predict subsequent returns before transaction costs.
• Type II hedge funds have a signal that can predict subsequent returns after transaction costs.
• Type III hedge funds have a signal that can be used to profitably alter the signals of the best hedge funds in the world.
• Type IV hedge funds have a signal so good that no other signal can be used to profitably alter it.

Type I hedge funds have the ability to predict future price movements; however, these predictions cannot be profitable because transaction costs, such as market impact costs, offset the gains. Type II hedge funds can make money after costs, thus violating the efficient market hypothesis.

Type III is where things get interesting, so let me explain Type III hedge funds with an example. Suppose we can identify the best hedge fund in the world, let's say it is Renaissance Technologies. In some parallel universe, the CEO of Two Sigma (another hedge fund) decides that all he wants to do is improve Renaissance's signals. He decides to give Renaissance Two Sigma's best signals—their highest quality stock predictions. He uploads Two Sigma's best signals to an FTP server that Renaissance can now see and use. Now, if Renaissance believes their own signals are so good that there is no way to use Two Sigma's signals to profitably alter their trading, then Two Sigma is not a Type III hedge fund. However, if Renaissance can use Two Sigma's signals to make profitable changes to their trading, then Two Sigma is a Type III hedge fund. Even if Renaissance still conducts 96% of the same trades they would typically make without Two Sigma's signals, the 4% of profitable trading changes that improve Renaissance's signals are enough to make Two Sigma a Type III hedge fund.

Becoming a Type III hedge fund is much more difficult than becoming a Type II hedge fund because Type II hedge funds only need to make profitable trades in the market, while Type III hedge funds must make profitable changes to already generated signals that lead to very profitable trades in the market.

Type IV hedge funds have a signal that no one can profitably alter. Type IV hedge funds are not only the best hedge funds in the world, but they are also hedge funds for which there are no known signals anywhere in the universe that can be combined with their signals for improvement. (In the example above, if Two Sigma is a Type III hedge fund under Renaissance's trading changes, then Renaissance is not a Type IV hedge fund.) Type IV hedge funds do not necessarily trade signals with perfect stock market prediction accuracy; it simply means that the signals they trade provide the greatest benefit compared to all currently known signals. It perfectly integrates all known signals.

Type IV hedge funds are like alien superintelligence in the stock market. It's somewhat akin to the best version of DeepMind's AlphaZero playing Go, where any changes made by humans (or older versions of AlphaZero or AlphaGo) cannot improve it.

I believe that Type IV hedge funds do not yet exist. But I am the founder of a new type of hedge fund called Numerai, and for the past few years, I have been conducting a thought experiment: What are the necessary attributes of a Type IV hedge fund? Is it possible to establish one?

Clearly, a Type IV hedge fund needs to be able to quickly and automatically receive any new signals that prevent it from becoming a Type IV hedge fund. If there were a Type IV hedge fund, it would be the best hedge fund, and thus there could be no Type III hedge fund signals that are not absorbed by the Type IV hedge fund. Type IV hedge funds must be able to instantly absorb any Type III signals; otherwise, they are not Type IV hedge funds. For this reason, Type IV hedge funds need to be an open system where anyone can upload new signals and make all changes to trading so that they can profit from them.

Renaissance, Two Sigma, or any other hedge fund operates under closed organizational designs that predate the internet and blockchain, which cannot create a Type IV hedge fund, just as Citibank's organizational design cannot create Bitcoin. Type IV hedge funds are indeed a completely new thing. They may look and feel more like Bitcoin than Two Sigma.

Of course, one obvious characteristic of a Type IV hedge fund is that it needs to be wealthy enough to provide sufficient capital to purchase all Type III signals. If Two Sigma can make profitable changes to the trades of a candidate Type IV hedge fund, then that Type IV hedge fund needs to have the funds to incentivize Two Sigma to completely shut down its trading operations and sell its signals to the Type IV hedge fund. Getting Two Sigma to change their signals is indeed very expensive. Perhaps even if Renaissance truly wanted to become a Type IV hedge fund, it could not afford the cost of becoming one.

Type IV hedge funds must be an open market for purchasing signals. Numerai is already such a system. Anyone can submit signals to Numerai using our free obfuscated data or their own data. Numerai does not have that much money, but we have about $150 million in cryptocurrency NMR, which makes Numerai the highest-earning data science competition in the world to date and the largest buyer of stock market signals on the internet.

By receiving signals in an open manner and using cryptocurrency to incentivize people to submit new signals, it seems that Numerai possesses the right attributes to become a Type IV hedge fund.

But there is a lingering, harmful assumption in this argument that hides out of sight. It is an assumption that is omitted in thought experiments but appears in actual reality. That is: Numerai would know how to evaluate whether some new signals would improve our existing good signals.

Numerai has combined thousands of signals submitted by our data scientists into what we call a meta-model. So, given a new signal, how do we know we can include it in this already large set of signals and that this inclusion will lead to profitable changes in trading strategies? Without a good technical solution to the signal evaluation problem, it is impossible to know whether a signal belongs to Type III, and thus it cannot reach Type IV.

In Numerai's years, we have had to learn how to excel at signal evaluation. Since Numerai's inception, we have made many improvements to signal evaluation. For example, we launched MMC and had Numerai data scientists stake NMR on their models to prove they believe their models can work out of sample (generalize). Both MMC and staking have improved the quality of signals on Numerai.

But we have never solved the signal evaluation problem. We cannot determine whether a signal is a Type III signal regarding our existing meta-model in our end-to-end system. But today we announce a new system we have spent years building called True Contribution, which addresses the signal evaluation problem.

It seems natural to assume that signals highly correlated with future stock returns may be most helpful to Numerai's meta-model. For this reason, Numerai has been paying data scientists based on their performance in generating signals closely related to Numerai's objectives (similar to residual returns) over the years. However, this incentivizes Type I signals, and rewarding signals based on their correlation with the target does not accurately represent how much that signal truly contributes to Numerai's optimized portfolio returns. Rewarding users based on their correlation with the target simply overlooks too many very important details, such as: the correlation of the signal with other signals, its interaction with the existing meta-model, or the hundreds of portfolio optimization parameters that Numerai uses to transform the meta-model into a balanced portfolio consisting of hundreds of stocks.

Clearly, we need a signal evaluation method that considers every detail in the system in order to assess whether a signal can make a profitable change for Numerai.

To do this end-to-end, we need to map out how signals and the NMR shares associated with those signals affect various aspects of the final portfolio constructed by the Numerai optimizer.

As you can see in the diagram above, Numerai first combines the signals generated by data scientists' machine learning models. We do this by creating a stake-weighted meta-model by calculating the equity-weighted average of each signal. Data scientists who stake a significant amount of NMR on their models will have a greater weight in the Stake-Weighted Meta Model.

The stake-weighted meta-model is still just a predictive signal for about 5,000 global stocks. It still needs to be transformed into a realistic portfolio with hundreds of risk constraints (such as market, country, and sector risk neutralization), which is precisely what the optimization step does. Once the optimizer creates a realistic hypothetical portfolio that meets all risk constraints, Numerai can observe the subsequent returns of the portfolio.

To properly evaluate signals, we must consider the impact of signals on the entire system described above, from the signal to the stake-weighted meta-model, and then to the returns of the optimized portfolio. This is what True Contribution does.

In short, True Contribution answers the question: If data scientists stake more on their models (thereby increasing their weight in the stake-weighted meta-model), how will the returns of the optimized portfolio change?

For those in quantitative finance, you can think of True Contribution as a complex form of signal attribution.

For those in machine learning, the diagram above about how Numerai works may remind you of neural network architectures. And if you have ever built a neural network, you may wonder if it is possible to adopt gradients regarding equity for optimized portfolio returns. That is precisely True Contribution.

But how do we compute the gradient through the portfolio optimization layer?

It turns out this can be done using new techniques developed by Stephen Boyd from Stanford University, Brandon Amos from Facebook AI, and others (see their paper: Differentiable Convex Optimization Layers).

By using cvxpylayers, we can define convex portfolio optimization as a layer in a PyTorch model. This allows us to effectively compute the gradient of optimized portfolio returns with respect to equity values and determine the true contribution of each signal submitted to Numerai.

In True Contribution portfolio construction, equity scale, model originality, and signal strength are all considered in exact proportion, as they are crucial for generating returns in the realistic portfolios that Numerai actually trades.

Original signals can help Numerai make trades that are different and more profitable than the ones we originally had, and now they will receive the highest NMR rewards, thus often having increasing weight in the stake-weighted meta-model. This reward feedback is also important for every data scientist on Numerai, as they can now improve their models to maximize their true contribution.

With True Contribution, Numerai is creating a feedback loop designed to continuously incentivize the creation and submission of signals, leading to profitable changes for our hedge fund and suppressing all other signals. Each round of Numerai will become another backpropagation in Numerai's overall cybernetic system. Feedback and correction propagate through distributed AI model layers, blockchain staking layers, meta-models, and convex optimization. In other words, with each round of Numerai, we are taking a step toward becoming a Type IV hedge fund.

Staking for True Contribution will begin on April 9, but in the meantime, we have backfilled True Contribution for every user on Numerai over the past approximately two years.

The results show that True Contribution has great potential as a new signal evaluation metric on Numerai.

For example, there are many data scientists (like LANCEALOT) who have very high true contribution rankings but rank much lower on other metrics (like correlation with the target). It is clear that, at least for some time recently, LANCEALOT has a model that helps Numerai the most, but their contribution has not been properly rewarded.

HB is an engineer at NASA's Jet Propulsion Laboratory working on the Europa Clipper mission and has been a long-time Numerai data scientist. HB has submitted multiple models to Numerai, but the true contribution of the model in which he staked the most NMR (765 NMR worth $22,000) is much lower than that of his other models. In terms of true contribution, some of his best models have no stake at all, meaning those excellent models have a weight of zero in the meta-model and are not rewarded at all.

With the ability to start staking for true contribution, data scientists will begin to shift their stakes to the models with the highest expected true contribution. In this dynamic system, as data scientists like HB and LANCEALOT adjust their stakes for more true contribution, there is no reason to believe that Numerai's stake-weighted meta-model will not become smarter.

Of course, we do not know if we will reach Type IV. But if we are on the path to becoming Type IV, what kind of things would you want to see? I think the key to look for is that risk-adjusted performance increases over time and with asset management scale.

Over time, hedge funds tend to get worse. How sad. The reason this happens is signal decay (as market efficiency increases, the trading signals that hedge funds start with deteriorate over time) and capacity constraints (a trading strategy that works well at $10 million may not work at all at $100 million).

But so far, Numerai has not gotten worse over time—it is getting better, which is a very good sign.

During this time in the chart, Numerai's hedge fund AUM has nearly increased tenfold, from about $7 million to $64 million (still in the early stages). Over time, our signal decay and asset management scale growth should harm performance, but our risk-adjusted returns (Sharpe ratio) continue to increase over time. This is because during the same period, the number of models staked in the meta-model has increased from 300 to over 4,000. Numerai has no signal decay; we have continuous signal recovery—the meta-model is rebuilt weekly with the latest signals.

Without Numerai users receiving the correct feedback about their models, that is, without true contribution…

Thanks to the geniuses at Midjourney for the AI-generated cover art. (The prompt here is: "The purple cybernetic Dyson Sphere of Wall Street.")

*The Sharpe ratio calculation is based on total returns after fees and assumes a risk-free rate of 0%.

The Sharpe ratio information provided is historical and not indicative of future performance. Investors should be aware that investment losses are possible. It does not represent that any investor will or may achieve profits or losses similar to those shown.