Numerai CLI 0.3.0: A tool to simplify the building and deployment of machine learning models

Author: Numerai

Compiled by: ChainCatcher

If you've heard of Numerai and read any of our documentation, you may have heard of Numerai Compute and its related tool numerai-cli (if you haven't heard of us, please check us out). If you want to create some resources in AWS, deploy your models to the cloud, and allow Numerai to automatically trigger your models, letting you sip your drink of choice, put your feet up on the desk, and watch the scores roll in every day.

Currently, it is supported anywhere Python 3 and Docker are available, including MacOS/OSX, Windows 10 (and 8, with some extra work), and Ubuntu 18+. We recently released the 0.2.* version series, which provides some setup scripts to improve accessibility, allowing you to start automatically submitting from the cloud with just 5 commands. Creating autonomous Numerai models has never been easier.

That said, our CLI (and the compute clusters you set up with it) has limited functionality; you can only deploy 1 codebase to 1 cloud provider, and the sample Python code is mom's spaghetti [code]. If users want to parallelize their models, they have to figure it out. If users want to debug their deployed prediction nodes, they will have to figure it out. If users want to modify the sample Python code, they will have to figure it out themselves. These issues make Compute inaccessible to most existing and new users.

So what are we doing? In Numerai CLI 0.3.0, the sample Python code has been streamlined from 3 examples totaling over 500 lines of code to 1 example with fewer than 100 lines. Our cloud supply code has been modularized into a plug-and-play design, allowing us to continue adding new cloud providers like GCP. Finally, our CLI has been restructured to be prediction node-centric (more details later); this means you can run your models as you wish, deploy them anywhere you want, and submit to any tournament based on each model. Customizing compute has never been easier.

Streamlined, more readable, and no longer reliant on complex, deprecated packages, we have taken steps to simplify the learning curve by using every basic concept you need in one file. This includes introducing the concept of multiple models in the first few lines, as well as simple atomic functions for retrieving data, fitting, predicting, and submitting; all of this is done in fewer than 100 lines of code. We also include a new Numerai Signals sample model that introduces the concept of extracting your own financial data and generating your own features. It is certainly more complex than the tournament example, totaling fewer than 400 lines, but each component is still logically divided into a functional example to allow novice data scientists to read and understand how to analyze market data.

In the future, these examples will serve as stepping stones for both seasoned data scientists and newcomers, encouraging everyone to simply change a few lines of code to improve their models and continue making simple changes until they become Numerai Masters.

A useful and simple first example is crucial to our overall planning. We cannot monopolize intelligence, data, or markets unless we diversify our users, introduce new ideas, and make our strategies better than the current hedge fund dominators. Becoming a beginner has never been easier.

We envision a future where Numerai can execute any prediction on any AI on any infrastructure at any time. This is not feasible unless we design tools to build this future; Rome wasn't built in a day, nor was it built with a stone hammer. Numerai CLI 0.3.0 aims to provide users with a better way to build and deploy any model by restructuring around the principles of plug-and-play modules.

With this design, we introduce the concept of prediction nodes—a universal building block for submitting predictions. These applications are agnostic to cloud providers, tournaments, data sources, and languages. At its core is a basic Docker image deployed to the cloud, with an auto-scaling compute cluster that will run a container when triggered by Numerai. Want a Python XGBoost model that submits to Numerai Signals from AWS? No problem; make a few small changes to our sample code, and you’re up and running. Want to use a neural network created in C++? Sure, just add a Dockerfile and automate it with any supported cloud provider in a few commands.

By configuring prediction nodes, Numerai can call on the predictive capabilities of models when foresight into the market is needed, paying you for your performance or burning cash. This concept is crucial to our overall planning, as reliable submissions are essential for reliable performance. The CLI is the tool our data scientists use to build a reliable model network, thereby solidifying the emerging meta-model hegemony that controls the market. Paving the way to the future has never been easier.

With the introduction of updated examples and prediction nodes, the new 0.3.0 update now provides users with essential tools for testing, diagnosing, and debugging their local and deployed environments. The first of these tools is a complete end-to-end testing system for deployed prediction nodes, including checks to see if it can be triggered by Numerai, if it can run successfully, and if it submits correctly. Another tool is the new "numerai doctor," which can verify whether certain parts of the local environment are set up correctly; in the future, this tool will become the preferred self-service command for diagnosing, fixing, or reporting environmental issues. Finally, we managed to complete alpha testing for this release before launch, which helped us update documentation, error handling, and the overall user experience to handle errors and bugs more gracefully and report them correctly. Detecting model reliability has never been easier.

This update injects new vitality into Compute, marking the Numerai Compute experience's entry into the next era. We aim to make setup more streamlined and reliable, enhance configurability, improve the reliability of prediction nodes, increase submission transparency, and enable collaboration. It sounds like a lot, and it is, but it is necessary to ensure the development of our overall planning.

We want to make it easier for you to deploy models as you wish, so you can receive feedback, learn, and adjust your models without worrying about submission deadlines. Automated submissions mean you can reliably upload predictions; spend more time fine-tuning your models and less time ensuring model stability.

We hope you have more choices in how you configure prediction nodes and where you deploy them. We don't want to force you to become a cloud engineer, so we want to provide you with tools that allow you to deploy models to other compute providers (like GCP or Golem). Additionally, we hope you can leverage important resources like cloud GPUs for machine learning and analysis.

We want you to collaborate and learn with our amazing community of data scientists. Adding tools to enable this collaboration will allow you to create better models. We need more amazing models; specifically, we need more models with incredible predictive capabilities that can reliably submit, and more.

Whether you are a Numerai veteran or a first-timer, start using Numerai Compute 0.3.0 today to automate your submissions and provide your feedback on RocketChat, as Numerai Compute makes things simple.