From Computing Power to Intelligence: A Reinforcement Learning-Driven Decentralized AI Investment Map

Author: Jacob Zhao, IOSG

Artificial intelligence is transitioning from a statistical learning paradigm focused on "pattern fitting" to a capability system centered on "structured reasoning," with the importance of post-training rapidly rising. The emergence of DeepSeek-R1 marks a paradigm shift for reinforcement learning in the era of large models, forming a consensus in the industry: pre-training builds a general capability foundation for models, and reinforcement learning is no longer just a tool for value alignment but has been proven to systematically enhance the quality of reasoning chains and complex decision-making capabilities, gradually evolving into a technical path for continuously improving intelligence levels.

Meanwhile, Web3 is reconstructing the production relations of AI through decentralized computing networks and cryptographic incentive systems. The structural needs of reinforcement learning for rollout sampling, reward signals, and verifiable training align naturally with blockchain's computational collaboration, incentive distribution, and verifiable execution. This research report will systematically break down the AI training paradigm and the technical principles of reinforcement learning, demonstrating the structural advantages of reinforcement learning × Web3, and analyzing projects such as Prime Intellect, Gensyn, Nous Research, Gradient, Grail, and Fraction AI.

I. Three Stages of AI Training: Pre-training, Instruction Fine-tuning, and Post-training Alignment

The entire lifecycle of modern large language model (LLM) training is typically divided into three core stages: Pre-training, Supervised Fine-tuning (SFT), and Post-training (Post-training/RL). Each stage serves the functions of "building a world model --- injecting task capabilities --- shaping reasoning and values," with their computational structures, data requirements, and validation difficulties determining the degree of decentralization.

• Pre-training builds the language statistical structure and cross-modal world model of the model through large-scale self-supervised learning, forming the foundation of LLM capabilities. This stage requires training on trillions of tokens in a globally synchronized manner, relying on thousands to tens of thousands of homogeneous clusters of H100, with costs accounting for 80-95%, making it extremely sensitive to bandwidth and data copyright, thus it must be completed in a highly centralized environment.

• Fine-tuning is used to inject task capabilities and instruction formats, with a smaller data volume and costs accounting for about 5-15%. Fine-tuning can be performed with full parameter training or using parameter-efficient fine-tuning (PEFT) methods, among which LoRA, Q-LoRA, and Adapter are mainstream in the industry. However, it still requires synchronized gradients, limiting its decentralization potential.

• Post-training consists of multiple iterative sub-stages that determine the model's reasoning ability, values, and safety boundaries. Its methods include reinforcement learning systems (RLHF, RLAIF, GRPO) as well as non-RL preference optimization methods (DPO) and process reward models (PRM). This stage has lower data volume and costs (5-10%), mainly focusing on Rollout and policy updates; it naturally supports asynchronous and distributed execution, where nodes do not need to hold complete weights. Combined with verifiable computation and on-chain incentives, it can form an open decentralized training network, making it the most compatible training segment for Web3.

II. Overview of Reinforcement Learning Technology: Architecture, Framework, and Applications

System Architecture and Core Elements of Reinforcement Learning Reinforcement Learning (RL) drives the model to autonomously improve decision-making capabilities through "environment interaction --- reward feedback --- policy update." Its core structure can be viewed as a feedback loop composed of states, actions, rewards, and policies. A complete RL system typically includes three types of components: Policy (policy network), Rollout (experience sampling), and Learner (policy updater). The policy interacts with the environment to generate trajectories, and the Learner updates the policy based on reward signals, forming a continuous iterative and optimizing learning process:

1. Policy: Generates actions from the environment state and is the core of the system's decision-making. During training, centralized backpropagation is required to maintain consistency; during inference, it can be distributed to different nodes for parallel execution.

2. Rollout: Nodes execute environment interactions based on the policy, generating trajectories such as state --- action --- reward. This process is highly parallel, with very low communication, and is less sensitive to hardware differences, making it the most suitable segment for scaling in decentralization.

3. Learner: Aggregates all Rollout trajectories and performs policy gradient updates, being the only module with the highest requirements for computing power and bandwidth, thus typically maintaining centralized or lightly centralized deployment to ensure convergence stability.

Reinforcement Learning Stage Framework (RLHF → RLAIF → PRM → GRPO) Reinforcement learning can typically be divided into five stages, with the overall process described as follows: # Data Generation Stage (Policy Exploration) Under given input prompts, the policy model πθ generates multiple candidate reasoning chains or complete trajectories, providing a sample basis for subsequent preference evaluation and reward modeling, determining the breadth of policy exploration. # Preference Feedback Stage (RLHF / RLAIF)

• RLHF (Reinforcement Learning from Human Feedback) uses multiple candidate answers, human preference annotations, trains reward models (RM), and optimizes policies with PPO, making model outputs more aligned with human values, which is a key step from GPT-3.5 to GPT-4.

• RLAIF (Reinforcement Learning from AI Feedback) replaces human annotations with AI judges or constitutional rules to automate preference acquisition, significantly reducing costs and exhibiting scalability, becoming the mainstream alignment paradigm for Anthropic, OpenAI, DeepSeek, etc.

# Reward Modeling Stage (Reward Modeling) Preferences are input into the reward model, learning to map outputs to rewards. RM teaches the model "what is the correct answer," while PRM teaches the model "how to reason correctly."

• RM (Reward Model) is used to evaluate the quality of final answers, scoring only the outputs:

• Process Reward Model (PRM) evaluates not only the final answer but also scores each step of reasoning, each token, and each logical segment. It is also a key technology of OpenAI o1 and DeepSeek-R1, essentially "teaching the model how to think."

# Reward Verification Stage (RLVR / Reward Verifiability) In the process of generating and using reward signals, "verifiable constraints" are introduced to ensure that rewards come from reproducible rules, facts, or consensus as much as possible, thereby reducing the risks of reward hacking and bias, and enhancing auditability and scalability in open environments. # Policy Optimization Stage (Policy Optimization) This stage updates policy parameters θ based on signals provided by the reward model to achieve stronger reasoning capabilities, higher safety, and more stable behavior patterns for the policy πθ′. Mainstream optimization methods include:

• PPO (Proximal Policy Optimization): The traditional optimizer for RLHF, known for its stability, but often faces limitations such as slow convergence and insufficient stability in complex reasoning tasks.

• GRPO (Group Relative Policy Optimization): A core innovation of DeepSeek-R1, modeling the advantage distribution within candidate answer groups to estimate expected value rather than simple ranking. This method retains reward magnitude information, making it more suitable for reasoning chain optimization, with a more stable training process, regarded as an important reinforcement learning optimization framework for deep reasoning scenarios after PPO.

• DPO (Direct Preference Optimization): A non-reinforcement learning post-training method that does not generate trajectories or build reward models but directly optimizes on preference pairs, being low-cost and stable in effect, widely used for aligning open-source models like Llama and Gemma, but does not enhance reasoning capabilities.

# New Policy Deployment Stage The optimized model exhibits: stronger reasoning chain generation capabilities (System-2 Reasoning), behaviors more aligned with human or AI preferences, lower hallucination rates, and higher safety. The model continuously learns preferences, optimizes processes, and improves decision quality through ongoing iterations, forming a closed loop. Five Categories of Industrial Applications of Reinforcement Learning Reinforcement Learning has evolved from early game intelligence to a core framework for autonomous decision-making across industries. Its application scenarios can be summarized into five categories based on technological maturity and industrial implementation, driving key breakthroughs in their respective directions.

• Game & Strategy: This is the earliest validated direction for RL, where RL has demonstrated decision-making intelligence comparable to or even surpassing human experts in environments with "perfect information + clear rewards," such as AlphaGo, AlphaZero, AlphaStar, and OpenAI Five, laying the foundation for modern RL algorithms.

• Robotics & Embodied AI: RL enables robots to learn manipulation, motion control, and cross-modal tasks (e.g., RT-2, RT-X) through continuous control and dynamics modeling, rapidly advancing towards industrialization and becoming a key technological route for real-world robot deployment.

• Digital Reasoning (Digital Reasoning / LLM System-2): RL + PRM drives large models from "language imitation" to "structured reasoning," with representative achievements including DeepSeek-R1, OpenAI o1/o3, Anthropic Claude, and AlphaGeometry, essentially optimizing rewards at the reasoning chain level rather than merely evaluating final answers.

• Automated Scientific Discovery & Mathematical Optimization: RL seeks optimal structures or strategies in unlabeled, complex rewards, and vast search spaces, achieving foundational breakthroughs such as AlphaTensor, AlphaDev, and Fusion RL, showcasing exploration capabilities that surpass human intuition.

• Economic Decision-making & Trading Systems: RL is used for strategy optimization, high-dimensional risk control, and adaptive trading system generation, enabling continuous learning in uncertain environments compared to traditional quantitative models, forming an important component of intelligent finance.

III. Natural Match Between Reinforcement Learning and Web3

The high compatibility between Reinforcement Learning (RL) and Web3 stems from the fact that both are essentially "incentive-driven systems." RL relies on reward signals to optimize policies, while blockchain coordinates participant behavior through economic incentives, making their mechanisms inherently consistent. The core needs of RL—large-scale heterogeneous Rollout, reward distribution, and authenticity verification—are precisely the structural advantages of Web3. # Decoupling Reasoning and Training The training process of reinforcement learning can be clearly divided into two stages:

• Rollout (Exploration Sampling): The model generates a large amount of data based on the current policy, which is computation-intensive but communication-sparse. It does not require frequent communication between nodes, making it suitable for parallel generation on globally distributed consumer-grade GPUs.

• Update (Parameter Update): This stage updates model weights based on the collected data and requires high-bandwidth centralized nodes to complete.

"Decoupling reasoning and training" naturally fits the decentralized heterogeneous computing structure: Rollout can be outsourced to an open network, settled by contribution through a token mechanism, while model updates remain centralized to ensure stability. # Verifiability ZK and Proof-of-Learning provide means to verify whether nodes genuinely execute reasoning, addressing honesty issues in open networks. In deterministic tasks such as code and mathematical reasoning, verifiers only need to check the answers to confirm the workload, significantly enhancing the credibility of decentralized RL systems. # Incentive Layer: Feedback Production Mechanism Based on Token Economics The token mechanism of Web3 can directly reward contributors of preference feedback in RLHF/RLAIF, creating a transparent, accountable, and permissionless incentive structure for preference data generation; staking and slashing further constrain feedback quality, forming a more efficient and aligned feedback market than traditional crowdsourcing. # Potential of Multi-Agent Reinforcement Learning (MARL) Blockchain is essentially a public, transparent, and continuously evolving multi-agent environment, where accounts, contracts, and agents continuously adjust their strategies under incentive-driven conditions, naturally possessing the potential to build large-scale MARL experimental fields. Although still in its early stages, its characteristics of state transparency, verifiable execution, and programmable incentives provide principled advantages for the future development of MARL.

IV. Analysis of Classic Web3 + Reinforcement Learning Projects

Based on the theoretical framework above, we will briefly analyze the most representative projects in the current ecosystem: Prime Intellect: Asynchronous Reinforcement Learning Paradigm prime-rl Prime Intellect aims to build a global open computing power market, lowering training barriers, promoting collaborative decentralized training, and developing a complete open-source superintelligence technology stack. Its system includes: Prime Compute (unified cloud/distributed computing environment), INTELLECT model family (10B--100B+), Open Reinforcement Learning Environment Hub, and large-scale synthetic data engine (SYNTHETIC-1/2).

The core infrastructure component of Prime Intellect, the prime-rl framework, is designed for asynchronous distributed environments and is highly relevant to reinforcement learning. Other components include the OpenDiLoCo communication protocol that breaks bandwidth bottlenecks and the TopLoc verification mechanism that ensures computational integrity. # Overview of Prime Intellect Core Infrastructure Components # Technical Cornerstone: prime-rl Asynchronous Reinforcement Learning Framework prime-rl is the core training engine of Prime Intellect, designed for large-scale asynchronous decentralized environments, achieving high-throughput reasoning and stable updates through complete decoupling of Actor and Learner. Rollout Workers and Trainers no longer block synchronously, allowing nodes to join or leave at any time, only needing to continuously pull the latest policy and upload generated data:

• Rollout Workers: Responsible for model reasoning and data generation. Prime Intellect innovatively integrates the vLLM reasoning engine on the Actor side. The PagedAttention technology and continuous batching capability of vLLM enable Actors to generate reasoning trajectories with extremely high throughput.

• Learner (Trainer): Responsible for policy optimization. The Learner asynchronously pulls data from a shared experience replay buffer for gradient updates without waiting for all Actors to complete the current batch.

• Orchestrator: Responsible for scheduling model weights and data flow.

# Key Innovations of prime-rl

• True Asynchrony: prime-rl abandons the synchronous paradigm of traditional PPO, not waiting for slow nodes or requiring batch alignment, allowing any number and performance of GPUs to connect at any time, establishing the feasibility of decentralized RL.

• Deep Integration of FSDP2 and MoE: Through FSDP2 parameter slicing and MoE sparse activation, prime-rl enables efficient training of models with billions of parameters in distributed environments, with Actors only running active experts, significantly reducing memory and inference costs.

• GRPO+: GRPO eliminates the need for a Critic network, significantly reducing computational and memory overhead, naturally adapting to asynchronous environments. prime-rl's GRPO+ further ensures reliable convergence under high-latency conditions through stabilization mechanisms.

# INTELLECT Model Family: A Mark of Decentralized RL Technology Maturity

• INTELLECT-1 (10B, October 2024) first proves that OpenDiLoCo can efficiently train across heterogeneous networks spanning three continents (communication accounts for <2%, computing power utilization 98%), breaking the physical cognition of cross-regional training;

• INTELLECT-2 (32B, April 2025) as the first permissionless RL model, verifies the stable convergence capabilities of prime-rl and GRPO+ in multi-step delays and asynchronous environments, achieving decentralized RL with global open computing power participation;

• INTELLECT-3 (106B MoE, November 2025) adopts a sparse architecture that activates only 12B parameters, training on 512×H200 and achieving flagship-level reasoning performance (AIME 90.8%, GPQA 74.4%, MMLU-Pro 81.9%, etc.), with overall performance approaching or even surpassing centralized closed-source models of much larger scale.

Additionally, Prime Intellect has built several supporting infrastructures: OpenDiLoCo reduces the communication volume of cross-regional training by hundreds of times through time-sparse communication and quantized weight differences, maintaining 98% utilization for INTELLECT-1 across a three-continent network; TopLoc + Verifiers form a decentralized trusted execution layer, ensuring the authenticity of reasoning and reward data through activation fingerprints and sandbox verification; the SYNTHETIC data engine produces large-scale high-quality reasoning chains and efficiently runs 671B models on consumer-grade GPU clusters through pipeline parallelism. These components provide a critical engineering foundation for data generation, verification, and reasoning throughput in decentralized RL. The INTELLECT series proves that this technology stack can produce mature world-class models, marking the transition of decentralized training systems from the conceptual stage to practical application. Gensyn: Core RL Stack RL Swarm and SAPO Gensyn aims to aggregate global idle computing power into an open, trustless, and infinitely scalable AI training infrastructure. Its core includes a standardized execution layer across devices, a peer-to-peer coordination network, and a trustless task verification system, automatically allocating tasks and rewards through smart contracts. Focusing on the characteristics of reinforcement learning, Gensyn introduces core mechanisms such as RL Swarm, SAPO, and SkipPipe, decoupling the generation, evaluation, and updating processes, utilizing a "swarm" of globally heterogeneous GPUs for collective evolution. Its ultimate delivery is not merely computing power but verifiable intelligence. # Reinforcement Learning Applications in Gensyn Stack # RL Swarm: Decentralized Collaborative Reinforcement Learning Engine RL Swarm demonstrates a new collaborative model. It is no longer a simple task distribution but a decentralized "generate --- evaluate --- update" loop that simulates human social learning, analogous to collaborative learning processes, in an infinite loop:

• Solvers (Executors): Responsible for local model reasoning and Rollout generation, with heterogeneous nodes being no obstacle. Gensyn integrates high-throughput reasoning engines (such as CodeZero) locally, capable of outputting complete trajectories rather than just answers.

• Proposers: Dynamically generate tasks (math problems, coding questions, etc.), supporting task diversity and difficulty adaptability akin to Curriculum Learning.

• Evaluators: Use frozen "judge models" or rules to evaluate local Rollouts, generating local reward signals. The evaluation process can be audited, reducing the space for malicious behavior.

Together, these three form a P2P RL organizational structure that can achieve large-scale collaborative learning without centralized scheduling. # SAPO: Strategy Optimization Algorithm Reconstructed for Decentralization SAPO (Swarm Sampling Policy Optimization) focuses on "sharing Rollouts and filtering non-gradient signal samples rather than sharing gradients," maintaining stable convergence in environments with significant node delay differences and no central coordination through large-scale decentralized Rollout sampling. Compared to PPO, which relies on a Critic network and has high computational costs, or GRPO, which estimates advantages within groups, SAPO allows consumer-grade GPUs to effectively participate in large-scale reinforcement learning optimization with extremely low bandwidth.

Through RL Swarm and SAPO, Gensyn demonstrates that reinforcement learning (especially RLVR in the post-training phase) naturally adapts to decentralized architectures—because it relies more on large-scale, diverse exploration (Rollout) rather than high-frequency parameter synchronization. Combined with the verification systems of PoL and Verde, Gensyn provides an alternative path for training trillion-parameter models that no longer depends on a single tech giant: a self-evolving superintelligence network composed of millions of heterogeneous GPUs. Nous Research: Verifiable Reinforcement Learning Environment Atropos Nous Research is building a decentralized, self-evolving cognitive infrastructure. Its core components—Hermes, Atropos, DisTrO, Psyche, and World Sim—are organized into a continuously closed-loop intelligent evolution system. Unlike the traditional linear process of "pre-training --- post-training --- reasoning," Nous employs reinforcement learning techniques such as DPO, GRPO, and rejection sampling to unify data generation, verification, learning, and reasoning into a continuous feedback loop, creating a self-improving closed-loop AI ecosystem. # Overview of Nous Research Components # Model Layer: Hermes and the Evolution of Reasoning Capabilities The Hermes series is the main model interface for Nous Research, clearly showcasing the industry's transition from traditional SFT/DPO alignment to reasoning reinforcement learning (Reasoning RL):

• Hermes 1--3: Instruction alignment and early agent capabilities: Hermes 1--3 achieves robust instruction alignment through low-cost DPO, with Hermes 3 leveraging synthetic data and the newly introduced Atropos verification mechanism.

• Hermes 4 / DeepHermes: Writes System-2 style slow thinking into weights through thought chains, enhancing mathematical and coding performance via Test-Time Scaling, and constructs high-purity reasoning data relying on "rejection sampling + Atropos verification."

• DeepHermes further adopts GRPO to replace PPO, which is difficult to implement in distributed environments, allowing reasoning RL to run on the Psyche decentralized GPU network, laying the engineering foundation for the scalability of open-source reasoning RL.

# Atropos: Verifiable Reward-Driven Reinforcement Learning Environment Atropos is the true hub of the Nous RL system. It encapsulates prompts, tool calls, code execution, and multi-turn interactions into a standardized RL environment, allowing direct verification of whether outputs are correct, thus providing deterministic reward signals that replace expensive and unscalable human annotations. More importantly, in the decentralized training network Psyche, Atropos acts as a "judge" to verify whether nodes genuinely improve strategies, supporting auditable Proof-of-Learning, fundamentally addressing the issue of reward credibility in distributed RL. # DisTrO and Psyche: Optimizer Layer for Decentralized Reinforcement Learning Traditional RLHF/RLAIF training relies on centralized high-bandwidth clusters, which is a core barrier that open-source cannot replicate. DisTrO reduces the communication costs of RL by several orders of magnitude through momentum decoupling and gradient compression, allowing training to run on internet bandwidth; Psyche deploys this training mechanism on a blockchain network, enabling nodes to complete reasoning, verification, reward evaluation, and weight updates locally, forming a complete RL closed loop.

In Nous's system, Atropos verifies thought chains; DisTrO compresses training communication; Psyche runs the RL loop; World Sim provides complex environments; Forge collects real reasoning; Hermes writes all learning into weights. Reinforcement learning is not just a training phase but the core protocol connecting data, environments, models, and infrastructure in the Nous architecture, making Hermes a living system that can continuously self-improve on an open-source computing power network. Gradient Network: Reinforcement Learning Architecture Echo The core vision of Gradient Network is to reconstruct the computing paradigm of AI through an "Open Intelligence Stack." Gradient's technology stack consists of a set of independently evolving yet heterogeneously collaborative core protocols. Its system includes: Parallax (distributed reasoning), Echo (decentralized RL training), Lattica (P2P network), SEDM / Massgen / Symphony / CUAHarm (memory, collaboration, security), VeriLLM (trusted verification), and Mirage (high-fidelity simulation), collectively forming a continuously evolving decentralized intelligent infrastructure. Echo --- Reinforcement Learning Training Architecture Echo is Gradient's reinforcement learning framework, designed to decouple training, reasoning, and data (reward) paths in reinforcement learning, allowing Rollout generation, policy optimization, and reward evaluation to independently scale and schedule in heterogeneous environments. It operates in a heterogeneous network composed of reasoning and training nodes, maintaining training stability through lightweight synchronization mechanisms in wide-area heterogeneous environments, effectively alleviating the SPMD failure and GPU utilization bottlenecks caused by mixed running of reasoning and training in traditional DeepSpeed RLHF/VERL. Echo employs a "dual-group architecture for reasoning and training" to maximize computing power utilization, with each group operating independently without blocking each other:

• Maximizing Sampling Throughput: The inference swarm consists of consumer-grade GPUs and edge devices, constructing a high-throughput sampler through Parallax in a pipeline-parallel manner, focusing on trajectory generation;

• Maximizing Gradient Computing Power: The training swarm consists of consumer-grade GPU networks that can run on centralized clusters or globally, responsible for gradient updates, parameter synchronization, and LoRA fine-tuning, focusing on the learning process.

To maintain consistency between policies and data, Echo provides two types of lightweight synchronization protocols: sequential (Sequential) and asynchronous (Asynchronous), achieving bidirectional consistency management of policy weights and trajectories:

• Sequential Pull Mode | Precision First: The training side forces reasoning nodes to refresh model versions before pulling new trajectories, ensuring the freshness of trajectories, suitable for tasks highly sensitive to outdated policies;

• Asynchronous Push-Pull Mode | Efficiency First: The reasoning side continuously generates version-tagged trajectories, with the training side consuming at its own pace, while the coordinator monitors version deviations and triggers weight refreshes, maximizing device utilization.

At the base level, Echo is built on Parallax (heterogeneous reasoning in low-bandwidth environments) and lightweight distributed training components (such as VERL), relying on LoRA to reduce cross-node synchronization costs, enabling stable operation of reinforcement learning on global heterogeneous networks. Grail: Reinforcement Learning in the Bittensor Ecosystem Bittensor constructs a vast, sparse, and non-stationary reward function network through its unique Yuma consensus mechanism.

Covenant AI within the Bittensor ecosystem builds a vertically integrated pipeline from pre-training to RL post-training through SN3 Templar, SN39 Basilica, and SN81 Grail. Among them, SN3 Templar is responsible for pre-training the base model, SN39 Basilica provides a distributed computing power market, and SN81 Grail serves as the "verifiable reasoning layer" for RL post-training, completing the closed-loop optimization from the base model to the aligned policy. GRAIL aims to cryptographically prove the authenticity of each reinforcement learning rollout and bind it to the model identity, ensuring that RLHF can be securely executed in a trustless environment. The protocol establishes a trusted chain through three mechanisms:

1. Deterministic Challenge Generation: Using drand random beacons and block hashes to generate unpredictable yet reproducible challenge tasks (such as SAT, GSM8K), eliminating pre-computation cheating;

2. By indexing sampling and sketch commitments through PRF, verifiers can randomly check token-level logprob and reasoning chains at extremely low costs, confirming that the rollout is indeed generated by the declared model;

3. Model Identity Binding: Binding the reasoning process to the model weight fingerprint and the structural signature of token distribution, ensuring that any replacement of models or replay of results will be immediately recognized. This provides a foundation for the authenticity of reasoning trajectories (rollouts) in RL.

Based on this mechanism, the Grail subnet implements a GRPO-style verifiable post-training process: miners generate multiple reasoning paths for the same question, and verifiers score based on correctness, reasoning chain quality, and SAT satisfaction, writing the normalized results on-chain as TAO weights. Public experiments show that this framework has improved the MATH accuracy of Qwen2.5-1.5B from 12.7% to 47.6%, proving that it can prevent cheating while significantly enhancing model capabilities. In Covenant AI's training stack, Grail is the cornerstone of trust and execution for decentralized RLVR/RLAIF, and it has not yet officially launched on the mainnet. Fraction AI: Competition-Based Reinforcement Learning RLFC Fraction AI's architecture is explicitly built around Reinforcement Learning from Competition (RLFC) and gamified data labeling, replacing the static rewards and human annotations of traditional RLHF with an open, dynamic competitive environment. Agents compete in different Spaces, with their relative rankings and AI judge scores collectively forming real-time rewards, transforming the alignment process into a continuously online multi-agent game system.

The core differences between traditional RLHF and Fraction AI's RLFC: The core value of RLFC lies in that rewards no longer come from a single model but from continuously evolving opponents and evaluators, avoiding exploitation of the reward model and preventing the ecosystem from falling into local optima through strategy diversity. The structure of Spaces determines the nature of the game (zero-sum or positive-sum), promoting the emergence of complex behaviors through competition and cooperation.

In terms of system architecture, Fraction AI breaks down the training process into four key components:

• Agents: Lightweight strategy units based on open-source LLMs, expanded through QLoRA with differential weights for low-cost updates;

• Spaces: Isolated task domain environments where agents pay to enter and receive rewards based on wins and losses;

• AI Judges: An instant reward layer built with RLAIF, providing scalable and decentralized evaluation;

• Proof-of-Learning: Binding policy updates to specific competitive results, ensuring that the training process is verifiable and cheat-proof.

The essence of Fraction AI is to build an "evolution engine" for human-machine collaboration. Users act as "meta-optimizers" at the strategy layer, guiding exploration directions through prompt engineering and hyperparameter configuration; while agents automatically generate massive amounts of high-quality preference data pairs through micro-level competition. This model allows data labeling to achieve a commercial closed loop through "trustless fine-tuning." Comparison of Reinforcement Learning Web3 Project Architectures

V. Conclusion and Outlook: Paths and Opportunities of Reinforcement Learning × Web3

Based on the deconstruction analysis of the aforementioned cutting-edge projects, we observe that although the entry points of various teams (algorithm, engineering, or market) differ, when reinforcement learning (RL) combines with Web3, their underlying architectural logic converges into a highly consistent "decoupling-validation-incentive" paradigm. This is not only a technical coincidence but also an inevitable result of decentralized networks adapting to the unique properties of reinforcement learning. General Architectural Features of Reinforcement Learning: Addressing Core Physical Limitations and Trust Issues

1. Decoupling of Rollouts & Learning ------ Default Computational Topology

   Communication-sparse, parallel Rollouts are outsourced to global consumer-grade GPUs, while high-bandwidth parameter updates are centralized to a few training nodes, as seen from Prime Intellect's asynchronous Actor-Learner to Gradient Echo's dual-group architecture.

2. Verification-Driven Trust Layer ------ Infrastructure-based

   In permissionless networks, the authenticity of computations must be enforced through mathematical and mechanism design, with implementations including Gensyn's PoL, Prime Intellect's TOPLOC, and Grail's cryptographic verification.

3. Tokenized Incentive Loop ------ Market Self-regulation

   The supply of computing power, data generation, verification ranking, and reward distribution form a closed loop, driving participation through rewards and suppressing cheating through slashing, allowing the network to remain stable and continuously evolve in an open environment. Differentiated Technical Paths: Different "Breakthrough Points" Under a Consistent Architecture Although the architectures converge, each project has chosen different technical moats based on its own genetics:

• Algorithm Breakthrough Faction (Nous Research): Attempts to solve the fundamental contradictions of distributed training (bandwidth bottlenecks) from a mathematical bottom-up approach. Its DisTrO optimizer aims to compress gradient communication by thousands of times, targeting to allow household broadband to run large model training, representing a "dimensionality reduction strike" against physical limitations.

• Systems Engineering Faction (Prime Intellect, Gensyn, Gradient): Focuses on building the next generation of "AI runtime systems." Prime Intellect's ShardCast and Gradient's Parallax are designed to extract the highest efficiency from heterogeneous clusters through extreme engineering means under existing network conditions.

• Market Game Faction (Bittensor, Fraction AI): Concentrates on the design of reward functions. By designing ingenious scoring mechanisms, it guides miners to spontaneously seek optimal strategies to accelerate the emergence of intelligence.

Advantages, Challenges, and Future Outlook In the paradigm of combining reinforcement learning with Web3, system-level advantages are primarily reflected in the rewriting of cost structures and governance structures.

• Cost Restructuring: The demand for sampling (Rollout) in post-training (Post-training) is infinite, and Web3 can mobilize global long-tail computing power at extremely low costs, which is a cost advantage that centralized cloud vendors find difficult to match.

• Sovereign Alignment: Breaking the monopoly of large companies over AI values (Alignment), communities can vote through tokens to decide what constitutes a "good answer," achieving democratization of AI governance.

At the same time, this system also faces two major structural constraints.

• Bandwidth Wall: Despite innovations like DisTrO, physical latency still limits the full training of ultra-large parameter models (70B+), and current Web3 AI is more limited to fine-tuning and inference.

• Goodhart's Law (Reward Hacking): In highly incentivized networks, miners can easily "overfit" reward rules (score manipulation) rather than enhancing genuine intelligence. Designing robust reward functions to prevent cheating is an eternal game.

• Malicious Byzantine Worker Attacks: Actively manipulating and poisoning training signals to disrupt model convergence. The core issue lies not in continuously designing anti-cheating reward functions but in building mechanisms with adversarial robustness.

The combination of reinforcement learning and Web3 fundamentally rewrites the mechanisms of "how intelligence is produced, aligned, and valued." Its evolutionary path can be summarized into three complementary directions:

1. Decentralized Training Networks: From computing power miners to policy networks, outsourcing parallel and verifiable Rollouts to global long-tail GPUs, focusing in the short term on verifiable reasoning markets, and evolving in the medium term into reinforcement learning subnets clustered by tasks;

2. Assetization of Preferences and Rewards: From labeling labor to data equity. Realizing the assetization of preferences and rewards transforms high-quality feedback and Reward Models into governable and distributable data assets, upgrading from "labeling labor" to "data equity."

3. "Small but Beautiful" Evolution in Vertical Domains: Cultivating small yet powerful dedicated RL Agents in vertically scoped scenarios where results are verifiable and returns quantifiable, such as DeFi strategy execution and code generation, directly binding strategy improvements to value capture and potentially outperforming general closed-source models.

Overall, the true opportunity of reinforcement learning × Web3 lies not in replicating a decentralized version of OpenAI but in rewriting the "intelligent production relations": making training execution an open computing power market, making rewards and preferences governable on-chain assets, and redistributing the value brought by intelligence among trainers, aligners, and users. Recommended Reading:

Why the largest Bitcoin treasury company in Asia, Metaplanet, is not bottom-fishing?

Multicoin Capital: The arrival of the Fintech 4.0 era

Is Web3 social a false proposition? The forced transformation of Farcaster, a Web3 unicorn company backed by a16z