AIxDePIN: What new opportunities will emerge from the collision of this hot track?
Web3 x AI Series Research Report Volume 2.Author: Cynic Shigeru, CGV Research
With the power of algorithms, computing power, and data, advancements in AI technology are redefining the boundaries of data processing and intelligent decision-making. Meanwhile, DePIN represents a paradigm shift from centralized infrastructure to decentralized, blockchain-based networks.
As the world accelerates its pace towards digital transformation, AI and DePIN (Decentralized Physical Infrastructure) have become foundational technologies driving change across various industries. The integration of AI and DePIN not only facilitates rapid technological iteration and widespread application but also opens up safer, more transparent, and efficient service models, bringing profound changes to the global economy.
DePIN: Decentralization from Virtual to Physical, the Pillar of the Digital Economy
DePIN stands for Decentralized Physical Infrastructure. In a narrow sense, DePIN mainly refers to distributed networks of traditional physical infrastructure supported by distributed ledger technology, such as power networks, communication networks, and positioning networks. In a broader sense, all distributed networks supported by physical devices can be referred to as DePIN, such as storage networks and computing networks.
from: Messari
If Crypto has brought decentralization to the financial level, then DePIN is the decentralization solution in the real economy. One could say that PoW mining machines are a form of DePIN. From day one, DePIN has been a core pillar of Web3.
The Three Elements of AI—Algorithms, Computing Power, and Data, with DePIN Dominating Two
The development of artificial intelligence is often considered to rely on three key elements: algorithms, computing power, and data. Algorithms refer to the mathematical models and program logic that drive AI systems, computing power refers to the computational resources required to execute these algorithms, and data is the foundation for training and optimizing AI models.
Which of the three elements is the most important? Before the emergence of chatGPT, people generally believed it was algorithms; otherwise, academic conferences and journal papers would not be filled with one algorithm tweak after another. However, when chatGPT and the large language model (LLM) supporting its intelligence appeared, people began to realize the importance of the latter two. Massive computing power is a prerequisite for the model's birth, and the quality and diversity of data are crucial for building robust and efficient AI systems. In contrast, the demand for algorithms is no longer as stringent as before.
In the era of large models, AI has shifted from meticulous refinement to large-scale brute force, with increasing demands for computing power and data, which DePIN can precisely provide. Token incentives leverage the long-tail market, and vast consumer-grade computing power and storage will become the best nourishment for large models.
Decentralization of AI is Not an Option, but a Necessity
Of course, some may ask, computing power and data are available in AWS data centers, and they outperform DePIN in terms of stability and user experience. Why choose DePIN over centralized services?
This argument certainly has its merits, as almost all large models today are developed directly or indirectly by major internet companies. Behind chatGPT is Microsoft, behind Gemini is Google, and almost every major internet company in China has its own large model. Why? Because only large internet companies have sufficient high-quality data and robust financial backing for computing power. But this is incorrect; people no longer want to be manipulated by internet giants.
On one hand, centralized AI poses risks to data privacy and security, potentially subject to censorship and control; on the other hand, AI created by internet giants further entrenches dependency and leads to market centralization, raising barriers to innovation.
from: https://www.gensyn.ai/
Humanity should not need a Martin Luther for an AI era; people should have the right to communicate directly with the divine.
From a Business Perspective: Cost Reduction and Efficiency Improvement are Key
Even setting aside the ideological debate between decentralization and centralization, there are still advantages to using DePIN for AI from a business perspective.
First, we need to clearly recognize that although internet giants possess a large amount of high-end GPU resources, the consumer-grade GPUs scattered among the public can also form a considerable computing power network, which is the long-tail effect of computing power. The idle rate of these consumer-grade GPUs is actually very high. As long as the incentives provided by DePIN exceed electricity costs, users will be motivated to contribute computing power to the network. At the same time, all physical facilities are managed by the users themselves, and the DePIN network does not have to bear the operational costs that centralized suppliers cannot avoid, focusing solely on the protocol design itself.
For data, the DePIN network can release the potential usability of data and reduce transmission costs through edge computing and other methods. Additionally, most distributed storage networks have automatic deduplication functions, reducing the workload of cleaning AI training data.
Finally, the crypto-economics brought by DePIN enhances the system's fault tolerance, promising a win-win situation for providers, consumers, and platforms.
from: UCLA
In case you don't believe it, a recent study from UCLA shows that using decentralized computing achieves 2.75 times the performance of traditional GPU clusters at the same cost, specifically being 1.22 times faster and 4.83 times cheaper.
Challenges Ahead: What Challenges Will AIxDePIN Face?
We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard.
------John Fitzgerald Kennedy
Building AI models using DePIN's distributed storage and distributed computing without trust still presents many challenges.
Work Verification
Essentially, computing deep learning models and PoW mining are both general-purpose computations, with the underlying signal changes between logic gates. On a macro level, PoW mining is "useless computation," attempting to derive a hash value with n leading zeros through countless random number generations and hash function calculations; whereas deep learning computation is "useful computation," calculating the parameter values of each layer in deep learning through forward and backward propagation to build an efficient AI model.
The fact is that "useless computation" like PoW mining uses hash functions, which are easy to verify because it is easy to compute the image from the pre-image but hard to compute the pre-image from the image; however, for deep learning model computations, due to the hierarchical structure where the output of each layer serves as the input for the next layer, verifying the validity of the computation requires executing all previous work, making it difficult to verify simply and effectively.
from: AWS
Work verification is crucial; otherwise, the provider of the computation could simply submit a randomly generated result without performing any computation.
One idea is to have different servers execute the same computation task, verifying the validity of the work by repeating the execution and checking for consistency. However, most model computations are non-deterministic, meaning that even in identical computing environments, the same results cannot be reproduced, and only statistical similarity can be achieved. Additionally, repeated computations would lead to rapidly rising costs, which contradicts the key goal of DePIN to reduce costs and improve efficiency.
Another idea is the Optimistic mechanism, which initially assumes that the results are valid computations while allowing anyone to verify the results. If errors are found, a Fraud Proof can be submitted, and the protocol penalizes the fraudster while rewarding the whistleblower.
Parallelization
As mentioned earlier, DePIN primarily leverages the long-tail consumer-grade computing power market, which means that the computing power provided by a single device is relatively limited. For large AI models, training on a single device would take a very long time, necessitating the use of parallelization to shorten the training time.
The main difficulty in parallelizing deep learning training lies in the dependencies between tasks. This dependency can make parallelization challenging.
Currently, parallelization in deep learning training is mainly divided into data parallelism and model parallelism.
Data parallelism refers to distributing data across multiple machines, where each machine retains all parameters of a model and trains using local data, ultimately aggregating the parameters from each machine. Data parallelism works well when the data volume is large but requires synchronized communication to aggregate parameters.
Model parallelism is used when the model size is too large to fit into a single machine, allowing the model to be split across multiple machines, with each machine storing a portion of the model's parameters. Communication between different machines is required during forward and backward propagation. Model parallelism has advantages when the model is large, but it incurs high communication overhead during propagation.
For gradient information between different layers, it can be further divided into synchronous updates and asynchronous updates. Synchronous updates are simple and direct but increase waiting time; asynchronous updates have shorter waiting times but introduce stability issues.
from: Stanford University, Parallel and Distributed Deep Learning
Privacy
There is a global movement to protect personal privacy, with governments strengthening protections for personal data privacy and security. Although AI extensively uses public datasets, what truly distinguishes different AI models is the proprietary user data of various companies.
How can we benefit from proprietary data during training without exposing privacy? How can we ensure that the parameters of the constructed AI model are not leaked?
These are two aspects of privacy: data privacy protects users, while model privacy protects the organizations that build the models. In the current context, data privacy is far more important than model privacy.
Various solutions are attempting to address privacy issues. Federated learning trains at the source of the data, keeping the data local while transmitting model parameters to ensure data privacy; meanwhile, zero-knowledge proofs may emerge as a promising solution.
Case Analysis: What Quality Projects Are Available in the Market?
Gensyn
Gensyn is a distributed computing network for training AI models. The network uses a layer-1 blockchain based on Polkadot to verify whether deep learning tasks have been executed correctly and triggers payments via commands. Founded in 2020, it disclosed a $43 million Series A funding round in June 2023, led by a16z.
Gensyn constructs certificates for the executed work using metadata from gradient-based optimization processes, and executes them consistently through multi-granularity, graph-based precise protocols and cross-evaluators, allowing for re-running verification work and comparing consistency, ultimately confirmed by the chain itself to ensure the validity of the computation. To further enhance the reliability of work verification, Gensyn introduces staking to create incentives.
There are four types of participants in the system: submitters, solvers, verifiers, and whistleblowers.
- Submitters are the end users of the system, providing tasks to be computed and paying for completed work units.
- Solvers are the main workers of the system, executing model training and generating proofs for verifiers to check.
- Verifiers are the key link between the non-deterministic training process and deterministic linear computations, replicating parts of the solver's proofs and comparing distances with expected thresholds.
- Whistleblowers are the last line of defense, checking the verifiers' work and raising challenges, receiving rewards if the challenge is successful.
Solvers need to stake, and whistleblowers verify the work of solvers. If wrongdoing is discovered, they can challenge, and if the challenge is successful, the tokens staked by the solver are forfeited, and the whistleblower is rewarded.
According to Gensyn's predictions, this solution is expected to reduce training costs to one-fifth of centralized providers.
from: Gensyn
FedML
FedML is a decentralized collaborative machine learning platform for decentralized and collaborative AI at any scale, anywhere. More specifically, FedML provides an MLOps ecosystem to train, deploy, monitor, and continuously improve machine learning models while collaborating on combined data, models, and computing resources in a privacy-preserving manner. Founded in 2022, FedML disclosed a $6 million seed round in March 2023.
FedML consists of two key components: FedML-API and FedML-core, representing high-level APIs and underlying APIs, respectively.
FedML-core includes two independent modules: distributed communication and model training. The communication module is responsible for the underlying communication between different workers/clients, based on MPI; the model training module is based on PyTorch.
FedML-API is built on top of FedML-core. With FedML-core, new distributed algorithms can be easily implemented using client-facing programming interfaces.
The latest work from the FedML team demonstrates that using FedML Nexus AI for AI model inference on consumer-grade GPU RTX 4090 is 20 times cheaper and 1.88 times faster than A100.
from: FedML
Future Outlook: DePIN Brings AI Democratization
One day, as AI further develops into AGI, computing power will become the de facto universal currency, and DePIN will accelerate this process.
The integration of AI and DePIN opens up a new technological growth point, providing tremendous opportunities for the development of artificial intelligence. DePIN offers vast distributed computing power and data for AI, aiding in training larger models and achieving stronger intelligence. At the same time, DePIN also drives AI towards a more open, secure, and reliable direction, reducing dependence on a single centralized infrastructure.
Looking ahead, AI and DePIN will continue to develop in synergy. Distributed networks will provide a strong foundation for training ultra-large models, which will play important roles in the applications of DePIN. While protecting privacy and security, AI will also assist in optimizing DePIN network protocols and algorithms. We look forward to AI and DePIN bringing a more efficient, fair, and trustworthy digital world.
This article is the second installment of the Web3xAI series report, and the CGV Research team will continue to delve into the theme of "the integration of Web3 and artificial intelligence (AI)," uncovering quality content. Stay tuned.
Reference
https://web.cs.ucla.edu/~harryxu/papers/dorylus-osdi21.pdf
https://web.stanford.edu/~rezab/classes/cme323/S16/projectsreports/hedgeusmani.pdf
About AIFocus Accelerator
AIFocus Accelerator was established in December 2023, initiated by CGV and Web3 Labs in Hong Kong, focusing on startups at the intersection of Web3 and artificial intelligence (AI). With the service philosophy of "invest first, accelerate later," the accelerator only provides customized services such as media promotion, organizing participation in industry summits, online collaboration, and thematic roadshows after confirming investment intentions, aiming to discover and support innovative projects that integrate Web3 and AI with forward-thinking and commercial prospects, helping them achieve rapid development. As of now, about 30 investment institutions, incubators, and laboratories are participating in the AIFocus Accelerator, and the list will be gradually announced after screening and confirmation.
About Cryptogram Venture (CGV):
CGV (Cryptogram Venture) is a crypto investment institution headquartered in Tokyo, Japan, investing in and incubating the licensed Japanese yen stablecoin JPYW. At the same time, CGV FoF is an LP for several globally renowned crypto funds. Since 2022, CGV has successfully organized two editions of the Japan Web3 Hackathon (TWSH), receiving joint support from institutions and experts such as the Japanese Ministry of Education, Culture, Sports, Science and Technology, Keio University, and NTT Docomo. Currently, CGV has branches in regions such as Hong Kong, Singapore, and New York.
