Machine Learning For Web3: Realizing The Potential and The Challenge of Censorship Resistance

A few things we need to get out of the way. First of all, what are LLMs and why is it now such a big deal? LLMS stands for Large Language Models. But before we had large language models, we had basic language models and most of us have already been using them, for example, the smart compose option in google’s gmail is using a language model. This works relatively well. But it writes short phrases and sentences. To get something meatier, we need to up the ante, and with that comes large language models.

Enter LLMs. What is important about them is that they are models that have billions of parameters. By the way, you all are already familiar with parameters. One of the earliest ones we all got exposed to is the parameters in the equation of a line, y = mx + b. In the formulation a parameter is simply an unknown variable that we need to calculate. In the equation of the line, the parameters are m and b.

m: represents the slope of the line. 

b: This is the y-intercept, which is the value of y when x is equal to 0. It determines the point where the line intersects the y-axis.

For comparison, the equation of line has 2 parameters, while GPT4 has over a trillion parameters. That is a lot of calculations. 

The key thing that made all this possible was the invention of something called transformers. It was introduced in 2017, in the truly revolutionary paper Attention Is All You Need. Since that paper dropped, we have seen some amazing achievements in all tasks associated with human and machine languages.

Attention in the context of the "Attention Is All You Need" paper, it refers to a way for the computer to pay special attention to different parts of a sentence or piece of text. The paper introduced the idea that using attention alone can be really powerful. Think of attention as a spotlight that helps the model understand which words or parts of a sentence are important at any given moment. Basically it reduces the number of things it needs to remember. This breakthrough was important because it simplified the model's architecture. It SIGNIFICANTLY improved the model's ability to understand the relationships between different words in a sentence and generate more accurate and meaningful text. This paper opened up new possibilities for building models that can understand and generate human-like language. And those possibilities are exactly what OpenAI ran with. 

At Fimio, we have run experiments using LLMs to extract embeddings that we use in models that can identify malicious patterns in Ethereum smart contracts. In our quest to run that experiment we discovered a few things about doing ML in the context of web3. But before I get into that, let me unpack what I mean by web3. When I saw web3, I mean the next evolution of the internet based on a few key primitives: 

Distributed

  • Distributed version control system that makes strong claims about history of a project based on cryptography

  • Git commits are cryptographically authentication of project history

  • Content based file system that makes claims about the evolution of files contents

  • It is very much like a blockchain. For the most part they share very similar underlying data structures, the merkle tree. Where they differ is that most of us only used git system from centralized hosting services like GitHub, and GitLab.

Decentralized

  • Means that anyone can run the software on their own server. As long as a version of that software is live somewhere, it is good to go.

  • Decentralization refers to the distribution of control and authority across multiple nodes or entities instead of relying on a single central authority. 

  • Git supports decentralized workflows through its ability to create multiple repositories, often referred to as "remotes," which can be hosted on different servers or machines. 

  • This allows individuals or teams to work independently on their own copies of the project, making it more resilient to central failures and enabling collaboration without a central point of control.

Permissionless

Cryptographic authentication

  • When you make a commit in Git, it generates a unique identifier called a "commit hash" based on the contents of the commit. This hash is a cryptographic representation of the commit and ensures that the commit's contents cannot be tampered with without changing the hash. 

  • Note that all these primitives do not mention economics. When we add economic incentive to it, then we have a cryptoeconomic system that allows the creation of cryptocurrencies and more complex incentive alignment mechanisms to achieve coordination in a decentralized system. In that case you have something like Ethereum which is a computer EVM that works in conjunction with a cryptocurrency, Ether.

If you have a system that is distributed, decentralized, permissionless, and cryptographically  authenticated, you build something that is Censorship Resistance. And that my friend is awesome!! Because Ethereum is censorship resistant that is why we can use it for currency. It is credibly neutral, meaning it isn’t designed to implement policies of one kind of institution.

One thing all these web3 decentralized web systems do is generate an ungodly amount of data. Especially when they get massive adoption. One of the main reasons why ChatGPT is so successful on code is because it has been trained on all the code that is inside of public GitHub repositories. And friends, that is an awesome amount of code. In the case of repositories, git itself is a content addressed file storage system that lends itself to beautiful data organization. There are SQL based database where you can write a simple query and get all the information you want downloaded to any computer. 

Friends, here are a few things I found out when trying to run our malicious smart contract detection experiment at Fimio

Learnings from doing ML on web3 data

  1. Web3 data is not optimized for batch read, instead the few systems that occur are designed for write. The core usecase for a blockchain is for you to store data, and often to do that, you write to the chain. ML usecases require batch read.

  2. Standardized data schema. We are yet to decide what we should call all the tables if there are even tables. Ethereum has a very very different data model than Solana. They both are blockchains and supposedly should allow the creation of a metaverse, where there is a world where you can take your data with you anywhere you want on the internet. That day is still not today. Web3 hasn’t sorted that out

  3. Human readable code. Bytecode. You have to decode ABI. ABI stands for Application Binary Interface. It defines how to interact with a contract deployed on a blockchain. It lets you know what the functions are, their inputs, outputs, and other relevant details that external applications or users need to know in order to interact with the contract. It is basically the  comms protocol between different components of a system. The ABI typically includes the following information:

  4. Functions: The ABI lists all the functions that can be called on the smart contract. It specifies the name of each function, the type and order of its inputs (parameters), and the expected output (return value).

  5. Events: Smart contracts can emit events to notify external applications or other contracts about specific occurrences. The ABI includes the definition of events, their names, and the types of data associated with each event.

  6. Data Types: The ABI also specifies the data types used by the contract, such as integers, addresses, strings, arrays, and more. It defines how these data types are represented and encoded.

  7. Very nascent developer tooling. Want to decompile bytecode and get the source code itself? Good luck. Or another one, a new blockchain comes around and invents a new language. Great. Is there a standard parser for that language that can turn the source code into an AST? Probably not.

Web3 is beta-level web infrastructure but we have treated it as if it is ready for production. There are quite a few significant things that need to be invented before we should unleash it on the consumer web. Let’s pause here for a minute and take a step back and reexamine censorship resistance and credible neutrality. 

The first thing to notice in these network illustrations is that they consist of nodes and edges. In the centralized network, there is one central node that connects through edges to all the other nodes in the network. In the decentralized network, the structure is significantly different. We can’t tell who is in control. 

This unlock was so incredible that we started building peer to peer internet money. This put the focus on the internet of ownership. I own this much ether, I own this NFT and so on. We didn’t think about the internet of censorship resistant mathematically verifiable connections. We spent most of our intellectual effort on the nodes on the graph versus the edges. And I think the edges are just as important, perhaps even more important when you start to think about the types of applications we can build with a fully censorship resistant stack.

For each layer of the stack, there are questions we need to ask?

  1. Privacy preserving open records.

  2. Can we improve ZK technology to allow for quick resolution proofs on complex ML training algorithms?

  3. Do we bring the GPU to the data or do we move the data to the GPU?

We need solutions at every level of that pyramid to truly unlock the promise of web3.