Neo Protocol and Networking
The Neo network consists of two kind of protocols: a protocol to communicate with local clients and wallets, and an external protocol to communicate with other Neo nodes. To connect to a local node, JSON-RPC is used. This JSON-RPC can also be exposed to other external nodes.
+--------------+
+----------+ | +----------+ |
| | Neo Protocol | | | |
| Neo node +----------------+ Neo peer | |
| | | | | |
+----------+ | +----+-----+ |
| | |
| |JSON |
| |RPC |
| | |
| +----+-----+ |
| | | |
| |Local node| |
| | | |
| +----------+ |
+--------------+
Neo node
In this tutorial, we will focus on the other protocol, the Neo protocol . Using the Golang programming language, we will learn how to communicate with a Neo node.
Neo Ping with Golang
Although many core libraries of Neo are written in C# or Python, for this tutorial we will use Golang . The communication basics are the same for all languages.
The Neo protocol defines However, ia header and a payload. Every message needs to be sent with this specific format, with a 24 bytes header and its payload:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Magic: 0x00746E41 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| Command: 12 bytes |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-++-+-++-+-++-+-++-+-+-+-+-+-+-+-+
| Payload length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-++-+-++-+-++-+-++-+-+-+-+-+-+-+-+
| Payload checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-++-+-++-+-++-+-++-+-+-+-+-+-+-+-+
| |
| Payload |
Communication with other Neo nodes is handled via TCP or via WebSockets . The advantage of using WebSockets is that you could connect to a Neo node with a web browser, however in this tutorial we will use TCP.
The header contains a 4 bytes magic, which stands for "Ant" (little-endian), the name that was used before the Neo rebranding. The testnet uses slightly different magic bytes, allowing messages to be distinguished between the MainNet and TestNet.
Exercise 1 : Please answer the following question:
Although compatibility is very important and changing the format of the protocol could break many clients, what theoretical optimizations could be done in the protocol header?
In order to participate in the Neo distributed system, firstly we need to open a connection to a Neo node. Since we want to issue ping/pong commands, which was released in Neo version 2.10.1 on 4. April 2019, we need to make sure that our node has this version. A list of Neo nodes with versions can be found here: http://monitor.cityofzion.io/ . The following nodes announce this version at the time of writing:
-
node1.ledgercate.com:10333
-
seed.neoeconomy.io:10333
-
seed10.ngd.network:10333
Please note that peers can go offline at any time, and those peers above may not be online anymore. Any other Neo node above v2.10.1 may be substituted.
These nodes must be well reachable on the respective port. The default port in the mainnet is 10333. In case a node is behind a firewall, a node can use UPnP to configure the router to accept incoming connection to the node. NAT-PMP is currently not supported. In our tutorial we won't accept any incoming connections, that means we don't have any NAT issues.
Let's implement the protocol. Here is what we need to do: first we set the magic number, then we set the command followed by the payload length. Then we calculate the checksum and append the payload. Now, we can send our packet to one of the Neo nodes. Sounds easy, right?
Little-Endian vs. Big-Endian
For every protocol, it must be defined which byte order is used. In case of Neo its little-endian. Thus, the magic byte will be encoded as follows:
0x00746E41 -> [41][6E][74][00]
Big-endian encoding would like as follows: 00 6E. Many programming languages have utilities to handle the conversion, such as Golang:
binary.LittleEndian.PutUint32(b[0:], 0x00746E41)
While most CPUs use little-endian encoding, network protocols such as TCP or UDP use big-endian. More information on endianness can be found in https://en.wikipedia.org/wiki/Endianness . For this tutorial we will use little-endian, as most data is encoded in little-endian in Neo, and we won't touch the exceptions.
Checksum
The checksum field in the header is the checksum calculated over the payload. Although TCP also has a checksum over its payload, the TCP checksum is only 16 bits. The payload checksum in Neo is the first 4 bytes (32 bits) of the double SHA256 hash of the payload. In Golang, the checksum can be calculated as follows:
tmp := sha256.Sum256(payload)
hash := sha256.Sum256(tmp[:])
...
copy(b[20:], hash[0:4])
If the checksum does not match, the node should ignore the message.
Exercise 2 : Write an encoder and decoder for the header. Use the following "quick and dirty" template in Appendix A and complete the following two functions: encodeHeader(cmd string, payload []byte) []byte , where the return array is the header combined with the payload. The encoder should return the whole byte array that can be sent on the wire, and decodeHeader(b []byte) (uint32, uint32) , where the first return value is the length of the payload and the second value is the checksum.
Hint: the magic number can be encoded as: ` binary.LittleEndian.PutUint32(b[0:], 0x00746E41) and decoded as: `` fmt.Printf("magic: 0x%x\n", binary.LittleEndian.Uint32(b))
```
Payload protocol
After implementing the encoding and decoding of the headers for the Neo protocol, we can focus on the payload. The Neo protocol supports the following commands, though not every command has a payload.
-
version - information about the version of the node, including version number
-
verack - on successfully receiving of version information, send back verack (no payload)
-
getaddr - request to send a send active Neo nodes
-
addr - response to getaddr request
-
getblocks - request to send blocks
-
block - response to getblocks or getdata
-
consensus - response to getdata
-
filteradd - add data to the bloomfilter
-
filterclear - clear the bloomfilter
-
filterload - create the bloomfilter with its initial settings. The bloomfilter is used to return only blocks with transactions we are interested in
-
getdata - request a specific object. Response will be in tx, block, merkleblock, or consensus message
-
getheaders - request to send block headers
-
headers - response to getheaders
-
inv - send information about transactions, blocks, or consensus
-
mempool - request verified transactions from the mempool. Response will be in inv
-
ping - checks if the peer alive
-
pong - response to a ping message
-
tx - response to a getdata message
To write a Neo ping message it is important to understand that there is a strict order of commands that needs to be issued. For every connection, a version and its acknowledgment verack needs to be exchanged twice; once from your node and once from the remote node. The following flow diagram shows this sequence of commands:
+-------+ +-------+
| Neo1 | | Neo2 |
+-------+ +-------+
| |
| command version |
|-------------------->|
| |
| command version |
|<--------------------|
| |
| command verack |
|-------------------->|
| |
| command verack |
|<--------------------|
| |
| command ping |
|-------------------->|
| |
| command pong |
|<--------------------|
| |
For the Neo ping, we will only use the version and ping command with a payload. The verack command does not have any payload and will not be described. The payload of those two commands are defined as follows:
Command "version"
-
Version (uint32) specifies the version of the protocol, currently the version is 0
-
Services (uint64) - specifies the services, currently set to 1
-
Timestamp (uint32) - time in seconds since 01.01.1970
-
Port (uint16) - the port we are listening to, this can be 0 if your node does not handle incoming connections
-
Nonce (uint32) - random number
-
UserAgent max 1024 bytes - use 1 byte as the length in this tutorial, then the string. Don't go over 253 bytes (0xfd)
-
BlockHeight (uint32) - the block height, your block height can be 0
-
Relay (uint8) - if your node is a relay, set to false (0)
Command "ping"
-
BlockHeight (uint32) - the block height, your block height can be 0
-
Timestamp (uint32) - time in seconds since 01.01.1970
-
Nonce (uint32) - random number
Protocol
With this information we can implement the Neo node ping. First we need to write the encoders and decoders for the payload of version and ping.
Exercise 3 : Write an encoder and decoder for ping/version. Use the template and complete the following two functions: encodeVersion(userAgent string) []byte , encodePing() []byte , decodeVersion(b []byte) string , which returns the user agent, and decodePing(b []byte) . Print relevant information to the console in the decoder.
Hint: the timestamp can be encoded as:
binary.LittleEndian.PutUint32(b[12:], uint32(time.Now().Unix()))
and decoded as:
fmt.Printf("time: %v\n", time.Unix(int64(binary.LittleEndian.Uint32(b[12:])), 0))
Putting It together
First, connect to a Neo node that supports ping/pong. We will check for the correct version later on.
func main() {
remote, err := net.Dial("tcp", "node1.plutolo.gy:10333") //check: http://monitor.cityofzion.io/
if err != nil {
panic(err)
}
defer remote.Close()
fmt.Println("Conneced to: %v", remote.RemoteAddr())
Now we send our version to the remote Neo node. ` payloadVersion := encodeVersion("/Our NEO client:0.0.1/") packetVersion := encodeHeader("version", payloadVersion) n, err := remote.Write(packetVersion) if err != nil { panic(err) } fmt.Printf("wrote version packet: %v, %d\n", packetVersion, n) `
We wrote the version and now we can expect the version from the remote host. In fact, some clients also send it right after connection is established.
//we get the version from the remote, display it
read := make([]byte, 24)
n, err = io.ReadFull(remote, read) //read header
plen,rcvChecksum := decodeHeader(read)
read = make([]byte, plen)
n, err = io.ReadFull(remote, read) //read payload
userAgent := decodeVersion(read)
On receiving the version from the Neo node, we additionally check the checksum to ensure that they match. The checksum is the first 4 bytes of the double SHA256 hash of the payload.
tmp := sha256.Sum256(read)
hash := sha256.Sum256(tmp[:])
checksum := binary.LittleEndian.Uint32(hash[0:4])
fmt.Printf("read version payload: %v, %d\n", read, n)
if rcvChecksum != checksum {
panic(errors.New("checksum mismatch in version!"))
}
Since ping/pong was only implemented recently, we need to make sure that we ask a supported version. It looks that the versions use semantic versioning. However, the Python implementation uses different versioning, so we should also check which user agent is used. As there is no other implementation with such a high version 2.10.1, we can just check this version (its quick and dirty :).
//check if we have a good version
start := strings.Index(userAgent, ":")
end := strings.Index(userAgent[start:], "/")
if start < 0 && end < 0 {
panic(errors.New(fmt.Sprintf("cannot parse version in %s", userAgent)))
}
semVer := userAgent[start+1:start+end]
fmt.Printf("parsed semver: %v\n", semVer)
v1, err := version.NewVersion(semVer)
min, err := version.NewVersion("2.10.1")
if v1.LessThan(min) {
panic(errors.New(fmt.Sprintf("%s is less than %s", v1, min)))
}
Since we sent a version packet, we need to get the verack and we also need to send a verack, as we received the version as well:
////////// got version, send ack
packetVerack := encodeHeader("verack", []byte{})
n, err = remote.Write(packetVerack);
if err != nil {
panic(err)
}
///////// wait for verack confirmation
read = make([]byte, 24)
n, err = io.ReadFull(remote, read)
plen, rcvChecksum = decodeHeader(read)
fmt.Printf("read verack array: %v, %d\n", read, plen)
if rcvChecksum != 3806393949 {
panic(errors.New("checksum mismatch in verack!"))
}
After the version/verack, we can now send the ping. Without the versions, no commands can be sent.
/////// send ping
packet2 := encodeHeader("ping", encodePing())
n, err = remote.Write(packet2)
if err != nil {
panic(err)
}
fmt.Printf("wrote ping: %v, %d\n", packet2, n)
After sending the ping, we can expect a pong message.
//////// receive pong
read = make([]byte, 36)
_, err = io.ReadFull(remote, read)
_, rcvChecksum = decodeHeader(read)
decodePing(read[24 : 24+12])
fmt.Printf("read array: %v\n", read)
tmp = sha256.Sum256(read[24 : 24+12])
hash = sha256.Sum256(tmp[:])
checksum = binary.LittleEndian.Uint32(hash[0:4])
if rcvChecksum != checksum {
panic(errors.New("checksum mismatch in pong!"))
}
remote.Close()
}
Exercise 4 : Merge your encoder/decoder code with the main function from the template and run it on the MainNet. What can you see?
After successfully sending a Neo ping and receiving a pong back, we can send getaddr and receive further nodes in the Neo network. For that, you can use the connection you have already established. Finding other peers is crucial for P2P and distributed systems, as peers may go offline at any time, and having other nodes to connect to is vital.
Exercise 5 : After sending the ping message, send a getaddr message. Analyze the output.
Hint: The output contains a list of IP addresses (16 bytes IPv6/4, 2 bytes port) that are encoded big-endian. The data also contains a service flag (set to 1) and a timestamp.
You now have successfully implemented a Neo client that can initiate a connection to other Neo nodes, send a ping, receive a pong, and get a list of further peers.
Appendix A: Quick & Dirty Template
package main
import (
"bytes"
//"bytes"
"crypto/sha256"
"encoding/binary"
"errors"
"fmt"
"github.com/hashicorp/go-version"
"io"
//"math/rand"
"net"
"strings"
//"time"
)
func main() {
remote, err := net.Dial("tcp", "node1.plutolo.gy:10333") //check: http://monitor.cityofzion.io/
if err != nil {
panic(err)
}
defer remote.Close()
fmt.Println("Conneced to: %v", remote.RemoteAddr())
payloadVersion := encodeVersion("/The HSR NEO client:0.0.1/")
packetVersion := encodeHeader("version", payloadVersion)
n, err := remote.Write(packetVersion)
if err != nil {
panic(err)
}
fmt.Printf("wrote version packet: %v, %d\n", packetVersion, n)
//we get the version from the remote, display it
read := make([]byte, 24)
n, err = io.ReadFull(remote, read) //read header
plen, rcvChecksum := decodeHeader(read)
read = make([]byte, plen)
n, err = io.ReadFull(remote, read) //read payload
userAgent := decodeVersion(read)
tmp := sha256.Sum256(read)
hash := sha256.Sum256(tmp[:])
checksum := binary.LittleEndian.Uint32(hash[0:4])
fmt.Printf("read version payload: %v, %d\n", read, n)
if rcvChecksum != checksum {
panic(errors.New("checksum mismatch in version!"))
}
//check if we have a good version
start := strings.Index(userAgent, ":")
end := strings.Index(userAgent[start:], "/")
if start < 0 && end < 0 {
panic(errors.New(fmt.Sprintf("cannot parse version in %s", userAgent)))
}
semVer := userAgent[start+1 : start+end]
fmt.Printf("parsed semver: %v\n", semVer)
v1, err := version.NewVersion(semVer)
min, err := version.NewVersion("2.10.1")
if v1.LessThan(min) {
panic(errors.New(fmt.Sprintf("%s is less than %s", v1, min)))
}
////////// got version, send ack
packetVerack := encodeHeader("verack", []byte{})
n, err = remote.Write(packetVerack)
if err != nil {
panic(err)
}
///////// wait for verack confirmation
read = make([]byte, 24)
n, err = io.ReadFull(remote, read)
plen, rcvChecksum = decodeHeader(read)
fmt.Printf("read verack array: %v, %d\n", read, plen)
if rcvChecksum != 3806393949 {
panic(errors.New("checksum mismatch in verack!"))
}
/////// send ping
packet2 := encodeHeader("ping", encodePing())
n, err = remote.Write(packet2)
if err != nil {
panic(err)
}
fmt.Printf("wrote ping: %v, %d\n", packet2, n)
//////// receive pong
read = make([]byte, 36)
_, err = io.ReadFull(remote, read)
_, rcvChecksum = decodeHeader(read)
decodePing(read[24 : 24+12])
fmt.Printf("read array: %v\n", read)
tmp = sha256.Sum256(read[24 : 24+12])
hash = sha256.Sum256(tmp[:])
checksum = binary.LittleEndian.Uint32(hash[0:4])
if rcvChecksum != checksum {
panic(errors.New("checksum mismatch in pong!"))
}
remote.Close()
}
func encodeHeader(cmd string, payload []byte) []byte {
b := make([]byte, 24+len(payload))
binary.LittleEndian.PutUint32(b[0:], 0x00746E41)
//encoding here
copy(b[4:], cmd)
//payload length
binary.LittleEndian.PutUint32(b[16:], uint32(len(payload)))
//payload checksum
tmp := sha256.Sum256(payload)
hash := sha256.Sum256(tmp[:])
copy(b[20:], hash[0:4])
//payload
copy(b[24:], payload)
return b
}
func encodeVersion(userAgent string) []byte {
userAgentLen := len(userAgent)
b := make([]byte, 27+userAgentLen+1)
// encoding here
b[27+userAgentLen] = 0
return b
}
func encodePing() []byte {
b := make([]byte, 12)
// encoding here
return b
}
func decodeHeader(b []byte) (uint32, uint32) {
fmt.Printf("magic: 0x%x\n", binary.LittleEndian.Uint32(b))
fmt.Printf("command: %v\n", string(bytes.Trim(b[4:16], "\x00")))
len := binary.LittleEndian.Uint32(b[16:])
fmt.Printf("payload len: %d\n", len)
checksum := binary.LittleEndian.Uint32(b[20:])
fmt.Printf("checksum: 0x%x\n", checksum)
return len, checksum
}
func decodeVersion(b []byte) string {
// decoding here
//return userAgent
return ""
}
func decodePing(b []byte) {
// decoding here
}