Validating

This section describes how run a Exzo Network validator node.

There are several clusters available to connect to, see choosing a Cluster for an overview of each.

Hardware Requirements

• CPU
  • 12 cores / 24 threads, or more
  • 2.8GHz, or faster
  • AVX2 instruction support (to use official release binaries, self-compile otherwise)
  • Support for AVX512f and/or SHA-NI instructions is helpful
  • The AMD Zen3 series is popular with the validator community
• RAM
  • 128GB, or more
  • Motherboard with 256GB capacity suggested
• Disk
  • PCIe Gen3 x4 NVME SSD, or better
  • Accounts: 500GB, or larger. High TBW (Total Bytes Written)
  • Ledger: 1TB or larger. High TBW suggested
  • OS: (Optional) 500GB, or larger. SATA OK
  • The OS may be installed on the ledger disk, though testing has shown better performance with the ledger on its own disk
  • Accounts and ledger can be stored on the same disk, however due to high IOPS, this is not recommended
  • The Samsung 970 and 980 Pro series SSDs are popular with the validator community
• GPUs
  • Not strictly necessary at this time
  • Motherboard and power supply speced to add one or more high-end GPUs in the future suggested

Suggested RPC Node Requirements

The above should be considered the bare minimum hardware requirements for running a RPC node as these nodes require more in-depth details, historical data and increased reliability. To provide the full functionality of an RPC node please upgrade the requirements to the recommended ones below.

• CPU
  • 16 cores / 32 threads, or more
• RAM
  • 256 GB, or more
• Disk
  • Consider a larger ledger disk if longer transaction history is required
  • Accounts and ledger should not be stored on the same disk

Virtual Machines on Cloud Platforms

While you can run a validator on a cloud computing platform, it may not be cost-efficient over the long term.

However, it may be convenient to run non-voting api nodes on VM instances for your own internal usage. This use case includes exchanges and services built on the Exzo Network.

In fact, the mainnet-beta validators operated by the team are currently (Mar. 2023) run on GCE n2-standard-32 (32 vCPUs, 128 GB memory) instances with 2048 GB SSD for operational convenience.

For other cloud platforms, select instance types with similar specs.

Also note that egress internet traffic usage may turn out to be high, especially for the case of running delegated validators.

Docker

Running validator for live clusters (including mainnet-beta) inside Docker is not recommended and generally not supported. This is due to concerns of general Docker's containerzation overhead and resultant performance degradation unless specially configured.

We use Docker only for development purposes.

Software

• We build and run on Ubuntu 20.04.
• See Installing Exzo Network for the current Exzo Network software release.

Be sure to ensure that the machine used is not behind a residential NAT to avoid NAT traversal issues. A cloud-hosted machine works best. Ensure that IP ports 8000 through 10000 are not blocked for Internet inbound and outbound traffic.

Prebuilt binaries are available for Linux x86_64 on CPUs supporting AVX2 (Ubuntu 20.04 recommended). MacOS or WSL users may build from source.

GPU Requirements

CUDA is required to make use of the GPU on your system. The provided Exzo Network release binaries are built on Ubuntu 20.04 with CUDA Toolkit 10.1 update 1. If your machine is using a different CUDA version than you will need to rebuild from source.

Stake Requirements

Anyone can start a validator node, however it will not generate rewards before it's total delegated stake reaches 10,000 XZO or more.

Configure the Exzo Network CLI

The Exzo Network cli includes get and set configuration commands to automatically set the --url argument for cli commands. For example:

exzo-network config set --url https://testnet.exzoscan.io/rpc

While this section demonstrates how to connect to the Devnet cluster, the steps are similar for the other Exzo Network Clusters.

Confirm The Cluster Is Reachable

Before attaching a validator node, sanity check that the cluster is accessible to your machine by fetching the transaction count:

exzo-network transaction-count

Confirm your Installation

Try running following command to join the gossip network and view all the other nodes in the cluster:

exzo-gossip spy --entrypoint bootstrap.testnet.exzoscan.io:8001
# Press ^C to exit

Enabling CUDA

If your machine has a GPU with CUDA installed (Linux-only currently), include the --cuda argument to exzo-network-validator.

When your validator is started look for the following log message to indicate that CUDA is enabled: "[<timestamp> solana::validator] CUDA is enabled"

System Tuning

Linux

Automatic

The Exzo Network repo includes a daemon to adjust system settings to optimize performance (namely by increasing the OS UDP buffer limits, and scheduling PoH with realtime policy).

The daemon (solana-sys-tuner) is included in the Exzo Network binary release. Restart it, before restarting your validator, after each software upgrade to ensure that the latest recommended settings are applied.

To run it:

sudo solana-sys-tuner --user $(whoami) > sys-tuner.log 2>&1 &

Manual

If you would prefer to manage system settings on your own, you may do so with the following commands.

Increase UDP buffers

sudo bash -c "cat >/etc/sysctl.d/20-solana-udp-buffers.conf <<EOF
# Increase UDP buffer size
net.core.rmem_default = 134217728
net.core.rmem_max = 134217728
net.core.wmem_default = 134217728
net.core.wmem_max = 134217728
EOF"

sudo sysctl -p /etc/sysctl.d/20-solana-udp-buffers.conf

Increased memory mapped files limit

sudo bash -c "cat >/etc/sysctl.d/20-solana-mmaps.conf <<EOF
# Increase memory mapped files limit
vm.max_map_count = 500000
EOF"

sudo sysctl -p /etc/sysctl.d/20-solana-mmaps.conf

Add

LimitNOFILE=500000

to the [Service] section of your systemd service file, if you use one, otherwise add it to /etc/systemd/system.conf.

sudo systemctl daemon-reload

sudo bash -c "cat >/etc/security/limits.d/90-solana-nofiles.conf <<EOF
# Increase process file descriptor count limit
* - nofile 500000
EOF"

### Close all open sessions (log out then, in again) ###

The identity public key can now be viewed by running:

exzo-network-keygen pubkey ~/validator-keypair.json

Paper Wallet identity

You can create a paper wallet for your identity file instead of writing the keypair file to disk with:

exzo-network-keygen new --no-outfile

The corresponding identity public key can now be viewed by running:

exzo-network-keygen pubkey ASK

and then entering your seed phrase.

See Paper Wallet Usage for more info.

This page describes how to set up an on-chain vote account. Creating a vote account is needed if you plan to run a validator node on Velas.

Create a Vote Account

A vote account can be created with the create-vote-account command. The vote account can be configured when first created or after the validator is running. All aspects of the vote account can be changed except for the vote account address, which is fixed for the lifetime of the account.

Configure an Existing Vote Account

• To change the validator identity, use vote-update-validator.
• To change the vote authority, use vote-authorize-voter.
• To change the withdraw authority, use vote-authorize-withdrawer.
• To change the commission, use vote-update-commission.

Vote Account Structure

Vote Account Address

A vote account is created at an address that is either the public key of a keypair file, or at a derived address based on a keypair file's public key and a seed string.

The address of a vote account is never needed to sign any transactions, but is just used to look up the account information.

When someone wants to delegate tokens in a stake account, the delegation command is pointed at the vote account address of the validator to whom the token-holder wants to delegate.

Validator Identity

The validator identity is a system account that is used to pay for all the vote transaction fees submitted to the vote account. Because the validator is expected to vote on most valid blocks it receives, the validator identity account is frequently (potentially multiple times per second) signing transactions and paying fees. For this reason the validator identity keypair must be stored as a "hot wallet" in a keypair file on the same system the validator process is running.

Because a hot wallet is generally less secure than an offline or "cold" wallet, the validator operator may choose to store only enough VLX on the identity account to cover voting fees for a limited amount of time, such as a few weeks or months. The validator identity account could be periodically topped off from a more secure wallet.

This practice can reduce the risk of loss of funds if the validator node's disk or file system becomes compromised or corrupted.

The validator identity is required to be provided when a vote account is created. The validator identity can also be changed after an account is created by using the vote-update-validator command.

Vote Authority

The vote authority keypair is used to sign each vote transaction the validator node wants to submit to the cluster. This doesn't necessarily have to be unique from the validator identity, as you will see later in this document. Because the vote authority, like the validator identity, is signing transactions frequently, this also must be a hot keypair on the same file system as the validator process.

The vote authority can be set to the same address as the validator identity. If the validator identity is also the vote authority, only one signature per vote transaction is needed in order to both sign the vote and pay the transaction fee. Because transaction fees on Exzo Network are assessed per-signature, having one signer instead of two will result in half the transaction fee paid compared to setting the vote authority and validator identity to two different accounts.

The vote authority can be set when the vote account is created. If it is not provided, the default behavior is to assign it the same as the validator identity. The vote authority can be changed later with the vote-authorize-voter command.

The vote authority can be changed at most once per epoch. If the authority is changed with vote-authorize-voter, this will not take effect until the beginning of the next epoch. To support a smooth transition of the vote signing, Exzo Network-validator allows the --authorized-voter argument to be specified multiple times. This allows the validator process to keep voting successfully when the network reaches an epoch boundary at which the validator's vote authority account changes.

Withdraw Authority

The withdraw authority keypair is used to withdraw funds from a vote account using the withdraw-from-vote-account command. Any network rewards a validator earns are deposited into the vote account and are only retrievable by signing with the withdraw authority keypair.

The withdraw authority is also required to sign any transaction to change a vote account's commission, and to change the validator identity on a vote account.

Because the vote account could accrue a significant balance, consider keeping the withdraw authority keypair in an offline/cold wallet, as it is not needed to sign frequent transactions.

The withdraw authority can be set at vote account creation with the --authorized-withdrawer option. If this is not provided, the validator identity will be set as the withdraw authority by default.

The withdraw authority can be changed later with the vote-authorize-withdrawer command.

Commission

Commission is the percent of network rewards earned by a validator that are deposited into the validator's vote account. The remainder of the rewards are distributed to all of the stake accounts delegated to that vote account, proportional to the active stake weight of each stake account.

For example, if a vote account has a commission of 10%, for all rewards earned by that validator in a given epoch, 10% of these rewards will be deposited into the vote account in the first block of the following epoch. The remaining 90% will be deposited into delegated stake accounts as immediately active stake.

A validator may choose to set a low commission to try to attract more stake delegations as a lower commission results in a larger percentage of rewards passed along to the delegator. As there are costs associated with setting up and operating a validator node, a validator would ideally set a high enough commission to at least cover their expenses.

Commission can be set upon vote account creation with the --commission option. If it is not provided, it will default to 100%, which will result in all rewards deposited in the vote account, and none passed on to any delegated stake accounts.

Commission can also be changed later with the vote-update-commission command.

When setting the commission, only integer values in the set [0-100] are accepted. The integer represents the number of percentage points for the commission, so creating an account with --commission 10 will set a 10% commission.

Key Rotation

Rotating the vote account authority keys require special handling when dealing with a live validator.

Vote Account Validator Identity

You will need access to the withdraw authority keypair for the vote account to change the validator identity. The follow steps assume that ~/withdraw-authority.json is that keypair.

1. Create the new validator identity keypair, Exzo Network-keygen new -o ~/new-validator-keypair.json.
2. Ensure that the new identity account has been funded, Exzo Network transfer ~/new-validator-keypair.json 500.
3. Run Exzo Network vote-update-validator ~/vote-account-keypair.json ~/new-validator-keypair.json ~/withdraw-authority.json to modify the validator identity in your vote account
4. Restart your validator with the new identity keypair for the --identity argument

Vote Account Authorized Voter

The vote authority keypair may only be changed at epoch boundaries and requires some additional arguments to Exzo Network-validator for a seamless migration.

1. Run Exzo Network epoch-info. If there is not much time remaining time in the current epoch, consider waiting for the next epoch to allow your validator plenty of time to restart and catch up.
2. Create the new vote authority keypair, Exzo Network-keygen new -o ~/new-vote-authority.json.
3. Determine the current vote authority keypair by running Exzo Network vote-account ~/vote-account-keypair.json. It may be validator's identity account (the default) or some other keypair. The following steps assume that ~/validator-keypair.json is that keypair.
4. Run Exzo Network vote-authorize-voter ~/vote-account-keypair.json ~/validator-keypair.json ~/new-vote-authority.json. The new vote authority is scheduled to become active starting at the next epoch.
5. Exzo Network-validator now needs to be restarted with the old and new vote authority keypairs, so that it can smoothly transition at the next epoch. Add the two arguments on restart: --authorized-voter ~/validator-keypair.json --authorized-voter ~/new-vote-authority.json
6. After the cluster reaches the next epoch, remove the --authorized-voter ~/validator-keypair.json argument and restart Exzo Network-validator, as the old vote authority keypair is no longer required.

Vote Account Authorized Withdrawer

No special handling is required. Use the Exzo Network vote-authorize-withdrawer command as needed.

By default your validator will have no stake. This means it will be ineligible to become leader.

Monitoring Catch Up

To delegate stake, first make sure your validator is running and has caught up to the cluster. It may take some time to catch up after your validator boots. Use the catchup command to monitor your validator through this process:

exzonetwork catchup ~/validator-keypair.json

Until your validator has caught up, it will not be able to vote successfully and stake cannot be delegated to it.

Also if you find the cluster's slot advancing faster than yours, you will likely never catch up. This typically implies some kind of networking issue between your validator and the rest of the cluster.

Create Stake Keypair

If you haven’t already done so, create a staking keypair. If you have completed this step, you should see the “validator-stake-keypair.json” in your Exzo Network runtime directory.

exzonetwork-keygen new -o ~/validator-stake-keypair.json

Delegate Stake

Now delegate 1 XZO to your validator by first creating your stake account:

exzonetwork create-stake-account ~/validator-stake-keypair.json 1

and then delegating that stake to your validator:

exzonetwork delegate-stake ~/validator-stake-keypair.json ~/vote-account-keypair.json

Don’t delegate your remaining XZO, as your validator will use those tokens to vote.

Stakes can be re-delegated to another node at any time with the same command, but only one re-delegation is permitted per epoch:

exzonetwork delegate-stake ~/validator-stake-keypair.json ~/some-other-vote-account-keypair.json

Assuming the node is voting, now you're up and running and generating validator rewards. Rewards are paid automatically on epoch boundaries.

The rewards lamport earned are split between your stake account and the vote account according to the commission rate set in the vote account. Rewards can only be earned while the validator is up and running. Further, once staked, the validator becomes an important part of the network. In order to safely remove a validator from the network, first deactivate its stake.

At the end of each slot, a validator is expected to send a vote transaction. These vote transactions are paid for by lamport from a validator's identity account.

This is a normal transaction so the standard transaction fee will apply. The transaction fee range is defined by the genesis block. The actual fee will fluctuate based on transaction load. You can determine the current fee via the RPC API “getRecentBlockhash” before submitting a transaction.

Validator Stake Warm-up

Stakes need to warm up, and warmup increments are applied at Epoch boundaries, so it can take an hour or more for stake to come fully online.

To monitor your validator during its warmup period:

• View your vote account:exzonetwork vote-account ~/vote-account-keypair.json This displays the current state of all the votes the validator has submitted to the network.
• View your stake account, the delegation preference and details of your stake:solana stake-account ~/validator-stake-keypair.json
• exzonetwork validators displays the current active stake of all validators, including yours
• exzonetwork stake-history shows the history of stake warming up and cooling down over recent epochs
• Look for log messages on your validator indicating your next leader slot: [2019-09-27T20:16:00.319721164Z INFO exzonetwork_core::replay_stage] <VALIDATOR_IDENTITY_PUBKEY> voted and reset PoH at tick height ####. My next leader slot is ####
• Once your stake is warmed up, you will see a stake balance listed for your validator by running exzonetwork validators

Monitor Your Staked Validator

Confirm your validator becomes a leader

• After your validator is caught up, use the exzonetwork balance command to monitor the earnings as your validator is selected as leader and collects transaction fees
• Exzo Network nodes offer a number of useful JSON-RPC methods to return information about the network and your validator's participation. Make a request by using curl (or another http client of your choosing), specifying the desired method in JSON-RPC-formatted data. For example:

// Request
  curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getEpochInfo"}' http://localhost:8899
 
  // Result
  {"jsonrpc":"2.0","result":{"epoch":3,"slotIndex":126,"slotsInEpoch":256},"id":1}

Helpful JSON-RPC methods:

• getEpochInfo An epoch is the time, i.e. number of slots, for which a leader schedule is valid. This will tell you what the current epoch is and how far into it the cluster is.
• getVoteAccounts This will tell you how much active stake your validator currently has. A % of the validator's stake is activated on an epoch boundary. You can learn more about staking on Exzo Network here.
• getLeaderSchedule At any given moment, the network expects only one validator to produce ledger entries. The validator currently selected to produce ledger entries is called the “leader”. This will return the complete leader schedule (on a slot-by-slot basis) for currently activated stake, the identity pubkey will show up 1 or more times here.

Deactivating Stake

Before detaching your validator from the cluster, you should deactivate the stake that was previously delegated by running:

exzonetwork deactivate-stake ~/validator-stake-keypair.json

Stake is not deactivated immediately and instead cools down in a similar fashion as stake warm up. Your validator should remain attached to the cluster while the stake is cooling down. While cooling down, your stake will continue to earn rewards. Only after stake cooldown is it safe to turn off your validator or withdraw it from the network. Cooldown may take several epochs to complete, depending on active stake and the size of your stake.

Note that a stake account may only be used once, so after deactivation, use the cli's withdraw-stake command to recover the previously staked lamport.

Check Gossip

Confirm the IP address and identity pubkey of your validator is visible in the gossip network by running:

exzonetwork-gossip spy --entrypoint bootstrap.exzonetwork.com:8001

Check Your Balance

Your account balance should decrease by the transaction fee amount as your validator submits votes, and increase after serving as the leader. Pass the --lamports are to observe in finer detail:

exzonetwork balance --lamports

Check Vote Activity

The exzonetwork vote-account command displays the recent voting activity from your validator:

exzonetwork vote-account ~/vote-account-keypair.json

Get Cluster Info

There are several useful JSON-RPC endpoints for monitoring your validator on the cluster, as well as the health of the cluster:

# Similar to exzonetwork-gossip, you should see your validator in the list of cluster nodes
curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getClusterNodes"}' https://explorer.testnet.exzonetwork.com/rpc
# If your validator is properly voting, it should appear in the list of `current` vote accounts. If staked, `stake` should be > 0
curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getVoteAccounts"}' https://explorer.testnet.exzonetwork.com/rpc
# Returns the current leader schedule
curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getLeaderSchedule"}' https://explorer.testnet.exzonetwork.com/rpc
# Returns info about the current epoch. slotIndex should progress on subsequent calls.
curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getEpochInfo"}' https://explorer.testnet.exzonetwork.com/rpc

You can publish your validator information to the chain to be publicly visible to other users.

Run Exzo Network validator-info

Run the Exzo Network CLI to populate a validator info account:

exzonetwork validator-info publish --keypair ~/validator-keypair.json <VALIDATOR_INFO_ARGS> <VALIDATOR_NAME>

For details about optional fields for VALIDATOR_INFO_ARGS:

exzonetwork validator-info publish --help

Example Commands

Example publish command:

exzonetwork validator-info publish "Elvis Validator" -n elvis -w "https://elvis-validates.com"

Example query command:

exzonetwork validator-info get

which outputs

Validator info from 8WdJvDz6obhADdxpGCiJKZsDYwTLNEDFizayqziDc9ah
Validator pubkey: 6dMH3u76qZ7XG4bVboVRnBHR2FfrxEqTTTyj4xmyDMWo
Info: {"keybaseUsername":"elvis","name":"Elvis Validator","website":"https://elvis-validates.com"}

Keybase

Including a Keybase username allows client applications (like the Exzo Network Explorer) to automatically pull in your validator public profile, including cryptographic proofs, brand identity, etc. To connect your validator pubkey with Keybase:

1. Join https://keybase.io/ and complete the profile for your validator
2. Add your validator identity pubkey to Keybase:

• Create an empty file on your local computer called validator-<PUBKEY>
• In Keybase, navigate to the Files section, and upload your pubkey file to a exzonetwork subdirectory in your public folder: /keybase/public/<KEYBASE_USERNAME>/exzonetwork
• To check your pubkey, ensure you can successfully browse to https://keybase.pub/<KEYBASE_USERNAME>/exzonetwork/validator-<PUBKEY>

3. Add or update your exzonetwork validator-info with your Keybase username. The CLI will verify the validator-<PUBKEY> file

Ode to Linux System Administration Generic Best Practices

Useful Links & Discussion

Native chain Explorer

EVM chain Explorer