Glenn's Digital Garden

VAST NFS

VAST has its own NFS client, plainly called VAST NFS which implements a bunch of mount options that improve performance. VAST makes every effort to upstream these options, but not all of them are accepted into Linux.

nconnect and remoteports

nconnect

This option allows a single NFS mount to utilize more than one connection to break through the 2 GB/s limit of a single mount.

Upstream Linux now supports this, but the VAST NFS driver supports high maximum values (e.g., 64 connections per mount).

remoteports / multipath

VAST supports allowing a single mount to access via multiple IP addresses. In order for this to work correctly, all IP addresses (NFS servers) must have a consistent view of the data it is serving. This is possible with some clustered NFS implementations, but not all clustered NFSes achieve the performance uplift that VAST does by specifying remoteports.

Interactions

When both nconnect and remoteports are specified, nconnect connections are distributed across remoteports. This results in the following behavior:

  • If you have nconnect=8 and specify eight IPs for remoteports, you get one connection per remoteport.
  • If you have nconnect=7 and eight remoteports, I don’t know what happens. I think you only use seven of the eight remote ports.
  • If you have nconnect=16 and eight remoteports, each remoteport gets two connections.
  • If you have nconnect=9 and eight remoteports, I don’t know what happens.

These semantics mean that, in order to actually use multipathing, your nconnect must be higher than 1. Otherwise, you only use one connection and one remote port, which is no different than not specifying nconnect or remoteports.

It’s generally a good idea to have nconnect be evenly divisible by the number of remoteports. This way, connections are evenly balanced across remote ports.

Spreading I/O

The vastnfs driver also supports two non-upstream options, spread_reads and spread_writes, which allow single-file accesses from a single client to be sprayed across multiple remoteports. This occurs at the RPC level, so the size of sharding is dictated by the mount options (rsize/wsize) for large writes, write size (if direct I/O is at play), or read size.

Graph View