nvmeof API#

Controller and namespace#

Userspace NVMe over Fabrics RDMA with direct host and GPU I/O.

class nvmeof.Completion(result, sq_head, sq_id, command_id, status_field)#

Bases: object

Decoded 16-byte NVMe completion queue entry.

status_field retains the phase bit exactly as received. The derived status properties remove that bit and expose the NVMe status code, status code type, and do-not-retry flag.

Parameters:

result (int) –
sq_head (int) –
sq_id (int) –
command_id (int) –
status_field (int) –

result: int#

sq_head: int#

sq_id: int#

command_id: int#

status_field: int#

classmethod from_bytes(data)#

Decode the first 16 bytes of an NVMe completion queue entry.

Parameters:: data (bytes) –
Return type:: Completion

property status: int#: Return the status field with its phase bit removed.

property status_code: int#: Return the eight-bit NVMe status code (SC).

property status_type: int#: Return the three-bit status code type (SCT).

property do_not_retry: bool#: Whether the target marked this command as not retryable.

property succeeded: bool#: Whether both the status code and status code type are zero.

raise_for_status()#

Raise NVMeStatusError unless the command succeeded.

Return type:: None

class nvmeof.Controller(host, subsystem_nqn, *, port=4420, host_id=None, host_nqn=None, queue_depth=128, keep_alive_ms=0, timeout=30.0, source=None)#

Bases: object

A connected userspace NVMe-oF/RDMA controller.

Construction connects an admin queue, enables and identifies the remote controller, then creates one direct I/O queue. Use connect() for a descriptive constructor or instantiate the class directly. Controllers are context managers and should be closed after every registered buffer.

Parameters:

host (str) – Target hostname or IP address.
subsystem_nqn (str) – NQN of the target subsystem to connect to.
port – NVMe/RDMA service port, normally nvmeof.protocol.NVME_RDMA_PORT.
host_id – Stable initiator UUID. A random UUID is generated by default.
host_nqn – Initiator NQN. By default it is derived from host_id.
queue_depth – Requested I/O queue depth from 2 through 256. The target may negotiate a smaller depth.
keep_alive_ms – Keep-alive timeout. Only zero is currently supported.
timeout – Command timeout in seconds.
source – Optional initiator IP address used to select and bind the local RDMA device.

classmethod connect(host, subsystem_nqn, **kwargs)#

Connect to subsystem_nqn at host and return a controller.

Parameters:

host (str) –
subsystem_nqn (str) –

Return type:

Controller

identify_namespace(nsid)#

Identify nsid and return its high-level I/O interface.

Namespaces using separate metadata are rejected because this initiator currently transfers data blocks only.

Parameters:: nsid (int) –
Return type:: Namespace

namespace(nsid)#

Identify nsid and return its high-level I/O interface.

Namespaces using separate metadata are rejected because this initiator currently transfers data blocks only.

Parameters:: nsid (int) –
Return type:: Namespace

allocate(length)#

Allocate host memory registered for direct NVMe/RDMA I/O.

length is the positive allocation size in bytes. The returned buffer is owned by the controller’s I/O protection domain and must be closed before the controller.

Parameters:: length (int) –
Return type:: HostBuffer

register(tensor)#

The returned MR retains the allocation and must be closed before the controller. Use register_gpu() for CUDA tensors.

register_gpu(tensor)#

The returned ibverbs.cuda.GpuMR retains tensor and must be closed before the controller.

close()#

Close the I/O and admin queues and release their RDMA resources.

Return type:: None

class nvmeof.ControllerInfo(serial, model, firmware, mdts, controller_id, version, max_commands, namespace_count, sgls, ioccsz, iorcsz, icdoff)#

Bases: object

Controller fields consumed from a 4096-byte Identify response.

The record includes display identity, MDTS, controller and namespace limits, keyed-SGL capability bits, and NVMe-oF capsule sizing fields.

Parameters:

serial (str) –
model (str) –
firmware (str) –
mdts (int) –
controller_id (int) –
version (int) –
max_commands (int) –
namespace_count (int) –
sgls (int) –
ioccsz (int) –
iorcsz (int) –
icdoff (int) –

serial: str#

model: str#

firmware: str#

mdts: int#

controller_id: int#

version: int#

max_commands: int#

namespace_count: int#

sgls: int#

ioccsz: int#

iorcsz: int#

icdoff: int#

classmethod from_bytes(data)#

Parse an Identify Controller data structure.

Parameters:: data (bytes) –
Return type:: ControllerInfo

class nvmeof.HostBuffer(pd, length, access=None)#

Bases: object

Pinned host allocation registered with an NVMe/RDMA queue’s PD.

Parameters:

pd – Protection domain used to register the allocation.
length (int) – Allocation size in bytes.
access – Optional verbs access mask. By default the buffer permits the local and remote operations required by NVMe/RDMA.

The buffer is a context manager. Close it before closing the protection domain or controller that owns pd.

property rkey: int#: Return the remote key encoded in NVMe keyed SGL descriptors.

property lkey: int#: Return the local key used for initiator work requests.

property closed: bool#: Whether the underlying memory region has been deregistered.

write(data, offset=0)#

Copy data into this allocation at byte offset.

Parameters:

data (bytes) –
offset (int) –

Return type:

None

read(length=None, offset=0)#

Copy bytes from this allocation into a new bytes object.

length defaults to all bytes from offset through the end of the allocation.

Parameters:: offset (int) –
Return type:: bytes

close()#

Deregister the memory region.

Return type:: None

class nvmeof.Namespace(controller, info)#

Bases: object

An identified NVM namespace accessed through a controller’s I/O queue.

info contains the full identified geometry; nsid and lba_size provide the fields most I/O callers need.

Parameters:

controller (Controller) –
info (p.NamespaceInfo) –

read(buffer, slba, blocks=None, *, buffer_offset=0)#

Read namespace LBAs directly into a registered host/GPU buffer.

blocks defaults to the whole remaining buffer and must describe a whole number of LBAs. Transfers exceeding MDTS are split into serial commands. A GPU read flushes completed GPUDirect writes before return.

Parameters:

buffer – Registered host MR or ibverbs.cuda.GpuMR.
slba (int) – Starting logical block address in the namespace.
blocks – Logical block count, or None to use the remaining registered buffer.
buffer_offset (int) – Starting byte offset in buffer.

Return type:

None

write(buffer, slba, blocks=None, *, buffer_offset=0)#

Write namespace LBAs directly from a registered host/GPU buffer.

blocks defaults to the whole remaining buffer and must describe a whole number of LBAs. Transfers exceeding MDTS are split into serial commands. CUDA work is synchronized before a GPU write is submitted.

Parameters:

buffer – Registered host MR or ibverbs.cuda.GpuMR.
slba (int) – Starting logical block address in the namespace.
blocks – Logical block count, or None to use the remaining registered buffer.
buffer_offset (int) – Starting byte offset in buffer.

Return type:

None

flush()#

Commit volatile write data for this namespace to nonvolatile media.

Return type:: None

class nvmeof.NamespaceInfo(nsid, size_lbas, capacity_lbas, used_lbas, lba_size, metadata_size)#

Bases: object

Namespace capacity and active LBA-format information.

Parameters:

nsid (int) –
size_lbas (int) –
capacity_lbas (int) –
used_lbas (int) –
lba_size (int) –
metadata_size (int) –

nsid: int#

size_lbas: int#

capacity_lbas: int#

used_lbas: int#

lba_size: int#

metadata_size: int#

classmethod from_bytes(nsid, data)#

Parse an Identify Namespace data structure for nsid.

Parameters:

nsid (int) –
data (bytes) –

Return type:

NamespaceInfo

exception nvmeof.NVMeStatusError(completion)#

Bases: OSError

An NVMe command completed with a non-zero status.

Parameters:: completion (Completion) –

exception nvmeof.QueueFullError#

Bases: RuntimeError

Raised when every command ID on an NVMe/RDMA queue is in use.

class nvmeof.RDMAQueue(host, port, qid, depth, controller_id=0, source=None)#

Bases: object

One connected NVMe/RDMA submission/completion queue pair.

This is the low-level command transport used by Controller. Queue construction resolves the route, creates an RC QP, connects with NVMe/RDMA private data, and posts the response receives.

Parameters:

host (str) – Target hostname or IP address.
port (int) – Target NVMe/RDMA service port.
qid (int) – NVMe queue identifier. Zero creates an admin queue.
depth (int) – Queue depth from 2 through 256. At most depth - 1 commands may be outstanding.
controller_id (int) – Connected controller ID for an I/O queue.
source – Optional initiator IP address used to select the local HCA.

property closed: bool#: Whether queue shutdown has begun.

property outstanding: int#: Return the number of commands awaiting transport completion.

submit(command, data_owner=None)#

Submit one 64-byte command capsule without waiting.

The queue assigns and writes the command ID. data_owner should own any buffer addressed by the command’s keyed SGL so it remains alive until the returned request completes. Call poll() to advance and retrieve completed requests.

Raises:: QueueFullError – If depth - 1 commands are already outstanding.
Parameters:: command (bytes) –
Return type:: Request

poll(max_entries=None)#

Poll transport completions and return newly completed requests.

A request is returned only after its capsule SEND completion and matching NVMe response have both arrived. This method does not block; max_entries defaults to enough entries to drain the CQ.

execute(command, data_owner=None, timeout=30.0)#

Submit a command and wait for its successful NVMe completion.

Parameters:

command (bytes) – Exactly one 64-byte NVMe command capsule.
data_owner – Optional owner of the command’s keyed-SGL buffer.
timeout (float) – Maximum wait in seconds.

Returns:

The matching Completion.

Raises:

TimeoutError – If the request does not finish before timeout.
nvmeof.protocol.NVMeStatusError – If the target returns a non-zero NVMe status.

close()#

Disconnect and close every RDMA resource owned by this queue.

Return type:: None

class nvmeof.Request(command_id, data_owner=None, send_complete=False, response=None, done=False)#

Bases: object

State retained for an asynchronously submitted NVMe command.

A request becomes done only after both the capsule SEND and the matching NVMe response complete. data_owner keeps any keyed-SGL allocation alive while the command is outstanding. Callers should treat the state fields as read-only.

Parameters:

command_id (int) –
data_owner (object) –
send_complete (bool) –
response (Completion | None) –
done (bool) –

command_id: int#

data_owner: object = None#

send_complete: bool = False#

response: Completion | None = None#

done: bool = False#

NVMe/RDMA transport#

NVMe/RDMA queue transport built directly on ibverbs.

exception nvmeof.rdma.QueueFullError#

Bases: RuntimeError

Raised when every command ID on an NVMe/RDMA queue is in use.

class nvmeof.rdma.HostBuffer(pd, length, access=None)#

Bases: object

Pinned host allocation registered with an NVMe/RDMA queue’s PD.

Parameters:

pd – Protection domain used to register the allocation.
length (int) – Allocation size in bytes.
access – Optional verbs access mask. By default the buffer permits the local and remote operations required by NVMe/RDMA.

The buffer is a context manager. Close it before closing the protection domain or controller that owns pd.

property rkey: int#: Return the remote key encoded in NVMe keyed SGL descriptors.

property lkey: int#: Return the local key used for initiator work requests.

property closed: bool#: Whether the underlying memory region has been deregistered.

write(data, offset=0)#

Copy data into this allocation at byte offset.

Parameters:

data (bytes) –
offset (int) –

Return type:

None

read(length=None, offset=0)#

Copy bytes from this allocation into a new bytes object.

length defaults to all bytes from offset through the end of the allocation.

Parameters:: offset (int) –
Return type:: bytes

close()#

Deregister the memory region.

Return type:: None

class nvmeof.rdma.Request(command_id, data_owner=None, send_complete=False, response=None, done=False)#

Bases: object

State retained for an asynchronously submitted NVMe command.

Parameters:

command_id (int) –
data_owner (object) –
send_complete (bool) –
response (Completion | None) –
done (bool) –

command_id: int#

data_owner: object = None#

send_complete: bool = False#

response: Completion | None = None#

done: bool = False#

class nvmeof.rdma.RDMAQueue(host, port, qid, depth, controller_id=0, source=None)#

Bases: object

One connected NVMe/RDMA submission/completion queue pair.

This is the low-level command transport used by Controller. Queue construction resolves the route, creates an RC QP, connects with NVMe/RDMA private data, and posts the response receives.

Parameters:

host (str) – Target hostname or IP address.
port (int) – Target NVMe/RDMA service port.
qid (int) – NVMe queue identifier. Zero creates an admin queue.
depth (int) – Queue depth from 2 through 256. At most depth - 1 commands may be outstanding.
controller_id (int) – Connected controller ID for an I/O queue.
source – Optional initiator IP address used to select the local HCA.

property closed: bool#: Whether queue shutdown has begun.

property outstanding: int#: Return the number of commands awaiting transport completion.

submit(command, data_owner=None)#

Submit one 64-byte command capsule without waiting.

Raises:: QueueFullError – If depth - 1 commands are already outstanding.
Parameters:: command (bytes) –
Return type:: Request

poll(max_entries=None)#

Poll transport completions and return newly completed requests.

A request is returned only after its capsule SEND completion and matching NVMe response have both arrived. This method does not block; max_entries defaults to enough entries to drain the CQ.

execute(command, data_owner=None, timeout=30.0)#

Submit a command and wait for its successful NVMe completion.

Parameters:

command (bytes) – Exactly one 64-byte NVMe command capsule.
data_owner – Optional owner of the command’s keyed-SGL buffer.
timeout (float) – Maximum wait in seconds.

Returns:

The matching Completion.

Raises:

TimeoutError – If the request does not finish before timeout.
nvmeof.protocol.NVMeStatusError – If the target returns a non-zero NVMe status.

close()#

Disconnect and close every RDMA resource owned by this queue.

Return type:: None

Protocol#

NVMe, NVMe-oF, and NVMe/RDMA wire layouts used by the initiator.

nvmeof.protocol.NVME_RDMA_PORT = 4420#: IANA service port for NVMe over Fabrics/RDMA.

nvmeof.protocol.ADMIN_QUEUE_DEPTH = 32#: Queue depth used for the controller’s admin queue.

nvmeof.protocol.MAX_QUEUE_DEPTH = 256#: Largest queue supported by this initiator’s command-ID layout.

nvmeof.protocol.MAX_KEYED_SGL_LENGTH = 16777215#: Largest byte count representable by one keyed SGL descriptor.

nvmeof.protocol.OPC_FLUSH = 0#: NVM Flush opcode.

nvmeof.protocol.OPC_WRITE = 1#: NVM Write opcode.

nvmeof.protocol.OPC_READ = 2#: NVM Read opcode.

nvmeof.protocol.OPC_IDENTIFY = 6#: Admin Identify opcode.

nvmeof.protocol.OPC_SET_FEATURES = 9#: Admin Set Features opcode.

nvmeof.protocol.OPC_FABRICS = 127#: Fabrics command opcode.

nvmeof.protocol.FCTYPE_PROPERTY_SET = 0#: Fabrics Property Set command type.

nvmeof.protocol.FCTYPE_CONNECT = 1#: Fabrics Connect command type.

nvmeof.protocol.FCTYPE_PROPERTY_GET = 4#: Fabrics Property Get command type.

nvmeof.protocol.REG_CAP = 0#: Controller Capabilities property offset.

nvmeof.protocol.REG_VS = 8#: Controller Version property offset.

nvmeof.protocol.REG_CC = 20#: Controller Configuration property offset.

nvmeof.protocol.REG_CSTS = 28#: Controller Status property offset.

nvmeof.protocol.CC_ENABLE = 1#: Controller Configuration enable bit.

nvmeof.protocol.CC_CSS_CSI = 96#: Controller Configuration command-set selection for CSI.

nvmeof.protocol.CC_MPS_SHIFT = 7#: Controller Configuration memory-page-size shift.

nvmeof.protocol.CC_IOSQES = 393216#: Controller Configuration I/O submission queue entry size.

nvmeof.protocol.CC_IOCQES = 4194304#: Controller Configuration I/O completion queue entry size.

nvmeof.protocol.CSTS_READY = 1#: Controller Status ready bit.

nvmeof.protocol.CSTS_FATAL = 2#: Controller Status fatal-status bit.

nvmeof.protocol.FEAT_NUMBER_OF_QUEUES = 7#: Set Features identifier for Number of Queues.

nvmeof.protocol.PSDT_SGL = 64#: Command data-pointer flag selecting an SGL.

nvmeof.protocol.KEYED_DATA_BLOCK = 64#: NVMe/RDMA keyed data block SGL descriptor type.

exception nvmeof.protocol.NVMeStatusError(completion)#

Bases: OSError

An NVMe command completed with a non-zero status.

Parameters:: completion (Completion) –

class nvmeof.protocol.Completion(result, sq_head, sq_id, command_id, status_field)#

Bases: object

Decoded 16-byte NVMe completion queue entry.

status_field retains the phase bit exactly as received. The derived status properties remove that bit and expose the NVMe status code, status code type, and do-not-retry flag.

Parameters:

result (int) –
sq_head (int) –
sq_id (int) –
command_id (int) –
status_field (int) –

result: int#

sq_head: int#

sq_id: int#

command_id: int#

status_field: int#

classmethod from_bytes(data)#

Decode the first 16 bytes of an NVMe completion queue entry.

Parameters:: data (bytes) –
Return type:: Completion

property status: int#: Return the status field with its phase bit removed.

property status_code: int#: Return the eight-bit NVMe status code (SC).

property status_type: int#: Return the three-bit status code type (SCT).

property do_not_retry: bool#: Whether the target marked this command as not retryable.

property succeeded: bool#: Whether both the status code and status code type are zero.

raise_for_status()#

Raise NVMeStatusError unless the command succeeded.

Return type:: None

class nvmeof.protocol.ControllerInfo(serial, model, firmware, mdts, controller_id, version, max_commands, namespace_count, sgls, ioccsz, iorcsz, icdoff)#

Bases: object

Controller fields consumed from a 4096-byte Identify response.

The record includes display identity, MDTS, controller and namespace limits, keyed-SGL capability bits, and NVMe-oF capsule sizing fields.

Parameters:

serial (str) –
model (str) –
firmware (str) –
mdts (int) –
controller_id (int) –
version (int) –
max_commands (int) –
namespace_count (int) –
sgls (int) –
ioccsz (int) –
iorcsz (int) –
icdoff (int) –

serial: str#

model: str#

firmware: str#

mdts: int#

controller_id: int#

version: int#

max_commands: int#

namespace_count: int#

sgls: int#

ioccsz: int#

iorcsz: int#

icdoff: int#

classmethod from_bytes(data)#

Parse an Identify Controller data structure.

Parameters:: data (bytes) –
Return type:: ControllerInfo

class nvmeof.protocol.NamespaceInfo(nsid, size_lbas, capacity_lbas, used_lbas, lba_size, metadata_size)#

Bases: object

Namespace capacity and active LBA-format information.

Parameters:

nsid (int) –
size_lbas (int) –
capacity_lbas (int) –
used_lbas (int) –
lba_size (int) –
metadata_size (int) –

nsid: int#

size_lbas: int#

capacity_lbas: int#

used_lbas: int#

lba_size: int#

metadata_size: int#

classmethod from_bytes(nsid, data)#

Parse an Identify Namespace data structure for nsid.

Parameters:

nsid (int) –
data (bytes) –

Return type:

NamespaceInfo

nvmeof.protocol.set_keyed_sgl(command, buffer, length=None, offset=0)#

Point a command at a registered host MR or ibverbs.cuda.GpuMR.

buffer must expose addr, length, and rkey. The selected range must fit one 24-bit keyed data block descriptor.

Parameters:: command (bytearray) –
Return type:: None

nvmeof.protocol.rdma_cm_request(qid, depth, controller_id=0)#

Build the 32-byte NVMe/RDMA connection-management request record.

The admin queue (qid=0) always encodes controller ID zero. I/O queues encode the ID returned by the admin Fabrics Connect command.

Parameters:

qid (int) –
depth (int) –
controller_id (int) –

Return type:

bytes

nvmeof.protocol.parse_rdma_cm_response(data)#

Validate NVMe/RDMA CM response data and return its receive queue size.

Parameters:: data (bytes) –
Return type:: int

nvmeof.protocol.connect_data(host_id, subsystem_nqn, host_nqn, controller_id)#

Build the 1024-byte Fabrics Connect data structure.

host_id accepts a uuid.UUID or UUID string. Both NQNs must contain 11 through 223 non-NUL bytes when UTF-8 encoded.

Parameters:

subsystem_nqn (str) –
host_nqn (str) –
controller_id (int) –

Return type:

bytes

nvmeof.protocol.fabrics_connect(qid, depth, data_buffer, kato_ms=0)#

Build a Fabrics Connect command using data_buffer as its payload.

Parameters:

qid (int) –
depth (int) –
kato_ms (int) –

Return type:

bytes

nvmeof.protocol.property_get(offset, size)#

Build a Fabrics Property Get command for a 4- or 8-byte register.

Parameters:

offset (int) –
size (int) –

Return type:

bytes

nvmeof.protocol.property_set(offset, value, size=4)#

Build a Fabrics Property Set command for a 4- or 8-byte register.

Parameters:

offset (int) –
value (int) –
size (int) –

Return type:

bytes

nvmeof.protocol.identify(data_buffer, *, nsid=0, controller=False)#

Build an Identify Controller or Identify Namespace command.

data_buffer must provide at least 4096 registered bytes. Pass controller=True for controller data; otherwise nsid selects the namespace.

Parameters:

nsid (int) –
controller (bool) –

Return type:

bytes

nvmeof.protocol.set_features(feature_id, value)#

Build a Set Features command with command-dword 11 value.

Parameters:

feature_id (int) –
value (int) –

Return type:

bytes

nvmeof.protocol.rw_command(opcode, nsid, slba, blocks, buffer, *, lba_size, buffer_offset=0)#

Build one NVM Read or Write command with a keyed SGL.

A command may transfer 1 through 65536 logical blocks, subject to the 24-bit keyed-SGL length limit. Higher-level namespace I/O splits larger transfers before calling this builder.

Parameters:

opcode (int) –
nsid (int) –
slba (int) –
blocks (int) –
lba_size (int) –
buffer_offset (int) –

Return type:

bytes