ibverbs API#

Core API#

ibverbs: low-level Pythonic bindings for libibverbs (RDMA).

Quickstart:

import ibverbs as ib

dev = ib.get_device_list()[0]
ctx = dev.open()
pd = ctx.alloc_pd()
cq = ctx.create_cq(16)
mr = pd.reg_mr(buf_addr, nbytes,
               ib.AccessFlags.LOCAL_WRITE | ib.AccessFlags.REMOTE_WRITE)
qp = pd.create_qp(ib.QPInitAttr(send_cq=cq, recv_cq=cq))

The compiled fast paths live in ibverbs._ibverbs; enums in ibverbs.enums; thin RC helpers in ibverbs.helpers.

class ibverbs.AH#

Bases: object

An address handle (ibv_ah) for UD sends.

close(self)#: Destroy this address handle.

pd#

class ibverbs.CQ#

Bases: object

A completion queue (ibv_cq).

ack_events(self, unsigned int nevents=1)#: Acknowledge nevents events delivered via the channel.

channel#

close(self)#: Destroy this completion queue.

context#

cqe#

int

Return the completion queue’s actual entry capacity.

Type:: CQ.cqe

poll(self, int num_entries) → list#

Poll up to num_entries completions; return a list of WC.

Return type:: list

req_notify(self, bool solicited_only=False)#: Request a completion notification on the CQ’s channel.

class ibverbs.MR#

Bases: object

A registered memory region (ibv_mr).

addr#

int

Return the registered region’s starting virtual address.

Type:: MR.addr

close(self)#: Deregister this memory region.

closed#

bool

Whether this memory region has been deregistered.

Type:: MR.closed

handle#

int

Return the provider memory-region handle.

Type:: MR.handle

length#

int

Return the registered length in bytes.

Type:: MR.length

lkey#

int

Return the local key used by this process’s QPs.

Type:: MR.lkey

owner#

pd#

rkey#

int

Return the key that authorizes remote access.

Type:: MR.rkey

sge(self, length=None, offset=0)#: Return an SGE for a bounded range of this MR.

class ibverbs.PD#

Bases: object

A protection domain (ibv_pd).

close(self)#: Deallocate the protection domain when it has no open children.

context#

create_ah(self, AHAttr attr) → 'AH'#

Create an address handle from attr for UD communication.

Return type:: AH

create_qp(self, init_attr) → 'QP'#

Create a queue pair from QPInitAttr.

Return type:: QP

create_srq(self, max_wr=128, max_sge=1, srq_limit=0) → 'SRQ'#

Create a shared receive queue with the requested capacities.

max_wr is the posted receive limit, max_sge is the SGE limit per receive, and a non-zero srq_limit requests an asynchronous low-watermark event when the available receives fall below it.

Return type:: SRQ

reg_dmabuf_mr(self, offset, length, iova, int fd, int access) → 'MR'#

Return type:: MR

reg_mr(self, addr, length, int access) → 'MR'#

addr is an integer virtual address; it may be a host pointer or a CUDA device pointer (GPUDirect via nvidia_peermem).

Return type:: MR

class ibverbs.QP#

Bases: object

A queue pair (ibv_qp).

close(self)#: Destroy this queue pair.

modify(self, int attr_mask, ah_attr=None, **fields)#

Low-level ibv_modify_qp.

attr_mask is an ORed QPAttrMask. Scalar attributes are passed as keywords (e.g. qp_state=, port_num=); the address vector is passed as an AHAttr via ah_attr=.

pd#

post_recv(self, wrs)#: Post one or more RecvWR to the receive queue.

post_send(self, wrs)#: Post one or more SendWR to the send queue.

qp_num#

int

Return the provider-assigned queue-pair number.

Type:: QP.qp_num

qp_type#

int

Return this queue pair’s transport type.

Type:: QP.qp_type

query(self)#: Return (attrs, cap) for the queue pair as plain dict + QPCap.

recv_cq#

send_cq#

srq#

state#

int

The queue pair’s current state (authoritative, via query).

Type:: QP.state

to_init(self, int port, int access=0, int pkey_index=0, int qkey=0)#

Transition RESET -> INIT.

port selects the local physical port. access is an ORed AccessFlags mask for RC/UC; pkey_index selects the partition key and qkey authorizes UD datagrams.

to_rtr(self, remote, int sgid_index, mtu=None, int hop_limit=1, int min_rnr_timer=12, int max_dest_rd_atomic=1, int sl=0, int traffic_class=0)#

Transition INIT -> RTR using remote path information.

remote is a QPInfo; sgid_index picks the local source GID. mtu defaults to the peer’s advertised MTU. hop_limit, sl, and traffic_class configure the route. min_rnr_timer controls RNR NAK delay, and max_dest_rd_atomic limits responder resources for incoming RDMA Read/atomic requests. UD ignores these path parameters and only changes state.

to_rts(self, int psn, int timeout=14, int retry_cnt=7, int rnr_retry=7, int max_rd_atomic=1)#

Transition RTR -> RTS.

psn sets the starting send packet sequence number. For RC, timeout is the local ACK timeout exponent, retry_cnt controls transport retries, rnr_retry controls receiver-not-ready retries (7 means infinite), and max_rd_atomic limits outstanding RDMA Read/atomic requests. UD uses only psn.

class ibverbs.SRQ#

Bases: object

A shared receive queue (ibv_srq).

close(self)#: Destroy this shared receive queue.

modify(self, max_wr=None, srq_limit=None)#: Update max_wr and/or the event threshold srq_limit.

pd#

post_recv(self, wrs)#: Post one or more receive work requests to the shared queue.

query(self)#: Return the shared receive queue’s current attributes.

class ibverbs.AHAttr(dgid=None, sgid_index=0, dlid=0, is_global=1, port_num=1, hop_limit=1, traffic_class=0, sl=0, flow_label=0, src_path_bits=0, static_rate=0)#

Bases: object

Address-handle attributes (also used to reach RTR for RC/UD).

Parameters:

dgid – Remote 16-byte GID. Defaults to all zeroes.
sgid_index – Local source-GID table index.
dlid – Remote InfiniBand LID; zero for a global RoCE route.
is_global – Whether to populate the global route header.
port_num – Local physical port number.
hop_limit – Global-route hop limit.
traffic_class – Global-route traffic class.
sl – InfiniBand service level.
flow_label – Global-route flow label.
src_path_bits – InfiniBand source path bits.
static_rate – InfiniBand static-rate selector.

dgid#

bytes

Type:: dgid

dlid#

‘uint16_t’

Type:: dlid

flow_label#

‘uint32_t’

Type:: flow_label

hop_limit#

‘uint8_t’

Type:: hop_limit

is_global#

‘uint8_t’

Type:: is_global

port_num#

‘uint8_t’

Type:: port_num

sgid_index#

‘uint8_t’

Type:: sgid_index

sl#

‘uint8_t’

Type:: sl

src_path_bits#

‘uint8_t’

Type:: src_path_bits

static_rate#

‘uint8_t’

Type:: static_rate

traffic_class#

‘uint8_t’

Type:: traffic_class

class ibverbs.AsyncEvent#

Bases: object

An asynchronous device event from Context.get_async_event().

event_type#

event_type_str#

str

Return the human-readable asynchronous event type.

Type:: AsyncEvent.event_type_str

class ibverbs.CompChannel#

Bases: object

A completion event channel (ibv_comp_channel).

close(self)#: Destroy the completion channel after its CQs are closed.

context#

fd#

int

Return the completion channel’s pollable file descriptor.

Type:: CompChannel.fd

get_cq_event(self) → 'CQ'#

Block until a CQ event arrives; return the associated CQ.

The event must later be acknowledged with CQ.ack_events().

Return type:: CQ

class ibverbs.Context#

Bases: object

An open device context (ibv_context).

ack_async_event(self, AsyncEvent ev)#: Acknowledge an event returned by get_async_event().

alloc_pd(self) → 'PD'#

Allocate a protection domain owned by this context.

Return type:: PD

async_fd#

int

Return the file descriptor used for asynchronous device events.

Type:: Context.async_fd

close(self)#: Close the device context after all child resources are closed.

create_comp_channel(self) → 'CompChannel'#

Create a completion event channel owned by this context.

Return type:: CompChannel

create_cq(self, int cqe, channel=None, int comp_vector=0) → 'CQ'#

Create a completion queue with space for at least cqe entries.

channel optionally enables event notification, and comp_vector selects its interrupt vector from 0 through num_comp_vectors - 1.

Return type:: CQ

get_async_event(self) → 'AsyncEvent'#

Block until an asynchronous device event is available.

Return type:: AsyncEvent

name#

num_comp_vectors#

int

Return the number of completion interrupt vectors.

Type:: Context.num_comp_vectors

query_device(self) → DeviceAttr#

Query generic verbs capabilities for this device.

Return type:: DeviceAttr

query_gid(self, int port_num, int index) → Gid#

Query GID table entry index on port_num.

Return type:: Gid

query_port(self, int port_num) → PortAttr#

Query attributes for port_num.

Return type:: PortAttr

class ibverbs.Device(str name, uint64_t guid)#

Bases: object

An RDMA device discovered by get_device_list().

guid#

name#

open(self) → 'Context'#

Open this device, returning a Context.

Return type:: Context

class ibverbs.DeviceAttr#

Bases: object

Device capabilities from Context.query_device().

fw_ver#

hw_ver#

max_ah#

max_cq#

max_cqe#

max_mr#

max_mr_size#

max_pd#

max_pkeys#

max_qp#

max_qp_rd_atom#

max_qp_wr#

max_sge#

max_srq#

max_srq_sge#

max_srq_wr#

node_guid#

phys_port_cnt#

sys_image_guid#

vendor_id#

vendor_part_id#

class ibverbs.Gid(bytes raw)#

Bases: object

A 16-byte GID (RoCE/InfiniBand address).

interface_id#

int

Return the least-significant 64 bits in network byte order.

Type:: Gid.interface_id

raw#

subnet_prefix#

int

Return the most-significant 64 bits in network byte order.

Type:: Gid.subnet_prefix

class ibverbs.PortAttr#

Bases: object

Port attributes from Context.query_port().

active_mtu#

active_speed#

active_width#

gid_tbl_len#

lid#

link_layer#

lmc#

max_msg_sz#

max_mtu#

pkey_tbl_len#

port_cap_flags#

sm_lid#

state#

class ibverbs.QPCap#

Bases: object

Queue-pair capacity limits.

max_inline_data#

max_recv_sge#

max_recv_wr#

max_send_sge#

max_send_wr#

class ibverbs.QPInitAttr(send_cq, recv_cq, qp_type=2, max_send_wr=128, max_recv_wr=128, max_send_sge=1, max_recv_sge=1, max_inline_data=0, srq=None, sq_sig_all=False)#

Bases: object

Parameters for PD.create_qp().

Parameters:

send_cq – Completion queue for send work requests.
recv_cq – Completion queue for receive work requests.
qp_type – A QPType; defaults to RC.
max_send_wr – Requested maximum outstanding send work requests.
max_recv_wr – Requested maximum posted receive work requests. Ignored when srq is set.
max_send_sge – Requested SGE limit for each send request.
max_recv_sge – Requested SGE limit for each receive request. Ignored when srq is set.
max_inline_data – Requested maximum inline payload in bytes.
srq – Optional SRQ used instead of a per-QP receive queue.
sq_sig_all – Make every send produce a completion, even without SIGNALED.

class ibverbs.RecvWR(wr_id=0, sg_list=None)#

Bases: object

A receive work request.

Parameters:

wr_id – Application-defined value returned in the completion.
sg_list – Local SGE objects that receive the payload.

sg_list#

list

Type:: sg_list

wr_id#

‘uint64_t’

Type:: wr_id

class ibverbs.SendWR(wr_id=0, sg_list=None, opcode=0, send_flags=0, remote_addr=0, rkey=0, imm_data=0, compare_add=0, swap=0, ah=None, remote_qpn=0, remote_qkey=0)#

Bases: object

A send work request.

Parameters:

wr_id – Application-defined value returned in the completion.
sg_list – Local SGE objects. Defaults to an empty list.
opcode – A WROpcode value.
send_flags – ORed SendFlags values.
remote_addr – Remote virtual address for RDMA and atomic operations.
rkey – Remote memory key for RDMA and atomic operations.
imm_data – Immediate data for an opcode ending in WITH_IMM.
compare_add – Compare operand for Compare and Swap, or add operand for Fetch and Add.
swap – Swap operand for Compare and Swap.
ah – Address handle for a UD send.
remote_qpn – Destination queue-pair number for a UD send.
remote_qkey – Destination qkey for a UD send.

ah#

object

Type:: ah

compare_add#

‘uint64_t’

Type:: compare_add

imm_data#

‘uint32_t’

Type:: imm_data

opcode#

‘int’

Type:: opcode

remote_addr#

‘uint64_t’

Type:: remote_addr

remote_qkey#

‘uint32_t’

Type:: remote_qkey

remote_qpn#

‘uint32_t’

Type:: remote_qpn

rkey#

‘uint32_t’

Type:: rkey

send_flags#

‘unsigned int’

Type:: send_flags

sg_list#

list

Type:: sg_list

swap#

‘uint64_t’

Type:: swap

wr_id#

‘uint64_t’

Type:: wr_id

class ibverbs.SGE(target, length, lkey=0, offset=0)#

Bases: object

A scatter/gather entry.

SGE(target, length, lkey=0, offset=0) where target is either an MR (lkey/addr derived from it) or an integer address.

addr#

length#

lkey#

owner#

exception ibverbs.VerbsError(operation, err, detail=None)#

Bases: OSError

Raised when a libibverbs call fails.

Subclasses OSError, so .errno and .strerror are populated. .operation names the verb that failed.

exception ibverbs.CompletionError(wc)#

Bases: VerbsError

Raised by WC.raise_for_status() for a failed completion.

class ibverbs.CMID#

Bases: object

A synchronously resolved RDMA-CM endpoint.

resolve() performs address and route resolution. The returned context is borrowed from RDMA-CM and must not be closed directly; allocate the PD and CQs from it, then use create_qp() so CM owns the QP state transitions and destruction.

close(self)#: Destroy the endpoint after its CM-managed resources are closed.

closed#

bool

Whether this RDMA-CM endpoint has been destroyed.

Type:: CMID.closed

connect(self, private_data=b'', *, responder_resources=1, initiator_depth=1, retry_count=7, rnr_retry_count=7) → bytes#

Connect the QP and return the peer’s RDMA-CM private data.

private_data may contain up to 255 application bytes. responder_resources advertises incoming RDMA Read/atomic capacity; initiator_depth requests that capacity from the peer. retry_count controls transport retries and rnr_retry_count controls receiver-not-ready retries. Each numeric option is uint8.

Return type:: bytes

connected#

bool

Whether the endpoint currently has an established connection.

Type:: CMID.connected

context#

create_qp(self, PD pd, init_attr) → QP#

Create a CM-managed RC QP using pd and init_attr.

Return type:: QP

disconnect(self)#: Synchronously disconnect an established endpoint.

host#

port#

classmethod resolve(cls, str host, port=4420, source=None)#

Resolve an RDMA-CM route and return an endpoint.

host and port select the destination. source optionally binds a local IP address, which also selects the local RDMA device.

source#

class ibverbs.WC#

Bases: object

A work completion returned by CQ.poll().

byte_len#

imm_data#

is_success#

bool

Whether the work request completed successfully.

Type:: WC.is_success

opcode#

qp_num#

raise_for_status(self)#: Raise CompletionError if this completion did not succeed.

src_qp#

status#

status_str#

str

Return the provider’s human-readable completion status.

Type:: WC.status_str

vendor_err#

wc_flags#

wr_id#

ibverbs.get_device_list() → list#

Return the list of Device objects present on this host.

Return type:: list

class ibverbs.AccessFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Memory-region / QP access permissions (IBV_ACCESS_*).

LOCAL_WRITE = 1#: Permit the local HCA to write the registered memory.

REMOTE_WRITE = 2#: Permit a remote peer to write the region; also requires LOCAL_WRITE.

REMOTE_READ = 4#: Permit a remote peer to read the region.

REMOTE_ATOMIC = 8#: Permit remote atomic operations; also requires LOCAL_WRITE.

MW_BIND = 16#: Permit binding this region into a memory window.

ZERO_BASED = 32#: Interpret SGE addresses as offsets from the MR base address.

ON_DEMAND = 64#: Register the region with on-demand paging.

HUGETLB = 128#: Assert that the on-demand-paging region is backed by huge pages.

FLUSH_GLOBAL = 256#: Permit remote flushes to global visibility.

FLUSH_PERSISTENT = 512#: Permit remote flushes to persistent storage.

RELAXED_ORDERING = 1048576#: Allow relaxed inbound write ordering for higher performance.

class ibverbs.MTU(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Path MTU (IBV_MTU_*).

MTU_256 = 1#

MTU_512 = 2#

MTU_1024 = 3#

MTU_2048 = 4#

MTU_4096 = 5#

class ibverbs.NodeType(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Device node type (IBV_NODE_*).

UNKNOWN = -1#

CA = 1#

SWITCH = 2#

ROUTER = 3#

RNIC = 4#

USNIC = 5#

USNIC_UDP = 6#

UNSPECIFIED = 7#

class ibverbs.PortState(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Port state (IBV_PORT_*).

NOP = 0#

DOWN = 1#

INIT = 2#

ARMED = 3#

ACTIVE = 4#

ACTIVE_DEFER = 5#

class ibverbs.QPAttrMask(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Bitmask selecting which fields modify_qp applies (IBV_QP_*).

STATE = 1#: Apply qp_state.

CUR_STATE = 2#: Apply cur_qp_state as a transition precondition.

EN_SQD_ASYNC_NOTIFY = 4#: Apply send-queue-drained asynchronous notification.

ACCESS_FLAGS = 8#: Apply QP access permissions.

PKEY_INDEX = 16#: Apply the partition-key table index.

PORT = 32#: Apply the local physical port.

QKEY = 64#: Apply the datagram queue key.

AV = 128#: Apply the address vector.

PATH_MTU = 256#: Apply the path MTU.

TIMEOUT = 512#: Apply the local ACK timeout exponent.

RETRY_CNT = 1024#: Apply the transport retry count.

RNR_RETRY = 2048#: Apply the receiver-not-ready retry count.

RQ_PSN = 4096#: Apply the receive packet sequence number.

MAX_QP_RD_ATOMIC = 8192#: Apply the initiator RDMA Read/atomic depth.

ALT_PATH = 16384#: Apply the alternate path.

MIN_RNR_TIMER = 32768#: Apply the minimum receiver-not-ready timer.

SQ_PSN = 65536#: Apply the send packet sequence number.

MAX_DEST_RD_ATOMIC = 131072#: Apply the responder RDMA Read/atomic depth.

PATH_MIG_STATE = 262144#: Apply the path-migration state.

CAP = 524288#: Apply queue-pair capacity limits.

DEST_QPN = 1048576#: Apply the destination queue-pair number.

RATE_LIMIT = 33554432#: Apply the packet pacing rate limit.

class ibverbs.QPState(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Queue-pair state (IBV_QPS_*).

RESET = 0#

INIT = 1#

RTR = 2#

RTS = 3#

SQD = 4#

SQE = 5#

ERR = 6#

UNKNOWN = 7#

class ibverbs.QPType(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Queue-pair transport service type (IBV_QPT_*).

RC = 2#

UC = 3#

UD = 4#

RAW_PACKET = 8#

XRC_SEND = 9#

XRC_RECV = 10#

DRIVER = 255#

class ibverbs.SendFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Send work-request flags (IBV_SEND_*).

FENCE = 1#: Fence this request behind earlier RDMA Read or atomic operations.

SIGNALED = 2#: Generate a send completion for this request.

SOLICITED = 4#: Request a solicited receive event at the peer.

INLINE = 8#: Copy the payload into the work request instead of retaining its SGEs.

IP_CSUM = 16#: Ask a raw-packet QP to calculate the IP checksum.

class ibverbs.WCFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Work-completion flags (IBV_WC_* bit flags).

GRH = 1#: A global route header precedes the received payload.

WITH_IMM = 2#: The completion contains valid immediate data.

IP_CSUM_OK = 4#: The received IP checksum was verified successfully.

WITH_INV = 8#: The completion contains a valid invalidated rkey.

TM_SYNC_REQ = 16#: A tag-matching synchronization operation completed.

TM_MATCH = 32#: A tag-matching receive matched.

TM_DATA_VALID = 64#: Tag-matching completion data is valid.

class ibverbs.WCOpcode(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Work-completion opcode (IBV_WC_*).

SEND = 0#

RDMA_WRITE = 1#

RDMA_READ = 2#

COMP_SWAP = 3#

FETCH_ADD = 4#

BIND_MW = 5#

LOCAL_INV = 6#

TSO = 7#

RECV = 128#

RECV_RDMA_WITH_IMM = 129#

class ibverbs.WCStatus(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Work-completion status (IBV_WC_*).

SUCCESS = 0#

LOC_LEN_ERR = 1#

LOC_QP_OP_ERR = 2#

LOC_EEC_OP_ERR = 3#

LOC_PROT_ERR = 4#

WR_FLUSH_ERR = 5#

MW_BIND_ERR = 6#

BAD_RESP_ERR = 7#

LOC_ACCESS_ERR = 8#

REM_INV_REQ_ERR = 9#

REM_ACCESS_ERR = 10#

REM_OP_ERR = 11#

RETRY_EXC_ERR = 12#

RNR_RETRY_EXC_ERR = 13#

LOC_RDD_VIOL_ERR = 14#

REM_INV_RD_REQ_ERR = 15#

REM_ABORT_ERR = 16#

INV_EECN_ERR = 17#

INV_EEC_STATE_ERR = 18#

FATAL_ERR = 19#

RESP_TIMEOUT_ERR = 20#

GENERAL_ERR = 21#

TM_ERR = 22#

TM_RNDV_INCOMPLETE = 23#

class ibverbs.WROpcode(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Send work-request opcode (IBV_WR_*).

RDMA_WRITE = 0#

RDMA_WRITE_WITH_IMM = 1#

SEND = 2#

SEND_WITH_IMM = 3#

RDMA_READ = 4#

ATOMIC_CMP_AND_SWP = 5#

ATOMIC_FETCH_AND_ADD = 6#

LOCAL_INV = 7#

BIND_MW = 8#

SEND_WITH_INV = 9#

TSO = 10#

class ibverbs.QPInfo(qp_num, psn, lid, gid, port, mtu)#

Bases: object

The minimum a peer needs to connect an RC/UD queue pair to this one.

Exchange this out-of-band (e.g. over a TCP socket) during connection setup. to_bytes() / from_bytes() give a fixed 28-byte wire layout so both ends agree regardless of platform endianness.

Parameters:

qp_num (int) –
psn (int) –
lid (int) –
gid (bytes) –
port (int) –
mtu (int) –

qp_num: int#

psn: int#

lid: int#

gid: bytes#

port: int#

mtu: int#

to_bytes()#

Serialize this endpoint to its fixed network format.

Return type:: bytes

classmethod from_bytes(data)#

Parse a fixed-size endpoint description.

Parameters:: data (bytes) –
Return type:: QPInfo

ibverbs.connect_rc(qp, remote, *, port, sgid_index, access, local_psn=0, mtu=None, **rts_kwargs)#

Drive an RC qp all the way from RESET to RTS against remote.

Equivalent to calling QP.to_init(), QP.to_rtr() and QP.to_rts() in sequence with sensible defaults. Unless explicitly overridden, the path MTU is negotiated as the smaller of the local and remote active MTUs.

Parameters:

qp – Local RC queue pair in RESET state.
remote (QPInfo) – Peer path and queue-pair information.
port (int) – Local physical port number.
sgid_index (int) – Local source-GID table index.
access (int) – ORed AccessFlags for remote access.
local_psn (int) – Local starting send packet sequence number.
mtu (int | None) – Explicit path MTU, or None to negotiate the smaller endpoint MTU.
**rts_kwargs – Overrides forwarded to QP.to_rts(), such as timeout, retry_cnt, rnr_retry, or max_rd_atomic.

Return type:

None

ibverbs.local_qp_info(qp, port_attr, gid, *, port, psn=0)#

Build a QPInfo describing qp for sending to a peer.

port_attr comes from Context.query_port(), gid from Context.query_gid() (its .raw bytes are used). port records the local physical port and psn is the starting send packet sequence number that the peer must use as its receive PSN.

Parameters:

port (int) –
psn (int) –

Return type:

QPInfo

ibverbs.reg_tensor(pd, tensor, access)#

Convenience for a contiguous torch CPU tensor or numpy array. For CUDA tensors use ibverbs.cuda.register_tensor() (dma-buf/GPUDirect) instead. pd is the owning protection domain and access is an ORed AccessFlags mask.

ibverbs.tensor_addr_len(tensor)#

Return (addr, nbytes) for a tensor-like object (torch or numpy).

Works with a torch tensor (data_ptr() + numel() × element_size()) or a numpy array (ctypes.data + nbytes). No import of either library is required — the object is duck-typed.

Helpers#

Thin, optional helpers for the common RC connection dance.

These build on the raw verbs in ibverbs._ibverbs. The raw QP.modify / QP.to_init / QP.to_rtr / QP.to_rts remain available for callers that want full control.

class ibverbs.helpers.QPInfo(qp_num, psn, lid, gid, port, mtu)#

Bases: object

The minimum a peer needs to connect an RC/UD queue pair to this one.

Exchange this out-of-band (e.g. over a TCP socket) during connection setup. to_bytes() / from_bytes() give a fixed 28-byte wire layout so both ends agree regardless of platform endianness.

Parameters:

qp_num (int) –
psn (int) –
lid (int) –
gid (bytes) –
port (int) –
mtu (int) –

qp_num: int#

psn: int#

lid: int#

gid: bytes#

port: int#

mtu: int#

to_bytes()#

Serialize this endpoint to its fixed network format.

Return type:: bytes

classmethod from_bytes(data)#

Parse a fixed-size endpoint description.

Parameters:: data (bytes) –
Return type:: QPInfo

ibverbs.helpers.tensor_addr_len(tensor)#

Return (addr, nbytes) for a tensor-like object (torch or numpy).

Works with a torch tensor (data_ptr() + numel() × element_size()) or a numpy array (ctypes.data + nbytes). No import of either library is required — the object is duck-typed.

ibverbs.helpers.reg_tensor(pd, tensor, access)#

ibverbs.helpers.local_qp_info(qp, port_attr, gid, *, port, psn=0)#

Build a QPInfo describing qp for sending to a peer.

Parameters:

port (int) –
psn (int) –

Return type:

QPInfo

ibverbs.helpers.connect_rc(qp, remote, *, port, sgid_index, access, local_psn=0, mtu=None, **rts_kwargs)#

Drive an RC qp all the way from RESET to RTS against remote.

Parameters:

qp – Local RC queue pair in RESET state.
remote (QPInfo) – Peer path and queue-pair information.
port (int) – Local physical port number.
sgid_index (int) – Local source-GID table index.
access (int) – ORed AccessFlags for remote access.
local_psn (int) – Local starting send packet sequence number.
mtu (int | None) – Explicit path MTU, or None to negotiate the smaller endpoint MTU.
**rts_kwargs – Overrides forwarded to QP.to_rts(), such as timeout, retry_cnt, rnr_retry, or max_rd_atomic.

Return type:

None

CUDA#

Optional CUDA GPUDirect helpers for registering GPU tensors as memory regions.

This module lazily dlopen``s ``libcuda and imports no torch and links no CUDA at build time, so it adds no required dependency. It is duck-typed: anything exposing data_ptr() (e.g. a torch CUDA tensor) works.

For torch, allocate with the VMM allocator so the memory can be exported as a dma-buf fd (the fd libibverbs needs for ibv_reg_dmabuf_mr):

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Typical use:

import ibverbs, ibverbs.cuda, torch
t = torch.zeros(1 << 20, dtype=torch.uint8, device="cuda")
gmr = ibverbs.cuda.register_tensor(pd, t, ibverbs.AccessFlags.LOCAL_WRITE)
qp.post_send(ibverbs.SendWR(sg_list=[gmr.sge()], ...))

Before the NIC reads a tensor, synchronize the CUDA stream that produced it. After an inbound completion, call flush_gpudirect_writes() before CUDA work consumes the destination.

class ibverbs.cuda.GpuMR(mr, addr, length, tensor=None)#

Bases: object

A registered GPU memory region plus the info needed to address it.

ibv_mr.addr is not meaningful for dma-buf MRs, so this wraps the real device virtual address. Use sge() for the local scatter/gather list and addr / rkey for the remote side of an RDMA op.

Parameters:

addr (int) –
length (int) –

property tensor#: The tensor retained as this registration’s allocation owner.

property closed: bool#: Whether the underlying memory region has been deregistered.

property lkey: int#: Return the local key used in scatter/gather entries.

property rkey: int#: Return the remote key shared with an RDMA peer.

sge(length=None, offset=0)#: Build an SGE for a byte range of this GPU memory region.

close()#: Deregister the memory region and release the retained tensor.

ibverbs.cuda.dmabuf_fd(ptr, length)#

Export a dma-buf fd for the device VA range [ptr, ptr+length).

ptr and length must be page-aligned (register_tensor() handles alignment). The caller owns and must os.close the returned fd.

Parameters:

ptr (int) –
length (int) –

Return type:

int

ibverbs.cuda.flush_gpudirect_writes(*, all_devices=False)#

Make completed inbound GPUDirect RDMA writes visible to CUDA.

Call this after the relevant receive/completion or application-level remote-write notification, before launching CUDA work that consumes the destination tensor. The current thread must have the destination tensor’s CUDA context active. By default CUDA flushes writes only to the owning device; set all_devices=True to flush visibility to every GPU in the current context’s scope.

Parameters:: all_devices (bool) –
Return type:: None

ibverbs.cuda.synchronize()#

Wait for CUDA work in the current context before an RDMA device read.

This is the framework-independent ordering primitive for outbound GPUDirect buffers. The caller must make the buffer’s CUDA context current.

Return type:: None

ibverbs.cuda.tensor_ptr_len(tensor)#: Return (device_addr, nbytes) for a tensor-like object.

ibverbs.cuda.register_tensor(pd, tensor, access)#

Prefers the dma-buf path (no kernel module needed) and transparently handles page alignment. Falls back to the raw reg_mr (nvidia_peermem) path when dma-buf export or registration is unavailable. The returned object retains tensor so its allocation cannot be recycled while the MR is registered.

pd is the owning protection domain and access is an ORed AccessFlags mask.

Return type:: GpuMR

Enums#

Pythonic enums mirroring the libibverbs ABI constants.

These values are part of the stable rdma-core ABI. They are declared here as enum.IntEnum / enum.IntFlag so they interoperate with plain ints everywhere the low-level API accepts a flag or code.

class ibverbs.enums.AccessFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Memory-region / QP access permissions (IBV_ACCESS_*).

LOCAL_WRITE = 1#: Permit the local HCA to write the registered memory.

REMOTE_WRITE = 2#: Permit a remote peer to write the region; also requires LOCAL_WRITE.

REMOTE_READ = 4#: Permit a remote peer to read the region.

REMOTE_ATOMIC = 8#: Permit remote atomic operations; also requires LOCAL_WRITE.

MW_BIND = 16#: Permit binding this region into a memory window.

ZERO_BASED = 32#: Interpret SGE addresses as offsets from the MR base address.

ON_DEMAND = 64#: Register the region with on-demand paging.

HUGETLB = 128#: Assert that the on-demand-paging region is backed by huge pages.

FLUSH_GLOBAL = 256#: Permit remote flushes to global visibility.

FLUSH_PERSISTENT = 512#: Permit remote flushes to persistent storage.

RELAXED_ORDERING = 1048576#: Allow relaxed inbound write ordering for higher performance.

class ibverbs.enums.QPType(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Queue-pair transport service type (IBV_QPT_*).

RC = 2#

UC = 3#

UD = 4#

RAW_PACKET = 8#

XRC_SEND = 9#

XRC_RECV = 10#

DRIVER = 255#

class ibverbs.enums.QPState(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Queue-pair state (IBV_QPS_*).

RESET = 0#

INIT = 1#

RTR = 2#

RTS = 3#

SQD = 4#

SQE = 5#

ERR = 6#

UNKNOWN = 7#

class ibverbs.enums.QPAttrMask(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Bitmask selecting which fields modify_qp applies (IBV_QP_*).

STATE = 1#: Apply qp_state.

CUR_STATE = 2#: Apply cur_qp_state as a transition precondition.

EN_SQD_ASYNC_NOTIFY = 4#: Apply send-queue-drained asynchronous notification.

ACCESS_FLAGS = 8#: Apply QP access permissions.

PKEY_INDEX = 16#: Apply the partition-key table index.

PORT = 32#: Apply the local physical port.

QKEY = 64#: Apply the datagram queue key.

AV = 128#: Apply the address vector.

PATH_MTU = 256#: Apply the path MTU.

TIMEOUT = 512#: Apply the local ACK timeout exponent.

RETRY_CNT = 1024#: Apply the transport retry count.

RNR_RETRY = 2048#: Apply the receiver-not-ready retry count.

RQ_PSN = 4096#: Apply the receive packet sequence number.

MAX_QP_RD_ATOMIC = 8192#: Apply the initiator RDMA Read/atomic depth.

ALT_PATH = 16384#: Apply the alternate path.

MIN_RNR_TIMER = 32768#: Apply the minimum receiver-not-ready timer.

SQ_PSN = 65536#: Apply the send packet sequence number.

MAX_DEST_RD_ATOMIC = 131072#: Apply the responder RDMA Read/atomic depth.

PATH_MIG_STATE = 262144#: Apply the path-migration state.

CAP = 524288#: Apply queue-pair capacity limits.

DEST_QPN = 1048576#: Apply the destination queue-pair number.

RATE_LIMIT = 33554432#: Apply the packet pacing rate limit.

class ibverbs.enums.WROpcode(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Send work-request opcode (IBV_WR_*).

RDMA_WRITE = 0#

RDMA_WRITE_WITH_IMM = 1#

SEND = 2#

SEND_WITH_IMM = 3#

RDMA_READ = 4#

ATOMIC_CMP_AND_SWP = 5#

ATOMIC_FETCH_AND_ADD = 6#

LOCAL_INV = 7#

BIND_MW = 8#

SEND_WITH_INV = 9#

TSO = 10#

class ibverbs.enums.SendFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Send work-request flags (IBV_SEND_*).

FENCE = 1#: Fence this request behind earlier RDMA Read or atomic operations.

SIGNALED = 2#: Generate a send completion for this request.

SOLICITED = 4#: Request a solicited receive event at the peer.

INLINE = 8#: Copy the payload into the work request instead of retaining its SGEs.

IP_CSUM = 16#: Ask a raw-packet QP to calculate the IP checksum.

class ibverbs.enums.WCStatus(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Work-completion status (IBV_WC_*).

SUCCESS = 0#

LOC_LEN_ERR = 1#

LOC_QP_OP_ERR = 2#

LOC_EEC_OP_ERR = 3#

LOC_PROT_ERR = 4#

WR_FLUSH_ERR = 5#

MW_BIND_ERR = 6#

BAD_RESP_ERR = 7#

LOC_ACCESS_ERR = 8#

REM_INV_REQ_ERR = 9#

REM_ACCESS_ERR = 10#

REM_OP_ERR = 11#

RETRY_EXC_ERR = 12#

RNR_RETRY_EXC_ERR = 13#

LOC_RDD_VIOL_ERR = 14#

REM_INV_RD_REQ_ERR = 15#

REM_ABORT_ERR = 16#

INV_EECN_ERR = 17#

INV_EEC_STATE_ERR = 18#

FATAL_ERR = 19#

RESP_TIMEOUT_ERR = 20#

GENERAL_ERR = 21#

TM_ERR = 22#

TM_RNDV_INCOMPLETE = 23#

class ibverbs.enums.WCOpcode(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Work-completion opcode (IBV_WC_*).

SEND = 0#

RDMA_WRITE = 1#

RDMA_READ = 2#

COMP_SWAP = 3#

FETCH_ADD = 4#

BIND_MW = 5#

LOCAL_INV = 6#

TSO = 7#

RECV = 128#

RECV_RDMA_WITH_IMM = 129#

class ibverbs.enums.WCFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Work-completion flags (IBV_WC_* bit flags).

GRH = 1#: A global route header precedes the received payload.

WITH_IMM = 2#: The completion contains valid immediate data.

IP_CSUM_OK = 4#: The received IP checksum was verified successfully.

WITH_INV = 8#: The completion contains a valid invalidated rkey.

TM_SYNC_REQ = 16#: A tag-matching synchronization operation completed.

TM_MATCH = 32#: A tag-matching receive matched.

TM_DATA_VALID = 64#: Tag-matching completion data is valid.

class ibverbs.enums.MTU(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Path MTU (IBV_MTU_*).

MTU_256 = 1#

MTU_512 = 2#

MTU_1024 = 3#

MTU_2048 = 4#

MTU_4096 = 5#

class ibverbs.enums.PortState(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Port state (IBV_PORT_*).

NOP = 0#

DOWN = 1#

INIT = 2#

ARMED = 3#

ACTIVE = 4#

ACTIVE_DEFER = 5#

class ibverbs.enums.NodeType(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Device node type (IBV_NODE_*).

UNKNOWN = -1#

CA = 1#

SWITCH = 2#

ROUTER = 3#

RNIC = 4#

USNIC = 5#

USNIC_UDP = 6#

UNSPECIFIED = 7#