EFA API¶

Core API¶

efa: low-level Pythonic bindings for AWS EFA (SRD RDMA).

Quickstart (SRD is connectionless – no per-QP handshake, just exchange EndpointInfo once):

import efa

dev = efa.get_efa_device_list()[0]
ctx = dev.open()
pd = ctx.alloc_pd()
cq = ctx.create_cq(256)
qp = pd.create_qp(efa.QPInitAttr(send_cq=cq, recv_cq=cq)).prepare(qkey=0x1234)

info = efa.local_endpoint_info(qp, qkey=0x1234)
# ... exchange info.to_bytes() with the peer out-of-band ...
peer = efa.EndpointInfo.from_bytes(remote_bytes).peer(pd)

mr = efa.reg_tensor(pd, tensor, efa.AccessFlags.LOCAL_WRITE)
qp.post_send(efa.SendWR(sg_list=[mr.sge()], opcode=efa.WROpcode.SEND,
                        send_flags=efa.SendFlags.SIGNALED, dest=peer))

The compiled fast paths live in efa._efa; enums in efa.enums; address-exchange and tensor helpers in efa.helpers.

class efa.AH¶

Bases: object

An address handle naming a remote EFA device (by GID).

ahn¶

int

The EFA address handle number (efadv_query_ah).

Type:: AH.ahn

close(self)¶

pd¶

class efa.CQ¶

Bases: _CQBase

A classic completion queue (ibv_cq).

close(self)¶

poll(self, int num_entries) → list¶

Poll up to num_entries completions; return a list of WC.

Return type:: list

query_efa(self) → CQAttr¶

Return the provider’s direct CQ layout (EFA 1.4 or newer).

Return type:: CQAttr

class efa.CQAttr¶

Bases: object

An EFA completion queue’s direct layout from efadv_query_cq.

buffer_addr¶

doorbell_addr¶

entry_size¶

num_entries¶

class efa.CQEx¶

Bases: _CQBase

An extended EFA completion queue (efadv_create_cq).

Functionally a CQ, plus optional sender-GID reporting (sgid=True): SRD/UD receivers can identify previously unknown peers straight from the completion (see WC.sgid).

close(self)¶

poll(self, int num_entries) → list¶

Poll up to num_entries completions; return a list of WC.

Return type:: list

query_efa(self) → CQAttr¶

Return the provider’s direct CQ layout (EFA 1.4 or newer).

Return type:: CQAttr

sgid_enabled¶

bool

Type:: CQEx.sgid_enabled

unsolicited_enabled¶

bool

Type:: CQEx.unsolicited_enabled

class efa.MR¶

Bases: object

A registered memory region (ibv_mr).

addr¶

int

Type:: MR.addr

close(self)¶

closed¶

bool

Type:: MR.closed

handle¶

int

Type:: MR.handle

length¶

int

Type:: MR.length

lkey¶

int

Type:: MR.lkey

owner¶

pd¶

query_efa(self) → MRAttr¶

EFA-specific MR attributes (efadv_query_mr, rdma-core >= 44).

Return type:: MRAttr

rkey¶

int

Type:: MR.rkey

sge(self, length=None, offset=0)¶: Return an SGE for a bounded range of this MR.

class efa.PD¶

Bases: object

A protection domain (ibv_pd).

close(self)¶

context¶

create_ah(self, addr, int sgid_index=0, int port_num=1) → 'AH'¶

Create an address handle for a remote EFA device.

addr may be an AHAttr, a Gid, or 16 raw GID bytes.

Return type:: AH

create_qp(self, init_attr) → 'QP'¶

Create an SRD (default) or UD queue pair.

SRD QPs go through efadv_create_qp_ex; UD QPs through ibv_create_qp_ex. Both are created with the extended send-ops work-request API enabled (see QPInitAttr).

Return type:: QP

reg_dmabuf_mr(self, offset, length, iova, int fd, int access) → 'MR'¶

Return type:: MR

reg_mr(self, addr, length, int access) → 'MR'¶

addr is an integer virtual address; it may be a host pointer or a CUDA device pointer (GPUDirect via nvidia_peermem).

Return type:: MR

class efa.QP¶

Bases: object

An EFA queue pair (SRD or UD).

Sends go through the extended work-request API under the hood, so every supported opcode, including RDMA read/write on SRD, which the classic ibv_post_send cannot express on this provider, posts uniformly via post_send().

close(self)¶

modify(self, int attr_mask, **fields)¶

Low-level ibv_modify_qp.

attr_mask is an ORed QPAttrMask. Scalar attributes are passed as keywords (e.g. qp_state=, qkey=).

pd¶

post_recv(self, wrs)¶: Post one or more RecvWR to the receive queue.

post_send(self, wrs)¶

Post one or more SendWR as a single batch.

All work requests are queued between wr_start/wr_complete, so either the whole batch is submitted or none of it is.

EFA requires every send WR to carry SIGNALED (or the QP must be created with sq_sig_all=True); an unsignaled WR fails the batch with EINVAL.

prepare(self, qkey, int port=1, psn=0, rnr_retry=None)¶

Drive RESET -> INIT -> RTR -> RTS in one call.

SRD/UD are connectionless, so a QP is fully usable after this without any remote information; destinations are named per-send.

qp_num¶

int

Type:: QP.qp_num

qp_type¶

query(self)¶: Return (attrs, cap) for the queue pair as plain dict + QPCap.

query_wqs(self)¶: Return (sq, rq) WQAttr (efadv_query_qp_wqs, EFA >= 1.4).

recv_cq¶

send_cq¶

send_ops_flags¶

state¶

int

The queue pair’s current state (authoritative, via query).

Type:: QP.state

to_init(self, qkey, int port=1, int pkey_index=0)¶: Transition RESET -> INIT, binding the qkey.

to_rtr(self)¶: Transition INIT -> RTR (connectionless: no remote info needed).

to_rts(self, psn=0, rnr_retry=None)¶

Transition RTR -> RTS.

rnr_retry (0-7, 7 = infinite) is EFA’s receiver-not-ready retry count; needs RNR_RETRY.

class efa.AHAttr(dgid=None, sgid_index=0, port_num=1, hop_limit=1, traffic_class=0, flow_label=0, is_global=1)¶

Bases: object

Address-handle attributes.

EFA addresses are GIDs: only dgid (the remote device’s GID), sgid_index and port_num matter; the rest exists for parity with generic verbs.

dgid¶

bytes

Type:: dgid

flow_label¶

‘uint32_t’

Type:: flow_label

hop_limit¶

‘uint8_t’

Type:: hop_limit

is_global¶

‘uint8_t’

Type:: is_global

port_num¶

‘uint8_t’

Type:: port_num

sgid_index¶

‘uint8_t’

Type:: sgid_index

traffic_class¶

‘uint8_t’

Type:: traffic_class

class efa.AsyncEvent¶

Bases: object

An asynchronous device event from Context.get_async_event().

event_type¶

event_type_str¶

str

Type:: AsyncEvent.event_type_str

class efa.CompChannel¶

Bases: object

A completion event channel (ibv_comp_channel).

close(self)¶

context¶

fd¶

int

Type:: CompChannel.fd

get_cq_event(self)¶

Block until a CQ event arrives; return the associated CQ object.

The event must later be acknowledged with ack_events() on the returned CQ / CQEx.

exception efa.CompletionError(wc)¶

Bases: EfaError

Raised by WC.raise_for_status() for a failed completion.

class efa.Context¶

Bases: object

An open device context (ibv_context).

ack_async_event(self, AsyncEvent ev)¶

alloc_pd(self) → 'PD'¶

Return type:: PD

async_fd¶

int

Type:: Context.async_fd

close(self)¶

create_comp_channel(self) → 'CompChannel'¶

Return type:: CompChannel

create_cq(self, int cqe, channel=None, int comp_vector=0) → 'CQ'¶

Create a classic completion queue (ibv_create_cq).

Return type:: CQ

create_cq_ex(self, int cqe, channel=None, int comp_vector=0, wc_flags=None, bool sgid=False, bool unsolicited=False) → 'CQEx'¶

Create an extended EFA completion queue (efadv_create_cq).

sgid=True requests sender-GID reporting on receive completions (needs CQ_WITH_SGID); the GID shows up as WC.sgid for peers not matched by a local AH.

unsolicited=True requests an indicator for unsolicited write-with-immediate receive completions (needs UNSOLICITED_WRITE_RECV).

Return type:: CQEx

get_async_event(self) → 'AsyncEvent'¶

Return type:: AsyncEvent

is_efa(self) → bool¶

Return True if this device is driven by the EFA provider.

Return type:: bool

name¶

num_comp_vectors¶

int

Type:: Context.num_comp_vectors

query_device(self) → DeviceAttr¶

Return type:: DeviceAttr

query_efa_device(self) → EfaDeviceAttr¶

EFA capabilities (efadv_query_device).

Raises EfaError if the device is not an EFA device.

Return type:: EfaDeviceAttr

query_gid(self, int port_num=1, int index=0) → Gid¶

The device GID, which is the EFA device’s network address.

Return type:: Gid

query_port(self, int port_num=1) → PortAttr¶

Return type:: PortAttr

class efa.Device(str name, uint64_t guid)¶

Bases: object

An RDMA device discovered by get_device_list().

guid¶

name¶

open(self) → 'Context'¶

Open this device, returning a Context.

Return type:: Context

class efa.DeviceAttr¶

Bases: object

Generic verbs device capabilities from Context.query_device().

fw_ver¶

hw_ver¶

max_ah¶

max_cq¶

max_cqe¶

max_mr¶

max_mr_size¶

max_pd¶

max_pkeys¶

max_qp¶

max_qp_wr¶

max_sge¶

node_guid¶

phys_port_cnt¶

sys_image_guid¶

vendor_id¶

vendor_part_id¶

class efa.EfaDeviceAttr¶

Bases: object

EFA-specific capabilities from Context.query_efa_device().

device_caps¶

inline_buf_size¶

max_rdma_size¶

max_rq_sge¶

max_rq_wr¶

max_sq_sge¶

max_sq_wr¶

exception efa.EfaError(operation, err, detail=None)¶

Bases: OSError

Raised when a libibverbs/libefa call fails.

Subclasses OSError, so .errno and .strerror are populated. .operation names the call that failed.

class efa.Gid(bytes raw)¶

Bases: object

A 16-byte GID (the EFA device address).

interface_id¶

int

Type:: Gid.interface_id

raw¶

subnet_prefix¶

int

Type:: Gid.subnet_prefix

class efa.MRAttr¶

Bases: object

EFA MR attributes from MR.query_efa() (efadv_query_mr).

ic_id_validity¶

rdma_read_ic_id¶

rdma_recv_ic_id¶

recv_ic_id¶

class efa.PortAttr¶

Bases: object

Port attributes from Context.query_port().

max_msg_sz is the SEND size limit; RDMA read/write are limited by EfaDeviceAttr.max_rdma_size instead.

active_mtu¶

active_speed¶

active_width¶

gid_tbl_len¶

lid¶

link_layer¶

max_msg_sz¶

max_mtu¶

pkey_tbl_len¶

port_cap_flags¶

state¶

class efa.QPCap¶

Bases: object

Queue-pair capacity limits.

max_inline_data¶

max_recv_sge¶

max_recv_wr¶

max_send_sge¶

max_send_wr¶

class efa.QPInitAttr(send_cq, recv_cq, qp_type=None, max_send_wr=128, max_recv_wr=128, max_send_sge=1, max_recv_sge=1, max_inline_data=0, sq_sig_all=False, send_ops_flags=None, sl=0, unsolicited_write_recv=False)¶

Bases: object

Parameters for PD.create_qp().

qp_type is SRD (default) or UD. send_ops_flags defaults to every operation the QP type supports (SEND/SEND_WITH_IMM, plus the RDMA ops for SRD); pass an explicit QPExSendOpsFlags mask to restrict it; the provider rejects masks including ops the device cannot do, so pass e.g. SEND | SEND_WITH_IMM on RDMA-less instance types. unsolicited_write_recv opts in to RDMA-write-with-imm completions that consume no posted recv (check UNSOLICITED_WRITE_RECV).

class efa.RecvWR(wr_id=0, sg_list=None)¶

Bases: object

A receive work request.

sg_list¶

list

Type:: sg_list

wr_id¶

‘uint64_t’

Type:: wr_id

class efa.SendWR(wr_id=0, sg_list=None, opcode=2, send_flags=0, remote_addr=0, rkey=0, imm_data=0, ah=None, remote_qpn=0, remote_qkey=0, dest=None)¶

Bases: object

A send work request.

SRD/UD is connectionless: every send names its destination. Set either dest= (anything with .ah, .qp_num and .qkey attributes, e.g. efa.helpers.Peer) or the explicit ah/remote_qpn/ remote_qkey trio.

Opcodes: SEND, SEND_WITH_IMM, RDMA_READ, RDMA_WRITE, RDMA_WRITE_WITH_IMM (RDMA opcodes additionally need remote_addr/rkey).

ah¶

object

Type:: ah

imm_data¶

‘uint32_t’

Type:: imm_data

opcode¶

‘int’

Type:: opcode

remote_addr¶

‘uint64_t’

Type:: remote_addr

remote_qkey¶

‘uint32_t’

Type:: remote_qkey

remote_qpn¶

‘uint32_t’

Type:: remote_qpn

rkey¶

‘uint32_t’

Type:: rkey

send_flags¶

‘unsigned int’

Type:: send_flags

sg_list¶

list

Type:: sg_list

wr_id¶

‘uint64_t’

Type:: wr_id

class efa.SGE(target, length, lkey=0, offset=0)¶

Bases: object

A scatter/gather entry.

SGE(target, length, lkey=0, offset=0) where target is either an MR (lkey/addr derived from it) or an integer address.

addr¶

length¶

lkey¶

owner¶

class efa.WC¶

Bases: object

A work completion returned by CQ.poll() / CQEx.poll().

For completions from a CQEx created with sgid=True, sgid carries the sender’s GID on receive completions from peers not covered by a local AH (else None).

byte_len¶

imm_data¶

is_success¶

bool

Type:: WC.is_success

opcode¶

qp_num¶

raise_for_status(self)¶: Raise CompletionError if this completion did not succeed.

sgid¶

src_qp¶

status¶

status_str¶

str

Type:: WC.status_str

unsolicited¶

vendor_err¶

wc_flags¶

wr_id¶

class efa.WQAttr¶

Bases: object

A work queue’s layout from QP.query_wqs() (efadv_query_qp_wqs).

buffer_addr¶

doorbell_addr¶

entry_size¶

max_batch¶

num_entries¶

efa.get_device_list() → list¶

Return every RDMA Device on this host (EFA or not).

Use get_efa_device_list() to keep only EFA devices.

Return type:: list

efa.get_efa_device_list() → list¶

Return the Device list filtered to EFA devices.

Each device is briefly opened and probed with efadv_query_device.

Return type:: list

class efa.AccessFlags(*values)¶

Bases: IntFlag

Memory-region access permissions (IBV_ACCESS_*).

LOCAL_WRITE = 1¶

REMOTE_WRITE = 2¶

REMOTE_READ = 4¶

RELAXED_ORDERING = 1048576¶

class efa.CreateCQWCFlags(*values)¶

Bases: IntFlag

Extended-CQ per-completion fields (IBV_WC_EX_WITH_*).

STANDARD matches what classic ibv_poll_cq reports and is the default for create_cq_ex().

BYTE_LEN = 1¶

IMM = 2¶

QP_NUM = 4¶

SRC_QP = 8¶

SLID = 16¶

SL = 32¶

DLID_PATH_BITS = 64¶

STANDARD = 127¶

class efa.EfaDeviceCaps(*values)¶

Bases: IntFlag

EFA device capability bits (EFADV_DEVICE_ATTR_CAPS_*).

Test against device_caps.

RDMA_READ = 1¶

RNR_RETRY = 2¶

CQ_WITH_SGID = 4¶

RDMA_WRITE = 8¶

UNSOLICITED_WRITE_RECV = 16¶

CQ_WITH_EXT_MEM_DMABUF = 32¶

class efa.MTU(*values)¶

Bases: IntEnum

Path MTU (IBV_MTU_*).

MTU_256 = 1¶

MTU_512 = 2¶

MTU_1024 = 3¶

MTU_2048 = 4¶

MTU_4096 = 5¶

class efa.PortState(*values)¶

Bases: IntEnum

Port state (IBV_PORT_*).

NOP = 0¶

DOWN = 1¶

INIT = 2¶

ARMED = 3¶

ACTIVE = 4¶

ACTIVE_DEFER = 5¶

class efa.QPAttrMask(*values)¶

Bases: IntFlag

Bitmask selecting which fields modify_qp applies (IBV_QP_*).

STATE = 1¶

CUR_STATE = 2¶

PKEY_INDEX = 16¶

PORT = 32¶

QKEY = 64¶

RNR_RETRY = 2048¶

RQ_PSN = 4096¶

SQ_PSN = 65536¶

class efa.QPExSendOpsFlags(*values)¶

Bases: IntFlag

Extended-QP send operations (IBV_QP_EX_WITH_*).

Passed as QPInitAttr.send_ops_flags; the created QP can only post the opcodes enabled here.

RDMA_WRITE = 1¶

RDMA_WRITE_WITH_IMM = 2¶

SEND = 4¶

SEND_WITH_IMM = 8¶

RDMA_READ = 16¶

class efa.QPState(*values)¶

Bases: IntEnum

Queue-pair state (IBV_QPS_*).

RESET = 0¶

INIT = 1¶

RTR = 2¶

RTS = 3¶

SQD = 4¶

SQE = 5¶

ERR = 6¶

UNKNOWN = 7¶

class efa.QPType(*values)¶

Bases: IntEnum

EFA queue-pair type.

SRD (Scalable Reliable Datagram – reliable, connectionless, out-of-order; EFA’s native transport) or UD (unreliable datagram). SRD maps onto IBV_QPT_DRIVER and is created through the EFA provider.

UD = 4¶

SRD = 255¶

class efa.SendFlags(*values)¶

Bases: IntFlag

Send work-request flags (IBV_SEND_*).

FENCE = 1¶

SIGNALED = 2¶

SOLICITED = 4¶

INLINE = 8¶

class efa.WCFlags(*values)¶

Bases: IntFlag

Work-completion flags (IBV_WC_* bit flags).

GRH = 1¶

WITH_IMM = 2¶

class efa.WCOpcode(*values)¶

Bases: IntEnum

Work-completion opcode (IBV_WC_*).

SEND = 0¶

RDMA_WRITE = 1¶

RDMA_READ = 2¶

RECV = 128¶

RECV_RDMA_WITH_IMM = 129¶

class efa.WCStatus(*values)¶

Bases: IntEnum

Work-completion status (IBV_WC_*).

SUCCESS = 0¶

LOC_LEN_ERR = 1¶

LOC_QP_OP_ERR = 2¶

LOC_EEC_OP_ERR = 3¶

LOC_PROT_ERR = 4¶

WR_FLUSH_ERR = 5¶

MW_BIND_ERR = 6¶

BAD_RESP_ERR = 7¶

LOC_ACCESS_ERR = 8¶

REM_INV_REQ_ERR = 9¶

REM_ACCESS_ERR = 10¶

REM_OP_ERR = 11¶

RETRY_EXC_ERR = 12¶

RNR_RETRY_EXC_ERR = 13¶

LOC_RDD_VIOL_ERR = 14¶

REM_INV_RD_REQ_ERR = 15¶

REM_ABORT_ERR = 16¶

INV_EECN_ERR = 17¶

INV_EEC_STATE_ERR = 18¶

FATAL_ERR = 19¶

RESP_TIMEOUT_ERR = 20¶

GENERAL_ERR = 21¶

class efa.WROpcode(*values)¶

Bases: IntEnum

Send work-request opcode (IBV_WR_*, the subset EFA supports).

RDMA_WRITE = 0¶

RDMA_WRITE_WITH_IMM = 1¶

SEND = 2¶

SEND_WITH_IMM = 3¶

RDMA_READ = 4¶

class efa.EndpointInfo(gid, qp_num, qkey)¶

Bases: object

The minimum a peer needs to send to an SRD/UD queue pair.

Exchange this out-of-band (e.g. over a TCP socket or a distributed key-value store) during setup. to_bytes() / from_bytes() give a fixed 24-byte wire layout so both ends agree regardless of platform endianness.

Parameters:

gid (bytes)
qp_num (int)
qkey (int)

gid: bytes¶

qp_num: int¶

qkey: int¶

to_bytes()¶

Return type:: bytes

classmethod from_bytes(data)¶

Parameters:: data (bytes)
Return type:: EndpointInfo

peer(pd, sgid_index=0, port=1)¶

Create the Peer (AH + addressing) for this endpoint.

Parameters:

sgid_index (int)
port (int)

Return type:

Peer

class efa.Peer(ah, qp_num, qkey)¶

Bases: object

A resolved remote endpoint: an AH plus QP number and qkey.

Pass as dest= to SendWR. Owns the AH; close() (or use as a context manager) destroys it.

Parameters:

qp_num (int)
qkey (int)

close()¶

Return type:: None

efa.local_endpoint_info(qp, qkey, *, gid=None, gid_index=0, port=1)¶

Build an EndpointInfo describing qp for sending to a peer.

qkey must match what the QP was prepared with. The GID is queried from the QP’s device unless passed explicitly.

Parameters:

gid_index (int)
port (int)

Return type:

EndpointInfo

efa.read_wrs(mr, dest, remote_addr, rkey, *, length=None, offset=0, chunk=1073741824, wr_id=0)¶

Like write_wrs() but for RDMA reads into mr.

Parameters:

offset (int)
chunk (int)
wr_id (int)

efa.reg_tensor(pd, tensor, access)¶

Convenience for a contiguous torch CPU tensor or numpy array. For CUDA tensors use efa.cuda.register_tensor() (dma-buf/GPUDirect) instead.

efa.tensor_addr_len(tensor)¶

Return (addr, nbytes) for a tensor-like object (torch or numpy).

Works with a torch tensor (data_ptr() + numel() * element_size()) or a numpy array (ctypes.data + nbytes). No import of either library is required; the object is duck-typed.

efa.write_wrs(mr, dest, remote_addr, rkey, *, length=None, offset=0, chunk=1073741824, wr_id=0)¶

Build the SendWR list for a (possibly huge) RDMA write.

EFA caps a single RDMA operation at max_rdma_size (1 GiB on current hardware), so a tensor-sized transfer may need several work requests. This slices mr[offset:offset+length] into chunk-sized writes targeting remote_addr at matching offsets. Every WR is SIGNALED (an EFA requirement), so expect len(wrs) completions; they complete in posting order, and SRD completes a WR only once it is fully delivered.

mr may be an MR or a GpuMR-like object exposing addr/length/lkey. Returns a list of SendWRs to pass to post_send(); wr_id numbers them sequentially.

Parameters:

offset (int)
chunk (int)
wr_id (int)

Helpers¶

Thin, optional helpers for the common EFA/SRD workflow.

These build on the raw verbs in efa._efa. The raw QP.modify / QP.to_init / QP.to_rtr / QP.to_rts remain available for callers that want full control.

SRD is connectionless: unlike RC there is no per-QP connection handshake. Bootstrap is a one-shot exchange of EndpointInfo (GID + QP number + qkey) out-of-band, e.g. over a TCP socket or a torch distributed store, after which EndpointInfo.peer() gives a Peer that any number of sends can target.

class efa.helpers.EndpointInfo(gid, qp_num, qkey)¶

Bases: object

The minimum a peer needs to send to an SRD/UD queue pair.

Parameters:

gid (bytes)
qp_num (int)
qkey (int)

gid: bytes¶

qp_num: int¶

qkey: int¶

to_bytes()¶

Return type:: bytes

classmethod from_bytes(data)¶

Parameters:: data (bytes)
Return type:: EndpointInfo

peer(pd, sgid_index=0, port=1)¶

Create the Peer (AH + addressing) for this endpoint.

Parameters:

sgid_index (int)
port (int)

Return type:

Peer

class efa.helpers.Peer(ah, qp_num, qkey)¶

Bases: object

A resolved remote endpoint: an AH plus QP number and qkey.

Pass as dest= to SendWR. Owns the AH; close() (or use as a context manager) destroys it.

Parameters:

qp_num (int)
qkey (int)

close()¶

Return type:: None

efa.helpers.local_endpoint_info(qp, qkey, *, gid=None, gid_index=0, port=1)¶

Build an EndpointInfo describing qp for sending to a peer.

qkey must match what the QP was prepared with. The GID is queried from the QP’s device unless passed explicitly.

Parameters:

gid_index (int)
port (int)

Return type:

EndpointInfo

efa.helpers.tensor_addr_len(tensor)¶

Return (addr, nbytes) for a tensor-like object (torch or numpy).

Works with a torch tensor (data_ptr() + numel() * element_size()) or a numpy array (ctypes.data + nbytes). No import of either library is required; the object is duck-typed.

efa.helpers.reg_tensor(pd, tensor, access)¶

Convenience for a contiguous torch CPU tensor or numpy array. For CUDA tensors use efa.cuda.register_tensor() (dma-buf/GPUDirect) instead.

efa.helpers.write_wrs(mr, dest, remote_addr, rkey, *, length=None, offset=0, chunk=1073741824, wr_id=0)¶

Build the SendWR list for a (possibly huge) RDMA write.

mr may be an MR or a GpuMR-like object exposing addr/length/lkey. Returns a list of SendWRs to pass to post_send(); wr_id numbers them sequentially.

Parameters:

offset (int)
chunk (int)
wr_id (int)

efa.helpers.read_wrs(mr, dest, remote_addr, rkey, *, length=None, offset=0, chunk=1073741824, wr_id=0)¶

Like write_wrs() but for RDMA reads into mr.

Parameters:

offset (int)
chunk (int)
wr_id (int)

CUDA¶

Optional CUDA GPUDirect helpers for registering GPU tensors as memory regions.

This module lazily dlopen``s ``libcuda and imports no torch and links no CUDA at build time, so it adds no required dependency. It is duck-typed: anything exposing data_ptr() (e.g. a torch CUDA tensor) works.

For torch, allocate with the VMM allocator so the memory can be exported as a dma-buf fd (the fd libibverbs needs for ibv_reg_dmabuf_mr):

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Typical use:

import efa, efa.cuda, torch
t = torch.zeros(1 << 20, dtype=torch.uint8, device="cuda")
gmr = efa.cuda.register_tensor(pd, t, efa.AccessFlags.LOCAL_WRITE)
qp.post_send(efa.SendWR(sg_list=[gmr.sge()], dest=peer, ...))

Before the NIC reads a tensor, synchronize the CUDA stream that produced it. After an inbound completion, call flush_gpudirect_writes() before CUDA work consumes the destination.

class efa.cuda.GpuMR(mr, addr, length, tensor=None)¶

Bases: object

A registered GPU memory region plus the info needed to address it.

ibv_mr.addr is not meaningful for dma-buf MRs, so this wraps the real device virtual address. Use sge() for the local scatter/gather list and addr / rkey for the remote side of an RDMA op.

Parameters:

addr (int)
length (int)

property tensor¶: The tensor retained as this registration’s allocation owner.

property closed: bool¶

property lkey: int¶

property rkey: int¶

sge(length=None, offset=0)¶

close()¶

efa.cuda.dmabuf_fd(ptr, length)¶

Export a dma-buf fd for the device VA range [ptr, ptr+length).

ptr and length must be page-aligned (register_tensor() handles alignment). The caller owns and must os.close the returned fd.

Parameters:

ptr (int)
length (int)

Return type:

int

efa.cuda.flush_gpudirect_writes(*, all_devices=False)¶

Make completed inbound GPUDirect RDMA writes visible to CUDA.

Call this after the relevant receive/completion or application-level remote-write notification, before launching CUDA work that consumes the destination tensor. The current thread must have the destination tensor’s CUDA context active.

Parameters:: all_devices (bool)
Return type:: None

efa.cuda.tensor_ptr_len(tensor)¶: Return (device_addr, nbytes) for a tensor-like object.

efa.cuda.register_tensor(pd, tensor, access)¶

Prefers the dma-buf path (no kernel module needed) and transparently handles page alignment. Falls back to the raw reg_mr (nvidia_peermem) path when dma-buf export or registration is unavailable. The returned object retains tensor so its allocation cannot be recycled while the MR is registered.

Return type:: GpuMR

Enums¶

Pythonic enums mirroring the libibverbs / libefa ABI constants.

These values are part of the stable rdma-core ABI. They are declared here as enum.IntEnum / enum.IntFlag so they interoperate with plain ints everywhere the low-level API accepts a flag or code.

class efa.enums.AccessFlags(*values)¶

Bases: IntFlag

Memory-region access permissions (IBV_ACCESS_*).

LOCAL_WRITE = 1¶

REMOTE_WRITE = 2¶

REMOTE_READ = 4¶

RELAXED_ORDERING = 1048576¶

class efa.enums.QPType(*values)¶

Bases: IntEnum

EFA queue-pair type.

UD = 4¶

SRD = 255¶

class efa.enums.QPState(*values)¶

Bases: IntEnum

Queue-pair state (IBV_QPS_*).

RESET = 0¶

INIT = 1¶

RTR = 2¶

RTS = 3¶

SQD = 4¶

SQE = 5¶

ERR = 6¶

UNKNOWN = 7¶

class efa.enums.QPAttrMask(*values)¶

Bases: IntFlag

Bitmask selecting which fields modify_qp applies (IBV_QP_*).

STATE = 1¶

CUR_STATE = 2¶

PKEY_INDEX = 16¶

PORT = 32¶

QKEY = 64¶

RNR_RETRY = 2048¶

RQ_PSN = 4096¶

SQ_PSN = 65536¶

class efa.enums.QPExSendOpsFlags(*values)¶

Bases: IntFlag

Extended-QP send operations (IBV_QP_EX_WITH_*).

Passed as QPInitAttr.send_ops_flags; the created QP can only post the opcodes enabled here.

RDMA_WRITE = 1¶

RDMA_WRITE_WITH_IMM = 2¶

SEND = 4¶

SEND_WITH_IMM = 8¶

RDMA_READ = 16¶

class efa.enums.WROpcode(*values)¶

Bases: IntEnum

Send work-request opcode (IBV_WR_*, the subset EFA supports).

RDMA_WRITE = 0¶

RDMA_WRITE_WITH_IMM = 1¶

SEND = 2¶

SEND_WITH_IMM = 3¶

RDMA_READ = 4¶

class efa.enums.SendFlags(*values)¶

Bases: IntFlag

Send work-request flags (IBV_SEND_*).

FENCE = 1¶

SIGNALED = 2¶

SOLICITED = 4¶

INLINE = 8¶

class efa.enums.WCStatus(*values)¶

Bases: IntEnum

Work-completion status (IBV_WC_*).

SUCCESS = 0¶

LOC_LEN_ERR = 1¶

LOC_QP_OP_ERR = 2¶

LOC_EEC_OP_ERR = 3¶

LOC_PROT_ERR = 4¶

WR_FLUSH_ERR = 5¶

MW_BIND_ERR = 6¶

BAD_RESP_ERR = 7¶

LOC_ACCESS_ERR = 8¶

REM_INV_REQ_ERR = 9¶

REM_ACCESS_ERR = 10¶

REM_OP_ERR = 11¶

RETRY_EXC_ERR = 12¶

RNR_RETRY_EXC_ERR = 13¶

LOC_RDD_VIOL_ERR = 14¶

REM_INV_RD_REQ_ERR = 15¶

REM_ABORT_ERR = 16¶

INV_EECN_ERR = 17¶

INV_EEC_STATE_ERR = 18¶

FATAL_ERR = 19¶

RESP_TIMEOUT_ERR = 20¶

GENERAL_ERR = 21¶

class efa.enums.WCOpcode(*values)¶

Bases: IntEnum

Work-completion opcode (IBV_WC_*).

SEND = 0¶

RDMA_WRITE = 1¶

RDMA_READ = 2¶

RECV = 128¶

RECV_RDMA_WITH_IMM = 129¶

class efa.enums.WCFlags(*values)¶

Bases: IntFlag

Work-completion flags (IBV_WC_* bit flags).

GRH = 1¶

WITH_IMM = 2¶

class efa.enums.CreateCQWCFlags(*values)¶

Bases: IntFlag

Extended-CQ per-completion fields (IBV_WC_EX_WITH_*).

STANDARD matches what classic ibv_poll_cq reports and is the default for create_cq_ex().

BYTE_LEN = 1¶

IMM = 2¶

QP_NUM = 4¶

SRC_QP = 8¶

SLID = 16¶

SL = 32¶

DLID_PATH_BITS = 64¶

STANDARD = 127¶

class efa.enums.EfaDeviceCaps(*values)¶

Bases: IntFlag

EFA device capability bits (EFADV_DEVICE_ATTR_CAPS_*).

Test against device_caps.

RDMA_READ = 1¶

RNR_RETRY = 2¶

CQ_WITH_SGID = 4¶

RDMA_WRITE = 8¶

UNSOLICITED_WRITE_RECV = 16¶

CQ_WITH_EXT_MEM_DMABUF = 32¶

class efa.enums.MTU(*values)¶

Bases: IntEnum

Path MTU (IBV_MTU_*).

MTU_256 = 1¶

MTU_512 = 2¶

MTU_1024 = 3¶

MTU_2048 = 4¶

MTU_4096 = 5¶

class efa.enums.PortState(*values)¶

Bases: IntEnum

Port state (IBV_PORT_*).

NOP = 0¶

DOWN = 1¶

INIT = 2¶

ARMED = 3¶

ACTIVE = 4¶

ACTIVE_DEFER = 5¶