EFA API#

Core API#

efa: low-level Pythonic bindings for AWS EFA (SRD RDMA).

Quickstart (SRD is connectionless – no per-QP handshake, just exchange EndpointInfo once):

import efa

dev = efa.get_efa_device_list()[0]
ctx = dev.open()
pd = ctx.alloc_pd()
cq = ctx.create_cq(256)
qp = pd.create_qp(efa.QPInitAttr(send_cq=cq, recv_cq=cq)).prepare(qkey=0x1234)

info = efa.local_endpoint_info(qp, qkey=0x1234)
# ... exchange info.to_bytes() with the peer out-of-band ...
peer = efa.EndpointInfo.from_bytes(remote_bytes).peer(pd)

mr = efa.reg_tensor(pd, tensor, efa.AccessFlags.LOCAL_WRITE)
qp.post_send(efa.SendWR(sg_list=[mr.sge()], opcode=efa.WROpcode.SEND,
                        send_flags=efa.SendFlags.SIGNALED, dest=peer))

The compiled fast paths live in efa._efa; enums in efa.enums; address-exchange and tensor helpers in efa.helpers.

class efa.AH#

Bases: object

An address handle naming a remote EFA device (by GID).

ahn#

int

The EFA address handle number (efadv_query_ah).

Type:: AH.ahn

close(self)#: Destroy this address handle.

pd#

class efa.CQ#

Bases: _CQBase

A classic completion queue (ibv_cq).

close(self)#: Destroy this classic completion queue.

poll(self, int num_entries) → list#

Poll up to num_entries completions; return a list of WC.

Return type:: list

query_efa(self) → CQAttr#

Return the provider’s direct CQ layout (EFA 1.4 or newer).

Return type:: CQAttr

class efa.CQAttr#

Bases: object

An EFA completion queue’s direct layout from efadv_query_cq.

buffer_addr#

doorbell_addr#

entry_size#

num_entries#

class efa.CQEx#

Bases: _CQBase

An extended EFA completion queue (efadv_create_cq).

Functionally a CQ, plus optional sender-GID reporting (sgid=True): SRD/UD receivers can identify previously unknown peers straight from the completion (see WC.sgid).

close(self)#: Destroy this extended completion queue.

poll(self, int num_entries) → list#

Poll up to num_entries completions; return a list of WC.

Return type:: list

query_efa(self) → CQAttr#

Return the provider’s direct CQ layout (EFA 1.4 or newer).

Return type:: CQAttr

sgid_enabled#

bool

Whether polled completions request sender GID metadata.

Type:: CQEx.sgid_enabled

unsolicited_enabled#

bool

Whether unsolicited-write completion metadata is enabled.

Type:: CQEx.unsolicited_enabled

class efa.MR#

Bases: object

A registered memory region (ibv_mr).

addr#

int

Return the registered region’s starting virtual address.

Type:: MR.addr

close(self)#: Deregister this memory region.

closed#

bool

Whether this memory region has been deregistered.

Type:: MR.closed

handle#

int

Return the provider memory-region handle.

Type:: MR.handle

length#

int

Return the registered length in bytes.

Type:: MR.length

lkey#

int

Return the local key used by this process’s QPs.

Type:: MR.lkey

owner#

pd#

query_efa(self) → MRAttr#

EFA-specific MR attributes (efadv_query_mr, rdma-core >= 44).

Return type:: MRAttr

rkey#

int

Return the key that authorizes remote access.

Type:: MR.rkey

sge(self, length=None, offset=0)#: Return an SGE for a bounded range of this MR.

class efa.PD#

Bases: object

A protection domain (ibv_pd).

close(self)#: Deallocate the protection domain when it has no open children.

context#

create_ah(self, addr, int sgid_index=0, int port_num=1) → 'AH'#

Create an address handle for a remote EFA device.

addr may be an AHAttr, a Gid, or 16 raw GID bytes.

Return type:: AH

create_qp(self, init_attr) → 'QP'#

Create an SRD (default) or UD queue pair.

SRD QPs go through efadv_create_qp_ex; UD QPs through ibv_create_qp_ex. Both are created with the extended send-ops work-request API enabled (see QPInitAttr).

Return type:: QP

reg_dmabuf_mr(self, offset, length, iova, int fd, int access) → 'MR'#

Return type:: MR

reg_mr(self, addr, length, int access) → 'MR'#

addr is an integer virtual address; it may be a host pointer or a CUDA device pointer (GPUDirect via nvidia_peermem).

Return type:: MR

class efa.QP#

Bases: object

An EFA queue pair (SRD or UD).

Sends go through the extended work-request API under the hood, so every supported opcode, including RDMA read/write on SRD, which the classic ibv_post_send cannot express on this provider, posts uniformly via post_send().

close(self)#: Destroy this queue pair.

modify(self, int attr_mask, **fields)#

Low-level ibv_modify_qp.

attr_mask is an ORed QPAttrMask. Scalar attributes are passed as keywords (e.g. qp_state=, qkey=).

pd#

post_recv(self, wrs)#: Post one or more RecvWR to the receive queue.

post_send(self, wrs)#

Post one or more SendWR as a single batch.

All work requests are queued between wr_start/wr_complete, so either the whole batch is submitted or none of it is.

EFA requires every send WR to carry SIGNALED (or the QP must be created with sq_sig_all=True); an unsignaled WR fails the batch with EINVAL.

prepare(self, qkey, int port=1, psn=0, rnr_retry=None)#

Drive RESET -> INIT -> RTR -> RTS in one call.

SRD/UD are connectionless, so a QP is fully usable after this without any remote information; destinations are named per-send.

qkey, port, psn, and rnr_retry are forwarded to the corresponding state-transition methods.

qp_num#

int

Return the provider-assigned queue-pair number.

Type:: QP.qp_num

qp_type#

query(self)#: Return (attrs, cap) for the queue pair as plain dict + QPCap.

query_wqs(self)#: Return (sq, rq) WQAttr (efadv_query_qp_wqs, EFA >= 1.4).

recv_cq#

send_cq#

send_ops_flags#

state#

int

The queue pair’s current state (authoritative, via query).

Type:: QP.state

to_init(self, qkey, int port=1, int pkey_index=0)#

Transition RESET -> INIT.

qkey authorizes incoming datagrams; port selects the local physical port and pkey_index selects its partition-key entry.

to_rtr(self)#: Transition INIT -> RTR (connectionless: no remote info needed).

to_rts(self, psn=0, rnr_retry=None)#

Transition RTR -> RTS.

psn sets the starting send packet sequence number. rnr_retry (0-7, 7 = infinite) is EFA’s receiver-not-ready retry count and needs RNR_RETRY.

class efa.AHAttr(dgid=None, sgid_index=0, port_num=1, hop_limit=1, traffic_class=0, flow_label=0, is_global=1)#

Bases: object

Address-handle attributes.

EFA addresses are GIDs: only dgid (the remote device’s GID), sgid_index and port_num matter; the rest exists for parity with generic verbs.

Parameters:

dgid – Remote 16-byte GID. Defaults to all zeroes.
sgid_index – Local source-GID table index.
port_num – Local physical port number.
hop_limit – Global-route hop limit.
traffic_class – Global-route traffic class.
flow_label – Global-route flow label.
is_global – Whether to populate the global route header.

dgid#

bytes

Type:: dgid

flow_label#

‘uint32_t’

Type:: flow_label

hop_limit#

‘uint8_t’

Type:: hop_limit

is_global#

‘uint8_t’

Type:: is_global

port_num#

‘uint8_t’

Type:: port_num

sgid_index#

‘uint8_t’

Type:: sgid_index

traffic_class#

‘uint8_t’

Type:: traffic_class

class efa.AsyncEvent#

Bases: object

An asynchronous device event from Context.get_async_event().

event_type#

event_type_str#

str

Return the human-readable asynchronous event type.

Type:: AsyncEvent.event_type_str

class efa.CompChannel#

Bases: object

A completion event channel (ibv_comp_channel).

close(self)#: Destroy the completion channel after its CQs are closed.

context#

fd#

int

Return the completion channel’s pollable file descriptor.

Type:: CompChannel.fd

get_cq_event(self)#

Block until a CQ event arrives; return the associated CQ object.

The event must later be acknowledged with ack_events() on the returned CQ / CQEx.

exception efa.CompletionError(wc)#

Bases: EfaError

Raised by WC.raise_for_status() for a failed completion.

class efa.Context#

Bases: object

An open device context (ibv_context).

ack_async_event(self, AsyncEvent ev)#: Acknowledge an event returned by get_async_event().

alloc_pd(self) → 'PD'#

Allocate a protection domain owned by this context.

Return type:: PD

async_fd#

int

Return the file descriptor used for asynchronous device events.

Type:: Context.async_fd

close(self)#: Close the device context after all child resources are closed.

create_comp_channel(self) → 'CompChannel'#

Create a completion event channel owned by this context.

Return type:: CompChannel

create_cq(self, int cqe, channel=None, int comp_vector=0) → 'CQ'#

Create a classic completion queue (ibv_create_cq).

cqe is the requested minimum capacity. channel optionally enables event notification, and comp_vector selects its interrupt vector from 0 through num_comp_vectors - 1.

Return type:: CQ

create_cq_ex(self, int cqe, channel=None, int comp_vector=0, wc_flags=None, bool sgid=False, bool unsolicited=False) → 'CQEx'#

Create an extended EFA completion queue (efadv_create_cq).

sgid=True requests sender-GID reporting on receive completions (needs CQ_WITH_SGID); the GID shows up as WC.sgid for peers not matched by a local AH.

unsolicited=True requests an indicator for unsolicited write-with-immediate receive completions (needs UNSOLICITED_WRITE_RECV).

cqe, channel, and comp_vector have the same meaning as in create_cq(). wc_flags is an ORed CreateCQWCFlags mask and defaults to STANDARD.

Return type:: CQEx

get_async_event(self) → 'AsyncEvent'#

Block until an asynchronous device event is available.

Return type:: AsyncEvent

is_efa(self) → bool#

Return True if this device is driven by the EFA provider.

Return type:: bool

name#

num_comp_vectors#

int

Return the number of completion interrupt vectors.

Type:: Context.num_comp_vectors

query_device(self) → DeviceAttr#

Query generic verbs capabilities for this device.

Return type:: DeviceAttr

query_efa_device(self) → EfaDeviceAttr#

EFA capabilities (efadv_query_device).

Raises EfaError if the device is not an EFA device.

Return type:: EfaDeviceAttr

query_gid(self, int port_num=1, int index=0) → Gid#

The device GID, which is the EFA device’s network address.

Return type:: Gid

query_port(self, int port_num=1) → PortAttr#

Query attributes for port_num.

Return type:: PortAttr

class efa.Device(str name, uint64_t guid)#

Bases: object

An RDMA device discovered by get_device_list().

guid#

name#

open(self) → 'Context'#

Open this device, returning a Context.

Return type:: Context

class efa.DeviceAttr#

Bases: object

Generic verbs device capabilities from Context.query_device().

fw_ver#

hw_ver#

max_ah#

max_cq#

max_cqe#

max_mr#

max_mr_size#

max_pd#

max_pkeys#

max_qp#

max_qp_wr#

max_sge#

node_guid#

phys_port_cnt#

sys_image_guid#

vendor_id#

vendor_part_id#

class efa.EfaDeviceAttr#

Bases: object

EFA-specific capabilities from Context.query_efa_device().

device_caps#

inline_buf_size#

max_rdma_size#

max_rq_sge#

max_rq_wr#

max_sq_sge#

max_sq_wr#

exception efa.EfaError(operation, err, detail=None)#

Bases: OSError

Raised when a libibverbs/libefa call fails.

Subclasses OSError, so .errno and .strerror are populated. .operation names the call that failed.

class efa.Gid(bytes raw)#

Bases: object

A 16-byte GID (the EFA device address).

interface_id#

int

Return the least-significant 64 bits in network byte order.

Type:: Gid.interface_id

raw#

subnet_prefix#

int

Return the most-significant 64 bits in network byte order.

Type:: Gid.subnet_prefix

class efa.MRAttr#

Bases: object

EFA MR attributes from MR.query_efa() (efadv_query_mr).

ic_id_validity#

rdma_read_ic_id#

rdma_recv_ic_id#

recv_ic_id#

class efa.PortAttr#

Bases: object

Port attributes from Context.query_port().

max_msg_sz is the SEND size limit; RDMA read/write are limited by EfaDeviceAttr.max_rdma_size instead.

active_mtu#

active_speed#

active_width#

gid_tbl_len#

lid#

link_layer#

max_msg_sz#

max_mtu#

pkey_tbl_len#

port_cap_flags#

state#

class efa.QPCap#

Bases: object

Queue-pair capacity limits.

max_inline_data#

max_recv_sge#

max_recv_wr#

max_send_sge#

max_send_wr#

class efa.QPInitAttr(send_cq, recv_cq, qp_type=None, max_send_wr=128, max_recv_wr=128, max_send_sge=1, max_recv_sge=1, max_inline_data=0, sq_sig_all=False, send_ops_flags=None, sl=0, unsolicited_write_recv=False)#

Bases: object

Parameters for PD.create_qp().

Parameters:

send_cq – Completion queue for send work requests.
recv_cq – Completion queue for receive work requests.
qp_type – SRD (default) or UD.
max_send_wr – Requested maximum outstanding send work requests.
max_recv_wr – Requested maximum posted receive work requests.
max_send_sge – Requested SGE limit for each send request.
max_recv_sge – Requested SGE limit for each receive request.
max_inline_data – Requested maximum inline payload in bytes.
sq_sig_all – Make every send produce a completion, even without the SIGNALED flag. EFA requires one of these two signaling mechanisms for every send.
send_ops_flags – Enabled QPExSendOpsFlags. The default enables every operation supported by the QP type. Pass an explicit mask on devices without RDMA read/write support.
sl – EFA service level used when creating an SRD QP.
unsolicited_write_recv – Accept RDMA-write-with-immediate without a posted receive. Requires the matching EFA device capability and an extended CQ created with unsolicited=True.

class efa.RecvWR(wr_id=0, sg_list=None)#

Bases: object

A receive work request.

Parameters:

wr_id – Application-defined value returned in the completion.
sg_list – Local SGE objects that receive the payload.

sg_list#

list

Type:: sg_list

wr_id#

‘uint64_t’

Type:: wr_id

class efa.SendWR(wr_id=0, sg_list=None, opcode=2, send_flags=0, remote_addr=0, rkey=0, imm_data=0, ah=None, remote_qpn=0, remote_qkey=0, dest=None)#

Bases: object

A send work request.

SRD/UD is connectionless: every send names its destination. Set either dest= (anything with .ah, .qp_num and .qkey attributes, e.g. efa.helpers.Peer) or the explicit ah/remote_qpn/ remote_qkey trio.

Opcodes: SEND, SEND_WITH_IMM, RDMA_READ, RDMA_WRITE, RDMA_WRITE_WITH_IMM (RDMA opcodes additionally need remote_addr/rkey).

Parameters:

wr_id – Application-defined value returned in the completion.
sg_list – Local SGE objects. Defaults to an empty list.
opcode – A WROpcode value; defaults to SEND.
send_flags – ORed SendFlags values. EFA requires SIGNALED unless the QP uses sq_sig_all=True.
remote_addr – Remote virtual address for an RDMA operation.
rkey – Remote memory key for an RDMA operation.
imm_data – Immediate data for an opcode ending in WITH_IMM.
ah – Explicit destination address handle.
remote_qpn – Explicit destination queue-pair number.
remote_qkey – Explicit destination qkey.
dest – Convenience object supplying ah, qp_num, and qkey; mutually exclusive with the explicit destination fields.

ah#

object

Type:: ah

imm_data#

‘uint32_t’

Type:: imm_data

opcode#

‘int’

Type:: opcode

remote_addr#

‘uint64_t’

Type:: remote_addr

remote_qkey#

‘uint32_t’

Type:: remote_qkey

remote_qpn#

‘uint32_t’

Type:: remote_qpn

rkey#

‘uint32_t’

Type:: rkey

send_flags#

‘unsigned int’

Type:: send_flags

sg_list#

list

Type:: sg_list

wr_id#

‘uint64_t’

Type:: wr_id

class efa.SGE(target, length, lkey=0, offset=0)#

Bases: object

A scatter/gather entry.

SGE(target, length, lkey=0, offset=0) where target is either an MR (lkey/addr derived from it) or an integer address.

addr#

length#

lkey#

owner#

class efa.WC#

Bases: object

A work completion returned by CQ.poll() / CQEx.poll().

For completions from a CQEx created with sgid=True, sgid carries the sender’s GID on receive completions from peers not covered by a local AH (else None).

byte_len#

imm_data#

is_success#

bool

Whether the work request completed successfully.

Type:: WC.is_success

opcode#

qp_num#

raise_for_status(self)#: Raise CompletionError if this completion did not succeed.

sgid#

src_qp#

status#

status_str#

str

Return the provider’s human-readable completion status.

Type:: WC.status_str

unsolicited#

vendor_err#

wc_flags#

wr_id#

class efa.WQAttr#

Bases: object

A work queue’s layout from QP.query_wqs() (efadv_query_qp_wqs).

buffer_addr#

doorbell_addr#

entry_size#

max_batch#

num_entries#

efa.get_device_list() → list#

Return every RDMA Device on this host (EFA or not).

Use get_efa_device_list() to keep only EFA devices.

Return type:: list

efa.get_efa_device_list() → list#

Return the Device list filtered to EFA devices.

Each device is briefly opened and probed with efadv_query_device.

Return type:: list

class efa.AccessFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Memory-region access permissions (IBV_ACCESS_*).

LOCAL_WRITE = 1#: Permit the local HCA to write the registered memory.

REMOTE_WRITE = 2#: Permit a remote peer to write the region; also requires LOCAL_WRITE.

REMOTE_READ = 4#: Permit a remote peer to read the region.

RELAXED_ORDERING = 1048576#: Allow relaxed inbound write ordering for higher performance.

class efa.CreateCQWCFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Extended-CQ per-completion fields (IBV_WC_EX_WITH_*).

STANDARD matches what classic ibv_poll_cq reports and is the default for create_cq_ex().

BYTE_LEN = 1#: Request the completed byte count.

IMM = 2#: Request immediate data.

QP_NUM = 4#: Request the local queue-pair number.

SRC_QP = 8#: Request the source queue-pair number.

SLID = 16#: Request the source LID.

SL = 32#: Request the service level.

DLID_PATH_BITS = 64#: Request destination LID path bits.

STANDARD = 127#: Request every field reported by a classic completion queue.

class efa.EfaDeviceCaps(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

EFA device capability bits (EFADV_DEVICE_ATTR_CAPS_*).

Test against device_caps.

RDMA_READ = 1#: The provider supports SRD RDMA Read.

RNR_RETRY = 2#: The provider supports configurable RNR retry.

CQ_WITH_SGID = 4#: Extended completion queues can report sender GIDs.

RDMA_WRITE = 8#: The provider supports SRD RDMA Write.

UNSOLICITED_WRITE_RECV = 16#: QPs can receive unsolicited RDMA Write with Immediate.

CQ_WITH_EXT_MEM_DMABUF = 32#: Extended completion-queue memory can be supplied through dma-buf.

class efa.MTU(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Path MTU (IBV_MTU_*).

MTU_256 = 1#

MTU_512 = 2#

MTU_1024 = 3#

MTU_2048 = 4#

MTU_4096 = 5#

class efa.PortState(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Port state (IBV_PORT_*).

NOP = 0#

DOWN = 1#

INIT = 2#

ARMED = 3#

ACTIVE = 4#

ACTIVE_DEFER = 5#

class efa.QPAttrMask(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Bitmask selecting which fields modify_qp applies (IBV_QP_*).

STATE = 1#: Apply qp_state.

CUR_STATE = 2#: Apply cur_qp_state as a transition precondition.

PKEY_INDEX = 16#: Apply the partition-key table index.

PORT = 32#: Apply the local physical port.

QKEY = 64#: Apply the datagram queue key.

RNR_RETRY = 2048#: Apply the receiver-not-ready retry count.

RQ_PSN = 4096#: Apply the receive packet sequence number.

SQ_PSN = 65536#: Apply the send packet sequence number.

class efa.QPExSendOpsFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Extended-QP send operations (IBV_QP_EX_WITH_*).

Passed as QPInitAttr.send_ops_flags; the created QP can only post the opcodes enabled here.

RDMA_WRITE = 1#: Enable RDMA Write work requests.

RDMA_WRITE_WITH_IMM = 2#: Enable RDMA Write with Immediate work requests.

SEND = 4#: Enable Send work requests.

SEND_WITH_IMM = 8#: Enable Send with Immediate work requests.

RDMA_READ = 16#: Enable RDMA Read work requests.

class efa.QPState(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Queue-pair state (IBV_QPS_*).

RESET = 0#

INIT = 1#

RTR = 2#

RTS = 3#

SQD = 4#

SQE = 5#

ERR = 6#

UNKNOWN = 7#

class efa.QPType(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

EFA queue-pair type.

SRD (Scalable Reliable Datagram – reliable, connectionless, out-of-order; EFA’s native transport) or UD (unreliable datagram). SRD maps onto IBV_QPT_DRIVER and is created through the EFA provider.

UD = 4#

SRD = 255#

class efa.SendFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Send work-request flags (IBV_SEND_*).

FENCE = 1#: Fence this request behind earlier RDMA Read or atomic operations.

SIGNALED = 2#: Generate a send completion for this request.

SOLICITED = 4#: Request a solicited receive event at the peer.

INLINE = 8#: Copy the payload into the work request instead of retaining its SGEs.

class efa.WCFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Work-completion flags (IBV_WC_* bit flags).

GRH = 1#: A global route header precedes the received payload.

WITH_IMM = 2#: The completion contains valid immediate data.

class efa.WCOpcode(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Work-completion opcode (IBV_WC_*).

SEND = 0#

RDMA_WRITE = 1#

RDMA_READ = 2#

RECV = 128#

RECV_RDMA_WITH_IMM = 129#

class efa.WCStatus(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Work-completion status (IBV_WC_*).

SUCCESS = 0#

LOC_LEN_ERR = 1#

LOC_QP_OP_ERR = 2#

LOC_EEC_OP_ERR = 3#

LOC_PROT_ERR = 4#

WR_FLUSH_ERR = 5#

MW_BIND_ERR = 6#

BAD_RESP_ERR = 7#

LOC_ACCESS_ERR = 8#

REM_INV_REQ_ERR = 9#

REM_ACCESS_ERR = 10#

REM_OP_ERR = 11#

RETRY_EXC_ERR = 12#

RNR_RETRY_EXC_ERR = 13#

LOC_RDD_VIOL_ERR = 14#

REM_INV_RD_REQ_ERR = 15#

REM_ABORT_ERR = 16#

INV_EECN_ERR = 17#

INV_EEC_STATE_ERR = 18#

FATAL_ERR = 19#

RESP_TIMEOUT_ERR = 20#

GENERAL_ERR = 21#

class efa.WROpcode(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Send work-request opcode (IBV_WR_*, the subset EFA supports).

RDMA_WRITE = 0#

RDMA_WRITE_WITH_IMM = 1#

SEND = 2#

SEND_WITH_IMM = 3#

RDMA_READ = 4#

class efa.EndpointInfo(gid, qp_num, qkey)#

Bases: object

The minimum a peer needs to send to an SRD/UD queue pair.

Exchange this out-of-band (e.g. over a TCP socket or a distributed key-value store) during setup. to_bytes() / from_bytes() give a fixed 24-byte wire layout so both ends agree regardless of platform endianness.

Parameters:

gid (bytes) –
qp_num (int) –
qkey (int) –

gid: bytes#

qp_num: int#

qkey: int#

to_bytes()#

Serialize this endpoint to its fixed 24-byte network format.

Return type:: bytes

classmethod from_bytes(data)#

Parse a fixed 24-byte endpoint description.

Parameters:: data (bytes) –
Return type:: EndpointInfo

peer(pd, sgid_index=0, port=1)#

Create the Peer (AH + addressing) for this endpoint.

pd owns the new address handle; sgid_index and port select the local source GID and physical port used to reach the peer.

Parameters:

sgid_index (int) –
port (int) –

Return type:

Peer

class efa.Peer(ah, qp_num, qkey)#

Bases: object

A resolved remote endpoint: an AH plus QP number and qkey.

Pass as dest= to SendWR. Owns the AH; close() (or use as a context manager) destroys it.

Parameters:

qp_num (int) –
qkey (int) –

close()#

Destroy the owned address handle, if it is still open.

Return type:: None

efa.local_endpoint_info(qp, qkey, *, gid=None, gid_index=0, port=1)#

Build an EndpointInfo describing qp for sending to a peer.

qkey must match what the QP was prepared with. The GID is queried from the QP’s device unless passed explicitly. gid_index and port select that query when gid is omitted.

Parameters:

gid_index (int) –
port (int) –

Return type:

EndpointInfo

efa.read_wrs(mr, dest, remote_addr, rkey, *, length=None, offset=0, chunk=1073741824, wr_id=0)#

Build the work requests for an RDMA read into mr.

dest names the remote EFA peer; remote_addr and rkey select its memory region. length defaults to the local MR remainder after offset. chunk is the maximum bytes per WR, and wr_id seeds the sequential request identifiers.

Parameters:

offset (int) –
chunk (int) –
wr_id (int) –

efa.reg_tensor(pd, tensor, access)#

Convenience for a contiguous torch CPU tensor or numpy array. For CUDA tensors use efa.cuda.register_tensor() (dma-buf/GPUDirect) instead. pd is the owning protection domain and access is an ORed AccessFlags mask.

efa.tensor_addr_len(tensor)#

Return (addr, nbytes) for a tensor-like object (torch or numpy).

Works with a torch tensor (data_ptr() + numel() * element_size()) or a numpy array (ctypes.data + nbytes). No import of either library is required; the object is duck-typed.

efa.write_wrs(mr, dest, remote_addr, rkey, *, length=None, offset=0, chunk=1073741824, wr_id=0)#

Build the SendWR list for a (possibly huge) RDMA write.

EFA caps a single RDMA operation at max_rdma_size (1 GiB on current hardware), so a tensor-sized transfer may need several work requests. This slices mr[offset:offset+length] into chunk-sized writes targeting remote_addr at matching offsets. Every WR is SIGNALED (an EFA requirement), so expect len(wrs) completions; they complete in posting order, and SRD completes a WR only once it is fully delivered.

mr may be an MR or a GpuMR-like object exposing addr/length/lkey. Returns a list of SendWRs to pass to post_send(); wr_id numbers them sequentially.

dest names the remote EFA peer; remote_addr and rkey select its memory region. length defaults to the local MR remainder after offset. chunk is the maximum bytes per WR.

Parameters:

offset (int) –
chunk (int) –
wr_id (int) –

Helpers#

Thin, optional helpers for the common EFA/SRD workflow.

These build on the raw verbs in efa._efa. The raw QP.modify / QP.to_init / QP.to_rtr / QP.to_rts remain available for callers that want full control.

SRD is connectionless: unlike RC there is no per-QP connection handshake. Bootstrap is a one-shot exchange of EndpointInfo (GID + QP number + qkey) out-of-band, e.g. over a TCP socket or a torch distributed store, after which EndpointInfo.peer() gives a Peer that any number of sends can target.

class efa.helpers.EndpointInfo(gid, qp_num, qkey)#

Bases: object

The minimum a peer needs to send to an SRD/UD queue pair.

Parameters:

gid (bytes) –
qp_num (int) –
qkey (int) –

gid: bytes#

qp_num: int#

qkey: int#

to_bytes()#

Serialize this endpoint to its fixed 24-byte network format.

Return type:: bytes

classmethod from_bytes(data)#

Parse a fixed 24-byte endpoint description.

Parameters:: data (bytes) –
Return type:: EndpointInfo

peer(pd, sgid_index=0, port=1)#

Create the Peer (AH + addressing) for this endpoint.

pd owns the new address handle; sgid_index and port select the local source GID and physical port used to reach the peer.

Parameters:

sgid_index (int) –
port (int) –

Return type:

Peer

class efa.helpers.Peer(ah, qp_num, qkey)#

Bases: object

A resolved remote endpoint: an AH plus QP number and qkey.

Pass as dest= to SendWR. Owns the AH; close() (or use as a context manager) destroys it.

Parameters:

qp_num (int) –
qkey (int) –

close()#

Destroy the owned address handle, if it is still open.

Return type:: None

efa.helpers.local_endpoint_info(qp, qkey, *, gid=None, gid_index=0, port=1)#

Build an EndpointInfo describing qp for sending to a peer.

qkey must match what the QP was prepared with. The GID is queried from the QP’s device unless passed explicitly. gid_index and port select that query when gid is omitted.

Parameters:

gid_index (int) –
port (int) –

Return type:

EndpointInfo

efa.helpers.tensor_addr_len(tensor)#

Return (addr, nbytes) for a tensor-like object (torch or numpy).

Works with a torch tensor (data_ptr() + numel() * element_size()) or a numpy array (ctypes.data + nbytes). No import of either library is required; the object is duck-typed.

efa.helpers.reg_tensor(pd, tensor, access)#

efa.helpers.write_wrs(mr, dest, remote_addr, rkey, *, length=None, offset=0, chunk=1073741824, wr_id=0)#

Build the SendWR list for a (possibly huge) RDMA write.

mr may be an MR or a GpuMR-like object exposing addr/length/lkey. Returns a list of SendWRs to pass to post_send(); wr_id numbers them sequentially.

dest names the remote EFA peer; remote_addr and rkey select its memory region. length defaults to the local MR remainder after offset. chunk is the maximum bytes per WR.

Parameters:

offset (int) –
chunk (int) –
wr_id (int) –

efa.helpers.read_wrs(mr, dest, remote_addr, rkey, *, length=None, offset=0, chunk=1073741824, wr_id=0)#

Build the work requests for an RDMA read into mr.

Parameters:

offset (int) –
chunk (int) –
wr_id (int) –

CUDA#

Optional CUDA GPUDirect helpers for registering GPU tensors as memory regions.

This module lazily dlopen``s ``libcuda and imports no torch and links no CUDA at build time, so it adds no required dependency. It is duck-typed: anything exposing data_ptr() (e.g. a torch CUDA tensor) works.

For torch, allocate with the VMM allocator so the memory can be exported as a dma-buf fd (the fd libibverbs needs for ibv_reg_dmabuf_mr):

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Typical use:

import efa, efa.cuda, torch
t = torch.zeros(1 << 20, dtype=torch.uint8, device="cuda")
gmr = efa.cuda.register_tensor(pd, t, efa.AccessFlags.LOCAL_WRITE)
qp.post_send(efa.SendWR(sg_list=[gmr.sge()], dest=peer, ...))

Before the NIC reads a tensor, synchronize the CUDA stream that produced it. After an inbound completion, call flush_gpudirect_writes() before CUDA work consumes the destination.

class efa.cuda.GpuMR(mr, addr, length, tensor=None)#

Bases: object

A registered GPU memory region plus the info needed to address it.

ibv_mr.addr is not meaningful for dma-buf MRs, so this wraps the real device virtual address. Use sge() for the local scatter/gather list and addr / rkey for the remote side of an RDMA op.

Parameters:

addr (int) –
length (int) –

property tensor#: The tensor retained as this registration’s allocation owner.

property closed: bool#: Whether the underlying memory region has been deregistered.

property lkey: int#: Return the local key used in scatter/gather entries.

property rkey: int#: Return the remote key shared with an RDMA peer.

sge(length=None, offset=0)#: Build an SGE for a byte range of this GPU memory region.

close()#: Deregister the memory region and release the retained tensor.

efa.cuda.dmabuf_fd(ptr, length)#

Export a dma-buf fd for the device VA range [ptr, ptr+length).

ptr and length must be page-aligned (register_tensor() handles alignment). The caller owns and must os.close the returned fd.

Parameters:

ptr (int) –
length (int) –

Return type:

int

efa.cuda.flush_gpudirect_writes(*, all_devices=False)#

Make completed inbound GPUDirect RDMA writes visible to CUDA.

Call this after the relevant receive/completion or application-level remote-write notification, before launching CUDA work that consumes the destination tensor. The current thread must have the destination tensor’s CUDA context active. By default CUDA flushes writes only to the owning device; set all_devices=True to flush visibility to every GPU in the current context’s scope.

Parameters:: all_devices (bool) –
Return type:: None

efa.cuda.tensor_ptr_len(tensor)#: Return (device_addr, nbytes) for a tensor-like object.

efa.cuda.register_tensor(pd, tensor, access)#

Prefers the dma-buf path (no kernel module needed) and transparently handles page alignment. Falls back to the raw reg_mr (nvidia_peermem) path when dma-buf export or registration is unavailable. The returned object retains tensor so its allocation cannot be recycled while the MR is registered.

pd is the owning protection domain and access is an ORed AccessFlags mask.

Return type:: GpuMR

Enums#

Pythonic enums mirroring the libibverbs / libefa ABI constants.

These values are part of the stable rdma-core ABI. They are declared here as enum.IntEnum / enum.IntFlag so they interoperate with plain ints everywhere the low-level API accepts a flag or code.

class efa.enums.AccessFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Memory-region access permissions (IBV_ACCESS_*).

LOCAL_WRITE = 1#: Permit the local HCA to write the registered memory.

REMOTE_WRITE = 2#: Permit a remote peer to write the region; also requires LOCAL_WRITE.

REMOTE_READ = 4#: Permit a remote peer to read the region.

RELAXED_ORDERING = 1048576#: Allow relaxed inbound write ordering for higher performance.

class efa.enums.QPType(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

EFA queue-pair type.

UD = 4#

SRD = 255#

class efa.enums.QPState(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Queue-pair state (IBV_QPS_*).

RESET = 0#

INIT = 1#

RTR = 2#

RTS = 3#

SQD = 4#

SQE = 5#

ERR = 6#

UNKNOWN = 7#

class efa.enums.QPAttrMask(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Bitmask selecting which fields modify_qp applies (IBV_QP_*).

STATE = 1#: Apply qp_state.

CUR_STATE = 2#: Apply cur_qp_state as a transition precondition.

PKEY_INDEX = 16#: Apply the partition-key table index.

PORT = 32#: Apply the local physical port.

QKEY = 64#: Apply the datagram queue key.

RNR_RETRY = 2048#: Apply the receiver-not-ready retry count.

RQ_PSN = 4096#: Apply the receive packet sequence number.

SQ_PSN = 65536#: Apply the send packet sequence number.

class efa.enums.QPExSendOpsFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Extended-QP send operations (IBV_QP_EX_WITH_*).

Passed as QPInitAttr.send_ops_flags; the created QP can only post the opcodes enabled here.

RDMA_WRITE = 1#: Enable RDMA Write work requests.

RDMA_WRITE_WITH_IMM = 2#: Enable RDMA Write with Immediate work requests.

SEND = 4#: Enable Send work requests.

SEND_WITH_IMM = 8#: Enable Send with Immediate work requests.

RDMA_READ = 16#: Enable RDMA Read work requests.

class efa.enums.WROpcode(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Send work-request opcode (IBV_WR_*, the subset EFA supports).

RDMA_WRITE = 0#

RDMA_WRITE_WITH_IMM = 1#

SEND = 2#

SEND_WITH_IMM = 3#

RDMA_READ = 4#

class efa.enums.SendFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Send work-request flags (IBV_SEND_*).

FENCE = 1#: Fence this request behind earlier RDMA Read or atomic operations.

SIGNALED = 2#: Generate a send completion for this request.

SOLICITED = 4#: Request a solicited receive event at the peer.

INLINE = 8#: Copy the payload into the work request instead of retaining its SGEs.

class efa.enums.WCStatus(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Work-completion status (IBV_WC_*).

SUCCESS = 0#

LOC_LEN_ERR = 1#

LOC_QP_OP_ERR = 2#

LOC_EEC_OP_ERR = 3#

LOC_PROT_ERR = 4#

WR_FLUSH_ERR = 5#

MW_BIND_ERR = 6#

BAD_RESP_ERR = 7#

LOC_ACCESS_ERR = 8#

REM_INV_REQ_ERR = 9#

REM_ACCESS_ERR = 10#

REM_OP_ERR = 11#

RETRY_EXC_ERR = 12#

RNR_RETRY_EXC_ERR = 13#

LOC_RDD_VIOL_ERR = 14#

REM_INV_RD_REQ_ERR = 15#

REM_ABORT_ERR = 16#

INV_EECN_ERR = 17#

INV_EEC_STATE_ERR = 18#

FATAL_ERR = 19#

RESP_TIMEOUT_ERR = 20#

GENERAL_ERR = 21#

class efa.enums.WCOpcode(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Work-completion opcode (IBV_WC_*).

SEND = 0#

RDMA_WRITE = 1#

RDMA_READ = 2#

RECV = 128#

RECV_RDMA_WITH_IMM = 129#

class efa.enums.WCFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Work-completion flags (IBV_WC_* bit flags).

GRH = 1#: A global route header precedes the received payload.

WITH_IMM = 2#: The completion contains valid immediate data.

class efa.enums.CreateCQWCFlags(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

Extended-CQ per-completion fields (IBV_WC_EX_WITH_*).

STANDARD matches what classic ibv_poll_cq reports and is the default for create_cq_ex().

BYTE_LEN = 1#: Request the completed byte count.

IMM = 2#: Request immediate data.

QP_NUM = 4#: Request the local queue-pair number.

SRC_QP = 8#: Request the source queue-pair number.

SLID = 16#: Request the source LID.

SL = 32#: Request the service level.

DLID_PATH_BITS = 64#: Request destination LID path bits.

STANDARD = 127#: Request every field reported by a classic completion queue.

class efa.enums.EfaDeviceCaps(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntFlag

EFA device capability bits (EFADV_DEVICE_ATTR_CAPS_*).

Test against device_caps.

RDMA_READ = 1#: The provider supports SRD RDMA Read.

RNR_RETRY = 2#: The provider supports configurable RNR retry.

CQ_WITH_SGID = 4#: Extended completion queues can report sender GIDs.

RDMA_WRITE = 8#: The provider supports SRD RDMA Write.

UNSOLICITED_WRITE_RECV = 16#: QPs can receive unsolicited RDMA Write with Immediate.

CQ_WITH_EXT_MEM_DMABUF = 32#: Extended completion-queue memory can be supplied through dma-buf.

class efa.enums.MTU(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Path MTU (IBV_MTU_*).

MTU_256 = 1#

MTU_512 = 2#

MTU_1024 = 3#

MTU_2048 = 4#

MTU_4096 = 5#

class efa.enums.PortState(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Bases: IntEnum

Port state (IBV_PORT_*).

NOP = 0#

DOWN = 1#

INIT = 2#

ARMED = 3#

ACTIVE = 4#

ACTIVE_DEFER = 5#