EFA#
efa is a low-level wrapper for AWS Elastic Fabric Adapter, libibverbs, and
the EFA direct-verbs API. It supports reliable connectionless SRD traffic,
one-sided RDMA, and direct transfers involving CUDA tensors.
Setup#
The process needs libibverbs.so.1, libefa.so.1, and access to the host’s
/dev/infiniband/uverbs* devices.
import efa
device = efa.get_efa_device_list()[0]
context = device.open()
pd = context.alloc_pd()
cq = context.create_cq(256)
qp = pd.create_qp(
efa.QPInitAttr(send_cq=cq, recv_cq=cq)
).prepare(qkey=0x1234)
Exchange efa.local_endpoint_info(qp, qkey=0x1234).to_bytes() out of band.
Resolve the bytes received from the other process with
efa.EndpointInfo.from_bytes(...).peer(pd).
Configuration and feature flags#
QPInitAttr controls the send/receive CQs, queue capacities, SGE and inline
limits, service level, send signaling, enabled extended send operations, and
unsolicited-write support. Requested capacities are subject to provider and
device limits. Context.create_cq_ex independently controls completion fields,
sender-GID reporting, and unsolicited-write metadata.
The API reference documents every option and each member of
AccessFlags, QPAttrMask, QPExSendOpsFlags, SendFlags, WCFlags,
CreateCQWCFlags, and EfaDeviceCaps. Capability bits must be tested before
enabling their matching QP or CQ features.
EFA peer rule#
Each SEND names a destination address handle, QP number, and qkey. For RDMA read or write, the responder must also have created an address handle for the requester. Applications should resolve endpoint information in both directions even when data currently flows only one way.
Torch and GPUDirect#
Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before torch initializes
CUDA, then register a tensor without introducing a torch dependency into the
package:
import efa.cuda
gpu_mr = efa.cuda.register_tensor(pd, tensor, access)
Synchronize CUDA work that produced an outbound tensor before posting the
network operation. After an inbound completion or write notification, call
efa.cuda.flush_gpudirect_writes() in the destination CUDA context before
launching work that consumes the tensor.
The full installation, SRD, one-sided RDMA, and GPUDirect guide is maintained in the EFA package README.
For unsolicited RDMA write-with-immediate notifications, enable
unsolicited_write_recv on every communicating QP and unsolicited on the
extended receive CQ. EFA rejects peers that negotiate different QP feature
sets.