Huawei Ascend NPU backend for TensorPlay — a native, out-of-tree device backend that makes TensorPlay run on Ascend NPUs through the CANN runtime and the Ascend operator library.
import tensorplay as tp
import tensorplay_npu # compiles the native part on first load
x = tp.randn(64, 64, device="npu")
y = (x @ x).relu().sum()
print(y.item())TensorPlay reserves one out-of-tree device slot (PrivateUse1) for
accelerators that live outside the framework. This package fills that slot
for Huawei Ascend NPUs:
- memory comes from the Ascend runtime through a caching allocator;
- operators run as operator-library calls (single-operator
aclnnentries) on the device; - every operator without an NPU kernel yet runs on the host through the generic fallback, so models work end to end from day one;
- the device is spelled
"npu"in TensorPlay (tp.ones(3, device="npu")).
The same package ships a graph backend: tensorplay.compile(fn, backend="npu") compiles a captured region into an offline model through the
Ascend Graph Engine and runs it on the NPU — see
Graph compilation.
The native sources ship inside the wheel and are compiled against the
installed CANN toolkit the first time the backend is loaded, then cached
under ~/.cache/tensorplay_extensions.
- a Huawei Ascend NPU with the driver and firmware installed
(
npu-smi infolists the card) - the CANN toolkit (runtime, compiler and operator library — see the
Ascend documentation for your card); the
npugraph backend additionally uses the toolkit's ATC converter and its AscendCL Python bindings (import acl) - TensorPlay installed in the same environment (
pip install tensorplay)
export ASCEND_TOOLKIT_HOME=/usr/local/Ascend/cann-8.5.0 # CANN install path
export LD_LIBRARY_PATH=$ASCEND_TOOLKIT_HOME/lib64:$LD_LIBRARY_PATHgit clone https://github.com/ohtensorplay/tensorplay-npu.git
cd tensorplay-npu
pip install . # or: pip install -e .import tensorplay auto-loads the backend through the tensorplay.backends
entry point when $ASCEND_TOOLKIT_HOME holds the toolkit. You can also load
it explicitly:
import tensorplay_npu
tensorplay_npu.load()import tensorplay as tp
import tensorplay_npu
tensorplay_npu.load()
a = tp.randn(2, 3, device="npu")
print((a.abs().sum()).cpu().item()) # needs an Ascend NPUimport tensorplay as tp
import tensorplay_npu
tensorplay_npu.load()
tp.manual_seed(0)
x = tp.randn(256, 256, device="npu")
w = tp.randn(256, 256, device="npu")
y = x @ w # native matrix product
z = y.relu().softmax(-1) # activations and reductions
loss = z.sum()
loss.backward() # autograd works through the backend
print(z.cpu().shape, float(loss.cpu()))The backend implements the hot paths natively on the device:
| Area | Operators |
|---|---|
| Memory | empty, *_like, empty.memory_format, copy_ (host/device, casts, broadcasting), fill_, zero_, clone, contiguous, item |
| Binary | add/sub/mul/div with broadcasting, in-place and scalar forms, pow (tensor-scalar), rsub |
| Unary | exp, log, sqrt, rsqrt, neg, abs, relu, sigmoid, tanh, silu, gelu and their backward kernels |
| Blas | mm, addmm, bmm, matmul, linear (transposed operands read in place) |
| Reduce | sum, mean, softmax/log_softmax with backward |
| Profiling | per-operation device time, device memory events |
The device computes in float32, float16 and bfloat16. Everything else
runs on the host through cpu_fallback::register_host_fallbacks, so every
model runs; data types the device does not take (integers, float64,
bool) take the host path as well.
tensorplay.compile(fn, backend="npu") lowers a captured region through the
Ascend Graph Engine and runs it on the NPU. The region is exported to
ONNX, compiled into an offline model with the ATC converter, and cached
under ~/.cache/tensorplay-ascend. The cache key covers the graph, the
sample shapes and dtypes, the target SoC and the settings, so a rebuild only
happens when something that could change the binary changed.
import tensorplay as tp
import tensorplay_npu
tensorplay_npu.load()
def region(x, w, bias):
return tp.softmax(tp.relu(x @ w + bias), 1)
f = tp.compile(region, backend="npu")
x = tp.randn(8, 16, device="npu")
w = tp.randn(16, 32, device="npu")
b = tp.randn(32, device="npu")
print(f(x, w, b).device) # npuOptions go through options= (or as flat keywords):
| Option | Meaning |
|---|---|
precision |
"fp32" (keep the exported precision, default), "fp16" (force half) or "mix" (let the converter mix precisions) |
soc_version |
SoC the model is built for, for example "Ascend910B4"; unset reads it back off the running device |
atc |
path to the ATC converter; unset discovers it under the toolkit |
cache_dir |
directory for compiled models; None disables the disk cache |
extra_atc_args |
extra atc arguments, appended verbatim |
pass_through_build_failures |
raise on a build failure instead of running the region uncompiled |
The Graph Engine, the ATC converter and the AscendCL Python bindings are part
of the CANN toolkit. Without them the backend is reported as unavailable and
naming it in tensorplay.compile explains what to install. A region the
converter cannot build stays on its interpreter, so a coverage gap never
fails the whole compile.
benchmark/bench_npu_vs_torch.py in the TensorPlay repository runs this
backend against torch + torch_npu on the same card, one framework per
process, identical workloads. The chart below shows the ratio on an Ascend
910B4 (CANN 8.5.0, median of 50 timed iterations); values below 1.00 mean
TensorPlay is faster:
Representative numbers:
| Case | torch ms | tensorplay ms | tp/torch |
|---|---|---|---|
| mm 4096 fp32 | 2.135 | 2.102 | 0.98 |
| mm 1024 fp32 | 0.093 | 0.068 | 0.74 |
| add 4096² fp32 | 0.378 | 0.352 | 0.93 |
| relu 4096² fp32 | 0.259 | 0.236 | 0.91 |
| gelu 4096² fp32 | 0.257 | 0.240 | 0.94 |
| exp 4096² fp32 | 0.257 | 0.240 | 0.94 |
| softmax 4096² fp32 | 0.266 | 0.246 | 0.92 |
| bmm 32×512×64×512 fp32 | 0.075 | 0.054 | 0.72 |
| mm 4096 fp16 | 0.621 | 0.589 | 0.95 |
| mlp fwd b256 1024-4096 fp16 | 0.217 | 0.153 | 0.71 |
| mlp train step fp16 | 1.260 | 0.963 | 0.76 |
| h2d 64 MiB | 2.998 | 2.903 | 0.97 |
| d2h 64 MiB | 10.354 | 2.841 | 0.27 |
| mlp train step fp32 | 1.238 | 1.800 | 1.45 |
Elementwise, matrix, reduction and copy operators are at or ahead of
torch_npu; device→host copies are about 3.6× faster and the fp16 training
step is about 24% faster. The fp32 training step is the one case that still
trails. Full results are reproducible with:
cd <tensorplay-repo>
python benchmark/bench_npu_vs_torch.py --per-casefrom tensorplay.profiler import profile, ProfilerActivity
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.PrivateUse1]) as prof:
train_step()
print(prof.key_averages().table(sort_by="device_time_total", row_limit=10))Each operation gets its NPU time from an event pair recorded around it on the
device's stream; the elapsed times are read when the profile stops, so the
dispatch path never waits on the device. Device allocations show up in
prof.mem_events, and profile(use_device="npu") asks for the same device
timing. Kernel-level activity records are not collected — the per-operation
event pairs are the device timeline.
| Path | Role |
|---|---|
tensorplay_npu/csrc/AscendRuntime.* |
error checking, per-device stream, current device, byte copies |
tensorplay_npu/csrc/AscendAllocator.* |
caching allocator over aclrtMalloc, memory statistics |
tensorplay_npu/csrc/AscendHooks.cpp |
device count, synchronization, byte copies, memory info |
tensorplay_npu/csrc/AscendTensor.* |
ACL tensor descriptors, dtype and layout rules |
tensorplay_npu/csrc/AscendOpApi.* |
operator-library lookup, workspaces, execution |
tensorplay_npu/csrc/CopyKernels.cpp |
empty, *_like factories, copy_, fill_, clone |
tensorplay_npu/csrc/BinaryKernels.cpp |
add/sub/mul/div with broadcasting, in-place and scalar forms |
tensorplay_npu/csrc/UnaryKernels.cpp |
exp/log/sqrt/..., activations and their backward kernels |
tensorplay_npu/csrc/MatmulKernels.cpp |
mm, addmm, bmm, matmul, linear |
tensorplay_npu/csrc/ReduceKernels.cpp |
sum, mean, softmax/log-softmax with backward |
tensorplay_npu/csrc/NPUProfiler.cpp |
profiler device time: an event pair around each operation |
tensorplay_npu/csrc/Module.cpp |
C entry points: host fallbacks, memory statistics |
tensorplay_npu/compiler/ |
graph backend: ONNX down-lowering, the ATC build, a content-addressed model cache, NPU execution |
tests/ |
kernel-by-kernel comparison against the host |
tensorplay-npu is versioned independently of TensorPlay (semver, see
pyproject.toml). It targets the out-of-tree device slot TensorPlay
reserves for accelerators, so the two projects move at their own pace:
| Component | Version / branch |
|---|---|
tensorplay-npu |
0.1.0 (this repository, main) |
| TensorPlay | any release that ships the PrivateUse1 slot (pip install tensorplay) |
| CANN toolkit | 8.5.0 (the version this backend is tested against) |
| Tested card | Ascend 910B4 |
The native sources are compiled against the installed toolkit on first load
and cached under ~/.cache/tensorplay_extensions, so a pip install -U tensorplay-npu picks up new kernels without rebuilding TensorPlay.
The test suite compares every kernel against the host and needs an Ascend NPU with the CANN toolkit:
pip install pytest
pytest testsHow the out-of-tree slot works is documented in Adding a device backend in the TensorPlay repository.
Huawei and Ascend are trademarks of Huawei Technologies Co., Ltd. This project is not affiliated with or endorsed by Huawei.