docs(rfc): add FFI plugin ABI RFC - #8207
Conversation
|
@erickguan @Xuanwo here's my first shot at a RFC. have a look when you can. |
erickguan
left a comment
erickguan
left a comment
There was a problem hiding this comment.
Let's kick off some discussions firstly!
|
|
||
| Rust has no stable ABI. A service built on its own therefore cannot load into a separately built core, because the trait object layout, the tokio types, and the shared core types do not match across separate builds. The pyo3 capsule technique hides this problem inside one Python process only, but this RFC attempts to propose a solution beyond just one binding. | ||
|
|
||
| OpenDAL ships on two release trains. `opendal-core`, `opendal-service-*`, and `opendal-layer-*` share one fast train and release together. The bindings run a slower train, so a binding may run an older core than the newest service. The two trains meet at the plugin boundary, which must stay stable across versions (see [Versioning](#versioning)). |
There was a problem hiding this comment.
We could change release versioning strategy too. At least, a binding maintainer can set a versioning strategy along with each opendal release.
| - Each asynchronous operation returns a future handle. The handle has a `poll` function and a `drop` function. | ||
| - The `poll` function has the shape `poll(handle, waker, out) -> {Ready, Pending}`. On each poll the host passes a waker that borrows the current task waker for the duration of the call and writes the result into the caller-owned `out` on `Ready`. | ||
| - A `Ready` poll clones and allocates nothing: the plugin fills `out` and returns. The plugin clones the waker into an owned form only when it returns `Pending`, so it can wake the task once the parked work makes progress. The host then polls again. | ||
| - The host wraps the future handle in a Rust `Future`, so the plugin drives both the asynchronous operator and the blocking operator through the same handle. |
There was a problem hiding this comment.
I got a partial image of how this poll should work. Maybe draw a diagram?
|
|
||
| The RFC selects the poll model over a blocking-thread model. In the blocking-thread model each plugin operation blocks and the host runs it with `spawn_blocking`, which has a smaller ABI surface but blocks one runtime worker for each operation in flight. It does not scale for a service with high concurrency, such as S3. The poll model keeps true asynchronous behavior and uses no extra threads, at the cost of more ABI surface. | ||
|
|
||
| The host owns the one tokio runtime. A plugin never starts its own runtime and never blocks inside a `poll` call, so the whole process shares that single runtime. A plugin that wraps a native asynchronous dependency bridges it to the poll model behind its own handle rather than start a second runtime. |
There was a problem hiding this comment.
There is a problem we still need to fix. The host crate and ABI is straightforward. Once we load all these symbols in memory, e.g., tokio has per-thread runtime, and opendal executor will be able to "drive" (poll) execution. Because tokio uses local thread, so a service might use its compile-time tokio definition to find this thread local tokio runtime. This escapes our design.
One route we can go for is to forbid tokio timer use for layers and services and go to use asyncband. But we need to patrol dependencies, at least once, all the way to dependencies using tokio.
| The payload never crosses the boundary as a copy. A plugin produces its bytes in an allocation it owns and transfers ownership of that allocation to the host through `AbiBuf`; the host wraps the pointer as a `Buffer` and takes on the responsibility to free it later through the plugin's own free function. A write travels the same way in reverse: the host hands the plugin a borrowed slice for the duration of the call, and the plugin either consumes it in place or, for a buffered backend, appends it to its own allocation. No step memcpies the payload to move it across the boundary. | ||
|
|
||
| This matches how `opendal-core` already handles bytes internally. `Buffer` is a reference-counted, sliceable view over one or more `Bytes`, so a compiled-in read returns a slice of the backend's buffer without a copy. The ABI preserves that property across the boundary: a plugin-backed read is also a single owned allocation surfaced as a `Buffer`, so the data path of a plugin service and a compiled-in service are the same shape. | ||
|
|
|
|
||
| The RFC selects a stable ABI with run-time loading. This is the only option that supports the full set of services, including services with native dependencies such as RocksDB, HDFS, and FoundationDB, and also supports true run-time mixing of services in one process. | ||
|
|
||
| ## Implementation of the ABI |
There was a problem hiding this comment.
A related but not present in the RFC is that a service or layer might use opendal_core::raw. What cargo and Rust could guarantee when we are loading multiple plugins? Will these symbols be consistent? Are symbols in sync with release versions? Or they will overwrite each other if we load multiple versions of compiled opendal.
2 participants
Which issue does this PR close?
None. This PR opens a new RFC for discussion.
Links to:
Rationale for this change
RFC 6828 split the Rust core into
opendal-core, oneopendal-service-*crate per backend, and oneopendal-layer-*crate per layer. This split solved the Rust compile-time footprint. It did not solve the binding footprint. Each binding still selects services and layers with Cargo features, and Cargo links every selected service into one static artifact, so a binding ships one large native library and a user who needs one service still gets all of them.Rust has no stable ABI, so a service built on its own cannot load into a separately built core. This RFC proposes a stable ABI and a host crate that loads a service or a layer (together, a
plugin) from a separate native library at run time and registers it into theOperatorRegistry. A plugin-backed operator is an ordinaryOperator: it composes with other operators and with layers, and it moves payload bytes without a copy. The mechanism lives once in the core host, so every language binding uses it without per-binding code.What changes are included in this PR?
core/core/src/docs/rfcs/8207_ffi_plugin_abi.md, the RFC text.core/core/src/docs/rfcs/mod.rsso it renders in the rustdoc RFC index.This PR is documentation only. It changes no code, no public API, and no behavior.
Are there any user-facing changes?
No runtime or API changes. The only user-facing effect is a new RFC document in the rendered rustdoc RFC list.
AI Usage Statement
claude opus assisted this PR with grammar, external research, the ASCII diagrams, and tighten the language. It also cross-checked the
opendal-corereferences the RFC relies on against the current source, includingServicer,ServiceDyn, theServicetrait and its operation set,Operator::from_parts,ServiceInfo,Buffer, and the registry factory type.