module method results can be garbage collected during result wrapping in class_::invoke()

[kevinushey]

In Rcpp modules, class_<T>::invoke() wraps a freshly computed method result like this (inst/include/Rcpp/module/class.h):

return Rcpp::List::create( false, m->operator()( XP(object), args ) ) ;

m->operator()() returns a raw SEXP -- the wrapped result of the user's C++ method -- and nothing the garbage collector can see references it. List::create() then performs at least two allocations (the VECSXP for the result list, and Rf_ScalarLogical() for the voidness flag) before the result is stored in the (protected) list. A garbage collection inside that window frees the method result while it is still in use.

Module::invoke() has the same shape for module functions (inst/include/Rcpp/module/Module.h):

return List::create(
    _["result"] = fun->operator()( args ),
    _["void"]   = fun->is_void()
) ;

Here the window is wider still: the result list, the voidness flag, and the names attribute (a STRSXP plus two CHARSXPs) are all allocated while fun->operator()( args )'s result is unprotected.

Reachability

class_::invoke() is reached from ordinary R code whenever a module class method name has overloads with mixed voidness (some overloads void, some not): method_wrapper() in R/Module.R only routes through CppMethod__invoke -- and therefore class_::invoke() -- in that case, so that voidness can be resolved at call time. Method names whose overloads are all non-void dispatch through CppMethod__invoke_notvoid, which returns the result directly and is not affected.

Module__invoke (the entry point for Module::invoke()) is registered in rcpp_init.cpp, though Rcpp's own R code routes module functions through InternalFunction_invoke, which is not affected.

Why this usually goes unnoticed

When a method returns an Rcpp object (say a NumericVector), the result is kept on Rcpp's precious list until the method wrapper's temporaries are destroyed. Garbage collections that run while it is still protected mark the object and promote it out of generation 0, after which the (mostly level-0) collections inside the List::create() window usually skip it -- the same generational masking described in #1491. The window only bites when a deeper collection lands inside it, so on standard R builds the use-after-free is both rare and silent (the freed node's contents typically survive until the node is reused). A protect-checking build of R-devel detects it reliably. Methods returning a raw, freshly allocated SEXP are more exposed still: nothing protects the result at any point, so it is generation 0 and completely unreferenced when List::create() allocates.

Reproducible example

repro.cpp:

#include <Rcpp.h>

using namespace Rcpp;

class Gadget {
public:
    Gadget() {}

    // void overload: its presence forces dispatch through class_::invoke,
    // which wraps results as List::create(false, <method result>)
    void value(int x) {
        (void) x;
    }

    // non-void overload: returns a freshly allocated vector
    NumericVector value() {
        return NumericVector::create(1.0, 2.0, 3.0);
    }
};

RCPP_MODULE(gadget_module) {
    class_<Gadget>("Gadget")
        .constructor()
        .method("value", static_cast<void (Gadget::*)(int)>(&Gadget::value))
        .method("value", static_cast<NumericVector (Gadget::*)()>(&Gadget::value));
}

repro.R:

Rcpp::sourceCpp("repro.cpp")

g <- methods::new(Gadget)

stopifnot(identical(g$value(), c(1, 2, 3)))

gctorture(TRUE)
hit <- 0L
bad <- NULL
for (i in 1:100) {
    x <- g$value()
    if (!identical(x, c(1, 2, 3))) {
        hit <- i
        bad <- x
        break
    }
}
gctorture(FALSE)

if (hit > 0L) {
    cat("iteration", hit, "- corrupted method result!\n")
    cat("typeof:", typeof(bad), "(expected: double)\n")
    print(bad)
} else {
    cat("no corruption in 100 iterations\n")
}

On a protect-checking ASan/UBSan build of R-devel (r90339), this fails within the loop -- the freed REALSXP is the method result, caught when dealWith() in R/Module.R touches the result list:

Error in isTRUE(x[[1]]) : 
  unprotected object (0x625003d726a8) encountered (was REALSXP)
Calls: <Anonymous> -> <Anonymous> -> isTRUE -> .External
Execution halted

On a standard build of R 4.6.1 the loop runs "clean" (the freed node's contents happen to survive), which is exactly why this class of bug hides so well in practice.

I'll follow up with a PR that protects the freshly computed result in both class_::invoke() and Module::invoke().