Skip to content

protect module method and function results while wrapping in invoke - #1494

Open
kevinushey wants to merge 2 commits into
masterfrom
bugfix/module-invoke-gc
Open

protect module method and function results while wrapping in invoke#1494
kevinushey wants to merge 2 commits into
masterfrom
bugfix/module-invoke-gc

Conversation

@kevinushey

Copy link
Copy Markdown
Contributor

Fixes #1493.

In Rcpp modules, class_<T>::invoke() passed the freshly computed method result straight into Rcpp::List::create(), which allocates the result list and the voidness flag before the result is stored anywhere the garbage collector can see. A collection inside that window frees the result while it is still in use. Module::invoke() had the same shape for module functions, with a wider window (the result list's names are allocated too). See #1493 for the analysis, including why results returned as Rcpp objects are usually (but not reliably) masked by generational promotion.

Changes:

  • module/class.h: Shield the method result in class_::invoke() before wrapping it in the result list.
  • module/Module.h: same for the function result in Module::invoke().
  • inst/tinytest/cpp/Module.cpp, inst/tinytest/test_module.R: regression test exercising a method name with mixed-voidness overloads (the dispatch path that reaches class_::invoke(), previously uncovered) under gctorture().

Verification:

Checklist

  • Code compiles correctly
  • R CMD check still passes all tests (checked locally with RunAllRcppTests=yes on R 4.6.1; --no-manual --no-vignettes, so the only flagged items were the two vignette-packaging warnings from the local --no-build-vignettes build)
  • Preferably, new tests were added which fail without the change (fails on protect-checking builds of R; on standard builds the use-after-free is silent)
  • Document the changes by file in ChangeLog

} else {
return Rcpp::List::create( false, m->operator()( XP(object), args ) ) ;
Shield<SEXP> res( m->operator()( XP(object), args ) ) ;
return Rcpp::List::create( false, static_cast<SEXP>(res) ) ;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we cast the result of Shield<SEXP> back to SEXP in other spots?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only one other spot: Environment::assign() has a dedicated Shield<SEXP> overload casting back with (SEXP) (Environment.h:193, added in 8670bb6) -- required there because the templated assign(const WRAPPABLE&) out-matches the SEXP overload for a Shield argument (deduction ignores operator SEXP()) and its implementation would otherwise recurse. Everywhere else Shield feeds plain SEXP-taking C functions, where the implicit conversion applies and no cast is needed.

Strictly, the cast here isn't required: List::create() takes const T& and wrap() has a convertible-to-SEXP fallback, so the uncast form compiles and behaves correctly (verified, including under gctorture) -- Vector::create() itself relies on that path when assigning its Shielded names via res.attr("names"). But with the cast, _["result"] instantiates the named_object<SEXP> specialization rather than a generic named_object<Shield<SEXP>>, keeping this on the standard exact-SEXP paths, so I'd prefer to keep it.

I also checked whether other module-code sites need the same protect-then-cast treatment: invoke_notvoid() and getProperty() return the raw result directly with no allocation afterward, and Module::get_function()'s create() arguments are all rooted (XPtr via the preserve list, formals is a stored Rcpp::List), so these two sites appear to be the only ones with the unprotected window.

@eddelbuettel eddelbuettel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good too, left one question inline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

module method results can be garbage collected during result wrapping in class_::invoke()

2 participants