proc_macro support for reading files/strings to spanned TokenStream

[mystor]

Currently, as far as I am aware, there is no way to get spans for tokens parsed from a source other than the macro's callsite, and no good way to tell rustc & cargo a procedural macro depends on another file. It would be nice for macros which want to load other files to enable this.

It'd be nice to support some mechanism for loading a file as a TokenStream. Ideally this would take the form of a method in the crate directly parsing a file / string to a TokenStream. For example:

1. A function to load a file as a TokenStream with accurate span info etc, e.g.

   fn lex_file(path: &Path) -> io::Result<TokenStream>;

2. A function to parse a string with a given filename. This might need an additional mechanism to register the source file as a dependency, e.g.

   fn lex_str(src: &str, filename: &Path) -> Result<TokenStream, LexErr>;
   fn add_path_dependency(path: &Path) -> ...;

To make it generally useful & able to implement something like include!() we'd probably also want/need something like #54725 to get the file to load relative to.

[mystor]

CC @alexcrichton @dtolnay

[alexcrichton]

Agreed this would be a good idea to have! I don't know quite the shape that it should take, but it's something we'll want eventually

[TommasoBianchi]

I think that implementing a proc_macro_include system would fit very well with the procedural macro system we have right now, and it will allow to implement macros like include!() without resorting to compiler built-in code (plus any other similar utilities). I think that the signature might look something like:

#[proc_macro_include]
fn include_file(path: &Path) -> TokenStream

Erros (for example, file not existing) would be handled by panicking in the same way as they are handled for standard proc_macro.

I'm fairly new to this repo, but as far as I have understood there should be nothing significant holding this back.

[oddcoder]

Is there any progress done on this one?
This is issue is currently roadblocking some work I am doing where I needed exactly that, So if there is anyway I can contribute to make this issue resolved I am happy to help

[TommasoBianchi]

Hi @oddcoder, sorry for the late answer but lately my work kept me fairly busy and pretty far from Rust, unfortunately. Feel fre to pick up this issue if you want to, I did never manage to reach anything useful.

[oddcoder]

@rustbot claim

[Alexendoo]

Triage: Hi, are you still working on this issue @oddcoder?

[oddcoder]

ops sorry, I got stuck at some point, then got carried away with my still ongoing exams.

[Alexendoo]

No worries, good luck with the exams. Please claim the issue again if you want to work on it in the future

@rustbot release-assignment

[DaAitch]

It would be great if it's possible to create a spanned TokenStream from a file.
I think there are lots of use cases where it makes sense like e.g. libraries which parse different types of files at compile time, generating code and you want to point to a syntax error in the file which would be huge for DX.

Would this be something worth to contribute or are there other experimental concepts which will make this obsolete?
Something like what @mystor suggested?

fn lex_file(path: &Path) -> io::Result<TokenStream>;

[CAD97]

rust-lang/rfcs#3200 aims to address this, and as the author of the RFC I obviously believe that API to be a near-optimal shape.

(The RFC isn't crosslinked yet, so adding such.)