Add support for DistributedNext (and add DistributedNext as a weakdep)

[DilumAluthge]

No description provided.

[DilumAluthge]

@JamesWrigley @jpsamaroo Does this look correct?

[DilumAluthge]

Hmmm. Is this going to be sufficient?

Because, if I look here:

SlurmClusterManager.jl/src/slurmmanager.jl

Line 182 in 11d491c

_srun_cmd_without_env = `srun -D $exehome $exename $exeflags --worker`

I see that SlurmClusterManager is starting (under srun) each worker with julia --worker. But isn't julia --worker hardcoded to always use Distributed.jl?

[JamesWrigley]

Yeah that won't work, for DistributedNext you'd want to call get_worker_arg() and interpolate that into _srun_cmd_without_env: https://github.com/JuliaParallel/DistributedNext.jl/blob/7023c603014d0e78defedd2a3d753061faeabf32/src/managers.jl#L245

I might also suggest putting Distributed support in a package extension too, that way people only need to load one or the other packages.

[DilumAluthge]

I might also suggest putting Distributed support in a package extension too, that way people only need to load one or the other packages.

Hmmm. Unfortunately, the SlurmClusterManager.SlurmManager struct subtypes Distributed.ClusterManager. It might be breaking to change that.

[DilumAluthge]

Yeah that won't work, for DistributedNext you'd want to call get_worker_arg() and interpolate that into _srun_cmd_without_env: https://github.com/JuliaParallel/DistributedNext.jl/blob/7023c603014d0e78defedd2a3d753061faeabf32/src/managers.jl#L245

If we're going to need to make a change anyway, would it be worth exposing a public interface to the user, that provides a general interface for the user to specify the worker arg? E.g. something like #74?

My motivation for #74 was that I wanted to be able to start workers with julia -e 'import DilumsPackage; DilumsPackage.entrypoint()' (instead of julia --worker). It seems that DistributedNext would be a special case of this more general feature?

[JamesWrigley]

Hmmm. Unfortunately, the SlurmClusterManager.SlurmManager struct subtypes Distributed.ClusterManager. It might be breaking to change that.

Oh I forgot that's a requirement of implementing the cluster interface. But then won't it be necessary to subtype DistributedNext.ClusterManager? 🤔 Need to think about a clean way to do that...