Scheduler ABC class

[Gautzilla]

👔 Job module rework

The Job module was kind of a mess, I tried to split the logic a bit to make room for the integration of other schedulers (mostly SLURM in the next days)

What changed?

Before we only had the Job module with several classes:

class	function(s)
Job	Represent a single job Write/Submit jobs to a PBS queuing system
JobConfig	Represent job/server parameters
JobBuilder	Job factory for the Public API

Now, the Job logic is spread out between the actual job things and the scheduler part:

class	function(s)
Job	Represent a single job
Scheduler (ABC): Pbs(Scheduler) Slurm(Scheduler) (to do)	Represent a single job
JobConfig	Write jobs to file Submit job files in a specific scheduler
JobBuilder	Job factory for the Public API

What changed for the user?

Every PBS-related stuff that was in the Job and JobConfig classes was moved to the scheduler. e.g.:

Public API before changes

from osekit.utils.job import JobConfig, JobBuilder

job_config = JobConfig(
    nb_nodes=1,
    ncpus=28,
    ngpus=1,
    mem="60gb",
    walltime=Timedelta(hours=5),
    venv_name=os.environ["CONDA_DEFAULT_ENV"], 
    queue="omp" # This is a matter for the scheduler, not the job itself
)

project.job_builder = JobBuilder(
    config=job_config,
)

project.run(...)

Public API after changes

from osekit.job.builder import JobBuilder # New Job package split in modules
from osekit.job.config import JobConfig # New Job package split in modules
from osekit.job.scheduler.pbs import Pbs # Scheduler info

job_config = JobConfig(
    nb_nodes=1,
    ncpus=28,
    ngpus=1,
    mem="60gb",
    walltime=Timedelta(
        hours=5
    ),
    venv_name=os.environ["CONDA_DEFAULT_ENV"],
)

scheduler = Pbs(queue="omp")  # Scheduler in which the job is submitted

project.job_builder = JobBuilder(
    config=job_config,
    scheduler=scheduler, # We specify the scheduler in the JobBuilder
)

project.run(...)

Core API

The logic feels the same with the Core API, where write() and submit() methods are now done through the scheduler rather than the job itself:

# Job and server configuration
job_config = JobConfig(
    nb_nodes=1,
    ncpus=28,
    mem="60gb",
    walltime=Timedelta(hours=1),
    venv_name=os.environ["CONDA_DEFAULT_ENV"],
)

# Scheduler configuration
scheduler = Pbs(queue="omp")

job = Job(
    script_path=Path(export_transform.__file__),
    script_args=args,
    config=job_config,
    name="test_job_core",
    output_folder=Path(...),  # Path in which the .out and .err files are written
)

# Write the job  file and submit it through the scheduler
scheduler.write(job=job, path=Path(...) / f"{job.name}.pbs")
scheduler.submit(job=job)

[coveralls]

coverage: 99.097% (+0.008%) from 99.089% — Gautzilla:scheduler-ABC into Project-OSmOSE:main

[Gautzilla]

@mathieudpnt I've converted this PR back to draft cause I'll start working on the SLURM implementation and might do some changes here during the process