Add dialer-based RouteTaskHandler

[gabotechs]

Rewrite of the task routing API that unlocks the door for implementing retries and smarter routing.

Key changes to the task-routing API:

Fire on a per-task basis instead of per-stage

Previously, this API was fired once for each stage, and the expected returns where a list of URLs (one per task in the stage):

Original ExecutionPlan + 3 tasks -> [URL1, URL2, URL3]

Now, this is called once per task, and the passed plan is the task specialized one, not the original one.

Task-specialized ExecutionPlan + task index -> URL1 + Stablished connection

Returns a successfully established connection

There was a problem with the previous task routing API:

It prompted users and/or default implementations to return a URL, without knowing if something is listening at the other side of the URL.

There is a chance that the routed machine is unavailable, in which case, the new API gives the option to implementations to retry on a different machine.

The event handler is now async

Connections must be established in this event handler, which means that it must make network calls, therefore it's now made async.

Introduction of CoordinatorToWorkerDialer

The CoordinatorToWorkerDialer is a trait that users are not supposed to implement, instead, they are supposed to call its .dial() method for attempting to establish a connection from coordinator to remote worker, so implementations of the RouteTaskHandler would look like this:

#[async_trait]
impl RouteTaskHandler for RandomRouteTaskHandler {
    async fn handle(&self, ev: RouteTaskEvent<'_>) -> Option<Result<RouteTaskEventResponse>> {
        // get a URL somehow
        
        Some(ev.dialer.dial(url.clone()).await)
    }
}

If there's an error while dialing the worker, the user can choose to retry on another URL.

The CoordinatorToWorkerDialer under the hood will handle all the API churn for making API calls to workers, leaving a clean and narrow API to the routing task event handler implementor, just requiring the URL to which it needs to connect to.

What's not included yet

Any kind of retry mechanism on a different node in case of a connection failure.

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[gabotechs]

@JSOD11 would love some feedback about the new shape of this API.