Expand use of sql parsing string expressions in DataFrame

[timsaucer]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

This is a follow on to #1273

There are a number of places where it would be convenient to pass SQL strings as expressions. For example, it would be nice to do

df.select(
    "a",
    "a - b",
    col("c"),
)

This should intuitively know that we are getting column a, followed by col("a") - col("b") followed by column 'c'.

Describe the solution you'd like

Using the sql parsing on the DataFrame make the following functions handle SQL strings. We must be very careful that we do not break things like cases where users have a column name that is not SQL parseable.

Describe alternatives you've considered

Status quo

Additional context

DataFrame functions to update:

• select
• remove select_exprs
• with_column feat: with_column supports SQL expression #1284
• with_columns feat: Add SQL expression support for with_columns #1286
• aggregate
• repartition_by_hash feat: Add SQL expression for repartition_by_hash #1285

We do not want to apply this treatment to joins because there is no easy way to know which DataFrame to perform the SQL parsing against.

[timsaucer]

cc @milenkovicm @K-dash

[timsaucer]

I think when we have a string we want to first check to see if it is a column name. If so, create a column expression. If not, try sql parsing.

[milenkovicm]

Have you changed your mind about select Tim? I'm stuck next few days but may help if needed

[timsaucer]

I do think we want select but it just can't be the snippet I shared before. Instead I think it needs to check if the string matches a column name first. If so, make a column expression. Otherwise try parsing. What do you think?

[milenkovicm]

I have no strong opinion Tim, I'm split between keeping it as it is and change it

[milenkovicm]

i hope to tick off few of them off the list this weekend

[milenkovicm]

looking at spark it looks like we're aligned with select and selectExpr maybe we keep both as they are ?

[timsaucer]

I can see how it might be preferable to some to have it split out, but I also feel like just having one select is very ergonomic. Asking for another opinion from some users I know @ntjohnson1 @MrPowers

[milenkovicm]

if we do not have special handle for column names I would agree with you, one select may make more sense. If we have to check for column name before we parse_sql I believe it would be simpler from user perspective to leave them as they are. Anyway leats get consensus

[ntjohnson1]

I definitely think a single select would be nice ergonomically.

[milenkovicm]

aggregate may have the same issue like select as it expects column name as a string, I'll leave those two to you @timsaucer