EPIC: Organize CI with `xtask` to keep local and Github runs in sync

[2010YOUY01]

Is your feature request related to a problem or challenge?

Part of #21048
Follows-up: #24083

Describe the solution you'd like

#24083 introduces xtask to implement individual test commands, including setup steps such as environment variables. This allows both local and CI runs to use the same commands, making local reproduction easier and ensuring it is equivalent to CI.

See the issue write-up for how it works. It is essentially just a simple binary within the project; xtask is the conventional name for this pattern.

For example, the default test run command is:

datafusion/.github/workflows/rust.yml

Lines 324 to 335 in 26fbf4d

	cargo llvm-cov \
	--profile ci \
	--exclude datafusion-examples \
	--exclude ffi_example_table_provider \
	--exclude datafusion-cli \
	--workspace \
	--lib \
	--tests \
	--bins \
	--features serde,avro,json,backtrace,integration-tests,parquet_encryption,substrait \
	--codecov \
	--output-path target/codecov.json

Now both ci and local run can be executed with

# Use `cargo xtask help` for the full usage
# syntax: cargo xtask ci step [step-name] [extra-args]
> cargo xtask ci step test workspace
(...run tests)

# Use 'explain' to show the underlying command
> cargo xtask ci step test workspace --explain

cd /Users/yongting/Code/datafusion2/datafusion-ci-steps-xtask && \
cargo llvm-cov \
--profile ci \
--exclude datafusion-examples \
--exclude ffi_example_table_provider \
--exclude datafusion-cli \
--workspace \
--lib \
--tests \
--bins \
--features serde,avro,json,backtrace,integration-tests,parquet_encryption,substrait \
--codecov \
--output-path target/codecov.json

Next steps:
Using this rust workflow as reference, it is organized as:
workflow(entire file) -> job (several commands) -> step (command)
Link: https://github.com/apache/datafusion/blob/main/.github/workflows/rust.yml

• Introduce xtask orchestration for CI jobs/workflows, such as cargo xtask ci job rust, to simplify reproducing the full CI job/workflows locally.
• Fully migrate the rust.yml workflow so it is easy to reproduce locally.
• ...and then roll this out to other CI workflows.

Describe alternatives you've considered

No response

Additional context

No response