GH-50865: [Release][CI] Install conda-gcc-specs for Linux Conda builds

[tadeja]

Rationale for this change

Fix #50865
With conda-forge compilers 2.0 builds started failing at linking various C++ executables in Linux release verification (jobs verify-rc-source-{cpp/integration/python/ruby}-linux-conda-latest-amd64) and Conda C++ CI (jobs test-conda-cpp, test-conda-python-3.11-hdfs-{2.9.2/3.2.1}).

From Aug 14 - Aug 19 conda-forge dependencies resolved as:
compilers 2.0 -> cxx-compiler 2.0 hdab9b82_0 -> gxx 15.3.0 hd47ba16_1
-> gcc 15.3.0 hb7a6aa6_1 -> depends only on gcc_impl_linux-64, so missing conda-gcc-specs.
gxx build pinned no-specs GCC variant, and even though specs GCC variant existed didn't get selected.
This specs vs. no-specs got fixed upstream in ctng-compilers-feedstock #224

What changes are included in this PR?

Instead of indirect dependency, explicitly install conda-gcc-specs for Linux
in verify-release-candidate.sh,
in conda-cpp.dockerfile, in conda-integration.dockerfile.

Are these changes tested?

Yes, locally, and affected CI jobs also build successfully now.

Are there any user-facing changes?

No.

• GitHub Issue: [Release][CI] Conda verify-rc jobs fail build with compilers 2.0 #50865

[tadeja]

@github-actions crossbow submit verify-rc-source-*-linux-conda-latest-amd64

[github-actions]

⚠️ GitHub issue #50865 has been automatically assigned in GitHub to PR creator.

[github-actions]

Revision: df55908

Submitted crossbow builds: ursacomputing/crossbow @ actions-8dbb193cad

Task	Status
verify-rc-source-cpp-linux-conda-latest-amd64
verify-rc-source-integration-linux-conda-latest-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-ruby-linux-conda-latest-amd64

[raulcd]

We should probably add it here:

arrow/ci/docker/conda-cpp.dockerfile

Line 36 in 892c73d

compilers \

and here:

arrow/ci/docker/conda-integration.dockerfile

Line 39 in 892c73d

compilers \

[raulcd]

I am wondering why would this be required now 🤔, what has changed in conda to require this

[tadeja]

@github-actions crossbow submit test-conda-cpp verify-rc-source-cpp-macos-conda-amd64

[github-actions]

Revision: a8c2636

Submitted crossbow builds: ursacomputing/crossbow @ actions-eeb859535d

Task	Status
test-conda-cpp
verify-rc-source-cpp-macos-conda-amd64

[tadeja]

From what I've found so far, the difference comes from the compiler dependency chain as gxx now pins no-specs gcc variant.
The specs/no-specs gcc variants were introduced in conda-forge/ctng-compilers-feedstock/pull/180 (that fixed conda-forge/ctng-compilers-feedstock#176).

Previously, compilers 1.11 installed conda-gcc-specs indirectly:
compilers 1.11.0
-> cxx-compiler 1.11.0 hfcd1e18_0
-> gxx 14.4.0 hd47ba16_1
-> gcc-14.4.0-hc6a0c74_1 -> depends on both conda-gcc-specs and gcc_impl_linux-64
Successful job log
+ conda-gcc-specs 14.4.0 h611768e_1
+ gcc 14.4.0 hc6a0c74_1
+ gcc_impl_linux-64 14.4.0 h9711763_1
conda-forge gcc 14.4 package

But now compilers 2.0 dependency chain resolves to the no-specs gcc variant gcc-15.3.0-hb7a6aa6_1
compilers 2.0
-> cxx-compiler 2.0 hdab9b82_0
-> gxx 15.3.0 hd47ba16_1
-> gcc 15.3.0 hb7a6aa6_1 -> depends only on gcc_impl_linux-64
Failing job log
+ gcc 15.3.0 hb7a6aa6_1
+ gcc_impl_linux-64 15.3.0 h3a305e7_1
conda-forge gcc 15.3 package

I am still checking upstream why the published gxx package changed which gcc variant it pins.
Arrow previously relied on conda-gcc-specs being installed indirectly, so this PR would make dependency explicit.

[tadeja]

@github-actions crossbow submit test-conda-cpp-valgrind test-conda-python-3.11-hdfs-*

[github-actions]

Revision: a8c2636

Submitted crossbow builds: ursacomputing/crossbow @ actions-b2b8debfe2

Task	Status
test-conda-cpp-valgrind
test-conda-python-3.11-hdfs-2.9.2
test-conda-python-3.11-hdfs-3.2.1

[raulcd]

@tadeja I am happy merging this as it fixes the initial issue. The other failures for the conda-cpp and valgrind jobs are unrelated and we can tackle them separately, what do you think?

[tadeja]

@github-actions crossbow submit verify-rc-source--linux-conda-latest-amd64 test-conda-python-3.11-hdfs- test-conda-cpp

[github-actions]

Revision: a8c2636

Submitted crossbow builds: ursacomputing/crossbow @ actions-ab59739546

Task	Status
test-conda-cpp
test-conda-python-3.11-hdfs-2.9.2
test-conda-python-3.11-hdfs-3.2.1
verify-rc-source-cpp-linux-conda-latest-amd64
verify-rc-source-integration-linux-conda-latest-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-ruby-linux-conda-latest-amd64

[tadeja]

@raulcd so ... there was an upstream fix for conda specs on gxx 20 hours ago: conda-forge/ctng-compilers-feedstock#224 (issue conda-forge/compilers-feedstock/issues/80).

I've anyway updated the PR description and re-run the jobs. Now the passing jobs and those that cancel on timeout show correct gcc variant today: gcc 15.3.0 hc6a0c74_2 which depends also on conda-gcc-specs (instead of yesterday failing no-specs gcc hb7a6aa6_1)
https://conda-metadata-app.streamlit.app/?q=conda-forge%2Flinux-64%2Fgcc-15.3.0-hc6a0c74_1.conda

We should see nightlies passing soon without this PR's explicit install of conda-gcc-specs and that would confirm we don't need changes anymore (but we could still decide to add install anyway).

For the new S3 test failures I've opened #50930