[RFC] stop using conda for Python CI jobs

Proposal

We should stop using conda + conda-forge for Python-package CI and instead do something like:

• linting, docs, etc. --> pixi
• Python unit tests for end-of-life versions --> pixi
• all other Python build and unit test jobs --> pip

Motivation

This project uses conda + the conda-forge channel to set up environments for building and testing Python wheels.

https://github.com/lightgbm-org/LightGBM/blob/cf16013c30fa508e944f6ad6c46e5057969aa45e/.ci/test.sh#L114-L120

That has been helpful in many ways here, especially:

• strong runtime compatibility guarantees across all packages (GLIBC version, supported Python versions, etc.)
• broad platform support (including PowerPC by default, cross-ref #7345)
• ease of local development (in some ways)

But that setup has some significant drawbacks:

• elevated risk of building DSOs (.dll / .dylib / .so) that depend on conda-specific details by using conda-forge's compilation toolchain and shared libraries
  • details: https://github.com/lightgbm-org/LightGBM/pull/6566#pullrequestreview-2192846844
• elevated risk of failures from mixing conda, pip, and system-installed libraries
  • example: https://github.com/lightgbm-org/LightGBM/issues/5106#issuecomment-1121925896
  • another example: https://github.com/lightgbm-org/LightGBM/pull/7356
• CI fails to thoroughly cover the most common (I'm guessing) way lightgbm is used... by installing wheels into an environment managed with pip / uv
  • e.g., miss out on understanding which system libraries lightgbm and its dependencies expect to be installed
  • e.g., potentially miss issues from mixing manylinux wheels vendoring the same stuff, like #6596

In my opinion, those drawbacks significantly outweigh the benefits.

wheels, not conda packages, are built from this repo... they should be built and tested in environments built by installing wheels

Benefits of this work

See "Motivation" above, but in short:

• makes CI more closely match a very common workflow for users
• simplifies some details of building Python wheels (which may help with #6774 and #6828, among others)

Approach

It'd be convenient to get the CI image building finally moved into this repo (#7011, work in progress in #7109) before making other changes.

Then, remove conda from relevant places in .ci/ scripts, and do whatever's needed to get it passing again installing libraries from some mix of system package managers and pip.

Notes

Based on this conversation @borchero and I had recently: https://github.com/lightgbm-org/LightGBM/pull/7328#issuecomment-4725126251