[Tracking] Split #67470: route Azure data access through native Azure credentials

Search before asking

• I had searched in the issues and found no similar issues.

Description

#67470 ([fix](iceberg) Route Azure data files through native credentials) moves Azure Iceberg data access off the Hadoop ABFS compatibility path:

• FE preserves Azure account/container/object URIs instead of rewriting them to s3://container/path, and emits provider-owned native Azure credentials (AZURE_AUTH_TYPE, AZURE_ENDPOINT, AZURE_ACCOUNT_NAME, AZURE_SAS_TOKEN, ...) to BE.
• BE routes FILE_S3 ranges with provider=azure through the native Azure Blob client (SharedKey, SAS with expiry validation, OAuth2 client secret), with a separate bounded client cache and SAS-redacted diagnostics.
• Iceberg REST vended SAS credentials replace the static authentication group per request, for both the native data path (BE) and the Iceberg FileIO metadata path (FE, ADLSFileIO).
• Serialized metadata-table tasks carry a fixed credential expiry so the BE JNI scanner fails before opening an expired FileIO.

That PR touches 135 files (+13179 / -946) across BE, the FE filesystem SPI, fe-core, the Iceberg connector and the BE Java metadata scanner, which is too large to review as one change. It is being split into a series of independently reviewable PRs. This issue tracks the series; every PR in the series references this issue.

Split plan and merge order

Rules for the series:

• Consumers land before producers. BE accepts both the legacy AWS_* SharedKey map and the new AZURE_* map before FE starts emitting the new one.
• All SPI additions land in one PR with default implementations, so the plugin API version is bumped once (filesystem 1.0 -> 2.0, connector 9.0 -> 10.0); later PRs do not touch the frozen surface.
• Each PR compiles and passes its own unit tests on its own; no PR leaves half-wired behavior on master.

#	PR	Scope	Depends on
BE-1	[feature](be) Native Azure credentials, SAS expiry and client lifetime in S3ClientFactory	AzureCredentialOptions (SharedKey/SAS/OAuth2), AzureAuthFactory validation, S3ClientFactory native AZURE_* protocol + legacy AWS_* compatibility, separate expiring Azure client cache, S3URI for abfs[s]:// / wasb[s]:// / Azure HTTP(S), SAS redaction in logs	-
BE-2	[feature](be) Validate Azure credentials at reader/writer open and keep Azure display paths	access-time SAS validation in S3FileReader / S3FileSystem, Azure URI container/account checks in get_key, original URI in diagnostics	BE-1
FE-1	[feature](fe) Storage/connector SPI: access-time validation, backend view selection and request-local storage access	StorageProperties / FileSystemProperties / FileSystemProvider / ConnectorStorageContext / ConnectorScanRange default methods, ConnectorStorageAccess, ConnectorStorageAccessResolver, API version bump	-
FE-2	[fix](fe) Record the synthetic default HDFS binding at creation instead of re-guessing it	bindDefault / isSyntheticDefault, StorageAdapter.isExplicitlyConfigured, Iceberg selectEffectiveStorages drops the synthetic HDFS binding for object-storage-only catalogs	FE-1
FE-3	[fix](fe) Preserve Azure account authority in AzureUri and fix Azure listing pagination	AzureAccountHost, AzureUri literal ABFS object names + original authority, AzureFileSystem pagination fixes	-
FE-4	[feature](fe) Emit native Azure credentials (SharedKey/SAS/OAuth2) and route Azure locations by provider	AzureAuthType, AzureSasToken, AzureFileSystemProperties native AZURE_* backend map + NATIVE kind + URI preservation, fe-core StorageAdapter / LocationPath provider routing, StorageUriUtils Azure normalization removed, ObjectInfoAdapter SAS, S3SourceOffsetProvider object keys	FE-1, FE-3 (end-to-end: BE-1, BE-2)
FE-5	[feature](fe) Request-local storage bindings: vended credentials replace the static authentication group	AzureVendedSas, AzureFileIOSharedKey, bindVended, Iceberg FileIO views in the Azure provider; FileSystemPluginManager.bindVended/bindAllExcept, StorageAdapter.ofVended/ofAllExcept, DefaultConnectorContext.newStorageAccessResolver/resolveStorageProperties, PluginDrivenSplit backend file type	FE-1, FE-2, FE-4
ICE-1	[feature](iceberg) Select provider-owned Azure FileIO credentials in the REST catalog load path	IcebergRestFileIOClient / IcebergRestFileIOProperties, IcebergPostSuccessFileIOInitializationException, iceberg-azure dependency (JDK HTTP transport)	FE-1, FE-5
ICE-2	[feature](iceberg) Route Iceberg scan and write data access through request-local storage bindings	IcebergScanPlanProvider ReadStorageAccess, IcebergWritePlanProvider single resolved access per sink, IcebergStatementScope.readStorageAccess, IcebergScanRange.backendFileType	FE-5, ICE-1
ICE-3	[feature](iceberg) Carry a fixed FileIO credential expiry with serialized metadata tasks	TIcebergFileDesc.file_io_expiry_ms (optional), both BE JNI readers, IcebergSysTableJniScanner expiry check, scanner plugin Azure dependencies, IcebergMetadataTaskProperties	ICE-1 (ICE-2 first to avoid conflicts)
REG	[test](regression) Native Azure Iceberg acceptance suite	opt-in external_table_p2 suite against a pre-provisioned Azure REST catalog	all

Suggested serial order: BE-1, BE-2, FE-1, FE-3, FE-2, FE-4, FE-5, ICE-1, ICE-2, ICE-3, REG. BE-1, FE-1 and FE-3 are independent and can be opened in parallel.

Progress

• BE-1 — #68117
• BE-2
• FE-1
• FE-2
• FE-3
• FE-4
• FE-5
• ICE-1
• ICE-2
• ICE-3
• REG
• Close #67470 once the series is merged

Compatibility and rolling upgrade

• SharedKey: a new FE emits both the AZURE_* fields and the matching legacy AWS_* group, so an old BE keeps working during a rolling upgrade; a new BE verifies that the two groups agree and rejects conflicting values.
• SAS and OAuth2 have no legacy wire form. Upgrade all BEs before enabling native SAS/OAuth2 data access from a new FE. To roll back, stop native SAS/OAuth2 use first, then roll back the FE/BE pair.
• Genuine Fabric OneLake locations keep their explicit Hadoop (FILE_HDFS) path.

Carved out of the series

• The $files metadata-table projection fix (keep file_size when split_offsets is requested) that was bundled into #67470 is unrelated to Azure and will be submitted as a standalone fix.
• The two design notes under docs/ in #67470 are not merged; the acceptance fixture and rolling-upgrade notes move into the PR descriptions.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct