Azure Blob/ADLS Gen 2 and abfs - add instructions on using SAS token
Page
https://docs.alluxio.io/os/user/stable/en/ufs/Azure-Blob-Store.html https://docs.alluxio.io/os/user/stable/en/ufs/Azure-Data-Lake-Gen2.html
maybe also: https://docs.alluxio.io/os/user/stable/en/ufs/Azure-Data-Lake.html
Summary
Above page describes how to configure Azure Blob and Data Lake Gen2 as "Storage Integration"; they include instructions for using "Shared Key", "OAuth 2.0 Client Credentials", and "Azure Managed Identities" for authentication. However, "SAS token" is missing on both, while - in theory - this should be possible according to:
- https://learn.microsoft.com/en-us/azure/databricks/storage/azure-storage#access-azure-data-lake-storage-gen2-or-blob-storage-using-a-sas-token
- https://hadoop.apache.org/docs/stable/hadoop-azure/abfs.html
I have tried like so:
bin/alluxio fs mount \
--option fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net=SAS \
--option fs.azure.sas.token.provider.type.<storage-account>.dfs.core.windows.net=org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider \
--option fs.azure.sas.fixed.token.<storage-account>.dfs.core.windows.net="my sas token" \
/mnt/abfs abfs://<my container>@<storage-account>.dfs.core.windows.net/<my path>/and also (blob.core.windows.net instead of dfs.core.windows.net and wasb instead of abfs)
bin/alluxio fs mount \
--option fs.azure.account.auth.type.<storage-account>.blob.core.windows.net=SAS \
--option fs.azure.sas.token.provider.type.<storage-account>.blob.core.windows.net=org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider \
--option fs.azure.sas.fixed.token.<storage-account>.blob.core.windows.net="my sas token" \
/mnt/abfs wasb://<my container>@<storage-account>.blob.core.windows.net/<my path>/blob.core.windows.net/wasb result in error message after running into a timeout:
no route to host http://169.254.169.254/metadata/... this looks like it is falling back to MSI (managed identity) authentication (since 169.254.169.254 only responds from inside a VM running in Azure)
dfs.core.windows.net and abfs result in a non-descript, general error
Failed get FileSystem for abfs:I wonder if this can be done?
Source: Alluxio/alluxio