Spark: Flaky TestViews.readFromViewReferencingTempFunction fails with ROUTINE_ALREADY_EXISTS due to colliding random temp function names

Apache Iceberg version

main (development)

Query engine

Spark

Please describe the bug

TestViews.readFromViewReferencingTempFunction failed on the Spark 4.2 extensions tests in a main push build:

https://github.com/apache/iceberg/actions/runs/34809521107/job/103867881304

TestViews > readFromViewReferencingTempFunction() > catalogName = spark_hive_with_views, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, default-namespace=default, cache-enabled=false} FAILED
    org.apache.spark.sql.catalyst.analysis.FunctionAlreadyExistsException: [ROUTINE_ALREADY_EXISTS] Cannot create the routine `test_avg919406` because a routine of that name already exists.
    Choose a different name, drop or replace the existing routine, or add the IF NOT EXISTS clause to tolerate a pre-existing routine. SQLSTATE: 42723
        at app//org.apache.spark.sql.errors.QueryCompilationErrors$.functionAlreadyExistsError(QueryCompilationErrors.scala:1309)
        at app//org.apache.spark.sql.execution.command.CreateFunctionCommand.run(functions.scala:68)
        ...
        at app//org.apache.iceberg.spark.TestBase.sql(TestBase.java:131)
        at app//org.apache.iceberg.spark.extensions.TestViews.readFromViewReferencingTempFunction(TestViews.java:614)

Likely cause

The temp function name is generated with viewName("test_avg"), which appends new Random().nextInt(1000000). The temp function is never dropped.

TestViews runs each test once per catalog parameter (spark_with_views, spark_catalog, spark_hive_with_views) against the same SparkSession. The temp functions created by the earlier parameter runs therefore stay registered. When the random suffix repeats, CREATE TEMPORARY FUNCTION fails.

Spark 4.2's CreateFunctionCommand throws this error only when SessionCatalog.isRegisteredFunction finds the unqualified name in the session's temp function registry. It does not check the metastore, so the name must have been registered earlier in the same session. The collision is rare, which makes the test flaky.

The same pattern (random suffix, no DROP TEMPORARY FUNCTION) exists in readFromViewReferencingTempFunction, createViewReferencingTempFunction, and createViewReferencingQualifiedTempFunction. It is present in the Spark 3.5, 4.0, 4.1 and 4.2 versions of TestViews.

Possible fix

• Drop the temp function in a finally block (DROP TEMPORARY FUNCTION IF EXISTS ...), as createViewReferencingTemporaryVariable already does for its temporary variable; and/or
• generate collision-free names in viewName() (e.g. an incrementing counter or a UUID-based suffix) instead of new Random().nextInt(1000000).

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time