[RFC][Data] Add read_hive datasource for Apache Hive

Summary

This RFC proposes an alpha ray.data.read_hive() datasource for contribution to ray-project/ray.

The first implementation uses HiveServer2 (HS2) as the complete data plane. Impyla owns the HS2 client responsibilities—transport, authentication, sessions, statement execution, result fetching, and row decoding. Ray owns datasource planning and execution semantics, converts result batches to Arrow blocks, and does not directly access HMS, table locations, HDFS, object storage, manifests, or data files.

The RFC also defines a future capability-gated direct-read path as part of the target architecture. It is not part of the first HS2 implementation and will be delivered through follow-up PRs. write_hive() is intentionally outside this RFC and needs a separate write/commit design.

Context and motivation

Ray currently has generic read_sql() support but no Hive-specific datasource. Generic SQL does not define Hive table planning, the strict single-query contract, or the future capability-gated direct route.

Hive relations may be views or tables whose semantics depend on transactions, storage handlers, table-format metadata, partitions, InputFormat, SerDe properties, authorization, and server-side functions. HS2 provides the initial compatibility baseline by keeping those semantics on the Hive side.

Ray issue #51094 and PR #58253 explored a metadata-first/direct-storage route. PR #58253 was later closed after inactivity and did not merge. The HS2-only data-plane boundary is explicitly recorded in ray-hive issue #1. This RFC adopts HS2 first and keeps direct reading as a follow-up extension.

Goals and non-goals

Goals:

• Add an alpha native Ray Data read_hive() entry point.
• Provide HS2 as the strict compatibility path.
• Consume results incrementally in row batches and produce Arrow blocks.
• Execute one Hive data query for a strict read and prevent query replay after partial output.
• Target NOSASL, PLAIN/NONE, LDAP+TLS, and Kerberos/GSSAPI, with staged delivery.
• Define a direct-read seam that reuses existing Ray table/file readers.
• Fall back to HS2 when direct capability is unknown or incompatible.

Non-goals:

• Full Hadoop InputFormat or Hive SerDe compatibility.
• New physical-format readers in Hive support.
• Ray-side file parallelism for HS2 results; one HS2 operation is consumed through a sequential cursor even if Hive executes internally in parallel.
• Automatic query splitting or runtime direct-to-HS2 switching.
• Spark/Flink-level Hive compatibility.
• A Ray-owned TCLIService client.
• write_hive() in this read RFC.

Initial HS2 architecture

The driver validates the read, obtains table/view metadata through HS2 when it completely describes the requested result, and creates a serializable task specification. It does not execute a raw SQL probe to infer schema. The initial raw SQL path requires caller-supplied schema; arbitrary raw SQL schema inference is future scope. If table/view metadata is insufficient, the initial planner fails closed rather than executing a second SQL query.

Each strict read creates one Ray read task and executes one Hive data query. The worker creates one Impyla connection and cursor, obtains result metadata from that same operation, repeatedly fetches batches, converts them to Arrow blocks, and closes or cancels resources on terminal paths.

The implementation must configure the underlying Ray read task with zero task retries through the supported Ray task-options hook for the target Ray version. It must not rely on Ray’s default retry policy or enable exception retries. A failed connection, query, fetch, or decode fails the Dataset; the datasource must not append a second result stream.

Client, authentication, and schema scope

Impyla is an optional datasource dependency loaded only when Hive support is used. The exact dependency constraint is an implementation-PR decision and must be validated against Ray’s supported Python/runtime environments.

The RFC target profiles are:

NONE and NOSASL remain distinct. Ray delegates protocol and authentication mechanisms to Impyla and does not duplicate SASL/GSSAPI transports. Kerberos credentials are provisioned by the runtime environment; the datasource does not provide credential acquisition or renewal.

TLS-enabled profiles must explicitly enable certificate and server-identity verification; Ray must not inherit Impyla’s permissive TLS default.

The result schema comes from HS2 metadata. Ray only claims types, authentication profiles, Hive versions, and lifecycle behavior covered by real integration tests. Unsupported or ambiguous type mappings fail according to the accepted datasource contract rather than silently changing semantics.

Ray-owned default logs, exceptions, and metrics must not expose passwords, tokens, authentication headers, complete connection strings, SQL literals, or row values. Ray does not enable Impyla’s internal DEBUG logging.

Read semantics

• Dataset.filter() may use Ray’s normal predicate-pushdown path. The HS2 datasource pushes only predicates it can translate completely with equivalent Hive null/type semantics; otherwise the filter remains on the Ray side.
• Raw SQL is trusted caller input. Table-oriented SQL is generated only after validating table identifiers and supported typed predicates.
• A source-level limit and downstream Dataset.limit() are distinct contracts.
• Unsupported residual predicates must not be silently discarded.
• Impyla may retry transport opening before an RPC request is sent, but no second ExecuteStatement or FetchResults request may be issued after the original RPC request is sent.
• The source operation does not claim a stronger snapshot guarantee than the Hive server provides for its query.

Planned direct-read architecture

Direct mode is part of this RFC’s target architecture and is delivered through follow-up PRs:

HS2 metadata
    -> classify table format first
    -> evaluate direct capability contract
    -> select exactly one route before output

supported Iceberg/Delta/Hudi profile -> existing table-format reader
proven classic Hive profile          -> existing Ray file reader
unknown or unsupported profile       -> HS2-native read

Table format and physical file format are separate layers. Iceberg, Delta Lake, and Hudi are eligible only when the existing Ray reader covers the table’s required snapshot, transaction-log, timeline, delete, and schema semantics. They must not be downgraded to directory scans merely because their current files are Parquet or ORC.

For classic Hive tables, file extensions are not authoritative. Direct eligibility must account for InputFormat, SerDe and properties, schema, partitions, compression, split behavior, storage access, and authorization differences. TEXTFILE/CSV/JSON-compatible data is eligible only when an existing Ray reader exactly reproduces the tested Hive profile; this RFC does not add a Hadoop TextInputFormat or SerDe runtime.

The direct planner should reuse or extend Ray’s public alpha Catalog abstraction where its reader formats and credential handoff are sufficient. The capability planner remains responsible for table classification and fail-closed behavior for routes outside that abstraction.

Future route policies:

Fallback is planning-time only. Once a route starts producing data, it is fixed; a direct failure must not launch an HS2 query.

Ray integration and evolution

The first implementation uses Ray’s current V1 extension path: public Datasource/read_datasource and the DeveloperAPI ReadTask. It does not depend on ray.data._internal.datasource_v2.*. If a stable public V2 contract suitable for database and streaming sources becomes available, migration can be evaluated while preserving the architecture-level single-query, no-replay, cancellation, and Arrow-output semantics.

The implementation should reuse existing Ray readers and filesystem, projection, filter, partition, statistics, block, resource, and cancellation facilities. Hive-specific code should remain limited to the Impyla adapter, safe planning/SQL generation, HS2-to-Arrow conversion, lifecycle/error mapping, metadata normalization, and future route planning.

PR and validation boundaries

The expected contribution sequence is:

• Validate the selected Impyla release and add a small internal adapter.
• Add the HS2 datasource, public facade, incremental Arrow conversion, documentation, and NOSASL/PLAIN/Kerberos integration evidence.
• Add LDAP+TLS and HTTP profiles in separately reviewable PRs.
• Add the direct capability planner and table/file routing in later PRs.
• Design write_hive() separately.

The initial validation gate covers explicit-schema raw SQL, table/view planning, identifier validation, trusted-input boundaries, no driver-side SQL probe, empty and multi-batch results, type conversion, authentication/TLS profiles, failure/cancellation/cleanup, explicit zero-retry behavior, and absence of post-send query replay. Direct PRs additionally compare allowlisted direct and HS2 reads for schema, values, nulls, partitions, and relevant snapshot semantics.

Alternatives considered

Questions for Ray Data maintainers

• Is the HS2-first and follow-up direct-read scope appropriate for read_hive()?
• Once direct mode exists, should hs2 remain the default security boundary, or should capability-gated auto become default with an explicit hs2 override?

References

• Ray issue #51094
• Ray PR #58253
• ray-hive issue #1
• Ray DataSourceV2 RFC #61127
• Ray Datasource API
• Ray Catalog API
• Impyla
• Impyla HiveServer2 client
• Impyla transport and authentication implementation
• PyHive package status
• Apache HiveServer2 configuration
• Apache Hive DDL and storage formats