[draft] Master catalog spi 11 hive#65474
Open
morningman wants to merge 252 commits into
Open
Conversation
…pache#63641) ## Summary P1 batch A — close out scan-node SPI consolidation while keeping migration-period fallbacks in place. Three surgical changes route `PluginDrivenExternalTable` first in the nereids translator hot paths so already-migrated SPI connectors (JDBC, ES) take the SPI route, while the existing `instanceof XExternalTable` chains remain as fallbacks for connectors still pending migration (P3–P7). - **T3** — `PhysicalPlanTranslator.visitPhysicalFileScan`: move the existing `PluginDrivenExternalTable` branch from position 8 to position 1; the 7 connector-specific branches (HMS / Iceberg / Paimon / Trino / MaxCompute / LakeSoul / RemoteDoris) stay in place as migration-period fallbacks - **T4** — `PhysicalPlanTranslator.visitPhysicalHudiScan`: add a `PluginDrivenExternalTable` branch routed to `PluginDrivenScanNode.create(...)`, threading `tableSnapshot` + `scanParams` through `FileQueryScanNode` setters; `incrementalRelation` flagged as a P3 Hudi SPI extension TODO. The new branch is unreachable today (`PhysicalHudiScan` is only built for `HMSExternalTable + DLAType.HUDI`), so this is groundwork for P3 with zero current-day runtime impact - **T5** — `LogicalFileScan`: in `computeOutput()`, add a `PluginDrivenExternalTable` branch calling new helper `computePluginDrivenOutput()` — same shape as `computeIcebergOutput`, using `getFullSchema()` + virtualColumns; in `supportPruneNestedColumn()`, add an explicit `PluginDrivenExternalTable → false` branch. Both behaviorally equivalent for JDBC/ES today since they have no hidden cols and no virtualColumns P1 batch B (T1 — delete 13 legacy `Jdbc*Client` + `JdbcFieldSchema`) is deferred to P8 because the 3 fe-core callers — `PostgresResourceValidator`, `StreamingJobUtils`, `CdcStreamTableValuedFunction` — are live CDC streaming code that requires SPI extension for `getPrimaryKeys` / `getColumnsFromJdbc` / `listTables`, which is out of P1 surgical scope. Background and tracking docs live in `plan-doc/` (Master Plan §3.2 P1, tasks/P1-scan-node-cleanup.md, decisions log). ## Test plan - [x] `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` → BUILD SUCCESS - [x] `mvn -pl fe-core checkstyle:check` → 0 violations - [x] JDBC + ES regression-test passing — baseline established in P0 / PR apache#63582 - [ ] PR CI green on this PR - [ ] Manual scan-node smoke for an SPI connector — JDBC `SELECT *` should fall into the new `PluginDrivenExternalTable` branch first 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…apache#64096) ### What problem does this PR solve? Related PR: apache#63582 (P0 — SPI baseline), apache#63641 (P1 — nereids plugin-driven routing) Problem Summary: This is **P2** of the catalog SPI migration and targets the `branch-catalog-spi` feature branch (continuing P0 apache#63582 and P1 apache#63641). It fully migrates `trino-connector` off the legacy in-tree `fe-core/datasource/trinoconnector/` implementation and onto the connector SPI module `fe-connector-trino`, making `trino-connector` the first connector to complete the SPI consumption playbook that later connectors will reuse as a template. All five batches land together so there is no intermediate state where a newly-created trino catalog cannot be serialized. **Batch A — complete the SPI surface (`fe-connector-trino` only, no fe-core changes)** - `TrinoConnectorProvider.validateProperties`: enforce the required `trino.connector.name` property at `CREATE CATALOG` time (ported from the legacy `checkProperties`). - `TrinoDorisConnector.preCreateValidation`: call `ensureInitialized()` so plugin loading + connector-factory resolution happen at catalog creation instead of being deferred to the first `SELECT`. - `TrinoConnectorDorisMetadata.applyFilter` / `applyProjection`: bridge Trino native filter/projection pushdown, reusing `TrinoPredicateConverter` to translate a Doris `ConnectorExpression` into a Trino `TupleDomain`. `remainingFilter` is conservatively returned as the original expression to match legacy behavior (conjuncts are not stripped; BE re-evaluates them). **Batch B — fe-core bridge for image compatibility** - `GsonUtils`: atomically replace the three legacy `registerSubtype` entries (`TrinoConnectorExternalCatalog` / `Database` / `Table`) with `registerCompatibleSubtype` redirects onto the `PluginDrivenExternal*` hierarchy. This must be atomic — `RuntimeTypeAdapterFactory` rejects duplicate labels, so keeping both bindings would throw at static init. Mirrors what ES/JDBC already did. - `PluginDrivenExternalCatalog.gsonPostProcess`: extract a `legacyLogTypeToCatalogType()` helper that maps `Type.TRINO_CONNECTOR` → `"trino-connector"`; the generic `name().toLowerCase()` would otherwise produce the wrong `"trino_connector"` (underscore) that `CatalogFactory` does not recognize. - `PluginDrivenExternalTable.getEngine()` / `getEngineTableTypeName()`: add `trino-connector` branches that preserve the legacy engine-name / table-type display across `SHOW TABLE STATUS` and `information_schema`. **Batch C — flip the switch** - Add `"trino-connector"` to `CatalogFactory.SPI_READY_TYPES` so catalog creation routes through the SPI path. **Batch D — remove legacy code** - Drop the `instanceof TrinoConnectorExternalTable` scan branch in `PhysicalPlanTranslator` (the `PluginDrivenExternalTable` SPI branch already handles it). - Drop `case "trino-connector"` in `CatalogFactory`. - Delete `fe-core/datasource/trinoconnector/` (10 files) and the now-dead legacy `TrinoConnectorPredicateTest`. - Route the `TRINO_CONNECTOR` db-build case in `ExternalCatalog` to `PluginDrivenExternalDatabase` (mirrors the migrated JDBC case). - **Retained for image compatibility**: the `InitCatalogLog.Type.TRINO_CONNECTOR` and `TableType.TRINO_CONNECTOR_EXTERNAL_TABLE` enums, the GsonUtils redirects, and the `MetastoreProperties` trino-connector entry. **Batch E — tests + tracking docs** - 29 JUnit 5 unit tests over the plugin-free converters: - `TrinoPredicateConverterTest` — `ConnectorExpression` pushdown trees → Trino `TupleDomain` (EQ / range / NE / IN / IS [NOT] NULL / AND / OR, Slice encoding), plus graceful degradation to `TupleDomain.all()` on null/unsupported input. - `TrinoTypeMappingTest` — Trino SPI type → Doris `ConnectorType` (scalars, decimal precision/scale, timestamp precision clamp, array/map/struct, unsupported-type failure). - `TrinoConnectorProviderTest` — `validateProperties` fast-fails when `trino.connector.name` is missing/empty. - No Trino plugin/cluster required; plugin-dependent paths remain covered by the existing `external_table_p0/p2` `trino_connector` regression suites. - Sync the migration tracking docs under `plan-doc/` (already carried on this feature branch since P0). **Net effect**: 28 files, +1025 / −2681 (~1656 LOC net removed). Old FE images holding legacy trino catalogs / databases / tables deserialize onto the `PluginDrivenExternal*` hierarchy through the GsonUtils string-name redirect, with engine-name display preserved. **Deferred (follow-ups, not in this PR)**: - `trino_connector_migration_compat` regression test (old-image deserialization) — requires a running cluster + Trino plugin + docker, unavailable in this dev environment; tracked as a CI/cluster follow-up. - The plugin-install documentation update lives in the `doris-website` repo and is handled separately. ### Release note None ### Check List (For Author) - Test - [x] Unit Test — 29 new tests in `fe-connector-trino` (predicate converter / type mapping / property validation). - [ ] Regression test — existing `trino_connector` suites cover plugin paths; the new old-image compat regression is deferred to a CI/cluster follow-up. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. Internal routing moves from the legacy fe-core path to the SPI path; image compatibility, engine-name display, and pushdown semantics all mirror the legacy behavior. All batches land together, so there is no serialization-gap window. - Does this need documentation? - [x] Yes. The trino-connector plugin-install doc update is a follow-up in the `doris-website` repo. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
…tch design (hybrid, T02-T08) (apache#64143) ## Proposed changes testing with apache#64146 P3 of the catalog-SPI migration (base: `branch-catalog-spi`). Migrates the **hudi** connector following the **hybrid** strategy (D-019): harden the dormant HMS-over-SPI hudi connector to correctness parity, build a test baseline, and write the per-table dispatch design — **all behind the closed gate** (`SPI_READY_TYPES` unchanged). >⚠️ **No user-visible behavior change.** The SPI hudi path stays dormant (gate closed); hudi queries continue to use the legacy `HMSExternalTable.dlaType=HUDI` path. This PR removes correctness blockers ahead of the live cutover (deferred to P7 / batch E). ### What's included **Correctness fixes (hardening dormant code, behind gate):** - **T02** — fix hudi JNI `column_types` double bug: emit full Hive type strings (was Doris bare type names, losing precision/scale/subtypes) and send `column_names`/`column_types`/`delta_logs` as typed lists end-to-end (was comma join/split, which shattered `decimal(10,2)` / `struct<...>`). Matches the BE `hudi_jni_reader.cpp` contract (names `,` / types `#` / delta `,`). - **T04** — fail loud on time-travel / incremental read in the SPI `visitPhysicalHudiScan` branch (was silently returning the latest snapshot / silently full-scanning). - **T05** — real EQ/IN partition pruning in `HudiConnectorMetadata.applyFilter` (was a placeholder that ignored predicates and unconditionally switched the partition source from Hudi-metadata to HMS); faithfully mirrors `HiveConnectorMetadata.applyFilter`. - **T07** — column-name casing fix in `avroSchemaToColumns` (top-level lowercase, mirroring legacy `HMSExternalTable`). **Test baseline (all three connector modules started P3 with 0 tests):** - `fe-connector-hudi` (33): type-mapping / schema-parity (COW/MOR golden) / table-type / partition-pruning / scan-range. - `fe-connector-hms` (12): shared Hive-type-string parser tests. - `fe-connector-hive` (14): file-format / partition-pruning (mirrors T05). - COW/MOR schema is **type-agnostic** (golden parity vs legacy `initHudiSchema`); table type only affects scan planning. **Decisions / design (code-grounded, design-only):** - **T03** — defer `schema_id`/`history_schema_info` field-id evolution to batch E (DV-006; not a model-agnostic SPI fix). - **T06** — keep MVCC/snapshot SPI defaults (opt-out) + document (DV-007). - **T08** — `tableFormatType` dispatch design memo + **D-020**: single `hms` catalog per-table routing via a new backward-compatible `ConnectorMetadata.getScanPlanProvider(handle)` (per-table provider seam); refines D-005. The keystone gap is split into M1 (identity consumption, fe-core reads `tableFormatType` as an opaque string) and M2 (scan routing). ### Deferred to batch E / P7 (not in this PR) Gate flip (`SPI_READY_TYPES += hms/hudi`), fe-core `tableFormatType` consumption (M1+M2 implementation), live cutover, delete legacy `datasource/hudi/`, full incremental/time-travel/MVCC, Iceberg-on-hms via SPI (needs P6 `IcebergScanPlanProvider`), cluster/runtime validation. ### Verification Per task tracking, each code batch landed with: per-module compile + checkstyle 0 (incl. test sources) + connector import-gate pass + new unit tests green. The two most recent commits are docs-only (`plan-doc/`); the code is unchanged since the last green batch. Gate stays closed → the dormant SPI path is unreachable at runtime → zero live-path risk. CI re-verifies. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…core + make fe-core odps-free (T07-T09) (apache#64300) Follow-up to apache#64253 (the MaxCompute catalog-SPI cutover). After the cutover a `max_compute` catalog deserializes to `PluginDrivenExternalCatalog` and no legacy `MaxComputeExternal*` object is ever instantiated, so the legacy MaxCompute subsystem in fe-core is dead code. This removes it and makes fe-core's dependency tree fully odps-free. **1. Remove legacy subsystem** (`7a4db351100`) - Delete 20 fe-core files: `datasource/maxcompute/*` (incl. `MCTransaction`, `MaxComputeScanNode`/`Split`), the MaxCompute sink/insert/txn plumbing, and 2 legacy-only tests. - Clean ~21 reverse-reference sites (imports + dead `instanceof`/visitor/rule branches), keeping every `PluginDriven`/connector sibling branch and the image/replay keep-set (GsonUtils compat strings; `TableType`/`TransactionType`/`TableFormatType`/`InitCatalogLog.Type` `MAX_COMPUTE` enums; block-id thrift). - Rewire 3 tests; e.g. `FrontendServiceImplTest`'s block-id RPC test now mocks the generic `Transaction` SPI, since `getMaxComputeBlockIdRange` reads the PluginDriven connector transaction. **2. Make fe-core odps-free** (`409300a75b8`) - Drop the two odps deps from `fe-core/pom.xml`. - Move `MCUtils` from fe-common into `be-java-extensions/max-compute-connector` (its only consumer after the removal); keep `MCProperties` (odps-free constants) in fe-common. - Drop `odps-sdk-core` from fe-common — it was also leaking netty/protobuf transitively to fe-common's own `DorisHttpException`/`GsonUtilsBase`, so declare `netty-all` + `protobuf-java` directly (proper dependency hygiene). **3. Doc-sync** (`f8c305765e8`) — plan-doc PROGRESS/HANDOFF/deviations/design tracking notes. - `mvn -pl :fe-core -am test-compile` (main+test) passes; checkstyle 0 violations; connector import-gate passes. - `grep -rn com.aliyun.odps fe/fe-core/src` → empty. - `mvn -pl :fe-core dependency:tree | grep odps` → empty (no odps, direct or transitive). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…he#64446) (apache#64446) Migrates the legacy in-tree Paimon catalog (fe-core `datasource/paimon`, `metacache/paimon`, `property/metastore/*Paimon*` and their reverse references) onto the catalog SPI as a self-contained `fe-connector-paimon` plugin, following the maxcompute full-adopter + cutover template. `paimon` is added to `SPI_READY_TYPES`, so a Paimon catalog now deserializes to `PluginDrivenExternalCatalog` and all five functional areas — normal-table read, system-table read, DDL, MTMV/MVCC/time-travel (procedures are a doc-only no-op: fe-core has none) — flow through the generic `PluginDriven*` bridge with no Paimon-specific branch in fe-core. Paimon is the first lakehouse full-adopter to exercise the MVCC / sys-table / MTMV SPI surfaces, which become the reusable template for the upcoming iceberg/hudi migrations, so the cutover is held to byte-parity with the legacy path. The legacy `datasource/paimon/*` classes are left in place (now dead on the cutover path) and removed in a follow-up, mirroring the maxcompute cutover -> P4 removal. DefaultConnectorContext) Source-agnostic seams so any connector can express these capabilities; every new method is a default no-op/identity, so hive/iceberg/jdbc/maxcompute/ trino/es are byte-for-byte unaffected. - MVCC/time-travel: `resolveTimeTravel(session, handle, ConnectorTimeTravelSpec)` + `applySnapshot(...)` (threads a pinned snapshot into the handle before planScan); new immutable `ConnectorTimeTravelSpec` (SNAPSHOT_ID / TIMESTAMP / TAG / BRANCH / INCREMENTAL) that fe-core extracts from SQL; `ConnectorMvccSnapshot` carries `schemaId` for schema-evolution-aware time travel. - System tables: `listSupportedSysTables` / `getSysTableHandle`; fe-core composes the `{base}${sys}` reference name. - Scan: proportional split-weight (`getSelfSplitWeight`/`getTargetSplitSize`), COUNT(*) pushdown (`getPushDownRowCount`, 7-arg `planScan`), native-read accounting (`isNativeReadRange`), `getDeleteFiles` (VERBOSE merge-on-read), `ignorePartitionPruneShortCircuit` (predicate-driven connectors keep genuine-null partitions). - Metadata: `ConnectorPartitionInfo.fileCount` + `SUPPORTS_PARTITION_STATS` (rich SHOW PARTITIONS); `ConnectorColumn.withTimeZone` (WITH_TIMEZONE DESC marker, independent of the timestamp_tz mapping flag); connector cache control (`invalidateTable`/`invalidateAll`, `schemaCacheTtlSecondOverride`). - `ConnectorContext`: six engine-side hooks the connector must not implement itself (`loadHiveConfResources`, `vendStorageCredentials`, `getBackendStorageProperties`, `normalizeStorageUri` x2, `getStorageProperties`), bound in `DefaultConnectorContext` over fe-core's single-source-of-truth credential/URI/HiveConf utilities (no re-ported logic that could drift). The metastore-side counterpart of the fe-filesystem StorageProperties SPI: a Paimon catalog resolves its backend (hive/dlf/rest/jdbc/filesystem) by ServiceLoader dispatch on `paimon.catalog.type` instead of a fe-core switch. - `fe-connector-metastore-api`: hadoop/SDK-free `MetaStoreProperties` contracts (HMS/DLF/REST/JDBC/FileSystem) carrying only Map/scalar facts. - `fe-connector-metastore-spi`: `MetaStoreProvider<P>` + first-hit `MetaStoreProviders` dispatcher + five impls, registered via META-INF/services. `HmsMetaStorePropertiesImpl` reproduces the legacy `buildHmsHiveConf` key set and parity-critical ordering (storage overlay before the kerberos block; username/sasl last). Adding a backend = one provider class + one services line (no central enum/switch). - ServiceLoader is loaded with the SPI interface's own (plugin) classloader, NOT the thread-context CL: at CREATE CATALOG the static init first fires on an FE worker thread whose TCCL is the FE app loader, so a 1-arg load would cache an empty provider list process-wide. - fe-core bridges add `initExecutionAuthenticator(List<StorageProperties>)` so HDFS-Kerberos doAs is wired on the plugin path (legacy `initializeCatalog` is dead). Full read + DDL + partitions + statistics + system tables + MVCC/time-travel across all five metastore flavors, planning native (ORC/Parquet) vs JNI reads. - `PaimonCatalogFactory` (pure flavor switch: Options + Hadoop Configuration + HiveConf, classloader-pinned for child-first loading), `PaimonCatalogOps` (injection seam over the remote Catalog for offline-testable metadata), `PaimonConnectorMetadata` (~9 -> 28 methods: schema latest/at-snapshot, sys tables, MVCC/time-travel, DDL, partitions, statistics, table descriptor). - `PaimonScanPlanProvider` (~6 -> 30 methods): predicate/projection pushdown, native-vs-JNI routing, large-file sub-splitting with deletion vectors, COUNT(*) collapse, schema-evolution field-id dictionary, `$ro` unwrap to base FileStoreTable. - New: `PaimonSchemaBuilder` (CREATE TABLE), `PaimonIncrementalScanParams` (@incr), `PaimonTableResolver` (sys/branch-aware handle -> Table), `PaimonLatestSnapshotCache` + `PaimonSchemaAtMemo` (per-catalog caches on the long-lived connector). - `PaimonTypeMapping` gains the write direction (Doris -> Paimon) for CREATE TABLE. - NEW `PluginDrivenMvccExternalTable` (generic MvccTable + MTMV host: the runtime class for paimon tables), `PluginDrivenMvccSnapshot`, `PluginDrivenSysExternalTable` + `systable/PluginDrivenSysTable`. Behavior is selected by `ConnectorCapability` (SUPPORTS_MVCC_SNAPSHOT, SUPPORTS_PARTITION_STATS) — no paimon branch in fe-core. - `PluginDrivenScanNode`: MVCC pin threaded at every handle-consumption site, sys-table time-travel guard, COUNT pushdown, connector-agnostic EXPLAIN re-emission. - Cutover: `CatalogFactory` adds `paimon` to `SPI_READY_TYPES` and drops the built-in case; `GsonUtils` removes the 7 built-in `Paimon*` subtype registrations and adds replay compat so persisted `PaimonExternalCatalog` (all 5 flavors) / `...Database` -> `PluginDriven*` and `PaimonExternalTable` -> `PluginDrivenMvccExternalTable` on deserialization (existing clusters upgrade without losing catalogs). - Nereids/DDL: `PhysicalPlanTranslator` drops the `PaimonScanNode` branch; `UserAuthentication`, `CreateTableInfo`, `ShowPartitionsCommand` generalized; `Env.getDdlStmt` renders LOCATION/PROPERTIES for SHOW CREATE (paimon engine, no credential leak). - fe-filesystem: NEW `HdfsFileSystemProperties` / `HdfsConfigFileLoader` (typed HDFS; defaults-laden BE map vs defaults-free FE Hadoop-config map so it cannot clobber a co-bound object store's fs.s3a.* during a multi-backend merge); S3 MinIO support + legacy tuning defaults; OSS/OBS/COS ANONYMOUS provider-type; new `FileSystemFactory.bindAllStorageProperties` / `FileSystemPluginManager.bindAll`. - NEW `fe-kerberos` leaf module: dependency-free `AuthType` / `KerberosAuthSpec` facts shared across modules without pulling in Hadoop. - Packaging: NEW `fe-connector-paimon-hive-shade` relocates `org.apache.thrift` for HMS 2.3.7; the paimon plugin-zip excludes fe-thrift/libthrift (provided parent-first) to avoid a split `TBase` across loaders; self-contained bundle (hadoop-aws, AWS-SDK, hadoop-hdfs-client). `paimon.version` (1.3.1) pinned as the single property across FE connector / BE paimon-scanner / preload-extensions for FE<->BE Table/Split serde. - BE `PaimonJniScanner.getPredicates()` null-predicate backstop (no-filter scan; pairs with the FE producer now always emitting an empty predicate list). Parity fixes surfaced by the SPI cutover (each restores legacy semantics): - Storage/credentials: canonical `s3.*`/`oss.*` keys -> `fs.s3a.*`/`fs.oss.*` (STORAGE-CREDS); canonical `AWS_*` BE creds (STATIC-CREDS-BE); per-table vended DLF/STS token (REST-VENDED); `oss/cos/obs/s3a` -> `s3://` for data + deletion-vector paths (URI-NORMALIZE); external `hive.conf.resources` hive-site.xml (HMS-CONFRES); real doAs on filesystem/jdbc + wrapped HMS read RPCs (KERBEROS-DOAS). - Reads/types: paimon columns forced nullable (READ-NOTNULL); `VARCHAR(65533)` not widened to STRING (VARCHAR-BOUNDARY); dotted `enable.mapping.varbinary` / `.timestamp_tz` keys (MAPPING-FLAG-KEYS); per-type partition-value rendering (NATIVE-PARTVAL); genuine `\N` not coerced to NULL (PARTITION-NULL-SENTINEL); field-id history dict for rename-safe native reads (SCHEMA-EVOLUTION); paimon-native split encode under `enable_paimon_cpp_reader` (CPP-READER). - Planning/perf: restore the `force_jni_scanner` escape hatch (FORCE-JNI-SCANNER); intra-file native sub-splitting (NATIVE-SUBSPLIT); precomputed COUNT(*) row count (COUNT-PUSHDOWN); table row count for CBO / SHOW TABLE STATUS (TABLE-STATS). - Time travel / DDL: `FOR TIME AS OF` under CST/PST session zones (TZ-ALIAS); resolved + validated `driver_url` for jdbc catalogs (JDBC-DRIVER-URL); reject a local-only name conflict on CREATE TABLE (CREATE-TABLE-LOCAL-CONFLICT). plan-doc P5 design / task-list / HANDOFF / deviation tracking notes. - ~70 new FE unit suites: paimon connector (offline recording fakes: `RecordingPaimonCatalogOps` / `FakePaimonTable`), fe-core PluginDriven* MVCC / sys-table / scan-node / split-weight, metastore-spi dispatch + per-backend properties, fe-filesystem HDFS/MinIO/anonymous, fe-kerberos. - checkstyle 0 violations; connector import-gate passes; FE main+test compile clean. - End-to-end paimon behavior is docker-gated (`enablePaimonTest=true`); the five functional areas are covered by the `external_table_p0/paimon` regression suites, green on the latest CI run for this branch.
… make fe-core paimon-SDK-free (T29) (apache#64653) Follow-up to apache#64446 (the Paimon catalog-SPI cutover). After the cutover a `paimon` catalog deserializes to `PluginDrivenExternalCatalog` and its tables to `PluginDrivenMvccExternalTable`, so no legacy `Paimon*` catalog / table / scan-node object is ever instantiated — the entire legacy Paimon subsystem in fe-core is dead code. This PR removes it and makes fe-core's dependency tree fully paimon-SDK-free (zero `org.apache.paimon` imports, the 5 paimon maven deps dropped). Mirrors the P4 maxcompute cutover → legacy-removal (apache#64256). **1. Remove legacy subsystem + sever reverse-refs** (`7632a074e4b`) - Delete the dead fe-core paimon files: `datasource/paimon/*` (all 5 flavor catalogs, `PaimonExternalTable`, `PaimonMetadataOps`, `PaimonUtil`, `source/PaimonScanNode` / `PaimonSplit` / `PaimonPredicateConverter` / `PaimonSource` / `PaimonValueConverter`, `profile/*`), `datasource/metacache/paimon/*`, `systable/PaimonSysTable`, and the legacy-only tests. - Sever the live reverse-references to the deleted classes, keeping every `PluginDriven` / connector sibling branch and the image/replay keep-set: drop the `PAIMON` db-creation switch arm in `ExternalCatalog`; the paimon engine routing in `ExternalMetaCacheMgr` / `ExternalMetaCacheRouteResolver`; the dead `PAIMON_EXTERNAL_TABLE` branch in `Env.getDdlStmt` (keep the LIVE `PLUGIN_EXTERNAL_TABLE` SHOW CREATE LOCATION/PROPERTIES rendering); the `PaimonSysExternalTable` branch in `UserAuthentication`; the legacy clauses + `handleShowPaimonTablePartitions()` in `ShowPartitionsCommand` (keep `hasPartitionStatsCapability()` and the `TableType.PAIMON_EXTERNAL_TABLE` enum). Landed as one compiling unit (mirrors the P4 apache#64300 precedent: `PaimonUtils` still called the removed `ExternalMetaCacheMgr.paimon()`). - Inline the `getPaimonCatalogType()` string literals (`hms`/`filesystem`/`dlf`/`rest`/ `jdbc`) to decouple the still-consumed thin metastore-property classes from the deleted `PaimonExternalCatalog`. **2. Make fe-core paimon-SDK-free** (`32de7d16702`) - Strip the dead paimon-SDK catalog-building cluster from `AbstractPaimonProperties` + the 5 flavors (`initializeCatalog` / `buildCatalogOptions` / `appendCatalogOptions` / `getMetastoreType` / `normalizeS3Config` / … — 0 live callers; the plugin path builds catalogs connector-side). Keep every LIVE SDK-free duty: `warehouse` `@ConnectorProperty`, the Kerberos `executionAuthenticator` doAs wiring (read by `PluginDrivenExternalCatalog`), property normalization/validation, `getPaimonCatalogType`, `Type.PAIMON`. - Vended credentials: delete `PaimonVendedCredentialsProvider` + its test + the `VendedCredentialsFactory` `case PAIMON`. Its SDK methods were dead — reachable only via the iceberg-only `getStoragePropertiesMapWithVendedCredentials`; the real paimon vended path is the connector's `PaimonScanPlanProvider.extractVendedToken` (moved off fe-core at cutover). Relocate its one LIVE duty (the REST "skip static storage map" gate) to a new SDK-free `MetastoreProperties.isVendedCredentialsEnabled()` (base=false, `PaimonRestMetaStoreProperties`=true); `CatalogProperty` routes iceberg through its provider (byte-identical) and everything else through the metastore-props method. - pom: drop `paimon-core` / `-common` / `-format` / `-s3` / `-jindo` from `fe-core/pom.xml`. `fe/pom.xml` `paimon.version` is kept (fe-connector-paimon + BE paimon-scanner still consume it). **3. Doc-sync** (`58160cbcb56`, `895e214b2bd`) — plan-doc HANDOFF / PROGRESS / connectors / P5-T29 legacy-removal design tracking notes. - `mvn -pl :fe-core -am test-compile` (main + test) passes; checkstyle 0 violations; connector import-gate (`tools/check-connector-imports.sh`) passes. - `grep -rn org.apache.paimon fe/fe-core/src` → empty (main and test). - `mvn -pl :fe-core dependency:tree -Dincludes=org.apache.paimon` → empty (no paimon dep, direct or transitive). - End-to-end paimon regression (`external_table_p0/paimon`, docker `enablePaimonTest=true`) passed on this branch.
…kerberos (apache#64655) **P3b: consolidate the drifted Kerberos/Hadoop authentication implementations into the new top-level neutral leaf module `fe-kerberos`** as the single source of truth. Done as 3 commits: 1. **trino → JDK** (`4a740e1`) — replace the only external dependency in the auth path, trino's `KerberosTicketUtils`, with a JDK-only (`javax.security.auth.kerberos`) byte-for-byte equivalent, so the kerberos path is trino-free. 2. **relocate** (`8898e15`) — move the 13 `fe-common` `security.authentication.*` classes to `org.apache.doris.kerberos.*` in `fe-kerberos`; retarget all consumer imports (fe-core + 3 be-java-extensions scanners); merge the duplicate `AuthType`. 3. **unify interface** (`5e3e896`) — merge the two competing `HadoopAuthenticator` interfaces (fe-common's `PrivilegedExceptionAction` variant vs fe-filesystem-spi's `IOCallable` variant) into the single fe-kerberos one, and delete fe-filesystem-hdfs's own `KerberosHadoopAuthenticator`/`SimpleHadoopAuthenticator` copies (which had drifted from the canonical impls). `DFSFileSystem` now routes through the shared authenticators. `fe-kerberos` remains a top-level neutral leaf (no dependency cycle). HDFS filesystem access now uses the same authenticators as the HMS path (restoring parity). Two intentional behavior changes in fe-filesystem-hdfs: simple / no-`hadoop.username` now runs as remote user `hadoop` (was: FE process user, direct); kerberos uses the shared `LoginContext` + 80%-lifetime refresh. fe-filesystem-hdfs 79/0/0 (+fe-kerberos/spi), checkstyle 0, connector import-gate clean, whole-repo grep for the removed symbols = 0. >⚠️ docker kerberos e2e (HDFS kerberized + HMS) NOT yet run — the real gate; UGI login can't be exercised in unit tests.
…move legacy fe-core subsystem (apache#64688) Migrates the legacy in-tree Iceberg catalog (fe-core `datasource/iceberg`, its `property/metastore/*Iceberg*` clusters, and every reverse `instanceof IcebergExternal*` / `case ICEBERG` coupling scattered across nereids / planner / alter) onto the catalog SPI as a self-contained `fe-connector-iceberg` plugin, following the paimon full-adopter + cutover template. `iceberg` is added to `SPI_READY_TYPES`, so an Iceberg catalog now deserializes to `PluginDrivenExternalCatalog` and every functional area — read, write, row-level DML, `ALTER TABLE EXECUTE` procedures, system tables, DDL and MVCC/time-travel — flows through the generic `PluginDriven*` bridge with no Iceberg-specific branch in fe-core. (Iceberg tables surfaced *through* a `hive`/HMS catalog stay on the hive path until P7; this PR covers the dedicated `type=iceberg` catalog.) Iceberg is the widest lakehouse adopter migrated so far — 7 catalog flavors, merge-on-read row-level DML, 9 stored procedures, v3 row-lineage / deletion vectors — so the cutover is held to behavior-parity with the legacy path. Unlike the paimon migration (a cutover PR + a separate removal PR), this PR does the build-out, the live cutover **and** the full legacy-subsystem removal on one branch: fe-core `datasource/iceberg` and its property clusters are deleted, not just left dead. Part of the catalog-SPI migration tracked in apache#65185 (phase **P6**). Full read + write + row-level DML + procedures + system tables + DDL + MVCC/time-travel, planning native (Parquet/ORC) vs JNI reads. Key seams: - `IcebergCatalogFactory` (pure flavor switch, classloader-pinned for child-first loading), `IcebergCatalogOps` (metadata / DDL ops), `IcebergConnectorMetadata` (schema latest / at-snapshot, sys tables, MVCC/time-travel, DDL, partitions, statistics, table descriptor), `IcebergConnectorProvider` (ServiceLoader entry). - `IcebergScanPlanProvider` + `IcebergScanRange`: predicate/projection pushdown (`IcebergPredicateConverter` → Iceberg `Expression`), FileScanTask byte-offset sub-splitting, COUNT(\*) collapse, merge-on-read delete attachment (position delete / Puffin DV / equality delete), `IcebergColumnHandle` field-id column pruning, schema-evolution-safe reads (`IcebergSchemaUtils` field-id dictionary), synthetic rowid + v3 row-lineage metadata columns. - Write path: `IcebergWritePlanProvider` / `IcebergWriteContext` / `IcebergWriterHelper` / `IcebergConnectorTransaction` — INSERT / INSERT OVERWRITE (dynamic + static partition), branch-targeted writes, WRITE ORDERED BY, per-spec distribution, optimistic-concurrency conflict detection, one SDK transaction per statement, BE commit-fragment → DataFile/DeleteFile conversion. - Row-level DML: DELETE / UPDATE / MERGE INTO over position-delete (v2) and deletion-vector (v3) merge-on-read, baseSnapshot-anchored read. - `action/`: the 9 `ALTER TABLE EXECUTE` procedures (rollback_to_snapshot, rollback_to_timestamp, set_current_snapshot, cherrypick_snapshot, fast_forward, expire_snapshots, publish_changes, rewrite_manifests, rewrite_data_files) dispatched by `IcebergExecuteActionFactory`; `rewrite/` hosts the distributed bin-pack `rewrite_data_files` engine. - `IcebergSchemaBuilder` (CREATE TABLE), `IcebergTypeMapping` (both directions), `IcebergColumnChange` / `IcebergComplexTypeDiff` (schema + partition evolution), `IcebergLatestSnapshotCache` / `IcebergManifestCache` (per-catalog caches on the long-lived connector), `dlf/` (Aliyun DLF client). (`fe-connector-metastore-iceberg`) An Iceberg catalog resolves its backend by `MetaStoreProvider` ServiceLoader dispatch on `iceberg.catalog.type` (registered via `META-INF/services`), instead of a fe-core switch — **7 flavors**: REST, Hive Metastore, AWS Glue, Hadoop / filesystem, JDBC, Aliyun DLF, AWS S3Tables. Adding a backend = one provider class + one services line. The connector self-builds the HMS-metastore Kerberos authenticator, so metastore auth is owned entirely by the plugin. - `CatalogFactory` adds `iceberg` to `SPI_READY_TYPES` and drops the built-in case. - `GsonUtils` replay-compat: persisted `IcebergExternalCatalog` → `PluginDrivenExternalCatalog`, `IcebergExternalDatabase` → `PluginDrivenExternalDatabase`, `IcebergExternalTable` → `PluginDrivenMvccExternalTable` on deserialization, so existing clusters upgrade without losing catalogs (metadata backward-compat invariant). - Nereids/DDL/alter generalized off the concrete Iceberg types; SHOW CREATE renders LOCATION / PARTITION BY / ORDER BY / PROPERTIES via the generic path (iceberg engine name preserved, no credential leak). Per the 2026-07 architecture decision (fe-core holds **no** property parsing for migrated connectors; storage parsing → `fe-filesystem`, metastore parsing → `fe-connector`, both plugin-side; the plugin assembles → BE thrift → hands back to fe-core), iceberg's property/credential/auth resolution moves entirely into the connector: - Storage credentials bound to `fe-filesystem` on both scan and write paths (one source), retiring the second fe-core `StorageProperties.createAll` parse. - Vended temporary credentials, `warehouse` → `fs.defaultFS` derivation (hadoop flavor only, via the neutral `Connector.deriveStorageProperties` SPI), and Kerberos doAs are all connector-owned; the fe-core pre-execution authenticator is a no-op for plugin catalogs. - fe-core's shared static property/metastore helpers are kept alive only for the connectors still on the legacy path (Hive / Hudi / LakeSoul / HMS-iceberg). Net **−21.7k** lines from fe-core. Removes the 4 dead entity classes (`IcebergExternalTable` / `IcebergExternalDatabase` / `IcebergSysExternalTable` / `IcebergExternalCatalog`), the fe-core Iceberg `MetastoreProperties` cluster + its credential helpers, the 5 fe-core Iceberg connectivity probes and the coordinator branches, and un-registers the Iceberg property factories (fail-loud on any stray call). The now-dead paimon `MetastoreProperties` cluster left behind by P5 is swept in the same pass. `GsonUtils` retains only the three string compat labels for old-image upgrade. Parity fixes surfaced by the SPI cutover (each restores legacy semantics): - Classloader: TCCL pinned to the plugin classloader at all three loci where fe-core crosses into bundled Iceberg (scan call thread, write/DDL/commit engine thread via `TcclPinningConnectorContext`, and Iceberg's own manifest-writer worker pool), preventing a duplicate-copy `ClassCastException`. - Nested schema dictionary field names lowercased so a mixed-case nested struct (e.g. Iceberg `DROP_AND_ADD`) can't crash BE with `std::out_of_range`. - Session time zone: lazy `ZoneId` resolution tolerant of Doris CST/PST-style aliases in `FOR TIME AS OF` and datetime partition predicates. - Storage / URI normalization (`oss/cos/obs/s3a` → `s3`), SHOW CREATE secret redaction, and `ALTER TABLE EXECUTE ... WHERE` rejection kept at legacy wording. - ~68 FE unit test classes: iceberg connector (offline recording fakes for the remote Catalog / Table), metastore-iceberg per-flavor dispatch + properties, and the fe-core `PluginDriven*` MVCC / sys-table / scan-node / GSON-replay surfaces; many carry mutation checks. - checkstyle 0 violations; connector import-gate passes (no `fe-connector-iceberg → fe-core` import); FE main + test compile clean. - End-to-end iceberg behavior is docker-gated (`enableIcebergTest=true`); read, write, row-level DML, procedures, system tables, DDL and time-travel are covered by the `external_table_p0/iceberg` (+ `external_table_p2/iceberg`) regression suites. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…etaspace leak JdbcConnectorClient cached JDBC-driver classloaders in a ref-counted map that evicted the entry when refCount reached 0. A JDBC driver self-registers into the static java.sql.DriverManager on class load, and close() never deregisters it, so an evicted classloader stays pinned by DriverManager for the life of the process. Serial CREATE -> DROP -> CREATE churn of the same driver URL therefore rebuilt a fresh, separately-pinned classloader every cycle, leaking one driver's worth of Metaspace per cycle. Live FE evidence (external regression build 986696 OOM): only 4 jdbc catalogs yet 38 FactoryURLClassLoader instances survived a full GC (jmap -histo:live), tracking the DriverInfo count -> ~34 leaked, DriverManager-pinned driver classloaders. FE Metaspace grew 165MB -> 1565MB over the single-concurrency 5.5h run (multi-concurrency did not OOM: overlapping same-URL catalogs keep refCount > 0 so the loader was reused instead of evicted-and-rebuilt). Fix: revert to a keep-alive Map<URL, ClassLoader> (one loader per distinct driver URL, shared across catalogs, never evicted), mirroring the pre-SPI fe-core JdbcClient and the pattern IcebergConnector / PaimonConnector already use. Bounded by the number of distinct driver jars. Adds a regression test asserting repeated create/recreate cycles for one URL add exactly one cached classloader. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XKf5EzHW75qzgrXnkoBZcF
…TableTest after rebase Rebase-integration fix, belongs with P4 (apache#64300 "remove legacy maxcompute subsystem"); squash into it when finalizing that PR. Upstream apache#64937 added MaxComputeExternalTableTest as a unit test for the legacy MaxComputeExternalTable.parsePartitionValues. P4 deletes that class, but the new test file rode into the rebase base with no modify/delete conflict and then dangled: fe-core test-compile failed with "cannot find symbol: MaxComputeExternalTable". No coverage lost: the new connector's MaxComputeConnectorMetadata.listPartitionValues already resolves partition values by name via ODPS PartitionSpec.get(column) (structurally immune to the out-of-order/invalid-spec bug apache#64937 fixed in the legacy path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…es into SPI connectors After rebasing branch-catalog-spi onto upstream master (4dfcb4f), an audit of the 3 upstream commits touching fe-core datasource/ found behaviour that was dropped when P5/P6 removed the legacy fe-core paimon/iceberg subsystems. Port the bounded correctness gaps into the new fe-connector SPI code (main + tests): - apache#65332 (paimon JNI IOManager): PaimonScanPlanProvider.getBackendPaimonOptions returned emptyMap() for non-jdbc catalogs, so the three JNI IOManager options (paimon.doris.enable_jni_io_manager / .tmp_dir / .impl_class) never reached BE and Paimon primary-key merge reads could not spill (OOM risk). Collect the three options (strip the "paimon." prefix) before the metastore-type check and forward them for all catalog flavours; jdbc-only driver logic stays in the jdbc branch. - apache#65094 tier1 (build-time correctness): IcebergSchemaBuilder buildPartitionSpec / buildSortOrder and PaimonSchemaBuilder primary/partition keys used bare column names, so a partition/sort/key referenced with a different case than the (now case-preserving) schema column failed the engine's case-sensitive lookup. Add case-insensitive resolution back to the canonical column name (iceberg via Schema.caseInsensitiveFindField; paimon via a lower->canonical map), mirroring upstream getIcebergColumnName / getPaimonColumnNames. - apache#65094 tier2 (consistency, worse than upstream): PaimonConnectorMetadata emitted "partition_columns" with case-preserving keys while column names are lowercased, so PluginDrivenExternalTable's case-sensitive byName lookup silently dropped mixed-case paimon partition columns (table treated as non-partitioned). Lower-case partition_columns with the same bare toLowerCase() the columns use. Verified: mvn package -pl :fe-connector-paimon -am (95 paimon tests) + iceberg 23 tests, 0 failures. Read-path column-name case preservation (tier3) and the apache#63068 OIDC session-credential port are tracked separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…column-name case on the read path Align the iceberg + paimon catalog-SPI connectors with upstream apache#65094: external table top-level column names now keep their remote case on the read path instead of being lowercased. FE-only (BE untouched): the byte-match invariant "Doris scan slot name == TSchema top-level TField name" is preserved by flipping BOTH the scan slot names AND the field-id-dict / current-TSchema top-level names to case-preserving together, so BE (which matches top-level by field-id) needs no change. Nested struct children stay as-is (iceberg lowercase for BE StructNode.children.at; paimon paimon-cased). Reverts tier2's lowercasing of paimon partition_columns (bce28b2) now that the paimon columns are themselves case-preserving. iceberg (IcebergConnectorMetadata): partition_columns / getColumnHandles / parseSchema. iceberg (IcebergPartitionUtils): getIdentityPartitionColumns (path_partition_keys), getIdentityPartitionInfoMap (scan partition-value map) and generateRawPartition -- the listPartitions value-map key that PluginDrivenExternalTable.getNameToPartitionItems looks up by the case-preserved partition_columns remote name; omitting it would silently drop a mixed-case partition column's value in MTMV / partition pruning. paimon (PaimonConnectorMetadata): partition_columns / getColumnHandles / mapFields. paimon (PaimonScanPlanProvider): path_partition_keys / getPartitionInfoMap and the current (-1) TSchema entry (buildSchemaInfo lowercaseTopLevelNames -> false at the sole current-schema call site; the historical entries and nested names are unchanged). Tests: flip the connector tests that pinned mixed-case -> lowercase output to assert case preservation; add an IcebergSchemaUtils mixed-case byte-match case and an IcebergPartitionUtils.listPartitions regression pinning the case-preserved partition-value-map key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…l DTO + user-session capability Re-migrate the Iceberg REST OIDC per-user session-credential feature (upstream apache#63068 / e545f1a, dropped by the 2026-07-09 P6 rebase) onto the catalog SPI. This is the SPI layer (fe-connector-api): - ConnectorDelegatedCredential: neutral, immutable DTO carrying a user's per-connection OIDC/JWT/SAML token across the fe-core -> connector boundary so the connector never imports a fe-core type. The token is connection-scoped and in-memory only; toString() redacts it (never edit-logged / persisted / SHOWn). - ConnectorSession.getDelegatedCredential() (default empty) + getSessionId() (default = getQueryId): a connector reads the credential and the stable AuthSession key (preserved across FE observer->master forwarding) off the session. - ConnectorCapability.SUPPORTS_USER_SESSION: gates FE credential injection (least-privilege -- a connector that never consumes the token never receives it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…injection + per-user cache bypass FE bridge for the Iceberg REST per-user session feature (builds on the SPI DTO): - ConnectorSessionBuilder.from(ctx): copy the retained per-connection DelegatedCredential + sessionId onto the ConnectorSession ONLY when the target connector declares SUPPORTS_USER_SESSION (least-privilege); map fe-core DelegatedCredential -> neutral ConnectorDelegatedCredential by enum name so an added-but-unmapped type fails loud rather than being silently dropped. - PluginDrivenExternalCatalog.shouldBypassTableNameCache / shouldBypassDbNameCache = supportsUserSession() && ctx.hasDelegatedCredential(). Under session=user the REST server returns PER-USER metadata, so the shared (catalog+name-keyed, NOT user-keyed) db/table-name caches are bypassed to prevent cross-user leakage; a session with no credential keeps the shared cache (the fail-closed rejection then happens connector-side on the actual read, never by serving/poisoning a shared entry). - ExternalCatalog/ExternalDatabase: db-level live-path bypass (getFilteredDatabaseNames(false)/getDbNullableWithoutCache/matchesLocalDbName), still re-appending information_schema+mysql and never mutating the shared lowerCaseTo*Name lookup when bypassing (updateLookup=false). Tests: ConnectorSessionImplTest (capability-gated inject / non-capable skip / no-cred), PluginDrivenExternalCatalogSessionBypassTest (the bypass decision across capability + credential + null-context). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…outing (metadata + scan/write/procedure) Iceberg connector consumer for the per-user REST session feature: - IcebergConnectorProperties + IcebergRestMetaStoreProperties: iceberg.rest.session (none|user) + iceberg.rest.oauth2.delegated-token-mode (access_token|token_exchange) + optional session-timeout, with validation (session=user requires security.type=oauth2; relax the "oauth2 requires static token/credential" rule for user-session, whose identity is the per-request user token). - IcebergDelegatedCredentialUtils + IcebergSessionCatalogAdapter: bridge the neutral ConnectorDelegatedCredential to an iceberg SessionCatalog.SessionContext (access_token = verbatim OAuth2 bearer; token_exchange = the typed token-type key) over a SINGLE shared RESTSessionCatalog. delegatedCatalog/delegatedViewCatalog fail closed on a tokenless session=user request (never borrow a shared identity). - IcebergConnector: build a RESTSessionCatalog for a session=user REST catalog; newCatalogBackedOps(session) routes per-request metadata through the querying user's delegated catalog; declare SUPPORTS_USER_SESSION only for that config. - scan/write/procedure providers take a Function<ConnectorSession, IcebergCatalogOps> resolver (connector passes this::newCatalogBackedOps); the legacy ops constructors bind a constant s->ops so existing tests/behaviour stay byte-identical. This makes SELECT / INSERT-DELETE-MERGE / ALTER TABLE ... EXECUTE run per-user. The write COMMIT was already per-user (IcebergConnectorTransaction is opened by beginTransaction over the session-aware metadata ops). Tests: IcebergProviderSessionRoutingTest (each provider resolves ops with the call session + fail-closed), IcebergSessionCatalogAdapterTest (credential-key mapping per mode + fail-closed + fromString), IcebergConnectorValidatePropertiesTest (+session validation). IcebergConnectorTest/IcebergProcedureOpsTest adjusted for the resolver constructor (reflection reads catalogOpsResolver; a bare-null ctor arg is cast). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…research notes + HANDOFF (Tasks 0-7 DONE) - Add the authoritative design doc (Trino-aligned, SPI-retargeted) and the source-verified research notes (Trino reference + apache#63068 as-built + current SPI). - HANDOFF: mark Tasks 0-7 DONE with the 3-commit stack (SPI 55c991d / FE 930e477 / connector a07216e), the clean-room review result (routing completeness = zero gaps; cache-leakage = none), and the two follow-ups (background log-noise reduction; two minor case/consistency nits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…essionContextTest (cross-user cache-leakage data-flow) Closes the one remaining apache#63068 test gap: the DATA-FLOW proof (not just the bypass decision, which PluginDrivenExternalCatalogSessionBypassTest already pins) that a session=user catalog serves per-user database metadata LIVE and never through the shared (catalog+name-keyed, not user-keyed) name cache -- the cross-user leakage guard (Trino CVE-2026-34214). Retargeted off the deleted IcebergRestExternalCatalog onto a PluginDrivenExternalCatalog test subclass: mockStatic(SessionContext.current()) drives the per-token identity, the catalog overrides the remote listing to return each user's own databases and record the listing token. Two disjoint per-user results + a live re-list on EVERY read (including a repeat of an earlier token) prove no user's database set leaks to another and nothing was served from a shared cache; information_schema+mysql stay visible. apache#63068 asserted this via a no-token "bootstrap" read, which on this branch fail-closes (a session=user catalog has no shared identity to bootstrap) -- the per-read live token record is the architecture-correct equivalent observable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…a-flow test + Task 0 retained-base verify Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…er rebase to 4f48ccf Rebase branch-catalog-spi onto upstream-apache/master (10 upstream commits, 21 local commits replayed). Two files were touched by both sides; both are paimon + apache#65365 fallout: 1. Legacy fe-core PaimonScanNode.java: modify/delete conflict — upstream apache#65365 added "jni.enable_file_reader_async" to its BACKEND_PAIMON_OPTIONS forwarding list, while P5 (T29) deleted the whole legacy fe-core paimon subsystem. Resolution keeps the deletion and ports the new key to the connector-side mirror list PaimonScanPlanProvider.BACKEND_PAIMON_JNI_OPTIONS (+ intent test): the SPI path is the only forwarding path left, so without the port the new catalog property would silently never reach BE's PaimonJniScanner. 2. PaimonJniScanner.java: silent semantic conflict — apache#65365 removed the then-unused java.util.Collections import upstream, while this branch's null-predicate backstop uses Collections.emptyList(). git auto-merged the non-overlapping hunks without complaint, breaking compilation. Restore the import. Verified: fe-connector-paimon package GREEN (PaimonScanPlanProviderTest 65/0 incl. new backendOptionsForwardFileReaderAsyncOptOut); paimon-scanner Paimon* tests 11/0; full build.sh --fe BUILD SUCCESS (build cache disabled). Pre-existing local-env JVM crash in java-common JniScannerTest (StubRoutines::jbyte_disjoint_arraycopy_avx3, byte-identical module before and after rebase) is unrelated and left as-is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…progress → P7 hive P6 iceberg migration + tests are complete and squash-merged into upstream branch-catalog-spi as apache#64688 (8b391c7). Overwrite the rolling HANDOFF and sync all related progress trackers to reflect "P6 DONE → next = P7 hive/HMS". - HANDOFF.md: full rewrite to a P7-kickoff handoff (plan pointers to master-plan §3.8 + connectors/hive.md, sub-phases P7.1-P7.5, R-002 ACID gate, reusable ops/commit rules). - PROGRESS.md: §一 P6→100%/merged + P7→active + global ~64%; §二 kanban iceberg→100%/merged; §三 P6 block→P7 block; §四 new 2026-07-05 entry; §六 D-073/DV-049 counts; §七 session status. - connectors/iceberg.md: status rows → ✅ merged apache#64688; drop DV-038 flip-blocker label; progress-log entry. - connectors/hive.md: status → active P7 target; progress-log entry. - tasks/P6-iceberg-migration.md + P6.6-flip-blockers-tasklist.md: status banners → ✅ COMPLETE / merged apache#64688. Note: apache#64688 removed the native iceberg subsystem but 23 fe-core datasource/iceberg/ HMS-iceberg support classes remain (iceberg-on-HMS still routes through fe-core); they are deleted with P7 hive (阶段四, decision D5/Q3=B). Recorded in all touched docs. Docs only, no product code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
10-agent code-grounded recon of fe-core datasource/hive (52 files) + a supplemental type-coupling pass. Produces tasks/P7-hive-migration.md (P7.1-P7.5 split, old->new mapping, cutover mechanics, cross-connector deletion ordering, SPI gaps, 8 open decisions) and syncs HANDOFF / PROGRESS / connectors/hive.md. Key facts corrected vs stale plan: reverse instanceof 31->85 (33 files), HMSTransaction 1866->1895, HMSExternalTable 1293->1332. Coverage critic closed an instanceof-only blind spot (CatalogFactory:134, GsonUtils 366/447/471 compat, HudiUtils/IcebergHMSSource type coupling). Confirmed via decisions-log that iceberg/hudi-on-HMS ownership is already settled (D-020 per-table SPI provider delegation + D-019 hudi live cutover into P7) - not re-litigated. No product code, no new decisions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…n + lock config-threading decision P7.1 起步 recon（3 并行 agent + 直读 HEAD）核清并校正 spec： - 通用 DDL 桥 = PluginDrivenExternalCatalog（逐条 override + 内联 cache/editlog；连接器只实现纯 SPI）。 - TRUNCATE 硬缺口：SPI 无 truncateTable、桥未 override → 须加 seam + 桥 override（复用 OP_TRUNCATE_TABLE）。 - shared converter 逐字拷 properties（不 parse），但丢每列默认值（恒 null → 破坏 hive 列默认值 + DLF guard）+ 丢 LIST/RANGE partition value。 - stats/partition 写 4 法仅 HMSTransaction 消费 → 推 P7.3（本阶段加即死代码）；rename hive 今天不支持 → 保持 SPI default throw。 - 模板 = IcebergConnectorMetadata（HMS-backed + DLF guard + 插件侧 SchemaBuilder）。 产出 tasks/P7-hive-migration.md 末尾 P7.1 逐 task 拆解（T01–T11，范围=仅 DDL 写路径）+ recon 结论块。 唯一阻塞决策已由用户裁定：全局建表默认（默认文件格式 / 是否允许分桶表）迁到插件后取值来源 = 方案 A （fe-core 建连接器时把两个全局 Config 值注入为连接器属性默认，插件只读属性、零行为回归）。 无代码改动（仅文档 + 决策）。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…iven bridge route The generic SPI DDL bridge (PluginDrivenExternalCatalog) had no truncate route: the base ExternalCatalog.truncateTable throws "not supported" when metadataOps == null, so every SPI catalog (iceberg/paimon today) rejects TRUNCATE. Hive needs it after the P7 cutover. - ConnectorTableOps.truncateTable(session, handle, partitions): new default-throw seam (partitions null/empty = whole table). Default preserves iceberg/paimon "not supported". - PluginDrivenExternalCatalog.truncateTable(...): resolves the table by REMOTE names, dispatches to metadata.truncateTable, then emits the same TruncateTableInfo edit log the base op writes and refreshes the local table cache via RefreshManager.refreshTableInternal (mirroring legacy HiveMetadataOps.afterTruncateTable). forceDrop/rawTruncateSql carry no external semantics and are ignored, matching the legacy path. - PluginDrivenExternalCatalog.replayTruncateTable(info): refreshes the cache on follower replay (the base delegates to metadataOps.afterTruncateTable, a no-op for PluginDriven). No connector overrides truncateTable yet (HiveConnectorMetadata comes in a later P7.1 task), so behavior is unchanged pre-flip; verified by compile + checkstyle 0 + import-gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
… handoff → T02 Update rolling handoff + task spec: T01 (TRUNCATE SPI seam) done & green (c022297); next session resumes at T02 (plugin write DTOs) → T03/T04 (write client). Archive the T03 converter porting pointers (HiveUtil line refs + the fe-core couplings to break: dorisTypeToHiveType inverse over ConnectorType, DORIS_VERSION const, HiveProperties, hiveTextCompression session var, format/compression tables) so the next session does not re-recon. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…create/drop db+table, truncate) + converter Builds the SPI-clean write client in fe-connector-hms so the hive connector can issue CREATE/DROP DATABASE, CREATE/DROP TABLE and TRUNCATE without fe-core. The metadata layer (HiveConnectorMetadata overrides), config threading and the recording-fake client tests land in follow-up steps; this stays non-live (no cutover until the hive connector is flipped on). - HmsCreateTableRequest / HmsCreateDatabaseRequest: write-side specs mirroring legacy HiveTableMetadata / HiveDatabaseMetadata, connector-api + JDK types only. Columns carry data + partition columns; per-column defaults ride on ConnectorColumn.getDefaultValue(). Text-compression default and doris.version are threaded on the request (the plugin must not import fe-core Config/Version). - HmsTypeMapping.toHiveTypeString: reverse of the read mapping, SPI-clean equivalent of HiveMetaStoreClientHelper.dorisTypeToHiveType. Switches on the Doris PrimitiveType names ConnectorColumnConverter emits; unsupported types throw (parity with legacy) rather than emit a bogus type. - HmsWriteConverter: faithful port of HiveUtil.toHiveTable/toHiveDatabase plus HiveProperties.setTableProperties (serde vs table property split). Per-format input/output/serde + compression defaults copied verbatim; MANAGED_TABLE type and the "doris external hive table" storage tag preserved. - HmsClient: add createDatabase/dropDatabase/createTable/dropTable/truncateTable as default-throwing seams so read-only implementations (partition-pruning test fakes) keep compiling; ThriftHmsClient overrides all five via the existing execute(...) auth+pool framework, building SQLDefaultConstraint from column defaults (equivalent to legacy createTableWithConstraints). Verify: fe-connector-hms compile SUCCESS + checkstyle 0 (main+test) + check-connector-imports clean + 28 unit tests green (reverse type mapping + converter). No connector overrides these yet, so behavior is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…ndoff → T05 Record the plugin-side HMS DDL write client (fdf577e) as complete in the task breakdown + rolling handoff, and point the next unit at T05→T07 (HiveConnectorMetadata DDL overrides) with the two request fields the metadata layer / config-injection locus must populate (defaultTextCompression, dorisVersion). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…olumn defaults + explicit-partition-values signal Thread two pieces of the neutral CREATE TABLE request that the hive plugin needs to reproduce legacy HiveMetadataOps.createTableImpl parity; both are additive and ignored by connectors that build their schema from name/type/nullable/comment only (iceberg/paimon/maxcompute build-time paths do not read either). - Per-column default value: expose ColumnDefinition.getDefaultValueString() (Column.getDefaultValue() equivalent, via DefaultValue.getValue()) and thread it onto ConnectorColumn.defaultValue in CreateTableInfoToConnectorRequestConverter (was hardcoded null). Hive builds metastore default constraints from it and gates its DLF catalog on per-column defaults. - Explicit-partition-values signal: the converter drops LIST/RANGE partition value expressions (still not lowered), so add ConnectorPartitionSpec.hasExplicitPartitionValues() and set it from PartitionTableInfo.getPartitionDefs(). Lets hive keep rejecting `PARTITION BY LIST(dt) (PARTITION p VALUES IN (...))` (external tables discover partitions from the data layout) instead of silently ignoring it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…nfig defaults via connector environment The fe-connector-hive plugin cannot read FE Config, so surface the two FE-global CREATE TABLE toggles + the build version through the connector environment channel (DefaultConnectorContext.buildEnvironment) — the same "C4" mechanism already used for hive_metastore_client_timeout_second. Chosen over injecting them into the catalog property map so they are NOT persisted into the catalog image/edit-log, do not appear in SHOW CREATE CATALOG, and refresh from current Config on each connector rebuild. - hive_default_file_format <- Config.hive_default_file_format - enable_create_hive_bucket_table <- Config.enable_create_hive_bucket_table - doris_version <- Version build id (legacy ExternalCatalog.DORIS_VERSION_VALUE) A user-set per-table file_format property still wins (resolved plugin-side); the plugin reads these keys via ConnectorContext.getEnvironment(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…override (create/drop db+table, truncate) + tests Port legacy HiveMetadataOps create/drop database, create/drop/truncate table into the plugin so the "hms" SPI path reproduces its behavior (still off — cutover is P7.5). All property interpretation stays plugin-side (fe-core parses no hive properties). - T05 database: supportsCreateDatabase()=true; createDatabase (location -> URI, comment -> description, rest -> params); dropDatabase(force) cascades table drops then the db. - T06 createTable (faithful port of createTableImpl): owner default (session.getUser()), transactional-create reject, file_format default (env, user prop wins), doris.-prefixed round-trip params, LIST-only partitions + explicit-partition-values reject, DLF per-column-default guard, bucket gate (env enable + hash-only), doris.version + text compression default threaded onto the write request. - T07 dropTable (transactional-table reject, via handle params = AcidUtils parity without a hive-exec dep) + truncateTable; renameTable intentionally left as the SPI default throw (hive has no rename). HiveConnector now passes ConnectorContext to the metadata. - T10 tests: recording HmsClient fake + fake session/context (no Mockito), 20 DDL cases covering the guards, defaults, round-trip params, and force cascade. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
Contributor
TPC-DS: Total hot run time: 180225 ms |
Batch-0 (apache#65185 reverify) H4. After the HMS SPI cutover, a hudi-on-HMS table with mixed-case Avro columns (e.g. Id/Name/Addr) crashed every MOR/JNI split. Root cause: HudiScanPlanProvider.planScan built the JNI reader column list with `.map(Schema.Field::name)` (original case). BE HadoopHudiJniScanner.initRequiredColumnsAndTypes keys hudiColNameToType by these names, then resolves each requiredField with an EXACT lower-case containsKey; mixed-case "Id" misses lower-case "id" -> throw. The two sibling name paths were already lower-cased (avroSchemaToColumns:905, HudiSchemaUtils.buildField:137); legacy HudiScanNode:223 also emits lower-case names. This one path regressed. Fix: extract the inline transform into a package-private static `jniColumnNames(Schema)` that lower-cases each field name (toLowerCase(Locale.ROOT)); column ORDER preserved so the parallel columnTypes list stays positionally aligned. Extraction makes the transform offline-unit-testable (planScan needs a live metaClient). Test: HudiSchemaParityTest.jniColumnNamesAreLowerCased asserts the mixed-case Id/Name/Addr fixture lower-cases. MOR/JNI end-to-end read with mixed-case columns is live-gated (real hudi cluster). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
Contributor
ClickBench: Total hot run time: 25.02 s |
…/H2 hive scope H4 DONE (03f4c12). Records the batch-0 scope decision: adversarial verification confirmed H1/H2 are NOT hudi-only — HiveConnectorMetadata's byte-identical prune block silently drops rows too. User signed off on "fix both copies in place" (no shared-helper extraction); D-PRUNE stays deferred as tracked design debt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
Batch-0 (apache#65185 reverify) H1. After the HMS SPI cutover, a partitioned hive-on-HMS or hudi-on-HMS table silently DROPPED rows when a partition value contained a Hive-escaped character (e.g. ':' stored as '%3A') and the query carried an EQ/IN predicate on that column. Root cause: the connector partition-pruning decision uses a byte-identical block in both HiveConnectorMetadata and HudiConnectorMetadata. parsePartitionName stored the raw substring after '=' WITHOUT unescaping, while the predicate literal (extractLiteralValue) is unescaped, so matchesPredicates' string compare (escaped 'US%3ACA' vs unescaped 'US:CA') never matched -> the real partition was pruned out -> silent row loss. fe-core computes a correct typed requiredPartitions but the connector planScan ignores it and honors only applyFilter's prunedPartitions, so this pruning is authoritative. The sibling parse paths already unescape (HudiScanPlanProvider.parsePartitionValues, HiveWriteUtils.toPartitionValues); this pruning path regressed. Adversarial verification confirmed BOTH connectors are affected (not hudi-only). Fix (both copies in place, per scope decision): unescape the VALUE in each connector's parsePartitionName via its own same-package unescapePathName (widened private->package-private). The key (column name) is left as-is. parsePartitionName widened to package-private static for direct offline unit testing (mirrors the already-tested parsePartitionValues). Tests: direct parsePartitionName unescape assertions in HudiPartitionValuesTest and HiveConnectorMetadataPartitionPruningTest (durable across the later H3 restructure); plus a hive end-to-end applyFilter test proving an escaped partition value prunes-in instead of dropping. Escaped-value read on a real cluster is live-gated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
H1 DONE (39a279e). Unescape partition value in the prune-matching parse in both hive and hudi copies. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…e text Batch-0 (apache#65185 reverify) H2. After the HMS SPI cutover, an EQ/IN predicate on a DATETIME/TIMESTAMP partition column pruned a hive-on-HMS or hudi-on-HMS table to 0 rows (silent total row loss). DATE and STRING partition columns were unaffected. Root cause: the byte-identical prune block in both HiveConnectorMetadata and HudiConnectorMetadata renders the predicate literal via String.valueOf. ExprToConnectorExpressionConverter.convertDateLiteral wraps a non-DATE datetime literal as a LocalDateTime, and String.valueOf(LocalDateTime) == toString() == ISO "2024-01-01T10:00" (T separator, dropped zero seconds), which never string-equals the Hive-canonical stored partition value "2024-01-01 10:00:00" in matchesPredicates -> every partition is pruned out. Legacy pruned via typed Nereids comparison, never strings. Adversarial verification confirmed BOTH connectors are affected, and a real Hive fixture (run17.hql: `time_par timestamp` stored as time_par=2023-01-01 01%3A30%3A00) confirms the stored scale-0 form is exactly "yyyy-MM-dd HH:mm:ss". Fix (both copies in place, per scope decision): in extractLiteralValue, render a LocalDateTime value via a new package-private static hiveDateTimeString helper -> "yyyy-MM-dd HH:mm:ss" (space separator, full seconds), appending trailing-zero-trimmed microseconds only when present. All other value types keep String.valueOf; null stays null; LocalDate (DATE columns) is unaffected. No timezone conversion is involved (unlike the MaxCompute datetime-pushdown fix), so there is no ZoneId.of crash risk. This composes with H1 (H1 unescapes the stored value, H2 renders the predicate) so a datetime partition survives pruning. Tests: direct hiveDateTimeString render assertions (whole second, midnight, seconds, microseconds, trimmed) in both connectors; plus a hive end-to-end applyFilter test proving an escaped datetime partition value (H1) prunes-in against a datetime literal (H2). DATETIME partition read on a real cluster is live-gated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…ck-off H2 DONE (cf540ee). Render DATETIME/TIMESTAMP partition literals as Hive-canonical text in both hive and hudi copies. Adversarial design review verdict SOUND with a real Hive fixture (run17.hql) confirming the stored scale-0 form. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
… source Batch-0 (apache#65185 reverify) H3, hudi-only. After the HMS SPI cutover, a filtered query on a non-hive-style hudi-on-HMS table (hive_style_partitioning =false, the Hudi default) returned 0 splits (silent total row loss); the same query without a filter returned rows. Root cause: HudiConnectorMetadata.applyFilter set prunedPartitionPaths UNCONDITIONALLY from hmsClient.listPartitionNames (HMS hive-style names, "year=2024/month=01"). HudiScanPlanProvider feeds those verbatim to fsView.getLatestBaseFilesBeforeOrOn(partitionPath,...), which is keyed by Hudi RELATIVE STORAGE paths. For a non-hive-style table the physical layout is positional ("2024/01"), so fsView matched nothing -> 0 splits. The unpruned scan path (resolvePartitions -> listAllPartitionPaths) is use_hive_sync_partition-aware and returns the relative paths, which is why an unfiltered query worked. applyFilter bypassed that awareness (collectPartitions already had it); the stale HudiScanPlanProvider javadoc claiming the pitfall was "closed before the catalog flip" was outdated. Fix: make applyFilter's candidate source use_hive_sync_partition-aware, mirroring collectPartitions. hive-sync -> keep hmsClient.listPartitionNames (the hive-style name IS the relative layout, fsView accepts it) pruned by the existing prunePartitionNames (parsePartitionName, H1-unescaped). non-hive-sync (default) -> list the relative storage paths via metaClientExecutor.execute(listAllPartitionPaths(buildMetaClient(...))) — the SAME source the unpruned scan uses (net-neutral: resolvePartitions short-circuits once prunedPaths is set) — pruned by a new static prunePartitionPaths built on parsePartitionValues (handles positional AND hive-style layouts + unescapes). matchesPredicates made static so both prune helpers share it. No FSUtils/getPartitions/getLocation (relativizing custom HMS locations exceeds legacy/collectPartitions parity). Tests: migrated HudiPartitionPruningTest to a stub metaClient executor (feeds the canned listing offline; existing assertions unchanged, now exercising the non-hive-sync branch); added a positional-path test pinning that non-hive-style tables prune to RELATIVE paths (RED under the old HMS-name code), a hive-sync-branch test, and a direct prunePartitionPaths unit. Non-hive-style filtered read on a real cluster is live-gated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…verage Closes two low-risk coverage gaps flagged by the batch-0 adversarial review: - H2 design test-plan item 6 (DATE non-regression) was unimplemented: add a DATE partition + LocalDate predicate applyFilter test to both hive and hudi, proving a LocalDate literal is NOT diverted to the datetime render (stays String.valueOf = "2024-01-01") and prunes correctly. - The hive hiveDateTimeString test omitted the trailing-zero-trim ".1" case the hudi twin asserts; add it for parity. Test-only. hive 13/0-fail, hudi 12/0-fail, checkstyle clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…w + track-off H3 DONE (9c6fc58) + test-hardening (f0ee2ab). Batch-0 (H1-H4) all done. Final batch-wide adversarial review (3 skeptics) = CLEAN: four fixes correct, compose correctly, no regressions, scope complete (other connectors independently verified immune), all 10 new tests RED-capable. Registers one pre-existing residual (use_hive_sync_partition=true + non-hive-style layout, at legacy parity, out of H3 scope) for the D-PRUNE/relativization follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…atch 1 Records the apache#65185 reverify-fix series and batch-0 completion (H1-H4, all green, adversarial review CLEAN, one pre-existing residual registered). Next session: reverify batch 1 (connector-local M-series) + live-gated e2e. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
Contributor
FE UT Coverage ReportIncrement line coverage |
…gn current legacy) IcebergConnectorMetadata.computeRowCount reported total-records minus total-position-deletes with no equality-delete gate, so an iceberg MOR/CDC table with equality deletes surfaced an inflated row count to the CBO, while the sibling COUNT(*) pushdown (IcebergScanPlanProvider.getCountFromSummary) correctly declined -- an asymmetry that misleads join reorder / cardinality. Root cause is a parity target that moved under us, NOT a misread: the table- stats override was recon'd 2026-06-29 (HEAD 6252ecc) when legacy IcebergUtils.getIcebergRowCount was total-records - total-position-deletes with no gate, so the connector faithfully mirrored "no gate". Two days later upstream 32a2651 (apache#64648) refactored the count into getCountFromSummary(summary, ignoreDanglingDelete) and ADDED the equality gate (total-equality-deletes null or != "0" -> UNKNOWN), rewiring getIcebergRowCount to getCountFromSummary( summary, true). That commit is on this branch, so legacy now gates and the connector diverged. Cutover also routes iceberg-on-HMS (legacy HMSExternalTable :547 used the gated getIcebergRowCount) through computeRowCount -> newly activated regression there. Fix reproduces current legacy getCountFromSummary(summary, true) connector- locally (no fe-core edit): add TOTAL_EQUALITY_DELETES constant (byte-identical to the scan provider copy + fe-core), read + null-guard all three counters, and gate `!equalityDeletes.equals("0") -> -1` before the subtraction (-1 is already mapped to UNKNOWN by getTableStatistics). Position deletes still net out (legacy ignoreDanglingDelete = true). Corrected the three stale "does NOT gate" comments and rewrote the mutation-locked test equalityDeletesDoNotGate... -> equalityDeletesGateTableStatisticsToUnknown (asserts UNKNOWN; RED before fix). Reverses the prior clean-room-signed-off no-gate decision (P6.6-FIX-H4); user signed off the reversal 2026-07-11 (align current legacy over register-as- deviation). fe-core untouched; connector imports no fe-core; e2e live-gated. Build SUCCESS, 0 checkstyle, IcebergConnectorMetadataStatisticsTest 7/7 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
… H-4 docs + track-off M5 design (FIX-M5-design.md): frames the fix as a parity catch-up to upstream apache#64648 (legacy row count evolved to gate equality deletes after the 2026-06-29 recon), NOT a misread; records the user's 2026-07-11 sign-off to reverse the prior no-gate decision and align current legacy. Reconciles the three now-stale prior-decision docs with SUPERSEDED banners (preserving history, pointing to FIX-M5): P6.6-FIX-H4 design (key parity decision 1 + test #7 + mutation), its summary, and the iceberg flip-blockers tasklist H-4 entry. Ticks M5 in the reverify-fixes tracking table (+ batch-1 rolling notes: recon + adversarial red-team of all five items, all mechanisms confirmed at HEAD, no UNSOUND verdicts) and updates HANDOFF (M5 done; next M7->M6->M4->M2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…t to legacy parity
On the no-bound-S3 path (REST vended credentials: no static AK/SK/role -> chosenS3
empty), IcebergCatalogFactory.appendS3FileIO derives iceberg's client.region SOLELY
from firstNonBlank(props, S3_REGION_ALIASES). The array listed only {s3.region,
aws.region, region, client.region}, so a catalog whose region arrives via AWS_REGION,
iceberg.rest.signing-region, rest.signing-region, REGION, glue.region or
aws.glue.region yielded null -> client.region omitted -> iceberg S3FileIO falls to
the AWS DefaultAwsRegionProviderChain and the write commit fails "Unable to load
region". The comment claimed it mirrored legacy getRegionFromProperties but did not.
Widen S3_REGION_ALIASES to a verbatim connector-side copy of the fe-core S3Properties
@ConnectorProperty(isRegionField=true) region aliases (all 10, same declared order so
s3.region still wins on conflict). Connector-local literal copy (fe-connector must not
import fe-core). OSS/COS/OBS/Minio subclass region aliases are deliberately excluded
(irrelevant to an AWS-S3-backed vended REST catalog) and the comment is corrected to
say so rather than overclaiming a full getRegionFromProperties mirror.
New IcebergCatalogFactoryTest.buildCatalogPropertiesRestVendedResolvesRegionFromWidened
Aliases (region via AWS_REGION and via iceberg.rest.signing-region -> client.region);
RED before the widening. Existing s3.region test pins non-regression.
Build SUCCESS, 0 checkstyle, IcebergCatalogFactoryTest 62/62 green. e2e live-gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
FIX-M7-design.md (REST vended-cred client.region alias widening) + tick M7 in the reverify-fixes tracking table (progress row + detail + rolling log). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…chain (no-storage)
createS3TablesCatalog hard-failed when no fe-filesystem S3 storage was bound
("requires S3-compatible storage properties"). But an EC2 instance-profile
s3tables catalog carries only s3.region + a warehouse table-bucket ARN and no
static AK/SK/role, so S3FileSystemProvider.supports is false -> chosenS3 empty ->
throw on first access. Legacy IcebergS3TablesMetaStoreProperties supported this
via the SDK DefaultCredentialsProvider chain -- a fail-loud regression.
Invert the requirement: a REGION (from the bound storage or the raw props) is the
sole hard requirement for s3tables; credentials fall back to the SDK default chain
when no storage is bound.
- IcebergCatalogFactory.resolveS3Region(props): single source of truth over the
widened S3 region-alias set (M7); appendS3FileIO refactored to use it.
- IcebergConnector.resolveS3TablesRegion(chosenS3, props): static/package-visible
region gate (bound-storage region wins, else props), fail-loud only when neither
supplies a region. Unit-testable offline without a live S3TablesClient.
- createS3TablesCatalog drops the chosenS3-presence throw; buildS3TablesClient
re-signatured to (Optional<storage>, region) and derives credentials as
chosenS3.map(buildAwsCredentialsProvider).orElseGet(DefaultCredentialsProvider).
- Companion (data-plane): buildS3TablesCatalogProperties now emits client.region
from the props on the no-storage branch so S3FileIO honors an explicit s3.region
rather than only IMDS.
Tests (no Mockito): reframed s3TablesWithoutStorageFailsLoud ->
s3TablesWithoutStorageOrRegionFailsLoud (RED at HEAD); +4 resolveS3TablesRegion
gate tests (props fallback / widened alias / bound-storage-wins / fail-loud) and
+1 factory companion test (RED at HEAD). fe-core untouched; connector imports no
fe-core; no new production imports.
Build SUCCESS, 0 checkstyle, IcebergCatalogFactoryTest 63/63 + IcebergConnectorTest
19/19 green. Live-S3 e2e gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…ubgroup M5/M7/M6 done) FIX-M6-design.md (s3tables default-credential-chain fallback; companion promoted to required) + tick M6 in the tracking table (progress row + detail + rolling log) + HANDOFF: iceberg subgroup (M5/M7/M6) done, next M4 -> M2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…restore legacy)
Every query plan of a MaxCompute external table issued one full ODPS
getPartitions() round trip: fe-core PluginDrivenExternalTable.getNameToPartitionItems
(via initSelectedPartitions) calls ConnectorMetadata.listPartitions once per plan,
and the connector served it with a bare uncached ODPS SDK call (both fe-core and
connector layers uncached). On wide tables (tens of thousands of partitions) that
is multi-second per plan plus ODPS throttling risk -- a pure perf regression versus
the legacy MaxComputeExternalMetaCache.partitionValuesEntry (a TTL Caffeine cache)
the SPI migration deleted. Correctness is unaffected (BE re-filters). Hive/HMS
already re-homed its equivalent cache into the connector (HiveFileListingCache);
MaxCompute was the only connector still explicitly cache-less.
New MaxComputePartitionCache, a structural copy of HiveFileListingCache backed by
the shared fe-connector-cache framework (CacheSpec + MetaCacheEntry, byte-identical
contextual-only + manual-miss flags so the loader runs on the caller's TCCL-pinned
thread). Keyed by (db, table); loader injected as a PartitionLister for offline
testing. Held as a final field on the long-lived MaxComputeDorisConnector (metadata
is rebuilt per query), injected into the metadata; all three partition-listing
methods (listPartitions/listPartitionNames/listPartitionValues) route through it.
The four Connector REFRESH hooks (invalidateTable/Db/All/Partition) flush the cache
(invalidatePartition degrades to a whole-table flush). Config via
meta.cache.max_compute.partition.{enable,ttl-second,capacity}, defaults mirroring
legacy (ttl 86400s, capacity 10000). pom adds fe-connector-cache + Caffeine 2.9.3
(mirroring hive; the plugin zip bundles both automatically).
No fe-core edit; connector imports no fe-core; no Mockito (recording fakes). Tests
(MaxComputePartitionCacheTest, 9): 7 direct-cache + 2 metadata-integration encoding
"N plans / cross-method = 1 ODPS round trip" (behaviorally RED under mutation).
Build SUCCESS, 0 checkstyle, maxcompute module 113/113 (+1 live skip); plugin zip
bundles exactly one caffeine-2.9.3.jar + fe-connector-cache. Live-ODPS e2e gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
FIX-M4-design.md (maxcompute connector-side partition cache; records impl-subagent deviations + the verified caffeine-2.9.3 plugin-zip coherence) + tick M4 in the tracking table (progress row + detail + rolling log). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…utover) After the HMS cutover every hive table scans through the generic PluginDrivenScanNode, which enters batch/async split generation only when the connector opts in. HiveScanPlanProvider opted into neither flavor (supportsBatchScan=false, streamingSplitEstimate=-1), so a large partitioned hive scan materialized every file split of every selected partition into the FE heap in one synchronous pass -- worst on hive precisely because it is the biggest partition-count source. Legacy HiveScanNode went async at prunedPartitions.size() >= num_partitions_in_batch_mode (default 1024). Results stayed correct; FE heap + planning latency regressed. This is NOT a clean MaxCompute mimic: MaxCompute overrides only supportsBatchScan and relies on the SPI default planScanForPartitionBatch, correct only because its planScan is partition-set-scoped. Hive's planScan resolves from handle.getPrunedPartitions() and ignores the passed set, so inheriting the default would re-scan the whole pruned set per batch -> every partition's files emitted once per batch -> DUPLICATE ROWS. So hive overrides BOTH: - supportsBatchScan: true iff partitioned AND not transactional. ACID excluded deliberately (same isTransactional() accessor planScan branches on): the scan opens one metastore read transaction; per-batch resolution on background threads would open and leak a read transaction per batch. ACID partitioned tables keep the synchronous path (correct, not streamed). - planScanForPartitionBatch: resolves ONLY the batch via hmsClient.getPartitions(db, table, partitionBatch) (the batch strings are the HMS-rendered key=value/... names getPartitions accepts), then the SAME helpers as planScan (format detect, split size, hadoop conf, convertPartitions, listAndSplitFiles). No new imports. Deviations registered (fail-safe, results unaffected): - BATCH-ACID-SYNC (permanent, by-design): ACID partitioned tables stay synchronous. - BATCH-UNPRUNED-SYNC (resolved by M3, batch 2): a truly-unfiltered scan (no WHERE) keeps isPruned=false so fe-core's shouldUseBatchMode stays false; M3's ==NOT_PRUNED fix re-enables batch for it. No fe-core edit; connector imports no fe-core; no Mockito (real HmsClient / DirectoryLister / ConnectorSession fakes). New HiveScanBatchModeTest (4): supportsBatchScan true/false/false + batch-scoping size==1 (RED under the default -> 3 ranges, encoding the duplicate-splits bug). Build SUCCESS, 0 checkstyle, hive module 284/284 green. Large-partition async e2e gated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…all 5 done) FIX-M2-design.md (hive batch/async split path; registers BATCH-ACID-SYNC permanent + BATCH-UNPRUNED-SYNC resolved-by-M3) + tick M2 in the tracking table, mark batch 1 (M5/M7/M6/M4/M2) complete in the rolling log, and update HANDOFF (next = batch 2 M3->M1 fe-core generic node; M3 also resolves M2's unpruned-sync residual). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…acy parity (600s) Final adversarial review caught a parity miss in the M4 cache (c553c3c): the TTL default was copied from HiveFileListingCache's knob (external_cache_expire_time_seconds_after_access = 86400s / 24h), but the DELETED legacy MaxComputeExternalMetaCache.partitionValuesEntry used a DIFFERENT pair -- CacheSpec.of(Config.external_cache_refresh_time_minutes * 60, // default 10min = 600s Config.max_hive_partition_table_cache_num). // default 10000 Verified against the deletion commit (1da8836^) + Config.java defaults. The capacity (10000) coincidentally matched; the TTL was 144x too long -- a partition added directly in ODPS would have stayed invisible for up to 24h instead of the legacy ~10 min (both contextual-only expire-after-access, so this is a pure staleness regression vs the behavior this fix set out to restore). DEFAULT_TTL_SECOND 86400 -> 600; comment corrected to cite the real legacy knobs. No behavior change to capacity. MaxComputePartitionCacheTest 9/9 green (no test asserts the TTL value; the meta.cache.max_compute.partition.ttl-second override still works). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…(CLEAN) + M4 TTL fix Final review (wf_542c60b9-001, 5 per-fix skeptics + 1 cross-cut): M5/M6/M7/M2 and the whole-batch cross-cut CLEAN; M4 hit one medium defect (TTL default copied from the hive file cache's knob, 86400s, vs legacy MaxCompute partition cache's external_cache_refresh_time_minutes*60 = 600s) -> fixed in fca2884. Corrected FIX-M4-design.md, tracking table (rolling log + M4 detail), and HANDOFF. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…fix SecurityUtil split-brain) TeamCity #991951 (PR apache#65474, hive connector SPI migration) failed 49 external cases — hive / iceberg-on-HMS / hudi / mtmv / kerberos / tvf — all with the FE-side "Could not initialize class org.apache.hadoop.security.SecurityUtil". The cluster was healthy (366 passed, no BE crash/OOM); the "No backend available" lines are startup-only. Root cause: ThriftHmsClient.doAs pinned the TCCL to ClassLoader.getSystemClassLoader() before creating the metastore client. SecurityUtil.<clinit> -> new Configuration() captures that TCCL to reflectively load DNSDomainNameResolver: the system loader holds fe-core's hadoop copy while SecurityUtil/DomainNameResolver resolve from the plugin's child-first copy, so DomainNameResolver.isAssignableFrom(DNSDomainNameResolver) is false ("class DNSDomainNameResolver not DomainNameResolver") -> ExceptionInInitializerError, which permanently poisons SecurityUtil JVM-wide. doAs is the single choke point for both metastore creation (createFreshClient) and every RPC (execute), and both the Kerberos (ugi.doAs) and non-Kerberos (context.executeAuthenticated) authAction paths run inside it. Fix: pin getClass().getClassLoader() — the plugin child-first loader that loaded ThriftHmsClient, a strict superset of the system loader — matching iceberg/paimon TcclPinningConnectorContext and HiveConnectorMetadata's stats pins. Test: ThriftHmsClientDoAsClassLoaderTest drives DoAsTcclProbe through an isolated child-first loader (mirrors OdpsClassloaderIsolationTest) so getClass().getClassLoader() differs from the system loader, making the exact bug observable: RED before the fix (PIN_WRONG_SYSTEM_LOADER), GREEN after. fe-connector-hms 40/40 green, 0 checkstyle. Real-cluster e2e rerun of the 49 cases still owed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…er (latent eager-UGI split-brain) Latent-edge companion to the ThriftHmsClient.doAs fix (not hit by the 49 failing #991951 cases, hardened here per request). HiveConnector.buildPluginAuthenticator runs on the (unpinned) createClient thread; a catalog declaring hadoop.security.authentication=kerberos WITHOUT a principal/keytab falls back (AuthenticationConfig.getKerberosConfig) to a HadoopSimpleAuthenticator whose ctor EAGERLY calls UserGroupInformation.createRemoteUser -> SecurityUtil.<clinit>. Its internal new Configuration() captures the TCCL, so an unpinned thread would load hadoop's DNSDomainNameResolver from fe-core's system-loader copy and split-brain-poison SecurityUtil against the plugin copy — the same failure doAs guards. buildHadoopConf sets only the OUTER conf's loader, not the TCCL that SecurityUtil's own Configuration reads, so a thread pin is required here too. Fix: pin HiveConnector.class.getClassLoader() around the whole resolution (try/finally), mirroring HudiConnector.metaClientExecutor and the doAs fix. Also refreshes the now-stale comment in createClient that described doAs as pinning the system classloader. Test: HiveConnectorPluginAuthenticatorTcclTest observes, via a TCCL-recording properties map, that the method body runs under the plugin loader and restores the caller's TCCL — RED without the pin (observed = caller marker), GREEN with it. fe-connector-hive 186/186 green (incl. the 5 existing buildPluginAuthenticator cases), 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
… + HANDOFF RCA, fix design, and per-fix summary for the ThriftHmsClient.doAs / buildPluginAuthenticator TCCL classloader split-brain (CLR1 92004ef, CLR2 15d3df1). Records the e2e owed (real-cluster rerun of the 49 SecurityUtil cases) and the outlier classification (test_hdfs_parquet_group0 = BE ASAN mem flake, unrelated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
only for testing #65473