[refactor](catalog) Catalog spi 11 hive#65473
Open
morningman wants to merge 230 commits into
Open
Conversation
…progress → P7 hive P6 iceberg migration + tests are complete and squash-merged into upstream branch-catalog-spi as apache#64688 (8b391c7). Overwrite the rolling HANDOFF and sync all related progress trackers to reflect "P6 DONE → next = P7 hive/HMS". - HANDOFF.md: full rewrite to a P7-kickoff handoff (plan pointers to master-plan §3.8 + connectors/hive.md, sub-phases P7.1-P7.5, R-002 ACID gate, reusable ops/commit rules). - PROGRESS.md: §一 P6→100%/merged + P7→active + global ~64%; §二 kanban iceberg→100%/merged; §三 P6 block→P7 block; §四 new 2026-07-05 entry; §六 D-073/DV-049 counts; §七 session status. - connectors/iceberg.md: status rows → ✅ merged apache#64688; drop DV-038 flip-blocker label; progress-log entry. - connectors/hive.md: status → active P7 target; progress-log entry. - tasks/P6-iceberg-migration.md + P6.6-flip-blockers-tasklist.md: status banners → ✅ COMPLETE / merged apache#64688. Note: apache#64688 removed the native iceberg subsystem but 23 fe-core datasource/iceberg/ HMS-iceberg support classes remain (iceberg-on-HMS still routes through fe-core); they are deleted with P7 hive (阶段四, decision D5/Q3=B). Recorded in all touched docs. Docs only, no product code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
10-agent code-grounded recon of fe-core datasource/hive (52 files) + a supplemental type-coupling pass. Produces tasks/P7-hive-migration.md (P7.1-P7.5 split, old->new mapping, cutover mechanics, cross-connector deletion ordering, SPI gaps, 8 open decisions) and syncs HANDOFF / PROGRESS / connectors/hive.md. Key facts corrected vs stale plan: reverse instanceof 31->85 (33 files), HMSTransaction 1866->1895, HMSExternalTable 1293->1332. Coverage critic closed an instanceof-only blind spot (CatalogFactory:134, GsonUtils 366/447/471 compat, HudiUtils/IcebergHMSSource type coupling). Confirmed via decisions-log that iceberg/hudi-on-HMS ownership is already settled (D-020 per-table SPI provider delegation + D-019 hudi live cutover into P7) - not re-litigated. No product code, no new decisions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…n + lock config-threading decision P7.1 起步 recon（3 并行 agent + 直读 HEAD）核清并校正 spec： - 通用 DDL 桥 = PluginDrivenExternalCatalog（逐条 override + 内联 cache/editlog；连接器只实现纯 SPI）。 - TRUNCATE 硬缺口：SPI 无 truncateTable、桥未 override → 须加 seam + 桥 override（复用 OP_TRUNCATE_TABLE）。 - shared converter 逐字拷 properties（不 parse），但丢每列默认值（恒 null → 破坏 hive 列默认值 + DLF guard）+ 丢 LIST/RANGE partition value。 - stats/partition 写 4 法仅 HMSTransaction 消费 → 推 P7.3（本阶段加即死代码）；rename hive 今天不支持 → 保持 SPI default throw。 - 模板 = IcebergConnectorMetadata（HMS-backed + DLF guard + 插件侧 SchemaBuilder）。 产出 tasks/P7-hive-migration.md 末尾 P7.1 逐 task 拆解（T01–T11，范围=仅 DDL 写路径）+ recon 结论块。 唯一阻塞决策已由用户裁定：全局建表默认（默认文件格式 / 是否允许分桶表）迁到插件后取值来源 = 方案 A （fe-core 建连接器时把两个全局 Config 值注入为连接器属性默认，插件只读属性、零行为回归）。 无代码改动（仅文档 + 决策）。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…iven bridge route The generic SPI DDL bridge (PluginDrivenExternalCatalog) had no truncate route: the base ExternalCatalog.truncateTable throws "not supported" when metadataOps == null, so every SPI catalog (iceberg/paimon today) rejects TRUNCATE. Hive needs it after the P7 cutover. - ConnectorTableOps.truncateTable(session, handle, partitions): new default-throw seam (partitions null/empty = whole table). Default preserves iceberg/paimon "not supported". - PluginDrivenExternalCatalog.truncateTable(...): resolves the table by REMOTE names, dispatches to metadata.truncateTable, then emits the same TruncateTableInfo edit log the base op writes and refreshes the local table cache via RefreshManager.refreshTableInternal (mirroring legacy HiveMetadataOps.afterTruncateTable). forceDrop/rawTruncateSql carry no external semantics and are ignored, matching the legacy path. - PluginDrivenExternalCatalog.replayTruncateTable(info): refreshes the cache on follower replay (the base delegates to metadataOps.afterTruncateTable, a no-op for PluginDriven). No connector overrides truncateTable yet (HiveConnectorMetadata comes in a later P7.1 task), so behavior is unchanged pre-flip; verified by compile + checkstyle 0 + import-gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
… handoff → T02 Update rolling handoff + task spec: T01 (TRUNCATE SPI seam) done & green (c022297); next session resumes at T02 (plugin write DTOs) → T03/T04 (write client). Archive the T03 converter porting pointers (HiveUtil line refs + the fe-core couplings to break: dorisTypeToHiveType inverse over ConnectorType, DORIS_VERSION const, HiveProperties, hiveTextCompression session var, format/compression tables) so the next session does not re-recon. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…create/drop db+table, truncate) + converter Builds the SPI-clean write client in fe-connector-hms so the hive connector can issue CREATE/DROP DATABASE, CREATE/DROP TABLE and TRUNCATE without fe-core. The metadata layer (HiveConnectorMetadata overrides), config threading and the recording-fake client tests land in follow-up steps; this stays non-live (no cutover until the hive connector is flipped on). - HmsCreateTableRequest / HmsCreateDatabaseRequest: write-side specs mirroring legacy HiveTableMetadata / HiveDatabaseMetadata, connector-api + JDK types only. Columns carry data + partition columns; per-column defaults ride on ConnectorColumn.getDefaultValue(). Text-compression default and doris.version are threaded on the request (the plugin must not import fe-core Config/Version). - HmsTypeMapping.toHiveTypeString: reverse of the read mapping, SPI-clean equivalent of HiveMetaStoreClientHelper.dorisTypeToHiveType. Switches on the Doris PrimitiveType names ConnectorColumnConverter emits; unsupported types throw (parity with legacy) rather than emit a bogus type. - HmsWriteConverter: faithful port of HiveUtil.toHiveTable/toHiveDatabase plus HiveProperties.setTableProperties (serde vs table property split). Per-format input/output/serde + compression defaults copied verbatim; MANAGED_TABLE type and the "doris external hive table" storage tag preserved. - HmsClient: add createDatabase/dropDatabase/createTable/dropTable/truncateTable as default-throwing seams so read-only implementations (partition-pruning test fakes) keep compiling; ThriftHmsClient overrides all five via the existing execute(...) auth+pool framework, building SQLDefaultConstraint from column defaults (equivalent to legacy createTableWithConstraints). Verify: fe-connector-hms compile SUCCESS + checkstyle 0 (main+test) + check-connector-imports clean + 28 unit tests green (reverse type mapping + converter). No connector overrides these yet, so behavior is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…ndoff → T05 Record the plugin-side HMS DDL write client (fdf577e) as complete in the task breakdown + rolling handoff, and point the next unit at T05→T07 (HiveConnectorMetadata DDL overrides) with the two request fields the metadata layer / config-injection locus must populate (defaultTextCompression, dorisVersion). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…olumn defaults + explicit-partition-values signal Thread two pieces of the neutral CREATE TABLE request that the hive plugin needs to reproduce legacy HiveMetadataOps.createTableImpl parity; both are additive and ignored by connectors that build their schema from name/type/nullable/comment only (iceberg/paimon/maxcompute build-time paths do not read either). - Per-column default value: expose ColumnDefinition.getDefaultValueString() (Column.getDefaultValue() equivalent, via DefaultValue.getValue()) and thread it onto ConnectorColumn.defaultValue in CreateTableInfoToConnectorRequestConverter (was hardcoded null). Hive builds metastore default constraints from it and gates its DLF catalog on per-column defaults. - Explicit-partition-values signal: the converter drops LIST/RANGE partition value expressions (still not lowered), so add ConnectorPartitionSpec.hasExplicitPartitionValues() and set it from PartitionTableInfo.getPartitionDefs(). Lets hive keep rejecting `PARTITION BY LIST(dt) (PARTITION p VALUES IN (...))` (external tables discover partitions from the data layout) instead of silently ignoring it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…nfig defaults via connector environment The fe-connector-hive plugin cannot read FE Config, so surface the two FE-global CREATE TABLE toggles + the build version through the connector environment channel (DefaultConnectorContext.buildEnvironment) — the same "C4" mechanism already used for hive_metastore_client_timeout_second. Chosen over injecting them into the catalog property map so they are NOT persisted into the catalog image/edit-log, do not appear in SHOW CREATE CATALOG, and refresh from current Config on each connector rebuild. - hive_default_file_format <- Config.hive_default_file_format - enable_create_hive_bucket_table <- Config.enable_create_hive_bucket_table - doris_version <- Version build id (legacy ExternalCatalog.DORIS_VERSION_VALUE) A user-set per-table file_format property still wins (resolved plugin-side); the plugin reads these keys via ConnectorContext.getEnvironment(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…override (create/drop db+table, truncate) + tests Port legacy HiveMetadataOps create/drop database, create/drop/truncate table into the plugin so the "hms" SPI path reproduces its behavior (still off — cutover is P7.5). All property interpretation stays plugin-side (fe-core parses no hive properties). - T05 database: supportsCreateDatabase()=true; createDatabase (location -> URI, comment -> description, rest -> params); dropDatabase(force) cascades table drops then the db. - T06 createTable (faithful port of createTableImpl): owner default (session.getUser()), transactional-create reject, file_format default (env, user prop wins), doris.-prefixed round-trip params, LIST-only partitions + explicit-partition-values reject, DLF per-column-default guard, bucket gate (env enable + hash-only), doris.version + text compression default threaded onto the write request. - T07 dropTable (transactional-table reject, via handle params = AcidUtils parity without a hive-exec dep) + truncateTable; renameTable intentionally left as the SPI default throw (hive has no rename). HiveConnector now passes ConnectorContext to the metadata. - T10 tests: recording HmsClient fake + fake session/context (no Mockito), 20 DDL cases covering the guards, defaults, round-trip params, and force cascade. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…ecord 2 decisions; handoff → P7.2/P7.3 P7.1 (hive DDL metadata write path: create/drop db+table, truncate) is done and green across fe-core + fe-connector-hive (28 unit tests, checkstyle 0, import-gate clean; non-live, cutover is P7.5). Records the two user-signed decisions this round — config threading via the connector environment channel (not the persisted catalog property map), and keeping the legacy "explicit partition values not supported" rejection (via a threaded flag, not silent ignore) — and repoints the next session at P7.2 (events) / P7.3 (HMSTransaction, critical path, R-002). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…reakdown (no impl code) Enter the hive transaction/write-path subphase (critical path, R-002). Code-grounded recon (4 parallel agents, HEAD-verified) + user sign-off on three open decisions; append the P7.3 task breakdown to the phase spec and roll the HANDOFF forward. No implementation code this session. Key recon findings (baked into tasks/P7-hive-migration.md): - The generic write path already exists (built during iceberg P6): PhysicalConnectorTableSink -> PluginDrivenTableSink -> PluginDrivenInsertExecutor + ConnectorTransaction/ConnectorWritePlanProvider /PluginDrivenTransactionManager. fe-core bridge needs ZERO changes. P7.3 = fold hive INSERT into it + mirror IcebergConnectorTransaction/WriteContext. - Missing HMS write primitives (addPartitions/dropPartition/stats + ACID openTxn/allocateWriteId/lock/commitTxn/abortTxn/getValidWriteIds) all already exist on the vendored HiveMetaStoreClient -> just surface them on HmsClient + one-line execute() wrappers; no new thrift plumbing. - Latent regression to fix (T07): plugin HiveScanRange.populateTransactional HiveParams fills the delete-delta directory but drops file names (old fe-core filled both) -> transactional tables would silently mis-read once hive is flipped. Signed decisions (spec OQ section marked done): - OQ-RTX = add a neutral per-query finish callback in fe-core (Trino-aligned), replacing the hard QeProcessorImpl:210 -> Env.getHiveTransactionMgr call; hive read-txn manager moves off Env into the plugin. - OQ-ACID-WRITE = preserve current behavior (non-ACID write + ACID read migrated; full-ACID table write stays rejected) to control R-002. - OQ-LOCK = keep as-is (read-side shared HMS lock, no heartbeat). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…e names in HiveScanRange The plugin scan-range builder encodes each delete-delta as "dir|file1,file2" (HiveScanRange.Builder#acidInfo), but populateTransactionalHiveParams only decoded the directory and dropped everything after '|'. The old fe-core path (HiveScanNode.setScanParams) set BOTH directoryLocation and fileNames on TTransactionalHiveDeleteDeltaDesc; without the file names the BE cannot locate the delete records and silently under-deletes on transactional Hive reads. Parse the file names after '|' and set them; a directory-only encoding (no '|') still sets the directory and carries no file names. Add HiveScanRangeAcidTest covering the encode/decode round-trip, the directory-only case, and the non-transactional (no params emitted) case. This is the isolated must-fix regression called out in the P7.3 recon; the remaining T07 work (moving AcidUtil/HiveTransaction/AcidInfo into the plugin and wiring the producer that calls acidInfo) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…resh HANDOFF for next session Mark the isolated ACID delete-delta file-name regression as landed (c30fa15) in both HANDOFF and the P7.3 task table; note the remaining T07 work (move AcidUtil/HiveTransaction/AcidInfo/DeleteDeltaInfo into the plugin + wire the producer) and keep T01-T05 (atomic write/transaction batch) as the next task. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…-finish callback seam Replace the hive-specific hook hardcoded in the generic query-cleanup path (QeProcessorImpl.unregisterQuery -> Env.getCurrentHiveTransactionMgr().deregister) with a connector-agnostic QueryFinishCallbackRegistry: a connector registers a per-query cleanup callback and fe-core drains all callbacks exactly once at query finish. The hive read path now registers its read-transaction commit through the seam instead of relying on fe-core naming it. Aligns with the engine-driven query/transaction lifecycle model (e.g. Trino); implements OQ-RTX = (a). - QueryFinishCallbackRegistry: register / runAndClear (idempotent, exception-isolated). - QeProcessor: add registerQueryFinishCallback SPI; QeProcessorImpl drains generically. - HiveScanNode.doInitialize: register the deregister callback via the seam. - 6 unit tests (no Mockito): run-once, no-op-when-empty, order, idempotency, failure isolation, per-query scoping. Behavior preserved: drain key printId(queryId) matches the callback key hiveTransaction.getQueryId() (same equality the old direct deregister relied on); the early lock-release path (HiveScanNode) still commits eagerly and the finish callback then no-ops via idempotent deregister. HiveTransactionMgr stays in Env for now (still used by the legacy fe-core hive read path); it moves into the plugin later with the read-side ACID migration. Build SUCCESS / checkstyle 0 / import-gate net / 6 tests pass. Non-live (cutover P7.5). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…ing + refresh HANDOFF; correct T07 read-side ACID boundary - Mark T08 connector-agnostic query-finish seam as landed (21aa306); HiveTransactionMgr stays in Env, moves to plugin later with T07. - Record this session's recon correction to T07: the read-side ACID *produce* half is not an independent slice -- it is entangled with fe-core filesystem/ StorageProperties/FileCacheValue and shares ACID HmsClient primitives with the write batch (T01/T02); the plugin scan path currently skips ACID dirs, so producing ACID reads needs new scan-descent logic. Should advance with the write batch, not in parallel. - Roll HANDOFF forward for next session (atomic write/txn batch remains the core). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…implementation design (no impl code) Converts the P7.3 write/transaction batch from spec-level task breakdown into a single executable design doc + ordered build sequence, after a 6-agent HEAD-accurate porting recon (HMSTransaction, iceberg template, SPI surface, HmsClient, read-side ACID). Key recon corrections (now locked in the design): - Write path is a FOUR-class split mirroring iceberg (HiveConnectorTransaction + immutable HiveWriteContext + separate HiveWritePlanProvider + one-line HiveConnectorMetadata.beginTransaction); planWrite is NOT in the metadata. - fe-core insert/txn bridge needs ZERO changes; profileLabel()="HMS" reuses the existing TransactionType.HMS enum. - Read-side ACID (production half) is NOT entangled with the write-transaction machinery; the feared fe-core drag drops out at the plugin boundary — only the pure directory-name parse + hive-common ValidWriteIdList algorithm ports. - hive is not in SPI_READY_TYPES, so the whole plugin batch is dormant/safe (compiles + unit-tested, zero live routing), same as iceberg/paimon pre-cutover. - Cutover (fe-core retype, catalog identity flip, Env HiveTransactionMgr removal, SPI_READY flip, delete legacy) is explicitly OUT of this batch (later P7.4/P7.5). Delivery: user decided the whole plugin-side batch lands as ONE atomic commit, built internally in compilable dependency-order increments INC-1..INC-5 that are not individually committed. Design doc §6 is the resumable build order; §8 is the atomic-commit management strategy across sessions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
…mitives, uncommitted in tree) + refresh HANDOFF Atomic-batch increment INC-1 is implemented and green in the working tree but intentionally NOT committed (the batch lands as one feature commit once INC-1..INC-5 are all green; the uncommitted tree is the cross-session WIP carrier per the design doc §8). INC-1 (fe-connector-hms): 9 HmsClient metastore primitives (5 write: addPartitions / updateTable+PartitionStatistics / dropPartition / partitionExists; 4 read-ACID: openTxn / commitTxn / getValidWriteIds / acquireSharedLock) + 4 DTOs (HmsCommonStatistics, HmsPartitionStatistics, HmsPartitionWithStatistics, HmsAcidConstants) + HmsWriteConverter partition/ stats helpers + a recording-fake IMetaStoreClient unit harness. Module test SUCCESS (49 hms cases), checkstyle 0, connector import-gate clean; verified by a 3-lens adversarial review (0 production defects; 2 test-coverage gaps fixed). This commit updates only the design-doc INC-1 checklist (marked done, with the four faithful-migration deviations) and the rolling HANDOFF. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
…+ NameMapping in fe-connector-hive, uncommitted) + refresh HANDOFF INC-2 lands, inside fe-connector-hive (uncommitted in the working tree per the atomic-batch model), the minimal write-path leaf material the connector transaction needs: - HiveWriteUtils (new, package-private): byte-faithful port of the pure path helpers (isSubDirectory/getImmediateChildPath/pathsEqual + private sameFileSystem/normalizeUriPart/normalizePath) and mergePartitions from HEAD HMSTransaction. fe-core-free (Hadoop Path + java.net.URI + shared thrift only; no guava/lombok). mergePartitions made static. - NameMapping (new, co-move): JDK-only plugin copy of fe-core datasource.NameMapping (no lombok/guava), the action-map key of the connector transaction. Only leaf type co-moved -- it has no plugin peer. R6 resolution after reading plugin-side first: the statistics trio already has INC-1 plugin peers (HmsCommonStatistics/HmsPartitionStatistics/ HmsPartitionWithStatistics, with full merge/reduce ops), so they are NOT co-moved; HivePartition is deferred to INC-3 (its fe-core Column drag is dead on the write path, and HmsPartitionWithStatistics supersedes its SD role, so the PartitionAndMore descriptor shape is an INC-3 body-port decision). Tests: HiveWriteUtilsTest (12) + NameMappingTest (4), each pinning the contract's intent. fe-connector-hive build SUCCESS, 16 new tests green, checkstyle 0, import-gate clean. Zero changes to existing files. Only this doc/HANDOFF commit is made; INC-1+INC-2 code stays uncommitted (single atomic feature commit at end of batch). Next: INC-3 (HiveWriteContext + HiveConnectorTransaction core + beginTransaction). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz
…ted) + land port-map + refresh HANDOFF INC-3 = fe-core HMSTransaction(1895) faithfully ported to plugin HiveConnectorTransaction(1697) + HiveWriteContext(89) + beginTransaction one-liner + HiveWriteUtils equalsIgnoreSchemeIfOneIsS3/toPartitionValues + HiveConnectorTransactionTest(9, FS-free half) + FakeConnectorContext. All feature code stays UNCOMMITTED in the working tree (atomic-batch model, design §8); this commit is docs only. Verified independently: fe-connector-hive module 62 tests green (no regression), checkstyle 0, import-gate clean, compile+test BUILD SUCCESS; riskiest spans (D6 MPU/FS build, commit/rollback order, classification / NEW->APPEND downgrade, full-ACID-write guard) spot-checked faithful. Remaining INC-3: FS-heavy committer tests (via resolveObjectStoreFileSystem seam) + full adversarial fidelity review; then INC-4, INC-5, one atomic feature commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
…2 faithful) + dup-key fail-loud fix + refresh HANDOFF 12-agent adversarial clean-room review of HiveConnectorTransaction vs HEAD HMSTransaction: 11/12 dimensions faithful; 4 low discrepancies, only 1 CONFIRMED defect -- convertToInsertExistingPartitionAction silently deduped via HashMap.put where HEAD's Collectors.toMap fails loud on a duplicate key (silent dropped write vs abort, Rule 12). Fixed with two put(...)!=null -> IllegalStateException guards + testDuplicatePartitionValuesFromHmsFailsLoud. Other 3 adjudicated BENIGN (getTable begin-snapshot; redundant auth wrap; dropped fe-core doNothing() debug-point test seam). Code fix rides in the uncommitted INC-1..INC-3 WIP batch (atomic model, design doc S8); this commit is docs only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
… complete; refresh HANDOFF + design doc Added 4 committer filesystem unit tests to HiveConnectorTransactionTest (injected via a resolveObjectStoreFileSystem override returning a recording ObjFileSystem/ObjStorage): MPU completeMultipartUpload on commit (ETags sorted by part number), MPU abortMultipartUpload on rollback, rollback idempotency (second rollback no-throw, abort exactly once), and addPartitions called once with the full list + SD columns rebuilt from the table (GAP-7 / GAP-4). fe-connector-hive now 67 tests green (was 63), checkstyle 0, import-gate clean. INC-3 is fully done (main + fidelity review + FS tests); next is INC-4 -> INC-5. Feature code stays uncommitted (atomic-batch model); this commit is docs only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
…+ DEC-1 hash-no-sort + DEC-2 guard widen, uncommitted) + land INC-4 port-map + refresh HANDOFF INC-4 complete (uncommitted in tree, atomic-batch model): HiveWritePlanProvider + HiveSinkHelper (byte-faithful port of planner.HiveTableSink.bindDataSink) + getWritePlanProvider registration + capability gates + 20 tests. Two user decisions: DEC-1 = new generic connector-agnostic requiresPartitionHashWrite capability + a hash-without-sort distribution arm in PhysicalConnectorTableSink (byte-exact to legacy PhysicalHiveTableSink; the one authorized fe-core change, supersedes design section 4.5 "zero fe-core change"); DEC-2 = widen the transactional-write reject to any transactional table (full-ACID + insert-only), matching legacy InsertIntoTableCommand. Verified: fe-connector-hive 87 tests green (67+20); fe-core distribution/contract tests 16 green (no MaxCompute regression); checkstyle 0; import-gate clean. Clean-room adversarial review 3/8 dimensions ran independently (all faithful); remaining 5 truncated by an account session limit, covered by manual diff + strong behavior-pinning tests; sole finding low/benign/pre-existing. Docs only (feature code stays uncommitted until INC-5 + FINAL atomic commit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
… read-txn lifecycle, uncommitted) + clean-room review + refresh HANDOFF/design + land INC-5 port-map INC-5 (read-side ACID, dormant) implemented + green in the working tree (feature code stays uncommitted per the atomic-batch model; this commit is docs only): - HiveAcidUtil = pure port of AcidUtil.getAcidState onto raw Hadoop FileSystem - HiveReadTransaction + HiveReadTransactionManager (port of HiveTransaction/Mgr) - HiveScanPlanProvider.planAcidScan descent + acidInfo producer wiring - HiveTableHandle.isTransactional/isFullAcid from table params (D8) fe-connector-hive 104 tests green (87 + 17 new), checkstyle 0, import-gate clean. Clean-room adversarial fidelity review (6 dimensions, 13 agents): 1 CONFIRMED defect fixed (insert-only tables were mis-marked transactional_hive; now gated on isFullAcid), 1 self-caught defect fixed (per-table read txn, no cross-table reuse), test coverage hardened (visibility filter / isValidBase discrimination / not-enough-history). All five increments (INC-1..INC-5) now done + green; next step is the FINAL atomic feature commit (see HANDOFF). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
…+ write plan provider + ACID read producer (dormant) Port the Hive DML/write path and the ACID read path into the self-contained fe-connector-hive / fe-connector-hms plugins, mirroring the iceberg connector template, so fe-core no longer needs native Hive write logic. Dormant: HIVE is not in SPI_READY_TYPES yet, so this compiles and is unit-tested but not routed at runtime; cutover (flip the flag, retype the fe-core write chain, swap the read-txn manager, remove legacy datasource/hive) lands in a later atomic batch. fe-connector-hms (write-side HMS client): - HmsClient/ThriftHmsClient: open/commit/abort transaction, acquire locks, get valid write ids, add/alter partitions, table & partition statistics; convertTable now carries bucket columns / bucket count. - HmsWriteConverter: partition + statistics conversion for writes. - New HmsAcidConstants, HmsCommonStatistics, HmsPartitionStatistics, HmsPartitionWithStatistics. fe-connector-hive: - Write chain: HiveConnectorTransaction (HMS commit protocol + rollback), HiveWriteContext, HiveSinkHelper, HiveWritePlanProvider, HiveWriteUtils, NameMapping; HiveConnectorMetadata.beginTransaction override; full-ACID writes are hard-rejected. - Read chain: HiveAcidUtil (pure directory-name parsing + snapshot selection), HiveReadTransaction / HiveReadTransactionManager (per-table read-txn lifecycle); HiveScanPlanProvider.planAcidScan emits acidInfo only for full-ACID tables, insert-only tables read plain files by snapshot. - HiveTableHandle.isTransactional/isFullAcid derived from table parameters. Connector-agnostic write-plan seam (fe-connector-api + fe-core), default-off: - ConnectorWritePlanProvider on Connector + ConnectorContractValidator; PluginDrivenExternalTable / PhysicalConnectorTableSink consume it with no source-specific code in fe-core. Verified fresh: fe-connector-hive 104 tests, fe-connector-hms 49, fe-core write-seam 16 (all 0 failures/skips); checkstyle 0 violations across the touched modules; connector import-gate clean. Refs: apache#65185 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
… → refresh HANDOFF, next = cutover INC-1..INC-5 (hive plugin-side write transaction + committer + write plan provider + ACID read producer, dormant) are now committed as one atomic feature commit 0b19506 (40 files, 22 new + 18 modified). Working tree no longer carries the batch as uncommitted WIP. Refresh HANDOFF: current-state + this-session log now record the FINAL commit and its fresh verification (hive 104 / hms 49 / api 55 / fe-core 16 tests, checkstyle 0, import-gate clean); next-session task retargeted from FINAL to cutover (P7.4/P7.5): flip SPI_READY_TYPES, retype fe-core write chain, wire the plugin read-txn manager + query-finish deregister, drop legacy HiveTransactionMgr + datasource/hive (guarding cross-connector delete order), and run the ACID integration suite (R-002). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
…ormat dispatch keystone, dormant) Add Connector.getScanPlanProvider(ConnectorTableHandle) so one connector can select a different ConnectorScanPlanProvider per table. Selection MUST happen at provider-acquisition time (not inside a single dispatching provider) because ConnectorScanPlanProvider has handle-less methods (e.g. appendExplainInfo) and providers are built fresh/stateless per call — a returned provider must already be bound to the right backing scanner for the handle. fe-core: PluginDrivenScanNode routes all 11 provider look-ups through a single resolveScanProvider() helper keyed on currentHandle. The SPI default ignores the handle and returns the connector-level provider, so every existing connector (es/jdbc/trino/max_compute/ paimon/iceberg/hive) is byte-identical and this change is fully dormant (no connector overrides it yet). This is the keystone that lets a post-cutover heterogeneous hms catalog (plain-hive + iceberg-on-HMS + hudi-on-HMS under one gateway connector) route each table to the right scanner; the first real delegation lands in a later sub-batch. Tests: ConnectorScanProviderSelectionTest (SPI default delegation, null-passthrough, per-handle override); PluginDrivenScanNodeScanProviderSelectionTest (node passes currentHandle). Updated PluginDrivenScanNodeDeleteFilesTest/BatchModeTest to stub the arg overload (bare Mockito mocks do not run the real default method). fe-connector-api 58 tests, fe-core PluginDrivenScanNode* 76 tests, checkstyle 0 (api + core), import-gate clean. Refs: apache#65185 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
… decompose cutover into 6 sub-batches + refresh HANDOFF Land the cutover scope map (P7-cutover-scope-map-2026-07-06.md, from the 6-agent recon wf_536a2968-2c8): the "cutover" is not a compact flip but the entire P7 tail decomposed into sub-batches A-F (multi-format dispatch seam / iceberg-on-HMS delegation / hudi live cutover / event pipeline / DLA+write retype / flip+delete). Land sub-batch A design doc (P7.4-scan-provider-per-table-seam-design.md), marked DONE for commit 0923077. Refresh HANDOFF: current state = A done + dormant, next = sub-batch B (iceberg-on-HMS delegation), with the delete-order constraint and each sub-batch's deferred user decisions recorded. Refs: apache#65185 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
…o HMSExternalCatalog ExternalMetaIdMgr.replayMetaIdMappingsLog propagates the master's synced-event-id cursor on every FE by casting the live catalog to HMSExternalCatalog. The cast is gratuitous: updateMasterLastSyncedEventId only needs the catalog id, which the log already carries. Once an HMS catalog is served by a generic PluginDrivenExternalCatalog, that cast throws ClassCastException during edit-log replay and wedges FE startup. Key the cursor update by catalogId (long) instead. Behavior is byte-identical today (the live catalog is still HMSExternalCatalog); this removes a latent replay crash that would otherwise block routing an HMS catalog through the plugin connector SPI. - ExternalMetaIdMgr: drop the (HMSExternalCatalog) cast + now-unused import; pass catalogId. - MetastoreEventsProcessor.updateMasterLastSyncedEventId(HMSExternalCatalog, long) -> (long catalogId, long eventId); its sole caller is the replay path above. - Test: replaying a fromHmsEvent cursor log against a non-HMSExternalCatalog live catalog must not throw and must propagate the cursor keyed by catalogId. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
… findings + refresh HANDOFF Two code-grounded recons this session (iceberg-on-HMS delegation; HMS metastore-event pipeline) plus direct HEAD verification establish that both are FLIP-GATED: the per-table scan-provider seam only fires inside PluginDrivenScanNode (instanceof PluginDrivenExternalTable), and only a PluginDrivenExternalCatalog holds a plugin Connector. A legacy HMSExternalCatalog has neither, so neither delegation nor the SPI-driven event poller can be wired until the HMS catalog/table are retyped to the generic plugin-driven classes. The prior "6 independent pre-flip sub-batches" framing is corrected: the remaining HMS migration is essentially one flip, and format-delegation / events / hudi are facets of it. - iceberg-on-hms-delegation-findings: why flip-gated; cross-plugin sibling-connector handoff (co-packaging rejected — 2nd AWS SDK poisons S3 JVM-wide); design sketch; deferred deletions / per-column stats (recommend DROP) / IcebergUtils 6-member extraction. - hms-event-pipeline-findings: why flip-gated; primary blocker (events do structural Env object-graph mutation, not just cache invalidation, which the neutral ConnectorMetaInvalidator cannot express); recommended Model B (thin fe-core role driver + plugin poll-once SPI); ExternalMetaIdMgr droppable for HMS but opcode 470 must stay replayable. - HANDOFF: next = execute the flip (retype HMS catalog/table to generic), starting with a recon+design of the retype; the replay-CCE fix (a6dc782) is the one pre-flip de-risking item already landed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
…s post-HMS-cutover branch The third-party apache#65185 review targeted branch-catalog-spi (P1-P6 only, no hive/hms). This branch already completed the HMS SPI cutover ("hms" in SPI_READY_TYPES), so every finding is re-verified against current code (located by symbol; report line numbers are stale) via a multi-agent workflow: 3 recon + 34 finding-cluster verifiers + adversarial refutation of each surviving high/medium, cross-checked against deviations/decisions-log. 79 findings classified. Headline: the report's "dormant P7 hudi landmines" are now ACTIVATED by the cutover into 4 live high-severity regressions on hudi-on-HMS (partition unescape / datetime-ISO / HMS-name-as-storage-path / mixed-case-Avro JNI crash); several others fixed by P7 work (HD-A4 COW/MOR) or refuted (P3b-1, P3b-3 kerberos). Adds both the third-party input report and the dated reverify (severity-sorted, per-finding current status + evidence + fix, plus a "not-applicable / already-fixed / parity" section). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…M8/L1-L20/D-series) Tracking doc for fixing the reverify findings one-by-one in later sessions. Follows the existing task-list-*-fixes.md convention: master progress table (32 tracked tasks) + per-task blocks (current file:line / fix / files / test intent), suggested batches by blast radius (4 hudi highs first), AGENT-PLAYBOOK single-task loop, module build/verify reminders. Uses the same H/M/L ids as the reverify report so the two cross-reference. Flags 4 decision-gated tasks (ask-user-first) and an explicit "do NOT touch" list (registered deviations + refuted / already-fixed findings) so later sessions don't mis-edit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…rload getWritePlanProvider() returns a non-void value, so the method-reference form Assertions.assertDoesNotThrow(connector::getWritePlanProvider, msg) matched both assertDoesNotThrow(Executable, String) and assertDoesNotThrow(ThrowingSupplier<T>, String), failing testCompile on stricter javac (CI build 991899, pull/65474). Use a block-bodied lambda with no value-returning statement so it is void-compatible only and binds Executable unambiguously; the test only asserts no-throw, so the return value is intentionally discarded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
Batch-0 (apache#65185 reverify) H4. After the HMS SPI cutover, a hudi-on-HMS table with mixed-case Avro columns (e.g. Id/Name/Addr) crashed every MOR/JNI split. Root cause: HudiScanPlanProvider.planScan built the JNI reader column list with `.map(Schema.Field::name)` (original case). BE HadoopHudiJniScanner.initRequiredColumnsAndTypes keys hudiColNameToType by these names, then resolves each requiredField with an EXACT lower-case containsKey; mixed-case "Id" misses lower-case "id" -> throw. The two sibling name paths were already lower-cased (avroSchemaToColumns:905, HudiSchemaUtils.buildField:137); legacy HudiScanNode:223 also emits lower-case names. This one path regressed. Fix: extract the inline transform into a package-private static `jniColumnNames(Schema)` that lower-cases each field name (toLowerCase(Locale.ROOT)); column ORDER preserved so the parallel columnTypes list stays positionally aligned. Extraction makes the transform offline-unit-testable (planScan needs a live metaClient). Test: HudiSchemaParityTest.jniColumnNamesAreLowerCased asserts the mixed-case Id/Name/Addr fixture lower-cases. MOR/JNI end-to-end read with mixed-case columns is live-gated (real hudi cluster). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…/H2 hive scope H4 DONE (03f4c12). Records the batch-0 scope decision: adversarial verification confirmed H1/H2 are NOT hudi-only — HiveConnectorMetadata's byte-identical prune block silently drops rows too. User signed off on "fix both copies in place" (no shared-helper extraction); D-PRUNE stays deferred as tracked design debt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
Batch-0 (apache#65185 reverify) H1. After the HMS SPI cutover, a partitioned hive-on-HMS or hudi-on-HMS table silently DROPPED rows when a partition value contained a Hive-escaped character (e.g. ':' stored as '%3A') and the query carried an EQ/IN predicate on that column. Root cause: the connector partition-pruning decision uses a byte-identical block in both HiveConnectorMetadata and HudiConnectorMetadata. parsePartitionName stored the raw substring after '=' WITHOUT unescaping, while the predicate literal (extractLiteralValue) is unescaped, so matchesPredicates' string compare (escaped 'US%3ACA' vs unescaped 'US:CA') never matched -> the real partition was pruned out -> silent row loss. fe-core computes a correct typed requiredPartitions but the connector planScan ignores it and honors only applyFilter's prunedPartitions, so this pruning is authoritative. The sibling parse paths already unescape (HudiScanPlanProvider.parsePartitionValues, HiveWriteUtils.toPartitionValues); this pruning path regressed. Adversarial verification confirmed BOTH connectors are affected (not hudi-only). Fix (both copies in place, per scope decision): unescape the VALUE in each connector's parsePartitionName via its own same-package unescapePathName (widened private->package-private). The key (column name) is left as-is. parsePartitionName widened to package-private static for direct offline unit testing (mirrors the already-tested parsePartitionValues). Tests: direct parsePartitionName unescape assertions in HudiPartitionValuesTest and HiveConnectorMetadataPartitionPruningTest (durable across the later H3 restructure); plus a hive end-to-end applyFilter test proving an escaped partition value prunes-in instead of dropping. Escaped-value read on a real cluster is live-gated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
H1 DONE (39a279e). Unescape partition value in the prune-matching parse in both hive and hudi copies. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…e text Batch-0 (apache#65185 reverify) H2. After the HMS SPI cutover, an EQ/IN predicate on a DATETIME/TIMESTAMP partition column pruned a hive-on-HMS or hudi-on-HMS table to 0 rows (silent total row loss). DATE and STRING partition columns were unaffected. Root cause: the byte-identical prune block in both HiveConnectorMetadata and HudiConnectorMetadata renders the predicate literal via String.valueOf. ExprToConnectorExpressionConverter.convertDateLiteral wraps a non-DATE datetime literal as a LocalDateTime, and String.valueOf(LocalDateTime) == toString() == ISO "2024-01-01T10:00" (T separator, dropped zero seconds), which never string-equals the Hive-canonical stored partition value "2024-01-01 10:00:00" in matchesPredicates -> every partition is pruned out. Legacy pruned via typed Nereids comparison, never strings. Adversarial verification confirmed BOTH connectors are affected, and a real Hive fixture (run17.hql: `time_par timestamp` stored as time_par=2023-01-01 01%3A30%3A00) confirms the stored scale-0 form is exactly "yyyy-MM-dd HH:mm:ss". Fix (both copies in place, per scope decision): in extractLiteralValue, render a LocalDateTime value via a new package-private static hiveDateTimeString helper -> "yyyy-MM-dd HH:mm:ss" (space separator, full seconds), appending trailing-zero-trimmed microseconds only when present. All other value types keep String.valueOf; null stays null; LocalDate (DATE columns) is unaffected. No timezone conversion is involved (unlike the MaxCompute datetime-pushdown fix), so there is no ZoneId.of crash risk. This composes with H1 (H1 unescapes the stored value, H2 renders the predicate) so a datetime partition survives pruning. Tests: direct hiveDateTimeString render assertions (whole second, midnight, seconds, microseconds, trimmed) in both connectors; plus a hive end-to-end applyFilter test proving an escaped datetime partition value (H1) prunes-in against a datetime literal (H2). DATETIME partition read on a real cluster is live-gated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…ck-off H2 DONE (cf540ee). Render DATETIME/TIMESTAMP partition literals as Hive-canonical text in both hive and hudi copies. Adversarial design review verdict SOUND with a real Hive fixture (run17.hql) confirming the stored scale-0 form. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
… source Batch-0 (apache#65185 reverify) H3, hudi-only. After the HMS SPI cutover, a filtered query on a non-hive-style hudi-on-HMS table (hive_style_partitioning =false, the Hudi default) returned 0 splits (silent total row loss); the same query without a filter returned rows. Root cause: HudiConnectorMetadata.applyFilter set prunedPartitionPaths UNCONDITIONALLY from hmsClient.listPartitionNames (HMS hive-style names, "year=2024/month=01"). HudiScanPlanProvider feeds those verbatim to fsView.getLatestBaseFilesBeforeOrOn(partitionPath,...), which is keyed by Hudi RELATIVE STORAGE paths. For a non-hive-style table the physical layout is positional ("2024/01"), so fsView matched nothing -> 0 splits. The unpruned scan path (resolvePartitions -> listAllPartitionPaths) is use_hive_sync_partition-aware and returns the relative paths, which is why an unfiltered query worked. applyFilter bypassed that awareness (collectPartitions already had it); the stale HudiScanPlanProvider javadoc claiming the pitfall was "closed before the catalog flip" was outdated. Fix: make applyFilter's candidate source use_hive_sync_partition-aware, mirroring collectPartitions. hive-sync -> keep hmsClient.listPartitionNames (the hive-style name IS the relative layout, fsView accepts it) pruned by the existing prunePartitionNames (parsePartitionName, H1-unescaped). non-hive-sync (default) -> list the relative storage paths via metaClientExecutor.execute(listAllPartitionPaths(buildMetaClient(...))) — the SAME source the unpruned scan uses (net-neutral: resolvePartitions short-circuits once prunedPaths is set) — pruned by a new static prunePartitionPaths built on parsePartitionValues (handles positional AND hive-style layouts + unescapes). matchesPredicates made static so both prune helpers share it. No FSUtils/getPartitions/getLocation (relativizing custom HMS locations exceeds legacy/collectPartitions parity). Tests: migrated HudiPartitionPruningTest to a stub metaClient executor (feeds the canned listing offline; existing assertions unchanged, now exercising the non-hive-sync branch); added a positional-path test pinning that non-hive-style tables prune to RELATIVE paths (RED under the old HMS-name code), a hive-sync-branch test, and a direct prunePartitionPaths unit. Non-hive-style filtered read on a real cluster is live-gated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…verage Closes two low-risk coverage gaps flagged by the batch-0 adversarial review: - H2 design test-plan item 6 (DATE non-regression) was unimplemented: add a DATE partition + LocalDate predicate applyFilter test to both hive and hudi, proving a LocalDate literal is NOT diverted to the datetime render (stays String.valueOf = "2024-01-01") and prunes correctly. - The hive hiveDateTimeString test omitted the trailing-zero-trim ".1" case the hudi twin asserts; add it for parity. Test-only. hive 13/0-fail, hudi 12/0-fail, checkstyle clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…w + track-off H3 DONE (9c6fc58) + test-hardening (f0ee2ab). Batch-0 (H1-H4) all done. Final batch-wide adversarial review (3 skeptics) = CLEAN: four fixes correct, compose correctly, no regressions, scope complete (other connectors independently verified immune), all 10 new tests RED-capable. Registers one pre-existing residual (use_hive_sync_partition=true + non-hive-style layout, at legacy parity, out of H3 scope) for the D-PRUNE/relativization follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…atch 1 Records the apache#65185 reverify-fix series and batch-0 completion (H1-H4, all green, adversarial review CLEAN, one pre-existing residual registered). Next session: reverify batch 1 (connector-local M-series) + live-gated e2e. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…gn current legacy) IcebergConnectorMetadata.computeRowCount reported total-records minus total-position-deletes with no equality-delete gate, so an iceberg MOR/CDC table with equality deletes surfaced an inflated row count to the CBO, while the sibling COUNT(*) pushdown (IcebergScanPlanProvider.getCountFromSummary) correctly declined -- an asymmetry that misleads join reorder / cardinality. Root cause is a parity target that moved under us, NOT a misread: the table- stats override was recon'd 2026-06-29 (HEAD 6252ecc) when legacy IcebergUtils.getIcebergRowCount was total-records - total-position-deletes with no gate, so the connector faithfully mirrored "no gate". Two days later upstream 32a2651 (apache#64648) refactored the count into getCountFromSummary(summary, ignoreDanglingDelete) and ADDED the equality gate (total-equality-deletes null or != "0" -> UNKNOWN), rewiring getIcebergRowCount to getCountFromSummary( summary, true). That commit is on this branch, so legacy now gates and the connector diverged. Cutover also routes iceberg-on-HMS (legacy HMSExternalTable :547 used the gated getIcebergRowCount) through computeRowCount -> newly activated regression there. Fix reproduces current legacy getCountFromSummary(summary, true) connector- locally (no fe-core edit): add TOTAL_EQUALITY_DELETES constant (byte-identical to the scan provider copy + fe-core), read + null-guard all three counters, and gate `!equalityDeletes.equals("0") -> -1` before the subtraction (-1 is already mapped to UNKNOWN by getTableStatistics). Position deletes still net out (legacy ignoreDanglingDelete = true). Corrected the three stale "does NOT gate" comments and rewrote the mutation-locked test equalityDeletesDoNotGate... -> equalityDeletesGateTableStatisticsToUnknown (asserts UNKNOWN; RED before fix). Reverses the prior clean-room-signed-off no-gate decision (P6.6-FIX-H4); user signed off the reversal 2026-07-11 (align current legacy over register-as- deviation). fe-core untouched; connector imports no fe-core; e2e live-gated. Build SUCCESS, 0 checkstyle, IcebergConnectorMetadataStatisticsTest 7/7 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
… H-4 docs + track-off M5 design (FIX-M5-design.md): frames the fix as a parity catch-up to upstream apache#64648 (legacy row count evolved to gate equality deletes after the 2026-06-29 recon), NOT a misread; records the user's 2026-07-11 sign-off to reverse the prior no-gate decision and align current legacy. Reconciles the three now-stale prior-decision docs with SUPERSEDED banners (preserving history, pointing to FIX-M5): P6.6-FIX-H4 design (key parity decision 1 + test #7 + mutation), its summary, and the iceberg flip-blockers tasklist H-4 entry. Ticks M5 in the reverify-fixes tracking table (+ batch-1 rolling notes: recon + adversarial red-team of all five items, all mechanisms confirmed at HEAD, no UNSOUND verdicts) and updates HANDOFF (M5 done; next M7->M6->M4->M2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…t to legacy parity
On the no-bound-S3 path (REST vended credentials: no static AK/SK/role -> chosenS3
empty), IcebergCatalogFactory.appendS3FileIO derives iceberg's client.region SOLELY
from firstNonBlank(props, S3_REGION_ALIASES). The array listed only {s3.region,
aws.region, region, client.region}, so a catalog whose region arrives via AWS_REGION,
iceberg.rest.signing-region, rest.signing-region, REGION, glue.region or
aws.glue.region yielded null -> client.region omitted -> iceberg S3FileIO falls to
the AWS DefaultAwsRegionProviderChain and the write commit fails "Unable to load
region". The comment claimed it mirrored legacy getRegionFromProperties but did not.
Widen S3_REGION_ALIASES to a verbatim connector-side copy of the fe-core S3Properties
@ConnectorProperty(isRegionField=true) region aliases (all 10, same declared order so
s3.region still wins on conflict). Connector-local literal copy (fe-connector must not
import fe-core). OSS/COS/OBS/Minio subclass region aliases are deliberately excluded
(irrelevant to an AWS-S3-backed vended REST catalog) and the comment is corrected to
say so rather than overclaiming a full getRegionFromProperties mirror.
New IcebergCatalogFactoryTest.buildCatalogPropertiesRestVendedResolvesRegionFromWidened
Aliases (region via AWS_REGION and via iceberg.rest.signing-region -> client.region);
RED before the widening. Existing s3.region test pins non-regression.
Build SUCCESS, 0 checkstyle, IcebergCatalogFactoryTest 62/62 green. e2e live-gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
FIX-M7-design.md (REST vended-cred client.region alias widening) + tick M7 in the reverify-fixes tracking table (progress row + detail + rolling log). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…chain (no-storage)
createS3TablesCatalog hard-failed when no fe-filesystem S3 storage was bound
("requires S3-compatible storage properties"). But an EC2 instance-profile
s3tables catalog carries only s3.region + a warehouse table-bucket ARN and no
static AK/SK/role, so S3FileSystemProvider.supports is false -> chosenS3 empty ->
throw on first access. Legacy IcebergS3TablesMetaStoreProperties supported this
via the SDK DefaultCredentialsProvider chain -- a fail-loud regression.
Invert the requirement: a REGION (from the bound storage or the raw props) is the
sole hard requirement for s3tables; credentials fall back to the SDK default chain
when no storage is bound.
- IcebergCatalogFactory.resolveS3Region(props): single source of truth over the
widened S3 region-alias set (M7); appendS3FileIO refactored to use it.
- IcebergConnector.resolveS3TablesRegion(chosenS3, props): static/package-visible
region gate (bound-storage region wins, else props), fail-loud only when neither
supplies a region. Unit-testable offline without a live S3TablesClient.
- createS3TablesCatalog drops the chosenS3-presence throw; buildS3TablesClient
re-signatured to (Optional<storage>, region) and derives credentials as
chosenS3.map(buildAwsCredentialsProvider).orElseGet(DefaultCredentialsProvider).
- Companion (data-plane): buildS3TablesCatalogProperties now emits client.region
from the props on the no-storage branch so S3FileIO honors an explicit s3.region
rather than only IMDS.
Tests (no Mockito): reframed s3TablesWithoutStorageFailsLoud ->
s3TablesWithoutStorageOrRegionFailsLoud (RED at HEAD); +4 resolveS3TablesRegion
gate tests (props fallback / widened alias / bound-storage-wins / fail-loud) and
+1 factory companion test (RED at HEAD). fe-core untouched; connector imports no
fe-core; no new production imports.
Build SUCCESS, 0 checkstyle, IcebergCatalogFactoryTest 63/63 + IcebergConnectorTest
19/19 green. Live-S3 e2e gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…ubgroup M5/M7/M6 done) FIX-M6-design.md (s3tables default-credential-chain fallback; companion promoted to required) + tick M6 in the tracking table (progress row + detail + rolling log) + HANDOFF: iceberg subgroup (M5/M7/M6) done, next M4 -> M2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…restore legacy)
Every query plan of a MaxCompute external table issued one full ODPS
getPartitions() round trip: fe-core PluginDrivenExternalTable.getNameToPartitionItems
(via initSelectedPartitions) calls ConnectorMetadata.listPartitions once per plan,
and the connector served it with a bare uncached ODPS SDK call (both fe-core and
connector layers uncached). On wide tables (tens of thousands of partitions) that
is multi-second per plan plus ODPS throttling risk -- a pure perf regression versus
the legacy MaxComputeExternalMetaCache.partitionValuesEntry (a TTL Caffeine cache)
the SPI migration deleted. Correctness is unaffected (BE re-filters). Hive/HMS
already re-homed its equivalent cache into the connector (HiveFileListingCache);
MaxCompute was the only connector still explicitly cache-less.
New MaxComputePartitionCache, a structural copy of HiveFileListingCache backed by
the shared fe-connector-cache framework (CacheSpec + MetaCacheEntry, byte-identical
contextual-only + manual-miss flags so the loader runs on the caller's TCCL-pinned
thread). Keyed by (db, table); loader injected as a PartitionLister for offline
testing. Held as a final field on the long-lived MaxComputeDorisConnector (metadata
is rebuilt per query), injected into the metadata; all three partition-listing
methods (listPartitions/listPartitionNames/listPartitionValues) route through it.
The four Connector REFRESH hooks (invalidateTable/Db/All/Partition) flush the cache
(invalidatePartition degrades to a whole-table flush). Config via
meta.cache.max_compute.partition.{enable,ttl-second,capacity}, defaults mirroring
legacy (ttl 86400s, capacity 10000). pom adds fe-connector-cache + Caffeine 2.9.3
(mirroring hive; the plugin zip bundles both automatically).
No fe-core edit; connector imports no fe-core; no Mockito (recording fakes). Tests
(MaxComputePartitionCacheTest, 9): 7 direct-cache + 2 metadata-integration encoding
"N plans / cross-method = 1 ODPS round trip" (behaviorally RED under mutation).
Build SUCCESS, 0 checkstyle, maxcompute module 113/113 (+1 live skip); plugin zip
bundles exactly one caffeine-2.9.3.jar + fe-connector-cache. Live-ODPS e2e gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
FIX-M4-design.md (maxcompute connector-side partition cache; records impl-subagent deviations + the verified caffeine-2.9.3 plugin-zip coherence) + tick M4 in the tracking table (progress row + detail + rolling log). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…utover) After the HMS cutover every hive table scans through the generic PluginDrivenScanNode, which enters batch/async split generation only when the connector opts in. HiveScanPlanProvider opted into neither flavor (supportsBatchScan=false, streamingSplitEstimate=-1), so a large partitioned hive scan materialized every file split of every selected partition into the FE heap in one synchronous pass -- worst on hive precisely because it is the biggest partition-count source. Legacy HiveScanNode went async at prunedPartitions.size() >= num_partitions_in_batch_mode (default 1024). Results stayed correct; FE heap + planning latency regressed. This is NOT a clean MaxCompute mimic: MaxCompute overrides only supportsBatchScan and relies on the SPI default planScanForPartitionBatch, correct only because its planScan is partition-set-scoped. Hive's planScan resolves from handle.getPrunedPartitions() and ignores the passed set, so inheriting the default would re-scan the whole pruned set per batch -> every partition's files emitted once per batch -> DUPLICATE ROWS. So hive overrides BOTH: - supportsBatchScan: true iff partitioned AND not transactional. ACID excluded deliberately (same isTransactional() accessor planScan branches on): the scan opens one metastore read transaction; per-batch resolution on background threads would open and leak a read transaction per batch. ACID partitioned tables keep the synchronous path (correct, not streamed). - planScanForPartitionBatch: resolves ONLY the batch via hmsClient.getPartitions(db, table, partitionBatch) (the batch strings are the HMS-rendered key=value/... names getPartitions accepts), then the SAME helpers as planScan (format detect, split size, hadoop conf, convertPartitions, listAndSplitFiles). No new imports. Deviations registered (fail-safe, results unaffected): - BATCH-ACID-SYNC (permanent, by-design): ACID partitioned tables stay synchronous. - BATCH-UNPRUNED-SYNC (resolved by M3, batch 2): a truly-unfiltered scan (no WHERE) keeps isPruned=false so fe-core's shouldUseBatchMode stays false; M3's ==NOT_PRUNED fix re-enables batch for it. No fe-core edit; connector imports no fe-core; no Mockito (real HmsClient / DirectoryLister / ConnectorSession fakes). New HiveScanBatchModeTest (4): supportsBatchScan true/false/false + batch-scoping size==1 (RED under the default -> 3 ranges, encoding the duplicate-splits bug). Build SUCCESS, 0 checkstyle, hive module 284/284 green. Large-partition async e2e gated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…all 5 done) FIX-M2-design.md (hive batch/async split path; registers BATCH-ACID-SYNC permanent + BATCH-UNPRUNED-SYNC resolved-by-M3) + tick M2 in the tracking table, mark batch 1 (M5/M7/M6/M4/M2) complete in the rolling log, and update HANDOFF (next = batch 2 M3->M1 fe-core generic node; M3 also resolves M2's unpruned-sync residual). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…acy parity (600s) Final adversarial review caught a parity miss in the M4 cache (c553c3c): the TTL default was copied from HiveFileListingCache's knob (external_cache_expire_time_seconds_after_access = 86400s / 24h), but the DELETED legacy MaxComputeExternalMetaCache.partitionValuesEntry used a DIFFERENT pair -- CacheSpec.of(Config.external_cache_refresh_time_minutes * 60, // default 10min = 600s Config.max_hive_partition_table_cache_num). // default 10000 Verified against the deletion commit (1da8836^) + Config.java defaults. The capacity (10000) coincidentally matched; the TTL was 144x too long -- a partition added directly in ODPS would have stayed invisible for up to 24h instead of the legacy ~10 min (both contextual-only expire-after-access, so this is a pure staleness regression vs the behavior this fix set out to restore). DEFAULT_TTL_SECOND 86400 -> 600; comment corrected to cite the real legacy knobs. No behavior change to capacity. MaxComputePartitionCacheTest 9/9 green (no test asserts the TTL value; the meta.cache.max_compute.partition.ttl-second override still works). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…(CLEAN) + M4 TTL fix Final review (wf_542c60b9-001, 5 per-fix skeptics + 1 cross-cut): M5/M6/M7/M2 and the whole-batch cross-cut CLEAN; M4 hit one medium defect (TTL default copied from the hive file cache's knob, 86400s, vs legacy MaxCompute partition cache's external_cache_refresh_time_minutes*60 = 600s) -> fixed in fca2884. Corrected FIX-M4-design.md, tracking table (rolling log + M4 detail), and HANDOFF. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…fix SecurityUtil split-brain) TeamCity #991951 (PR apache#65474, hive connector SPI migration) failed 49 external cases — hive / iceberg-on-HMS / hudi / mtmv / kerberos / tvf — all with the FE-side "Could not initialize class org.apache.hadoop.security.SecurityUtil". The cluster was healthy (366 passed, no BE crash/OOM); the "No backend available" lines are startup-only. Root cause: ThriftHmsClient.doAs pinned the TCCL to ClassLoader.getSystemClassLoader() before creating the metastore client. SecurityUtil.<clinit> -> new Configuration() captures that TCCL to reflectively load DNSDomainNameResolver: the system loader holds fe-core's hadoop copy while SecurityUtil/DomainNameResolver resolve from the plugin's child-first copy, so DomainNameResolver.isAssignableFrom(DNSDomainNameResolver) is false ("class DNSDomainNameResolver not DomainNameResolver") -> ExceptionInInitializerError, which permanently poisons SecurityUtil JVM-wide. doAs is the single choke point for both metastore creation (createFreshClient) and every RPC (execute), and both the Kerberos (ugi.doAs) and non-Kerberos (context.executeAuthenticated) authAction paths run inside it. Fix: pin getClass().getClassLoader() — the plugin child-first loader that loaded ThriftHmsClient, a strict superset of the system loader — matching iceberg/paimon TcclPinningConnectorContext and HiveConnectorMetadata's stats pins. Test: ThriftHmsClientDoAsClassLoaderTest drives DoAsTcclProbe through an isolated child-first loader (mirrors OdpsClassloaderIsolationTest) so getClass().getClassLoader() differs from the system loader, making the exact bug observable: RED before the fix (PIN_WRONG_SYSTEM_LOADER), GREEN after. fe-connector-hms 40/40 green, 0 checkstyle. Real-cluster e2e rerun of the 49 cases still owed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
…er (latent eager-UGI split-brain) Latent-edge companion to the ThriftHmsClient.doAs fix (not hit by the 49 failing #991951 cases, hardened here per request). HiveConnector.buildPluginAuthenticator runs on the (unpinned) createClient thread; a catalog declaring hadoop.security.authentication=kerberos WITHOUT a principal/keytab falls back (AuthenticationConfig.getKerberosConfig) to a HadoopSimpleAuthenticator whose ctor EAGERLY calls UserGroupInformation.createRemoteUser -> SecurityUtil.<clinit>. Its internal new Configuration() captures the TCCL, so an unpinned thread would load hadoop's DNSDomainNameResolver from fe-core's system-loader copy and split-brain-poison SecurityUtil against the plugin copy — the same failure doAs guards. buildHadoopConf sets only the OUTER conf's loader, not the TCCL that SecurityUtil's own Configuration reads, so a thread pin is required here too. Fix: pin HiveConnector.class.getClassLoader() around the whole resolution (try/finally), mirroring HudiConnector.metaClientExecutor and the doAs fix. Also refreshes the now-stale comment in createClient that described doAs as pinning the system classloader. Test: HiveConnectorPluginAuthenticatorTcclTest observes, via a TCCL-recording properties map, that the method body runs under the plugin loader and restores the caller's TCCL — RED without the pin (observed = caller marker), GREEN with it. fe-connector-hive 186/186 green (incl. the 5 existing buildPluginAuthenticator cases), 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
… + HANDOFF RCA, fix design, and per-fix summary for the ThriftHmsClient.doAs / buildPluginAuthenticator TCCL classloader split-brain (CLR1 92004ef, CLR2 15d3df1). Records the e2e owed (real-cluster rerun of the 49 SecurityUtil cases) and the outlier classification (test_hdfs_parquet_group0 = BE ASAN mem flake, unrelated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #65185.
Part of P7 — hive (+HMS) in the Catalog SPI migration tracking issue.
This PR starts the hive/HMS connector migration on
branch-catalog-spi, covering the HMS metadata path, metastore-event pipeline, ACID transaction write path, and the hudi live cutover dependency tracked in P7.