Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] dynamic partition is not work #48989

Closed
3 tasks done
qidaye opened this issue Mar 12, 2025 · 0 comments · Fixed by #48988
Closed
3 tasks done

[Bug] dynamic partition is not work #48989

qidaye opened this issue Mar 12, 2025 · 0 comments · Fixed by #48988

Comments

@qidaye
Copy link
Contributor

qidaye commented Mar 12, 2025

Search before asking

  • I had searched in the issues and found no similar issues.

Version

3.0.4

What's Wrong?

mysql> alter table ods_kafka_ai_chat_prompt_rt add partition p20250316 values [('2025-03-16 00:00:00'),('2025-03-17 00:00:00'));
ERROR 1105 (HY000): Unexpected exception: Cannot invoke "java.util.List.iterator()" because "this.columnUniqueIds" is null

What You Expected?

Add partition normally.

How to Reproduce?

Create table in 3.0.2.

CREATE TABLE `ods_kafka_ai_chat_prompt_rt` (
  `msg_id` largeint NULL COMMENT "消息id",
  `api_key` varchar(2048) NULL COMMENT "apiKey",
  `api_key_alias` varchar(4096) NULL COMMENT "apiKey 别名",
  `model` varchar(2048) NULL COMMENT "模型",
  `messages` text NULL COMMENT "消息内容",
  `qpm` int NULL COMMENT "每分钟请求数限制",
  `temperature` decimal(10,2) NULL,
  `max_tokens` int NULL COMMENT "每分钟请求数限制",
  `token_per_min` bigint NULL COMMENT "每分钟token限额",
  `token_per_msg` bigint NULL COMMENT "单条token限额",
  `create_time` datetime NULL COMMENT "创建时间",
  `update_time` datetime NULL COMMENT "修改时间",
  INDEX idx_msg_id (`msg_id`) USING INVERTED,
  INDEX idx_api_key (`api_key`) USING INVERTED,
  INDEX idx_api_key_alias (`api_key_alias`) USING INVERTED,
  INDEX idx_model (`model`) USING INVERTED
) ENGINE=OLAP
DUPLICATE KEY(`msg_id`, `api_key`, `api_key_alias`, `model`)
PARTITION BY RANGE(`create_time`)()
DISTRIBUTED BY RANDOM BUCKETS AUTO
PROPERTIES (
"file_cache_ttl_seconds" = "0",
"bloom_filter_columns" = "api_key, model, msg_id, api_key_alias",
"is_being_synced" = "false",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "DAY",
"dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.start" = "-2147483648",
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p",
"dynamic_partition.buckets" = "10",
"dynamic_partition.create_history_partition" = "false",
"dynamic_partition.history_partition_num" = "-1",
"dynamic_partition.hot_partition_num" = "0",
"dynamic_partition.reserved_history_periods" = "NULL",
"storage_medium" = "hdd",
"storage_format" = "V2",
"inverted_index_storage_format" = "V2",
"light_schema_change" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false",
"group_commit_interval_ms" = "10000",
"group_commit_data_bytes" = "134217728"
);

Upgrade to 3.0.4

Then dynamic partition will not work.


ALTER TABLE ods_kafka_ai_chat_prompt_rt SET ("dynamic_partition.enable" = "false");
alter table ods_kafka_ai_chat_prompt_rt add partition p20250316 values [('2025-03-16 00:00:00'),('2025-03-17 00:00:00'));

Anything Else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@qidaye qidaye mentioned this issue Mar 12, 2025
Merged
16 tasks
qidaye added a commit to qidaye/incubator-doris that referenced this issue Mar 20, 2025
@qidaye
[fix](Index)Make column unique ids in index dynamically computed (apa…
4e40a4d 
…che#48988)

Issue Number: close apache#48989

Related PR: apache#46648 apache#42882

Problem Summary:

We introduced `Index.columnUniqueIds` in 3.0.3, and make sure
`Index.columnUniqueIds` is not empty in 3.0.4.
But when we upgrade Doris from 3.0.2 and before to 3.0.4, the
`Index.columnUniqueIds` will be `null`.

Versions 302 and earlier do not have this variable, and gson
deserialization will assign it to null.
We need to initialize the column unique ids associated with the index
correctly.

Key Changes

Removed stored `columnUniqueIds` from `Index` class:

Previously, column unique IDs were stored as a field in the `Index`
class
Now they're dynamically computed when needed rather than stored
redundantly
Added dynamic computation method:

New method `getColumnUniqueIds(List<Column> schema)` computes IDs at
runtime
Column IDs are looked up by matching index column names with schema
columns

Updated method signatures:

Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)`
Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer>
indexColumnUniqueIds)`
Updated all callers to pass column IDs as parameters
qidaye added a commit to qidaye/incubator-doris that referenced this issue Mar 20, 2025
@qidaye
[fix](Index)Make column unique ids in index dynamically computed (apa…
97a928c 
…che#48988)

Issue Number: close apache#48989

Related PR: apache#46648 apache#42882

Problem Summary:

We introduced `Index.columnUniqueIds` in 3.0.3, and make sure
`Index.columnUniqueIds` is not empty in 3.0.4.
But when we upgrade Doris from 3.0.2 and before to 3.0.4, the
`Index.columnUniqueIds` will be `null`.

Versions 302 and earlier do not have this variable, and gson
deserialization will assign it to null.
We need to initialize the column unique ids associated with the index
correctly.

Key Changes

Removed stored `columnUniqueIds` from `Index` class:

Previously, column unique IDs were stored as a field in the `Index`
class
Now they're dynamically computed when needed rather than stored
redundantly
Added dynamic computation method:

New method `getColumnUniqueIds(List<Column> schema)` computes IDs at
runtime
Column IDs are looked up by matching index column names with schema
columns

Updated method signatures:

Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)`
Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer>
indexColumnUniqueIds)`
Updated all callers to pass column IDs as parameters
qidaye added a commit to qidaye/incubator-doris that referenced this issue Mar 20, 2025
@qidaye
[fix](Index)Make column unique ids in index dynamically computed (apa…
c5aadc4 
…che#48988)

Issue Number: close apache#48989

Related PR: apache#46648 apache#42882

Problem Summary:

We introduced `Index.columnUniqueIds` in 3.0.3, and make sure
`Index.columnUniqueIds` is not empty in 3.0.4.
But when we upgrade Doris from 3.0.2 and before to 3.0.4, the
`Index.columnUniqueIds` will be `null`.

Versions 302 and earlier do not have this variable, and gson
deserialization will assign it to null.
We need to initialize the column unique ids associated with the index
correctly.

Key Changes

Removed stored `columnUniqueIds` from `Index` class:

Previously, column unique IDs were stored as a field in the `Index`
class
Now they're dynamically computed when needed rather than stored
redundantly
Added dynamic computation method:

New method `getColumnUniqueIds(List<Column> schema)` computes IDs at
runtime
Column IDs are looked up by matching index column names with schema
columns

Updated method signatures:

Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)`
Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer>
indexColumnUniqueIds)`
Updated all callers to pass column IDs as parameters
qidaye added a commit to qidaye/incubator-doris that referenced this issue Mar 20, 2025
@qidaye
[fix](Index)Make column unique ids in index dynamically computed (apa…
e172ed3 
…che#48988)

Issue Number: close apache#48989

Related PR: apache#46648 apache#42882

Problem Summary:

We introduced `Index.columnUniqueIds` in 3.0.3, and make sure
`Index.columnUniqueIds` is not empty in 3.0.4.
But when we upgrade Doris from 3.0.2 and before to 3.0.4, the
`Index.columnUniqueIds` will be `null`.

Versions 302 and earlier do not have this variable, and gson
deserialization will assign it to null.
We need to initialize the column unique ids associated with the index
correctly.

Key Changes

Removed stored `columnUniqueIds` from `Index` class:

Previously, column unique IDs were stored as a field in the `Index`
class
Now they're dynamically computed when needed rather than stored
redundantly
Added dynamic computation method:

New method `getColumnUniqueIds(List<Column> schema)` computes IDs at
runtime
Column IDs are looked up by matching index column names with schema
columns

Updated method signatures:

Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)`
Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer>
indexColumnUniqueIds)`
Updated all callers to pass column IDs as parameters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[fix](Index)Make column unique ids in index dynamically computed qidaye/incubator-doris
1 participant
@qidaye