[Bug]: alter table add fulltext index report 'objectio: too large object size 3517242331'.

[Ariznawlll]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

main

Commit ID

ab9421f

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

log: https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22GYP%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhost%3D%5C%2210-222-1-129%5C%22%7D%20%7C%3D%20%60too%20large%20object%20size%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221739346600000%22,%22to%22:%221739347320000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

create table fulltext_test (content text);
load data local infile '/root/weilu/zhwiki-latest-pages-articles-multistream6.xml-p7096380p8596379.csv' into table fulltext_test lines terminated by '\n' ignore 1 lines;
load data local infile '/root/weilu/zhwiki-latest-pages-articles-multistream1.xml-p1p187712.csv' into table fulltext_test lines terminated by '\n' ignore 1 lines;

alter table fulltext_test add column id int auto_increment first;
alter table fulltext_test add primary key(id);
alter table fulltext_test add fulltext f01(content);


csv file is huge, please contact me privately.

Additional information

No response

[cpegeric]

it is related to objectio/writer.go. Please ask them to take a look first.

[ouyuanning]

徐鹏先帮忙看看，这个是要在哪里切比较合适吧。
看是切object，还是要在block之前就切

[gouhongshen]

workspace 里面的 batch 堆积到一定数量都会 dump 到磁盘，但这个case 中这一机制失效了，导致最后将堆积的大量 batch 一次性的 dump 下去，生成的 object size 就超过了阈值。

为什么会失效？

每次尝试 dump batch 的时候都是根据 workspace snapshotOffset 来决定统计哪些 batch。构建全文索引时，会执行 insert data into secondary_index，将数据写入 workspace，每次都会申请一批自增值，每次申请都会更新 snapshotOffset。这样一来每次dump的时候就只统计了最近插入的几个 batch，size 就达不到 dump threshold，导致堆积。