[fix](deltawriter) Enhance file_size validation in _request_slave_tablet_pull_rowset method #48866
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
TPC-H: Total hot run time: 32580 ms |
sollhui
reviewed
Mar 10, 2025
be/src/olap/delta_writer.cpp
Outdated
| (*(request->mutable_inverted_indices_size()))[segment_id]; | ||
| // Add the new index size to the map value. | ||
| *index_size_map_value.mutable_index_sizes()->Add() = std::move(index_size); | ||
| int64_t size = safe_get_file_size(inverted_index_file); |
There was a problem hiding this comment.
Should return directly instead of not executing the logic if there is an exception?
TPC-DS: Total hot run time: 185736 ms |
ClickBench: Total hot run time: 30.79 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
added 2 commits
March 10, 2025 19:18
|
run buildall |
added 2 commits
March 10, 2025 19:33
|
run buildall |
TPC-H: Total hot run time: 32752 ms |
TPC-DS: Total hot run time: 186036 ms |
ClickBench: Total hot run time: 30.94 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
added 2 commits
March 11, 2025 15:58
|
run buildall |
sollhui
reviewed
Mar 11, 2025
be/src/olap/delta_writer.cpp
Outdated
| for (auto&& node_info : slave_tablet_nodes.slave_nodes()) { | ||
| _request_slave_tablet_pull_rowset(node_info); | ||
| Status status = _request_slave_tablet_pull_rowset(node_info); |
There was a problem hiding this comment.
RETURN_IF_ERROR(_request_slave_tablet_pull_rowset(node_info))
sollhui
reviewed
Mar 11, 2025
be/src/olap/delta_writer.cpp
Outdated
| @@ -231,7 +239,26 @@ int64_t BaseDeltaWriter::mem_consumption(MemType mem) { | |||
| return _memtable_writer->mem_consumption(mem); | |||
| } | |||
|
|
|||
| void DeltaWriter::_request_slave_tablet_pull_rowset(const PNodeInfo& node_info) { | |||
| Status safe_get_file_size(const std::string& file_path, int64_t* file_size) { | |||
There was a problem hiding this comment.
already simplify
TPC-H: Total hot run time: 32983 ms |
TPC-DS: Total hot run time: 186255 ms |
ClickBench: Total hot run time: 30.76 s |
morningman
reviewed
Mar 12, 2025
be/src/olap/delta_writer.h
Outdated
| @@ -144,7 +144,9 @@ class DeltaWriter final : public BaseDeltaWriter { | |||
| private: | |||
| void _init_profile(RuntimeProfile* profile) override; | |||
|
|
|||
| void _request_slave_tablet_pull_rowset(const PNodeInfo& node_info); | |||
| int64_t safe_get_filesize(const std::string& file_path); | |||
There was a problem hiding this comment.
Suggested change
| int64_t safe_get_filesize(const std::string& file_path); | |
| int64_t _safe_get_filesize(const std::string& file_path); |
be/src/olap/delta_writer.cpp
Outdated
| void DeltaWriter::_request_slave_tablet_pull_rowset(const PNodeInfo& node_info) { | ||
| Status safe_get_file_size(const std::string& file_path, int64_t* file_size) { | ||
| if (file_size == nullptr) { | ||
| return Status::InvalidArgument("Null output parameter in safe_get_file_size"); |
There was a problem hiding this comment.
I think you can just use CHECK(file_size != nullptr) << "Null output parameter in safe_get_file_size"
added 2 commits
March 14, 2025 09:28
|
run buildall |
morningman
previously approved these changes
Mar 14, 2025
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
github-actions
bot
removed
the
approved
morningman
previously approved these changes
Mar 14, 2025
|
run buildall |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
liaoxin01
reviewed
Mar 16, 2025
be/src/olap/delta_writer.cpp
Outdated
| CHECK(file_size != nullptr) << "Null output parameter in safe_get_file_size"; | ||
|
|
||
| try { | ||
| if (!std::filesystem::exists(file_path)) { |
There was a problem hiding this comment.
There is no need to check if the file exists. If the file does not exist, the file_size function will throw an exception.
There was a problem hiding this comment.
the issue has been fixed
be/src/olap/delta_writer.cpp
Outdated
| @@ -281,7 +299,9 @@ void DeltaWriter::_request_slave_tablet_pull_rowset(const PNodeInfo& node_info) | |||
| for (int segment_id = 0; segment_id < cur_rowset->rowset_meta()->num_segments(); segment_id++) { | |||
| auto seg_path = | |||
| local_segment_path(tablet_path, cur_rowset->rowset_id().to_string(), segment_id); | |||
| int64_t segment_size = std::filesystem::file_size(seg_path); | |||
| int64_t segment_size = 0; | |||
| RETURN_IF_ERROR(safe_get_file_size(seg_path, &segment_size)); | |||
There was a problem hiding this comment.
Suggested change
| RETURN_IF_ERROR(safe_get_file_size(seg_path, &segment_size)); | |
| RETURN_IF_ERROR(_safe_get_file_size(seg_path, &segment_size)); |
compile error.
github-actions
bot
removed
the
approved
|
run buildall |
TPC-H: Total hot run time: 32645 ms |
TPC-DS: Total hot run time: 192548 ms |
ClickBench: Total hot run time: 30.82 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
added 2 commits
March 29, 2025 12:53
morningman
approved these changes
Mar 29, 2025
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close
Related PR:
Problem Summary:Enhance file_size validation when synchronizing tablet replicas to other slave nodes check in _request_slave_tablet_pull_rowset on BE.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)