[feat](nereids)set runtime filter wait time according to table row count and table type #42640

englefly · 2024-10-28T10:57:00Z

Proposed changes

adjust compute rf wait time according to max table row count, if wait time is not default value

olap:
-- row < 1G: 1 sec
-- 1G <= row < 10G: 5 sec
-- 10G < row: 20 sec
external:
-- row < 1G: 5 sec
-- 1G <= row < 10G: 10 sec
-- 10G < row: 50 sec

Issue Number: close #xxx

doris-robot · 2024-10-28T10:57:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

englefly · 2024-10-28T10:57:10Z

run buildall

github-actions · 2024-10-28T10:57:44Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In tools/tpch-tools/bin/run-tpch-queries.sh line 179:
run_sql "source ${backup_session_variables_file};"
                ^-- SC2154 (warning): backup_session_variables_file is referenced but not assigned.

For more information:
  https://www.shellcheck.net/wiki/SC2154 -- backup_session_variables_file is ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

englefly · 2024-10-28T11:07:30Z

run buildall

github-actions · 2024-10-28T11:07:50Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- tools/tpch-tools/bin/run-tpch-queries.sh.orig
+++ tools/tpch-tools/bin/run-tpch-queries.sh
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

morrySnow · 2024-10-28T12:11:09Z

add desc please

fe/.idea/vcs.xml

fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java

englefly · 2024-10-29T10:24:03Z

run buildall

github-actions · 2024-10-29T10:24:12Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- tools/tpch-tools/bin/run-tpch-queries.sh.orig
+++ tools/tpch-tools/bin/run-tpch-queries.sh
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions · 2024-10-30T15:53:41Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- tools/tpch-tools/bin/run-tpch-queries.sh.orig
+++ tools/tpch-tools/bin/run-tpch-queries.sh
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

englefly · 2024-10-31T09:03:35Z

run buildall

github-actions · 2024-11-04T05:50:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-04T05:50:49Z

PR approved by anyone and no changes requested.

xzj7019 · 2024-11-04T06:30:17Z

fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java

@@ -1136,7 +1136,7 @@ public class SessionVariable implements Serializable, Writable {
    private int runtimeBloomFilterMaxSize = 16777216;

    @VariableMgr.VarAttr(name = RUNTIME_FILTER_WAIT_TIME_MS, needForward = true)
-    private int runtimeFilterWaitTimeMs = 1000;
+    public int runtimeFilterWaitTimeMs = 1000;


why change this but use the setRuntimeFilterWaitTimeMs instead?

fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java

englefly · 2024-11-04T07:24:26Z

run buildall

englefly · 2024-11-04T11:17:49Z

run p0

englefly · 2024-11-04T11:17:56Z

run cloud_p0

englefly · 2024-11-05T10:20:42Z

run buildall

englefly · 2024-11-05T10:24:28Z

run buildall

englefly · 2024-11-08T02:03:52Z

run buildall

englefly · 2024-11-08T06:10:45Z

run external

englefly · 2024-11-08T06:10:51Z

run p0

englefly · 2024-11-08T06:10:57Z

run cloud_p0

github-actions · 2024-11-11T06:48:22Z

PR approved by at least one committer and no changes requested.

…unt and table type (apache#42640) adjust compute rf wait time according to max table row count, if wait time is not default value - olap: -- row < 1G: 1 sec -- 1G <= row < 10G: 5 sec -- 10G < row: 20 sec - external: -- row < 1G: 5 sec -- 1G <= row < 10G: 10 sec -- 10G < row: 50 sec Issue Number: close #xxx

…unt and table type #42640 branch-3.0 (#44031) pick #42640

…unt and table type #42640 branch-2.1 (#45114) ### What problem does this PR solve? pick#42640

morrySnow reviewed Oct 28, 2024

View reviewed changes

fe/.idea/vcs.xml Outdated Show resolved Hide resolved

xzj7019 reviewed Oct 28, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java Outdated Show resolved Hide resolved

englefly force-pushed the rf-wait-rowcount branch from 78691d1 to 5a4772a Compare October 29, 2024 10:23

morrySnow added dev/2.1.x dev/3.0.x labels Oct 31, 2024

starocean999 previously approved these changes Nov 4, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 4, 2024

github-actions bot added the reviewed label Nov 4, 2024

xzj7019 reviewed Nov 4, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java Outdated Show resolved Hide resolved

englefly dismissed starocean999’s stale review via f1cd9a7 November 4, 2024 07:23

englefly force-pushed the rf-wait-rowcount branch from 8335099 to f1cd9a7 Compare November 4, 2024 07:23

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 4, 2024

englefly force-pushed the rf-wait-rowcount branch from f1cd9a7 to 2050178 Compare November 5, 2024 10:20

set rf wait time according to table row count and table type

5ce6154

englefly force-pushed the rf-wait-rowcount branch from 4a0ed88 to 5ce6154 Compare November 8, 2024 02:02

starocean999 approved these changes Nov 11, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 11, 2024

LiBinfeng-01 approved these changes Nov 11, 2024

View reviewed changes

englefly merged commit 1d8f6d7 into apache:master Nov 11, 2024
26 of 28 checks passed

englefly deleted the rf-wait-rowcount branch November 11, 2024 08:00

github-actions bot added cherry-pick-conflict-in-3.0 cherry-pick-conflict-in-2.1.x labels Nov 11, 2024

englefly mentioned this pull request Nov 15, 2024

[feat](nereids)set runtime filter wait time according to table row count and table type #42640 branch-3.0 #44031

Merged

16 tasks

dataroaring pushed a commit that referenced this pull request Nov 29, 2024

[feat](nereids)set runtime filter wait time according to table row co…

ec09d89

…unt and table type #42640 branch-3.0 (#44031) pick #42640

dataroaring added dev/3.0.4-merged and removed dev/3.0.x cherry-pick-conflict-in-3.0 labels Nov 29, 2024

englefly added a commit that referenced this pull request Dec 9, 2024

[feat](nereids)set runtime filter wait time according to table row co…

b2637e4

…unt and table type #42640 branch-2.1 (#45114) ### What problem does this PR solve? pick#42640

wm1581066 added dev/2.1.8-merged and removed dev/2.1.x labels Dec 9, 2024

morrySnow removed the cherry-pick-conflict-in-2.1.x label Dec 20, 2024

morrySnow mentioned this pull request Jan 6, 2025

[feat](nereids)set runtime filter wait time according to table row count and table type #42640 branch-2.1 #45114

Merged

16 tasks

yiguolei mentioned this pull request Jan 19, 2025

Release Note 2.1.8 #47198

Open

gavinchou mentioned this pull request Feb 18, 2025

Release Notes 3.0.4 #48013

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat](nereids)set runtime filter wait time according to table row count and table type #42640

[feat](nereids)set runtime filter wait time according to table row count and table type #42640

englefly commented Oct 28, 2024 •

edited

Loading

doris-robot commented Oct 28, 2024

englefly commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

englefly commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

morrySnow commented Oct 28, 2024

englefly commented Oct 29, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 30, 2024

englefly commented Oct 31, 2024

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

xzj7019 Nov 4, 2024

englefly commented Nov 4, 2024

englefly commented Nov 4, 2024

englefly commented Nov 4, 2024

englefly commented Nov 5, 2024

englefly commented Nov 5, 2024

englefly commented Nov 8, 2024

englefly commented Nov 8, 2024

englefly commented Nov 8, 2024

englefly commented Nov 8, 2024

github-actions bot commented Nov 11, 2024

[feat](nereids)set runtime filter wait time according to table row count and table type #42640

[feat](nereids)set runtime filter wait time according to table row count and table type #42640

Conversation

englefly commented Oct 28, 2024 • edited Loading

Proposed changes

doris-robot commented Oct 28, 2024

englefly commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

sh-checker report

englefly commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

sh-checker report

morrySnow commented Oct 28, 2024

englefly commented Oct 29, 2024

github-actions bot commented Oct 29, 2024

sh-checker report

github-actions bot commented Oct 30, 2024

sh-checker report

englefly commented Oct 31, 2024

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

xzj7019 Nov 4, 2024

Choose a reason for hiding this comment

englefly commented Nov 4, 2024

englefly commented Nov 4, 2024

englefly commented Nov 4, 2024

englefly commented Nov 5, 2024

englefly commented Nov 5, 2024

englefly commented Nov 8, 2024

englefly commented Nov 8, 2024

englefly commented Nov 8, 2024

englefly commented Nov 8, 2024

github-actions bot commented Nov 11, 2024

englefly commented Oct 28, 2024 •

edited

Loading

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`