Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids) prune not required window expressions on window operator #35504

Merged
merged 1 commit into from
May 29, 2024
Merged

[fix](Nereids) prune not required window expressions on window operator #35504

merged 1 commit into from
May 29, 2024

Conversation

morrySnow
Copy link
Contributor

if window expression is not required by its parent, we should prune this column. If all window expressions of window operator are pruned, we remove this window operator directly.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morrySnow
Copy link
Contributor Author

run buildall

@morrySnow morrySnow changed the title [fix](Nereids) prune not require window expressions on window operator [fix](Nereids) prune not required window expressions on window operator May 28, 2024
@morrySnow morrySnow force-pushed the fix_window_column_pruning branch from 771dce1 to 23967e7 Compare May 28, 2024 07:11
@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41066 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 23967e781ca50af6c4cb9975cdc25f7562441be4, data reload: false

------ Round 1 ----------------------------------
q1	17583	4308	4211	4211
q2	2025	198	195	195
q3	10449	1187	1211	1187
q4	10200	769	842	769
q5	7517	2658	2731	2658
q6	226	144	139	139
q7	959	605	614	605
q8	9225	2139	2079	2079
q9	10011	6682	6759	6682
q10	9233	3830	4009	3830
q11	440	269	244	244
q12	463	224	222	222
q13	17547	3168	3131	3131
q14	262	225	235	225
q15	504	480	481	480
q16	488	406	401	401
q17	980	738	768	738
q18	8383	7771	7833	7771
q19	5576	1588	1570	1570
q20	637	322	321	321
q21	5069	4091	3323	3323
q22	354	292	285	285
Total cold run time: 118131 ms
Total hot run time: 41066 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4587	4426	4422	4422
q2	384	286	279	279
q3	3164	2958	2838	2838
q4	1851	1639	1613	1613
q5	5467	5499	5518	5499
q6	215	126	125	125
q7	2202	1833	1841	1833
q8	3207	3384	3346	3346
q9	8622	8630	8651	8630
q10	3899	3729	3870	3729
q11	610	516	486	486
q12	805	646	619	619
q13	17178	3155	3150	3150
q14	309	288	264	264
q15	519	480	485	480
q16	474	404	414	404
q17	1763	1489	1481	1481
q18	7813	7597	7342	7342
q19	1640	1526	1491	1491
q20	1993	1779	1821	1779
q21	7824	4742	4747	4742
q22	581	485	494	485
Total cold run time: 75107 ms
Total hot run time: 55037 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169634 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 23967e781ca50af6c4cb9975cdc25f7562441be4, data reload: false

query1	916	382	382	382
query2	6427	2368	2233	2233
query3	6632	213	203	203
query4	19404	17552	17670	17552
query5	4148	427	420	420
query6	247	155	156	155
query7	4593	296	293	293
query8	242	197	176	176
query9	8536	2457	2480	2457
query10	468	277	276	276
query11	10444	10177	9993	9993
query12	140	93	88	88
query13	1638	370	391	370
query14	9516	6794	7308	6794
query15	214	168	169	168
query16	7764	260	263	260
query17	1725	537	533	533
query18	1954	304	287	287
query19	204	165	153	153
query20	89	81	84	81
query21	195	132	132	132
query22	4135	3942	3947	3942
query23	33659	33068	33129	33068
query24	8856	2861	2847	2847
query25	565	349	359	349
query26	698	152	153	152
query27	2226	328	326	326
query28	5765	2084	2099	2084
query29	881	601	595	595
query30	245	148	153	148
query31	969	764	738	738
query32	90	51	54	51
query33	634	271	258	258
query34	893	473	486	473
query35	711	604	609	604
query36	1087	947	898	898
query37	101	66	66	66
query38	2898	2781	2805	2781
query39	857	801	775	775
query40	194	125	131	125
query41	46	44	43	43
query42	102	102	100	100
query43	561	550	525	525
query44	1108	729	749	729
query45	178	161	160	160
query46	1082	770	711	711
query47	1863	1795	1772	1772
query48	381	309	299	299
query49	852	373	388	373
query50	766	387	391	387
query51	6840	6847	6760	6760
query52	97	94	96	94
query53	350	284	286	284
query54	785	440	417	417
query55	79	71	70	70
query56	261	238	240	238
query57	1121	1060	1044	1044
query58	232	211	211	211
query59	3261	3147	3005	3005
query60	269	259	254	254
query61	90	92	86	86
query62	605	474	444	444
query63	321	286	279	279
query64	8500	2208	1873	1873
query65	3159	3084	3126	3084
query66	780	330	327	327
query67	15210	15129	14945	14945
query68	4503	561	553	553
query69	450	268	264	264
query70	1219	1117	1157	1117
query71	374	272	269	269
query72	7485	2745	2561	2561
query73	715	326	324	324
query74	6062	5610	5683	5610
query75	3301	2643	2674	2643
query76	2311	944	1041	944
query77	413	269	261	261
query78	10215	9988	9631	9631
query79	1816	526	511	511
query80	1098	441	478	441
query81	515	221	219	219
query82	900	92	92	92
query83	224	166	169	166
query84	244	84	85	84
query85	1223	274	273	273
query86	461	312	287	287
query87	3344	3089	3115	3089
query88	3360	2438	2438	2438
query89	471	392	384	384
query90	1978	187	187	187
query91	123	101	99	99
query92	61	47	49	47
query93	1863	523	500	500
query94	1266	190	192	190
query95	395	318	310	310
query96	597	276	270	270
query97	3199	3040	3020	3020
query98	237	232	217	217
query99	1158	840	833	833
Total cold run time: 261440 ms
Total hot run time: 169634 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 23967e781ca50af6c4cb9975cdc25f7562441be4, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.68	0.07	0.07
query5	0.52	0.49	0.52
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.54	0.54
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.79	0.78	0.76
query15	0.83	0.80	0.80
query16	0.37	0.37	0.37
query17	1.00	1.02	1.04
query18	0.24	0.24	0.25
query19	1.83	1.71	1.73
query20	0.01	0.02	0.01
query21	15.73	0.67	0.66
query22	4.73	6.73	1.86
query23	18.31	1.27	1.18
query24	1.36	0.34	0.30
query25	0.14	0.09	0.08
query26	0.26	0.16	0.16
query27	0.07	0.08	0.08
query28	13.37	1.01	0.99
query29	13.81	3.32	3.30
query30	0.25	0.06	0.06
query31	2.86	0.38	0.37
query32	3.31	0.47	0.47
query33	2.79	2.94	2.89
query34	17.01	4.39	4.44
query35	4.49	4.45	4.68
query36	0.65	0.46	0.46
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.03
Total cold run time: 110.68 s
Total hot run time: 30.5 s

@morrySnow morrySnow added dev/2.0.x dev/2.1.x usercase Important user case type label labels May 28, 2024
@morrySnow morrySnow force-pushed the fix_window_column_pruning branch from 23967e7 to 9031fce Compare May 28, 2024 10:07
@morrySnow
Copy link
Contributor Author

run buildall

@morrySnow morrySnow requested review from 924060929 and englefly May 28, 2024 10:15
@doris-robot
Copy link

TPC-H: Total hot run time: 41397 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9031fce5fffba2785d346b9ddc767b42a7512caa, data reload: false

------ Round 1 ----------------------------------
q1	17600	4338	4245	4245
q2	2031	202	197	197
q3	10424	1307	1223	1223
q4	10204	788	810	788
q5	7491	2691	2733	2691
q6	223	141	136	136
q7	968	616	613	613
q8	9226	2142	2124	2124
q9	9440	6715	6718	6715
q10	9294	3870	3858	3858
q11	447	253	239	239
q12	492	232	232	232
q13	17295	3293	3183	3183
q14	283	238	237	237
q15	526	480	477	477
q16	531	402	417	402
q17	1001	749	726	726
q18	8280	7929	7788	7788
q19	7192	1567	1554	1554
q20	643	329	322	322
q21	5214	3364	4025	3364
q22	339	283	284	283
Total cold run time: 119144 ms
Total hot run time: 41397 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4519	4374	4425	4374
q2	369	272	269	269
q3	3148	2927	2890	2890
q4	1992	1747	1723	1723
q5	5410	5348	5498	5348
q6	210	121	126	121
q7	2175	1811	1811	1811
q8	3232	3394	3384	3384
q9	8691	8682	8605	8605
q10	4112	3839	3869	3839
q11	590	484	502	484
q12	757	581	592	581
q13	15935	3074	3138	3074
q14	300	275	264	264
q15	526	474	468	468
q16	502	428	431	428
q17	1773	1516	1486	1486
q18	8168	7753	7391	7391
q19	1653	1547	1561	1547
q20	1994	1791	1779	1779
q21	10216	4811	4758	4758
q22	561	505	502	502
Total cold run time: 76833 ms
Total hot run time: 55126 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169745 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9031fce5fffba2785d346b9ddc767b42a7512caa, data reload: false

query1	922	381	379	379
query2	6913	2474	2267	2267
query3	6636	207	207	207
query4	19773	17430	17102	17102
query5	4130	442	416	416
query6	246	162	149	149
query7	4591	297	285	285
query8	232	179	177	177
query9	8524	2448	2416	2416
query10	454	281	252	252
query11	10472	10039	10091	10039
query12	129	89	87	87
query13	1628	369	363	363
query14	10067	7506	7595	7506
query15	195	171	166	166
query16	7724	263	259	259
query17	1705	529	536	529
query18	1953	276	269	269
query19	199	147	150	147
query20	92	87	88	87
query21	191	130	130	130
query22	4144	3961	3865	3865
query23	33313	33142	33029	33029
query24	10629	2914	2852	2852
query25	566	349	357	349
query26	705	153	156	153
query27	2235	320	345	320
query28	5730	2105	2087	2087
query29	853	598	610	598
query30	241	150	149	149
query31	947	738	753	738
query32	87	51	60	51
query33	645	269	287	269
query34	877	484	476	476
query35	711	595	601	595
query36	1079	930	904	904
query37	101	67	64	64
query38	2911	2790	2732	2732
query39	842	779	787	779
query40	190	124	124	124
query41	46	44	47	44
query42	105	96	96	96
query43	582	555	540	540
query44	1078	732	741	732
query45	174	167	163	163
query46	1057	715	714	714
query47	1858	1792	1794	1792
query48	371	291	289	289
query49	850	371	379	371
query50	771	382	392	382
query51	6980	6761	6756	6756
query52	98	88	90	88
query53	353	279	280	279
query54	838	487	433	433
query55	75	71	71	71
query56	261	239	237	237
query57	1143	1040	1031	1031
query58	234	199	216	199
query59	3416	3212	3275	3212
query60	272	256	256	256
query61	88	86	88	86
query62	611	461	447	447
query63	316	279	285	279
query64	8531	2276	1707	1707
query65	3150	3118	3111	3111
query66	798	334	325	325
query67	15430	14715	15065	14715
query68	4588	551	564	551
query69	455	268	267	267
query70	1136	1115	1060	1060
query71	402	276	268	268
query72	7558	4696	2602	2602
query73	728	332	324	324
query74	5955	5572	5600	5572
query75	3412	2661	2602	2602
query76	2721	956	1029	956
query77	579	260	263	260
query78	10241	9903	9798	9798
query79	1616	525	513	513
query80	1061	435	427	427
query81	524	223	221	221
query82	949	94	91	91
query83	245	173	167	167
query84	247	89	84	84
query85	1196	276	263	263
query86	411	314	309	309
query87	3322	3141	3142	3141
query88	3132	2444	2431	2431
query89	491	418	386	386
query90	1945	202	190	190
query91	132	109	110	109
query92	60	56	50	50
query93	1297	524	510	510
query94	1173	188	190	188
query95	403	313	313	313
query96	590	277	271	271
query97	3158	3021	3030	3021
query98	236	233	218	218
query99	1147	873	861	861
Total cold run time: 264178 ms
Total hot run time: 169745 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.39 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9031fce5fffba2785d346b9ddc767b42a7512caa, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.07	0.06
query4	1.65	0.08	0.08
query5	0.53	0.50	0.51
query6	1.12	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.54	0.48	0.50
query10	0.55	0.55	0.56
query11	0.15	0.11	0.12
query12	0.15	0.11	0.12
query13	0.61	0.61	0.60
query14	0.80	0.81	0.79
query15	0.83	0.82	0.81
query16	0.36	0.36	0.37
query17	1.01	1.02	1.01
query18	0.21	0.26	0.24
query19	1.86	1.81	1.68
query20	0.01	0.01	0.01
query21	15.42	0.72	0.69
query22	4.83	6.52	1.69
query23	18.28	1.41	1.22
query24	1.59	0.28	0.26
query25	0.14	0.08	0.08
query26	0.25	0.16	0.16
query27	0.08	0.08	0.08
query28	13.31	1.03	1.00
query29	12.67	3.34	3.24
query30	0.23	0.06	0.05
query31	2.87	0.38	0.38
query32	3.30	0.49	0.47
query33	2.86	2.92	2.88
query34	17.23	4.53	4.43
query35	4.51	4.49	4.60
query36	0.69	0.48	0.46
query37	0.17	0.16	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.03
query40	0.16	0.13	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 109.76 s
Total hot run time: 30.39 s

@morrySnow
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 41077 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9031fce5fffba2785d346b9ddc767b42a7512caa, data reload: false

------ Round 1 ----------------------------------
q1	17600	4357	4202	4202
q2	2026	194	202	194
q3	10436	1240	1164	1164
q4	10191	793	755	755
q5	7541	2711	2715	2711
q6	230	131	136	131
q7	942	618	615	615
q8	9211	2100	2088	2088
q9	9443	6674	6703	6674
q10	9108	3898	3918	3898
q11	459	250	248	248
q12	489	223	245	223
q13	17954	3242	3183	3183
q14	280	240	236	236
q15	546	492	474	474
q16	507	389	373	373
q17	943	663	631	631
q18	8370	7801	7877	7801
q19	7331	1595	1529	1529
q20	659	334	308	308
q21	5155	3362	3928	3362
q22	361	277	285	277
Total cold run time: 119782 ms
Total hot run time: 41077 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4556	4454	4417	4417
q2	381	280	280	280
q3	3159	2932	2946	2932
q4	2013	1685	1576	1576
q5	5321	5521	5500	5500
q6	212	119	121	119
q7	2204	1854	1812	1812
q8	3211	3383	3377	3377
q9	8653	8628	8730	8628
q10	4045	3876	3725	3725
q11	574	487	498	487
q12	796	626	623	623
q13	16053	3160	3132	3132
q14	299	281	279	279
q15	520	499	469	469
q16	507	433	426	426
q17	1833	1548	1503	1503
q18	7926	7333	7328	7328
q19	1664	1538	1591	1538
q20	2002	1770	1792	1770
q21	11921	4642	4813	4642
q22	572	501	493	493
Total cold run time: 78422 ms
Total hot run time: 55056 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168642 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9031fce5fffba2785d346b9ddc767b42a7512caa, data reload: false

query1	897	372	362	362
query2	6428	2346	2460	2346
query3	6633	201	206	201
query4	19877	17382	17334	17334
query5	4105	419	410	410
query6	249	153	149	149
query7	4580	296	289	289
query8	235	188	185	185
query9	8411	2448	2416	2416
query10	460	282	266	266
query11	10628	10089	10007	10007
query12	139	95	91	91
query13	1658	373	370	370
query14	9422	7659	6492	6492
query15	210	168	169	168
query16	7883	265	271	265
query17	1711	530	500	500
query18	2003	269	267	267
query19	207	151	164	151
query20	89	86	84	84
query21	194	135	129	129
query22	4106	3923	4097	3923
query23	33375	33098	32738	32738
query24	10298	2870	2871	2870
query25	570	351	354	351
query26	705	150	155	150
query27	2195	325	322	322
query28	5734	2080	2084	2080
query29	877	596	596	596
query30	221	152	164	152
query31	980	764	724	724
query32	92	51	54	51
query33	658	269	257	257
query34	859	482	481	481
query35	738	621	600	600
query36	1075	883	917	883
query37	109	68	67	67
query38	2908	2741	2773	2741
query39	826	785	781	781
query40	197	126	123	123
query41	45	43	43	43
query42	102	94	97	94
query43	581	544	522	522
query44	1078	728	745	728
query45	181	191	155	155
query46	1062	730	729	729
query47	1825	1753	1786	1753
query48	363	289	303	289
query49	819	376	383	376
query50	767	403	385	385
query51	6916	6693	6716	6693
query52	101	97	90	90
query53	347	287	289	287
query54	872	429	427	427
query55	78	71	72	71
query56	258	237	244	237
query57	1104	1045	1087	1045
query58	225	211	242	211
query59	3507	3049	3359	3049
query60	274	250	258	250
query61	88	84	84	84
query62	601	466	453	453
query63	312	289	280	280
query64	8520	2207	1815	1815
query65	3186	3100	3141	3100
query66	846	332	322	322
query67	15463	14858	14645	14645
query68	4538	553	564	553
query69	448	264	269	264
query70	1111	1092	1127	1092
query71	387	265	267	265
query72	7745	5919	2684	2684
query73	727	330	330	330
query74	6038	5704	5639	5639
query75	3326	2631	2623	2623
query76	2301	1045	975	975
query77	433	267	271	267
query78	10296	9817	9878	9817
query79	2386	520	507	507
query80	1145	446	473	446
query81	535	226	212	212
query82	958	96	90	90
query83	263	168	166	166
query84	245	85	84	84
query85	1207	268	264	264
query86	493	300	281	281
query87	3249	3111	3123	3111
query88	3542	2439	2421	2421
query89	487	386	367	367
query90	1976	192	188	188
query91	123	98	95	95
query92	57	48	49	48
query93	1823	526	518	518
query94	1268	184	182	182
query95	403	310	358	310
query96	582	268	266	266
query97	3175	3009	3010	3009
query98	249	218	215	215
query99	1142	849	848	848
Total cold run time: 264721 ms
Total hot run time: 168642 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9031fce5fffba2785d346b9ddc767b42a7512caa, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.67	0.07	0.07
query5	0.50	0.47	0.48
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.54	0.55	0.53
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.60	0.59
query14	0.77	0.76	0.79
query15	0.82	0.84	0.82
query16	0.36	0.37	0.38
query17	0.92	0.96	0.94
query18	0.22	0.24	0.25
query19	1.77	1.69	1.65
query20	0.01	0.00	0.01
query21	15.47	0.70	0.67
query22	4.81	7.45	1.88
query23	18.32	1.31	1.28
query24	1.92	0.22	0.20
query25	0.14	0.07	0.08
query26	0.27	0.17	0.16
query27	0.08	0.08	0.08
query28	13.32	1.03	1.00
query29	12.66	3.36	3.26
query30	0.23	0.05	0.05
query31	2.93	0.39	0.37
query32	3.83	0.47	0.47
query33	2.94	2.92	2.91
query34	17.14	4.41	4.42
query35	4.52	4.51	4.54
query36	0.68	0.49	0.47
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.15	0.14	0.16
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.53 s
Total hot run time: 30.43 s

@morrySnow
[fix](Nereids) prune not require window expressions on window operator
fbadb21 
if window expression is not required by its parent, we should prune this
column. If all window expressions of window operator are pruned, we
remove this window operator directly.
@morrySnow morrySnow force-pushed the fix_window_column_pruning branch from 9031fce to fbadb21 Compare May 29, 2024 03:36
@morrySnow
Copy link
Contributor Author

run buildall

@morrySnow morrySnow mentioned this pull request May 29, 2024
Merged
@englefly
Copy link
Contributor

does this rule apply before the rule to choose column for count()?
example
select count()
from (
select 1.1 as A,
ROW_NUMBER() OVER() as B
from T ) V
in current version, count(*) will be replaced by count(B) because B is int and A is double.

@doris-robot
Copy link

TPC-H: Total hot run time: 40768 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fbadb21ae45ac6d534282a4788c7f14ee4755cfa, data reload: false

------ Round 1 ----------------------------------
q1	17887	4745	4412	4412
q2	2519	196	199	196
q3	11047	1185	1246	1185
q4	11113	810	791	791
q5	7909	2738	2663	2663
q6	221	139	135	135
q7	970	616	610	610
q8	9610	2081	2085	2081
q9	9016	6527	6517	6517
q10	8931	3790	3718	3718
q11	452	242	236	236
q12	454	222	217	217
q13	17765	2991	2966	2966
q14	259	222	231	222
q15	513	461	455	455
q16	538	378	389	378
q17	970	683	632	632
q18	8149	7446	7539	7446
q19	8060	1567	1462	1462
q20	656	307	305	305
q21	4979	3867	3956	3867
q22	326	280	274	274
Total cold run time: 122344 ms
Total hot run time: 40768 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4406	4226	4288	4226
q2	382	256	267	256
q3	2980	2764	2757	2757
q4	1888	1626	1569	1569
q5	5283	5272	5284	5272
q6	212	125	124	124
q7	2100	1758	1760	1758
q8	3215	3285	3314	3285
q9	8385	8336	8312	8312
q10	3931	3646	3646	3646
q11	582	485	479	479
q12	752	586	571	571
q13	16360	2982	3009	2982
q14	288	258	260	258
q15	522	457	472	457
q16	478	411	410	410
q17	1762	1494	1496	1494
q18	7609	7673	7487	7487
q19	2471	1582	1538	1538
q20	1978	1792	1759	1759
q21	4983	4785	4808	4785
q22	538	484	504	484
Total cold run time: 71105 ms
Total hot run time: 53909 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 167867 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fbadb21ae45ac6d534282a4788c7f14ee4755cfa, data reload: false

query1	929	392	375	375
query2	6466	2327	2257	2257
query3	6651	202	204	202
query4	19071	17429	17323	17323
query5	4138	415	431	415
query6	258	152	158	152
query7	4584	293	287	287
query8	250	180	182	180
query9	8501	2371	2340	2340
query10	455	279	266	266
query11	10699	10033	10079	10033
query12	144	97	93	93
query13	1663	365	378	365
query14	10038	6080	6961	6080
query15	215	163	165	163
query16	7827	264	252	252
query17	1645	513	502	502
query18	1941	287	267	267
query19	197	147	148	147
query20	90	87	82	82
query21	197	134	133	133
query22	4145	4054	4056	4054
query23	33397	32873	32807	32807
query24	12111	2837	2830	2830
query25	679	378	350	350
query26	1795	160	154	154
query27	2972	321	318	318
query28	7461	2046	2039	2039
query29	1099	610	601	601
query30	285	147	150	147
query31	970	749	754	749
query32	97	54	57	54
query33	767	271	261	261
query34	1001	488	464	464
query35	755	605	601	601
query36	1065	912	907	907
query37	149	71	66	66
query38	2869	2775	2737	2737
query39	860	787	788	787
query40	279	124	122	122
query41	45	44	44	44
query42	101	97	94	94
query43	610	537	570	537
query44	1211	722	743	722
query45	178	163	162	162
query46	1076	718	738	718
query47	1856	1786	1772	1772
query48	378	296	295	295
query49	1156	373	407	373
query50	775	392	388	388
query51	6926	6726	6765	6726
query52	101	92	91	91
query53	348	280	283	280
query54	969	425	425	425
query55	73	70	72	70
query56	261	236	255	236
query57	1148	1043	1041	1041
query58	239	249	213	213
query59	3308	3158	3025	3025
query60	268	252	253	252
query61	89	89	89	89
query62	662	466	434	434
query63	310	281	280	280
query64	9820	2208	1705	1705
query65	3199	3120	3114	3114
query66	1383	327	342	327
query67	15312	15048	14766	14766
query68	4548	532	526	526
query69	434	268	263	263
query70	1077	1094	1113	1094
query71	416	267	268	267
query72	7554	2708	2549	2549
query73	709	321	321	321
query74	6123	5703	5681	5681
query75	3520	2667	2633	2633
query76	2876	974	998	974
query77	423	260	259	259
query78	10186	9769	9645	9645
query79	2305	501	504	501
query80	1069	435	426	426
query81	510	221	221	221
query82	963	90	90	90
query83	245	167	175	167
query84	239	86	83	83
query85	1306	259	252	252
query86	494	296	318	296
query87	3232	3118	3159	3118
query88	3568	2340	2327	2327
query89	469	383	374	374
query90	1950	190	194	190
query91	135	110	108	108
query92	63	51	52	51
query93	1969	508	487	487
query94	1244	256	181	181
query95	398	309	331	309
query96	614	272	275	272
query97	3224	3002	3018	3002
query98	234	225	219	219
query99	1098	840	864	840
Total cold run time: 273712 ms
Total hot run time: 167867 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fbadb21ae45ac6d534282a4788c7f14ee4755cfa, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.06	0.07
query5	0.49	0.49	0.49
query6	1.13	0.75	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.50	0.48
query10	0.54	0.56	0.54
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.58	0.59	0.60
query14	0.78	0.77	0.80
query15	0.84	0.80	0.81
query16	0.36	0.36	0.37
query17	1.02	0.99	1.02
query18	0.24	0.22	0.25
query19	1.77	1.68	1.66
query20	0.02	0.01	0.01
query21	15.46	0.70	0.68
query22	5.08	7.42	1.48
query23	18.25	1.25	1.23
query24	2.02	0.24	0.20
query25	0.14	0.08	0.08
query26	0.26	0.16	0.17
query27	0.08	0.08	0.08
query28	13.33	1.01	1.00
query29	13.08	3.31	3.26
query30	0.24	0.06	0.06
query31	2.85	0.38	0.38
query32	3.28	0.49	0.46
query33	2.84	2.89	2.85
query34	16.96	4.48	4.43
query35	4.54	4.50	4.57
query36	0.66	0.46	0.46
query37	0.16	0.15	0.16
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.48 s
Total hot run time: 29.99 s

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 29, 2024
@morrySnow morrySnow merged commit 23aa430 into apache:master May 29, 2024
26 of 28 checks passed
@morrySnow morrySnow deleted the fix_window_column_pruning branch May 29, 2024 05:59
morrySnow added a commit that referenced this pull request May 29, 2024
@morrySnow
[fix](Nereids) prune not required window expressions on window operat…
1647702 
…or (#35577)

pick from master #35504 

if window expression is not required by its parent, we should prune this
column.
If all window expressions of window operator are pruned, we remove this
window operator directly.
@morrySnow morrySnow mentioned this pull request May 29, 2024
Merged
morrySnow added a commit that referenced this pull request May 29, 2024
@morrySnow
[fix](Nereids) prune not required window expressions on window operat…
1f5edae 
…or (#35593)

pick from master #35504

if window expression is not required by its parent, we should prune this
column. If all window expressions of window operator are pruned, we
remove this window operator directly.
dataroaring pushed a commit that referenced this pull request May 31, 2024
@morrySnow @dataroaring
[fix](Nereids) prune not required window expressions on window operat…
563b154 
…or (#35504)

if window expression is not required by its parent, we should prune this
column. If all window expressions of window operator are pruned, we
remove this window operator directly.
@morningman morningman mentioned this pull request Jun 22, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morrySnow
[fix](Nereids) prune not required window expressions on window operat…
4ab797d 
…or (apache#35577)

pick from master apache#35504

if window expression is not required by its parent, we should prune this
column.
If all window expressions of window operator are pruned, we remove this
window operator directly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@englefly englefly englefly approved these changes

@xzj7019 xzj7019 xzj7019 approved these changes

@924060929 924060929 Awaiting requested review from 924060929

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.11-merged dev/2.1.4-merged dev/3.0.0-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@morrySnow @doris-robot @englefly @xzj7019 @yiguolei @dataroaring