Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[branch-3.0](auto bucket) fix calc using the first k partition #41675 #41835

Merged
merged 1 commit into from
Oct 16, 2024
Merged

[branch-3.0](auto bucket) fix calc using the first k partition #41675 #41835

merged 1 commit into from
Oct 16, 2024
+63 −19

Conversation

yujun777
Copy link
Collaborator

cherry pick: #41675

@yujun777
[fix](auto bucket) fix auto buckets calc using the first k partition (a…
Loading
Loading status checks…
0af48dc 
…pache#41675)

If the first k (at most 7) partition data size is ascending, the result
will be partion_size[k-1] + ema(first k partitons delta).

This is a bug, should use the last k partitions, but not the first k
partitions to calculate.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39889 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0af48dc8f44fb0a260634212c918b4f8c401e903, data reload: false

------ Round 1 ----------------------------------
q1	17602	7354	7226	7226
q2	2030	151	147	147
q3	10756	1080	1107	1080
q4	10538	707	803	707
q5	7744	2746	2762	2746
q6	227	149	152	149
q7	1022	616	595	595
q8	9552	1915	1932	1915
q9	7454	6385	6395	6385
q10	6961	2287	2270	2270
q11	434	241	246	241
q12	399	220	219	219
q13	17784	2971	2943	2943
q14	240	219	207	207
q15	563	502	513	502
q16	482	415	395	395
q17	969	567	528	528
q18	7159	6518	6554	6518
q19	1429	982	945	945
q20	540	278	275	275
q21	3911	2939	2957	2939
q22	1050	986	957	957
Total cold run time: 108846 ms
Total hot run time: 39889 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7531	7188	7178	7178
q2	319	229	227	227
q3	2880	2820	2818	2818
q4	1984	1718	1711	1711
q5	5721	5717	5681	5681
q6	226	148	144	144
q7	2183	1773	1749	1749
q8	3342	3463	3479	3463
q9	8813	8874	8768	8768
q10	3575	3491	3485	3485
q11	568	482	488	482
q12	805	599	590	590
q13	16435	3216	3108	3108
q14	306	283	278	278
q15	565	536	520	520
q16	520	459	455	455
q17	1833	1598	1584	1584
q18	8177	7755	7561	7561
q19	7576	1564	1462	1462
q20	2069	1795	1836	1795
q21	5550	5258	5281	5258
q22	1089	1006	1011	1006
Total cold run time: 82067 ms
Total hot run time: 59323 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193173 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0af48dc8f44fb0a260634212c918b4f8c401e903, data reload: false

query1	1258	870	858	858
query2	6287	2044	1959	1959
query3	10790	3909	3725	3725
query4	62418	29061	23578	23578
query5	5128	450	450	450
query6	374	160	162	160
query7	5404	309	300	300
query8	273	204	201	201
query9	8130	2669	2656	2656
query10	446	286	272	272
query11	16842	15104	15897	15104
query12	150	112	109	109
query13	1440	456	444	444
query14	9642	8156	6847	6847
query15	227	187	176	176
query16	6723	501	518	501
query17	1179	587	588	587
query18	1014	309	332	309
query19	225	150	149	149
query20	119	115	113	113
query21	206	103	108	103
query22	4778	4743	4820	4743
query23	34568	33745	33723	33723
query24	5981	2860	2791	2791
query25	510	400	398	398
query26	649	158	158	158
query27	1653	303	292	292
query28	4160	2198	2160	2160
query29	655	429	432	429
query30	232	145	163	145
query31	962	766	775	766
query32	69	56	53	53
query33	442	290	296	290
query34	902	485	506	485
query35	876	713	733	713
query36	1061	957	945	945
query37	136	74	81	74
query38	3988	3849	3974	3849
query39	1503	1432	1427	1427
query40	212	97	97	97
query41	47	43	42	42
query42	124	99	99	99
query43	512	487	474	474
query44	1134	771	771	771
query45	194	162	164	162
query46	1124	697	683	683
query47	1931	1845	1852	1845
query48	430	342	334	334
query49	678	390	404	390
query50	804	400	417	400
query51	7061	6942	6868	6868
query52	96	89	85	85
query53	254	181	181	181
query54	565	476	449	449
query55	75	74	74	74
query56	271	245	243	243
query57	1237	1137	1102	1102
query58	216	231	244	231
query59	3045	2895	2796	2796
query60	293	264	275	264
query61	103	100	99	99
query62	774	656	652	652
query63	212	181	181	181
query64	1361	648	603	603
query65	3271	3154	3157	3154
query66	677	298	328	298
query67	15806	15190	15360	15190
query68	3746	527	526	526
query69	757	287	270	270
query70	1124	1070	1115	1070
query71	518	273	271	271
query72	8393	3835	3659	3659
query73	767	347	333	333
query74	10691	8939	8918	8918
query75	5093	2626	2643	2626
query76	4223	896	845	845
query77	791	289	287	287
query78	9916	9295	9707	9295
query79	4368	577	578	577
query80	1553	434	432	432
query81	556	223	223	223
query82	569	126	125	125
query83	396	134	135	134
query84	299	76	79	76
query85	1656	290	276	276
query86	458	293	269	269
query87	4602	4347	4257	4257
query88	4482	2456	2376	2376
query89	402	280	281	280
query90	2151	187	185	185
query91	152	102	104	102
query92	63	44	46	44
query93	4315	532	515	515
query94	1103	288	291	288
query95	350	247	244	244
query96	629	271	274	271
query97	3294	3116	3189	3116
query98	216	204	191	191
query99	1668	1310	1283	1283
Total cold run time: 323208 ms
Total hot run time: 193173 ms

@yujun777
Copy link
Collaborator Author

run p0

@yujun777
Copy link
Collaborator Author

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 40073 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0af48dc8f44fb0a260634212c918b4f8c401e903, data reload: false

------ Round 1 ----------------------------------
q1	17590	7354	7276	7276
q2	2054	155	147	147
q3	10719	1084	1106	1084
q4	10541	705	786	705
q5	7747	2761	2835	2761
q6	230	149	146	146
q7	989	643	644	643
q8	9584	1885	1918	1885
q9	8137	6425	6387	6387
q10	7008	2279	2304	2279
q11	440	249	252	249
q12	418	216	217	216
q13	17777	2949	2940	2940
q14	246	217	203	203
q15	566	518	512	512
q16	509	410	408	408
q17	970	544	536	536
q18	7096	6512	6517	6512
q19	1890	1000	971	971
q20	559	275	263	263
q21	3898	3021	2978	2978
q22	1084	995	972	972
Total cold run time: 110052 ms
Total hot run time: 40073 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7292	7254	7257	7254
q2	325	225	228	225
q3	2865	2856	2859	2856
q4	1997	1766	1769	1766
q5	5684	5672	5692	5672
q6	225	147	146	146
q7	2187	1794	1769	1769
q8	3366	3478	3545	3478
q9	8677	8859	8773	8773
q10	3526	3520	3552	3520
q11	564	484	475	475
q12	766	586	580	580
q13	16535	3154	3165	3154
q14	307	284	264	264
q15	575	522	545	522
q16	525	473	461	461
q17	1843	1627	1634	1627
q18	8115	7772	7551	7551
q19	1728	1355	1475	1355
q20	2147	1880	1865	1865
q21	5335	5208	5258	5208
q22	1123	1047	995	995
Total cold run time: 75707 ms
Total hot run time: 59516 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193463 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0af48dc8f44fb0a260634212c918b4f8c401e903, data reload: false

query1	1303	892	890	890
query2	6268	1963	1900	1900
query3	10811	3964	3645	3645
query4	66809	25039	23587	23587
query5	5581	463	462	462
query6	480	175	194	175
query7	6300	312	310	310
query8	306	209	219	209
query9	9683	2659	2663	2659
query10	528	279	273	273
query11	18221	15097	15760	15097
query12	162	98	106	98
query13	1645	478	431	431
query14	11451	7472	7341	7341
query15	235	182	175	175
query16	7290	485	497	485
query17	1058	582	569	569
query18	1684	288	301	288
query19	194	149	141	141
query20	113	101	105	101
query21	202	99	98	98
query22	4342	4177	4339	4177
query23	34157	33723	34396	33723
query24	5582	2966	2898	2898
query25	503	406	426	406
query26	676	159	169	159
query27	1709	297	289	289
query28	3895	2171	2159	2159
query29	657	434	435	434
query30	231	151	149	149
query31	957	781	769	769
query32	68	53	53	53
query33	477	295	308	295
query34	880	509	506	506
query35	894	747	745	745
query36	1027	954	938	938
query37	129	80	76	76
query38	3926	3845	3906	3845
query39	1491	1415	1554	1415
query40	213	101	98	98
query41	45	43	42	42
query42	119	98	99	98
query43	528	464	465	464
query44	1117	762	779	762
query45	195	164	162	162
query46	1110	730	709	709
query47	1886	1833	1830	1830
query48	441	364	332	332
query49	706	405	409	405
query50	791	414	409	409
query51	7202	6821	6937	6821
query52	106	90	98	90
query53	257	196	185	185
query54	586	466	466	466
query55	76	82	76	76
query56	288	260	253	253
query57	1213	1187	1146	1146
query58	236	244	262	244
query59	3033	2890	2908	2890
query60	308	270	272	270
query61	124	119	117	117
query62	792	643	649	643
query63	223	193	190	190
query64	1762	729	587	587
query65	3246	3144	3161	3144
query66	681	302	299	299
query67	15773	15306	15260	15260
query68	3823	543	540	540
query69	687	294	289	289
query70	1166	1104	1084	1084
query71	457	283	266	266
query72	7667	3921	3981	3921
query73	757	340	341	340
query74	10316	8910	8876	8876
query75	4206	2663	2642	2642
query76	3422	918	884	884
query77	772	289	283	283
query78	9988	9191	9105	9105
query79	2662	597	583	583
query80	2623	450	443	443
query81	572	226	221	221
query82	826	128	128	128
query83	317	131	148	131
query84	296	86	83	83
query85	1819	307	291	291
query86	459	295	296	295
query87	4620	4192	4307	4192
query88	4336	2366	2401	2366
query89	431	297	292	292
query90	2174	190	182	182
query91	134	104	109	104
query92	62	48	45	45
query93	5372	534	521	521
query94	1069	292	296	292
query95	361	256	257	256
query96	643	281	281	281
query97	3336	3103	3139	3103
query98	213	198	198	198
query99	1595	1264	1299	1264
Total cold run time: 332163 ms
Total hot run time: 193463 ms

@dataroaring dataroaring merged commit ae3db25 into apache:branch-3.0 Oct 16, 2024
21 of 25 checks passed
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@yujun777 @doris-robot @dataroaring