Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement](inverted index) apply inverted index when has any #41547 #41585

Merged
merged 1 commit into from
Oct 9, 2024
Merged

[Enhancement](inverted index) apply inverted index when has any #41547 #41585

merged 1 commit into from
Oct 9, 2024

Conversation

airborne12
Copy link
Member

cherry pick from #41547

@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

TPC-H: Total hot run time: 40172 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 257ac14caf9d7fc2f279fff8fac76d5a299f2750, data reload: false

------ Round 1 ----------------------------------
q1	17937	7462	7301	7301
q2	2054	155	148	148
q3	10774	1057	1145	1057
q4	10539	762	767	762
q5	7742	2807	2774	2774
q6	227	144	144	144
q7	1006	671	618	618
q8	9348	1906	1942	1906
q9	6601	6416	6427	6416
q10	6981	2263	2293	2263
q11	444	251	255	251
q12	408	218	220	218
q13	17783	2999	2968	2968
q14	251	216	208	208
q15	563	520	504	504
q16	493	409	397	397
q17	977	533	600	533
q18	7303	6637	6497	6497
q19	2573	984	907	907
q20	577	266	279	266
q21	3886	3205	3070	3070
q22	1091	982	964	964
Total cold run time: 109558 ms
Total hot run time: 40172 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7354	7251	7230	7230
q2	330	236	233	233
q3	2858	2841	2859	2841
q4	2022	1802	1841	1802
q5	5703	5687	5702	5687
q6	220	141	140	140
q7	2213	1736	1825	1736
q8	3341	3520	3557	3520
q9	8759	8833	8826	8826
q10	3549	3504	3498	3498
q11	586	477	476	476
q12	801	597	587	587
q13	16476	3108	3180	3108
q14	311	275	270	270
q15	567	520	520	520
q16	502	459	464	459
q17	1870	1647	1600	1600
q18	8211	7835	7564	7564
q19	5013	1555	1460	1460
q20	2109	1856	1890	1856
q21	5387	5210	5253	5210
q22	1121	1011	1004	1004
Total cold run time: 79303 ms
Total hot run time: 59627 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193605 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 257ac14caf9d7fc2f279fff8fac76d5a299f2750, data reload: false

query1	1265	879	918	879
query2	6264	2007	1992	1992
query3	10777	3849	3725	3725
query4	68130	25653	23511	23511
query5	5688	458	475	458
query6	485	170	166	166
query7	6316	298	302	298
query8	299	219	203	203
query9	9400	2645	2599	2599
query10	506	308	280	280
query11	18348	15249	16017	15249
query12	155	102	111	102
query13	1634	477	462	462
query14	11458	7472	7290	7290
query15	232	178	178	178
query16	6908	520	499	499
query17	1073	553	553	553
query18	1698	306	301	301
query19	225	150	139	139
query20	113	103	108	103
query21	210	99	99	99
query22	4564	4596	4463	4463
query23	34349	34042	33782	33782
query24	5584	2901	2841	2841
query25	519	416	400	400
query26	671	169	162	162
query27	1699	298	309	298
query28	3889	2182	2174	2174
query29	662	420	425	420
query30	225	149	150	149
query31	971	766	796	766
query32	71	53	53	53
query33	433	284	296	284
query34	901	491	486	486
query35	845	742	730	730
query36	1068	938	918	918
query37	139	77	80	77
query38	3957	3967	3847	3847
query39	1488	1483	1425	1425
query40	199	109	98	98
query41	45	42	43	42
query42	124	95	97	95
query43	517	476	476	476
query44	1129	779	791	779
query45	204	177	171	171
query46	1116	706	705	705
query47	1928	1856	1833	1833
query48	433	346	326	326
query49	703	385	384	384
query50	831	397	400	397
query51	7028	6793	6968	6793
query52	98	91	86	86
query53	257	186	180	180
query54	569	465	446	446
query55	77	73	73	73
query56	263	258	244	244
query57	1235	1141	1115	1115
query58	219	215	229	215
query59	3301	2774	2873	2774
query60	287	262	256	256
query61	103	101	102	101
query62	783	678	675	675
query63	220	187	177	177
query64	1678	614	592	592
query65	3227	3149	3167	3149
query66	731	302	298	298
query67	15579	15352	15375	15352
query68	4218	538	541	538
query69	616	283	285	283
query70	1179	1115	1142	1115
query71	435	277	270	270
query72	7110	3826	3913	3826
query73	761	354	339	339
query74	10396	8957	8902	8902
query75	4321	2618	2644	2618
query76	3421	900	865	865
query77	753	284	285	284
query78	9945	9076	9232	9076
query79	3571	591	596	591
query80	2301	427	432	427
query81	556	225	245	225
query82	863	129	124	124
query83	308	137	136	136
query84	296	81	81	81
query85	1905	283	284	283
query86	442	307	299	299
query87	4503	4268	4234	4234
query88	4863	2358	2376	2358
query89	405	287	295	287
query90	2107	179	184	179
query91	149	104	103	103
query92	58	47	46	46
query93	5370	533	521	521
query94	1079	277	293	277
query95	349	254	247	247
query96	614	283	285	283
query97	3358	3142	3117	3117
query98	213	195	198	195
query99	1746	1288	1283	1283
Total cold run time: 334314 ms
Total hot run time: 193605 ms

@airborne12
[Enhancement](inverted index) apply inverted index when has any (apac…
e6e30d5 
…he#41547)

## Proposed changes

This PR optimizes the inverted index logic by skipping the execution of
inverted index conditions when none are present. This change improves
performance by avoiding unnecessary inverted index condition flow.
@airborne12 airborne12 force-pushed the pick_41547_to_origin_branch-3.0 branch from 257ac14 to e6e30d5 Compare October 9, 2024 07:35
@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40538 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e6e30d531039e036ec7d10a214c41cc375ea1ca4, data reload: false

------ Round 1 ----------------------------------
q1	18156	8209	7529	7529
q2	2725	187	182	182
q3	11417	1118	1146	1118
q4	11178	763	783	763
q5	8425	2837	2899	2837
q6	227	151	164	151
q7	1034	670	617	617
q8	9339	1859	1998	1859
q9	6618	6390	6351	6351
q10	6988	2278	2280	2278
q11	447	255	252	252
q12	411	223	216	216
q13	17776	2951	2973	2951
q14	237	215	204	204
q15	559	502	505	502
q16	491	418	402	402
q17	989	599	531	531
q18	7361	6543	6604	6543
q19	3037	966	924	924
q20	569	275	282	275
q21	3840	3155	3093	3093
q22	1066	969	960	960
Total cold run time: 112890 ms
Total hot run time: 40538 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7347	7271	7336	7271
q2	323	232	230	230
q3	2932	2702	2708	2702
q4	1887	1663	1692	1663
q5	5374	5440	5406	5406
q6	226	143	144	143
q7	2099	1632	1662	1632
q8	3213	3409	3406	3406
q9	8543	8494	8479	8479
q10	3436	3369	3406	3369
q11	566	464	488	464
q12	759	571	581	571
q13	16912	2935	2982	2935
q14	303	264	255	255
q15	555	502	508	502
q16	496	436	436	436
q17	1776	1563	1538	1538
q18	7494	7323	7318	7318
q19	1670	1623	1504	1504
q20	1988	1785	1780	1780
q21	5113	5029	4894	4894
q22	1072	950	951	950
Total cold run time: 74084 ms
Total hot run time: 57448 ms

@airborne12 airborne12 merged commit ba825ef into apache:branch-3.0 Oct 9, 2024
14 of 21 checks passed
@airborne12 airborne12 deleted the pick_41547_to_origin_branch-3.0 branch October 9, 2024 07:55
@doris-robot
Copy link

TPC-DS: Total hot run time: 188686 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e6e30d531039e036ec7d10a214c41cc375ea1ca4, data reload: false

query1	968	374	354	354
query2	6548	1997	1920	1920
query3	6702	212	220	212
query4	34085	23455	23388	23388
query5	4241	441	445	441
query6	256	166	176	166
query7	4610	299	303	299
query8	243	198	208	198
query9	9627	2634	2614	2614
query10	473	283	267	267
query11	18151	15218	15125	15125
query12	149	99	98	98
query13	1681	440	445	440
query14	10468	7409	7363	7363
query15	224	180	173	173
query16	7650	470	492	470
query17	1622	543	521	521
query18	1835	289	287	287
query19	229	136	145	136
query20	116	106	103	103
query21	209	99	97	97
query22	4437	4311	4236	4236
query23	34615	33824	33775	33775
query24	12023	2807	2787	2787
query25	667	374	373	373
query26	1745	157	156	156
query27	2795	294	292	292
query28	7842	2120	2079	2079
query29	1001	421	421	421
query30	317	161	140	140
query31	1033	793	802	793
query32	96	53	55	53
query33	758	276	288	276
query34	996	478	485	478
query35	892	713	723	713
query36	1111	959	958	958
query37	262	81	75	75
query38	3935	3890	3850	3850
query39	1460	1413	1379	1379
query40	298	93	92	92
query41	43	44	40	40
query42	119	96	93	93
query43	513	471	488	471
query44	1235	775	773	773
query45	199	163	166	163
query46	1134	712	707	707
query47	1876	1773	1771	1771
query48	479	336	329	329
query49	1242	380	390	380
query50	818	392	415	392
query51	7056	6818	6986	6818
query52	97	99	86	86
query53	259	188	191	188
query54	1261	456	445	445
query55	73	76	77	76
query56	269	241	246	241
query57	1221	1116	1149	1116
query58	252	232	248	232
query59	3039	3020	2755	2755
query60	295	269	270	269
query61	121	116	121	116
query62	874	686	660	660
query63	214	186	189	186
query64	5398	700	679	679
query65	3210	3145	3160	3145
query66	1404	307	314	307
query67	15575	15229	15150	15150
query68	4629	543	543	543
query69	446	292	290	290
query70	1188	1053	1126	1053
query71	339	273	284	273
query72	6406	4013	4061	4013
query73	750	335	330	330
query74	10105	8883	8941	8883
query75	3351	2651	2624	2624
query76	2811	841	846	841
query77	415	295	292	292
query78	10034	9219	9146	9146
query79	2478	602	578	578
query80	2877	420	434	420
query81	579	220	226	220
query82	973	126	124	124
query83	304	137	137	137
query84	292	86	79	79
query85	2285	285	279	279
query86	499	316	295	295
query87	4444	4271	4190	4190
query88	4409	2370	2387	2370
query89	410	297	288	288
query90	1886	185	182	182
query91	135	103	122	103
query92	71	48	48	48
query93	2292	535	523	523
query94	960	281	293	281
query95	359	251	249	249
query96	600	274	275	274
query97	3285	3083	3119	3083
query98	218	198	197	197
query99	1831	1303	1292	1292
Total cold run time: 306307 ms
Total hot run time: 188686 ms

@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@airborne12 @doris-robot