[feature](paimon)support paimon with dlf

[wuwenchi]

Proposed changes

We now support reading the paimon table on dlf.
We can create a catalog using dlf for paimon in the following way:

CREATE CATALOG `dlf_paimon` PROPERTIES (
"type" = "paimon",
"paimon.catalog.type" = "dlf",
"warehouse" = "oss://xx/yy/",
"dlf.proxy.mode" = "DLF_ONLY",
"dlf.uid" = "xxxxx",
"dlf.region" = "cn-beijing",
"dlf.access_key" = "ak",
"dlf.secret_key" = "sk"

-- "dlf.endpoint" = "dlf.cn-beijing.aliyuncs.com",  -- optional
-- "dlf.catalog.id" = "xxxx", -- optional
);

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[wuwenchi]

run buildall

[wuwenchi]

run compile

[wuwenchi]

run performance

[wuwenchi]

run buildall

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[morningman]

Need to add PaimonDLFExternalCatalog to GsonUtils

[wuwenchi]

run buildall

[doris-robot]

TPC-H: Total hot run time: 40885 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 52a2465633e3a040f6fc8cf445f2dce17827cb61, data reload: false

------ Round 1 ----------------------------------
q1	17570	7425	7233	7233
q2	2023	276	275	275
q3	12133	1051	1214	1051
q4	10569	742	708	708
q5	7748	2844	2811	2811
q6	237	148	147	147
q7	965	637	602	602
q8	9353	1920	1967	1920
q9	6505	6431	6460	6431
q10	6969	2304	2296	2296
q11	445	243	243	243
q12	406	219	211	211
q13	17798	2974	2981	2974
q14	233	212	221	212
q15	578	532	531	531
q16	632	584	587	584
q17	955	529	552	529
q18	7157	6647	6759	6647
q19	1407	1169	1046	1046
q20	496	206	210	206
q21	3945	3228	3209	3209
q22	1114	1040	1019	1019
Total cold run time: 109238 ms
Total hot run time: 40885 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7211	7246	7215	7215
q2	321	227	226	226
q3	2954	2928	2954	2928
q4	2021	1898	1791	1791
q5	5704	5715	5763	5715
q6	235	148	143	143
q7	2225	1846	1794	1794
q8	3351	3603	3410	3410
q9	8925	8916	8839	8839
q10	3591	3557	3500	3500
q11	597	488	489	488
q12	867	664	606	606
q13	9283	3165	3181	3165
q14	309	271	277	271
q15	569	543	529	529
q16	680	651	646	646
q17	1851	1627	1601	1601
q18	8178	7720	7583	7583
q19	1739	1433	1450	1433
q20	2110	1874	1932	1874
q21	5675	5368	5398	5368
q22	1137	1084	1120	1084
Total cold run time: 69533 ms
Total hot run time: 60209 ms

[doris-robot]

TPC-DS: Total hot run time: 192759 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 52a2465633e3a040f6fc8cf445f2dce17827cb61, data reload: false

query1	924	385	409	385
query2	6265	2127	2081	2081
query3	8693	198	203	198
query4	33630	23660	23423	23423
query5	3507	462	464	462
query6	276	158	159	158
query7	4203	313	305	305
query8	275	224	223	223
query9	9241	2679	2681	2679
query10	466	272	292	272
query11	17754	15220	15269	15220
query12	150	99	97	97
query13	1539	421	414	414
query14	9755	7371	7375	7371
query15	250	170	178	170
query16	8027	442	433	433
query17	1667	624	615	615
query18	2157	322	317	317
query19	368	159	153	153
query20	125	119	112	112
query21	219	111	108	108
query22	5109	4900	4724	4724
query23	35644	34275	33975	33975
query24	11428	2857	2835	2835
query25	588	425	429	425
query26	1334	160	164	160
query27	2601	301	303	301
query28	7668	2453	2425	2425
query29	705	441	438	438
query30	252	154	155	154
query31	1037	815	779	779
query32	101	54	58	54
query33	774	298	311	298
query34	915	505	491	491
query35	898	742	732	732
query36	1080	912	944	912
query37	152	89	87	87
query38	4014	3880	3898	3880
query39	1466	1423	1481	1423
query40	264	99	100	99
query41	52	50	49	49
query42	116	94	96	94
query43	539	481	489	481
query44	1265	813	799	799
query45	203	171	166	166
query46	1148	729	730	729
query47	1952	1873	1862	1862
query48	473	367	370	367
query49	997	438	407	407
query50	833	421	427	421
query51	7043	6831	6957	6831
query52	100	89	88	88
query53	251	187	183	183
query54	1260	487	471	471
query55	81	77	79	77
query56	300	280	270	270
query57	1205	1117	1081	1081
query58	248	237	282	237
query59	3252	3042	3078	3042
query60	297	265	269	265
query61	106	104	109	104
query62	835	660	669	660
query63	218	183	182	182
query64	4985	648	618	618
query65	3229	3179	3232	3179
query66	1089	312	305	305
query67	16113	15872	15724	15724
query68	4399	564	556	556
query69	552	296	292	292
query70	1128	1064	1094	1064
query71	412	278	269	269
query72	7427	4169	4044	4044
query73	775	339	357	339
query74	10526	9033	9157	9033
query75	3803	2677	2696	2677
query76	3342	923	947	923
query77	629	296	308	296
query78	10616	9680	9612	9612
query79	2673	604	603	603
query80	1198	453	435	435
query81	578	239	238	238
query82	663	135	137	135
query83	304	134	135	134
query84	272	80	84	80
query85	1806	297	288	288
query86	443	296	298	296
query87	4417	4336	4355	4336
query88	3858	2406	2376	2376
query89	406	283	277	277
query90	2090	185	189	185
query91	172	147	141	141
query92	67	50	49	49
query93	2412	568	552	552
query94	1054	290	288	288
query95	368	260	261	260
query96	623	279	279	279
query97	3264	3121	3138	3121
query98	223	216	196	196
query99	1682	1315	1305	1305
Total cold run time: 305893 ms
Total hot run time: 192759 ms

[doris-robot]

ClickBench: Total hot run time: 32.23 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 52a2465633e3a040f6fc8cf445f2dce17827cb61, data reload: false

query1	0.05	0.05	0.05
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.64	0.10	0.11
query5	0.52	0.51	0.52
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.02
query9	0.55	0.51	0.50
query10	0.55	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.59	0.59
query14	2.74	2.79	2.69
query15	0.90	0.83	0.82
query16	0.39	0.36	0.37
query17	1.08	1.05	1.06
query18	0.20	0.19	0.19
query19	1.93	1.87	2.05
query20	0.01	0.02	0.01
query21	15.35	0.57	0.57
query22	2.32	1.88	1.52
query23	16.80	1.03	0.82
query24	2.58	0.76	1.49
query25	0.21	0.19	0.11
query26	0.32	0.14	0.14
query27	0.05	0.04	0.04
query28	10.86	1.09	1.08
query29	12.59	3.24	3.23
query30	0.27	0.08	0.07
query31	2.84	0.38	0.38
query32	3.28	0.46	0.46
query33	3.12	3.14	3.08
query34	16.94	4.42	4.45
query35	4.48	4.44	4.52
query36	0.65	0.51	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.14	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.06 s
Total hot run time: 32.23 s

[wuwenchi]

run buildall

[doris-robot]

TPC-H: Total hot run time: 40636 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c8f6df505b862029ca7d340d9c3c2b55a9f2dda, data reload: false

------ Round 1 ----------------------------------
q1	18144	7489	7349	7349
q2	2442	173	157	157
q3	12104	1150	1123	1123
q4	10590	772	703	703
q5	8698	2899	2870	2870
q6	228	157	154	154
q7	981	624	597	597
q8	9343	1849	1969	1849
q9	6637	6388	6371	6371
q10	6991	2292	2310	2292
q11	448	246	246	246
q12	405	219	213	213
q13	17815	2989	2971	2971
q14	241	216	216	216
q15	565	523	530	523
q16	651	592	601	592
q17	966	599	568	568
q18	7239	6705	6687	6687
q19	1401	1059	996	996
q20	476	200	196	196
q21	4042	3098	2985	2985
q22	1106	991	978	978
Total cold run time: 111513 ms
Total hot run time: 40636 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7272	7233	7166	7166
q2	323	230	229	229
q3	2923	2754	2845	2754
q4	1953	1724	1720	1720
q5	5471	5466	5457	5457
q6	224	141	137	137
q7	2066	1743	1691	1691
q8	3245	3373	3430	3373
q9	8476	8525	8482	8482
q10	3494	3413	3424	3413
q11	581	473	469	469
q12	778	563	589	563
q13	7265	3015	2994	2994
q14	284	270	254	254
q15	561	516	507	507
q16	669	642	651	642
q17	1793	1588	1540	1540
q18	7838	7413	7264	7264
q19	1658	1559	1532	1532
q20	2046	1813	1853	1813
q21	5510	5219	5321	5219
q22	1090	1023	985	985
Total cold run time: 65520 ms
Total hot run time: 58204 ms

[doris-robot]

TPC-DS: Total hot run time: 192032 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0c8f6df505b862029ca7d340d9c3c2b55a9f2dda, data reload: false

query1	984	367	378	367
query2	6533	2113	2103	2103
query3	6703	214	221	214
query4	33975	23377	23329	23329
query5	4304	463	450	450
query6	264	168	156	156
query7	4609	304	302	302
query8	273	236	220	220
query9	9427	2645	2635	2635
query10	472	293	293	293
query11	18222	14992	15089	14992
query12	158	103	97	97
query13	1641	418	406	406
query14	10596	7358	7380	7358
query15	321	170	189	170
query16	8030	471	473	471
query17	1806	580	589	580
query18	2130	307	314	307
query19	366	154	149	149
query20	121	109	112	109
query21	214	105	109	105
query22	4543	4538	4484	4484
query23	34698	34278	34138	34138
query24	11168	2919	2835	2835
query25	654	411	416	411
query26	1420	162	160	160
query27	2829	287	294	287
query28	8000	2413	2399	2399
query29	883	432	425	425
query30	327	163	169	163
query31	1049	806	840	806
query32	100	57	58	57
query33	785	313	313	313
query34	937	508	485	485
query35	916	728	751	728
query36	1087	948	951	948
query37	160	91	90	90
query38	3976	3869	3867	3867
query39	1609	1446	1407	1407
query40	279	100	98	98
query41	52	52	50	50
query42	122	97	99	97
query43	533	494	499	494
query44	1282	834	800	800
query45	200	169	169	169
query46	1137	714	724	714
query47	1940	1843	1881	1843
query48	469	384	388	384
query49	1148	416	393	393
query50	829	410	416	410
query51	7083	7075	7028	7028
query52	105	87	87	87
query53	257	188	181	181
query54	1338	466	486	466
query55	79	78	80	78
query56	283	263	263	263
query57	1226	1146	1143	1143
query58	240	246	237	237
query59	3281	3010	2965	2965
query60	304	270	280	270
query61	137	103	102	102
query62	872	681	666	666
query63	220	189	187	187
query64	5281	635	618	618
query65	3339	3183	3204	3183
query66	1436	308	325	308
query67	16030	15769	15763	15763
query68	4851	567	555	555
query69	526	284	296	284
query70	1150	1130	1072	1072
query71	367	279	265	265
query72	7119	4107	4045	4045
query73	772	343	341	341
query74	9780	9020	8917	8917
query75	3416	2697	2675	2675
query76	3346	910	851	851
query77	442	303	296	296
query78	10710	9724	9548	9548
query79	2858	596	632	596
query80	1640	452	453	452
query81	598	244	241	241
query82	868	152	138	138
query83	261	148	141	141
query84	255	91	77	77
query85	1280	283	279	279
query86	382	267	291	267
query87	4478	4240	4284	4240
query88	3691	2396	2340	2340
query89	409	294	281	281
query90	1977	189	188	188
query91	206	145	167	145
query92	66	49	51	49
query93	1474	549	555	549
query94	1078	264	313	264
query95	346	252	256	252
query96	612	280	278	278
query97	3316	3130	3143	3130
query98	222	197	197	197
query99	1534	1309	1266	1266
Total cold run time: 305822 ms
Total hot run time: 192032 ms

[doris-robot]

ClickBench: Total hot run time: 32.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0c8f6df505b862029ca7d340d9c3c2b55a9f2dda, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.03
query3	0.22	0.06	0.06
query4	1.66	0.10	0.10
query5	0.51	0.51	0.50
query6	1.13	0.72	0.71
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.54	0.57	0.54
query11	0.13	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.59	0.61
query14	2.71	2.72	2.71
query15	0.89	0.82	0.82
query16	0.36	0.40	0.38
query17	1.05	1.07	1.05
query18	0.24	0.20	0.22
query19	1.95	1.81	1.96
query20	0.01	0.01	0.02
query21	15.37	0.62	0.60
query22	2.81	2.27	2.50
query23	16.84	0.98	0.87
query24	3.37	2.29	0.35
query25	0.35	0.13	0.06
query26	0.45	0.14	0.15
query27	0.05	0.03	0.03
query28	10.22	1.10	1.06
query29	12.58	3.27	3.19
query30	0.24	0.06	0.05
query31	2.87	0.39	0.37
query32	3.29	0.49	0.46
query33	2.99	3.02	3.02
query34	17.12	4.45	4.50
query35	4.51	4.59	4.51
query36	0.67	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.11	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.07 s
Total hot run time: 32.49 s

[morningman]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.