Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cdc task auto detect db/tbl #20701

Merged
merged 5 commits into from
Dec 18, 2024
Merged

Cdc task auto detect db/tbl #20701

merged 5 commits into from
Dec 18, 2024

Conversation

ck89119
Copy link
Contributor

@ck89119 ck89119 commented Dec 10, 2024 •
edited
Loading

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue #20539 #20542

What this PR does / why we need it:

  1. 支持account，db级别同步，具体格式参考文档：https://doc.weixin.qq.com/doc/w3_AVQAQwZ_AEgrnqKicSrR0CJ1Dqxe8?scode=AJsA6gc3AA8P3rA66vAVQAQwZ_AEg
  2. 支持exclude选项，排除指定范围内某些db/table，格式同上
  3. 定期扫描所有库表，若检测到同步范围的新db/table，会自动拉起同步任务，下游若不存在该db/table会尝试进行自动创建
  4. 其他微调：新建任务时不再会check db/table是否存在；同一任务内不允许同一张表同时作为多个同步源

@matrix-meow matrix-meow added the size/XXL Denotes a PR that changes 2000+ lines label Dec 10, 2024
@ck89119 ck89119 added this to the 2.1.0 milestone Dec 10, 2024
@mergify mergify bot requested a review from sukki37 December 10, 2024 11:21
@mergify mergify bot added the kind/feature label Dec 10, 2024
@ck89119 ck89119 changed the title Auto detect db/tbl Cdc task auto detect db/tbl Dec 10, 2024
@ck89119 ck89119 requested a review from ouyuanning as a code owner December 11, 2024 10:54
@mergify mergify bot merged commit a7f27c7 into matrixorigin:2.0-dev Dec 18, 2024
18 checks passed
@ck89119 ck89119 deleted the auto_detect_db_tbl branch December 18, 2024 02:51
ck89119 added a commit to ck89119/matrixone that referenced this pull request Dec 18, 2024
@ck89119
Cdc task auto detect db/tbl (matrixorigin#20701)
d2a64dd 
1. 支持account，db级别同步，具体格式参考文档：https://doc.weixin.qq.com/doc/w3_AVQAQwZ_AEgrnqKicSrR0CJ1Dqxe8?scode=AJsA6gc3AA8P3rA66vAVQAQwZ_AEg
2. 支持exclude选项，排除指定范围内某些db/table，格式同上
3. 定期扫描所有库表，若检测到同步范围的新db/table，会自动拉起同步任务，下游若不存在该db/table会尝试进行自动创建
4. 其他微调：新建任务时不再会check db/table是否存在；同一任务内不允许同一张表同时作为多个同步源

Approved by: @daviszhen, @zhangxu19830126, @XuPeng-SH, @sukki37
ck89119 added a commit to ck89119/matrixone that referenced this pull request Dec 26, 2024
@ck89119
Revert "Cdc task auto detect db/tbl (matrixorigin#20701)"
364a590 
This reverts commit a7f27c7.
@ck89119 ck89119 mentioned this pull request Dec 26, 2024
Merged
7 tasks
mergify bot pushed a commit that referenced this pull request Dec 26, 2024
@ck89119
Revert "Cdc task auto detect db/tbl (#20701)" (#20952)
e33eb96 
Revert "Cdc task auto detect db/tbl (#20701)"

Approved by: @daviszhen, @zhangxu19830126, @XuPeng-SH, @sukki37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhangxu19830126 zhangxu19830126 zhangxu19830126 approved these changes

@XuPeng-SH XuPeng-SH XuPeng-SH approved these changes

@daviszhen daviszhen daviszhen approved these changes

@sukki37 sukki37 sukki37 approved these changes

@qingxinhome qingxinhome Awaiting requested review from qingxinhome qingxinhome is a code owner

Assignees
No one assigned
Labels
kind/feature size/XXL Denotes a PR that changes 2000+ lines
Projects
None yet
Milestone
2.1.0
Development

Successfully merging this pull request may close these issues.
6 participants
@ck89119 @zhangxu19830126 @XuPeng-SH @daviszhen @sukki37 @matrix-meow