-
Notifications
You must be signed in to change notification settings - Fork 20.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/geth:Implement freezer truncation as a subcommand(fixes #31135) #31351
cmd/geth:Implement freezer truncation as a subcommand(fixes #31135) #31351
Conversation
|
||
for low <= high { | ||
mid := (low + high) / 2 | ||
header := rawdb.ReadHeader(db, rawdb.ReadCanonicalHash(db, mid), mid) | ||
if header == nil { | ||
return fmt.Errorf("header %d not found", mid) | ||
} | ||
|
||
if header.Difficulty.Sign() == 0 { | ||
// This is a post-merge block, look earlier | ||
high = mid - 1 | ||
mergeBlock = mid | ||
found = true | ||
} else { | ||
// This is a pre-merge block, look later | ||
low = mid + 1 | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could do something like this here
for low <= high { | |
mid := (low + high) / 2 | |
header := rawdb.ReadHeader(db, rawdb.ReadCanonicalHash(db, mid), mid) | |
if header == nil { | |
return fmt.Errorf("header %d not found", mid) | |
} | |
if header.Difficulty.Sign() == 0 { | |
// This is a post-merge block, look earlier | |
high = mid - 1 | |
mergeBlock = mid | |
found = true | |
} else { | |
// This is a pre-merge block, look later | |
low = mid + 1 | |
} | |
} | |
sort.Search(*headNumber, func(index int) bool { | |
header := rawdb.ReadHeader(db, rawdb.ReadCanonicalHash(db, index), index) | |
if header == nil { | |
panic(fmt.Sprintf("header %d not found", index)) | |
} | |
return header.Difficulty.Sign() == 0 | |
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't we just assume the merge block is hardcoded to ethereum mainnet value? , and assume that we are running on a pre-merged chain?
|
||
// First, read all headers up to the merge block | ||
log.Info("Reading headers to preserve them", "count", mergeBlock) | ||
headers := make([][]byte, mergeBlock) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So this will be 15 Gbs of memory on mainnet
} | ||
|
||
// Re-insert the headers and hashes | ||
log.Info("Re-inserting headers", "count", len(headers)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this will not work. Yes you can re-insert the headers, but the freezer has the assumption that all of its tables have the same length. So upon next boot it will truncate the headers to match that of blocks, receipts etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems we have two options:
- relax this assumption in the freezer
- Move the headers and hashes to a fresh freezer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thankyou for pointing out the issues.
…nfigurable batch size
I've updated the PR with optimizations that address all the concerns raised: 1. Memory Usage Issue
2. Freezer Table Length Assumption
3. Hardcoded Merge Block Values
Additional Improvements
Please let me know your thoughts on it. |
…instead of progress reporter
Hii @s1na , I fixed the issues you have mentioned and made some modifications to the codebase to make it more readable and maintainable. Can you please review it and share your feedback. |
We decided to work on our own version of this. Thank you for your contribution! |
Hii, I have implemented a new subcommand
geth db truncate-freezer
that truncates the freezer at the merge block, keeping headers but removing bodies. This addresses issue #31135.These are the Implementation Details
truncate-freezer
to thedb
commandThis implementation follows the same pattern as the existing
prune-history
command and uses the same underlyingtruncateAncientStore
method.Fixes #31135