Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support external blob #1672

Open
wants to merge 32 commits into
base: master
Choose a base branch
from
Open

Conversation

BraveY
Copy link

@BraveY BraveY commented Feb 28, 2025

Relevant Issue (if applicable)

If there are Issues related to this PullRequest, please list it.

Details

Please describe the details of PullRequest.

Types of changes

What types of changes does your PullRequest introduce? Put an x in all the boxes that apply:

  • Bug fix (non-breaking change which fixes an issue)
  • [x ] New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation Update (if none of the other choices apply)

Checklist

Go over all the following points, and put an x in all the boxes that apply.

  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.

@BraveY BraveY requested a review from a team as a code owner February 28, 2025 04:48
@BraveY BraveY requested review from bergwolf, imeoer and Desiki-high and removed request for a team February 28, 2025 04:48
Copy link

codecov bot commented Feb 28, 2025

Codecov Report

Attention: Patch coverage is 56.87402% with 825 lines in your changes missing coverage. Please review.

Project coverage is 55.16%. Comparing base (7e51479) to head (9592ed1).

Files with missing lines Patch % Lines
contrib/nydusify/pkg/converter/converter.go 71.31% 76 Missing and 29 partials ⚠️
builder/src/directory.rs 0.00% 88 Missing ⚠️
builder/src/core/node.rs 23.76% 72 Missing and 5 partials ⚠️
contrib/nydusify/pkg/external/modctl/remote.go 58.77% 36 Missing and 18 partials ⚠️
contrib/nydusify/pkg/external/modctl/modctl.go 80.08% 36 Missing and 9 partials ⚠️
...trib/nydusify/pkg/snapshotter/external/external.go 53.60% 30 Missing and 15 partials ⚠️
storage/src/backend/external/meta.rs 0.00% 40 Missing ⚠️
builder/src/core/context.rs 51.28% 31 Missing and 7 partials ⚠️
src/bin/nydus-image/main.rs 0.00% 37 Missing ⚠️
storage/src/backend/external/mod.rs 10.00% 36 Missing ⚠️
... and 24 more

❗ There is a different number of reports uploaded between BASE (7e51479) and HEAD (9592ed1). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (7e51479) HEAD (9592ed1)
3 1
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1672      +/-   ##
==========================================
- Coverage   60.42%   55.16%   -5.26%     
==========================================
  Files         146      199      +53     
  Lines       48943    55359    +6416     
  Branches    46380    46978     +598     
==========================================
+ Hits        29573    30541     +968     
- Misses      17906    23242    +5336     
- Partials     1464     1576     +112     
Files with missing lines Coverage Δ
api/src/config.rs 88.78% <100.00%> (+0.03%) ⬆️
builder/src/core/bootstrap.rs 77.63% <100.00%> (+0.29%) ⬆️
builder/src/core/chunk_dict.rs 70.18% <ø> (ø)
builder/src/core/tree.rs 80.86% <100.00%> (-0.02%) ⬇️
builder/src/merge.rs 70.73% <100.00%> (ø)
builder/src/stargz.rs 75.00% <100.00%> (+0.14%) ⬆️
builder/src/tarball.rs 63.26% <100.00%> (+0.57%) ⬆️
contrib/nydusify/pkg/cache/cache.go 39.90% <100.00%> (ø)
...trib/nydusify/pkg/chunkdict/generator/generator.go 0.00% <ø> (ø)
contrib/nydusify/pkg/committer/commiter.go 0.00% <ø> (ø)
... and 44 more

... and 37 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@BraveY BraveY force-pushed the external-blob branch 5 times, most recently from 9ca650d to 24a62f1 Compare March 1, 2025 13:02
@imeoer imeoer changed the title [WIP]External blob Support external blob Mar 21, 2025
imeoer and others added 17 commits March 22, 2025 16:23
- copy snapshotter code to internal package.
- add modctl handler to produce nydus attribute pattern.
- support push manifest to registry in convert

Signed-off-by: Yang Kaiyong <[email protected]>
Ensure that the parent directories of external files are added to the external tree.
This change preserves the directory structure and marks all necessary parent directories as external.

Signed-off-by: Yang Kaiyong <[email protected]>
- Added detailed comments to the chunk implementation in modctl.go.
- Formatted the code to improve readability and consistency.

Signed-off-by: Yang Kaiyong <[email protected]>
…nd bootstrap

- Eliminate the merge step and directly integrate backend.json and bootstrap into the final bootstrap during the packing process.

- When assembling the manifest file, include the original model image's config information and layers.

Signed-off-by: Yang Kaiyong <[email protected]>
Reformat the codebase to align with Clippy's linting suggestions.

Signed-off-by: Yang Kaiyong <[email protected]>
The chunk size of model image can be configured by `--chunk-size` flag
in convert command.

Signed-off-by: Yang Kaiyong <[email protected]>
Yang Kaiyong added 7 commits March 22, 2025 16:23
Introduce the ability to use plain HTTP during the nydusify convert command when pushing blobs.
This feature is controlled via a dedicated flag, allowing users to opt for plain HTTP communication when needed.

Signed-off-by: Yang Kaiyong <[email protected]>
Signed-off-by: Yang Kaiyong <[email protected]>
Set file size before calculating block count.

Signed-off-by: Yang Kaiyong <[email protected]>
This update enables the `nydusify mount` subcommand to support mounting external large model images.

Signed-off-by: Yang Kaiyong <[email protected]>
- Add a new `goreleaser` job in GitHub Actions to handle RPM and DEB packaging.
- Introduce a `goreleaser.sh` script to dynamically generate `.goreleaser.yml` configuration.

This change enables automated RPM/DEB package generation, improving release efficiency and consistency.

Signed-off-by: Yang Kaiyong <[email protected]>
@BraveY BraveY closed this Mar 22, 2025
@BraveY BraveY reopened this Mar 22, 2025
@BraveY BraveY closed this Mar 22, 2025
@BraveY BraveY reopened this Mar 22, 2025
@BraveY BraveY requested a review from a team as a code owner March 22, 2025 08:40
@BraveY BraveY requested review from changweige and removed request for a team March 22, 2025 08:40
@BraveY
Copy link
Author

BraveY commented Mar 22, 2025

@codecov retest

Yang Kaiyong added 2 commits March 22, 2025 20:47
Add unit test in external modctl package.

Signed-off-by: Yang Kaiyong <[email protected]>
Remove snapshotter convert package to third party.

Just a workaround to avoid copying nydus-snapshotter code directly.

Signed-off-by: Yang Kaiyong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants