Backup fails with "Content-MD5 OR x-amz-checksum- HTTP header is required ..." when S3 object lock is enabled

[kevinmingtarja]

Describe the bug

S3 Object Lock can help prevent S3 objects from being deleted or overwritten for a fixed amount of time or indefinitely. It is typically enabled to help meet regulatory requirements that require WORM storage, or to add another layer of protection against object changes or deletion.

If a backup is initiated to an S3 bucket with Object Lock enabled, it fails with the following error:

...
I0314 03:32:08.564337      16 log.go:33] Sent range 23 for iteration: [, 0000000000000000000009506167652e6e616d650000000000000166e9fffffffffff4d15c) of size: 775 KiB
I0314 03:32:08.888820      16 log.go:33] Dgraph.Backup Sent data of size 115 KiB
I0314 03:32:08.889177      16 backup_ee.go:553] Backup group 1 version: 770046
I0314 03:32:08.889300      16 backup_handler.go:341] Backup waiting for upload to complete.
I0314 03:32:08.994170      16 backup_handler.go:357] Backup sent 0 bytes. Time elapsed: 0s
E0314 03:32:08.994217      16 backup_handler.go:362] Backup: Closing RW pipe due to error: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Part requests with Object Lock parameters

This is because the Content-MD5 or x-amz-sdk-checksum- header is required (docs) for any request to upload an object with a retention period configured using Object Lock. Theses headers are a way to verify the integrity of the object during upload.

To Reproduce

Steps to reproduce the behavior:

1. Spin up a single Alpha and Zero
2. Initiate a backup, as shown here

Expected behavior

The backup should succeed.

Environment

• OS: Ubuntu
• Version: v24.1.0

Fix

I think we simply need to set SendContentMd5 to true in minio.PutObjectOptions, in our s3Writer.upload method in backup_handler.go.

That should set the Content-MD5 header on the request. https://github.com/minio/minio-go/blob/01fe40d263e94a91edc2333697cd56fe04e7bd52/api.go#L907

[linear]

DGR-868 Backup fails with "Content-MD5 OR x-amz-checksum- HTTP header is required ..." when S3 object lock is enabled