Auto Rclone - rc: connection failed

[dropitlikecross]

Hello @shirooo39

This is for auto rclone, everything runs well until I try to copy from source to destination and I get the below error.
Any idea what it could be? I'm putting 1 shared drive's ID in the source and another's in the destination. Both have been setup in the config.

/content/tools/AutoRclone
rclone is detected: /usr/bin/rclone
generating rclone config file.
rclone config file generated.
Start: 04:32:22
rclone --config ./rclone.conf copy --drive-server-side-across-configs --rc --rc-addr="localhost:5572" -vv --ignore-existing --tpslimit 3 --transfers 3 --drive-chunk-size 32M --drive-acknowledge-abuse --log-file=log_rclone.txt "src001:" "dst001:" &
>> Let us go dst001: 04:32:22
2021/04/17 04:32:32 Failed to rc: connection failed: Post "http://localhost:5572/core/pid": dial tcp 127.0.0.1:5572: connect: connection refused
2021/04/17 04:32:32 Failed to rc: connection failed: Post "http://localhost:5572/core/stats": dial tcp 127.0.0.1:5572: connect: connection refused
2021/04/17 04:32:32 Failed to rc: connection failed: Post "http://localhost:5572/core/stats": dial tcp 127.0.0.1:5572: connect: connection refused
2021/04/17 04:32:32 Failed to rc: connection failed: Post "http://localhost:5572/core/stats": dial tcp 127.0.0.1:5572: connect: connection refused
No rclone task detected (possibly done for this account). (1/3)
rclone --config ./rclone.conf copy --drive-server-side-across-configs --rc --rc-addr="localhost:5572" -vv --ignore-existing --tpslimit 3 --transfers 3 --drive-chunk-size 32M --drive-acknowledge-abuse --log-file=log_rclone.txt "src002:" "dst002:" &
>> Let us go dst002: 04:32:32
2021/04/17 04:32:42 Failed to rc: connection failed: Post "http://localhost:5572/core/pid": dial tcp 127.0.0.1:5572: connect: connection refused
2021/04/17 04:32:42 Failed to rc: connection failed: Post "http://localhost:5572/core/stats": dial tcp 127.0.0.1:5572: connect: connection refused
2021/04/17 04:32:42 Failed to rc: connection failed: Post "http://localhost:5572/core/stats": dial tcp 127.0.0.1:5572: connect: connection refused
2021/04/17 04:32:42 Failed to rc: connection failed: Post "http://localhost:5572/core/stats": dial tcp 127.0.0.1:5572: connect: connection refused
No rclone task detected (possibly done for this account). (2/3)
rclone --config ./rclone.conf copy --drive-server-side-across-configs --rc --rc-addr="localhost:5572" -vv --ignore-existing --tpslimit 3 --transfers 3 --drive-chunk-size 32M --drive-acknowledge-abuse --log-file=log_rclone.txt "src003:" "dst003:" &
>> Let us go dst003: 04:32:42
2021/04/17 04:32:52 Failed to rc: connection failed: Post "http://localhost:5572/core/pid": dial tcp 127.0.0.1:5572: connect: connection refused
2021/04/17 04:32:52 Failed to rc: connection failed: Post "http://localhost:5572/core/stats": dial tcp 127.0.0.1:5572: connect: connection refused
2021/04/17 04:32:52 Failed to rc: connection failed: Post "http://localhost:5572/core/stats": dial tcp 127.0.0.1:5572: connect: connection refused
2021/04/17 04:32:52 Failed to rc: connection failed: Post "http://localhost:5572/core/stats": dial tcp 127.0.0.1:5572: connect: connection refused
No rclone task detected (possibly done for this account). (3/3)
All done (3/3).
Elapsed Time: 00:00:30.62

[shirooo39]

To be honest with you, AutoRclone is still yet to be tested.
I've been trying to replicate the method manually for almost a week already by now but still no result.

My target is from G-Suite to a Team Drive, though... so, it's more difficult, compared to Team Drive <-> Team Drive.
I was able to perform 2 x 740GB but then I kept getting limit reached. Been thinking for days, where's the issue...

so, yeah, I'll try to keep updating the Notebook once I'm able to use AutoRclone properly and understand how it works (because at its state, it's quite confusing to use. I'll try to simplify it so everyone can use it easily Hopefully, it's easy to do...).

[dropitlikecross]

Yeah I'm messing with it too for the first time. It's really clever but a bit confusing if you're like me and unfamiliar with gcp and want it running on colab. I've checked other books but yours is definitely the cleanest.

Also any chance I can get your contact details @shirooo39? A lot of your saved repos are in my stars. Would love to hit you up about ideas sometimes.

[shirooo39]

so, upon further testing, I think I was able to replicate the method manually simply by switching SA on the Destination drive (by simply changing which json file to use for the next operation), was able to perform almost 2TB server-side copy but then I got hit by another limiation. Asked r/DataHoarder and someone replied to me that I got hit by multiple limits like Drive usage and API limit.

There's also this other 2TB limit which you can read more in here: https://www.reddit.com/r/DataHoarder/comments/ir2tst/google_drive_now_counts_copying_as_part_of_the/ and it looks like this is the limit I hit and not the 750GB one.

GDrive 2021 got so many undocumented limitations, jeez...

[shirooo39]

I found this on a forum

"Are you trying to copy from a 3rd party TD as source? If yes, that’s the problem. You need to add your sa to the source TD also."

Make sure you have added your Service Accounts into both Drives (Source and Destination).
Adding them into a Google Group is way prefered as you'd only need to just add the Group's email and that's way easier.

The Google Group has a limiation tho...
you can only do 10 invites per-click and then you have to do another 10, with a total of 100 invites per-day (doesn't matter which account you use to invite the SAs, the 100 invites per-day limitation is being taken by the Group).

[dropitlikecross]

I saw that same one, I'm about to test it out. I'm sure there's a work around.

[dropitlikecross]

Just for other usesrs:
Service accounts are added to both source and destination.
Both have manager access.

The google groups way let me copy 6.6TB. It does take a while to set up though. Also I've had better success using option 2 rather than the rclone area copy.

At what speeds are you able to copy at?

[shirooo39]

The google groups way let me copy 6.6TB. It does take a while to set up though. Also I've had better success using option 2 rather than the rclone area copy.

as of now, you can't do copy for more than 2 TB. this limitation is applied to the Drive itself, rather than the user doing the operation (I'm still not sure if it applied to the Source drive, or the Destination drive).

At what speeds are you able to copy at?

Since it's a server-side operation, I once saw it maxed out at around 1 GB/s.