2020年6月6日 星期六

Rclone將您的文件同步到雲存儲

https://rclone.org/

Rclone將您的文件同步到雲存儲


About rclone

Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.

Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support, and --dry-run protection. It can be used at the command line, in scripts or via its API.

Users have called rclone "The Swiss army knife of cloud storage" and "Technology indistinguishable from magic".

Rclone really looks after your data. It preserves timestamps and verifies your data at all times. Transfers over limited bandwidth; intermittent connections, or subject to quota can be restarted, from the last good file transferred. You can check the integrity of your files. Where possible, rclone employs server side transfers to minimise local bandwidth use and transfers from one provider to another without using your local disk.

Virtual backends wrap local and cloud file systems to apply encryption, caching, chunking and joining.

Rclone can mount any local, cloud or virtual filesystem as a disk on Windows, macOS, linux and FreeBSD, and also serve these over SFTP, HTTP, WebDAV, FTP and DLNA.

Rclone is mature, open source software originally inspired by rsync and written in Go. The friendly support community are familiar with varied use cases. Official Ubuntu, Debian, Fedora, Brew and Chocolatey repos. include rclone. For the latest version downloading from rclone.org is recommended.

Rclone is widely used on Linux, Windows and Mac. Third party developers have built innovative backup, restore, GUI and business process solutions using the rclone command line or API.

Let rclone do the heavy lifting of communicating with cloud storage.

What can rclone do for you

Rclone can help you:

  • Backup (and encrypt) files to cloud storage
  • Restore (and decrypt) files from cloud storage
  • Mirror cloud data to other cloud services or locally
  • Migrate data to cloud, or between cloud storage vendors
  • Mount multiple, encrypted, cached or diverse cloud storage as a disk
  • Analyse and account for data held on cloud storage using lsf, ljson, size, ncdu
  • Union file systems together to present multiple local and/or cloud file systems as one

Features

  • Transfers
    • MD5, SHA1 hashes are checked at all times for file integrity
    • Timestamps are preserved on files
    • Operations can be restarted at any time
    • Can be to and from network, eg two different cloud providers
    • Can use multi-threaded downloads to local disk
  • Copy new or changed files to cloud storage
  • Sync (one way) to make a directory identical
  • Move files to cloud storage deleting the local after verification
  • Check hashes and for missing/extra files
  • Mount your cloud storage as a network disk
  • Serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna
  • Experimental Web based GUI

Supported providers

(There are many other providers, built on standard protocols such as WebDAV or S3, that work out of the box.)

Links

Share and Enjoy

 Twitter
 Facebook
 Reddit

Links

 Rclone forum
 GitHub project
 Rclone slack
 Rclone Wiki
 Donate
 @njcw

張貼者： 於 沒有留言:

使用服務帳戶（自動）進行rclone複製/移動/同步

使用服務帳戶（自動）進行rclone複製/移動/同步



非常感謝rclone和folderclone。

  •  使用腳本創建服務帳戶
  •  將大量服務帳戶添加到rclone配置文件中
  •  將大量服務帳戶添加到您的組織的組中
  •  rclone複製/移動/同步時自動切換帳戶
  •  支持Windows系統


Step 1. Copy code to your VPS or local machine

Before everything, install python3. Because we use python as our programing language.

For Linux system: Install screen, git and latest rclone. If in Debian/Ubuntu, directly use this command

sudo apt-get install screen git && curl https://rclone.org/install.sh | sudo bash

After all dependency above are successfully installed, run this command

sudo git clone https://github.com/xyou365/AutoRclone && cd AutoRclone && sudo pip3 install -r requirements.txt

For Windows system: Directly download this project then install latest rclone. Then run this command (type in cmd command windows or PowerShell windows) in our project folder

pip3 install -r requirements.txt

Step 2. Generate service accounts What is service account How to use service account in rclone.

Let us create only the service accounts that we need. Warning: abuse of this feature is not the aim of autorclone and we do NOT recommend that you make a lot of projects, just one project and 100 sa allow you plenty of use, its also possible that overabuse might get your projects banned by google.

Enable the Drive API in Python Quickstart and save the file credentials.json into project directory.

If you do not have any project in your account then

  • create 1 projec
  • enable the required services
  • create 100 (1 project, each with 100) Service Accounts
  • and download their credentials into a folder named accounts
Note: 1 service account can copy around 750gb a day, 1 project makes 100 service accounts so thats 75tb a day, for most users this should easily suffice.

The command would look something like python3 gen_sa_accounts.py --quick-setup 1 replace "1" with the number of projects you want

If you have already N projects and want to create service accounts only in newly created projects,

to

  • create additional 1 project (project N+1 to project N+2)
  • enable the required services
  • create 100 (1 project, with 100) Service Accounts
  • and download their credentials into a folder named accounts

run

python3 gen_sa_accounts.py --quick-setup 1 --new-only

If you want to create some service accounts using existing projects (do not create more projects), run python3 gen_sa_accounts.py --quick-setup -1. Note that this will overwrite the existing service accounts.

After it is finished, there will be many json files in one folder named accounts.

Step 3. Add service accounts to Google Groups (Optional but recommended for hassle free long term use)

We use Google Groups to manager our service accounts considering the
Official limits to the members of Team Drive (Limit for individuals and groups directly added as members: 600).

For GSuite Admin

  1. Turn on the Directory API following official steps (save the generated json file to folder credentials).

  2. Create group for your organization in the Admin console. After create a group, you will have an address for examplesa@yourdomain.com.

  3. Run python3 add_to_google_group.py -g sa@yourdomain.com

For meaning of above flags, please run python3 add_to_google_group.py -h

For normal user

Create Google Group then add the service accounts as members by hand. Limit is 10 at a time, 100 a day but if you read our warning and notes above, you would have 1 project and hence easily in your range.

Step 4. Add service accounts or Google Groups into Team Drive

If you do not use Team Drive, just skip. Warning: It is NOT recommended to use service accounts to clone "to" folders that are not in teamdrives, SA work best for teamdrives.

If you have already created Google Groups (Step 2) to manager your service accounts, add the group address sa@yourdomain.com or sa@googlegroups.com to your source Team Drive (tdsrc) and destination Team Drive (tddst).

Otherwise, add service accounts directly into Team Drive.

Enable the Drive API in Python Quickstart and save the credentials.json into project root path if you have not done it in Step 2.

  • Add service accounts into your source Team Drive: python3 add_to_team_drive.py -d SharedTeamDriveSrcID
  • Add service accounts into your destination Team Drive: python3 add_to_team_drive.py -d SharedTeamDriveDstID

Step 5. Start your task

Let us copy hundreds of TB resource using service accounts. Note: Sarcasm, over abuse of this (regardless of what cloning script you use) may get you noticed by google, we recommend you dont be a glutton and clone what is important instead of downloading entire wikipedia.

For server side copy

  •  publicly shared folder to Team Drive
  •  Team Drive to Team Drive
  •  publicly shared folder to publicly shared folder (with write privilege)
  •  Team Drive to publicly shared folder
python3 rclone_sa_magic.py -s SourceID -d DestinationID -dp DestinationPathName -b 1 -e 600

  • For meaning of above flags, please run python3 rclone_sa_magic.py -h

  • Add --disable_list_r if rclone cannot read all contents of public shared folder.

  • Please make sure the Rclone can read your source and destination directory. Check it using rclone size:

  1. rclone --config rclone.conf size --disable ListR src001:

  2. rclone --config rclone.conf size --disable ListR dst001:

For local to Google Drive (needs some testing)

  •  local to Team Drive
  •  local to private folder
  •  private folder to any (think service accounts cannot do anything about private folder)
python3 rclone_sa_magic.py -sp YourLocalPath -d DestinationID -dp DestinationPathName -b 1 -e 600
  • Run command tail -f log_rclone.txt to see what happens in details (linux only).

Also let's talk about this project in Telegram Group AutoRclone

[Blog（中文）](Blog (中文) https://gsuitems.com/index.php/archives/13/) | Google Drive Group | Google Drive Channel

張貼者： 於 沒有留言:
標籤：
訂閱： 文章 (Atom)