What is rclone?

rclone is a command-line program to manage files on cloud storage. It’s like rsync for cloud storage, supporting over 40 cloud storage products including Google Drive, Amazon S3, Dropbox, and more.

Installation

macOS

1
brew install rclone

Linux

1
curl https://rclone.org/install.sh | sudo bash

Windows

Download from rclone.org/downloads or use:

1
choco install rclone

Verify Installation

1
rclone version

Configuration

Configure a New Remote (Google Drive)

  1. Start the configuration wizard:
1
rclone config

  1. Follow the prompts:

    • Type n for new remote
    • Enter a name (e.g., gdrive)
    • Choose storage type (e.g., drive for Google Drive)
    • Leave Client ID and Secret blank (unless you have your own)
    • Choose scope (usually 1 for full access)
    • Leave root folder ID blank
    • Leave service account file blank
    • Choose n for advanced config
    • Choose y to auto-config (opens browser)
    • Authenticate in browser
    • Choose n for team drive (or y if needed)
    • Type y to confirm

  2. Test the connection:

1
rclone listremotes

List All Configured Remotes

1
rclone listremotes

Show Config Location

1
rclone config file

Configure for Shared/Team Drives

If you need to access Google Shared Drives (Team Drives):

  1. Run rclone config again
  2. Select your existing remote to edit it
  3. Choose advanced config
  4. Set team_drive to the ID of your shared drive
  5. Or configure a new remote specifically for a team drive

Alternatively, you can access shared files directly using the --drive-shared-with-me flag (see below).

Basic Commands

List Files and Directories

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# List directories in top level
rclone lsd gdrive:

# List all files and directories
rclone ls gdrive:

# List with more details (size, modification time)
rclone lsl gdrive:

# List only file names
rclone lsf gdrive:

# List files in a specific folder
rclone ls gdrive:folder_name/

Downloading Files

Download a Single File

1
rclone copy gdrive:file.txt /local/path/

Download a Folder

1
2
3
4
5
# Copy entire folder
rclone copy gdrive:folder_name /local/path/

# Sync folder (deletes files in destination that don't exist in source)
rclone sync gdrive:folder_name /local/path/

Download with Progress

1
rclone copy gdrive:folder_name /local/path/ --progress

Download with Bandwidth Limit

1
2
# Limit to 10 MB/s
rclone copy gdrive:folder_name /local/path/ --bwlimit 10M

Download Only New/Modified Files

1
rclone copy gdrive:folder_name /local/path/ --update

Accessing Shared Drives and Shared Files

List Shared Drives (Team Drives)

1
2
# List available shared drives
rclone backend drives gdrive:

Access Files Shared With Me

Use the --drive-shared-with-me flag to access files that others have shared with you:

1
2
3
4
5
6
7
8
# List files shared with you
rclone lsf gdrive: --drive-shared-with-me

# Download from shared location
rclone copy "gdrive:Shared Folder/file.zip" /local/path/ --drive-shared-with-me --progress

# Download from shared drive with specific path
rclone copy "gdrive:Max Project Data/eda_data/PBench/cosmos1_14b_v1.zip" . --drive-shared-with-me --progress

Configure a Specific Shared Drive

To work with a specific shared drive permanently:

  1. Get the shared drive ID:
1
rclone backend drives gdrive:
  1. Create a new remote for that shared drive:
1
rclone config create shared_drive drive team_drive=DRIVE_ID
  1. Use it normally:
1
2
rclone ls shared_drive:
rclone copy shared_drive:folder/file.txt /local/path/

Uploading Files

Upload a Single File

1
rclone copy /local/path/file.txt gdrive:

Upload a Folder

1
2
3
4
5
# Copy entire folder
rclone copy /local/path/folder_name gdrive:

# Upload to specific remote folder
rclone copy /local/path/folder_name gdrive:remote_folder/

Upload with Progress

1
rclone copy /local/path/folder_name gdrive: --progress

Upload Only New/Modified Files

1
rclone copy /local/path/folder_name gdrive: --update

Upload and Skip Existing Files

1
rclone copy /local/path/folder_name gdrive: --ignore-existing

Upload to Shared Drive

When uploading to shared drives, you can use the same --drive-shared-with-me flag:

1
2
# Upload to shared drive location
rclone copy /local/path/file.tar.gz "gdrive:Shared Folder/" --drive-shared-with-me --progress

⚠️ Important: Avoiding Infinite Loop on Shared Drive Uploads

When uploading to shared drives, rclone may sometimes enter an infinite loop. To prevent this, limit transfers and checkers to 1:

1
2
3
4
5
6
7
# Safe upload to shared drive (prevents infinite loop)
rclone copy eval_data.tar.gz "gdrive:Max Project Data/" \
    --drive-shared-with-me \
    -P \
    --stats 10s \
    --transfers 1 \
    --checkers 1

The key flags to prevent infinite loops:

  • --transfers 1 - Only 1 file transfer at a time
  • --checkers 1 - Only 1 file checker at a time
  • --stats 10s - Show stats every 10 seconds for monitoring
  • -P - Short for --progress

Advanced Operations

Sync vs Copy

Copy: Copies files from source to destination, never deletes anything

1
rclone copy source:path dest:path

Sync: Makes destination identical to source (deletes extra files)

1
rclone sync source:path dest:path

Move Files

1
2
3
4
5
# Move file (copy then delete from source)
rclone move gdrive:old_location/ gdrive:new_location/

# Move from local to cloud
rclone move /local/path/ gdrive:

Delete Files

1
2
3
4
5
6
7
8
# Delete a file
rclone delete gdrive:file.txt

# Delete a folder
rclone purge gdrive:folder_name/

# Delete empty directories
rclone rmdirs gdrive:folder_name/

Check and Compare

1
2
3
4
5
6
7
8
# Check files in source and destination match
rclone check source:path dest:path

# Get size of remote folder
rclone size gdrive:folder_name/

# Show differences between source and destination
rclone check source:path dest:path --one-way

Mount Remote as Local Drive

1
2
3
4
5
6
# Mount Google Drive to local folder
rclone mount gdrive: /local/mount/point &

# Unmount
fusermount -u /local/mount/point  # Linux
umount /local/mount/point         # macOS

Useful Flags and Options

Common Flags

1
2
3
4
5
6
7
8
9
10
11
12
13
14
--progress              # Show progress during transfer
--dry-run              # Test run without making changes
--verbose (-v)         # Verbose output
--transfers N          # Number of parallel transfers (default 4)
--checkers N           # Number of checkers (default 8)
--bwlimit BANDWIDTH    # Bandwidth limit (e.g., 10M)
--update               # Skip files that are newer on destination
--ignore-existing      # Skip files that already exist
--exclude PATTERN      # Exclude files matching pattern
--include PATTERN      # Include only files matching pattern
--log-file FILE        # Log to file
--max-size SIZE        # Don't transfer files bigger than SIZE
--min-size SIZE        # Don't transfer files smaller than SIZE
--drive-shared-with-me # Access files shared with you on Google Drive

Example with Multiple Flags

1
2
3
4
5
6
7
8
rclone copy /local/path/ gdrive:backup/ \
  --progress \
  --transfers 8 \
  --checkers 16 \
  --bwlimit 50M \
  --exclude "*.tmp" \
  --exclude ".DS_Store" \
  --log-file rclone.log

Filtering Examples

Exclude Patterns

1
2
3
4
5
6
7
8
9
10
11
12
# Exclude specific file types
rclone copy /local/path/ gdrive: --exclude "*.tmp"
rclone copy /local/path/ gdrive: --exclude "*.log"

# Exclude multiple patterns
rclone copy /local/path/ gdrive: \
  --exclude "*.tmp" \
  --exclude "*.log" \
  --exclude ".DS_Store"

# Exclude directories
rclone copy /local/path/ gdrive: --exclude "node_modules/**"

Include Only Specific Files

1
2
# Include only specific file types
rclone copy /local/path/ gdrive: --include "*.jpg" --include "*.png"

Using Filter Files

Create a file filter-rules.txt:

1
2
3
4
5
6
- *.tmp
- *.log
- node_modules/**
+ *.jpg
+ *.png
+ **

Use it:

1
rclone copy /local/path/ gdrive: --filter-from filter-rules.txt

Performance Optimization

Faster Transfers

1
2
3
4
5
# Increase parallel transfers and checkers
rclone copy source:path dest:path \
  --transfers 32 \
  --checkers 64 \
  --drive-chunk-size 128M

Resume Failed Transfers

1
2
# rclone automatically resumes, but you can also use:
rclone copy source:path dest:path --max-backlog 200000

Common Use Cases

Backup Local Files to Cloud

1
2
3
4
# Daily backup with timestamp
rclone copy /important/data/ gdrive:backups/$(date +%Y-%m-%d)/ \
  --progress \
  --log-file backup.log

Sync Two Cloud Services

1
2
# Sync from Google Drive to Dropbox
rclone sync gdrive:folder dropbox:folder --progress

Download Specific File Types

1
2
3
4
5
6
# Download only images
rclone copy gdrive:photos/ /local/photos/ \
  --include "*.jpg" \
  --include "*.png" \
  --include "*.jpeg" \
  --progress

Upload with Exclusions

1
2
3
4
5
6
7
# Upload project folder excluding build artifacts
rclone copy /local/project/ gdrive:projects/myproject/ \
  --exclude "node_modules/**" \
  --exclude "build/**" \
  --exclude ".git/**" \
  --exclude "*.log" \
  --progress

Download from Shared Drive

1
2
3
4
5
6
7
8
9
10
# Download a specific file from a shared location
rclone copy "gdrive:Project Folder/data/model.zip" ./downloads/ \
  --drive-shared-with-me \
  --progress

# Download entire shared folder
rclone copy "gdrive:Team Resources/" ./team_resources/ \
  --drive-shared-with-me \
  --progress \
  --transfers 8

Troubleshooting

Reset Configuration

1
2
rclone config delete remote_name
rclone config

Check Connection

1
rclone about gdrive:

Verbose Output for Debugging

1
rclone copy source:path dest:path -vv

Force Reconfiguration

1
rclone config reconnect gdrive:

Infinite Loop When Uploading to Shared Drive

Problem: rclone gets stuck in an infinite loop when uploading to shared drives.

Solution: Limit transfers and checkers to 1:

1
2
3
4
5
6
rclone copy your_file.tar.gz "gdrive:Shared Folder/" \
    --drive-shared-with-me \
    --transfers 1 \
    --checkers 1 \
    -P \
    --stats 10s

This forces sequential processing and prevents the loop issue.

Quick Reference

Command Description
rclone config Configure remotes
rclone listremotes List configured remotes
rclone ls remote: List files
rclone copy source: dest: Copy files
rclone sync source: dest: Sync (one-way)
rclone move source: dest: Move files
rclone delete remote:path Delete files
rclone purge remote:path Delete folder
rclone check source: dest: Check if files match
rclone mount remote: /path Mount as drive
rclone size remote:path Show size

Tips and Best Practices

  1. Always test with --dry-run before syncing or deleting

    1
    rclone sync source: dest: --dry-run

  2. Use copy instead of sync unless you specifically want to delete files

  3. Set up logging for important operations

    1
    --log-file /path/to/log.txt

  4. Use bandwidth limits on slow connections

    1
    --bwlimit 10M

  5. Increase transfers for faster operations on fast connections

    1
    --transfers 16 --checkers 32

  6. For shared drive uploads, use single transfer/checker to avoid infinite loops

    1
    --transfers 1 --checkers 1

  7. Create aliases for frequently used commands in your .bashrc or .zshrc:

    1
    2
    alias rclone-upload='rclone copy --progress --transfers 16'
    alias rclone-download='rclone copy --progress --transfers 16'

Resources