Welcome to Skyplane!

Note

This project is under active development.

🔥 Blazing fast bulk data transfers between any cloud 🔥

Skyplane is a tool for blazingly fast bulk data transfers in the cloud. Skyplane manages parallelism, data partitioning, and network paths to optimize data transfers, and can also spin up VM instances to increase transfer throughput.

You can use Skyplane to transfer data: * Between buckets within a cloud provider * Between object stores across multiple cloud providers * (experimental) Between local storage and cloud object stores

Copy a large dataset in the cloud in a minute, not hours:

$ pip install skyplane
$ skyplane init
$ skyplane [sync/cp] [local/s3/gcp/azure]://mybucket/big_dataset [local/s3/gcp/azure]://mybucket2/

Contents

Overview

• Getting Started
  • Installation
  • Cloud Credentials
  • Transferring Data
• Benchmarks
  • Benchmark: Large file benchmark
  • Benchmark: Cost comparison w/ compression
  • Benchmark: ImageNet transfer on AWS
  • Benchmark: ImageNet transfer on GCP
  • Benchmark: Impact of Chunk Sizes on Transfer time and Throughput
  • Instructions to replicate benchmarks
• Configuration
  • Increasing performance of transfers via paralllelism
  • Reusing Gateways
  • Spot Instances to reduce instance costs
  • Configuring networking between gateways
• Architecture
  • On prem support
  • Transfer Integrity and Checksumming
  • Security
  • Firewalls
  • Large Objects
• Performance Stats Collection
  • What data is collected
  • How to disable it
  • How does it work
  • Requesting removal of collected data
  • Frequently Asked Questions (FAQ)
  • Contact us
• FAQ
  • I get InsufficientVCPUException when running a transfer
  • Does Skyplane support local transfers?
  • How does Skyplane map object keys from source to destination?
  • Troubleshooting MissingObjectException
  • Troubleshooting MissingBucketException

Developer documentation

• Building from source
  • Setting up a developer environment
  • Building and testing Skyplane
    • Building and pushing a Skyplane Docker image
      • First time setup: make sure ghcr image is “public”
    • Building the Skyplane client
    • Testing a transfer
  • Development Tips
• Contributing to Skyplane
  • Ways to Contribute
  • Code Contributions
    • Submitting pull requests
• Roadmap
  • Skyplane 0.1.0 release (current)
  • Skyplane 0.2.0 release
  • Upcoming
• Debugging Tools
  • Logs
    • Client Logs
    • Gateway Logs
    • Chunk API
  • Gateway ssh

Package documentation

• CLI documentation
  • skyplane
• Internal CLI documentation
  • skyplane
• Internal package documentation
  • skyplane.replicate.replicator_client module
  • skyplane.chunk module
  • skyplane.config module
  • skyplane.exceptions module

Community

• Slack