Building from source#

Prerequisite: install development dependencies#

For Skyplane development, we use a variety of tools to check for code quality and to build documentation. To install these tools, run the following command:

pip install -r requirements-dev.txt

Setting up a developer environment#

Skyplane is composed of the client (runs locally on a user’s laptop) and gateway VMs (runs in respective clouds). Normally, the gateways use a pre-built nightly Docker image containing the latest build of the Skyplane gateway code (public.ecr.aws/s6m1p0n8/skyplane:edge). However, if you modify the gateway source (under skyplane/gateway), you will need to rebuild the gateway Docker image and push it to a container registry.

Ensure you have authenticated your Github account with Docker:

  • Install docker

curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
echo <PERSONAL_ACCESS_TOKEN> | sudo docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin

Building and testing Skyplane#

Building and pushing a Skyplane Docker image#

To package the code into a Docker image and push it to a container registry on your account, run the following command (substitute YOUR_GITHUB_USERNAME_HERE for your Github username):

export SKYPLANE_DOCKER_IMAGE=$(bash scripts/pack_docker.sh <YOUR_GITHUB_USERNAME_HERE>)

This will build the Skyplane Docker image for the gateway and push it ghcr.io under your user account. When running a Skyplane transfer, any provisioned gateways will pull this image from the SKYPLANE_DOCKER_IMAGE environment variable to ensure a reproducible environment.

First time setup: make sure ghcr image is “public”#

By default, new packages on ghcr are private. To make the package public so gateways can download the image, convert the package to public (you only need to do this once):

Package settings

  • Click the “Make public” button.

Make public

Building the Skyplane client#

We use Poetry to manage package dependencies during development. For convenience, we provide a Poetry wrapper via setup.py. To build the client, install the Skyplane package in development mode. The package points to your current checked-out version of the code, and any edits to the Skyplane client will immediately apply to the skyplane CLI command.

pip install -e ".[aws,azure,gcp]"

Testing a transfer#

We can run the skyplane CLI to test a transfer. The CLI will read your SKYPLANE_DOCKER_IMAGE environment variable and use that Docker image when launching gateways.

skyplane init
skyplane cp s3://... s3://...

Development Tips#

If testing transfers repeatedly, we recommend using the --reuse-gateways to reduce setup time.

Skyplane has debugging tools to ssh into gateway instances, view detailed transfer logs, and query chunk states during transfers. See Debugging Tools for more.