Browse All Tools
Platform/OS
Linux, macOS, Windows
Category
Data Science & Analytics

PyTorch

PyTorch (created by Meta AI) is the leading deep-learning framework almost all major AI breakthroughs today are built using it. It integrates natively with S3-compatible storage for massive datasets, model checkpoints, and distributed training data lakes. It lets you create apps that recognize photos, understand speech, or predict future sales without needing a PhD in math. It uses S3-compatible object storage as a simple digital warehouse to keep all the training data, finished models, and results so they are safe and easy to grab from anywhere. This storage approach means your team can share huge datasets and trained models without filling up local hard drives or dealing with complicated file servers. Overall it makes building and improving intelligent software much faster, cheaper, and more collaborative for real business needs.

Connection / Setup

  1. Create an account and log in to DiskSpace
  2. Click I'm ready to set up my storage
  3. Enter Bucket Name following the instructions
  4. Click Create Bucket and wait for it to be generated
  5. Find your Endpoint URL, Access Key, and Secret Key in the S3 Client Configuration section for use below

Connect PyTorch to DiskSpace.sh (S3-compatible Storage)

This guide explains how to configure PyTorch to use DiskSpace.sh as an S3-compatible object storage backend. DiskSpace.sh is fully S3 API compatible and requires path-style addressing.

Connection Steps
  1. Install the required packages:
    pip install boto3 s3fs
  2. Gather your DiskSpace.sh credentials:
    • Endpoint URL (e.g. https://s3.diskspace.sh)
    • Access Key ID
    • Secret Access Key
  3. Configure the storage backend using one of the methods below.
Credential Fields
Region: Can be left empty or set to us-east-1
Path Style Requests (Required)

DiskSpace.sh requires path-style requests. The correct URL format is:

https://s3.diskspace.sh/bucket-name/key

Virtual-hosted style (bucket.s3.diskspace.sh) is not supported.

Example Configuration
import torch
from torch.utils.data import Dataset
import s3fs
import boto3

# Configure S3 filesystem
fs = s3fs.S3FileSystem(
    key="YOUR_ACCESS_KEY_ID",
    secret="YOUR_SECRET_ACCESS_KEY",
    endpoint_url="https://s3.diskspace.sh",
    use_ssl=True,
    client_kwargs={
        "region_name": "us-east-1"
    }
)

# Example: Load a model from DiskSpace.sh
s3_path = "s3://my-bucket/models/resnet50.pth"

with fs.open(s3_path, "rb") as f:
    model = torch.load(f, weights_only=False)

# Or using boto3 session for torch.hub / custom datasets
session = boto3.Session(
    aws_access_key_id="YOUR_ACCESS_KEY_ID",
    aws_secret_access_key="YOUR_SECRET_ACCESS_KEY",
    region_name="us-east-1"
)

s3 = session.client(
    "s3",
    endpoint_url="https://s3.diskspace.sh",
    use_ssl=True
)
Troubleshooting
  • SignatureDoesNotMatch: Ensure you are using path-style requests and the correct endpoint URL.
  • Connection Error: Verify your Endpoint URL includes https:// and that your firewall allows outbound connections.
  • Access Denied: Double-check your Access Key ID and Secret Access Key. Ensure the bucket policy allows your credentials.
  • Region Issues: Try setting region to us-east-1 if you encounter region-related errors.
Tip: You can also set these values as environment variables:
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL, AWS_S3_ENDPOINT
DiskSpace.sh S3 Integration for PyTorch • Path-style requests required