Browse All Tools
Platform/OS
Linux, macOS, Windows
Category
Data Science & Analytics

Apache Spark / Hadoop

Apache Spark is a data processing engine for big data sets. Apache Hadoop is a software utility that allows users to manage big data sets (from gigabytes to petabytes) by enabling a network of computers (or “nodes”) to solve vast and intricate data problems. They are powerful software tools that let companies process huge amounts of data that would overwhelm regular computers. They work like a giant team of helpers that split up a massive pile of information, analyze it in parallel, and quickly find useful patterns or answers. They use S3-compatible object storage as a giant, cheap digital warehouse to keep all the raw data safe and easily accessible without needing expensive specialized hardware. This storage approach means Spark and Hadoop can read and write data directly from S3 buckets while automatically scaling up or down based on the job size. The combination delivers fast insights for real-world tasks like recommending movies, detecting fraud, or understanding customer behavior without breaking the bank on infrastructure.

Connection / Setup

  1. Create an account and log in to DiskSpace
  2. Click I'm ready to set up my storage
  3. Enter Bucket Name following the instructions
  4. Click Create Bucket and wait for it to be generated
  5. Find your Endpoint URL, Access Key, and Secret Key in the S3 Client Configuration section for use below

Connecting Apache Spark / Hadoop to DiskSpace.sh

DiskSpace.sh is a fully S3 API-compatible object storage service. Follow the steps below to configure Apache Spark or Hadoop to use DiskSpace.sh with path-style addressing.

Connection Steps
  1. Obtain your credentials from the DiskSpace.sh dashboard: Endpoint URL, Access Key ID, and Secret Access Key.
  2. Configure Hadoop/Spark core-site.xml with the properties shown below.
  3. Use path-style URLs when referencing buckets (e.g. s3a://bucket-name/path/to/data).
  4. Set the region to us-east-1 or leave it empty.
Required Configuration Properties
Path Style Requests (Required)
DiskSpace.sh requires path-style requests. Virtual-hosted style URLs are not supported.
Example Configuration (core-site.xml)
<property>
  <name>fs.s3a.endpoint</name>
  <value>s3.diskspace.sh</value>
</property>
<property>
  <name>fs.s3a.access.key</name>
  <value>YOUR_ACCESS_KEY_ID</value>
</property>
<property>
  <name>fs.s3a.secret.key</name>
  <value>YOUR_SECRET_ACCESS_KEY</value>
</property>
<property>
  <name>fs.s3a.path.style.access</name>
  <value>true</value>
</property>
<property>
  <name>fs.s3a.connection.ssl.enabled</name>
  <value>true</value>
</property>
<property>
  <name>fs.s3a.aws.credentials.provider</name>
  <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
</property>
Troubleshooting
  • Connection refused / SSL errors: Ensure you are using https:// and that your endpoint is set to s3.diskspace.sh (without the https:// prefix in the fs.s3a.endpoint property).
  • Signature mismatch / 403 errors: Double-check your Access Key ID and Secret Access Key. Confirm path.style.access is set to true.
  • Region issues: Set fs.s3a.endpoint.region to us-east-1 or leave the region property empty.
  • Spark specific: When using Spark, also set spark.hadoop.fs.s3a.* properties with the same values.
Last updated: 2025 • DiskSpace.sh S3-compatible storage integration for Apache Spark and Hadoop