Apache Spark / Hadoop user guide

Platform/OS
Linux, macOS, Windows
Category
Data Science & Analytics

Apache Spark / Hadoop

Apache Spark and Hadoop are powerful tools that let companies process huge amounts of data that would overwhelm normal computers. They work like a giant team of helpers that split up a massive job, give each person a small piece to work on, and then combine all the answers quickly.
They use DiskSpace S3-compatible object storage as a giant, reliable warehouse to keep all the raw data and final results. Instead of being limited by the hard drives inside the computers doing the work, Spark and Hadoop can pull data from S3, process it, and write the answers straight back to S3.
This combination gives businesses the ability to analyze enormous datasets at low cost while easily scaling up or down whenever they need to.

Connection / Setup

How to get your DiskSpace S3 credentials
New Accounts
  1. Create an account and log in to DiskSpace
  2. Click I'm ready to set up my storage
  3. Enter Bucket Name following the instructions
  4. Click Create Bucket and wait for it to be generated
  5. Find your Endpoint URL, Access Key, and Secret Key in the S3 Client Configuration section for use below
Existing Accounts
  1. Log in to DiskSpace
  2. Scroll to DiskSpace S3 Connection Info
  3. Find your Endpoint URL, Access Key, Secret Key, and Bucket Name for use below

Sign up or log in to set up your DiskSpace S3 Storage for Apache Spark / Hadoop.