---
title: Apache Hadoop
description: "# How to Connect PeaSoup S3 to Apache Hadoop Follow these steps to configure Apache Hadoop to use PeaSoup S3 as a cloud storage backend for Hadoop Distributed File System (HDFS) operations. Since PeaSoup is S3-compatible, you can integrate it…"
url: "https://peasoup.cloud/docs/apache-hadoop/"
updated: "2024-11-06"
category: "S3-compatible Storage Knowledge Base / Connecting S3 Clients"
---

# Apache Hadoop

# How to Connect PeaSoup S3 to Apache Hadoop

Follow these steps to configure Apache Hadoop to use PeaSoup S3 as a cloud storage backend for Hadoop Distributed File System (HDFS) operations. Since PeaSoup is S3-compatible, you can integrate it similarly to Amazon S3 for data storage.

## Prerequisites

- **PeaSoup S3 Access**: Ensure you have the following details:
  - Access key
  - Secret key
  - Bucket name
  - PeaSoup S3 endpoint URL https://**s3.eu-west-1.peasoup.cloud**
- **Apache Hadoop** cluster installed and running with the ability to use S3-compatible storage as a file system.

## Steps to Connect PeaSoup S3 to Apache Hadoop

1. **Edit Hadoop Configuration Files**:
   - To enable PeaSoup S3 as a storage backend, you need to edit the Hadoop configuration files. These are typically located in the Hadoop configuration directory (e.g., `/etc/hadoop/`).
   - Open the `core-site.xml` file using a text editor:
   - Add the following configuration details to `core-site.xml` to integrate PeaSoup S3:<configuration> <property> <name>fs.s3a.endpoint</name> <value>https://s3.eu-west-1.peasoup.cloud</value> </property> <property> <name>fs.s3a.access.key</name> <value>your-access-key</value> </property> <property> <name>fs.s3a.secret.key</name> <value>your-secret-key</value> </property> <property> <name>fs.s3a.impl</name> <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value> </property> <property> <name>fs.s3a.path.style.access</name> <value>true</value> </property> </configuration>
2. **Verify Hadoop S3A Support**:
   - Ensure that Hadoop is using the S3A connector, which is required for integrating S3-compatible storage. This connector is available in Hadoop 2.7 and later.
   - If not installed, make sure to include the required libraries for S3A by adding the appropriate JARs to the Hadoop classpath or installing them via your package manager.
3. **Set Up the PeaSoup S3 Bucket**:
   - Ensure that the PeaSoup S3 bucket is created and accessible. You can create the bucket through the PeaSoup web console or API.
   - Make sure that the access keys configured in `core-site.xml` have the necessary permissions to read and write to the bucket.
4. **Test the Connection**:
   - After configuring Hadoop, you can test the connection by running Hadoop commands to interact with PeaSoup S3.
   - For example, you can use the following command to list the contents of your S3 bucket:hadoop fs -ls s3a://your-bucket-name/
   - If everything is configured correctly, Hadoop will list the contents of your PeaSoup S3 bucket.
5. **Use PeaSoup S3 with Hadoop Jobs**:
   - Once the configuration is successful, you can start using PeaSoup S3 as a storage location for Hadoop MapReduce, Hive, Spark, or other Hadoop ecosystem tools.
   - Submit jobs or transfer data to and from the PeaSoup S3 bucket by specifying the bucket URL in the `s3a://` format.

## Optional Configuration

- You can fine-tune the performance of PeaSoup S3 integration by adjusting parameters such as:
  - **fs.s3a.connection.maximum**: Maximum number of connections to S3.
  - **fs.s3a.multipart.size**: Size of parts in a multipart upload.
  - **fs.s3a.fast.upload**: Enables faster uploads for large datasets.

## Notes

- Ensure that the network settings and firewall rules allow Hadoop to access PeaSoup S3 endpoints.
- PeaSoup's S3-compatible API is fully supported by Hadoop’s S3A connector, making it easy to integrate for large-scale data processing and storage needs.