---
title: Databricks
description: "# How to Connect PeaSoup S3 to Databricks Follow these steps to configure Databricks to access and store data on PeaSoup S3. Since PeaSoup uses an S3-compatible API, you can easily integrate it as a data source and storage backend…"
url: "https://peasoup.cloud/docs/databricks/"
updated: "2024-11-06"
category: "S3-compatible Storage Knowledge Base / Connecting S3 Clients"
---

# Databricks

# How to Connect PeaSoup S3 to Databricks

Follow these steps to configure Databricks to access and store data on PeaSoup S3. Since PeaSoup uses an S3-compatible API, you can easily integrate it as a data source and storage backend for your Databricks workflows.

## Prerequisites

- **PeaSoup S3 Access**: Ensure you have the following information:
  - Access key
  - Secret key
  - Bucket name
  - PeaSoup S3 endpoint URL https://**s3.eu-west-1.peasoup.cloud**
- **Databricks** workspace with access to configure external data sources.

## Steps to Connect PeaSoup S3 to Databricks

1. **Log into Databricks**:
   - Open your Databricks workspace by going to the [Databricks portal](https://databricks.com/) and logging in with your credentials.
2. **Set Up AWS S3 Credentials for PeaSoup**:
   - In Databricks, navigate to your notebook or cluster settings.
   - Set the following environment variables with your PeaSoup S3 credentials:spark.conf.set("fs.s3a.access.key", "your-access-key") spark.conf.set("fs.s3a.secret.key", "your-secret-key") spark.conf.set("fs.s3a.endpoint", "https://s3.eu-west-1.peasoup.cloud") spark.conf.set("fs.s3a.path.style.access", "true")
   - These settings will allow Databricks to authenticate and interact with PeaSoup S3 via the S3A protocol.
3. **Access PeaSoup S3 Data**:
   - Once the credentials are configured, you can read and write data to PeaSoup S3 using Databricks notebooks.
   - For example, to read a CSV file from PeaSoup S3:val df = spark.read.option("header", "true") .csv("s3a://your-bucket-name/path-to-your-file.csv") df.show()
   - To write data to PeaSoup S3:df.write.mode("overwrite").csv("s3a://your-bucket-name/output-path/")
4. **Configure PeaSoup S3 for Delta Lake (Optional)**:
   - Databricks integrates with Delta Lake for optimized storage. You can use PeaSoup S3 as a storage backend for Delta Lake by specifying the S3 path.
   - To read and write Delta Lake data to PeaSoup S3:df.write.format("delta").save("s3a://your-bucket-name/delta-lake-output/") val deltaDF = spark.read.format("delta").load("s3a://your-bucket-name/delta-lake-output/")
5. **Secure PeaSoup S3 with IAM Roles (Optional)**:
   - For enhanced security, you can use IAM roles to authenticate with PeaSoup S3 instead of hardcoding access keys. This is especially useful if PeaSoup supports IAM-like functionality for access control.
   - In the Databricks workspace, configure the cluster to assume an IAM role that has the necessary permissions for the PeaSoup S3 bucket.
6. **Test and Monitor Data Access**:
   - Once the setup is complete, run tests to read from and write to PeaSoup S3 via Databricks to ensure that everything is working as expected.
   - Use Databricks’ monitoring tools to track job progress and data access performance when interacting with PeaSoup S3.

## Optional: Tuning Performance for S3 Access

- For improved performance when working with large datasets, you can adjust some of the S3A connector settings:
  - **spark.hadoop.fs.s3a.connection.maximum**: Increase the maximum number of connections for high throughput.
  - **spark.hadoop.fs.s3a.multipart.size**: Adjust the multipart upload size for larger data transfers.
  - **spark.hadoop.fs.s3a.fast.upload**: Enable faster uploads for large files.

## Notes

- Ensure that the network and firewall settings on PeaSoup S3 allow access from Databricks to the specified S3 endpoint.
- PeaSoup’s S3-compatible API makes it easy to use with Databricks, enabling cloud-based data processing and storage for advanced analytics and machine learning workflows.