How to Connect PeaSoup S3 to Splunk

Follow these steps to configure Splunk to archive and store data on PeaSoup S3. PeaSoup provides an S3-compatible API, which allows it to be used similarly to Amazon S3 for storing archived data or backups from your Splunk environment.

Prerequisites

• PeaSoup S3 Access: Ensure you have the following details:
  • Access key
  • Secret key
  • Bucket name
  • PeaSoup S3 endpoint URL https://s3.eu-west-1.peasoup.cloud
• Splunk configured with administrative access and necessary permissions to manage indexes.

Steps to Connect PeaSoup S3 to Splunk

1. Log into Splunk Web Interface:
   • Open the Splunk web interface by navigating to your Splunk instance in a web browser.
   • Log in with your administrator credentials.
2. Configure Splunk Index for Archiving:
   • Go to Settings > Indexes to create or modify an index that will be archived to PeaSoup S3.
   • Click Edit or Create New Index based on your setup.
   • Configure the following:
     • Max Hot Bucket Size: Specify the size before the index rolls over from the hot bucket (active) to the warm bucket (nearline).
     • Frozen Path: Configure a frozen path where older data is archived. This will be set to PeaSoup S3 in the next step.
   • Save the index settings.
3. Configure PeaSoup S3 as the Archiving Destination:
   • To configure PeaSoup S3 as the destination for archiving frozen data:
     • Access your Splunk server via SSH or a terminal with administrative access.
     • Edit the indexes.conf file located in the Splunk configuration directory (typically $SPLUNK_HOME/etc/system/local/indexes.conf).
     • Add the following configuration under the desired index to configure PeaSoup S3 as the archiving (frozen) path:[] coldToFrozenScript = $SPLUNK_HOME/bin/scripts/s3-frozen-archive.sh frozenTimePeriodInSecs = # e.g., 31536000 for 1 year
     • Create a custom script called s3-frozen-archive.sh in the $SPLUNK_HOME/bin/scripts/ directory to handle the upload to PeaSoup S3.
   • The script will use your PeaSoup S3 credentials and upload the frozen data to PeaSoup. You can customize the script with the following sample:#!/bin/bash # s3-frozen-archive.sh: Script to archive Splunk data to PeaSoup S3 # PeaSoup S3 configuration S3_BUCKET=your-bucket-name S3_ACCESS_KEY=your-access-key S3_SECRET_KEY=your-secret-key S3_ENDPOINT=https://s3.eu-west-1.peasoup.cloud # Archive path ARCHIVE_PATH=$1 # Upload to PeaSoup S3 using AWS CLI or another tool aws s3 cp “$ARCHIVE_PATH” “s3://$S3_BUCKET/” –endpoint-url “$S3_ENDPOINT” –recursive
   • Ensure the script has executable permissions by running:chmod +x $SPLUNK_HOME/bin/scripts/s3-frozen-archive.sh
4. Test the Archiving Process:
   • After configuring the script and the index settings, force an index to roll over and archive data to PeaSoup S3.
   • Monitor the script execution and ensure that the data is successfully uploaded to your PeaSoup S3 bucket.
5. Retrieve Archived Data from PeaSoup S3 (Optional):
   • To retrieve archived data from PeaSoup S3, download the data using the PeaSoup S3 portal or via the AWS CLI:aws s3 cp s3://<peasoup-bucket-name>/archived-data-path” /local-path –endpoint-url https://s3.eu-west-1.peasoup.cloud
   • You can then restore the data back to your Splunk instance or other storage systems as needed.

Optional: Fine-Tune Frozen Data Archiving

• In the indexes.conf file, you can configure additional options such as:
  • Compression: Compress data before archiving it to PeaSoup S3 for reduced storage costs.
  • Retention Period: Adjust the frozenTimePeriodInSecs to define how long data should remain active before being archived.
  • Delete After Archiving: Ensure the data is deleted from Splunk’s local storage after it is successfully archived to PeaSoup S3.

Notes

• Ensure that your Splunk server can communicate with PeaSoup S3 by verifying network and firewall settings.
• PeaSoup’s S3-compatible API makes it easy to integrate with Splunk for scalable cloud storage and long-term data archiving.