How to Connect PeaSoup S3 to Graylog

Follow these steps to configure Graylog to archive logs and data to PeaSoup S3. PeaSoup provides an S3-compatible API, allowing Graylog to offload and store log data in the cloud, similar to Amazon S3.

Prerequisites

• PeaSoup S3 Access: Ensure you have the following details:
  • Access key
  • Secret key
  • Bucket name
  • PeaSoup S3 endpoint URL https://s3.eu-west-1.peasoup.cloud
• Graylog installed and running, with administrative access.
• Graylog Archive Plugin (optional, depending on your version of Graylog).

Steps to Connect PeaSoup S3 to Graylog

1. Log into Graylog Web Interface:
   • Open the Graylog web interface by navigating to your Graylog instance in a web browser.
   • Log in using your administrator credentials.
2. Install the Graylog S3 Plugin (if needed):
   • If your Graylog version does not come with native S3 archiving support, install the Graylog S3 plugin to enable this feature. You can download the plugin from the Graylog Marketplace.
   • Once installed, restart Graylog to activate the plugin.
3. Configure PeaSoup S3 in Graylog:
   • In the Graylog web interface, navigate to System > Configurations > Archiving (or Manage Archives depending on your version).
   • Select Amazon S3 as the storage provider and configure the following details to connect to PeaSoup S3:
     • Bucket Name: Enter your PeaSoup S3 bucket name.
     • Access Key: Enter your PeaSoup S3 access key.
     • Secret Key: Enter your PeaSoup S3 secret key.
     • Region: Leave this blank unless PeaSoup specifies a region.
     • Custom Endpoint: Enter the PeaSoup S3 endpoint https://s3.eu-west-1.peasoup.cloud
   • Click Save to apply the configuration.
4. Configure Archiving Policies:
   • After setting up PeaSoup S3, configure archiving policies for your logs in Graylog.
   • Navigate to System > Indices > Index Set and select the index set you wish to archive.
   • Under the Retention settings, choose Archive as the action when indices are closed or data reaches a specific retention period.
   • Set how frequently Graylog should archive the data and the duration for which logs should remain stored in PeaSoup S3.
   • Save the configuration to apply the archiving rules.
5. Test Archiving to PeaSoup S3:
   • Once the archiving rules are set, force an index to roll over by manually closing an index, or wait until Graylog automatically archives data based on your retention policies.
   • Monitor the Graylog logs and the PeaSoup S3 bucket to verify that the log data is successfully archived in the specified bucket.
6. Retrieve Archived Data from PeaSoup S3 (Optional):
   • To retrieve archived log data from PeaSoup S3, you can use the Graylog interface or download the data directly using the PeaSoup S3 console or AWS CLI:aws s3 cp “s3://your-bucket-name/path-to-archive” /local-path –endpoint-url https://s3.pscloud.io
   • Import the archived data back into Graylog if needed for analysis or recovery.

Optional: Fine-Tune Archive Settings

• In Graylog’s archiving configuration, you can fine-tune additional settings:
  • Compression: Enable compression to reduce the size of archived logs before uploading them to PeaSoup S3.
  • Encryption: Ensure that archived data is encrypted before it is transferred to PeaSoup S3 for enhanced security.
  • Retention Policies: Set specific rules for how long archived data should be retained in PeaSoup S3.

Notes

• Ensure that Graylog has network access to PeaSoup S3 by verifying your network and firewall settings.
• PeaSoup’s S3-compatible API allows for seamless integration with Graylog, providing scalable and reliable cloud storage for log data archiving.