How to Connect PeaSoup S3 to Elasticsearch

Follow these steps to configure Elasticsearch to store snapshots on PeaSoup S3. Since PeaSoup provides an S3-compatible API, it can be configured as a repository for backups and snapshots in Elasticsearch.

Prerequisites

• PeaSoup S3 Access: Ensure you have the following information:
  • Access key
  • Secret key
  • Bucket name
  • PeaSoup S3 endpoint URL https://s3.eu-west-1.peasoup.cloud
• Elasticsearch cluster running and accessible with the ability to install plugins.
• Elasticsearch S3 Repository Plugin installed on your Elasticsearch nodes.

Steps to Connect PeaSoup S3 to Elasticsearch

1. Install the S3 Repository Plugin (if not installed):
   • If you haven’t already installed the S3 repository plugin, you can do so by running the following command on your Elasticsearch nodes:sudo bin/elasticsearch-plugin install repository-s3
   • Restart your Elasticsearch service after the plugin installation is complete:sudo systemctl restart elasticsearch
2. Configure PeaSoup S3 in Elasticsearch:
   • Edit the Elasticsearch elasticsearch.yml configuration file to add PeaSoup S3 credentials:cloud: aws: s3: access_key: your-access-key secret_key: your-secret-key endpoint: https://s3.pscloud.io
   • Restart Elasticsearch to apply the changes:sudo systemctl restart elasticsearch
3. Create an S3 Snapshot Repository in Elasticsearch:
   • Once the plugin is installed and PeaSoup S3 credentials are configured, you can create a snapshot repository. Use the following Elasticsearch API call to register the repository:PUT _snapshot/pea_soup_backup { “type”: “s3”, “settings”: { “bucket”: “your-bucket-name”, “endpoint”: “https://s3.pscloud.io”, “access_key”: “your-access-key”, “secret_key”: “your-secret-key”, “protocol”: “https” } }
   • This command will create a snapshot repository named `pea_soup_backup` that uses PeaSoup S3 for storing snapshots.
4. Create a Snapshot:
   • Once the snapshot repository is created, you can take a snapshot of your Elasticsearch data. Use the following API command to create a snapshot:PUT _snapshot/pea_soup_backup/snapshot_1 { “indices”: “your-index-name”, “ignore_unavailable”: true, “include_global_state”: false }
   • Replace `your-index-name` with the index you want to back up, or you can omit the `indices` field to snapshot all indices.
   • Monitor the snapshot status with the following command:GET _snapshot/pea_soup_backup/snapshot_1/_status
5. Restore Data from PeaSoup S3:
   • To restore data from a snapshot stored in PeaSoup S3, use the following API command:POST _snapshot/pea_soup_backup/snapshot_1/_restore { “indices”: “your-index-name”, “include_global_state”: true }
   • This will restore the specified index or indices from the snapshot stored in PeaSoup S3.

Optional: Configure Snapshot Lifecycle Management (SLM)

• Elasticsearch offers Snapshot Lifecycle Management (SLM) to automate the process of taking snapshots. You can create an SLM policy to schedule regular snapshots using the PeaSoup S3 repository:PUT _slm/policy/daily-snapshot { “schedule”: “0 30 1 * * ?”, // Daily at 1:30 AM “name”: “”, “repository”: “pea_soup_backup”, “config”: { “indices”: [“your-index-name”], “ignore_unavailable”: false, “include_global_state”: false } }

Notes

• Ensure that Elasticsearch has network access to PeaSoup S3 by verifying your network and firewall settings.
• PeaSoup’s S3-compatible API allows seamless integration with Elasticsearch, providing scalable and reliable storage for snapshots and backups.