Table of Contents
How to Connect PeaSoup S3 to Elasticsearch #
Follow these steps to configure Elasticsearch to store snapshots on PeaSoup S3. Since PeaSoup provides an S3-compatible API, it can be configured as a repository for backups and snapshots in Elasticsearch.
Prerequisites #
- PeaSoup S3 Access: Ensure you have the following information:
- Access key
- Secret key
- Bucket name
- PeaSoup S3 endpoint URL (e.g.,
https://s3.pscloud.io
)
- Elasticsearch cluster running and accessible with the ability to install plugins.
- Elasticsearch S3 Repository Plugin installed on your Elasticsearch nodes.
Steps to Connect PeaSoup S3 to Elasticsearch #
- Install the S3 Repository Plugin (if not installed):
- If you haven’t already installed the S3 repository plugin, you can do so by running the following command on your Elasticsearch nodes:sudo bin/elasticsearch-plugin install repository-s3
- Restart your Elasticsearch service after the plugin installation is complete:sudo systemctl restart elasticsearch
- Configure PeaSoup S3 in Elasticsearch:
- Edit the Elasticsearch
elasticsearch.yml
configuration file to add PeaSoup S3 credentials:cloud: aws: s3: access_key: your-access-key secret_key: your-secret-key endpoint: https://s3.pscloud.io - Restart Elasticsearch to apply the changes:sudo systemctl restart elasticsearch
- Edit the Elasticsearch
- Create an S3 Snapshot Repository in Elasticsearch:
- Once the plugin is installed and PeaSoup S3 credentials are configured, you can create a snapshot repository. Use the following Elasticsearch API call to register the repository:PUT _snapshot/pea_soup_backup { “type”: “s3”, “settings”: { “bucket”: “your-bucket-name”, “endpoint”: “https://s3.pscloud.io”, “access_key”: “your-access-key”, “secret_key”: “your-secret-key”, “protocol”: “https” } }
- This command will create a snapshot repository named `pea_soup_backup` that uses PeaSoup S3 for storing snapshots.
- Create a Snapshot:
- Once the snapshot repository is created, you can take a snapshot of your Elasticsearch data. Use the following API command to create a snapshot:PUT _snapshot/pea_soup_backup/snapshot_1 { “indices”: “your-index-name”, “ignore_unavailable”: true, “include_global_state”: false }
- Replace `your-index-name` with the index you want to back up, or you can omit the `indices` field to snapshot all indices.
- Monitor the snapshot status with the following command:GET _snapshot/pea_soup_backup/snapshot_1/_status
- Restore Data from PeaSoup S3:
- To restore data from a snapshot stored in PeaSoup S3, use the following API command:POST _snapshot/pea_soup_backup/snapshot_1/_restore { “indices”: “your-index-name”, “include_global_state”: true }
- This will restore the specified index or indices from the snapshot stored in PeaSoup S3.
Optional: Configure Snapshot Lifecycle Management (SLM) #
- Elasticsearch offers Snapshot Lifecycle Management (SLM) to automate the process of taking snapshots. You can create an SLM policy to schedule regular snapshots using the PeaSoup S3 repository:PUT _slm/policy/daily-snapshot { “schedule”: “0 30 1 * * ?”, // Daily at 1:30 AM “name”: “”, “repository”: “pea_soup_backup”, “config”: { “indices”: [“your-index-name”], “ignore_unavailable”: false, “include_global_state”: false } }
Notes #
- Ensure that Elasticsearch has network access to PeaSoup S3 by verifying your network and firewall settings.
- PeaSoup’s S3-compatible API allows seamless integration with Elasticsearch, providing scalable and reliable storage for snapshots and backups.