View Categories

Elasticsearch

How to Connect PeaSoup S3 to Elasticsearch #

Follow these steps to configure Elasticsearch to store snapshots on PeaSoup S3. Since PeaSoup provides an S3-compatible API, it can be configured as a repository for backups and snapshots in Elasticsearch.

Prerequisites #

  • PeaSoup S3 Access: Ensure you have the following information:
    • Access key
    • Secret key
    • Bucket name
    • PeaSoup S3 endpoint URL (e.g., https://s3.pscloud.io)
  • Elasticsearch cluster running and accessible with the ability to install plugins.
  • Elasticsearch S3 Repository Plugin installed on your Elasticsearch nodes.

Steps to Connect PeaSoup S3 to Elasticsearch #

  1. Install the S3 Repository Plugin (if not installed):
    • If you haven’t already installed the S3 repository plugin, you can do so by running the following command on your Elasticsearch nodes:sudo bin/elasticsearch-plugin install repository-s3
    • Restart your Elasticsearch service after the plugin installation is complete:sudo systemctl restart elasticsearch
  2. Configure PeaSoup S3 in Elasticsearch:
    • Edit the Elasticsearch elasticsearch.yml configuration file to add PeaSoup S3 credentials:cloud: aws: s3: access_key: your-access-key secret_key: your-secret-key endpoint: https://s3.pscloud.io
    • Restart Elasticsearch to apply the changes:sudo systemctl restart elasticsearch
  3. Create an S3 Snapshot Repository in Elasticsearch:
    • Once the plugin is installed and PeaSoup S3 credentials are configured, you can create a snapshot repository. Use the following Elasticsearch API call to register the repository:PUT _snapshot/pea_soup_backup { “type”: “s3”, “settings”: { “bucket”: “your-bucket-name”, “endpoint”: “https://s3.pscloud.io”, “access_key”: “your-access-key”, “secret_key”: “your-secret-key”, “protocol”: “https” } }
    • This command will create a snapshot repository named `pea_soup_backup` that uses PeaSoup S3 for storing snapshots.
  4. Create a Snapshot:
    • Once the snapshot repository is created, you can take a snapshot of your Elasticsearch data. Use the following API command to create a snapshot:PUT _snapshot/pea_soup_backup/snapshot_1 { “indices”: “your-index-name”, “ignore_unavailable”: true, “include_global_state”: false }
    • Replace `your-index-name` with the index you want to back up, or you can omit the `indices` field to snapshot all indices.
    • Monitor the snapshot status with the following command:GET _snapshot/pea_soup_backup/snapshot_1/_status
  5. Restore Data from PeaSoup S3:
    • To restore data from a snapshot stored in PeaSoup S3, use the following API command:POST _snapshot/pea_soup_backup/snapshot_1/_restore { “indices”: “your-index-name”, “include_global_state”: true }
    • This will restore the specified index or indices from the snapshot stored in PeaSoup S3.

Optional: Configure Snapshot Lifecycle Management (SLM) #

  • Elasticsearch offers Snapshot Lifecycle Management (SLM) to automate the process of taking snapshots. You can create an SLM policy to schedule regular snapshots using the PeaSoup S3 repository:PUT _slm/policy/daily-snapshot { “schedule”: “0 30 1 * * ?”, // Daily at 1:30 AM “name”: “”, “repository”: “pea_soup_backup”, “config”: { “indices”: [“your-index-name”], “ignore_unavailable”: false, “include_global_state”: false } }

Notes #

  • Ensure that Elasticsearch has network access to PeaSoup S3 by verifying your network and firewall settings.
  • PeaSoup’s S3-compatible API allows seamless integration with Elasticsearch, providing scalable and reliable storage for snapshots and backups.