Version: Unreleased

10.1 -> 10.2

Summary

During the migration from Monitor 10.1 to Monitor 10.2, Elasticsearch is upgraded from 7.x → 8.x → 9.x. Because a direct upgrade from 7.x to 9.x isn’t supported, the process uses an intermediate 8.x cluster to keep data compatible and safe.

The intermediate step is required by Elasticsearch itself because a direct upgrade from 7.x to 9.x is not supported.

the reason for the intermediate is defined by Elasticsearch it self as direct upgrade from 7.x to 9.x is not supported.

Detailed Migration step by step instructions

Monitor 10.2 introduces changes to the database schema that require a migration process. Follow these steps to ensure a smooth transition:

Back up your data: Before starting the migration, ensure you have a complete backup of your current Monitor data and Elasticsearch indices. This is crucial to prevent data loss in case anything goes wrong during the migration process.
- Guide to create and restore cluster

Start temporary Elasticsearch 8.x migration cluster: Start a temporary Elasticsearch 8.x instance for the migration. Monitor 10.2 ultimately runs on Elasticsearch 9.x, but the upgrade from 7.x → 9.x must pass through 8.x.

tip

Copy the Elasticsearch 8.x compose file to your working directory where your compose file is located so the volume are going to match between the production and the migration volume.

monitor/
├─ docker-compose.yml             # production compose
├─ .env                           # production env file
├─ docker-compose.esUpgrade8.yml  # migration compose for Elasticsearch 8.x
└─ docker-compose.esUpgrade9.yml  # migration compose for Elasticsearch 9.x

If the folder is different then the volumes references are prepending the folder name to the volume, so make sure the volume names are matching.

Start the Elasticsearch service using Docker Compose:

docker-compose -f docker-compose.esUpgrade8.yml up -d

Show example docker-compose.yml

⬇️ Download docker-compose.esUpgrade8.yml

docker-compose.yml
services:
  elastic:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.19.5
    container_name: elastic-migration-container-8
    ports:
      - "9200:9200" # HTTP API
      - "9300:9300" # Transport protocol
    environment:
      - node.name=elastic-node-migration8
      - discovery.type=single-node # Run as single node
      - xpack.ml.enabled=false # Disable ML features
      - xpack.security.enabled=false # Disable security
      - xpack.security.http.ssl.enabled=false # Disable HTTP SSL
      - logger.level=info # Set log level to info
      - ES_JAVA_OPTS=-Xms6g -Xmx6g # Set JVM heap size
    volumes:
      - es-data:/usr/share/elasticsearch/data # Data volume
volumes:
  es-data:

wait until Elasticsearch is fully started and operational. Open the Elasticsearch web interface at http://<server_ip>:9200/_cat/indices?v=true --> all of the indices should be in green state.

Run the migration script on Elasticsearch 8.x: Use the provided migration script located in the migrations directory of the Monitor repository. Execute the script with the following command:

warning
Make sure the script at the end of the migration from the version 8.x to 9.x is executed and not exited as the same instance will be used in the step #6.
```
./migrateToEs9.sh
```
tip
if you cannot run the script as it is not an executable, you can change the permissions with:
chmod +x ./migrateToEs9.sh
- during the full process the script will be also outputting the logs into the file runtime.log
- at the end of the script execution, you should see a message indicating that the migration to Elasticsearch 8.x was successful and Elasticsearch 9.x can be started.
info
Verify migration and deprecations: After the migration script has completed, verify that all data has been successfully migrated by checking the Monitor application and ensuring that all previous data is accessible.
- when chekcing the URL http://<server_ip>:9200/_migration/deprecations?pretty --> no entry should be mentioned in the output
```
{
 "cluster_settings" : [ ],
 "node_settings" : [ ],
 "ilm_policies" : { },
 "data_streams" : { },
 "index_settings" : { },
 "templates" : { }
 }
```
Start Elasticsearch 9.x: After confirming that the migration to Elasticsearch 8.x was successful, you can now upgrade to Elasticsearch 9.x.
- Run the Elasticsearch 9.x compose file to start Elasticsearch version 9.x.
- Start the Elasticsearch service using Docker Compose:
```
docker-compose -f docker-compose.esUpgrade9.yml up -d
```
  Show example docker-compose.yml
  ⬇️ Download docker-compose.esUpgrade9.yml
  docker-compose.yml
  services: elastic: image: docker.elastic.co/elasticsearch/elasticsearch:9.1.5 container_name: elastic-migration-container-9 ports: - "9200:9200" # HTTP API - "9300:9300" # Transport protocol environment: - node.name=elastic-node-migration9 - discovery.type=single-node # Run as single node - xpack.ml.enabled=false # Disable ML features - xpack.security.enabled=false # Disable security - xpack.security.http.ssl.enabled=false # Disable HTTP SSL - logger.level=info # Set log level to info - ES_JAVA_OPTS=-Xms6g -Xmx6g # Set JVM heap size volumes: - es-data:/usr/share/elasticsearch/data # Data volume volumes: es-data:
- wait until Elasticsearch is fully started and operational. Open the Elasticsearch web interface at http://<server_ip>:9200/_cat/indices?v=true --> all of the indices should be in green state.
note
The loading of the Elasticsearch application may take some time depending on your system resources and the size of your database. Please be patient while the migration process completes.
You can monitor the progress by going to the http://<server_ip>:9200/_cat/indices?v=true as the states of the indices will change from yellow/red to green after some time, so please be patient.
Continue migration checks on Elasticsearch 9.x: Now you can continue with the next steps of the Elasticsearch upgrade process to version 9.x by pressing enter in the window
- at the end of the script it will show you that all shards/tasks are completed
info
- if the previous sessions was stopped you can re-run the script again with the same command and it will continue from the last step
Stop migration containers and start regular Monitor stack: Stop the Elasticsearch migration containers mentioned above and start your regular Monitor containers as per your standard setup.

warning
Make sure you don't remove the volume that houses your migrated Elasticsearch data.

Summary​

Detailed Migration step by step instructions​

Summary

Detailed Migration step by step instructions