Docker Migration Guide

This document is a guide on migrating a Metatron Discovery service across docker instances.

I suppose that you use https://github.com/teamsprint/docker-metatron.git/ for convenience on docker commands. Refer to https://metatron.app/2020/01/21/deploying-metatron-with-the-fully-engineered-docker-image/

I assume that you use MySQL as the metadata store.

1. Stop Metatron Service

Run the following command to get into the docker instance.

git clone https://github.com/teamsprint/docker-metatron.git/
cd docker-metatron
./attach.sh

Stop Metatron service with the following command.

cd $METATRON_HOME
bin/metatron.sh stop

2. Backup Metadata Store

We must backup all metadata used in Metatron like datasets, dataflows, etc. Run the following commands from host machine. (Container name is “metatron”, the database name of metadata store is “polaris”.

sudo docker exec metatron /usr/bin/mysqldump -uroot -pMetatron123$ polaris > metadata_store_backup.sql

3. Backup configuration files and run scripts.

sudo docker cp metatron:/servers/metatron-discovery/conf/application-config.yaml .
sudo docker cp metatron:/servers/metatron-discovery/conf/metatron-env.sh .
sudo docker cp metatron:/servers/metatron-discovery/conf/logback-console.sh .
sudo docker cp metatron:/servers/metatron-discovery/bin/metatron.sh .
sudo docker cp metatron:/servers/metatron-discovery/bin/common.sh .

4. Backup uploaded file datasets, data snapshots.

sudo docker cp metatron:/servers/metatron-discovery/dataprep/uploads .
sudo docker cp metatron:/servers/metatron-discovery/dataprep/snapshots .

Generally, you don’t need to backup data snapshots. If the snapshot is small enough, you can easily remake the snapshots. Or if it’s too big, backup size might be also too big.

By the way, you cannot backup snapshots stored in internal databases. If you didn’t modify configurations about staging DB (if the configuration is the default of initial image), then you cannot backup staging DB type snapshots.

5. Remove old docker instance

Run the following commands to remove the old docker instance.

./destroy.sh

6. Run new docker instance

Run the following commands from the host machine.

./run.sh

In case you patch the binary, you might need to edit run.sh to modify IMAGE_NAME.

Run the following commands inside the docker instance.

./prepare-all-metatron.sh

Normally, Metatron service will be ready in about 2~3 minutes. Check the service running, then shutdown right after. Let’s start to restore.

./stop-metatron.sh

7. Restore Metadata Store

Run the following commands from the host machine.

cat metadata_store_backup.sql | sudo docker exec -i metatron /usr/bin/mysql -uroot -pMetatron123$ polaris

8. Restore configrations, run scripts

Run the following commands from the host machine. In case you patch the binary, you should apply the changes to the corresponding files.

sudo docker cp application-config.yaml metatron:/servers/metatron-discovery/conf/
sudo docker cp metatron-env.sh metatron:/servers/metatron-discovery/conf/
sudo docker cp logback-console.sh metatron:/servers/metatron-discovery/conf/
sudo docker cp metatron.sh metatron:/servers/metatron-discovery/bin/
sudo docker cp common.sh metatron:/servers/metatron-discovery/bin/

9. Restore file datasets and snapshots

Run the following commands from the host machine.

sudo docker exec metatron mkdir -p /servers/metatron-discovery/dataprep
sudo docker cp uploads metatron:/servers/metatron-discovery/dataprep/
sudo docker cp snapshots metatron:/servers/metatron-discovery/dataprep/

10. Start New Metatron Service

Run the following commands from the host machine.

./attach.sh

Run the following commands inside the docker instance.

./start-metatron.sh

Generally, Metatron service will be ready in about 1~2 minuts.