Redis cluster command line deployment tool

Preparation before use:
1) Configure the public redis.conf file unrelated to the port, and place it in the same directory as the tool
2) Configure the PORT related template redis-PORT.conf file, and put it in the same directory as the tool (PORT will be replaced with specific PORT number during deployment)
3) Configure the node file redis? Cluster.nodes, which is also in the same directory as the tool
The file format of Redis cluster.nodes is one node of Redis cluster for each line. It supports comment lines starting with "ා". Example format:
127.0.0.1 6381
127.0.0.1 6382
127.0.0.1 6383
127.0.0.1 6384
127.0.0.1 6385
127.0.0.1 6386
4) Create the directory where redis is installed (the build batch tool Mooon? SSH is completed, and deploy? Redis? Cluster.sh mainly uses this batch tool)
5) Other more detailed can directly see the source code, there are detailed instructions.

Set up to download https://github.com/eyjian/redis-tools/tree/master/deploy to a directory. When you run the deploy ﹣ redis ﹣ cluster.sh tool, it will prompt various preconditions, such as whether redis cli is available.

Source code (available from https://github.com/eyjian/redis-tools Download):

#!/bin/bash
# Source code: https://github.com/eyjian/redis-tools
# a tool to deploy a redis cluster
#
# Automatically deploy redis cluster tools,
# Remote operation is enough without logging in to any machine in the Redis cluster.
#
# To batch create user redis as root:
# export H=192.168.0.5,192.168.0.6,192.168.0.7,192.168.0.8,192.168.0.9
# export U=root
# export P='root^1234'
# mooon_ssh -c='groupadd redis; useradd -g redis -m redis; echo "redis:redis#1234"|chpasswd'
#
# Batch create the redis installation directory / data/redis-4.0.11, and set the owner as user redis, and the user group as redis example:
# mooon_ssh -c='mkdir /data/redis-4.0.11;ln -s /data/redis-4.0.11 /data/redis;chown redis:redis /data/redis*'
#
# Process monitor.sh can be used to monitor the redis server process restart:
# https://github.com/eyjian/libmooon/blob/master/shell/process_monitor.sh
# Use example:
# * * * * * /usr/local/bin/process_monitor.sh "/usr/local/redis/bin/redis-server 6379" "/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis-6379.conf"
# * * * * * /usr/local/bin/process_monitor.sh "/usr/local/redis/bin/redis-server 6380" "/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis-6380.conf"
# You can find the operation log of process_monitor.sh in the / tmp directory. When the process of the corresponding port is not available, it will restart the process of the corresponding port within 5 seconds.
#
# Operation parameters:
# Parameter 1 SSH port
# Parameter 2 installation user
# Parameter 3 install user password
# Parameter 4 installation directory
#
# Preconditions (you can use the batch tools Mooon? SSH and Mooon? Upload to complete):
# 1) The installation user has created
# 2) Setup user password has been set
# 3) The installation directory has been created, and the owner of the directory is the installation user
# 4) ruby is installed on the machine where the tool is executed, and the version number is no less than 2.0.0
# 5) redis-X.Y.Z.gem is installed on the machine executing this tool, and the version number is no less than redis-3.0.0.gem
#
# 6) The following executables exist in the same directory:
# 6.1)redis-server
# 6.2)redis-cli
# 6.3)redis-check-rdb
# 6.4)redis-check-aof
# 6.5)redis-trib.rb
#
# 7) There are two configuration files in the same directory:
# 7.1)redis.conf
# 7.2)redis-PORT.conf
# redis.conf is the public configuration file,
# redis-PORT.conf is the profile template of the specified port,
# At the same time, you need to replace the directory and port in the redis-PORT.conf file with INSTALLDIR and REDISPORT respectively. For example:
# include INSTALLDIR/conf/redis.conf
# pidfile INSTALLDIR/bin/redis-REDISPORT.pid
# logfile INSTALLDIR/log/redis-REDISPORT.log
# port REDISPORT
# dbfilename dump-REDISPORT.rdb
# dir INSTALLDIR/data/REDISPORT
#
# Where INSTALLDIR is replaced with the value of parameter 4,
# And REDISPORT will use the port number in redis? Cluster.nodes instead
#
# The configuration file redis? Cluster.nodes defines the nodes where redis is installed
# File format (in“#”Note at the beginning):
# Each line consists of IP and port numbers separated by spaces, commas, semicolons, or TAB characters
#
# Dependence:
# 1) Mooon SSH batch command tool for remote operation of multiple machines
# 2) Mooon? Upload remote operation multiple machines batch upload tool
# 3)https://raw.githubusercontent.com/eyjian/libmooon
# 4) libmooon also relies on libssh2 (http://www.libssh2.org/)

BASEDIR=$(dirname $(readlink -f $0))
REDIS_CLUSTER_NODES=$BASEDIR/redis_cluster.nodes

# Batch command tool
MOOON_SSH=mooon_ssh
# Bulk upload tool
MOOON_UPLOAD=mooon_upload
# Create redis cluster tool
REDIS_TRIB=$BASEDIR/redis-trib.rb
# redis-server
REDIS_SERVER=$BASEDIR/redis-server
# redis-cli
REDIS_CLI=$BASEDIR/redis-cli
# redis-check-aof
REDIS_CHECK_AOF=$BASEDIR/redis-check-aof
# redis-check-rdb
REDIS_CHECK_RDB=$BASEDIR/redis-check-rdb
# redis.conf
REDIS_CONF=$BASEDIR/redis.conf
# redis-PORT.conf
REDIS_PORT_CONF=$BASEDIR/redis-PORT.conf

# global variable
# Total nodes of the redis cluster
num_nodes=0
# All IP arrays that make up the redis cluster
redis_node_ip_array=()
# Array of all nodes that make up the redis cluster (IP+port constructs a redis node)
redis_node_array=()

# usage
function usage()
{
    echo -e "\033[1;33mUsage\033[m: `basename $0` \033[0;32;32mssh-port\033[m install-user \033[0;32;32minstall-user-password\033[m install-dir"
    echo -e "\033[1;33mExample\033[m: `basename $0` \033[0;32;32m22\033[m redis \033[0;32;32mredis^1234\033[m /usr/local/redis-4.0.11"
}

# Five parameters need to be specified
if test $# -ne 4; then
    usage
    echo ""
    exit 1
fi

ssh_port="$1"
install_user="$2"
install_user_password="$3"
install_dir="$4"
echo -e "[ssh port] \033[1;33m$ssh_port\033[m"
echo -e "[install user] \033[1;33m$install_user\033[m"
echo -e "[install directory] \033[1;33m$install_dir\033[m"
echo ""

# Check if ruby is available
which ruby > /dev/null 2>&1
if test $? -eq 0; then
    echo -e "Checking ruby OK"
else
    echo -e "ruby \033[0;32;31mnot exists or not executable\033[m"
    echo "https://www.ruby-lang.org"
    echo -e "Exit now\n"
    exit 1
fi

# Check if gem is available
which gem > /dev/null 2>&1
if test $? -eq 0; then
    echo -e "Checking gem OK"
else
    echo -e "gem \033[0;32;31mnot exists or not executable\033[m"
    echo "https://rubygems.org/pages/download"
    echo -e "Exit now\n"
    exit 1
fi

# Check whether Mooon? SSH is available
which "$MOOON_SSH" > /dev/null 2>&1
if test $? -eq 0; then
    echo -e "Checking $MOOON_SSH OK"
else
    echo -e "$MOOON_SSH \033[0;32;31mnot exists or not executable\033[m"
    echo "There are two versions: C++ and GO:"
    echo "https://github.com/eyjian/libmooon/releases"
    echo "https://raw.githubusercontent.com/eyjian/libmooon/master/tools/mooon_ssh.cpp"
    echo "https://raw.githubusercontent.com/eyjian/libmooon/master/tools/mooon_ssh.go"    
    echo -e "Exit now\n"
    exit 1
fi

# Check whether Mooon? Upload is available
which "$MOOON_UPLOAD" > /dev/null 2>&1
if test $? -eq 0; then
    echo -e "Checking $MOOON_UPLOAD OK"
else
    echo -e "$MOOON_UPLOAD \033[0;32;31mnot exists or not executable\033[m"
    echo "There are two versions: C++ and GO:"
    echo "https://github.com/eyjian/libmooon/releases"
    echo "https://raw.githubusercontent.com/eyjian/libmooon/master/tools/mooon_upload.cpp"
    echo "https://raw.githubusercontent.com/eyjian/libmooon/master/tools/mooon_upload.go"
    echo -e "Exit now\n"
    exit 1
fi

# Check whether redis-trib.rb is available
which "$REDIS_TRIB" > /dev/null 2>&1
if test $? -eq 0; then
    echo -e "Checking $REDIS_TRIB OK"
else
    echo -e "$REDIS_TRIB \033[0;32;31mnot exists or not executable\033[m"
    echo -e "Exit now\n"
    exit 1
fi

# Check whether redis server is available
which "$REDIS_SERVER" > /dev/null 2>&1
if test $? -eq 0; then
    echo -e "Checking $REDIS_SERVER OK"
else
    echo -e "$REDIS_SERVER \033[0;32;31mnot exists or not executable\033[m"
    echo -e "Exit now\n"
    exit 1
fi

# Check whether redis cli is available
which "$REDIS_CLI" > /dev/null 2>&1
if test $? -eq 0; then
    echo -e "Checking $REDIS_CLI OK"
else
    echo -e "$REDIS_CLI \033[0;32;31mnot exists or not executable\033[m"
    echo -e "Exit now\n"
    exit 1
fi

# Check whether redis check AOF is available
which "$REDIS_CHECK_AOF" > /dev/null 2>&1
if test $? -eq 0; then
    echo -e "Checking $REDIS_CHECK_AOF OK"
else
    echo -e "$REDIS_CHECK_AOF \033[0;32;31mnot exists or not executable\033[m"
    echo -e "Exit now\n"
    exit 1
fi

# Check whether redis check RDB is available
which "$REDIS_CHECK_RDB" > /dev/null 2>&1
if test $? -eq 0; then
    echo -e "Checking $REDIS_CHECK_RDB OK"
else
    echo -e "$REDIS_CHECK_RDB \033[0;32;31mnot exists or not executable\033[m"
    echo -e "Exit now\n"
    exit 1
fi

# Check whether redis.conf is available
if test -r "$REDIS_CONF"; then
    echo -e "Checking $REDIS_CONF OK"
else
    echo -e "$REDIS_CONF \033[0;32;31mnot exists or not readable\033[m"
    echo -e "Exit now\n"
    exit 1
fi

# Check whether redis-PORT.conf is available
if test -r "$REDIS_PORT_CONF"; then
    echo -e "Checking $REDIS_PORT_CONF OK"
else
    echo -e "$REDIS_PORT_CONF \033[0;32;31mnot exists or not readable\033[m"
    echo -e "Exit now\n"
    exit 1
fi

# Resolve the redis? Cluster.nodes file,
# All the nodes that make up the redis cluster are obtained.
function parse_redis_cluster_nodes()
{    
    redis_nodes_str=
    redis_nodes_ip_str=
    while read line
    do
        # Remove leading and trailing spaces
        line=`echo "$line" | xargs`
        if test -z "$line" -o "$line" = "#"; then
            continue
        fi

        # Skip annotations
        begin_char=${line:0:1}
        if test "$begin_char" = "#"; then
            continue
        fi

        # Get IP and port
        eval $(echo "$line" | awk -F[\ \:,\;\t]+ '{ printf("ip=%s\nport=%s\n",$1,$2); }')

        # Both IP and port must have
        if test ! -z "$ip" -a ! -z "$port"; then
            if test -z "$redis_nodes_ip_str"; then
                redis_nodes_ip_str=$ip
            else
                redis_nodes_ip_str="$redis_nodes_ip_str,$ip"
            fi

            if test -z "$redis_nodes_str"; then
                redis_nodes_str="$ip:$port"
            else
                redis_nodes_str="$redis_nodes_str,$ip:$port"
            fi          
        fi
    done < $REDIS_CLUSTER_NODES   
    
    if test -z "$redis_nodes_ip_str"; then
        num_nodes=0
    else
        # Get the IP array redis? Node? IP? Array    
        redis_node_ip_array=`echo "$redis_nodes_ip_str" | tr ',' '\n' | sort | uniq`

        # Get the node array
        redis_node_array=`echo "$redis_nodes_str" | tr ',' '\n' | sort | uniq`
        
        for redis_node in ${redis_node_array[@]};
        do
            num_nodes=$((++num_nodes))
            echo "$redis_node"
        done
    fi
}

# check redis_cluster.nodes
if test ! -r $REDIS_CLUSTER_NODES; then
    echo -e "File $REDIS_CLUSTER_NODES \033[0;32;31mnot exits\033[m"
    echo ""
    echo -e "\033[0;32;32mFile format\033[m (columns delimited by space, tab, comma, semicolon or colon):"
    echo "IP1 port1"
    echo "IP2 port2"
    echo ""
    echo -e "\033[0;32;32mExample\033[m:"
    echo "127.0.0.1 6381"
    echo "127.0.0.1 6382"
    echo "127.0.0.1 6383"
    echo "127.0.0.1 6384"
    echo "127.0.0.1 6385"
    echo "127.0.0.1 6386"
    echo -e "Exit now\n"
    exit 1
else    
    echo -e "\033[0;32;32m"
    parse_redis_cluster_nodes
    echo -e "\033[m"

    if test $num_nodes -lt 1; then
        echo -e "Checking $REDIS_CLUSTER_NODES \033[0;32;32mfailed\033[m: no any node"
        echo -e "Exit now\n"
        exit 1
    else
        echo -e "Checking $REDIS_CLUSTER_NODES OK, the number of nodes is \033[1;33m${num_nodes}\033[m"
    fi
fi

# Continue after confirmation
while true
do
    # At least six nodes are needed to form a redis cluster
    if test $num_nodes -lt 6; then
        echo -e "\033[0;32;32mAt least 6 nodes are required to create a redis cluster\033[m"
    fi

    # Prompt whether to continue
    echo -en "Are you sure to continue? [\033[1;33myes\033[m/\033[1;33mno\033[m]"
    read -r -p " " input

    if test "$input" = "no"; then
        echo -e "Exit now\n"
        exit 1
    elif test "$input" = "yes"; then
        echo "Starting to install ..."
        echo ""
        break
    fi
done

# Do you want to empty the installation directory before installing?
clear_install_directory=
while true
do
    echo -en "Clear install directory? [\033[1;33myes\033[m/\033[1;33mno\033[m]"
    read -r -p " " clear_install_directory

    if test "$clear_install_directory" = "no"; then
        echo ""
        break
    elif test "$clear_install_directory" = "yes"; then        
        echo ""
        break
    fi
done

# Install public, including executable files and public configuration files
function install_common()
{
    redis_ip="$1"
    
    # Check whether the installation directory exists and has read-write permission
    echo "$MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c=\"test -d $install_dir && test -r $install_dir && test -w $install_dir && test -x $install_dir\""
    $MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c="test -d $install_dir && test -r $install_dir && test -w $install_dir && test -x $install_dir"
    if test $? -ne 0; then
        echo ""
        echo -e "Directory $install_dir \033[1;33mnot exists or no (rwx) permission\033[m"
        echo -e "Exit now\n"
        exit 1
    fi

    # Empty installation directory
    if test "$clear_install_directory" = "yes"; then
        echo ""
        echo "$MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c=\"killall -q -w -u $install_user redis-server\""
        $MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c="killall -q -w -u $install_user redis-server"

        echo "$MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c=\"rm -fr $install_dir/*\""
        $MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c="rm -fr $install_dir/*"
        if test $? -ne 0; then
            echo -e "Exit now\n"
            exit 1
        fi
    fi

    # create directory
    echo ""
    echo "$MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c=\"cd $install_dir;mkdir -p bin conf log data\""
    $MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c="cd $install_dir;mkdir -p bin conf log data"
    if test $? -ne 0; then
        echo -e "Exit now\n"
        exit 1
    fi

    # upload configuration files
    echo ""
    echo "$MOOON_UPLOAD -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -s=redis.conf -d=$install_dir/conf"
    $MOOON_UPLOAD -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -s=redis.conf -d=$install_dir/conf 
    if test $? -ne 0; then
        echo -e "Exit now\n"
        exit 1
    fi

    # upload executable files
    echo ""
    echo "$MOOON_UPLOAD -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -s=redis-server,redis-cli,redis-check-aof,redis-check-rdb -d=$install_dir/bin"    
    $MOOON_UPLOAD -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -s=redis-server,redis-cli,redis-check-aof,redis-check-rdb,redis-trib.rb -d=$install_dir/bin
    if test $? -ne 0; then
        echo -e "Exit now\n"
        exit 1
    fi
}

# Install node profile
function install_node_conf()
{
    redis_ip="$1"
    redis_port="$2"

    # Generate node profile
    cp redis-PORT.conf redis-$redis_port.conf
    sed -i "s|INSTALLDIR|$install_dir|g;s|REDISPORT|$redis_port|g" redis-$redis_port.conf

    # create data directory for the given node
    echo ""
    echo "$MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c=\"cd $install_dir;mkdir -p data/$redis_port\""
    $MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c="cd $install_dir;mkdir -p data/$redis_port"
    if test $? -ne 0; then
        rm -f redis-$redis_port.conf
        echo -e "Exit now\n"
        exit 1
    fi

    # upload configuration files
    echo ""
    echo "$MOOON_UPLOAD -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -s=redis-$redis_port.conf -d=$install_dir/conf"
    $MOOON_UPLOAD -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -s=redis-$redis_port.conf -d=$install_dir/conf    
    if test $? -ne 0; then
        rm -f redis-$redis_port.conf
        echo -e "Exit now\n"
        exit 1
    fi

    rm -f redis-$redis_port.conf
}

function start_redis_node()
{
    redis_ip="$1"
    redis_port="$2"

    # start redis instance
    echo ""
    echo "$MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c=\"$install_dir/bin/redis-server $install_dir/conf/redis-$redis_port.conf\""    
    $MOOON_SSH -h=$redis_ip -P=$ssh_port -u=$install_user -p=$install_user_password -c="nohup $install_dir/bin/redis-server $install_dir/conf/redis-$redis_port.conf > /dev/null 2>&1 &"
    if test $? -ne 0; then
        echo -e "Exit now\n"
        exit 1
    fi
}

# Install public, including executable files and public configuration files
echo ""
echo -e "\033[1;33m================================\033[m"
for redis_node_ip in $redis_node_ip_array;
do
    echo -e "[\033[1;33m$redis_node_ip\033[m] Installing common ..."
    install_common $redis_node_ip
done

# Install node profile
echo ""
echo -e "\033[1;33m================================\033[m"
for redis_node in ${redis_node_array[@]};
do
    node_ip=
    node_port=

    eval $(echo "$redis_node" | awk -F[\ \:,\;\t]+ '{ printf("node_ip=%s\nnode_port=%s\n",$1,$2); }')
    if test -z "$node_ip" -o -z "$node_port"; then
        continue
    fi
    
    echo -e "[\033[1;33m$node_ip:$node_port\033[m] Installing node ..."
    install_node_conf $node_ip $node_port
done

# Continue after confirmation
echo ""
echo -e "\033[1;33m================================\033[m"
while true
do
    echo -en "Start redis? [\033[1;33myes\033[m/\033[1;33mno\033[m]"
    read -r -p " " input

    if test "$input" = "no"; then
        echo ""
        exit 1
    elif test "$input" = "yes"; then
        echo "Starting to start redis ..."
        echo ""
        break
    fi
done

# start redis instance
for redis_node in ${redis_node_array[@]};
do
    eval $(echo "$redis_node" | awk -F[\ \:,\;\t]+ '{ printf("node_ip=%s\nnode_port=%s\n",$1,$2); }')
    if test -z "$node_ip" -o -z "$node_port"; then
        continue
    fi

    echo -e "[\033[1;33m$node_ip:$node_port\033[m] Starting node ..."
    start_redis_node $node_ip $node_port
done

echo ""
echo -e "\033[1;33m================================\033[m"
echo "Number of nodes: $num_nodes"
if test $num_nodes -lt 6; then
    echo "Number of nodes less than 6, can not create redis cluster"
    echo -e "Exit now\n"
    exit 1
else
    redis_nodes_str=`echo "$redis_nodes_str" | tr ',' ' '`

    # Continue after confirmation
    echo ""
    while true
    do
        echo -en "Create redis cluster? [\033[1;33myes\033[m/\033[1;33mno\033[m]"
        read -r -p " " input

        if test "$input" = "no"; then
            echo ""
            exit 1
        elif test "$input" = "yes"; then
            echo "Starting to create redis cluster with $redis_nodes_str ... ..."
            echo ""
            break
        fi
    done

    # Create a redis cluster
    # redis-trib.rb create --replicas 1    
    $REDIS_TRIB create --replicas 1 $redis_nodes_str
    echo -e "Exit now\n"
    exit 0
fi

 

Keywords: Redis ssh github Ruby

Added by tony.j.jackson@o2.co.uk on Thu, 02 Jan 2020 08:13:53 +0200