starrocks 部署-存算一体集群|逐字航行

规划

IP	操作系统	角色	cpu	mem
192.168.0.241	rocky9.4	FE+BE+jdk	4	6G
192.168.0.242	rocky9.4	FE+BE+jdk	4	6G
192.168.0.244	rocky9.4	FE+BE+jdk	4	6G

部署

环境检查

最小化安装后，初始化命令如下：

systemctl disable --now firewalld

sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

echo 'b104Ak9YsLgB8vekfeDd' | passwd root --stdin 

timedatectl set-timezone Asia/Shanghai

dnf install vim wget zip unzip net-tools chrony -y

sed -i '3,6d' /etc/chrony.conf 

sed -i '2a pool time1.aliyun.com' /etc/chrony.conf 

systemctl enable --now chronyd
reboot

通过starrocks 环境检查脚本进行检查，检查后结果：


############################ CPU检查 #############################
success

########################## Linux版本检查 ##########################
若无特殊原因，建议您更换使用CentOS7部署StarRocks，该系统当前测试最为充分

########################## Glibc版本检查 ##########################
success

############################ Swap检查 ############################
检查项1：使用swap分区可能影响查询性能，建议配置为不优先使用，临时配置命令：echo 0 | sudo tee /proc/sys/vm/swappiness
success

########################### 内核参数检查 ##########################
检查项1：推荐调整overcommit_memory=1，以允许内核分配所有的物理内存来保障程序稳定性，临时调整命令：echo 1 | sudo tee /proc/sys/vm/overcommit_memory
检查项2：推荐调整max_map_count=262144，来调大进程可拥有的内存映射区域的最大数量以保障程序稳定性，临时调整命令：echo 262144 | sudo tee /proc/sys/vm/max_map_count

####################### 进程最大打开文件数检查 ######################
句柄数限制过小可能导致服务异常退出，完整调整命令：ulimit -n 655350 && echo -e '* soft nofile 655350\n* hard nofile 655350' >> /etc/security/limits.conf

####################### 用户最大可用进程数检查 ######################
进程数限制过小可能导致服务异常退出，完整调整命令：ulimit -u 655350 && echo -e '* soft nproc 655350\n* hard nproc 655350' >> /etc/security/limits.conf && sed -i 's/4096/655350/' /etc/security/limits.d/20-nproc.conf

########################### FE端口检查 ###########################
success

########################### BE及CN端口检查 ###########################
success

######################### Broker端口检查 #########################
success

########################### 防火墙检查 ###########################
success

########################## TCP参数检查 ###########################
推荐调整tcp_abort_on_overflow参数值为1，临时调整命令：echo 1 | sudo tee /proc/sys/net/ipv4/tcp_abort_on_overflow

######################### Somaxconn检查 #########################
success

########################## SELinux检查 ###########################
success

########################## Hugepage检查 ##########################
检查项1：推荐禁用透明大页，临时禁用命令：echo never > /sys/kernel/mm/transparent_hugepage/enabled
检查项2：推荐禁用碎片整理，临时禁用命令：echo never > /sys/kernel/mm/transparent_hugepage/defrag

########################## 时钟同步检查 ##########################
未检测到ntp命令，StarRocks各FE节点间的时钟差大于5秒将无法启动，建议在部署前使用ntp对各节点进行时钟同步

############################ 时区检查 ############################
success

########################## 磁盘容量检查 ##########################
success

########################## 内存大小检查 ##########################
服务器内存较小，为保证集群性能和稳定性，生产环境的建议内存为32G+

######################### Netstat命令检查 ########################
success

需要关注Swap、内核参数、用户最大可用进程数、TCP参数、Hugepage的检查提示，时钟同步通过chronyd配置，内存和操作系统无法修改。进行如下修改：

# swap处理
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

# 内核参数处理
echo 'vm.overcommit_memory=1' >> /etc/sysctl.conf
echo 'vm.max_map_count=262144' >> /etc/sysctl.conf

# 最大打开文件数处理
echo -e '* soft nofile 655350\n* hard nofile 655350' >> /etc/security/limits.conf

# 最大可用进程数处理
echo -e '* soft nproc 655350\n* hard nproc 655350' >> /etc/security/limits.conf

# 修改TCP参数
echo 'net.ipv4.tcp_abort_on_overflow=1' >> /etc/sysctl.conf

# 修改Hugepage
cat >> /etc/rc.local <<EOF
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
EOF
chmod u+x /etc/rc.d/rc.local

# 重启生效
sysctl -p
shutdown -r now

重启后再次检查，没有问题再进行安装

安装

环境准备

dnf install java-11-openjdk -y
mkdir /data/starrocks/{meta,data} -p

安装包准备

cd /usr/local/src && wget https://releases.starrocks.io/starrocks/StarRocks-3.3.7-centos-amd64.tar.gz
tar xzvf StarRocks-3.3.7-centos-amd64.tar.gz
# 文件层级如下：StarRocks-3.3.7-centos-amd64
├── LICENSE.txt
├── NOTICE.txt
├── apache_hdfs_broker
│   ├── bin
│   ├── conf
│   └── lib
├── be
│   ├── bin
│   ├── conf
│   ├── datadog
│   ├── lib
│   └── www
└── fe
    ├── arthas
    ├── bin
    ├── conf
    ├── datadog
    ├── hive-udf
    ├── lib
    ├── spark-dpp
    └── webroot

FE安装

mv StarRocks-3.3.7-centos-amd64 /usr/local/starrocks
cd /usr/local/starrocks
vim fe/conf/fe.conf
# 添加内容：
meta_dir = /data/starrocks/meta
# 每台都需要修改
priority_networks = 192.168.0.241/24
# 修改JAVA_OPTS配置，我这里内存只有6个G，但默认Xmx为8192，调整为2048

启动fe实例：

# 192.168.0.241节点执行
./fe/bin/start_fe.sh --daemon
# 192.168.0.242节点执行, 第一次启动follower节点需要添加--helper参数，后面就不需要了
./fe/bin/start_fe.sh --helper 192.168.0.241:9010 --daemon
# 192.168.0.244节点执行, 第一次启动follower节点需要添加--helper参数，后面就不需要了
./fe/bin/start_fe.sh --helper 192.168.0.241:9010 --daemon

通过mysql命令客户端连接主节点进行FE集群配置，默认无密码

mysql> show proc '/frontends';
+----------------------------------+---------------+-------------+----------+-----------+---------+--------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+
| Name                             | IP            | EditLogPort | HttpPort | QueryPort | RpcPort | Role   | ClusterId | Join | Alive | ReplayedJournalId | LastHeartbeat       | IsHelper | ErrMsg | StartTime           | Version       |
+----------------------------------+---------------+-------------+----------+-----------+---------+--------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+
| 192.168.0.241_9010_1733132642680 | 192.168.0.241 | 9010        | 8030     | 9030      | 9020    | LEADER | 175412200 | true | true  | 151               | 2024-12-02 17:52:24 | true     |        | 2024-12-02 17:44:31 | 3.3.7-00177de |
+----------------------------------+---------------+-------------+----------+-----------+---------+--------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+
1 row in set (0.02 sec)

mysql> alter system add follower "192.168.0.242:9010";
Query OK, 0 rows affected (0.02 sec)

mysql> alter system add follower "192.168.0.244:9010";
Query OK, 0 rows affected (0.02 sec)

mysql> show proc '/frontends';
+----------------------------------+---------------+-------------+----------+-----------+---------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+
| Name                             | IP            | EditLogPort | HttpPort | QueryPort | RpcPort | Role     | ClusterId | Join | Alive | ReplayedJournalId | LastHeartbeat       | IsHelper | ErrMsg | StartTime           | Version       |
+----------------------------------+---------------+-------------+----------+-----------+---------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+
| 192.168.0.241_9010_1733132642680 | 192.168.0.241 | 9010        | 8030     | 9030      | 9020    | LEADER   | 175412200 | true | true  | 186               | 2024-12-02 17:54:14 | true     |        | 2024-12-02 17:44:31 | 3.3.7-00177de |
| 192.168.0.244_9010_1733133225323 | 192.168.0.244 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | 175412200 | true | true  | 185               | 2024-12-02 17:54:14 | true     |        | 2024-12-02 17:54:11 | 3.3.7-00177de |
| 192.168.0.242_9010_1733133222032 | 192.168.0.242 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | 175412200 | true | true  | 185               | 2024-12-02 17:54:14 | true     |        | 2024-12-02 17:54:09 | 3.3.7-00177de |
+----------------------------------+---------------+-------------+----------+-----------+---------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+
3 rows in set (0.03 sec)

BE安装

vim be/conf/be.conf
# 新增如下内容
storage_root_path = /data/starrocks/data
# 每台都需要修改
priority_networks = 192.168.0.241/24

启动be服务

./be/bin/start_be.sh --daemon
# 查看be日志
tail -f be/log/be.INFO

be节点添加到集群：

# mysql客户端连接到leader节点，添加到集群
mysql> ALTER SYSTEM ADD BACKEND "192.168.0.241:9050","192.168.0.242:9050","192.168.0.244:9050";
Query OK, 0 rows affected (0.02 sec)

mysql> SHOW PROC '/backends'\G;
*************************** 1. row ***************************
            BackendId: 10002
                   IP: 192.168.0.241
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2024-12-02 18:13:49
        LastHeartbeat: 2024-12-02 18:14:29
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 61
     DataUsedCapacity: 0.000 B
        AvailCapacity: 89.704 GB
        TotalCapacity: 98.957 GB
              UsedPct: 9.35 %
       MaxDiskUsedPct: 9.35 %
               ErrMsg: 
              Version: 3.3.7-00177de
               Status: {"lastSuccessReportTabletsTime":"2024-12-02 18:13:49"}
    DataTotalCapacity: 89.704 GB
          DataUsedPct: 0.00 %
             CpuCores: 4
             MemLimit: 4.687GB
    NumRunningQueries: 0
           MemUsedPct: 3.03 %
           CpuUsedPct: 0.0 %
     DataCacheMetrics: Status: Normal, DiskUsage: 0B/0B, MemUsage: 0B/0B
             Location: 
*************************** 2. row ***************************
            BackendId: 10003
                   IP: 192.168.0.242
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2024-12-02 18:13:49
        LastHeartbeat: 2024-12-02 18:14:29
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 62
     DataUsedCapacity: 0.000 B
        AvailCapacity: 89.765 GB
        TotalCapacity: 98.957 GB
              UsedPct: 9.29 %
       MaxDiskUsedPct: 9.29 %
               ErrMsg: 
              Version: 3.3.7-00177de
               Status: {"lastSuccessReportTabletsTime":"2024-12-02 18:13:49"}
    DataTotalCapacity: 89.765 GB
          DataUsedPct: 0.00 %
             CpuCores: 4
             MemLimit: 4.687GB
    NumRunningQueries: 0
           MemUsedPct: 3.05 %
           CpuUsedPct: 0.2 %
     DataCacheMetrics: Status: Normal, DiskUsage: 0B/0B, MemUsage: 0B/0B
             Location: 
*************************** 3. row ***************************
            BackendId: 10004
                   IP: 192.168.0.244
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2024-12-02 18:13:49
        LastHeartbeat: 2024-12-02 18:14:29
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 62
     DataUsedCapacity: 0.000 B
        AvailCapacity: 89.764 GB
        TotalCapacity: 98.957 GB
              UsedPct: 9.29 %
       MaxDiskUsedPct: 9.29 %
               ErrMsg: 
              Version: 3.3.7-00177de
               Status: {"lastSuccessReportTabletsTime":"2024-12-02 18:13:50"}
    DataTotalCapacity: 89.764 GB
          DataUsedPct: 0.00 %
             CpuCores: 4
             MemLimit: 4.687GB
    NumRunningQueries: 0
           MemUsedPct: 3.06 %
           CpuUsedPct: 0.2 %
     DataCacheMetrics: Status: Normal, DiskUsage: 0B/0B, MemUsage: 0B/0B
             Location: 
3 rows in set (0.00 sec)

# 如果字段Alive为true，说明be节点正常启动并加入集群