{
    "componentChunkName": "component---src-templates-blog-blog-detail-tsx",
    "path": "/blog/chaos-mesh-with-greatdb",
    "result": {"pageContext":{"blog":{"id":"Blogs_369","title":"Chaos Mesh 实战分享丨通过混沌工程验证 GreatDB 分布式部署模式的稳定性","tags":["Chaos Mesh"],"category":{"name":"案例实践"},"summary":"Chaos Mesh 最初作为开源分布式数据库 TiDB 的测试平台而创建，是一个多功能混沌工程平台，通过混沌测试验证分布式系统的稳定性。本文以万里安全数据库软件 GreatDB 分布式部署模式为例，介绍了通过 Chaos Mesh 进行混沌测试的全流程。 ","body":"Chaos Mesh 最初作为开源分布式数据库 TiDB 的测试平台而创建，是一个多功能混沌工程平台，通过混沌测试验证分布式系统的稳定性。本文以万里安全数据库软件 GreatDB 分布式部署模式为例，介绍了通过 Chaos Mesh 进行混沌测试的全流程。 \n\n## 需求背景与 GreatDB 介绍\n\n### 需求背景\n\n混沌测试是检测分布式系统不确定性、建立系统弹性信心的一种非常好的方式，因此我们采用开源工具 Chaos Mesh 来做 GreatDB 分布式集群的混沌测试。\n\n### GreatDB 分布式部署模式介绍\n\n万里安全数据库软件 GreatDB 是一款关系型数据库软件，同时支持集中式和分布式的部署方式，本文涉及的是分布式部署方式。\n\n分布式部署模式采用 shared-nothing 架构；通过数据冗余与副本管理确保数据库无单点故障；数据 sharding 与分布式并行计算实现数据库系统高性能；可无限制动态扩展数据节点，满足业务需要。\n\n整体架构如下图所示：\n\n![1.jpg](https://img1.www.pingcap.com/prod/1_a164021e9d.jpg)\n\n\n## 环境准备\n\n### Chaos Mesh 安装\n\n在安装 Chaos Mesh 之前请确保已经预先安装了 helm，docker，并准备好了一个 kubernetes 环境。\n\n-  使用 Helm 安装 \n\n1）在 Helm 仓库中添加 Chaos Mesh 仓库：\n\n```shell\nhelm repo add chaos-mesh https://charts.chaos-mesh.org\n```\n\n\n\n2）查看可以安装的 Chaos Mesh 版本：\n\n```shell\nhelm search repo chaos-mesh\n```\n\n\n\n3）创建安装 Chaos Mesh 的命名空间：\n\n```shell\nkubectl create ns chaos-testing\n```\n\n\n\n4）在 docker 环境下安装 Chaos Mesh：\n\n```shell\nhelm install chaos-mesh chaos-mesh/chaos-mesh -n=chaos-testing\n```\n\n\n\n-  验证安装 \n\n执行以下命令查看 Chaos Mesh 的运行情况：\n\n`kubectl get pod -n chaos-testing`\n\n下面是预期输出：\n\n```shell\nNAME                                       READY   STATUS    RESTARTS   AGE\n\nchaos-controller-manager-d7bc9ccb5-dbccq   1/1     Running   0          26d\n\nchaos-daemon-pzxc7                         1/1     Running   0          26d\n\nchaos-dashboard-5887f7559b-kgz46           1/1     Running   1          26d\n```\n\n\n\n如果 3 个 pod 的状态都是 Running，表示 Chaos Mesh 已经成功安装。\n\n### 准备测试需要的镜像\n\n#### 准备 MySQL 镜像\n\n一般情况下，MySQL 使用官方 5.7 版本的镜像，MySQL 监控采集器使用的是 mysqld-exporter，也可以直接从 docker hub 下载：\n```shell\ndocker pull mysql:5.7\n\ndocker pull prom/mysqld-exporter\n```\n\n#### 准备 ZooKeeper 镜像\n\nZooKeeper 使用的是官方 3.5.5 版本镜像，ZooKeeper 组件涉及的监控有 jmx-prometheus-exporter 和 zookeeper-exporter，均从 docker hub 下载：\n```shell\ndocker pull zookeeper:3.5.5\n\ndocker pull sscaling/jmx-prometheus-exporter\n\ndocker pull josdotso/zookeeper-exporter\n\n```\n\n#### 准备 GreatDB 镜像\n\n选择一个 GreatDB 的 tar 包，将其解压得到一个 ./greatdb 目录，再将 greatdb-service-docker.sh 文件拷贝到这个解压出来的./greatdb 目录里：\n\n```shell\ncp greatdb-service-docker.sh ./greatdb/\n```\n\n\n\n将 greatdb Dockerfile 放到./greatdb 文件夹的同级目录下，然后执行以下命令构建 GreatDB 镜像：\n```shell\ndocker build -t greatdb/greatdb:tag2021 .\n\n```\n\n#### 准备 GreatDB 分布式集群部署/清理的镜像\n\n下载集群部署脚本 cluster-setup，集群初始化脚本 init-zk 以及集群 helm charts 包（可咨询 4.0 开发/测试组获取）\n\n将上述材料放在同一目录下，编写如下 Dockerfile:\n```dockerfile\nFROM debian:buster-slim as init-zk\n\n\n\nCOPY ./init-zk /root/init-zk\n\nRUN chmod +x /root/init-zk\n\n\n\nFROM debian:buster-slim as cluster-setup\n\n\\*# Set aliyun repo for speed*\n\nRUN sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list && \\\n\n  sed -i 's/security.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list\n\n\n\nRUN apt-get -y update && \\\n\n  apt-get -y install \\\n\n  curl \\\n\n  wget\n\n\n\nRUN curl -L https://storage.googleapis.com/kubernetes-release/release/v1.20.1/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && \\\n\n  chmod +x /usr/local/bin/kubectl && \\\n\n  mkdir /root/.kube && \\\n\n  wget https://get.helm.sh/helm-v3.5.3-linux-amd64.tar.gz && \\\n\n  tar -zxvf helm-v3.5.3-linux-amd64.tar.gz && \\\n\n  mv linux-amd64/helm /usr/local/bin/helm\n\n\n\nCOPY ./config /root/.kube/\n\nCOPY ./helm /helm\n\nCOPY ./cluster-setup /\n\n```\n\n执行以下命令构建所需镜像：\n\n```shell\ndocker build --target init-zk -t greatdb/initzk:latest .\n\n\ndocker build --target cluster-setup -t greatdb/cluster-setup:v1 .\n```\n\n\n#### 准备测试用例的镜像\n\n目前测试支持的用例有：bank、bank2、pbank、tpcc、flashback 等，每个用例都是一个可执行文件。\n\n以 flashback 测例为例构建测试用例的镜像，先将用例下载到本地，在用例的同一目录下编写如下内容的 Dockerfile：\n\n\n```dockerfile\nFROM debian:buster-slim\n\nCOPY ./flashback /\n\nRUN cd / && chmod +x ./flashback\n```\n\n\n执行以下命令构建测试用例镜像：\n```shell\ndocker build -t greatdb/testsuite-flashback:v1 .\n```\n\n\n### 将准备好的镜像上传到私有仓库中\n\n创建私有仓库和上传镜像操作请参考：<https://zhuanlan.zhihu.com/p/78543733>\n\n## Chaos Mesh 的使用\n\n### 搭建 GreatDB 分布式集群\n\n在上一章中 cluster-setup 目录下执行以下命令块去搭建测试集群：\n```shell\n./cluster-setup  \\\n\n-clustername=c0 \\\n\n-namespace=test \\\n\n-enable-monitor=true \\\n\n-mysql-image=mysql:5.7 \\\n\n-mysql-replica=3 \\\n\n-mysql-auth=1 \\\n\n-mysql-normal=1 \\\n\n-mysql-global=1 \\\n\n-mysql-partition=1 \\\n\n-zookeeper-repository=zookeeper \\\n\n-zookeeper-tag=3.5.5 \\\n\n-zookeeper-replica=3 \\\n\n-greatdb-repository=greatdb/greatdb \\\n\n-greatdb-tag=tag202110 \\\n\n-greatdb-replica=3 \\\n\n-greatdb-serviceHost=172.16.70.249\n```\n\n\n输出信息：\n```shell\nliuxinle@liuxinle-OptiPlex-5060:~/k8s/cluster-setup$ ./cluster-setup \\\n\n\\> -clustername=c0 \\\n\n\\> -namespace=test \\\n\n\\> -enable-monitor=true \\\n\n\\> -mysql-image=mysql:5.7 \\\n\n\\> -mysql-replica=3 \\\n\n\\> -mysql-auth=1 \\\n\n\\> -mysql-normal=1 \\\n\n\\> -mysql-global=1 \\\n\n\\> -mysql-partition=1 \\\n\n\\> -zookeeper-repository=zookeeper \\\n\n\\> -zookeeper-tag=3.5.5 \\\n\n\\> -zookeeper-replica=3 \\\n\n\\> -greatdb-repository=greatdb/greatdb \\\n\n\\> -greatdb-tag=tag202110 \\\n\n\\> -greatdb-replica=3 \\\n\n\\> -greatdb-serviceHost=172.16.70.249\n\nINFO[2021-10-14T10:41:52+08:00] SetUp the cluster ...                         NameSpace=test\n\nINFO[2021-10-14T10:41:52+08:00] create namespace ...                         \n\nINFO[2021-10-14T10:41:57+08:00] copy helm chart templates ...                \n\nINFO[2021-10-14T10:41:57+08:00] setup ...                                     Component=MySQL\n\nINFO[2021-10-14T10:41:57+08:00] exec helm install and update greatdb-cfg.yaml ... \n\nINFO[2021-10-14T10:42:00+08:00] waiting mysql pods running ...               \n\nINFO[2021-10-14T10:44:27+08:00] setup ...                                     Component=Zookeeper\n\nINFO[2021-10-14T10:44:28+08:00] waiting zookeeper pods running ...           \n\nINFO[2021-10-14T10:46:59+08:00] update greatdb-cfg.yaml                      \n\nINFO[2021-10-14T10:46:59+08:00] setup ...                                     Component=greatdb\n\nINFO[2021-10-14T10:47:00+08:00] waiting greatdb pods running ...             \n\nINFO[2021-10-14T10:47:21+08:00] waiting cluster running ...                  \n\nINFO[2021-10-14T10:47:27+08:00] waiting prometheus server running...         \n\nINFO[2021-10-14T10:47:27+08:00] Dump Cluster Info                            \n\nINFO[2021-10-14T10:47:27+08:00] SetUp success.                                ClusterName=c0 NameSpace=test\n\n```\n\n执行如下命令，查看集群 pod 状态：\n```shell\nkubectl get pod -n test -o wide\n```\n\n\n输出信息：\n```shell\nNAME                                    READY   STATUS      RESTARTS   AGE     IP             NODE                     NOMINATED NODE   READINESS GATES\n\nc0-auth0-mysql-0                        2/2     Running     0          10m     10.244.87.18   liuxinle-optiplex-5060   <none>           <none>\n\nc0-auth0-mysql-1                        2/2     Running     0          9m23s   10.244.87.54   liuxinle-optiplex-5060   <none>           <none>\n\nc0-auth0-mysql-2                        2/2     Running     0          8m39s   10.244.87.57   liuxinle-optiplex-5060   <none>           <none>\n\nc0-greatdb-0                            2/2     Running     1          5m3s    10.244.87.58   liuxinle-optiplex-5060   <none>           <none>\n\nc0-greatdb-1                            2/2     Running     0          4m57s   10.244.87.20   liuxinle-optiplex-5060   <none>           <none>\n\nc0-greatdb-2                            2/2     Running     0          4m50s   10.244.87.47   liuxinle-optiplex-5060   <none>           <none>\n\nc0-glob0-mysql-0                        2/2     Running     0          10m     10.244.87.51   liuxinle-optiplex-5060   <none>           <none>\n\nc0-glob0-mysql-1                        2/2     Running     0          9m23s   10.244.87.41   liuxinle-optiplex-5060   <none>           <none>\n\nc0-glob0-mysql-2                        2/2     Running     0          8m38s   10.244.87.60   liuxinle-optiplex-5060   <none>           <none>\n\nc0-nor0-mysql-0                         2/2     Running     0          10m     10.244.87.29   liuxinle-optiplex-5060   <none>           <none>\n\nc0-nor0-mysql-1                         2/2     Running     0          9m29s   10.244.87.4    liuxinle-optiplex-5060   <none>           <none>\n\nc0-nor0-mysql-2                         2/2     Running     0          8m45s   10.244.87.25   liuxinle-optiplex-5060   <none>           <none>\n\nc0-par0-mysql-0                         2/2     Running     0          10m     10.244.87.55   liuxinle-optiplex-5060   <none>           <none>\n\nc0-par0-mysql-1                         2/2     Running     0          9m26s   10.244.87.13   liuxinle-optiplex-5060   <none>           <none>\n\nc0-par0-mysql-2                         2/2     Running     0          8m42s   10.244.87.21   liuxinle-optiplex-5060   <none>           <none>\n\nc0-prometheus-server-6697649b76-fkvh9   2/2     Running     0          4m36s   10.244.87.37   liuxinle-optiplex-5060   <none>           <none>\n\nc0-zookeeper-0                          1/1     Running     1          7m35s   10.244.87.44   liuxinle-optiplex-5060   <none>           <none>\n\nc0-zookeeper-1                          1/1     Running     0          6m41s   10.244.87.30   liuxinle-optiplex-5060   <none>           <none>\n\nc0-zookeeper-2                          1/1     Running     0          6m10s   10.244.87.49   liuxinle-optiplex-5060   <none>           <none>\n\nc0-zookeeper-initzk-7hbfs               0/1     Completed   0          7m35s   10.244.87.17   liuxinle-optiplex-5060   <none>           <none>\n```\n\n\n看到 c0-zookeeper-initzk-7hbfs 的状态是 Completed，其他 pod 的状态为 Running，表示集群搭建成功。\n\n### 在 GreatDB 分布式集群中使用 Chaos Mesh 做混沌测试\n\nChaos Mesh 在 kubernetes 环境支持注入的故障类型包括：模拟 Pod 故障、模拟网络故障、模拟压力场景等，这里我们以模拟 Pod 故障中的 pod-kill 为例。\n\n将实验配置写入到文件中 pod-kill.yaml，内容示例如下：\n```yaml\napiVersion: chaos-mesh.org/v1alpha1\n\nkind: PodChaos   *# 要注入的故障类型*\n\nmetadata:\n\n  name: pod-failure-example\n\n  namespace: test   *# 测试集群pod所在的namespace*\n\nspec:\n\n  action: pod-kill   *# 要注入的具体故障类型*\n\n  mode: all    *# 指定实验的运行方式，all（表示选出所有符合条件的 Pod）*\n\n  duration: '30s'    *# 指定实验的持续时间* \n\n  selector: \n\n    labelSelectors:\n\n      \"app.kubernetes.io/component\": \"greatdb\"    *# 指定注入故障目标pod的标签，通过kubectl describe pod c0-greatdb-1 -n test 命令返回结果中Labels后的内容得到*\n```\n\n\n创建故障实验，命令如下：\n```shell\nkubectl create -n test -f pod-kill.yaml\n```\n\n\n创建完故障实验之后，执行命令 kubectl get pod -n test -o wide 结果如下：\n```shell\nNAME                                    READY   STATUS              RESTARTS   AGE     IP             NODE                     NOMINATED NODE   READINESS GATES\n\nc0-auth0-mysql-0                        2/2     Running             0          14m     10.244.87.18   liuxinle-optiplex-5060   <none>           <none>\n\nc0-auth0-mysql-1                        2/2     Running             0          14m     10.244.87.54   liuxinle-optiplex-5060   <none>           <none>\n\nc0-auth0-mysql-2                        2/2     Running             0          13m     10.244.87.57   liuxinle-optiplex-5060   <none>           <none>\n\nc0-greatdb-0                            0/2     ContainerCreating   0          2s      <none>         liuxinle-optiplex-5060   <none>           <none>\n\nc0-greatdb-1                            0/2     ContainerCreating   0          2s      <none>         liuxinle-optiplex-5060   <none>           <none>\n\nc0-glob0-mysql-0                        2/2     Running             0          14m     10.244.87.51   liuxinle-optiplex-5060   <none>           <none>\n\nc0-glob0-mysql-1                        2/2     Running             0          14m     10.244.87.41   liuxinle-optiplex-5060   <none>           <none>\n\nc0-glob0-mysql-2                        2/2     Running             0          13m     10.244.87.60   liuxinle-optiplex-5060   <none>           <none>\n\nc0-nor0-mysql-0                         2/2     Running             0          14m     10.244.87.29   liuxinle-optiplex-5060   <none>           <none>\n\nc0-nor0-mysql-1                         2/2     Running             0          14m     10.244.87.4    liuxinle-optiplex-5060   <none>           <none>\n\nc0-nor0-mysql-2                         2/2     Running             0          13m     10.244.87.25   liuxinle-optiplex-5060   <none>           <none>\n\nc0-par0-mysql-0                         2/2     Running             0          14m     10.244.87.55   liuxinle-optiplex-5060   <none>           <none>\n\nc0-par0-mysql-1                         2/2     Running             0          14m     10.244.87.13   liuxinle-optiplex-5060   <none>           <none>\n\nc0-par0-mysql-2                         2/2     Running             0          13m     10.244.87.21   liuxinle-optiplex-5060   <none>           <none>\n\nc0-prometheus-server-6697649b76-fkvh9   2/2     Running             0          9m24s   10.244.87.37   liuxinle-optiplex-5060   <none>           <none>\n\nc0-zookeeper-0                          1/1     Running             1          12m     10.244.87.44   liuxinle-optiplex-5060   <none>           <none>\n\nc0-zookeeper-1                          1/1     Running             0          11m     10.244.87.30   liuxinle-optiplex-5060   <none>           <none>\n\nc0-zookeeper-2                          1/1     Running             0          10m     10.244.87.49   liuxinle-optiplex-5060   <none>           <none>\n\nc0-zookeeper-initzk-7hbfs               0/1     Completed           0          12m     10.244.87.17   liuxinle-optiplex-5060   <none>           <none>\n```\n\n\n可以看到有带 greatdb 名字的 pod 正在被重启，说明注入故障成功。\n\n## 在 Argo 中编排测试流程\n\nArgo 是一个开源的容器本地工作流引擎，用于在 Kubernetes 上完成工作，可以将多步骤工作流建模为一系列任务，完成测试流程编排。\n\n我们使用 argo 定义一个测试任务，基本的测试流程是固定的，如下所示：\n\n![2.jpg](https://img1.www.pingcap.com/prod/2_309bb781d9.jpg)\n\n测试流程的 step1 是部署测试集群，接着开启两个并行任务，step2 跑测试用例，模拟业务场景，step3 同时使用 Chaos Mesh 注入故障，step2 的测试用例执行结束之后，step4 终止故障注入，最后 step5 清理集群环境。\n\n### 用 Argo 编排一个混沌测试工作流（以 flashback 测试用例为例)\n\n1）修改 cluster-setup.yaml 中的 image 信息，改成步骤“准备测试需要的镜像”中自己传上去的集群部署/清理镜像名和 tag\n\n2）修改 testsuite-flashback.yaml 中的 image 信息，改成步骤“准备测试需要的镜像”中自己传上去的测试用例镜像名和 tag\n\n3）将集群部署、测试用例和工具模板的 yaml 文件全部使用 kubectl apply -n argo -f xxx.yaml 命令创建资源 （这些文件定义了一些 argo template，方便用户写 workflow 时候使用）\n\n```shell\nkubectl apply -n argo -f cluster-setup.yaml\n\nkubectl apply -n argo -f testsuite-flashback.yaml\n\nkubectl apply -n argo -f tools-template.yaml\n```\n\n\n4）复制一份 workflow 模板文件 workflow-template.yaml，将模板文件中注释提示的部分修改为自己的设置即可，然后执行以下命令创建混沌测试工作流：\n\n```shell\nkubectl apply -n argo -f workflow-template.yaml\n```\n\n以下是一份 workflow 模板文件：\n```yaml\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n  generateName: chaostest-c0-0-\n  name: chaostest-c0-0\n  namespace: argo\nspec:\n  entrypoint: test-entry #测试入口，在这里传入测试参数，填写clustername、namespace、host、greatdb镜像名和tag名等基本信息\n  serviceAccountName: argo\n  arguments:\n    parameters:\n      - name: clustername\n        value: c0\n      - name: namespace\n        value: test\n      - name: host\n        value: 172.16.70.249\n      - name: port\n        value: 30901\n      - name: password\n        value: Bgview@2020\n      - name: user\n        value: root\n      - name: run-time\n        value: 10m\n      - name: greatdb-repository\n        value: greatdb/greatdb\n      - name: greatdb-tag\n        value: tag202110\n      - name: nemesis\n        value: kill_mysql_normal_master,kill_mysql_normal_slave,kill_mysql_partition_master,kill_mysql_partition_slave,kill_mysql_auth_master,kill_mysql_auth_slave,kill_mysql_global_master,kill_mysql_global_slave,kill_mysql_master,kill_mysql_slave,net_partition_mysql_normal,net_partition_mysql_partition,net_partition_mysql_auth,net_partition_mysql_global\n      - name: mysql-partition\n        value: 1\n      - name: mysql-global\n        value: 1\n      - name: mysql-auth\n        value: 1\n      - name: mysql-normal\n        value: 2\n  templates:\n    - name: test-entry\n      steps:\n        - - name: setup-greatdb-cluster  # step.1 集群部署. 请指定正确的参数，主要是mysql和zookeeper的镜像名、tag名\n            templateRef:\n              name: cluster-setup-template\n              template: cluster-setup\n            arguments:\n              parameters:\n                - name: namespace\n                  value: \"{{workflow.parameters.namespace}}\"\n                - name: clustername\n                  value: \"{{workflow.parameters.clustername}}\"\n                - name: mysql-image\n                  value: mysql:5.7.34\n                - name: mysql-replica\n                  value: 3\n                - name: mysql-auth\n                  value: \"{{workflow.parameters.mysql-auth}}\"\n                - name: mysql-normal\n                  value: \"{{workflow.parameters.mysql-normal}}\"\n                - name: mysql-partition\n                  value: \"{{workflow.parameters.mysql-partition}}\"\n                - name: mysql-global\n                  value: \"{{workflow.parameters.mysql-global}}\"\n                - name: enable-monitor\n                  value: false\n                - name: zookeeper-repository\n                  value: zookeeper\n                - name: zookeeper-tag\n                  value: 3.5.5\n                - name: zookeeper-replica\n                  value: 3\n                - name: greatdb-repository\n                  value: \"{{workflow.parameters.greatdb-repository}}\"\n                - name: greatdb-tag\n                  value: \"{{workflow.parameters.greatdb-tag}}\"\n                - name: greatdb-replica\n                  value: 3\n                - name: greatdb-serviceHost\n                  value: \"{{workflow.parameters.host}}\"\n                - name: greatdb-servicePort\n                  value: \"{{workflow.parameters.port}}\"\n        - - name: run-flashbacktest    # step.2 运行测试用例,请替换为你要运行的测试用例template并指定正确的参数，主要是测试使用的表个数和大小\n            templateRef:\n              name: flashback-test-template\n              template: flashback\n            arguments:\n              parameters:\n                - name: user\n                  value: \"{{workflow.parameters.user}}\"\n                - name: password\n                  value: \"{{workflow.parameters.password}}\"\n                - name: host\n                  value: \"{{workflow.parameters.host}}\"\n                - name: port\n                  value: \"{{workflow.parameters.port}}\"\n                - name: concurrency\n                  value: 16\n                - name: size\n                  value: 10000\n                - name: tables\n                  value: 10\n                - name: run-time\n                  value: \"{{workflow.parameters.run-time}}\"\n                - name: single-statement\n                  value: true\n                - name: manage-statement\n                  value: true\n          - name: invoke-chaos-for-flashabck-test    # step.3 注入故障，请指定正确的参数，这里run-time和interval分别定义了故障注入的时间和频次，因此省略掉了终止故障注入步骤\n            templateRef:\n              name: chaos-rto-template\n              template: chaos-rto\n            arguments:\n              parameters:\n                - name: user\n                  value: \"{{workflow.parameters.user}}\"\n                - name: host\n                  value: \"{{workflow.parameters.host}}\"\n                - name: password\n                  value: \"{{workflow.parameters.password}}\"\n                - name: port\n                  value: \"{{workflow.parameters.port}}\"\n                - name: k8s-config\n                  value: /root/.kube/config\n                - name: namespace\n                  value: \"{{workflow.parameters.namespace}}\"\n                - name: clustername\n                  value: \"{{workflow.parameters.clustername}}\"\n                - name: prometheus\n                  value: ''\n                - name: greatdb-job\n                  value: greatdb-monitor-greatdb\n                - name: nemesis\n                  value: \"{{workflow.parameters.nemesis}}\"\n                - name: nemesis-duration\n                  value: 1m\n                - name: nemesis-mode\n                  value: default\n                - name: wait-time\n                  value: 5m\n                - name: check-time\n                  value: 5m\n                - name: nemesis-scope\n                  value: 1\n                - name: nemesis-log\n                  value: true\n                - name: enable-monitor\n                  value: false\n                - name: run-time\n                  value: \"{{workflow.parameters.run-time}}\"\n                - name: interval\n                  value: 1m\n                - name: monitor-log\n                  value: false\n                - name: enable-rto\n                  value: false\n                - name: rto-qps\n                  value: 0.1\n                - name: rto-warm\n                  value: 5m\n                - name: rto-time\n                  value: 1m\n                - name: log-level\n                  value: debug\n        - - name: flashbacktest-output         # 输出测试用例是否通过的结果\n            templateRef:\n              name: tools-template\n              template: output-result\n            arguments:\n              parameters:\n                - name: info\n                  value: \"flashback test pass, with nemesis: {{workflow.parameters.nemesis}}\"\n        - - name: clean-greatdb-cluster           # step.4 清理测试集群，这里的参数和step.1的参数一致\n            templateRef:\n              name: cluster-setup-template\n              template: cluster-setup\n            arguments:\n              parameters:\n                - name: namespace\n                  value: \"{{workflow.parameters.namespace}}\"\n                - name: clustername\n                  value: \"{{workflow.parameters.clustername}}\"\n                - name: mysql-image\n                  value: mysql:5.7\n                - name: mysql-replica\n                  value: 3\n                - name: mysql-auth\n                  value: \"{{workflow.parameters.mysql-auth}}\"\n                - name: mysql-normal\n                  value: \"{{workflow.parameters.mysql-normal}}\"\n                - name: mysql-partition\n                  value: \"{{workflow.parameters.mysql-partition}}\"\n                - name: mysql-global\n                  value: \"{{workflow.parameters.mysql-global}}\"\n                - name: enable-monitor\n                  value: false\n                - name: zookeeper-repository\n                  value: zookeeper\n                - name: zookeeper-tag\n                  value: 3.5.5\n                - name: zookeeper-replica\n                  value: 3\n                - name: greatdb-repository\n                  value: \"{{workflow.parameters.greatdb-repository}}\"\n                - name: greatdb-tag\n                  value: \"{{workflow.parameters.greatdb-tag}}\"\n                - name: greatdb-replica\n                  value: 3\n                - name: greatdb-serviceHost\n                  value: \"{{workflow.parameters.host}}\"\n                - name: greatdb-servicePort\n                  value: \"{{workflow.parameters.port}}\"\n                - name: clean\n                  value: true\n        - - name: echo-result\n            templateRef:\n              name: tools-template\n              template: echo\n            arguments:\n              parameters:\n                - name: info\n                  value: \"{{item}}\"\n            withItems:\n              - \"{{steps.flashbacktest-output.outputs.parameters.result}}\"\n```\n\n至此，你已经成功使用 Chaos Mesh 进行了一次混沌测试，并成功验证了分布式系统的稳定性。\n\nNow enjoy GreatSQL, and enjoy Chaos Mesh :)\n","date":"2022-03-29","author":"GreatSQL 社区","fillInMethod":"writeDirectly","customUrl":"chaos-mesh-with-greatdb","file":null,"relatedBlogs":[]}}},
    "staticQueryHashes": ["1327623483","1820662718","3081853212","3430003955","3649515864","4265596160","63159454"]}