节点启动与leader选主分析

节点启动过程

node init的过程:

  • new NodeMetrics

  • 创建 timerManager

  • 创建需要的定时器:

    • voteTimer

    • electionTimer

      这两个 timer 的倒计时时间都是根据配置的 (ElectionTimeoutMs, ElectionTimeoutMs + maxElectionDelayMs) 之间进行随机出一个整数作为其倒计时时间。
      这两个timer看起来差不多,一个是voteTimer,一个是electionTimer。为啥? 成为候选人之后再投票? 没成为候选人时先触发选举?应该是对的, 看对应的回调handleVoteTimeout和handleElectionTimeout的处理,一个是检查STATE_FOLLOWER,另一个是STATE_CANDIDATE。 所以按此角色的判定,要electionTimer的回调逻辑先执行到才行

    • stepDownTimer

    • snapshotTimer

  • 创建 applyDisruptor 和 applyQueue

  • 创建 FSMCallerImpl

  • initLogStorage

  • initMetaStorage

  • initFSMCaller

  • 创建BallotBox 并初始化

  • initSnapshotStorage

  • logManager.checkConsistency

  • 创建ReplicatorGroupImpl 并在稍后初始化

  • 创建DefaultRaftClientService

  • 创建ReadOnlyServiceImpl并初始化

  • 节点状态初始化为STATE_FOLLOWER 这个对应leader选举 章节中讲到的节点状态机的初始状态

  • 如果这个group只有自己一个节点,那么触发选举自己

节点角色状态机

在理解投票与选举逻辑之前,先要搞清楚节点的状态变化的状态机,这个内容在论文的5.1章节有图讲到。

stateDiagram-v2
    [*] --> Follower:starts up
    Follower --> Candidate: time out, starts election
    Candidate --> Leader: receives votes from majority of servers
    Candidate --> Candidate: times out, new election
    Leader --> Follower: discovers server with higher term
    Candidate --> Follower: discovers current leader or new term

最简的理想的变换过程是:

Follower – Candidate – Leader

leader选举过程的简易图示

授权成功的解释

授权成功分为两部分逻辑:

  • 远端(即接收到请求的服务端)授权成功
  • 本端(即发起请求的客户端)授权成功

远端授权成功逻辑

分析远端授权其实就是分析服务端对预投票和投票的处理逻辑。

handlePreVoteRequest

主要逻辑有:

  • 一些合法性校验,比如请求发起端是否在配置的 peer 列表中。
  • 请求中的term是否已经小于当前的term,如果是的,那么此次请求授权失败,直接返回响应给发起方。
  • checkReplicator
  • 比较requestLastLogId和此时再次获取到的lastLogId,如果大于等于,则授权成功,返回响应给发起方。

handleRequestVoteRequest

服务端处理投票和预投票的逻辑还是有些不同的,服务端处理投票的主要逻辑有:

  • 一些合法性校验,比如请求发起端是否在配置的 peer 列表中。
  • 如果请求的term小于当前term,则此次请求授权失败,并将当前term返回给发起方。
  • 如果请求的term大于当前term,则要进入stepDown逻辑(TODO),后续逻辑继续执行。
  • 拿request中的lastLogId和当前获取到的lastLogId比较,如果是大于等于的则算是logIsOK
  • 这个时候如果logIsOk,term也是合法(请求term大于等于当前term),且votedId是空的,则处理stepDown逻辑(TODO)且将发起请求的serverid作为votedId,并用metaStorage存储为这个candidateId在投票。
  • 最后返回是否授权成功,此处,是否授权成功,还是再判断一次的,判断规则是:request.getTerm() == this.currTerm && candidateId.equals(this.votedId)

本端授权成功逻辑

分析本端授权主要是分析Ballot的授权逻辑。分析Ballot授权逻辑之前要分析其init逻辑。

Ballot init分析

Ballot的grant之前都会init, 且这两个同时出现执行,出现在preVote和electSelf中,preVote对应的是prevVoteCtx(Ballot的实例),electSelf对应的是voteCtx(Ballot的实例)。

init逻辑

  • init时将迭代conf中所有的peer构建为UnfoundPeerId,并用index按序标志。并将UnfoundPeerId实例添加到Ballot的peers字段中。
  • Ballot的quorum字段初始化为peer个数的一半加1。
  • 如果有oldConf则按上述逻辑同样处理,不过相应的结果都是维护在oldXX字段中。

Ballot grant分析

grant逻辑

  • 如果直接对某个peer grant,则用new 的PosHint,PosHint初始的pos0、pos1都是-1 , -1则意味着按serverid比对迭代查找(逻辑没啥区别,只是不是直接取到)。

  • 如果这个peer没有被found,那么quorm减1。这个peer包括发起预投票或者投票时的发起方自己,也包括应答了这个请求并且授权成功的对端。这个逻辑非常重要。
    详细的过程可以参见下面相关日志做了注释的部分:

    // node1的日志,这个节点接下来会被选为leader
    2021-10-23 22:46:59 [JRaft-ElectionTimer-<election_test/127.0.0.1:8081>0] INFO  NodeImpl:2669 - preVote before grant, serverId 127.0.0.1:8081 ,,, prevVoteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=false, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=false, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=false, index=2]], quorum=2, oldPeers=[], oldQuorum=0]
    2021-10-23 22:46:59 [JRaft-ElectionTimer-<election_test/127.0.0.1:8081>0] INFO  NodeImpl:2670 - preVote after grant, serverId 127.0.0.1:8081 ,,, prevVoteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=true, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=false, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=false, index=2]], quorum=1, oldPeers=[], oldQuorum=0] // 此时quorum 变成1    Ballot grant的是他自己: serverId 127.0.0.1:8081
        
        
        2021-10-23 22:46:59 [JRaft-RPC-Processor-0] INFO  NodeImpl:2577 - Node <election_test/127.0.0.1:8081> received PreVoteResponse from 127.0.0.1:8082, term=0, granted=true.
    2021-10-23 22:46:59 [JRaft-RPC-Processor-0] INFO  NodeImpl:2580 - handlePreVoteResponse before grant, peerId 127.0.0.1:8082 ,,, prevVoteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=true, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=false, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=false, index=2]], quorum=1, oldPeers=[], oldQuorum=0]
    2021-10-23 22:46:59 [JRaft-RPC-Processor-0] INFO  NodeImpl:2581 - handlePreVoteResponse after grant, peerId 127.0.0.1:8082 ,,, prevVoteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=true, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=true, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=false, index=2]], quorum=0, oldPeers=[], oldQuorum=0] // 此时quorum 变成0  那么表示pre vote成功    Ballot grant的是他发送的对端: peerId 127.0.0.1:8082
    //............
    2021-10-23 22:46:59 [JRaft-RPC-Processor-1] INFO  Replicator:897 - Replicator=Replicator [state=Created, statInfo=<running=null, firstLogIndex=0, lastLogIncluded=0, lastLogIndex=0, lastTermIncluded=0>, peerId=127.0.0.1:8082, type=Follower]@127.0.0.1:8082 is started
    //成为leader之后就是一直发空的AppendEntriesRequest请求了    此时per_log_term和pre_log_index都是0
    2021-10-23 22:46:59 [JRaft-RPC-Processor-1] INFO  AbstractClientService:195 - ===invokeWithDone, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$AppendEntriesRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8081"
    peer_id: "127.0.0.1:8082"
    term: 1
    prev_log_term: 0
    prev_log_index: 0
    committed_index: 0
    data: "" 
    //............
    2021-10-23 22:46:59 [JRaft-FSMCaller-Disruptor-0] INFO  StateMachineAdapter:62 - onLeaderStart: term=1.
    [ElectionBootstrap] Leader's ip is: 127.0.0.1, port: 8081
    [ElectionBootstrap] Leader start on term: 1
    2021-10-23 22:46:59 [JRaft-Rpc-Closure-Executor-1] INFO  AbstractClientService:195 - ===invokeWithDone, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$AppendEntriesRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8081"
    peer_id: "127.0.0.1:8082"
    term: 1
    prev_log_term: 1
    prev_log_index: 1
    committed_index: 1
    //............
    2021-10-23 22:47:55 [election_test/PeerPair[127.0.0.1:8081 -> 127.0.0.1:8083]-AppendEntriesThread0] INFO  NodeImpl:1878 - ===handle rpc request, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$AppendEntriesRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8083"
    peer_id: "127.0.0.1:8081"
    term: 2
    prev_log_term: 2
    prev_log_index: 2
    committed_index: 2
    data: ""
    // ............
     2021-10-23 22:47:56 [election_test/PeerPair[127.0.0.1:8081 -> 127.0.0.1:8083]-AppendEntriesThread0] INFO  NodeImpl:1878 - ===handle rpc request, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$AppendEntriesRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8083"
    peer_id: "127.0.0.1:8081"
    term: 2
    prev_log_term: 1
    prev_log_index: 1
    entries {
      term: 2
      type: ENTRY_TYPE_CONFIGURATION
      peers: "127.0.0.1:8081"
      peers: "127.0.0.1:8082"
      peers: "127.0.0.1:8083"
      data_len: 0
    }
    committed_index: 2   
        
        
    // node2的日志,他响应了node1的预投票和投票请求
    2021-10-23 22:46:58 [main] INFO  RaftGroupService:136 - Start the RaftGroupService successfully.
    2021-10-23 22:46:59 [Bolt-default-executor-6-thread-2] INFO  NodeImpl:1635 - ===handle rpc request, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$RequestVoteRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8081"
    peer_id: "127.0.0.1:8082"
    term: 1
    last_log_term: 0
    last_log_index: 0
    pre_vote: true  // 预投票的日志
    
    2021-10-23 22:46:59 [Bolt-default-executor-6-thread-2] INFO  NodeImpl:1689 - Node <election_test/127.0.0.1:8082> received PreVoteRequest from 127.0.0.1:8081, term=1, currTerm=0, granted=true, requestLastLogId=LogId [index=0, term=0], lastLogId=LogId [index=0, term=0].
    2021-10-23 22:46:59 [Bolt-default-executor-6-thread-3] INFO  NodeImpl:1736 - ===handle rpc request, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$RequestVoteRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8081"
    peer_id: "127.0.0.1:8082"
    term: 1
    last_log_term: 0
    last_log_index: 0
    pre_vote: false // 投票的日志
    2021-10-23 22:46:59 [Bolt-default-executor-6-thread-3] INFO  NodeImpl:1760 - Node <election_test/127.0.0.1:8082> received RequestVoteRequest from 127.0.0.1:8081, term=1, currTerm=0.
    2021-10-23 22:46:59 [Bolt-default-executor-6-thread-3] INFO  LocalRaftMetaStorage:132 - Save raft meta, path=/tmp/server2/meta, term=1, votedFor=0.0.0.0:0, cost time=155 ms
    2021-10-23 22:46:59 [Bolt-default-executor-6-thread-3] INFO  LocalRaftMetaStorage:132 - Save raft meta, path=/tmp/server2/meta, term=1, votedFor=127.0.0.1:8081, cost time=77 ms
    //............
    2021-10-23 22:46:59 [election_test/PeerPair[127.0.0.1:8082 -> 127.0.0.1:8081]-AppendEntriesThread0] INFO  NodeImpl:1878 - ===handle rpc request, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$AppendEntriesRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8081"
    peer_id: "127.0.0.1:8082"
    term: 1
    prev_log_term: 0
    prev_log_index: 0
    committed_index: 0
    data: ""
    //............    
    2021-10-23 22:46:59 [election_test/PeerPair[127.0.0.1:8082 -> 127.0.0.1:8081]-AppendEntriesThread0] INFO  NodeImpl:1878 - ===handle rpc request, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$AppendEntriesRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8081"
    peer_id: "127.0.0.1:8082"
    term: 1
    prev_log_term: 1
    prev_log_index: 1
    committed_index: 1
    //............        
    2021-10-23 22:47:35 [Bolt-conn-event-executor-10-thread-1] INFO  RpcRequestProcessor:279 - Destroyed peer request context for election_test/PeerPair[127.0.0.1:8082 -> 127.0.0.1:8081]
    2021-10-23 22:47:37 [Bolt-default-executor-6-thread-5] INFO  NodeImpl:1635 - ===handle rpc request, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$RequestVoteRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8083"
    peer_id: "127.0.0.1:8082"
    term: 2
    last_log_term: 1
    last_log_index: 1
    pre_vote: true
    
    2021-10-23 22:47:37 [Bolt-default-executor-6-thread-5] INFO  NodeImpl:1689 - Node <election_test/127.0.0.1:8082> received PreVoteRequest from 127.0.0.1:8083, term=2, currTerm=1, granted=true, requestLastLogId=LogId [index=1, term=1], lastLogId=LogId [index=1, term=1].
    2021-10-23 22:47:37 [Bolt-default-executor-6-thread-6] INFO  NodeImpl:1736 - ===handle rpc request, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$RequestVoteRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8083"
    peer_id: "127.0.0.1:8082"
    term: 2
    last_log_term: 1
    last_log_index: 1
    pre_vote: false
    
    2021-10-23 22:47:37 [Bolt-default-executor-6-thread-6] INFO  NodeImpl:1760 - Node <election_test/127.0.0.1:8082> received RequestVoteRequest from 127.0.0.1:8083, term=2, currTerm=1.
    2021-10-23 22:47:37 [JRaft-FSMCaller-Disruptor-0] INFO  StateMachineAdapter:84 - onStopFollowing: LeaderChangeContext [leaderId=127.0.0.1:8081, term=1, status=Status[EHIGHERTERMRESPONSE<10008>: Raft node receives higher term RequestVoteRequest.]].
    2021-10-23 22:47:37 [Bolt-default-executor-6-thread-6] INFO  LocalRaftMetaStorage:132 - Save raft meta, path=/tmp/server2/meta, term=2, votedFor=0.0.0.0:0, cost time=71 ms
    2021-10-23 22:47:37 [Bolt-default-executor-6-thread-6] INFO  LocalRaftMetaStorage:132 - Save raft meta, path=/tmp/server2/meta, term=2, votedFor=127.0.0.1:8083, cost time=71 ms
    2021-10-23 22:47:37 [election_test/PeerPair[127.0.0.1:8082 -> 127.0.0.1:8083]-AppendEntriesThread0] INFO  NodeImpl:1878 - ===handle rpc request, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$AppendEntriesRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8083"
    peer_id: "127.0.0.1:8082"
    term: 2
    prev_log_term: 1
    prev_log_index: 1
    committed_index: 1
    data: ""
    
    // node3 日志,主要体现当时做了一次leader切换的过程(此时停了node1了,node3上来接管了)
    2021-10-23 22:47:37 [JRaft-ElectionTimer-<election_test/127.0.0.1:8083>0] INFO  AbstractClientService:195 - ===invokeWithDone, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$RequestVoteRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8083"
    peer_id: "127.0.0.1:8082"
    term: 2
    last_log_term: 1
    last_log_index: 1
    pre_vote: true
    
    2021-10-23 22:47:37 [JRaft-ElectionTimer-<election_test/127.0.0.1:8083>0] INFO  BoltRpcClient:102 - ==== invokeSync 127.0.0.1:8081, class com.alipay.sofa.jraft.rpc.RpcRequests$PingRequest
    
    =============
    2021-10-23 22:47:37 [JRaft-ElectionTimer-<election_test/127.0.0.1:8083>0] INFO  NodeImpl:2669 - preVote before grant, serverId 127.0.0.1:8083 ,,, prevVoteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=false, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=false, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=false, index=2]], quorum=2, oldPeers=[], oldQuorum=0]
    2021-10-23 22:47:37 [JRaft-ElectionTimer-<election_test/127.0.0.1:8083>0] INFO  NodeImpl:2670 - preVote after grant, serverId 127.0.0.1:8083 ,,, prevVoteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=false, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=false, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=true, index=2]], quorum=1, oldPeers=[], oldQuorum=0]
    2021-10-23 22:47:37 [JRaft-RPC-Processor-0] INFO  NodeImpl:2577 - Node <election_test/127.0.0.1:8083> received PreVoteResponse from 127.0.0.1:8082, term=1, granted=true.
    =============
    2021-10-23 22:47:37 [JRaft-RPC-Processor-0] INFO  NodeImpl:2580 - handlePreVoteResponse before grant, peerId 127.0.0.1:8082 ,,, prevVoteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=false, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=false, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=true, index=2]], quorum=1, oldPeers=[], oldQuorum=0]
    2021-10-23 22:47:37 [JRaft-RPC-Processor-0] INFO  NodeImpl:2581 - handlePreVoteResponse after grant, peerId 127.0.0.1:8082 ,,, prevVoteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=false, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=true, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=true, index=2]], quorum=0, oldPeers=[], oldQuorum=0]
    =============
    2021-10-23 22:47:37 [JRaft-RPC-Processor-0] INFO  NodeImpl:1120 - Node <election_test/127.0.0.1:8083> start vote and grant vote self, term=1.
    2021-10-23 22:47:37 [JRaft-RPC-Processor-0] INFO  AbstractClientService:195 - ===invokeWithDone, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$RequestVoteRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8083"
    peer_id: "127.0.0.1:8082"
    term: 2
    last_log_term: 1
    last_log_index: 1
    pre_vote: false
        
    2021-10-23 22:47:37 [JRaft-RPC-Processor-0] INFO  LocalRaftMetaStorage:132 - Save raft meta, path=/tmp/server3/meta, term=2, votedFor=127.0.0.1:8083, cost time=95 ms
    2021-10-23 22:47:37 [JRaft-RPC-Processor-0] INFO  NodeImpl:1172 - electSelf before grant, serverId 127.0.0.1:8083 ,,, voteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=false, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=false, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=false, index=2]], quorum=2, oldPeers=[], oldQuorum=0]
    2021-10-23 22:47:37 [JRaft-RPC-Processor-0] INFO  NodeImpl:1173 - electSelf after grant, serverId 127.0.0.1:8083 ,,, voteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=false, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=false, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=true, index=2]], quorum=1, oldPeers=[], oldQuorum=0]
    2021-10-23 22:47:37 [JRaft-RPC-Processor-1] INFO  NodeImpl:2518 - handleRequestVoteResponse before grant, peerId 127.0.0.1:8082 ,,, voteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=false, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=false, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=true, index=2]], quorum=1, oldPeers=[], oldQuorum=0]
    2021-10-23 22:47:37 [JRaft-RPC-Processor-1] INFO  NodeImpl:2519 - handleRequestVoteResponse after grant, peerId 127.0.0.1:8082 ,,, voteCtx Ballot [peers=[UnfoundPeerId [peerId=127.0.0.1:8081, found=false, index=0], UnfoundPeerId [peerId=127.0.0.1:8082, found=true, index=1], UnfoundPeerId [peerId=127.0.0.1:8083, found=true, index=2]], quorum=0, oldPeers=[], oldQuorum=0]
    2021-10-23 22:47:37 [JRaft-RPC-Processor-1] INFO  NodeImpl:1217 - Node <election_test/127.0.0.1:8083> become leader of group, term=2, conf=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083, oldConf=.// node3变成leader
    2021-10-23 22:47:37 [JRaft-RPC-Processor-1] INFO  Replicator:897 - Replicator=Replicator [state=Created, statInfo=<running=null, firstLogIndex=0, lastLogIncluded=0, lastLogIndex=0, lastTermIncluded=0>, peerId=127.0.0.1:8082, type=Follower]@127.0.0.1:8082 is started
    2021-10-23 22:47:37 [JRaft-RPC-Processor-1] INFO  AbstractClientService:195 - ===invokeWithDone, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$AppendEntriesRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8083"
    peer_id: "127.0.0.1:8082"
    term: 2
    prev_log_term: 1
    prev_log_index: 1
    committed_index: 1
    data: ""
    // ............
    2021-10-23 22:47:35 [Bolt-conn-event-executor-10-thread-1] INFO  RpcRequestProcessor:279 - Destroyed peer request context for election_test/PeerPair[127.0.0.1:8083 -> 127.0.0.1:8081]
    2021-10-23 22:47:37 [JRaft-FSMCaller-Disruptor-0] INFO  StateMachineAdapter:84 - onStopFollowing: LeaderChangeContext [leaderId=127.0.0.1:8081, term=1, status=Status[ERAFTTIMEDOUT<10001>: Lost connection from leader 127.0.0.1:8081.]].
    2021-10-23 22:47:37 [JRaft-ElectionTimer-<election_test/127.0.0.1:8083>0] INFO  NodeImpl:2623 - Node <election_test/127.0.0.1:8083> term 1 start preVote.
    2021-10-23 22:47:37 [JRaft-ElectionTimer-<election_test/127.0.0.1:8083>0] INFO  BoltRpcClient:102 - ==== invokeSync 127.0.0.1:8082, class com.alipay.sofa.jraft.rpc.RpcRequests$PingRequest
    2021-10-23 22:47:37 [Bolt-conn-event-executor-5-thread-1] INFO  ClientServiceConnectionEventProcessor:50 - Peer 127.0.0.1:8082 is connected
    2021-10-23 22:47:37 [JRaft-ElectionTimer-<election_test/127.0.0.1:8083>0] INFO  AbstractClientService:195 - ===invokeWithDone, request is : class com.alipay.sofa.jraft.rpc.RpcRequests$RequestVoteRequest 
    group_id: "election_test"
    server_id: "127.0.0.1:8083"
    peer_id: "127.0.0.1:8082"
    term: 2
    last_log_term: 1
    last_log_index: 1
    pre_vote: true

    如上有两条重要的地方,如日志中的注释:

    此时quorum 变成1 Ballot grant的是他自己: serverId 127.0.0.1:8081

    此时quorum 变成0 那么表示pre vote成功 Ballot grant的是他发送的对端: peerId 127.0.0.1:8082