Mongo

From wiki.kungfootek.net
Jump to: navigation, search

Force a single Secondary to Primary

While doing some maintenance I had upgraded the two secondaries previously. One was ready and the other was still catching up. I knocked over the primary expecting the one good secondary server to become master. This never happened and I ended up with 1 lame secondary, 1 server booting to become a 2nd lame secondary, and a healthy secondary all by itself that refused to become master.

The steps below are what I used to force my sole secondary to become master by itself, then later add the two secondaries back to the replica-set.


With the manual as a guide:


Capture the current config.

Command:

db_server_name_mongo_env:PRIMARY> cfg = rs.conf()

Result:

{
   "_id" : "db_server_name_mongo_env",
   "version" : 30541,
   "members" : [
      {
       "_id" : 8,
       "host" : "ip-10-245-90-88.us-west-2.compute.internal:27017"
      },
      {
       "_id" : 9,
       "host" : "ip-10-245-90-76.us-west-2.compute.internal:27017"
      },
      {
       "_id" : 10,
       "host" : "ip-10-245-90-78.us-west-2.compute.internal:27017"
      }
   ]
}


Select the line that represents your working Mongo secondary. Ignore the "_id" : #, count the lines from 0. In this case I wanted the line identified by "_id" : 9 so I enter 1 in the array field. [cfg.members[1]]


Command:

db_server_name_mongo_env:SECONDARY> cfg.members = [cfg.members[1]]


Result:

[
   {
      "_id" : 9,
      "host" : "ip-10-245-90-76.us-west-2.compute.internal:27017"
   }
]


As long as the good working secondary is listed we are good to reconfigure the cluster by forcing the issue. There are implications to using force too liberally, please read all the documentation available on passing the force parameter to make sure your situation is a good candidate before continuing.


Command:

db_server_name_mongo_env:SECONDARY> rs.reconfig(cfg, {force : true})


Result:

{ "ok" : 1 }


Confirm your changes.


Command:

db_server_name_mongo_env:SECONDARY> db.isMaster()

Result:

{
   "setName" : "db_server_name_mongo_env",
   "setVersion" : 30539,
   "ismaster" : true,
   "secondary" : false,
   "hosts" : [
      "ip-10-245-90-76.us-west-2.compute.internal:27017"
   ],
   "primary" : "ip-10-245-90-76.us-west-2.compute.internal:27017",
   "me" : "ip-10-245-90-76.us-west-2.compute.internal:27017",
   "maxBsonObjectSize" : 16777216,
   "maxMessageSizeBytes" : 48000000,
   "maxWriteBatchSize" : 1000,
   "localTime" : ISODate("2016-06-18T21:49:32.938Z"),
   "maxWireVersion" : 2,
   "minWireVersion" : 0,
   "ok" : 1
}


db_server_name_mongo_env:MASTER>


From here you can re-enter your configuration options or allow your chef-client to run if you are provisioning via DevOps tools and reconfigure your cluster again:

Command:

rs.reconfig(cfg)

Result:

{ "ok" : 1 }


At this point I would like to add a note about the importance of having a reliable arbiter deployed with any replica set. This hadn't happened in Prod, but we were moving that direction when this popped up.