OdenData

Mongo: Install and Upgrade

by Mark Nielsen
Copyright August 2021

You can install Mongo from scratch and also upgrade Mongo in this document. In general, if you go the upgrade path, you have to be careful to not jump with upgrades to far. Check the version upgrade documentation. Also, we are mainly focused on MongoDB 4.4 to MongoDB 5.0.

---

1. Links
2. GUI install
3. Potential problems upgrading
4. Simple install.
5. Simple Upgrade
6. Multiple version upgrade.
7. Replica Set Upgrades
8. Shard upgrades.
9. Weird stuff or gotchas

Links

• https://docs.mongodb.com/manual/release-notes/
• https://docs.mongodb.com/manual/installation/
• https://docs.mongodb.com/manual/tutorial/upgrade-revision/
• https://www.guru99.com/top-20-mongodb-tools.html

---

GUI install

I like to do manual installs. The GUI programs just issue commands behind the scences. That being said, why not make it easier? But for my purposes, I won't be using for my articles.

The ones I am interested or used:

• ClusterControl by SeveralNines. This is worth an article or youtube.
• Robo 3T
• Mongo Compass
• Mongo Management Studio. Note certain parts of this is all free.

There are paid alternatives, I may list them here later.

---

Potential problems upgrading

You should do complete testing and have each department sign off on upgrades. Dev, QA, ETL, application developers, etc.

• An engine changes. Like MMAPv1 is mow depreciated. And unlikely. It could affect the upgrade process, backups, restores, failovers, high availability. The worst is the data was in a storage engine that is now gone.
• A configuration is depreciated. This could affect restarts, upgrades, etc.
• A command or method or function is now gone. This could affect cronjobs, a procedure, monitoring, etc.
• A procedure changed. This can affect anything that uses that procedure.

---

Simple Install

We are installing on Ubuntu.

We will install 5.0 for now. For the upgrade we will remove and install 4.4.

  apt-get -y install gnupg
  wget -qO - https://www.mongodb.org/static/pgp/server-5.0.asc | sudo apt-key add -

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu \
 focal/mongodb-org/5.0 multiverse" \
 | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list

apt-get update

#apt-get install -y mongodb-org=5.0.5 mongodb-org-database=5.0.5 \
#   mongodb-org-server=5.0.5 mongodb-org-shell=5.0.5 \
#   mongodb-org-mongos=5.0.5 mongodb-org-tools=5.0.5

  # also this would work
apt-get install -y mongodb-org
sleep 5

service mongod restart

sleep 5

# Test some commands
mongo --eval "printjson(db.serverStatus())"
mongo --eval "Date()"

  # because I want to. 
apt-get -y install python2
  

---

Simple Upgrades

First remove mongo if installed.

apt-get remove  mongodb-database-tools mongodb-mongosh mongodb-org-database \
  mongodb-org-database-tools-extra mongodb-org-mongos mongodb-org-server \
  mongodb-org-shell mongodb-org-tools mongodb-org

rm -vrf /var/lib/mongodb
rm -rvf /etc/mongo*

Now install 4.4

wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list

apt-get update

apt-get install -y mongodb-org=4.4.11 mongodb-org-server=4.4.11 \
   mongodb-org-shell=4.4.11 mongodb-org-mongos=4.4.11\
   mongodb-org-tools=4.4.11
sleep 2

service mongod restart
sleep 2

ps auxw | grep mongo

mongo --eval "Date()"


Now make some stuff
mongo Junk --eval 'db.junk.insert( { "junk":1, "junkid" : 1 } ) '
mongo Junk --eval 'db.junk.createIndex(  { "junkid": 1  }, { unique: true, sparse: true  }) '
mongo Junk --eval 'db.junk.count() '



Now upgrade the standalone server.

  # even though we have 4.4 and 5 in apt, it will upgrade.
  # It will also restart mongo

   # this did not work. 
#apt-get install -y mongodb-org
   # You might have to remove 4.4 from apt and reload. That would work. 

apt list --installed | grep -i mongo

service mongod stop

  # did this instead. 
sudo apt-get install -y mongodb-org=5.0.5 mongodb-org-database=5.0.5 \ 
  mongodb-org-server=5.0.5 mongodb-org-shell=5.0.5 \ 
  mongodb-org-mongos=5.0.5 mongodb-org-tools=5.0.5 mongodb-org=5.0.5

apt list --installed | grep -i mongo
service mongod restart
sleep 5

mongo Junk --eval 'db.junk.count() '


Multiple version upgrade.

I suggest you read
https://www.percona.com/blog/2018/03/12/mongodb-mass-version-upgrading-upgrading-2-6-3-6/
and
https://docs.mongodb.com/manual/tutorial/upgrade-revision/
and be careful of it. Research how to do it. 

I really don't want to figure this out. Try to figure how to upgrade to 4.4 and then 
5.0 is much easier. 


Upgrading Replica Set
First, lets setup a cluster. We are also going to some fun unnecessary commands. 

function printPrimary { 
  count=`mongo --quiet --port 3001 mongo/functions.js| wc -l`
  while [ $count -lt 1 ]
  do
    echo "No primary yet: $count"
    sleep 2
    count=`mongo --quiet --port 3001 mongo/functions.js| wc -l`
  done
  port=`mongo --quiet --port 3001 mongo/functions.js | cut -d ':' -f2 `
  echo "port is on $port"
}

  # If restarting, kill the mogo processes and then do 
rm -rf  mongo/d1 mongo/d2 mongo/d3 mongo/logs
  # Setup 3 directories:
mkdir -p mongo/d1 mongo/d2 mongo/d3 mongo/logs
  # If not AWS,	  ignore.  
sudo  -s -- bash -c   '  echo "127.0.0.1 localhost ">> /etc/hosts '
  # Start once mongo instance
mongod --dbpath mongo/d1 --port 3001 --oplogSize 100 --wiredTigerCacheSizeGB 0.25 --replSet r --bind_ip 127.0.0.1 > mongo/logs/1.log 2>&1  & 
sleep 2
  # In  the mongo shell
mongo --port 3001 --eval "rs.initiate( { _id: 'r', version: 1, members: [ {_id :0, host: 'localhost:3001' } ] } )"

  # Start and add two other replica sets
mongod --dbpath mongo/d2 --port 3002 --oplogSize 100 --wiredTigerCacheSizeGB 0.25 --replSet r --bind_ip 127.0.0.1 > mongo/logs/2.log 2>&1  &
mongod --dbpath mongo/d3 --port 3003 --oplogSize 100 --wiredTigerCacheSizeGB 0.25 --replSet r --bind_ip 127.0.0.1 > mongo/logs/3.log 2>&1  &
sleep 2
mongo --port 3001 --eval "rs.add('localhost:3002')"
sleep 2
mongo --port 3001 --eval "rs.add('localhost:3003')"
sleep 2
  # Look at the status
mongo --port 3001 --eval "rs.status()"

echo "Let us sleep a really long time, 30 seconds"
sleep 30

/home/mark/temp# mongo --port 3001 --eval "rs.status()" | grep -i state

  # Lets setup the functions. 
echo "
cfg = rs.conf();
cfg.members[0].priority = 3;
cfg.members[1].priority = 2;
cfg.members[2].priority = 1;
rs.reconfig(cfg);
" >> mongo/reconfig.js

echo '
function pcheck() {
  s = rs.status()
  s.members.forEach( 
    function(myDoc) { 
      if (myDoc.stateStr=="PRIMARY") 
        { print( myDoc.stateStr + " " + myDoc.name ) }
      }
    );
}
pcheck()
' > mongo/functions.js

  # Lets print out the hosts in the replica set
mongo --port 3001 --eval "rs.status()" | grep name
  # Remove a host
mongo --port 3001 --eval "rs.remove('localhost:3003')"
sleep 1
mongo --port 3001 --eval "rs.status()" | grep name

  # Add host back in
mongo --port 3001 --eval "rs.add('localhost:3003')"
sleep 2
  # Add print out who is primary and secondary
mongo --port 3001 --eval "rs.status()" | egrep 'name|stateStr'

  # Redo the prorities, but in practice its no promise for stepdowns.
cat mongo/reconfig.js | mongo --port 3001 
mongo --port 3001 --eval "rs.conf()" | egrep 'host|priority'

mongo --port 3001 --eval "rs.stepDown({secondaryCatchUpPeriodSecs: 10 })"
sleep 5
printPrimary
mongo --port 3001 --eval "rs.status()" | egrep 'name|stateStr'

  # Not let's force it back to the first one --- this is bad in production. 
mongo --quiet  --port 3001 --eval "rs.remove('localhost:3003')"
sleep 5
printPrimary
mongo --quiet --port 3001 --eval "rs.remove('localhost:3002')"
sleep 5
printPrimary
mongo --quiet --port 3001 --eval "rs.status()" | egrep 'name|stateStr'

   # Now add them back in
mongo --quiet --port 3001 --eval "rs.add('localhost:3002')"
sleep 2
mongo --quiet --port 3001 --eval "rs.add('localhost:3003')"
sleep 2
  # Look at the status
mongo --quiet --port 3001 --eval "rs.status()" | egrep 'name|stateStr'


  echo "
  cfg = rs.conf();
  cfg.members[0].priority = 3;
  cfg.members[1].priority = 2;
  cfg.members[2].priority = 1;
  rs.reconfig(cfg);
  " > mongo/reconfig.js
  
cat mongo/reconfig.js | mongo --port 3001 --quiet
cat mongo/reconfig.js | mongo --port 3002 --quiet
cat mongo/reconfig.js | mongo --port 3003 --quiet




Since we have all the mongo servers using the same binary, we have to shutdown all
the servers. Normally, you would shutdown one server at a time and upgrade and restart
mongodb. You would start with the secondaries, then the primary and make the original
primary server primary again. 
Here we will just do one big upgrade. 

We start with 4.4
ps auxw  | grep mongod

killall mongod

sleep 10

ps auxw  | grep mongod

 apt-get install -y mongodb-org=5.0.5 mongodb-org-database=5.0.5 \
  mongodb-org-server=5.0.5 mongodb-org-shell=5.0.5 \
  mongodb-org-mongos=5.0.5 mongodb-org-tools=5.0.5 mongodb-org=5.0.5

apt list --installed | grep -i mongo
service mongod restart
sleep 5

mongo --port 3001 --eval "rs.status()" | grep name



Upgrading Shards

With sharding you do he following which follows mongodb notes on upgrading 
sharding. I would do it in a different order than whats documented unless you want 
little downtime, then the documentation is fine. 

Schedule downtime. 
Disable the balancer. 
Stop mongos services. 
Upgrade mongos.
Stop config servers. 
Upgrade config servers. 
Upgrade each replica set and restart the replica sets. Look at error logs. 
Start config servers. Look at error logs. 
Start the mongos services. Look at error logs. 
Enable the balancer. 
Test and verify before ending downtime. 




Weird or gotchas
Weird stuff I experiences or heard of:

  
 MMAPv1 storage engine no longer is in the latest versions. However, I worked
    in a project to explain why Mongo was slower with the new engine and version of
    MySQL. So I timed the process with the different storage engines and found out a
    single thread processing a lot of data was faster under MMAPv1 storage engine
    than WiredTiger. A single process may be slower in WiredTiger but WiredTiger handles
    multiple processes much better (locking and other). Still they didn't like I didn't
    make it faster even though I found out why. They just wanted the upgraded version
    for political reasons. I stay away from politics.
  
In an interview, a person asked how to can get lost data. They were pointing at
    bad shards or putting data in the wrong shard and so it remains invisible. But they
    there are other ways. If there is a imperfect failover in a replica set some data
    may be stored in the "rollback" directory. They didn't like I had better answers
    than they did. Its bad in an interview to correct the interviewer.
    Its often better
    to answer what they want to hear than being correct. 

    
https://docs.mongodb.com/manual/core/replica-set-rollbacks/#std-label-replica-set-rollback