05 Mar 2018 in VMware, vRealize Network Insight

vRealize Network Insight Upgrade fails, 'No Space Left on Device'

In many cases your usual version of the software is either at n-1 level, or the greatest and the latest.Sometimes you need to upgrade software to meet compatibility levels and or to fix a bug, or to utilise new feature sets.

Upgrading vRNI from 3.x to 3.7 requires some specific set of instructions.

User accounts to use to upgrade is consoleuser [this is the only account that has privilages to perform application level configurations.]

vRNI Users

vRNI Users

Note: It’s highy recommended that the default passwords be changed.

Updating vRNI from 3.6 to 3.7

If your vRNI appliance doesn’t have internet access [due to security reasons], offline method of upgrading is available, but only via command line.

There are many artciles on the net explaining on how to do this, so i’ll jump straight to the issue.

Always perform a snapshot, prior to performing any upgrade/config changes to vRNI.

Upgrade Order:

  1. vRNI Platform Appliance
  2. vRNI Proxy Appliances [as many you have]

Logon into vRNI platform via SSH, check services and version

vRealize Network Insight Command Line Interface
(cli) show-service-status
CollectorMain is running
IpfixProcessor is running
Postgresql is running
Nginx is running
Nfcapd is running
NTP is running and healthy.
All required services are running.
(cli) show-version
3.6.0.1509991432

Attempt to apply upgrade;

Note in this case, the offline upgrade file VMware-vRealize-Network-Insight.3.7.0.1519211678.upgrade.bundle has been uplaoded to /tmp/ folder of the VRNI Appliance.

Use WinScp Tool, and can only be done by using the support username/password.

(cli) package-installer upgrade --name /tmp/VMware-vRealize-Network-Insight.3.7.0.1519211678.upgrade.bundle
Do you want to continue with upgrade? (y/n) y
It will take some time...

Failed to upgrade. Please contact support and provide support bundle.
To get more specific details try "log-trace display 100 cli.log"

It fails, but atleast states how to review the log files,

Upon Reviewing;

(cli) log-trace display 100 cli.log
Removed 
2018-03-05 17:28:31,466 - INFO - Starting handle file: samzajobs-1519211678-a97e930f0acb2a14f090e579f257a23f.tgz.bundle
2018-03-05 17:28:31,466 - INFO - gpg -o samzajobs.tar.gz  /tmp/upgrade-bits/samzajobs-1519211678-a97e930f0acb2a14f090e579f257a23f.tgz.bundle
2018-03-05 17:28:40,867 - ERROR - Error decrypt and verify bundle
2018-03-05 17:28:40,868 - ERROR -
2018-03-05 17:28:40,868 - ERROR - gpg: WARNING: unsafe ownership on configuration file `/home/consoleuser/.gnupg/gpg.conf'
gpg: encrypted with 2048-bit RSA key, ID C3F055C3, created 2016-09-04
      "vRNI-App (Client side key pair for platform and proxy)"
gpg: Error writing to `samzajobs.tar.gz': No space left on device
gpg: handle plaintext failed: file write error
2018-03-05 17:28:40,868 - ERROR - Do not continue because we cannot handle file
2018-03-05 17:28:40,882 - ERROR - Error running upgrade main [python offline-upgrade/main-upgrade.py]
2018-03-05 17:28:40,883 - ERROR - Running Pre-steps
cmd: python offline-upgrade/pre-upgrade.py failed
2018-03-05 17:28:40,883 - ERROR -
2018-03-05 17:28:40,883 - INFO - Cleaning up files under /tmp/upgrade-bits and /home/consoleuser/downloads
2018-03-05 17:28:43,400 - ERROR - Failed to upgrade
(cli)

Failure states; No space left on device

Review the Disk Usage, and there is plenty of Disk Space, atleast on /tmp and also on /home

(cli) disk-usage

Filesystem                    Size  Used Avail Use% Mounted on
udev                          4.9G  4.0K  4.9G   1% /dev
tmpfs                        1001M  776K 1000M   1% /run
/dev/dm-0                      30G  5.3G   23G  20% /
none                          4.0K     0  4.0K   0% /sys/fs/cgroup
none                          5.0M     0  5.0M   0% /run/lock
none                          4.9G  4.0K  4.9G   1% /run/shm
none                          100M     0  100M   0% /run/user
/dev/sda1                     184M   88M   87M  51% /boot
/dev/mapper/vg-home            15G  5.7G  8.2G  41% /home
/dev/mapper/vg-tmp             15G  287M   14G   3% /tmp
/dev/mapper/vg-var             53G   17G   35G  32% /var
/dev/mapper/vg-var+log         25G  5.9G   18G  25% /var/log
/dev/mapper/vg-var+log+audit   15G   53M   14G   1% /var/log/audit

Notice in the logs, atleast the last 2 lines states, that it’s cleaning the directories s

  • 2018-03-05 17:28:40,883 - INFO - Cleaning up files under /tmp/upgrade-bits and /home/consoleuser/downloads
  • 2018-03-05 17:28:43,400 - ERROR - Failed to upgrade

From what i gather, it appears, when the upgrade command is applied, the file VMware-vRealize-Network-Insight.3.7.0.1519211678.upgrade.bundle is copied from /tmp to /home/consoleuser/downloads , and is also extracted to path /tmp/upgrade-bits/

File Size is 2.6GB in Size, and copied to 1 more location and extracted in /tmp as well. This could possbily be the issue.

Now i delete the file VMware-vRealize-Network-Insight.3.7.0.1519211678.upgrade.bundle from /tmp and attempt to scp it to /home/consoleuser/downloads, and it fails, as support account doesn’t have rights to consoleuser home directory.

Now i also atempted to copy the file VMware-vRealize-Network-Insight.3.7.0.1519211678.upgrade.bundle to /home/support/downloads folder via WinSCP, this worked, but now it appears consoleuser doesn’t have access to /home/support/downloads directory, or not able to expand the files.

By now your drives and or /tmp has some bloated files extracted onto it. Best to perform a reboot to clear these files.

(cli) appliance-reboot now

Broadcast message from consoleuser@vrni-proxy-release
        (/dev/pts/1) at 17:34 ...

The Fix

After the reboot , i continue to investigate and browse the diectory structure of the vRNI Appliance via WinSCP. There’s a /downloads directoy on the root partition.

Log into the appliance using the support account/password AND apply the below commands to review the /downloads folder details

support@vrni-proxy-release:~$ cd /
support@vrni-proxy-release:/$ ls
bin        etc             lib         mnt             proc  srv  var
boot       home            lib64       nginx.conf.tmp  root  sys  vmlinuz
dev        initrd.img      lost+found  nonexistent     run   tmp  vmlinuz.old
downloads  initrd.img.old  media       opt             sbin  usr  vnera.tmp

support@vrni-proxy-release:/$ ls -la
total 113
drwxr-xr-x  24 root root       4096 Mar  5 22:54 .
drwxr-xr-x   2 root root       4096 Mar  5 20:54 bin
drwxr-xr-x   4 root root       1024 Mar  5 20:54 boot
drwxr-xr-x  17 root root       4040 Mar  5 20:18 dev
drwxrwxr-x   2 root vrniusers  4096 Mar  5 20:57 downloads
drwxr-xr-x 106 root root      12288 Mar  5 22:50 etc
drwxr-xr-x   6 root root       4096 May 17  2017 home
----------10 other lines removed-------------------
support@vrni-proxy-release:/$

Great, vrniusers have access to folder /downloads on the root partition.

Now upload the file VMware-vRealize-Network-Insight.3.7.0.1519211678.upgrade.bundle to /downloads via WinScp, and then apply upgrade commands

(cli) package-installer upgrade --name /downloads/VMware-vRealize-Network-Insight.3.7.0.1519211678.upgrade.bundle
Do you want to continue with upgrade? (y/n) y
It will take some time...
 [\]

After 25mins or so, the upgrade completes.

Check Version and Service Status

(cli) show-version
3.7.0.1519211678

(cli) show-service-status
ElasticSearch is running
HMaster is running
HRegionServer is running
Kafka is running
QuorumPeerMain is running
ResourceManager is running
NodeManager is running
SaasListener is running
Restapilayer is running
ClusterBasedJobCoordinator is running
SamzaContainer is running
Launcher is running
Postgresql is running
Nginx is running
Patroni is running
ExpressJSApp is running
NTP is running and healthy.
All required services are running.

You can now perform offline upgrades by uploading the upgrade bundle file to /downloads folder on root partition.

Upgrading vRNI 3.7 to new releases

From version 3.7 onwards, you can now perform offline upgrades via the WebGUI of vRNI.

Wait until a new update/bugfix release bundle is made available for download.

Log into vRNI, Settings, Install and Support;

vRNI 3.7 Upgrade

Click on the Click here

and below screen comes up, and rest is self explainatory.

vRNI 3.7 Upgrade Offline

Hopefully in future releases, we get to upload a single upgrade bundle file, and it upgrades the both platform and proxy appliances in a sequence.

Thank You For Reading
Roshan Mohammed

Cloud Engineering Officer, allclouds.nz