26 Jan 2018 in Storage, Hardware, Software Defined Storage

OSNexus Quantastor Support - Is Awesome

Late Dec 2017

I had built by Storage Server with OSNexus Quantastor with v4.4.1.011 released 19 Dec 2017, with Community Licenses, limited to 20TB only.
Created NFS Exports, and setup NFS share access to Hosts.
Also exported 2x Physical Disks as Pass-thru Storage Volumes presented via Fibre Channel. Connected the Cisco Nexus 9124 Fabric and did the zoning as well.

Two days later, i can no longer log in in WebUI.

Error Message;

Error
Connection Refused

The default admin password for QuantaStor is simply ‘password’. I had not changed it. If there is a wrong password or username, the error message would be different. Now i’m getting nowhere and perform the trouble shooting as below

So i followed your wiki as per; https://wiki.osnexus.com/index.php?title=QuantaStor_Troubleshooting_Guide#Login_.26_Upgrade_Issues

I tried resetting the admin password, as per wiki,


However when i test it

root@n54l-osnexus:/opt/osnexus/quantastor/bin# qs sys-get --server=localhost,admin,newpass ERROR: Server request to [http://localhost:5152] failed: "Connection refused"

---------------------------------------------------- desperate times now---------------------------------------------------- I tried the following as well; Resetting all the security configuration data A more brute force way of removing all user accounts and recreating the admin account with the default password ('password') is to do a full --security-reset like so:

root@n54l-osnexus:/opt/osnexus/quantastor/bin# service quantastor stop * Stopping OSNEXUS QuantaStor Service qs_service [ OK ]
root@n54l-osnexus:/opt/osnexus/quantastor/bin# ./qs_service --reset-security OSNEXUS QuantaStor Service 4.4.1.011 Copyright (c) 2009-2017 OSNEXUS Corporation. All rights reserved. root@n54l-osnexus:/opt/osnexus/quantastor/bin# service quantastor start
* Starting OSNEXUS QuantaStor Service qs_service OSNEXUS QuantaStor Service 4.4.1.011 Copyright (c) 2009-2017 OSNEXUS Corporation. All rights reserved. [ OK ]
root@n54l-osnexus:/opt/osnexus/quantastor/bin# root@n54l-osnexus:/opt/osnexus/quantastor/bin# qs sys-get --server=localhost,admin,password
ERROR: Server request to [http://localhost:5152] failed: "Connection refused" root@n54l-osnexus:/opt/osnexus/quantastor/bin#

Still the same. I simply power off the Microserver, and replace the boot disk with another SSD Disk, and disconnect the data drives. Install and configure the OSNexus Quantastor with v4.4.1.011, and run updates, basuically the updates for Ubuntu Linux [it’s the base os used by OSNexus Quantastor.] 06-01-2018 Apply community license, and power off the microserver. Plug in the data disks, power it on, OSNexus Quantastor dectets the disks and UuiD, and informs in log section, that disks/nfs exports can be imported. Attempt to import, now issue is can’t import with Community license. Jumped online to OSNexus Quantastor website and requesting trail licenses and also adding a comment that can’t import with community license. I receive the Trial license, and apply it. Log out of web console and log back in. Duh the problem has started again. OS Nexus Business Development, team on the same day log a suport case on my behalf. I apply the admin password reset methodology again as per wiki, no avail. I power it off. This time thinking the issue is still SSD boot Disk. * One day later support starts to communicate via email , and we are now in trobleshooting mode. Suport requests i upload logs. I uploaded logs and was advised to reset the admin password by this wiki link Please review the the steps in this KB article and make sure that you follow the steps exactly: How To Reset The Admin Password Via SSH. https://support.osnexus.com/hc/en-us/articles/211417523-How-to-reset-the-admin-password-via-SSH- That didn’t resolve the issue. Supoprt enageges in a remote suport sessions and runs further diagostics; and upload log files By this time, i am desperate to get my storage OS up and runing.
I replace the boot disk with conventional 2.5” WD Spining disk and re-install the OSNexus Quantastor with v4.4.1.011. Apply trial license a and power off the microserver. Plug in the data disks, power it on,
OSNexus Quantastor dectets the disks and UuiD, and infirm in log section, that disks/nfs exports can be imported.
I import disks and shares. By now the response from OSNexus Supoprt: Hello Roshan, I just wanted to give you an update and let you know that I have found a "SEGFAULT" error in the logs so this may still be the root cause of the issue. I will need to have my Engineering team review the stack trace to determine what is causing this. Please see below for an example of the error:

{Sat Jan 13 16:22:59 2018, INFO, c8ff9700:hwadapter_util:382} Perform initial hw correlation verify ... {Sat Jan 13 16:22:59 2018, INFO, c8ff9700:hwadapter_util:765} Initial correlation verify complete. {Sat Jan 13 16:23:15 2018, ERROR, SEGFAULT} {Sat Jan 13 16:23:15 2018, ERROR, FAULTINFO} stack trace: {Sat Jan 13 16:23:15 2018, ERROR, FAULTINFO} /opt/osnexus/quantastor/bin/qs_service() [0xdc532d] {Sat Jan 13 16:23:15 2018, ERROR, FAULTINFO} /lib/x86_64-linux-gnu/libpthread.so.0 : ()+0x10330 {Sat Jan 13 16:23:15 2018, ERROR, FAULTINFO} /lib/x86_64-linux-gnu/libc.so.6 : ()+0x9aa5e {Sat Jan 13 16:23:15 2018, ERROR, FAULTINFO} /opt/osnexus/quantastor/bin/qs_service() [0x280f420] {Sat Jan 13 16:23:15 2018, ERROR, FAULTINFO} /opt/osnexus/quantastor/bin/qs_service() [0x280f86c] {Sat Jan 13 16:23:15 2018, ERROR, FAULTINFO} /opt/osnexus/quantastor/bin/qs_ser vice() [0x40b335]

Jan 18, 2018 Support comes back with update; (OSNEXUS Customer Support) Jan 18, 12:40 PST Hello Roshan, Our Engineering team has been able to determine the root cause of the SEGFAULT so there is no need for you to collect any additional information or continue testing. We have a software fix for this issue that we are currently testing and should it be in our next release that is scheduled for sometime next week. I can let you know when the new Quantastor version is available or you can check this link for the new release to be available. https://wiki.osnexus.com/index.php?title=QuantaStor_Version_ChangeLog Jan 22, 2018 Support comes back with news; (OSNEXUS Customer Support) Jan 22, 10:27 PST Greetings, We have just released the v4.4.2.004 version of Quantastor which should address the issues you have been running into. Please review the link below for the new release and download the new version as soon as possible to correct your issue: https://wiki.osnexus.com/index.php?title=QuantaStor_4.4.2_Upgrade_Instructions Please let me now if you have any questions at this time. 26 Jan 2018 I update the OSNexus quantastor SDS with instructions as per https://wiki.osnexus.com/index.php?title=QuantaStor_4.4.2_Upgrade_Instructions and so far no lockouts.

Whats great is that for an Organisation like OSNexus, they have gone beyond expectation to resolve a product issue, despite me not being on a paid Support Subscription. This shows the true image of OSNexus, OSNEXUS focuses on continuous innovation with a strong focus on product quality and industry leading support to ensure customer and partner success with every deployment. https://www.osnexus.com/about/

Thank You For Reading
Roshan Mohammed

Cloud Engineering Officer, allclouds.nz