Hi,
I am using Nodequery and Nixstats. But they get data at certain intervals. So if anything happens in between that interval and my server went unresponsive or crashes then it’s not recorded. Now, it is very hard to detect what exactly caused the issue
Question 1: Can you suggest any self-hosted monitoring tool that very good at monitoring and alerting? Question 2: How to setup a system that can reboot Ubuntu if it goes out of memory.
The first tool that comes to mind is Monit, it solves both questions 1 + 2.
As described on their website;
Monit can act if an error situation should occur, e.g.; if sendmail is not running, Monit can start sendmail again automatically or if apache is using too much resources (e.g. if a DoS attack is in progress) Monit can stop or restart apache and send you an alert message. Monit can also monitor process characteristics, such as; how much memory or cpu cycles a process is using.
Monit for being proactive + New Relic (Lite) for alerting is a lethal combo. 
Hope that covers your monitoring & alerting needs.