MySQL suddenly "died"

Murio · June 15, 2015, 7:18am

Hi there,

all was working fine until today around 3:00 AM. When I woke up I found an e-mail from Uptime Robot informing me that my site is offline. Quick check confirmed that it’s not working (“Cannot connect to database” error).

I checked ee status and indeed - MySQL isn’t running and cannot start.

ee debug showed (2003, “Can’t connect to MySQL server on ‘localhost’ (111)”). Which - I guess - means, that’s offline What’s not very helpful.

Nothing was updating (EE was already updated to latest version), I did nothing. It just died by itself

Where to start looking what’s the reason and how to repair it? I couldn’t find log files. (Simple “methods” did not work - restarting server and stack did not help)

From “weird things” I spotted, that server is using more RAM that it usually did after rebooting (usually it was ~300-500MB and it was buiding up from that point if necessary. Now from the start it’s 1,5-1,6GB RAM used and free -m is showing that most of it is a garbade data.)

Regards, Kamil

harshadyeola · June 15, 2015, 8:57am

@Murio

Check your disk space (df -h) if its full 100% then your mysql server may crash.
check logs /var/log/mysql/error.log for reason.

Murio · June 15, 2015, 9:42am

@harshadyeola

I have 72% disc space usage.
I haven’t got that file in that location.

Regards, Kamil

EDITED: I checked EE log file and I have found something strange at 3:00 AM. Is it helpful?

Log file:

 2015-06-15 02:48:07,392 (DEBUG) ee : logging initialized for 'ee' using LoggingLogHandler
2015-06-15 02:48:09,940 (DEBUG) ee : ['/usr/local/bin/ee', 'stack', 'status']
2015-06-15 02:48:09,940 (DEBUG) ee : collecting arguments/commands for <ee.cli.controllers.base.EEBaseController object at 0x7fb4043e7630>
2015-06-15 02:48:09,945 (DEBUG) ee : collecting arguments/commands for <ee.cli.plugins.stack.EEStackController object at 0x7fb4043e7828>
2015-06-15 02:48:09,948 (DEBUG) ee : collecting arguments/commands for <ee.cli.plugins.stack_services.EEStackStatusController object at 0x7fb4043e7fd0>
2015-06-15 02:48:09,953 (INFO) ee : Initializing EasyEngine Database
2015-06-15 02:48:27,787 (INFO) ee : e[94mHHVM is not installede[0m
2015-06-15 02:48:27,834 (INFO) ee : e[94mnginx     :  Runninge[0m
2015-06-15 02:48:27,856 (INFO) ee : e[94mphp5-fpm  :  Runninge[0m
2015-06-15 02:48:27,902 (DEBUG) ee : e[95m * MariaDB is stopped.e[0m
2015-06-15 02:48:27,939 (INFO) ee : e[94mpostfix   :  Runninge[0m
2015-06-15 02:49:26,865 (DEBUG) ee : logging initialized for 'ee' using LoggingLogHandler
2015-06-15 02:49:27,176 (DEBUG) ee : ['/usr/local/bin/ee', 'stack', 'restart']
2015-06-15 02:49:27,177 (DEBUG) ee : collecting arguments/commands for <ee.cli.controllers.base.EEBaseController object at 0x7f28b534e630>
2015-06-15 02:49:27,181 (DEBUG) ee : collecting arguments/commands for <ee.cli.plugins.stack.EEStackController object at 0x7f28b534eb70>
2015-06-15 02:49:27,183 (DEBUG) ee : collecting arguments/commands for <ee.cli.plugins.stack_services.EEStackStatusController object at 0x7f28b534eef0>
2015-06-15 02:49:27,188 (INFO) ee : Initializing EasyEngine Database
2015-06-15 02:49:37,202 (INFO) ee : e[94mHHVM is not installede[0m
2015-06-15 02:49:37,203 (DEBUG) ee : e[95mRestarting service: nginxe[0m
2015-06-15 02:49:37,207 (INFO) ee : e[94mRestart : nginx     e[0m
2015-06-15 02:49:38,666 (INFO) ee : e[94m[e[0mOKe[94m]e[0m
2015-06-15 02:49:38,666 (DEBUG) ee : e[95mRestarting service: php5-fpme[0m
2015-06-15 02:49:38,667 (INFO) ee : e[94mRestart : php5-fpm  e[0m
2015-06-15 02:49:38,766 (INFO) ee : e[94m[e[0mOKe[94m]e[0m
2015-06-15 02:49:38,766 (DEBUG) ee : e[95mRestarting service: mysqle[0m
2015-06-15 02:49:38,766 (INFO) ee : e[94mRestart : mysql     e[0m
2015-06-15 02:50:09,382 (DEBUG) ee : e[95m * Stopping MariaDB database server mysqld
   ...done.
 * Starting MariaDB database server mysqld
   ...fail!e[0m
2015-06-15 02:50:09,382 (INFO) ee : e[94m[e[91mFailede[94m]e[0m
2015-06-15 02:50:09,383 (DEBUG) ee : e[95mRestarting service: postfixe[0m
2015-06-15 02:50:09,383 (INFO) ee : e[94mRestart : postfix   e[0m
2015-06-15 02:50:09,599 (INFO) ee : e[94m[e[0mOKe[94m]e[0m
2015-06-15 02:50:26,846 (DEBUG) ee : logging initialized for 'ee' using LoggingLogHandler
2015-06-15 02:50:27,154 (DEBUG) ee : ['/usr/local/bin/ee', 'stack', 'start', '--mysql']
2015-06-15 02:50:27,155 (DEBUG) ee : collecting arguments/commands for <ee.cli.controllers.base.EEBaseController object at 0x7f0cb02b9668>
2015-06-15 02:50:27,159 (DEBUG) ee : collecting arguments/commands for <ee.cli.plugins.stack.EEStackController object at 0x7f0cb02b9470>
2015-06-15 02:50:27,161 (DEBUG) ee : collecting arguments/commands for <ee.cli.plugins.stack_services.EEStackStatusController object at 0x7f0cb02b9ef0>
2015-06-15 02:50:27,168 (INFO) ee : Initializing EasyEngine Database
2015-06-15 02:50:31,474 (DEBUG) ee : e[95mStarting service: mysqle[0m
2015-06-15 02:50:31,474 (INFO) ee : e[94mStart : mysql     e[0m
2015-06-15 02:51:02,050 (DEBUG) ee : e[95m * Starting MariaDB database server mysqld
   ...fail!e[0m
2015-06-15 02:51:02,051 (INFO) ee : e[94m[e[91mFailede[94m]e[0m
2015-06-15 02:52:19,711 (DEBUG) ee : logging initialized for 'ee' using LoggingLogHandler
2015-06-15 02:52:20,019 (DEBUG) ee : ['/usr/local/bin/ee', 'debug']
2015-06-15 02:52:20,019 (DEBUG) ee : collecting arguments/commands for <ee.cli.controllers.base.EEBaseController object at 0x7f6eb4f475f8>
2015-06-15 02:52:20,024 (DEBUG) ee : collecting arguments/commands for <ee.cli.plugins.debug.EEDebugController object at 0x7f6eb4f47cf8>
2015-06-15 02:52:20,028 (INFO) ee : Initializing EasyEngine Database
2015-06-15 02:53:02,283 (DEBUG) ee : logging initialized for 'ee' using LoggingLogHandler
2015-06-15 02:53:02,604 (DEBUG) ee : ['/usr/local/bin/ee', 'debug', 'gerendi.pl', '--mysql']
2015-06-15 02:53:02,605 (DEBUG) ee : collecting arguments/commands for <ee.cli.controllers.base.EEBaseController object at 0x7fef66eb7630>
2015-06-15 02:53:02,610 (DEBUG) ee : collecting arguments/commands for <ee.cli.plugins.debug.EEDebugController object at 0x7fef66eb77f0>
2015-06-15 02:53:02,615 (INFO) ee : Initializing EasyEngine Database
2015-06-15 02:53:49,790 (DEBUG) ee : logging initialized for 'ee' using LoggingLogHandler
2015-06-15 02:53:50,105 (DEBUG) ee : ['/usr/local/bin/ee', 'debug', '--mysql']
2015-06-15 02:53:50,105 (DEBUG) ee : collecting arguments/commands for <ee.cli.controllers.base.EEBaseController object at 0x7f9c066fe630>
2015-06-15 02:53:50,110 (DEBUG) ee : collecting arguments/commands for <ee.cli.plugins.debug.EEDebugController object at 0x7f9c066fe780>
2015-06-15 02:53:50,115 (INFO) ee : Initializing EasyEngine Database
2015-06-15 02:53:50,118 (DEBUG) ee : e[95mRunning command: mysql -e "show variables like 'slow_query_log';" | grep ONe[0m
2015-06-15 02:53:50,211 (DEBUG) ee : e[95mCommand Output: , 
Command Error: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")
e[0m
2015-06-15 02:53:50,211 (INFO) ee : e[94mSetting up MySQL slow loge[0m
2015-06-15 02:57:54,795 (DEBUG) ee : logging initialized for 'ee' using LoggingLogHandler
2015-06-15 02:57:55,106 (DEBUG) ee : ['/usr/local/bin/ee', 'update']
2015-06-15 02:57:55,106 (DEBUG) ee : collecting arguments/commands for <ee.cli.controllers.base.EEBaseController object at 0x7fc7e5fb85f8>
2015-06-15 02:57:55,110 (DEBUG) ee : collecting arguments/commands for <ee.cli.plugins.update.EEUpdateController object at 0x7fc7e5fb84e0>
2015-06-15 02:57:55,114 (INFO) ee : Initializing EasyEngine Database
2015-06-15 02:57:55,117 (INFO) ee : e[94mDownloading update script       e[0m
2015-06-15 02:57:56,069 (INFO) ee : e[94m[e[0mDonee[94m]e[0m
2015-06-15 02:57:56,069 (INFO) ee : e[94mupdating EasyEngine, please wait...e[0m
2015-06-15 02:58:06,869 (DEBUG) ee : logging initialized for 'ee' using LoggingLogHandler
2015-06-15 02:58:07,178 (DEBUG) ee : ['/usr/local/bin/ee', '-v']
2015-06-15 02:58:07,178 (DEBUG) ee : collecting arguments/commands for <ee.cli.controllers.base.EEBaseController object at 0x7f8001c555f8>

harshadyeola · June 16, 2015, 12:36pm

@Kamil

can you check for mysql logs in /var/log/ directory.

Murio · June 16, 2015, 12:55pm

@harshadyeola

I have 2 files there: mysql.log and mysql.err. Both are empty.

Regards, Kamil

harshadyeola · June 16, 2015, 1:00pm

@Murio

Have you tried to start you r mysql service. service mysql start?

what does it say?

Murio · June 16, 2015, 1:13pm

@harshadyeola

It says nothing. There is only “[fail]” with “O_O” icon (my server is configured by your team )

Regards, Kamil

dineshjain2911 · June 16, 2015, 8:18pm

@Murio:

Community forum is for providing general support to the queries of EasyEngine.

The issue which you are facing require debugging on the server. You can check Premium Server Support (https://rtcamp.com/products/easyengine-premium-support/) to resolve this issue.

Murio · June 16, 2015, 9:47pm

@dineshjain2911

this is on “Easy Engine” tag, so why I can’t ask Easy Engine specific questions?

Regards, Kamil

dineshjain2911 · June 17, 2015, 9:55am

@Murio

You can surely post queries regarding EasyEngine here.

Considering the problem you are facing the logs does not make clear what issue has occurred on the server.

Possible reason for this issue might be either there is some problem with MySQL server or the site database is corrupted.

To know the exact cause of the issue debugging on the server will be required. For this you need to hire some System Admin. Or else you can try anyone of the following solution at your end which might help to fix the problem.

http://dba.stackexchange.com/questions/8011/any-better-way-out-of-mysql-innodb-log-in-the-future
Remove MySQL and install again. You can restore database from the previous backup of database.

system · November 9, 2018, 1:50pm