How to solve Error establishing a database connection

Rishabhss · April 15, 2021, 9:19pm

I have a wordpress website hosted on easyengine, today it started showing Error establishing database connection all of a sudden.

I tried running df -h command on my VPS, and got this result:

Filesystem      Size  Used Avail Use% Mounted on
udev            464M     0  464M   0% /dev
tmpfs            99M 1004K   98M   1% /run
/dev/vda1        14G   14G     0 100% /
tmpfs           491M     0  491M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           491M     0  491M   0% /sys/fs/cgroup
overlay          14G   14G     0 100% /var/lib/docker/overlay2/50997f18ec12291e23f8f5ee4b3c62f568a8eadb9bf86e38e588eb589f91b24b/merged
overlay          14G   14G     0 100% /var/lib/docker/overlay2/5bd89e5ff66de830590264681dc7c6cfec44bb2857a1a6d85ce1803ae50951a6/merged
overlay          14G   14G     0 100% /var/lib/docker/overlay2/1f80db31b70c8bd9bf84fe7acc11d867362a33c54ccd9ebd16af570e65e31536/merged
overlay          14G   14G     0 100% /var/lib/docker/overlay2/5122c787f1e2ca4c00e8624fdaa883eebaa05622234a5315bf194783b5e33427/merged
overlay          14G   14G     0 100% /var/lib/docker/overlay2/22151c6d902e9e81a66d1532f8d7b87a07e0ef79cde53157feeb719fd79e4615/merged
overlay          14G   14G     0 100% /var/lib/docker/overlay2/3c5042ecc72b84175a13f40a261012b7c387f064aca8d7cc28825e0d44e4ebab/merged
shm              64M     0   64M   0% /var/lib/docker/containers/0c3c5ae8a534813469535610a60c18ec44887bd0e6504e4303952f992e84c0cb/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/a09417ad0d96e5e5f91b88c863cee59f7031e7bac9c78bc553baf5e308d4128d/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/83cca897d31edabd2a74828a5562c578ac6fffe66f511c38c65e417661d6e70f/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/173215364fb4c9df612ef4ec0b7c79de634e51439f8de65ce4b73510294f260d/mounts/shm
tmpfs            99M     0   99M   0% /run/user/0
shm              64M     0   64M   0% /var/lib/docker/containers/5512994e4fddbeaf0e8579511ac261d55b8c64a86b94d18965a9b243f819f6a2/mounts/shm

as you can see this overlay is using all 14G of space.
Is that the reason for database error?
If So, then how to free up this space without breaking my website.

freedog96150 · April 15, 2021, 11:50pm

Chances are this is a log file that has consumed all the available space. If the sql server or any other process is failing and logging the results, it does not take long for that storage to fill up. Can you access the server via SSH?

If so then some quick du -h commands on various directories should start to shed some light. I would start in /var/log or any of the directories where log files are stored.

Once you know what daemon/service is using up log space then it gets easier to clear the logs and change the config files so reduce logging so this does not happen again. Then it is onto fixing the issue that is causing the daemon/service to fail!!

EDIT: Forgot to mention that I like to use

sudo du -h --max-depth=2

just to keep the results compact enough to read and keep drilling into the largest directories until you zero in on the folder that is using all the space.

Rishabhss · April 16, 2021, 6:05am

Should I just manually delete all those log files using Filezilla?

Rishabhss · April 16, 2021, 7:01am

Ok, so I managed to free up a lot of space.

This is the result now:

Filesystem      Size  Used Avail Use% Mounted on
udev            464M     0  464M   0% /dev
tmpfs            99M  924K   98M   1% /run
/dev/vda1        14G  8.9G  4.3G  68% /
tmpfs           491M     0  491M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           491M     0  491M   0% /sys/fs/cgroup
overlay          14G  8.9G  4.3G  68% /var/lib/docker/overlay2/50997f18ec12291e23f8f5ee4b3c62f568a8eadb9bf86e38e588eb589f91b24b/merged
overlay          14G  8.9G  4.3G  68% /var/lib/docker/overlay2/1f80db31b70c8bd9bf84fe7acc11d867362a33c54ccd9ebd16af570e65e31536/merged
overlay          14G  8.9G  4.3G  68% /var/lib/docker/overlay2/5bd89e5ff66de830590264681dc7c6cfec44bb2857a1a6d85ce1803ae50951a6/merged
overlay          14G  8.9G  4.3G  68% /var/lib/docker/overlay2/3c5042ecc72b84175a13f40a261012b7c387f064aca8d7cc28825e0d44e4ebab/merged
shm              64M     0   64M   0% /var/lib/docker/containers/0c3c5ae8a534813469535610a60c18ec44887bd0e6504e4303952f992e84c0cb/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/a09417ad0d96e5e5f91b88c863cee59f7031e7bac9c78bc553baf5e308d4128d/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/173215364fb4c9df612ef4ec0b7c79de634e51439f8de65ce4b73510294f260d/mounts/shm
overlay          14G  8.9G  4.3G  68% /var/lib/docker/overlay2/5122c787f1e2ca4c00e8624fdaa883eebaa05622234a5315bf194783b5e33427/merged
shm              64M     0   64M   0% /var/lib/docker/containers/83cca897d31edabd2a74828a5562c578ac6fffe66f511c38c65e417661d6e70f/mounts/shm
tmpfs            99M     0   99M   0% /run/user/0

But still I am getting Error establishing a database connection error.
Any idea what is causing this issue?

I also tried ee service restart db, but still the same error.

Rishabhss · April 16, 2021, 7:40am

when I use docker ps command, I see this:

CONTAINER ID   IMAGE                           COMMAND                  CREATED       STATUS                          PORTS                                                                      NAMES
*********   easyengine/cron:v4.0.0          "/usr/bin/ofelia dae…"   2 weeks ago   Up 10 minutes                                                                                              ee-cron-scheduler
*********   easyengine/nginx:v4.1.4         "/usr/bin/openresty …"   2 weeks ago   Up 10 minutes                   80/tcp                                                                     9to6livein_nginx_1
*********   easyengine/postfix:v4.1.5       "postfix start-fg"       2 weeks ago   Up 10 minutes                   25/tcp                                                                     9to6livein_postfix_1
*********   easyengine/php:v4.1.6           "docker-entrypoint.s…"   2 weeks ago   Up 10 minutes                   9000/tcp                                                                   9to6livein_php_1
*********   easyengine/mariadb:v4.1.3       "docker-entrypoint.s…"   2 weeks ago   Restarting (1) 37 seconds ago                                                                              services_global-db_1
*********   easyengine/nginx-proxy:v4.1.4   "/app/docker-entrypo…"   2 weeks ago   Up 6 minutes                    0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   services_global-nginx-proxy_1

The mariadb container is restarting continuously. How do I fix this?

Rishabhss · April 16, 2021, 10:08am

I also tried these commands:

#sudo docker-compose down
#sudo docker-compose pull
#sudo docker-compose up -d

, but it didn’t solved the problem.

I checked the Log of docker logs services_global-db_1:

2021-04-16 11:09:28+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.4+maria~focal started.
2021-04-16 11:09:29+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2021-04-16 11:09:29+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.4+maria~focal started.
2021-04-16 11:09:29 0 [Note] mysqld (mysqld 10.5.4-MariaDB-1:10.5.4+maria~focal) starting as process 1 ...
2021-04-16 11:09:29 0 [Warning] You need to use --log-bin to make --binlog-format work.
2021-04-16 11:09:29 0 [Note] InnoDB: Using Linux native AIO
2021-04-16 11:09:29 0 [Note] InnoDB: Uses event mutexes
2021-04-16 11:09:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-04-16 11:09:29 0 [Note] InnoDB: Number of pools: 1
2021-04-16 11:09:29 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
2021-04-16 11:09:29 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2021-04-16 11:09:30 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2021-04-16 11:09:30 0 [Note] InnoDB: Completed initialization of buffer pool
2021-04-16 11:09:30 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction 955232 was in the XA prepared state.
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction 955166 was in the XA prepared state.
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction 955167 was in the XA prepared state.
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction 955173 was in the XA prepared state.
2021-04-16 11:09:30 0 [Note] InnoDB: 4 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo
2021-04-16 11:09:30 0 [Note] InnoDB: Trx id counter is 955233
2021-04-16 11:09:30 0 [Note] InnoDB: 128 rollback segments are active.
2021-04-16 11:09:30 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-04-16 11:09:30 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-04-16 11:09:30 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-04-16 11:09:30 0 [Note] InnoDB: 10.5.4 started; log sequence number 3046186820; transaction id 955235
2021-04-16 11:09:30 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-04-16 11:09:30 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-04-16 11:09:30 0 [Note] InnoDB: Buffer pool(s) load completed at 210416 11:09:30
2021-04-16 11:09:30 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2021-04-16 11:09:30 0 [Note] InnoDB: Starting recovery for XA transactions...
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction 955167 in prepared state after recovery
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction contains changes to 1 rows
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction 955232 in prepared state after recovery
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction contains changes to 1 rows
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction 955173 in prepared state after recovery
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction contains changes to 1 rows
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction 955166 in prepared state after recovery
2021-04-16 11:09:30 0 [Note] InnoDB: Transaction contains changes to 1 rows
2021-04-16 11:09:30 0 [Note] InnoDB: 4 transactions in prepared state after recovery
2021-04-16 11:09:30 0 [Note] Found 4 prepared transaction(s) in InnoDB
2021-04-16 11:09:30 0 [ERROR] Found 4 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions.
2021-04-16 11:09:30 0 [ERROR] Aborting

then attempted to manually run the MariaDB container with
docker run -it --rm --entrypoint /bin/bash easyengine/mariadb:v4.1.3
‘mysqld_safe --tc-heuristic-recover=COMMIT’

It gave me this result:

root@9bbf5a8f4354:/# mysqld_safe --tc-heuristic-recover=COMMIT
210416 11:13:26 mysqld_safe Logging to syslog.
210416 11:13:26 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql

But still I am getting Error establishing a database connection

system · May 16, 2021, 10:08am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.