Websites have become extremely slow. All websites on my server are not responding to any HTTP/S requests (or respond after a 1 minute delay)

[msarhan]

So this issue just began an hour ago, when I was investigating this and writing the post about it Files from my domain are being served with various "X-Powered-By" headers

A new problem has come up, but now this is happening on all the websites hosted on this server. For many of the js/css/image files, no headers for the requests are being sent at all for at least ~50 seconds. The only time any of the websites will load quickly is if the browser has already cached the files (ie a previous visit). If I open the developer console (in any browser) and disable browser caching, the majority of requests (instead of being Status: 200) will remain pending for almost a minute, or they will return a status of FAILED.

I’ve tried different browsers on my 2nd computer, first entirely clearing the browser caches then visiting the websites, and the same thing happens. It will take a minute to load, many of the requests remain pending (or fail) but if it does end up loading, then each subsequent visit will be very fast (unless you open your developer console and disable browser caching then refresh the page. Again this is happening on pretty much every browser across multiple computers.)

So my page load times on my websites have gone from ~2s to almost 1 minute. I don’t have any plugins installed besides WooCommerce and a basic theme. Everything is up to date. At first I thought this issue was only happening to the 1 domain that I linked above (where I was getting served with different “Powered-By” headers, but then I tested as I was writing this and I realised it is happening to all 5 domains on my server. This is totally new, I browse these sites every day and I’d never noticed the issue before. I have not made any changes to the server at all or to any of the sites since they’ve gone online, the only thing is I always update EE to the latest version, and I’ve been on 4.0.9 since it was released.

Below is pics of the page load time of 1 of the sites when your browser happens to have all the files already cached (ie if you happen to be a previous visitor and your browser has cached all the files from your last visit):

If you are a new visitor, or if you happen to open your dev console and disable browser caching then refresh the page, these are the results you’ll get:

The files that are taking the longest for the requests to be sent are always random. Across all of the websites I’ve refreshed several times to see if there are any files in particular that are causing it to stall, but there is nothing consistent about it. It can be a css or js file from the WP core itself, to a file from WooCommerce, to a file from the installed theme. I’ve power cycled the server, disabled/re-enabled all sites, cleaned caches on all sites.

Also as I’ve mentioned, no changes have been made to the config on the server recently, although I did change php execution time but that was months ago, and even then I’ve only changed the execution time for 1 of my sites, the rest have no custom PHP config. And none of the sites have any custom nginx, redis or memcached config. Everything is all basically default.

Speaking of the server, here is how the performance is looking if I am refreshing across multiple browsers & computers & using online load testing tools at the same time. So I don’t think server performance is the issue here.

[cim]

Did you check to see if it’s a code issue? My sites are all running fine with WordPress, no cache, not even plugin cache.

[msarhan]

Yes. Like I said though I’ve been running these sites for months with no code changes (either server side or web side) made ever since they’ve gone online.

[msarhan]

I should I also mention I have tried multiple internet connections. I have 1 fiber optic connection, 2 ADSL2+ connections, all with different providers. Same issue persists across all computers/browsers/ISPs (although 1 ISP takes a slightly different route to the server, but that’s just a fun fact)

[purbeckpixels]

I’ve noticed similar behaviour a handful of times with the sites on my EE server too. Visit a site only for it to show ‘Waiting for example.com…’ for so long that I think my server’s experiencing an issue, only to check another site on the same server and find it loads normally. It’s not happening consistently though, but it’s happened enough to concern me. It’s somewhat of a relief to discover it’s not only me experiencing the issue.

[msarhan]

Have you noticed it happening again lately? I don’t know if it’s coincidence but it seems to be when I am editing a site, I save a page update every minute and refresh to see what it looks like, then make more changes to the design etc. Maybe something is limiting my access — but that wouldn’t really make sense for some of the requests being served fine and some stalling for a minute. And it’s not like ee has ufw in place. I’m on a dedicated server, I tried a fresh install too and the issue did come back at some point. It’s very odd but good to know it’s not just me going crazy.

For now I have started using the staging feature on blogvault, I subscribed to them for real time backups for e-commerce specifically but I didn’t know they had a staging feature included in the plan. I’m doing that just in case it is the server blocking me somehow. But I doubt this is the case. I’ve even contacted the data centre to confirm and there’s nothing on their end blocking or rate limiting me from their end, plus when this is happening I will usually grab an LTE connection and try some of my domains on mobile, for the most part will load quickly. Not 2 second quick as I would usually expect. But at least under 10s.

On my 3 hard wired internet connections, I get the issue. And across multiple computers. And browsers. Very strange indeed.

[purbeckpixels]

Yes, similar issue - pretty sure I’ve been logged into WordPress to make changes to content and design every time it has happened. Was updating a site - https://shafferoffshoresolutions.com - last night and twice I went to check the updates in a tab while logged in and the site hung at ‘Waiting for…’, but when I then clicked on a link in the main menu it loaded immediately. It has happened again once so far this afternoon. It’s an odd and frustrating issue.

Edit: Sites are on a 4GB Hetzner Cloud server - never had an issue with them before EE4. Wondering if Docker’s involved somewhere (EE4 is my first experience with Docker).

[msarhan]

Damn what an annoying problem.

I have also been logged in to WP to make changes to content and design every time this has happened!

I can’t even save files via WP dashboard any more. Like if I want to edit my stylesheet or header.php or whatever via the dashboard, it will just stall when I click save. I’m not sure if the two issues here are related but I am guessing they probably are.

I really like ee / can’t be bothered moving over to wordops so I’m gonna stick this one out until I figure out what the problem is. The network hanging issue has only happened once again since I made this thread, hopefully it will remain that way. But it is weird that now I need to SSH/SCP in to make changes to files all of a sudden. I mean this is across all of my domains, not just 1 in particular. And I used to edit files manually via WP dashboard all the time and out of no where now it stalls (either Err_network_reset) or it’ll just say waiting for … forever. Actually can you check on your domains if you can save changes via theme/plugin editor in WP dashboard? Just add a comment somewhere, no need to make actual code changes. I wonder if we could have the same problem or if mine is completely unrelated? Since you mentioned docker I am curious to see

Edit: I was curious to test out the site you linked, see it loaded fine for me at 2.44 seconds – your 4GB server is definitely more than enough for EE4, I don’t think that’s the issue here. I’ve got 5 woocommerce sites and 2 general sites running on my server all with their own db and redis containers, and I’m barely hitting 1.5GB ram used (out of 16), and my memory settings for each site are not on default (heavily increased PHP ram allowance, redis too, but they’re barely used anyway. My point being, I think your 4GB server is way more than enough.)

[purbeckpixels]

I seem to have fixed whatever was causing the slowness on my EE4 server with a reboot. I don’t like rebooting servers containing live websites unless absolutely necessary, but on this occasion the issue was driving me nuts. Since the reboot everything appears to be normal again, no random slowness in the last several hours and server load is down a lot too. I did notice that during the ‘hangs’ one or two php-fpm processes were maxing CPU out at 100% and then one or two php process would start and do likewise. My guess is that an EE update caused an issue somewhere down the line - I’ve gone from 4.0.1 to 4.0.9 via all point releases without a single server reboot.

I did try editing a theme file via the WordPress admin (usually I use the wp-config.php constant to disable it because of the security implications) and it worked fine both before and after the reboot which, given the CPU/server load I was seeing, suggests the issues are unrelated.

My 4GB server has 15 sites on it! 10 low traffic WordPress sites, a dev Grav site, my own WPackagist fork, a HTML/CDN serving podcast mp3s plus a couple other projects, and it tops out at about 1.8GB of RAM usage. It’s amazing what you can squeeze out of cloud servers these days.

Hope you get to the bottom of what’s causing your server issue. I’ll let you know if mine return…

[cim]

I seem to have fixed whatever was causing the slowness on my EE4 server with a reboot.

Sometimes a reboot fixes most of my problems ¯_(ツ)_/¯

[juliocorrea]

idão aleatória nas últimas horas e a carga do servidor está muito baixa também. Eu notei que durante os ‘hangs’ um ou dois php-fpm processos estavam maximizando a CPU em 100% e então um ou dois php processos começavam e faziam o mesmo. Meu palpite é que uma atualização do EE causou um problema em algum ponto da linha - eu passei de 4.0.1 para 4.0.9 através de todos os lançamentos pontuais sem uma única reinicial

Which command do you use to restart on ee4? reboot?

[purbeckpixels]

Just a quick update to say that the performance issues returned yesterday - same symptoms, random ‘hangs’, slow wp-admin, higher CPU use and server load etc. Another server reboot fixed it. The only change between the two periods of instability was the 4.0.9 to 4.0.10 update. That’s the only pattern I’ve noticed so far. I’m considering moving away from EE4 because of this issue…

[msarhan]

Happened to me yesterday again as well. Lasted about an hour. I’ve already been on 4.0.10 though. I’m going to try increasing the amount of PHP workers per site, see if that helps. If that doesn’t work I will create each site with its own cache and db containers; seems tedious but luckily I can 1 click restore each of those sites after I re-create them…but still, last resort.

You know what’s strange, given the slow down occurs across all of my computers/phones at the same time – yesterday when it happened I randomly thought to connect to a VPN (in my own country) and I browsed to all of the sites and they were all performing normally. I disconnected, and the hanging comes back (again, across all devices and browsers. Connecting via VPN on multiple devices = sites performing normally. Once disconnected, even on my phones, the hangs are back.) I don’t know if that’s just a weird coincidence or what. But if you have a VPN subscription, or if you can boot up a small server and install a OpenVPN on it (check out Streisand on github, super simple to install) – next time it happens to you can you connect via VPN and see if you can access your server normally?

I think it was just a strange coincidence. Because I’m with multiple ISPs anyway and these hangs happen across all of them…although all of my connections use a static IP – and these slow downs tend to happen when I am putting a lot of my own activity on the sites (very frequent page edits etc)…could the server be rate limiting requests from our IP? I will have to go through all the nginx config files to see if there’s anything set to do that.