How to use robots.txt with EE3?


#1

So I used to edit the robots.txt file using the Yoast plugin, but with NGINX the spot to edit the file isn’t there.

I am looking to disallow 4 woocommerce categories on my EE3 install. I’m not sure how to do this so could someone help guide me on what code needs to be used and which file that code needs to be put in?


#2

You can edit a text file named robots.txt. Please refer to this tutorial to write in the file. https://varvy.com/robottxt.html Then put the file in /var/www/example.com/htdocs . Make sure people can see the file by http(s)://example.com/robots.txt

For me, I would use nano editor to do that: nano /var/www/example.com/htdocs/robots.txt


#3

Hrmm, if it’s that simply with EE3/NGINX, then I wonder why the feature is no longer there in Yoast.

Oh well, thank you


#4

Are you using the latest version of Yoast? I don’t know how this error manifests but the latest update included a fix for “Fixes a bug where a dependency wasn’t loaded for the SEO - Tools page.”

(That said I can see the ‘file editor’ option in the previous version)

Soooooo many errors/bug fixes with this plugin :slight_smile:


#5

I am on Version 9.1 of Yoast right now (I keep all of my themes and plugins updated), and my Tools page looks like this. . .

Soooooo many errors/bug fixes with this plugin

And yea. . .I’ve used Yoast for so many years and now it’s starting to become garbage. I tried some SEO Framework plugin that I saw people on Reddit discussing. But it didn’t let me edit individual pages in a way (like all pages with variables) like Yoast does.

I’ve also had Yoast revert a custom meta description for the frontpage of a website I had setup a couple of months back. The next option I try out will be all-in-one SEO I guess.

These types of bugs are annoying but the big one they let slip through that tanked people’s SERPs earlier this year was really bad. I can understand why so many people are switching away from the plugin these days.


#6

I can see the option on both the current and previous Yoast versions (both sites on ee3). But Yoast settings are different (text link counter is turned off).

It’ll probably reappear again in the next round of bug fixes… shouldn’t be more than a couple days for yet another bug fix release :wink:

I’ve used SEO Framework too, it is a lot more stable - mainly because they don’t keep trying to ram more features in every week. It does have page/post level settings too though, just like Yoast.