How to Configure Robots.txt for WordPress

robots.txt is a text file that is used to restrict robots (more importantly, search engine bots) from indexing and crawling certain parts of your website. Without a robots.txt file, the bots have free reign around your website which is not a good thing at all.

By default WordPress does not come with a robots.txt, so this is something you will need to add and it is beneficial to do so.

The whole point of restricting access to certain areas of your website, is so that you can focus on indexing content that is important.

What happens when you don’t have a robots.txt file?

In short, anything and everything can be indexed.

For WordPress that means the following.

  • /cgi-bin
  • /wp-admin
  • /wp-includes
  • /wp-content/themes
  • /wp-content/plugins/

As you can probably guess, non of the above have any real SEO relevance  and can even hurt your site. Google allows only a finite amount of what is known as link juice, so it makes sense to focus our efforts on our content and pages.

How can we control the indexation of our site?

The solution is really very simple.

Create a text file using your favorite text editor, or if you have Cpanel, just create a new file in the file manager and call is robots.txt.

Copy and paste the following.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/themes
Disallow: /wp-content/plugins/
Allow: /wp-content/uploads
Allow: /feed*

Save the file, if you created it off the server just upload it to the root directory. This is the same directory that wp-admin, includes and content is stored.

That’s it, now you will start to notice some de-indexation of irrelevant information over the next few days and weeks.

If you would like to add other files to the above, just add another rule like so.

Allow: /path_to/folder

Just remember that we omit the site name, the add the path to the folder starting with a forward slash (/).

Meet the Author

Matthew Horne

Matthew Horne is web developer who specializes in optimized development. He also builds custom solutions instead of reverting to plugins. Matthew Has a strong understanding of PHP, JavaScript, jQuery.

12 comments… add one
  • Aditya Nath Jha Apr 24, 2013, 9:03 am

    Most people ignore such basic stuff. But this small drops add up to be the ocean at the end. Thanks for helping newbies like me 😀
    Well, do we have to specify the bots too individually in the Robots.txt ?

    • Matthew Horne Apr 24, 2013, 9:08 am

      No problem man, no you don’t have to specify individually, the above is more than sufficient enough to ensure that only wp-content, posts, pages etc are indexed and crawled.

    • Matthew Horne Apr 24, 2013, 9:09 am

      Saves you having to install another plugin also. 🙂

  • Praveen May 11, 2013, 9:26 am

    Could you please advice regarding my Wordpress Robots.txt file

    • Matthew Horne May 12, 2013, 9:35 am

      You would need to send the robots.txt to me or paste here so I can see.

  • Tony Aug 30, 2013, 3:38 am

    I don’t understand how to get the content from the text editor to wp.. I have notepad++ Thanks

    • Matthew Horne Aug 30, 2013, 11:31 am

      You have to create a file called robots.txt and upload that to your root folder which is where wp-admin, wp-content etc is.

  • Charjan Sep 7, 2013, 1:15 am

    Great post..I will certainly try to help is out

  • Jasmeet Singh Sep 21, 2014, 8:03 pm

    Dear Matthew,

    I didn’t create any robots.txt for my site which I have been using it about one year. It makes by default but now I am not able to change or add anything. If you know about my site running on WordPress then please tell me.

    Thanks

Leave a Comment