Help:Controlling search engine indexing

<languages/>

There are a variety of ways in which MediaWiki allows users to attempt to control search engine indexing, commonly termed "noindexing", as the default position is that pages are indexed. Most of the methods rely on using the noindex HTML meta tag, which tells search engines not to index certain pages. Respecting the tag, or the robots.txt file, especially in terms of removing already indexed content, is up to the individual search engine, and in theory they may be ignored entirely.

The control methods are:

1. Controlling an entire namespace, via MediaWiki software settings
2. Controlling classes of pages, via <tvar|robots>MediaWiki:Robots.txt
3. Controlling individual pages by adding the <tvar|code>__NOINDEX__ [[<tvar|magic>Special:MyLanguage/Help:magic word|magic word]] into them, either directly or using a template made to contain it.
4. Controlling pages by adding the <tvar|code>__NOINDEX__ magic word into standard templates used in certain situations.

Software settings and robots.txt

Software settings

Entire namespaces can be noindexed via $wgNamespaceRobotPolicies.

At the same time, <tvar|code>__NOINDEX__ can be disabled in certain namespaces. This is controlled by the MediaWiki software setting <tvar|ExemptFromUserRobotsControl>$wgExemptFromUserRobotsControl, which defaults to <tvar|ContentNamespaces>$wgContentNamespaces.

$wgArticleRobotPolicies can be used to specify different policies for individual pages.

Robots.txt noindexing

System message <tvar|Robots>MediaWiki:Robots.txt can be used to noindex pages. Unlike the software settings, it can be directly edited.

NOINDEX magic word

Individual pages

Individual pages can be noindexed by adding the <tvar|code>__NOINDEX__ [[<tvar|magic>Special:MyLanguage/Help:Magic words|magic word]] into that page. This can be done either directly or using a template designed to contain the magic word (eg <tvar|NOINDEX>{{NOINDEX}}); this can make use of the keyword easier to track.

Pages with the keyword are tracked by MediaWiki, and automatically categorized in a tracking category. The category name is determined by MediaWiki:Noindex-category.

Standard template noindexing

Standard templates can include the <tvar|code>__NOINDEX__ keyword, thereby noindexing pages to which the templates are applied. A template used to flag a particularly sensitive type of page can then automatically noindex it.

INDEX magic word

Individual pages

Individual pages can override namespace noindexing by adding the <tvar|code>__INDEX__ [[<tvar|magic>Special:MyLanguage/Help:magic word|magic word]] into that page. Pages with the keyword are tracked by MediaWiki, and automatically categorized in a tracking category. The category name is determined by <tvar|Index>MediaWiki:Index-category. However, INDEX does not override noindexing via <tvar|Robots>MediaWiki:Robots.txt. It does override <tvar|ArticleRobotsPolicies>$wgArticleRobotPolicies.

Current issues

• bug 24169http://bugzilla.wikimedia.org/show_bug.cgi?id=24169 – "Create an <tvar|code>__NOINDEX__ equivalent to prevent indexing by internal search engine"

This page was moved from mw:Help:Controlling search engine indexing. It's edit history can be viewed at Help:Controlling search engine indexing/edithistory