![]() | MediaWiki:Robots.txt provides the
Robots.txt file for English Wikipedia, telling search engines not to index the specified pages. See the
documentation of {{
NOINDEX}} for a survey of noindexing methods. This interface message or skin may also be documented on MediaWiki.org or translatewiki.net. The page forms part of the MediaWiki interface, and can only be edited by administrators and interface editors. To request a change to the page, add {{ edit fully-protected}} to this page, followed by a description of your request. Consider announcing discussions you add here at Wikipedia:Village pump (technical) to bring more people to the discussion. |
Perhaps these should be included?
Along with the associated talk pages and all archives? rootology ( C)( T) 13:17, 13 September 2008 (UTC)
I strongly disagree with the inclusion of all of AN. Very few AN discussions involve people identifiable in real life. In previous discussions I asked for examples where a Google search on just someone's name showed a major result from AN (top 20 or so) and no one could give me any examples. Being able to search AN is useful, and I don't believe there is sufficient evidence that content on AN creates harm for people in order to justify this inclusion. Dragons flight ( talk) 15:21, 13 September 2008 (UTC)
{{
edit protected}}
Can you disallow my user and user talk?
NonvocalScream (
talk)
16:40, 13 September 2008 (UTC)
{{
NOINDEX}}
on them. Also note,
user-talk pages are automatically noindexed. -
Rjd0060 (
talk)
16:42, 13 September 2008 (UTC)
{{
NOINDEX}}
, is that {{
NOINDEX}}
supposed to automatically apply to all subdirectories?
Geo Swan (
talk)
00:10, 11 December 2009 (UTC)
Is there a reason to keep the bugzilla links? As far as I can see, they are in the original robots.txt to show why various entries have been added and who requested them, etc. Here we don't need to file bugzilla reports, of course, we can use the talk page or just edit the page ourselves, so I don't really see why we should keep those links. We should probably use the comments to explain why the various pages are in the list instead. -- Conti| ✉ 01:25, 15 September 2008 (UTC)
This page (and its subpages and related pages) are being indexed by Google and probably shouldn't be. -- MZMcBride ( talk) 00:56, 8 September 2009 (UTC)
Does WikiProject_Deletion_sorting need to be here? It only contains current and very recently closed deletion discussions, and I was surprised when I couldn't find one by Googling for i.e. 'deletion sorting china'.-- Apoc2400 ( talk) 23:02, 10 January 2010 (UTC)
{{ editprotected}} Could someone update TFD as it has been renamed to "Templates for discussion" (though I'm not sure it's really needed, as templates hardly would end up as the number 1 Google hit for some person) and add Files for deletion and Possibly unfree files, where I see more danger than in templates. An image about oneself might be something to avoid in Google results. However, file deletion discussions don't seem to be too popular (e.g. they weren't in Xfd today until recently), so no one has added them yet. In this table, you'll find the new syntax:
code to add
|
---|
Disallow: /wiki/Wikipedia:Templates_for_deletion/ Disallow: /wiki/Wikipedia%3ATemplates_for_deletion/ Disallow: /wiki/Wikipedia:Files_for_deletion/ Disallow: /wiki/Wikipedia%3AFiles_for_deletion/ Disallow: /wiki/Wikipedia:Possibly_unfree_files/ Disallow: /wiki/Wikipedia%3APossibly_unfree_files/ |
Thank you, -- The Evil IP address ( talk) 20:27, 4 April 2010 (UTC)
{{
editprotected}}
Could you replace the <pre> tag with <source lang="robots">. This would highlight the syntax and thus make it easier to read. Thanks, --
The Evil IP address (
talk)
20:28, 3 June 2010 (UTC)
Hi! Could someone more experienced take a look at b:pt:MediaWiki:Robots.txt to see if I've created it right, please? Any suggestions? Helder 15:23, 21 November 2010 (UTC)
Curtis J Neeley v NAMEMEDIA INC et al, (5:09-cv-05151-JLH)
https://ecf.arwd.uscourts.gov/cgi-bin/DktRpt.pl?33207
Curtis J Neeley has been ordered to attempt to see if googlebot-images can be directed to stay out of the images donated here. The Plaintiff removed them from the articles but they were reverted back in overnight by others. Must Curtis J Neeley sue the Wikipedia Foundation to force the googlebot-images exclusion? It is now either voluntarily exclude this bot or Curtis J Neeley will ask that the Wikipedia Foundation be added for US Title 17 § 106A violations in the ongoing litigation with Google et al.
CurtisNeeley (
talk)
21:14, 7 December 2010 (UTC)
Can someone confirm that the subpages of WP:Copyright problems and WP:Suspected copyright violations (which I just added) won't get indexed? It seems that the CP pages (which have been listed here for a while) don't show up on Google/elsewhere but I'd just like someone else to confirm that the daily subpages won't be indexed individually. Help? VernoWhitney ( talk) 00:09, 20 March 2011 (UTC)
Suggested addition. – xeno talk 19:17, 6 April 2011 (UTC)
Please disallow Category:Wikipedia sock puppetry and all of its subcategories. – xeno talk 12:50, 16 September 2011 (UTC)
Could we have a short description at MediaWiki:Robots.txt making it clear whether this is the robots.txt file for en.wikipedia.org or the robots.txt file for www.mediawiki.org? Even if the two are identical now that may change in the future.
Also, the text does not match http://en.wikipedia.org/robots.txt or http://www.wikimedia.org/robots.txt. Is this an old version? If so, is there a way to keep it synchronized with the actual robots.txt file? -- Guy Macon ( talk) 18:58, 2 May 2012 (UTC)
Would be cool if an admin could fix these:
/wiki/Wikipedia%3Mediation_Committee/ /wiki/Wikipedia_talk%3Mediation_Committee/ /wiki/Wikipedia%3Mediation_Cabal/Cases/
There has to be an A after all the `%3`s to be a valid encoded URL:
/wiki/Wikipedia%3AMediation_Committee/ /wiki/Wikipedia_talk%3AMediation_Committee/ /wiki/Wikipedia%3AMediation_Cabal/Cases/
Thanks! — Preceding unsigned comment added by Cebe.cc ( talk • contribs) 21:47, 6 January 2013 (UTC)
http://en.wikipedia.org/robots.txt
Line 122, is this an error or..? Shouldn't it be Wayback Machine instead of maschine? It doesn't show up in this page either, just the actual robots.txt page. -- 108.211.193.185 ( talk) 14:39, 12 May 2013 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
There are two entries for each of these:
One of them can be removed safely. -- Dalba 16:40, 26 November 2013 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
i am requesting a change in this sites robots.txt due to a error in it here is a description of it
Line 145 Allow: /w/api.php?action=mobileview& Unknown command. Acceptable commands are "User-agent" and "Disallow". A robots.txt file doesn't say what files/directories you can allow but just what you can disallow.
thank you for your time Aarongaming100 ( talk) 16:03, 13 May 2014 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
The old bugzilla.wikimedia.org links need to be replaced with the new phabricator.wikimedia.org links, given that Bugzilla was replaced with Phabricator. Basically, the old links of the form http://bugzilla.wikimedia.org/show_bug.cgi?id=[id] need to be replaced with links of the form https://phabricator.wikimedia.org/T[id+2000]. Gparyani ( talk) 21:35, 28 February 2015 (UTC)
-- [[
User:Edokter]] {{
talk}}
20:04, 1 March 2015 (UTC)
It is noted that a number of users sensibly use their userspace to develop article drafts and to create sandbox content for test edits.
It would therefore be appreciated if consideration be given to adding such sandboxes and drafts to the exclusions here.
The alternative is to place a {{ user sandbox}} or {{ userpace draft}} manually, which I've been informed upsets people who like to trest their userspace with a degree of privacy. Sfan00 IMG ( talk) 12:02, 26 April 2015 (UTC)
Lately a large part of my Google searches give url's like https://en.wikipedia.org/?title=Denmark and https://en.wikipedia.org/?title=Woman while our preferred /wiki/ url is not listed. I guess it's removed as duplicate content of /?title=. Is it possible to disallow /?title= without a big risk of not having it replaced by another url like /wiki/ ? So far I only see /?title= for en.wikipedia.org so I haven't posted to meta:MediaWiki talk:Robots.txt. PrimeHunter ( talk) 15:02, 22 June 2015 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
Per
Wikipedia:Village pump (proposals)/Archive_126#Userpage drafts shown in search engines, there is a consensus to disable indexing for userspace. This is easiest done by adding Disallow: /wiki/User:
immediately below the last entry in the list (Disallow: /wiki/Category%3ANoindexed_pages
).
Thanks,
Mdann52 (
talk)
10:40, 5 July 2015 (UTC)
Thanks for the ping, TheDJ! Is the indexing of user space a recent change? In the distant past I added the __INDEX__ magic word to my user page since I didn't mind having it indexed by search engines, which would imply that something's changed since then. If there's some other cause of this then it'd be good to know what it is rather than piling on quick hacks on top of some other problem, as this doesn't seem emergent enough to require immediate action. I'll start a thread on wikitech-l to see if anyone knows. With respect to this specific request, I have no issue with this request as the __NOINDEX__ magic word doesn't affect our search functionality at all, so you can still patrol the projects in that way. Per TheDJ's recommendation, we can do this via a configuration change; I can have an engineer in the Search Team take a look at that after some initial investigation is performed. -- Dan Garry, Wikimedia Foundation ( talk) 17:11, 6 July 2015 (UTC)
Isn't this bit treated as a comment rather than a rule since it's preceded by #'s?
Wayback Machine entry
|
---|
# Don't allow the Wayback Machine to index user-pages #User-agent: ia_archiver #Disallow: /wiki/User #Disallow: /wiki/Benutzer |
86.90.39.63 ( talk) 22:32, 3 October 2015 (UTC)
-- [[
User:Edokter]] {{
talk}}
22:36, 3 October 2015 (UTC)
User-agent: archive.org_bot Disallow: /wiki/User: Disallow: /wiki/Benutzer:
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
YO 1.23.216.65 ( talk) 19:27, 26 February 2016 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
The Disallow: /wiki/Wikipedia:Long_term_abuse section needs to be updated to refer to Wikipedia:Long-term abuse due to a mass rename of the entire project years ago. The difference is a hyphen, but search engines are now picking up on the reports which previously were excluded. The same goes for the Disallow: /wiki/Wikipedia:Abuse_reports/ section, which was renamed to Wikipedia:Abuse response years ago. It might be better to keep both the old and new names, because there are some straggler subpages on both names. The corresponding talk pages and subpages would also need to be updated. Pteroinae ( talk) 07:08, 3 April 2016 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
Could we also perhaps add Wikipedia:Wikiquette assistance and its subpages/talkpages to NOINDEX because it's materially similar to the other noticeboards already NOINDEXed and could out/pose a privacy concern to those being discussed (or were being discussed, since the place is inactive)? Pteroinae ( talk) 07:08, 3 April 2016 (UTC)
Could you please add
Disallow: /wiki/Wikipedia:Archive.is_RFC Disallow: /wiki/Wikipedia_talk:Archive.is_RFC
These RFC pages mistakenly not placed under already disallowed folders;
Disallow: /wiki/Wikipedia:Requests_for_comment/ Disallow: /wiki/Wikipedia_talk:Requests_for_comment/
PS. I added RFC_5. There is no such page yet, it is just to avoid the extra work when it will be created.
PPS. I read the spec about robots.txt and removed lines like "Disallow: /wiki/Wikipedia:Archive.is_RFC_4", "Disallow: /wiki/Wikipedia:Archive.is_RFC" should cover all pages with this prefix including "Disallow: /wiki/Wikipedia:Archive.is_RFC_4". Only two lines needed. — Preceding unsigned comment added by 78.139.174.106 ( talk) 13:30, 26 May 2016 (UTC)
Google results for "Sarah Beck Mather" brings up this link: Sarah Beck Mather, which is technically not disallowed, because the slash is escaped. If I am not reading this correctly, I'd like a pointer to what's actually happening, but if I'm right, please add %2F counterparts for the appropriate rules with slashes.
Note: This was brought up in the #wikipedia-en-help channel on IRC, and while the article is now blanked (thanks, User:DragonflySixtyseven), other AfDs may be indexed in this manner, against our wishes.
Thanks! -- MarkTraceur ( talk) 16:20, 8 December 2016 (UTC)
Wikipedia%3AArticles_for_deletion%2F
would fix this, however. In my admittedly very basic understanding of this, by escaping the slash, we are now referring to the page Wikipedia%3AArticles_for_deletion%2FSarah_Beck_Mather
as a subpage of wiki/
, instead of Sarah_Beck_Mather
as a subpage of Wikipedia%3AArticles_for_deletion/
. In other words, for this to work, we would have to add Wikipedia%3AArticles_for_deletion%2FSarah_Beck_Mather
to the robots.txt in order to pull it from Google. It would be easier if the MediaWiki developers could somehow prevent our URLs from being able to escape that slash with a %2F.
Mz7 (
talk)
23:15, 31 December 2016 (UTC)![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
I was just coming here to say/note the same. This search has the result:
Wikipedia:Articles for deletion/Anil Dash - Wikipedia /info/en/?search=Wikipedia%3AArticles_for_deletion%2FAnil_Dash This page is an archive of the discussion about the proposed deletion of the article below. This page is no longer live. Further comments should be made on the ...
We currently specify these lines:
Disallow: /wiki/Wikipedia:Articles_for_deletion/ Disallow: /wiki/Wikipedia%3AArticles_for_deletion/
These lines do not match "Wikipedia%3AArticles_for_deletion%2FAnil_Dash".
Do we care about the root page (i.e., Wikipedia:Articles for deletion) being indexed? If not, we could just remove the trailing slashes from these two rules, which would then catch the Anil Dash case and others.
Otherwise, we'll need to add more permutations to the list of disallow directives, which is kind of gross. In either case, we need to act here. -- MZMcBride ( talk) 16:26, 9 January 2017 (UTC)
Disallow: /wiki/Wikipedia%3AArticles_for_deletion%2F
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
Hi Od Mishehu and Legoktm and any other passing admin. Thanks for the recent edits. Can someone please remove the trailing slashes from the other rules? I'm worried about cases like this search, which have < /info/en/?search=Wikipedia%3ARequests_for_comment%2FHipocrite> in the results. -- MZMcBride ( talk) 05:11, 11 January 2017 (UTC)
Hi. I found this piece interesting: < http://www.archiveteam.org/index.php?title=Robots.txt>. -- MZMcBride ( talk) 07:06, 12 January 2017 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
My page Ujwal Ghimire needs indexing so search engines find it. Please add Indexing. Thanks -- Rohkum ( talk) 18:36, 30 January 2017 (UTC) Rohkum ( talk) 18:36, 30 January 2017 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
Please add:
Disallow: /wiki/Module:Sandbox Disallow: /wiki/Module%3ASandbox
Unlike normal templates, Scribunto modules only work in the Module namespace, so what would otherwise be created in the User namespace get created under Module:Sandbox/. Nardog ( talk) 10:28, 2 January 2019 (UTC)
Also add:
Disallow: /wiki/Template:TemplateStyles sandbox Disallow: /wiki/Template%3ATemplateStyles sandbox
for a similar reason. Nardog ( talk) 05:30, 6 September 2020 (UTC)
TemplateStyles sandbox
→ TemplateStyles_sandbox
). Apologies for the inconvenience.
Nardog (
talk)
11:09, 12 September 2020 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
Please add Disallow: /wiki/Talk:
and Disallow: /wiki/Talk%3A
per
Wikipedia:Village_pump_(proposals)/Archive_169#Thoughts_on_deindexing_(some)_non-content_namespaces. (I know the discussion is from a month ago, but I did not know if there was consensus for the move, but looking a second time, it appears that there is a rough consensus to deindex article talk pages.)
A
a
s
i
m
05:57, 19 August 2020 (UTC)
$wgNamespaceRobotPolicies
parameters - which will require a phab request - which will also be requiring a well-attended, strongly supported discussion. —
xaosflux
Talk
14:15, 19 August 2020 (UTC)Someone may want to remove the syntax validator URL from the comments, as it now redirects to a completely different site. Trivialist ( talk) 16:06, 29 May 2021 (UTC)
COIBot is creating reports related to spamming/link abuse which are currently {{ NOINDEX}}ed by the addition of a template. User:Asartea suggested to have them added here, therefore: can the following 4 pages and subpages of them (thousands of reports) be NOINDEXed through robots.txt please: Wikipedia:WikiProject Spam/COIReports, Wikipedia:WikiProject Spam/LinkReports, Wikipedia:WikiProject Spam/UserReports, and Wikipedia:WikiProject Spam/PageReports? Dirk Beetstra T C 12:48, 23 January 2022 (UTC)
Google can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippetSuffusion of Yellow ( talk) 20:10, 23 January 2022 (UTC)
<meta name="robots" content="noindex,follow"/>
in the {{
NOINDEX}}ed page? It's not supposed to access it.
Suffusion of Yellow (
talk)
20:46, 24 January 2022 (UTC)
![]() | MediaWiki:Robots.txt provides the
Robots.txt file for English Wikipedia, telling search engines not to index the specified pages. See the
documentation of {{
NOINDEX}} for a survey of noindexing methods. This interface message or skin may also be documented on MediaWiki.org or translatewiki.net. The page forms part of the MediaWiki interface, and can only be edited by administrators and interface editors. To request a change to the page, add {{ edit fully-protected}} to this page, followed by a description of your request. Consider announcing discussions you add here at Wikipedia:Village pump (technical) to bring more people to the discussion. |
Perhaps these should be included?
Along with the associated talk pages and all archives? rootology ( C)( T) 13:17, 13 September 2008 (UTC)
I strongly disagree with the inclusion of all of AN. Very few AN discussions involve people identifiable in real life. In previous discussions I asked for examples where a Google search on just someone's name showed a major result from AN (top 20 or so) and no one could give me any examples. Being able to search AN is useful, and I don't believe there is sufficient evidence that content on AN creates harm for people in order to justify this inclusion. Dragons flight ( talk) 15:21, 13 September 2008 (UTC)
{{
edit protected}}
Can you disallow my user and user talk?
NonvocalScream (
talk)
16:40, 13 September 2008 (UTC)
{{
NOINDEX}}
on them. Also note,
user-talk pages are automatically noindexed. -
Rjd0060 (
talk)
16:42, 13 September 2008 (UTC)
{{
NOINDEX}}
, is that {{
NOINDEX}}
supposed to automatically apply to all subdirectories?
Geo Swan (
talk)
00:10, 11 December 2009 (UTC)
Is there a reason to keep the bugzilla links? As far as I can see, they are in the original robots.txt to show why various entries have been added and who requested them, etc. Here we don't need to file bugzilla reports, of course, we can use the talk page or just edit the page ourselves, so I don't really see why we should keep those links. We should probably use the comments to explain why the various pages are in the list instead. -- Conti| ✉ 01:25, 15 September 2008 (UTC)
This page (and its subpages and related pages) are being indexed by Google and probably shouldn't be. -- MZMcBride ( talk) 00:56, 8 September 2009 (UTC)
Does WikiProject_Deletion_sorting need to be here? It only contains current and very recently closed deletion discussions, and I was surprised when I couldn't find one by Googling for i.e. 'deletion sorting china'.-- Apoc2400 ( talk) 23:02, 10 January 2010 (UTC)
{{ editprotected}} Could someone update TFD as it has been renamed to "Templates for discussion" (though I'm not sure it's really needed, as templates hardly would end up as the number 1 Google hit for some person) and add Files for deletion and Possibly unfree files, where I see more danger than in templates. An image about oneself might be something to avoid in Google results. However, file deletion discussions don't seem to be too popular (e.g. they weren't in Xfd today until recently), so no one has added them yet. In this table, you'll find the new syntax:
code to add
|
---|
Disallow: /wiki/Wikipedia:Templates_for_deletion/ Disallow: /wiki/Wikipedia%3ATemplates_for_deletion/ Disallow: /wiki/Wikipedia:Files_for_deletion/ Disallow: /wiki/Wikipedia%3AFiles_for_deletion/ Disallow: /wiki/Wikipedia:Possibly_unfree_files/ Disallow: /wiki/Wikipedia%3APossibly_unfree_files/ |
Thank you, -- The Evil IP address ( talk) 20:27, 4 April 2010 (UTC)
{{
editprotected}}
Could you replace the <pre> tag with <source lang="robots">. This would highlight the syntax and thus make it easier to read. Thanks, --
The Evil IP address (
talk)
20:28, 3 June 2010 (UTC)
Hi! Could someone more experienced take a look at b:pt:MediaWiki:Robots.txt to see if I've created it right, please? Any suggestions? Helder 15:23, 21 November 2010 (UTC)
Curtis J Neeley v NAMEMEDIA INC et al, (5:09-cv-05151-JLH)
https://ecf.arwd.uscourts.gov/cgi-bin/DktRpt.pl?33207
Curtis J Neeley has been ordered to attempt to see if googlebot-images can be directed to stay out of the images donated here. The Plaintiff removed them from the articles but they were reverted back in overnight by others. Must Curtis J Neeley sue the Wikipedia Foundation to force the googlebot-images exclusion? It is now either voluntarily exclude this bot or Curtis J Neeley will ask that the Wikipedia Foundation be added for US Title 17 § 106A violations in the ongoing litigation with Google et al.
CurtisNeeley (
talk)
21:14, 7 December 2010 (UTC)
Can someone confirm that the subpages of WP:Copyright problems and WP:Suspected copyright violations (which I just added) won't get indexed? It seems that the CP pages (which have been listed here for a while) don't show up on Google/elsewhere but I'd just like someone else to confirm that the daily subpages won't be indexed individually. Help? VernoWhitney ( talk) 00:09, 20 March 2011 (UTC)
Suggested addition. – xeno talk 19:17, 6 April 2011 (UTC)
Please disallow Category:Wikipedia sock puppetry and all of its subcategories. – xeno talk 12:50, 16 September 2011 (UTC)
Could we have a short description at MediaWiki:Robots.txt making it clear whether this is the robots.txt file for en.wikipedia.org or the robots.txt file for www.mediawiki.org? Even if the two are identical now that may change in the future.
Also, the text does not match http://en.wikipedia.org/robots.txt or http://www.wikimedia.org/robots.txt. Is this an old version? If so, is there a way to keep it synchronized with the actual robots.txt file? -- Guy Macon ( talk) 18:58, 2 May 2012 (UTC)
Would be cool if an admin could fix these:
/wiki/Wikipedia%3Mediation_Committee/ /wiki/Wikipedia_talk%3Mediation_Committee/ /wiki/Wikipedia%3Mediation_Cabal/Cases/
There has to be an A after all the `%3`s to be a valid encoded URL:
/wiki/Wikipedia%3AMediation_Committee/ /wiki/Wikipedia_talk%3AMediation_Committee/ /wiki/Wikipedia%3AMediation_Cabal/Cases/
Thanks! — Preceding unsigned comment added by Cebe.cc ( talk • contribs) 21:47, 6 January 2013 (UTC)
http://en.wikipedia.org/robots.txt
Line 122, is this an error or..? Shouldn't it be Wayback Machine instead of maschine? It doesn't show up in this page either, just the actual robots.txt page. -- 108.211.193.185 ( talk) 14:39, 12 May 2013 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
There are two entries for each of these:
One of them can be removed safely. -- Dalba 16:40, 26 November 2013 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
i am requesting a change in this sites robots.txt due to a error in it here is a description of it
Line 145 Allow: /w/api.php?action=mobileview& Unknown command. Acceptable commands are "User-agent" and "Disallow". A robots.txt file doesn't say what files/directories you can allow but just what you can disallow.
thank you for your time Aarongaming100 ( talk) 16:03, 13 May 2014 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
The old bugzilla.wikimedia.org links need to be replaced with the new phabricator.wikimedia.org links, given that Bugzilla was replaced with Phabricator. Basically, the old links of the form http://bugzilla.wikimedia.org/show_bug.cgi?id=[id] need to be replaced with links of the form https://phabricator.wikimedia.org/T[id+2000]. Gparyani ( talk) 21:35, 28 February 2015 (UTC)
-- [[
User:Edokter]] {{
talk}}
20:04, 1 March 2015 (UTC)
It is noted that a number of users sensibly use their userspace to develop article drafts and to create sandbox content for test edits.
It would therefore be appreciated if consideration be given to adding such sandboxes and drafts to the exclusions here.
The alternative is to place a {{ user sandbox}} or {{ userpace draft}} manually, which I've been informed upsets people who like to trest their userspace with a degree of privacy. Sfan00 IMG ( talk) 12:02, 26 April 2015 (UTC)
Lately a large part of my Google searches give url's like https://en.wikipedia.org/?title=Denmark and https://en.wikipedia.org/?title=Woman while our preferred /wiki/ url is not listed. I guess it's removed as duplicate content of /?title=. Is it possible to disallow /?title= without a big risk of not having it replaced by another url like /wiki/ ? So far I only see /?title= for en.wikipedia.org so I haven't posted to meta:MediaWiki talk:Robots.txt. PrimeHunter ( talk) 15:02, 22 June 2015 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
Per
Wikipedia:Village pump (proposals)/Archive_126#Userpage drafts shown in search engines, there is a consensus to disable indexing for userspace. This is easiest done by adding Disallow: /wiki/User:
immediately below the last entry in the list (Disallow: /wiki/Category%3ANoindexed_pages
).
Thanks,
Mdann52 (
talk)
10:40, 5 July 2015 (UTC)
Thanks for the ping, TheDJ! Is the indexing of user space a recent change? In the distant past I added the __INDEX__ magic word to my user page since I didn't mind having it indexed by search engines, which would imply that something's changed since then. If there's some other cause of this then it'd be good to know what it is rather than piling on quick hacks on top of some other problem, as this doesn't seem emergent enough to require immediate action. I'll start a thread on wikitech-l to see if anyone knows. With respect to this specific request, I have no issue with this request as the __NOINDEX__ magic word doesn't affect our search functionality at all, so you can still patrol the projects in that way. Per TheDJ's recommendation, we can do this via a configuration change; I can have an engineer in the Search Team take a look at that after some initial investigation is performed. -- Dan Garry, Wikimedia Foundation ( talk) 17:11, 6 July 2015 (UTC)
Isn't this bit treated as a comment rather than a rule since it's preceded by #'s?
Wayback Machine entry
|
---|
# Don't allow the Wayback Machine to index user-pages #User-agent: ia_archiver #Disallow: /wiki/User #Disallow: /wiki/Benutzer |
86.90.39.63 ( talk) 22:32, 3 October 2015 (UTC)
-- [[
User:Edokter]] {{
talk}}
22:36, 3 October 2015 (UTC)
User-agent: archive.org_bot Disallow: /wiki/User: Disallow: /wiki/Benutzer:
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
YO 1.23.216.65 ( talk) 19:27, 26 February 2016 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
The Disallow: /wiki/Wikipedia:Long_term_abuse section needs to be updated to refer to Wikipedia:Long-term abuse due to a mass rename of the entire project years ago. The difference is a hyphen, but search engines are now picking up on the reports which previously were excluded. The same goes for the Disallow: /wiki/Wikipedia:Abuse_reports/ section, which was renamed to Wikipedia:Abuse response years ago. It might be better to keep both the old and new names, because there are some straggler subpages on both names. The corresponding talk pages and subpages would also need to be updated. Pteroinae ( talk) 07:08, 3 April 2016 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
Could we also perhaps add Wikipedia:Wikiquette assistance and its subpages/talkpages to NOINDEX because it's materially similar to the other noticeboards already NOINDEXed and could out/pose a privacy concern to those being discussed (or were being discussed, since the place is inactive)? Pteroinae ( talk) 07:08, 3 April 2016 (UTC)
Could you please add
Disallow: /wiki/Wikipedia:Archive.is_RFC Disallow: /wiki/Wikipedia_talk:Archive.is_RFC
These RFC pages mistakenly not placed under already disallowed folders;
Disallow: /wiki/Wikipedia:Requests_for_comment/ Disallow: /wiki/Wikipedia_talk:Requests_for_comment/
PS. I added RFC_5. There is no such page yet, it is just to avoid the extra work when it will be created.
PPS. I read the spec about robots.txt and removed lines like "Disallow: /wiki/Wikipedia:Archive.is_RFC_4", "Disallow: /wiki/Wikipedia:Archive.is_RFC" should cover all pages with this prefix including "Disallow: /wiki/Wikipedia:Archive.is_RFC_4". Only two lines needed. — Preceding unsigned comment added by 78.139.174.106 ( talk) 13:30, 26 May 2016 (UTC)
Google results for "Sarah Beck Mather" brings up this link: Sarah Beck Mather, which is technically not disallowed, because the slash is escaped. If I am not reading this correctly, I'd like a pointer to what's actually happening, but if I'm right, please add %2F counterparts for the appropriate rules with slashes.
Note: This was brought up in the #wikipedia-en-help channel on IRC, and while the article is now blanked (thanks, User:DragonflySixtyseven), other AfDs may be indexed in this manner, against our wishes.
Thanks! -- MarkTraceur ( talk) 16:20, 8 December 2016 (UTC)
Wikipedia%3AArticles_for_deletion%2F
would fix this, however. In my admittedly very basic understanding of this, by escaping the slash, we are now referring to the page Wikipedia%3AArticles_for_deletion%2FSarah_Beck_Mather
as a subpage of wiki/
, instead of Sarah_Beck_Mather
as a subpage of Wikipedia%3AArticles_for_deletion/
. In other words, for this to work, we would have to add Wikipedia%3AArticles_for_deletion%2FSarah_Beck_Mather
to the robots.txt in order to pull it from Google. It would be easier if the MediaWiki developers could somehow prevent our URLs from being able to escape that slash with a %2F.
Mz7 (
talk)
23:15, 31 December 2016 (UTC)![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
I was just coming here to say/note the same. This search has the result:
Wikipedia:Articles for deletion/Anil Dash - Wikipedia /info/en/?search=Wikipedia%3AArticles_for_deletion%2FAnil_Dash This page is an archive of the discussion about the proposed deletion of the article below. This page is no longer live. Further comments should be made on the ...
We currently specify these lines:
Disallow: /wiki/Wikipedia:Articles_for_deletion/ Disallow: /wiki/Wikipedia%3AArticles_for_deletion/
These lines do not match "Wikipedia%3AArticles_for_deletion%2FAnil_Dash".
Do we care about the root page (i.e., Wikipedia:Articles for deletion) being indexed? If not, we could just remove the trailing slashes from these two rules, which would then catch the Anil Dash case and others.
Otherwise, we'll need to add more permutations to the list of disallow directives, which is kind of gross. In either case, we need to act here. -- MZMcBride ( talk) 16:26, 9 January 2017 (UTC)
Disallow: /wiki/Wikipedia%3AArticles_for_deletion%2F
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
Hi Od Mishehu and Legoktm and any other passing admin. Thanks for the recent edits. Can someone please remove the trailing slashes from the other rules? I'm worried about cases like this search, which have < /info/en/?search=Wikipedia%3ARequests_for_comment%2FHipocrite> in the results. -- MZMcBride ( talk) 05:11, 11 January 2017 (UTC)
Hi. I found this piece interesting: < http://www.archiveteam.org/index.php?title=Robots.txt>. -- MZMcBride ( talk) 07:06, 12 January 2017 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
My page Ujwal Ghimire needs indexing so search engines find it. Please add Indexing. Thanks -- Rohkum ( talk) 18:36, 30 January 2017 (UTC) Rohkum ( talk) 18:36, 30 January 2017 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
Please add:
Disallow: /wiki/Module:Sandbox Disallow: /wiki/Module%3ASandbox
Unlike normal templates, Scribunto modules only work in the Module namespace, so what would otherwise be created in the User namespace get created under Module:Sandbox/. Nardog ( talk) 10:28, 2 January 2019 (UTC)
Also add:
Disallow: /wiki/Template:TemplateStyles sandbox Disallow: /wiki/Template%3ATemplateStyles sandbox
for a similar reason. Nardog ( talk) 05:30, 6 September 2020 (UTC)
TemplateStyles sandbox
→ TemplateStyles_sandbox
). Apologies for the inconvenience.
Nardog (
talk)
11:09, 12 September 2020 (UTC)
![]() | This
edit request has been answered. Set the |answered= or |ans= parameter to no to reactivate your request. |
Please add Disallow: /wiki/Talk:
and Disallow: /wiki/Talk%3A
per
Wikipedia:Village_pump_(proposals)/Archive_169#Thoughts_on_deindexing_(some)_non-content_namespaces. (I know the discussion is from a month ago, but I did not know if there was consensus for the move, but looking a second time, it appears that there is a rough consensus to deindex article talk pages.)
A
a
s
i
m
05:57, 19 August 2020 (UTC)
$wgNamespaceRobotPolicies
parameters - which will require a phab request - which will also be requiring a well-attended, strongly supported discussion. —
xaosflux
Talk
14:15, 19 August 2020 (UTC)Someone may want to remove the syntax validator URL from the comments, as it now redirects to a completely different site. Trivialist ( talk) 16:06, 29 May 2021 (UTC)
COIBot is creating reports related to spamming/link abuse which are currently {{ NOINDEX}}ed by the addition of a template. User:Asartea suggested to have them added here, therefore: can the following 4 pages and subpages of them (thousands of reports) be NOINDEXed through robots.txt please: Wikipedia:WikiProject Spam/COIReports, Wikipedia:WikiProject Spam/LinkReports, Wikipedia:WikiProject Spam/UserReports, and Wikipedia:WikiProject Spam/PageReports? Dirk Beetstra T C 12:48, 23 January 2022 (UTC)
Google can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippetSuffusion of Yellow ( talk) 20:10, 23 January 2022 (UTC)
<meta name="robots" content="noindex,follow"/>
in the {{
NOINDEX}}ed page? It's not supposed to access it.
Suffusion of Yellow (
talk)
20:46, 24 January 2022 (UTC)