<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>Great!</div>
<div><br>
</div>
<div>I just went down this road here at Georgian Court.</div>
<div>We launched our new site about a month ago, and old media files were coming up in search results. Our old media directory was /uploadedFiles (thousands of .pdfs, .jpgs, etc.)</div>
<div><br>
</div>
<div>I built the new site from scratch, as if we never had a website, so any files that went on the new site were provided by their department, guaranteeing that they were the most updated.</div>
<div><br>
</div>
<div>Wordpress does not use this directory, so I blocked the directory in webmaster tools/Remove URLs
<i><b>and</b></i> Disallow: /uploadedFiles/ in robots.txt</div>
<div><br>
</div>
<div>I am currently going through analytics/content drill down to find other unused directories. I also took screen shots of our old web directory structure. Comparing the two, I can remove additional directories.</div>
<div><br>
</div>
<div>Glad this worked out for you Ben.</div>
<div>
<div><b style="font-family: 'Times New Roman', serif; font-size: 16px;"><span style="font-family: 'Lucida Calligraphy'; color: rgb(79, 129, 189);"><br>
</span></b></div>
<div><span style="color: rgb(79, 129, 189); font-size: 16px;"><font face="Arial"><b>Richard Berardi</b></font></span></div>
<div><span style="font-size: 10pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);"><b>Web Administrator</b><br>
Marketing & Communications<br>
Georgian Court University<br>
900 Lakewood Ave.<br>
Lakewood, NJ 08701<br>
p: <a href="tel:732-987-2469" target="_blank">732-987-2469</a><br>
e: <a href="mailto:rberardi@georgian.edu" target="_blank" title="mailto:rberardi@georgian.edu
mailto:greenhalgh@georgian.edu">rberardi@georgian.edu</a></span></div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>wp-edu <<a href="mailto:wp-edu-bounces@lists.automattic.com">wp-edu-bounces@lists.automattic.com</a>> on behalf of Ben Bakelaar <<a href="mailto:bakelaar@rutgers.edu">bakelaar@rutgers.edu</a>><br>
<span style="font-weight:bold">Reply-To: </span>"Low-traffic list discussing WordPress in education." <<a href="mailto:wp-edu@lists.automattic.com">wp-edu@lists.automattic.com</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, September 16, 2015 at 4:04 PM<br>
<span style="font-weight:bold">To: </span>"Low-traffic list discussing WordPress in education." <<a href="mailto:wp-edu@lists.automattic.com">wp-edu@lists.automattic.com</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:jon.oliver@rutgers.edu">jon.oliver@rutgers.edu</a>" <<a href="mailto:jon.oliver@rutgers.edu">jon.oliver@rutgers.edu</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [wp-edu] Uploads folder content indexed in Google?<br>
</div>
<div><br>
</div>
<div>
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New","serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1f497d">Yes, that was it! Thanks Richard.</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">Created robots.txt file:</span></p>
<p class="MsoNormal"><span style="font-size: 10pt; font-family: 'Courier New', serif; color: black;">User-agent: *</span></p>
<p class="MsoNormal"><span style="font-size: 10pt; font-family: 'Courier New', serif; color: black;">Disallow: /wp-content/uploads/sites/7/</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">Tested with Fetch as Googlebot. Went to Google Index > Remove URLs within Google Webmaster Tools. Clicked on “Create a new removal request”. Entered directory name only:</span></p>
<p class="MsoNormal"><a href="http://eclipse.rutgers.edu/wp-content/uploads/sites/7" target="_blank"><span style="font-size: 10pt; font-family: Arial, sans-serif; color: rgb(102, 17, 204); background-color: white; text-decoration: none; background-position: initial initial; background-repeat: initial initial;">/wp-content/uploads/sites/7<img border="0" width="12" height="12" id="Picture_x0020_1" src="cid:image001.png@01D0F099.57048180" alt="https://www.google.com/webmasters/tools/images/url_icon.png"></span></a><span style="color:#1f497d"></span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">On the next screen, the third option on the drop-down menu is “Remove directory” which normally is not there if you enter a full URL. Submitted and pending!</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">I also turned off “Indexes” in httpd.conf, not sure if it was on for “all” sub-sites before. But that’s my working theory, that directory listings were turned on and somehow search bots got to those pages and
then they indexed each file and sub-dir listed.</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">I guess my outdated conception was that search engine bots only scour the web via finding extant links within HTML documents – so if there is no public link to that content, it will never get indexed. It appears
they may be more aggressive now, using algorithms to predict sub-directories (perhaps based on CMS detection?) and then scan for available content? Just a working theory since I still can’t explain it.</span></p>
<p class="MsoNormal"><span style="font-family: Tahoma, sans-serif; color: rgb(127, 127, 127);"> </span></p>
<p class="MsoNormal"><span style="font-family: Tahoma, sans-serif; color: rgb(127, 127, 127);">---------------------------------<br>
BEN BAKELAAR | IT Services<br>
School of Communication and Information </span></p>
<p class="MsoNormal"><span style="font-family: Tahoma, sans-serif; color: rgb(127, 127, 127);">Rutgers, The State University of New Jersey<br>
p 848.932.8710</span><span style="color:#7f7f7f"></span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;"> wp-edu [mailto:<a href="mailto:wp-edu-bounces@lists.automattic.com">wp-edu-bounces@lists.automattic.com</a>]
<b>On Behalf Of </b>Berardi, Richard<br>
<b>Sent:</b> Tuesday, September 15, 2015 9:22 PM<br>
<b>To:</b> Low-traffic list discussing WordPress in education.<br>
<b>Cc:</b> <a href="mailto:jon.oliver@rutgers.edu">jon.oliver@rutgers.edu</a><br>
<b>Subject:</b> Re: [wp-edu] Uploads folder content indexed in Google?</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal"><b>Removing an entire directory or site</b><br>
<br>
In order for a directory or site-wide removal to be successful, the directory or site must be <i>disallowed in the site's <a href="http://www.google.com/support/webmasters/bin/answer.py?answer=35302"><span style="text-decoration:none">robots.txt file</span></a></i>.
For example, in order to remove the <a href="http://www.example.com/secret/">http://www.example.com/secret/</a> directory, your robots.txt file would need to include:<br>
User-agent: *<br>
Disallow: /secret/<br>
It isn't enough for the root of the directory to return a 404 status code, because it's possible for a directory to return a 404 but still serve out files underneath it. Using robots.txt to block a directory (or an entire site) ensures that all the URLs under
that directory (or site) are blocked as well. You can test whether a directory has been blocked correctly using either the <a href="http://www.google.com/support/webmasters/bin/answer.py?answer=158587"><span style="text-decoration:none">Fetch as Googlebot</span></a> or <a href="http://www.google.com/support/webmasters/bin/answer.py?answer=156449"><span style="text-decoration:none">Test
robots.txt</span></a> features in Webmaster Tools.<br>
<br>
Only verified owners of a site can request removal of an entire site or directory in Webmaster Tools. To request removal of a directory or site, click on the site in question, then go to <i>Site configuration > Crawler access > Remove URL</i>. If you enter
the root of your site as the URL you want to remove, you'll be asked to confirm that you want to remove the entire site. If you enter a subdirectory, select the "Remove directory" option from the drop-down menu.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><a href="http://googlewebmastercentral.blogspot.com/2010/03/url-removal-explained-part-i-urls.html?m=1">http://googlewebmastercentral.blogspot.com/2010/03/url-removal-explained-part-i-urls.html?m=1</a><br>
<br>
Hope this helps.<br>
<br>
Sent from my iPhone 6</p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
On Sep 15, 2015, at 5:59 PM, Ben Bakelaar <<a href="mailto:bakelaar@rutgers.edu">bakelaar@rutgers.edu</a>> wrote:</p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hello all, it appears we have had some of the files on our Wordpress network indexed in Google search results. I had assumed security through obscurity here, but it appears I was wrong.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Our network runs sites as sub-directories, and we also use domain mapping for some of them. I haven’t quite figured out how yet, but one of the mapped domains (xyz, not
<a href="http://root.url.com">root.url.com</a>) which points to site A has shown up in search results with absolute paths to files in a completely different site B (which is actually a sub-dir site, not masked). And they load just fine – this must be an unanticipated
quirk of DNS records + the Wordpress code that routes requests.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">So we have URLs like xyz.domain/wp-content/uploads/sites/x/xxxx/xx/filename.doc coming up in results! Eek! I have already started the removal requests via Google Webmaster Tools. Again no explanation yet for how these URLs were located
by the search engines, but I’m working on possible theories.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Aside from getting to the bottom of this, I’m trying to figure out the best way to block this from happening in the future. Apache .htaccess rules are one option. Robots.txt could be another? Has anyone run into this issue before, and what
have you done as a solution? I’m a little surprised this isn’t addressed “in code”. There are many plugins that allow uploads, this is a desired and supported user behavior by default. But there are no conceivable use cases I can think of where those uploads
should be able to be indexed by bots.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Could I simply place robots.txt in the root of the WP codebase, and tell it to avoid indexing ALL files under /wp-content? Would that cover all the various access cases with direct-linked files (like graphics), domain masking/mapping, etc.?
And to fully prevent opening any uploads from outside the university network (as a decent but arbitrary perimeter), can I do the same with .htaccess or do I have to make dozens of .htaccess files per /wp-content/uploads/sites/X – in each little sub-directory?</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><span style="font-family: Tahoma, sans-serif; color: rgb(127, 127, 127);"> </span></p>
<p class="MsoNormal"><span style="font-family: Tahoma, sans-serif; color: rgb(127, 127, 127);">---------------------------------<br>
BEN BAKELAAR | IT Services<br>
School of Communication and Information </span></p>
<p class="MsoNormal"><span style="font-family: Tahoma, sans-serif; color: rgb(127, 127, 127);">Rutgers, The State University of New Jersey<br>
p 848.932.8710</span></p>
<p class="MsoNormal"> </p>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size: 12pt; font-family: 'Times New Roman', serif;">_______________________________________________<br>
wp-edu mailing list<br>
<a href="mailto:wp-edu@lists.automattic.com">wp-edu@lists.automattic.com</a><br>
<a href="http://lists.automattic.com/mailman/listinfo/wp-edu">http://lists.automattic.com/mailman/listinfo/wp-edu</a></span></p>
</div>
</blockquote>
</div>
</div>
</div>
</span>
</body>
</html>