[wp-hackers] wp_remote_request not telling me the 301'd URL

[Scott Kingsley Clark]

The spidering process can really take a lot of time for a large site, and 
can end up eating resources and adding time to the infamous php 
max_execution_time so I was looking to cut corners. If I've gotta do two 
requests to do this, I'll do it. Thanks for the advice and attention.

-Scott

On Monday, February 28, 2011 5:28:54 PM UTC-6, Jacob Santos wrote:
>
> Not really. The wp_remote_request simply defaults to GET, you can change it
> to be HEAD, which is what it seems like you are wanting anyway. You can
> check to see if it is a redirect and then send another request. It does not
> sound like speed is a concern (albeit one factor since many sites can quite
> frankly get up there with the amount of redirects given Canonical URLs 
> might
> give you (Hint: Should be at most 2 requests, one for the redirect and one
> for the actual page).
>
> You'll probably want to use wp_remote_head() instead, since
> wp_remote_request() is a generic function made to accommodated the rest of
> the HTTP and HTTP extensions (there isn't any built-in calls support for
> Subversion or webdav).
>
> Jacob Santos
>
> On Mon, Feb 28, 2011 at 5:22 PM, Scott Kingsley Clark <sc... at skcdev.com
> >wrote:
>
> > Actually, this is in regards to a plugin I'm currently developing. It's 
> in
> > Beta right now but it's available on WP.org. It's called Search Engine 
> and
> > it's like a mini-Google on your site. It spiders your site (or other 
> sites
> > too) and indexes content into the DB.
> >
> > http://wordpress.org/extend/plugins/search-engine/
> >
> > <http://wordpress.org/extend/plugins/search-engine/>The use-case is that 
> I
> > want to be able to tell whether a page that's linked to on a site, is
> > really
> > redirected elsewhere. Right now, since I switched to wp_remote_request, I
> > only get the content of the final destination page, without any knowledge
> > of
> > the path it's taken. So the best my script (or any script) can tell is 
> that
> > when you get content using wp_remote_request and it's redirected, there
> > page
> > exists at the URL requested -- oblivious to the real redirect happening.
> > Previously I was using a home-brewed version similar
> > to wp_remote_request but calling cURL and others manually).
> >
> > So it looks like right now I'll need to do a little extra code to make my
> > own wp_remote_request like function which does both the 301/302 redirect
> > headers check and the body content return.
> >
> > -Scott
> >
> > On Monday, February 28, 2011 5:11:22 PM UTC-6, Dion Hulse (dd32) wrote:
> > >
> > > 2 separate requests will be 2 separate requests.
> > > What's the use-case you're working on here?
> > > Personally, I'd do a normal fetch, followed by a head if it was a
> > > exceeded-redirects error if you want the body, otherwise, the url..
> > > But i cant think of a case where you'd want one or the other..
> > >
> > > On 1 March 2011 04:06, Scott Kingsley Clark <sc... at skcdev.com> wrote:
> > >
> > > > Not sure if anyone knows this, but does the page get loaded twice or 
> is
> > > the
> > > > second time getting loaded from some sort of cache? I'm specifically
> > > > calling
> > > > to the idea of using wp_remote_head on a URL to check for a redirect,
> > and
> > > > then using wp_remote_request on the same URL to get the content / 
> etc.
> > > > _______________________________________________
> > > > wp-hackers mailing list
> > > > wp-h... at lists.automattic.com
> > > > http://lists.automattic.com/mailman/listinfo/wp-hackers
> > > >
> > > >
> > > _______________________________________________
> > > wp-hackers mailing list
> > > wp-h... at lists.automattic.com
> > > http://lists.automattic.com/mailman/listinfo/wp-hackers
> > >
> > >
> >
> > _______________________________________________
> > wp-hackers mailing list
> > wp-ha... at lists.automattic.com
> > http://lists.automattic.com/mailman/listinfo/wp-hackers
> >
> >
> _______________________________________________
> wp-hackers mailing list
> wp-ha... at lists.automattic.com
> http://lists.automattic.com/mailman/listinfo/wp-hackers
>
>