Search Engines browser pwd protected sites

[Trusten]

How do you enable a search engine (such as google or something like that) to search pwd protected sites. Is there a code to enter into a page or something?

[Feyd]

You don't...

It defeats the purpose. However, you could do some overly complex things that just would be ridiculously ornate for this task...like detecting the footprint of SE robots and serving them different robots.txt file and removing the password protection so that only robots can index that page. Or backdooring your own password protected area (so why have the password protection to begin with).

If you are talking about useless password protection (like virtually any password protection done on the client side, ie. javascript, hidden URLs, form tricks, etc.), then you might have some more options. However, if you are talking about real password protection, such as htpasswd or login forms and sessions/cookies based on login, its kind of pointless.

Maybe if you explain why it needs to be password protected and why (at the same time) it needs to be searchable via external SEs, we can give you some better architecture philosophy on this...

[Trusten]

why do you keep telling me that my idea and/or need is 'pointless'

i have the pwd protection for stories those who are on my mailinglist can read. and to promote memebership (and also make it easier for those ON the mailinglist to search) I would like to have those stories indexed.

i did however consider taking the pwd off so that the spiders could do their stuff, but no idea when google or any others would spider it.

why have the mailinglist? because some books have adult content and that way, if they sign up to view them, they know what they are viewing.

a simple no will be fine.

thank you for your time.

good day

[Feyd]

You misunderstand...

It is pointless to try and get password protected pages to be indexed...spiders can't index them if they are password protected...see what I mean? In order to get them to be indexed, you have to loosen up (virtually remove entirely) your password protection. It's like a person wanting to break the record for hand walking after they cut off their hands...kinda going the wrong way, sparky!

This is why I asked what specifically you were trying to do since there may be simpler ways you have not considered...

For instance, create a gallery of headlines/abstracts (brief descriptions) outside of your password protected directory than is public to spiders and you'll get some indexing (if you link from this gallery to the articles, spidering won't happen, but that page will show up with search possibilities on articles that haven't been searched), which could all point to your registration/membership area.

[Trusten]

not a bad idea. i don't suppose you know php, huh? well if you do, please do not hesitate to help me out in my php topic. *grin*

anywho, i see what you mean. that way keywords would point to key files.

not a bad idea. i'm still looking for a cheap if not 'free' PDF indexing search engine. so far i use google but you know how that is.

Thanks.

I think i have a good idea of what i'm going to do.

[BrainJar]

In addition to Feyd's suggestion, you could also add a search engine to your site. Most web servers have one built in, like Index Server in Microsoft IIS.

This would allow you to index your site locally and your members could search it once they've logged in.

[joh6nn]

it's always a good idea to add a seach utility to your site ( or, well, i can imagine that there are times when this could be bad, but none occur to me at the moment ). however, that doesn't really server his purpose at the moment. he wants search engines to index his content, so that people will come to his site, but he wants people to have to sign up, to get in.

in which case, as far as i know, Feyd's way is the only one that will work for all search engines.

[Trusten]

hi, i'm a girl.


well some of the books don't require membership, and those ones are indexed. the only thing is that some are in PDF format, so ya know, i'd need a search engine that supports that. google is the only thing that's even slightly adeqet.

luckily.

(

but the idea of getting my own search engine on my site is a great one though!

thanks a lot. if you have any other suggestions on how to make this easier for me, i'd really appreciate it.

[brothercake]

Atomz search can index PDF documents

[knightcon]

spiders can't index them if they are password protected...see what I mean? In order to get them to be indexed, you have to loosen up (virtually remove entirely) your password protection.

For the purposes of any other poor sod wanting to know if anything has changed in recent years it has. Namely with Google but if you use Google Webmaster Tools you can specify your site to be indexed and you can also provide the login details for the site to index password protected pages. It is still not recommended as providing a publicly accessible list of your protected pages is like cutting yourself shaving while swimming in the shark tank with a gallon of chum down your jocks but the service is still available through Google. It supports HTTP authentication, Form based authentication as well as cookie and URL based authentication.