|
|
This is only a preview of the paper
Click here to register and get the full text.
Existing members click here to login
|
|
|
Robots Exclusion Sometimes people find they have been indexed by an indexing robot, or that a resource discovery robot has visited part of a site that for some reason shouldn't be visited by robots. In recognition of this problem, many Web Robots offer facilities for Web site administrators and content providers to limit what the robot does. This is achieved through two mechanisms: The Robots Exclusion Protocol A Web site administrator can indicate which parts of the site should not be vistsed by a robot, by providing a specially formatted file on their site, in http://.../robots.txt. The Robots META tag A Web author can indicate if a page may or may not be indexed, or analysed for links, through the use of a special HTML META tag.
Approximate Word count = 414
Approximate Pages = 1.7
(250 words per page double spaced)
|
|
|
|
|
|