# Name: Robots.txt # Purpose: Control Webcrawling # # 10JUL06 djd - add folder /info to Disallow list # 03JUL06 djd - copied from old site and added folders /Ads, /App_Data, and Maint. # 27Dec04 - Set to allow crawl, disallow images # Note - this will ALLOW the PDFs in the TS subdir to be crawled. User-agent: * Disallow: /info Disallow: /images Disallow: /Ads Disallow: /App_Data Disallow: /Maint