Great work Lee, I was considering something similar which never really got started. I was looking at making use of wwwRobotRules on CodePlex http://robotrules.codeplex.com/ - may be of interest or use to you if you decided to do anything further to this package.
(I'd completely forgotten that I'd set-up this forum **yikes**)
Hi Simon,
I looked at WWWRobotRules to do the validation... but it was slightly overkill for our needs. That library is more targetted towards crawler apps, downloading a remote robots.txt, parsing it and checking/verifying links/URLs against the allow/disallow rules.
If you take a look at the source-code for our Robots.txt Editor - I do a simple validation, looping through each line checking for valid keywords (i.e. "User-Agent", "Disallow", etc).
Taking it forward
Great work Lee, I was considering something similar which never really got started. I was looking at making use of wwwRobotRules on CodePlex http://robotrules.codeplex.com/ - may be of interest or use to you if you decided to do anything further to this package.
(I'd completely forgotten that I'd set-up this forum **yikes**)
Hi Simon,
I looked at WWWRobotRules to do the validation... but it was slightly overkill for our needs. That library is more targetted towards crawler apps, downloading a remote robots.txt, parsing it and checking/verifying links/URLs against the allow/disallow rules.
If you take a look at the source-code for our Robots.txt Editor - I do a simple validation, looping through each line checking for valid keywords (i.e. "User-Agent", "Disallow", etc).
- Lee
is working on a reply...