generate robots.txt
full reference, searchable
all standard headers explained
parse URL into components