Please add this sooner than "in the future". Obeying the robots.txt is a _MUST_ for any robots out there. This also from a content owner. Thank you.