Most indexers obey the robots.txt file, but they're generally harmless even if they don't.