e-axe`s small home :: blog
-------------------------------------------------------------------------
menu :: blog | projects | books | wishlists | about :: prev | next
-------------------------------------------------------------------------
the "404 file not found" problem... [Fri Jan 8 22:36:05 CET 2010]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
some of you might be already stumbled over the so called
"404 file not found" problem - which means, the webserver (oh, in some
cases it is the application itself by modifying the http header) sends
back a "200 OK" with the 404 error inside the content. or, which i have
seen even more often, you get an "302 Moved" back which redirects
to a custom 404 page.
well, the problem with this is, that a lot of crawling and security
tools are relying on the http status codes! which in case means, that
those tools think, that the requested resource is existent even if its
not! this brings in a lot of false positives.
i thought about on how to solve this "problem" and came to some
solutions. today, i will share one of them with you:
http://mytty.org/nopaste/?pid=427
execute it like this: ruby SCRIPT www.THE-DOMAIN.org "/resource1.html" "/resource/2.html" "/foobar/"
the script will perform a check if those pages of the given domain
are existent or not by fetching 10 "not existing pages" and
calculating the average size of the response bodys. then it fetches
the given resources and checks if the body sizes are equal
(+- a few bytes; this maybe need to be adjusted (can partly be done
automatically!)). this method could be used if you stumble over a
webserver/application answering with "200 OK 404 not found" pages ;)
your feedback is very welcome!
so long,
e-axe
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
there is always something stupid to do...