Get All URLs on a Page

In this article, I show a class that can be used to find and display all of the urls on a web page. What for you may ask? Well, in my experience as a web developer, I have found a class like this to be very useful. Sometimes, you may want to use this class a a basis for a more complex application that crawls your site checking for bad or broken links. In other cases, you may simply want to check an individual page to make sure your links are formatted correctly, or don’t contain any obsolete pages. You could also easily change this class to look for other items within your page, like specific text or tags. Who knows, this may be the start of a specialized spider that crawls sites on the internet looking for something specific.

I think you get the picture. Of course, to make this class do all those wonderful things, you would have to expand on what I am presenting here. However, I believe this is a good start. The class has one public method - RetrieveUrls. The method calls two private methods. The RetrieveContents method will issue a request to the web page, and retreive the contents. The GetAllUrls method will use a regular expression to find all of the urls on the page. This method writes the matches to the screen, as well as saving them in a log file. Of course, if you prefer, you could modify the method to save the matches somewhere else, like an array or a database table.

Using the code

</p> <p>GetUrls urls = new GetUrls();</p> <p>urls.RetrieveUrls(”http://www.microsoft.com”);</p> <p>

The class is listed below. Have fun!