Web crawlers. Googlebot.

A Web Crawler is a computer program that automatically browses the World Wide Web in a methodical way. Web Crawlers is also called ant, bot, worm or Web spider. The process of scanning the WWW is called Web crawling or spidering.

What Web Crawlers do?

Web Crawling is used by Search engines to provide up-to-date data to the users. What Web Crawlers essentially do is to create a copy of all the visited pages for later processing by a Search Engine. The search engine will then index the downloaded pages in order to provide fast searches.

Web Crawlers are also used for automating tasks on websites such as checking links or validating HTML code.

A Web crawler usually starts with a list of URLs to visit (called the seeds). As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit (crawl frontier). URLs from the frontier are then recursively visited according to a set of policies.

Here is a picture that I did to show you the architecture of a Web Crawler:

Web Crawler architecture

Web Crawler architecture


Tagged with: browsersoftwaretutorialWeb

Ninite Multi-App Installer

Ninite is a great solution for Windows that provides you a very easy way to get multiple applications installed on a computer with a single click! All you have to do is to select the apps you want installed from the list available at the Ninite’s homepage, start the customized installer that you get from the website and you’re done!

Ninite Multi-App Installer homepage

Ninite Multi-App Installer homepage

Ninite runs on Windows XP/Vista/7, it allows to install up to 59 different programs (all installed with default values). Therefore especially if you have to set up more than a single computer with a set of basic programs, Ninite will make life easier .

Tagged with: browsersoftwaretutorialWebWindows

FileZilla FTP client

FileZilla Client is a free and open source FTP client. It supports FTP, SFTP, and FTPS (FTP over SSL/TLS). The download of the FileZilla client is available for many platforms, Windows, Linux and Mac OS X.

FileZilla FTP Client

FileZilla FTP Client

FileZilla has a straightforward interface available with translations in most of the world’s major languages. Browsing FTP sites is fast. It’s possible to bookmark sites using the Site Manager for all the major FTP locations and QuickConnect for fast connections on the fly.

FileZilla FTP Client Site Manager

FileZilla FTP Client Site Manager

It’s possible to download or upload multiple files at once from and to several servers. There is even a cool feature that lets you automatically adjust transfer speeds based on the time of day or date (if your bandwidth is somewhat limited) and a¬†filter to only see the files you need.

FileZilla is clearly one of the best FTP tools around and it’s totally free!

If you are interested in an FTP client a bit more limited in terms of functionalities but more easy to use, please visit Cyberduck homepage!

Tagged with: AppledownloadLeopardMACsoftwareWindows

Evernote – Remember everything!

You can use Evernote to save your ideas, things you see, and things you like. Then you can find them all on any computer or device you use. For free!

Either type a text note, clip a web page, snap a photo or grab a screenshot and save it forever in Evernote:

Evernote - Remember everything!

Evernote - MAC OS X Simple and easy user interface.


Tagged with: AppleBlackberryiPhoneiPodLeopardMACnotesPalmsoftwareWindows

Need an easy way to keep track of your bills? Chronicle may be for you!

Chronicle is a good solution if you don’t wanna miss a payment and if have ever found sometimes yourself in a panic trying to remember if you paid or not your rent or mortgage this month.. or if just want to get a better idea of how much you are spending on your bills! Designed to be incredibly easy to use, Chronicle will remind you to pay all your recurring bills on time, and keep track of what and how much you have paid:

Chronicle - Keep track of your bills.


Tagged with: ApplebusinessLeopardMACsoftware

How Windows Live Messenger works.

How does Windows Live Messenger works? there are millions of users typing messages everyday but maybe few of them ever asked themselves how the messenger really works! so here I wrote a little post about it.

Windows Live Messenger it’s an hybrid Client-Server / Peer-to-Peer application.

It basically started as a client-server application.

Let’s say that the client A wants to contact the client B. The client A logs in a CS (Connection Server) through a persistent TCP connection (eventually using proxy,gateway..). Behind the CS there are the PS (Presence Server).

Each person get always the same particular PS, which is where your personal status message, description of your user photo and similar things are stored.

Windows Live Messenger architecture

Another element of the architecture is the Address Book. The client A gets directly from the Address Book his list of contacts.

Then the client A tells to his CS who his friends are, the CS subscribes to his friend’s PS to get the presence information that are sent up through the client server connection.

If the client A change his status to OffLine for example, the change goes up to the CS of A, then to the PS of A, then down to the CS of B through the subscription and then down to the client B.


If the client A wants to chat, tells to his CS that wants to contact somebody, and the CS tells A to contact a Mixer, which sends IM traffic to a destination, for example to B (passing through the CS of B). Then A and B can communicate through the Mixer. (more…)

Tagged with: chatsoftwaretutorialWebWindows

Php-stats installation tutorial.

Php-stats is an excellent site monitoring script and it’s free to download and definitely easy to install. It’s released under the GPL/GNU license. The requirements for the software are: PHP 4.0.6 or higher, MySQL 3.23 (*) or higher.

Let’s describe now the few steps for the installation of Php-stats:

  1. Modify the config.php with the values provided by Aruba when you activated the MySql service:
    option['host']=”xxx”; xxx is the IP address of the Aruba MySQL server. An example value is
    $option['database']=”xxx”; xxx is the MySQL database name.For example something like Sql123456_1
    $option['user_db']=”xxx”; xxx is the username of the database. For example Sql123456
    $option['pass_db']=”xxx”; xxx is the password of the database. Example: k1526g0f
    $option['script_url']=”xxx”; xxx is the URL of the installation path of Php-Stats. For example: www.domainname.com/stats
    $option['exc_pass']=”xxx”; xxx is the password used to enable the dynamic exclusion of the IP addresses. Es. dynamic.
  2. Now let’s upload via FTP the Php-stats files to the www.domainname.com/stats URL, eventually creating the folder stats if it doesn’t exist.
  3. Make sure the folder option has read,write,execute permissions for everyone (chmod 777). Everytime we modify something from the control panel Php-stats will write in that folder.
  4. Make sure the file config.php has read and write permissions for everyone (chmod 666) to be able to use the Config-Editor di PHP-Stats.
  5. Run the setup.php using the URL www.domainname.com/stats/setup.php and follow the instruction displayed.
  6. At the end of the installation set the options parameters displayed, as for example for example the URL of the website, your email for password recovering and reports and other settings for the reports.

That’s it! Now just wait for people to visit your website and check the statistics! ;)

Tagged with: downloadsoftwareWeb