Working of a Search engine

All search engines work using a 3 phase approach to managing, ranking and returning search results. But a lot of people have no idea what is happening behind that search box when they type in their search queries. So just how do Google, Bing and the rest of them work out what is on the web, what is relevant to your general query and which specific websites should be ranked highly?

There are three functions which need to be done:

Web Crawling

This is the means by which search engines can find out what is published out on the World Wide Web. Essentially, crawling is copying what is on web pages and repeatedly checking the multitude of pages to see if they are changed and make a copy of any changes found.

The programs which have the job of doing this are variously referred to as robots, crawlers, spiders or some variation using ‘web’, e.g. web crawler.

Indexing

Once a spider has crawled a web page, the copy that is made is returned to the search engine and stored in a data center.  Data centers are huge, purpose-built collections of servers which act as a repository of all the copies of web pages being made by the crawlers. Google owns dozens of them dotted around the world, which it guards very closely and which are among the most hi-tech buildings in the world.

The repository of web pages is referred to as the ‘Index’, and it is this data store which is organized and used to provide the search results you see on the search engine. Indexing is the process of organizing the masses of data and pages so they can be searched quickly for relevant results to your search query.

The Algorithm

Finally, we have a huge collection of web page copies which are being constantly updated and organized so we can quickly find what you are looking for. But we need a means by which they can be ranked in order of relevance to your search term – this is where the Algorithm comes into play.

The algorithm is a very complex and lengthy equation which calculates a value for any given site in relation to a search term.  We don’t know what the algorithm actually is, because search engines tend to keep this a closely guarded secret from competitors and from people looking to game the search engine to get to the top spots.  That said, enough about the algorithm has been worked out to let SEOs advise website owners on how to improve their sites and SEO factors to move up in the rankings