How Search Engines Work

Every day we talk about search engines and searches but most of us may not know exactly how search engines work, so today I have decided to shade some light on how search engines work because a good understanding of how search engines work will help us build better search engine optimized blogs.

What are Search Engines?

Search engines are programs that search the internet or documents for specified keywords and return a list of the web pages where the keywords were found in an order of how the search engine deems them important and relevant to the searched query.

But often times the term search engines are used to describe systems like Google, Yahoo, Bing and Yandex etc. These systems are used to search for information, documents, files or anything at all on the internet. Search engines have become a very important part of us as the internet has become as well. But note that you cannot talk about the internet without the search engines and that is exactly why a good understanding of how search engines works can help you make use of the internet even better.

How Search Engines Work:

While I will not be diving into the complex mathematical algorithms of search engines in this article I will keep this concept pretty simply just to give you a clear understanding of how search engines work. There are basically three major components that powers the search engines and they are as follows;

1. Crawling
2. Indexing
3. Ranking

I will try and explain every component so you can grab a high-level understanding of the basics.

Crawling:

This is the first step to how search engines work, search engines simply crawl around the internet looking for new contents like your newly published blog posts or that new PDF document you just uploaded to Scribd with bits of computer code that find information on a web page called ‘Spiders’.

The Spiders crawl from one link to another looking what newly added contents, when they find something new they ‘Read’ it. Periodically these spiders return to look for changes in already crawled sites.

You can also allow or refuse the search engine spiders from crawling some sections of your website especially those areas that contain information you do not want to share in public like your login details etc. This is made possible with the robots.txt file present on your server.

Example of Robots.txt format includes;

Allow indexing of everything

User-agent: *

Disallow:

Disallow indexing of everything

User-agent: *

Disallow: /

Disallow indexing of a specific folder

User-agent: *

Disallow: /folder/

Disallow Googlebot from indexing of a folder, except for allowing the indexing of one file in that folder

User-agent: Googlebot

Disallow: /folder1/

Allow: /folder1/myfile.html

If the search engine spider can’t find your content or doesn’t understand what it is all about, it won’t proceed to the next two steps as discussed below.

Indexing:

Indexing is a concept that explains what the search engine spiders do when they find a new content on the internet. Once a new content has been found the spiders store this new information in it giant database.

So simply put indexing is the storing of newly found information in a database by the search engine spiders for the future benefit of searchers. It also gauges the relevance of that content to the keyword that searchers use to find it.

Ranking:

This is the final and most important aspect of how search engines work. This step involves the way search engines determines the relevance of results and how it is delivered to the searcher. Except you work in a place like Google or Bing no one knows exactly how search engines determine which result should come first or not when a particular keyword is searched.

This is where you apply all you have learnt in SEO to get your content ranked; I have covered a lot of SEO topic on this blog.

Follow the SEO advises I have been giving on this blog and you won’t have to worry about how to get your contents crawled, indexed and ranked. Consider the search engine as a toddler that has to be spoon-fed; you have to apply all the SEO principles that will invite the search engine spider to come and crawl your content, index and rank it.

I hope you find this article ‘How search engines work’ interesting, if you have any questions or contributions please use the comment section below. Remember to subscribe to my RSS feed.