How Web Search Engines Work
| This section does not cite any references or sources. Please help improve this section by adding citations to reliable sources. Unsourced material may be challenged and removed. |
A search engine operates in the following order:
- Web crawling
- Indexing
- Searching
Web search engines work by storing information about many web pages, which they retrieve from the page's HTML. These pages are retrieved by a Web crawler (sometimes also known as a spider) — an automated Web browser which follows every link on the site. The site owner can make exclusions by using robots.txt. The contents of each page are then analyzed to determine how it should be indexed (for example, words can be extracted from the titles, page content, headings, or special fields called meta tags). Data about web pages are stored in an index database for use in later queries. A query from a user can be a single word. The index helps find information relating to the query as quickly as possible. Some search engines, such as Google, store all or part of the source page (referred to as a cache) as well as information about the web pages, whereas others, such as AltaVista, store every word of every page they find. This cached page always holds the actual search text since it is the one that was actually indexed, so it can be very useful when the content of the current page has been updated and the search terms are no longer in it. This problem might be considered a mild form of linkrot, and Google's handling of it increases usability by satisfying user expectations that the search terms will be on the returned webpage. This satisfies the principle of least astonishment, since the user normally expects that the search terms will be on the returned pages. Increased search relevance makes these cached pages very useful, not just because they may contain data that may no longer be available elsewhere.
When a user enters a query into a search engine (typically by using keywords), the engine examines its index and provides a listing of best-matching web pages according to its criteria, usually with a short summary containing the document's title and sometimes parts of the text. The index is built from the information stored with the data and the method by which the information is indexed. From 2007 the Google.com search engine has allowed one to search by date by clicking 'Show search tools' in the leftmost column of the initial search results page, and then selecting the desired date range. Most search engines support the use of the boolean operators AND, OR and NOT to further specify the search query. Boolean operators are for literal searches that allow the user to refine and extend the terms of the search. The engine looks for the words or phrases exactly as entered. Some search engines provide an advanced feature called proximity search, which allows users to define the distance between keywords. There is also concept-based searching where the research involves using statistical analysis on pages containing the words or phrases you search for. As well, natural language queries allow the user to type a question in the same form one would ask it to a human. A site like this would be ask.com.
The usefulness of a search engine depends on the relevance of the result set it gives back. While there may be millions of web pages that include a particular word or phrase, some pages may be more relevant, popular, or authoritative than others. Most search engines employ methods to rank the results to provide the "best" results first. How a search engine decides which pages are the best matches, and what order the results should be shown in, varies widely from one engine to another. The methods also change over time as Internet usage changes and new techniques evolve. There are two main types of search engine that have evolved: one is a system of predefined and hierarchically ordered keywords that humans have programmed extensively. The other is a system that generates an "inverted index" by analyzing texts it locates. This first form relies much more heavily on the computer itself to do the bulk of the work.
Most Web search engines are commercial ventures supported by advertising revenue and thus some of them allow advertisers to have their listings ranked higher in search results for a fee. Search engines that do not accept money for their search results make money by running search related ads alongside the regular search engine results. The search engines make money every time someone clicks on one of these ads.
Read more about this topic: Search Engine
Famous quotes containing the words web, search, engines and/or work:
“The spider spinning his web for the unwary fly. The blood is the life, Mr. Renfield.”
—Garrett Fort (1900–1945)
“You may well ask how I expect to assert my privacy by resorting to the outrageous publicity of being one’s actual self on paper. There’s a possibility of it working if one chooses the terms, to wit: outshouting image-gimmick America through a quietly desperate search for self.”
—Kate Millett (b. 1934)
“America is like one of those old-fashioned six-cylinder truck engines that can be missing two sparkplugs and have a broken flywheel and have a crankshaft that’s 5000 millimeters off fitting properly, and two bad ball-bearings, and still runs. We’re in that kind of situation. We can have substantial parts of the population committing suicide, and still run and look fairly good.”
—Thomas McGuane (b. 1939)
“... anybody is as their land and air is. Anybody is as the sky is low or high, the air heavy or clear and anybody is as there is wind or no wind there. It is that which makes them and the arts they make and the work they do and the way they eat and the way they drink and the way they learn and everything.”
—Gertrude Stein (1874–1946)