What's With Those Search Engines?

If you're looking for something on the Internet and don't know a specific web site address, what do you do? You type in a search term at a Search Engine such as Google, MSN, Yahoo, etc., right?

How do all those web links end up in the search engines?

Search engines are searchable databases of web sites compiled by software called search "spiders" or "crawlers." These spiders index information they find in a web site, some of them also crawling through links found within a web site like a big chain. Search spiders do not crawl the web in real time; they simply gather the information and then index it in a database for a search engine. Only after a web page is indexed will that page be searchable in the search engine's database.

How do the search spiders find the web sites?

If a site or page is not linked to from a crawled web site, or is not directly submitted by someone (webmaster, page author, etc.), it will not be accessible from a search engine. Engines primarily use these two methods of finding out about new sites and pages.

In most cases, submission to search engines is free. However because of the enormous quantity of information out there on the Web, it can sometimes take weeks or months for the spiders to get around to crawling a web site. While you can often pay to speed up the submission process, for example to get your site looked at by Yahoo or Looksmart, this does not guarantee your site will be listed.

Different search engines have different criteria for inclusion in their databases

Some will only crawl the first 50 to 110KB of a page, so if you have excessively long web pages, there may be a lot of information that never makes it to the search engines. Still, 50 to 110KB leaves room for a lot of textual content. And CONTENT IS KING. Without plenty of good content on your site, there won't be much for the search engines to "pick up" and compare with what searches are entered.

Some search engines, such as Google, get much or most of their information from other databases. Google uses the Open Directory Project as a primary Web directory.

Some search engines (such as Google again) offer cached versions of web pages to speed up searches. This means it can take weeks or longer sometimes before Google finally includes an updated version of a page in its database.

How do search engines determine how high to rank a page?

Search engines "look" at several factors to determine relevancy in searches, which in turn determines in part how high a site will be ranked for a given search:

• Text on a page - you should have at LEAST 300 content-rich words on every single page in your site.
• Page title
• Page META description
• Page META keywords
• Link popularity (how many other relevant web sites have linked to the site containing this page)
• "Alt" text for images
• The more good content-rich pages you have, the higher your search engine rankings will be. All things considered, it's a lot easier for a 50 page web site with good relevant content to have great rankings than it is for a 2-page site.

Note that not all search engines look at all factors. Some search engines ignore alt text completely. Some search engines do not look at META tags; some search engines look only at the META description or META keywords. All search engines assign very high priority to the text on a web page. This means that if your web page is comprised mainly of images and very little text, there won't be much for the search engines to index -- which also means any web page out there with a higher density of relevant text will be ranked higher.

Does that mean you should create a page with nothing but keywords repeated over and over again, and submit it to 50,000 Free For All link sites? After all, if it's great to have some keywords and have your link on other web sites, then it must be even better to have thousands of keywords and thousands of other sites linking to you, right?

Nope. Absolutely NOT.

Create a web page with no real content except for a whole bunch of keywords, and your site will be ranked lower or even banned by the search engines for spamming. The same goes for web sites with a lot of linking from Free For All link sites (also known as FFA's). Web sites which are submitted to FFA's are increasingly being banned from search engines because the search engines rank relevancy - not just useless links with no real purpose except linking. If a search engine detects that your site is linked from FFA's, watch out. A single link from a web site that ranks high in Yahoo is worth more than 5000 FFA links. In fact if you have 5000 FFA links, don't plan to see your web site in the search engines very much longer!

OK, so HOW do you make a web site get a higher ranking in the search engines? It sounds pretty tricky.

Actually, it is. Good ranking in search engines is a combination of many factors. To complicate matters, search engines are continually updating their methods and algorithms so that what's effective today may not be as effective next week, and new factors may take priority. Additionally, many search engines are starting to give much more emphasis and higher rankings to paid submissions, purchased keywords and keyword phrases, and even promoting price bidding for keywords and ranking. Other search engines combine paid keyword positioning with pay-per-clicks, in other words web site owners pay the search engine a set amount for every click-through from the search engine to the listed web site. This can get fairly expensive pretty fast.

If you can't afford to pay hundreds or thousands of dollars every month for search engine positioning, it's still not hopeless, but it takes much more effort and time. At a minimum, this is what is necessary:

• Keyword phrases chosen carefully, based upon
  • Your target market -- who is the customer you are trying to sell to?
  • The popularity of the key word phrase -- how many people are searching for your key term?
  • Market analysis -- how many other sites are optimized for the same key word phrase?

One of our favorite tools for determining most-frequently searched keyword phrases for a particular product or service is the Overture Keyword Suggestion Tool. Try different variations of similar key words or keyword phrases. The results may surprise you.

• The next step is to incorporate the keywords and keyword phrases into your site. Remember: Content is King! If you don't have enough textual content on your page which provides relevant keyword-rich web content for the search engines to spider, it will be extremely difficult to get any visible placement or ranking in the search engines at all. Every single web page in your web site should have at minimum 300 relevant words visible as regular text (this does NOT include words on images such as buttons, logos, etc.).
  
  Writing for the search engines is a bit different than writing for a specific target customer or marketing campaign. Along with providing relevant high quality content to your site visitors and compelling them to purchase your product, the web developer must also be conscious of how the search engines will evaluate your site. The position of the key words on a web page and how the key words are used within the web page are very important for obtaining the best listings and positioning on the search engines.
  
  
• Page META tags must be carefully composed for each individual page to make optimum use of keywords and keyword phrases in conjunction with the page's title and specific page content. However, search engines are increasingly placing less emphasis on META tags (except the TITLE tag), and much more emphasis on page body text content and link popularity (see next bullet).
  
  
• YOU, the web site owner, should spend serious time surfing and finding non-compteting, high-quality, high-ranking web sites that are relevant to your own web site's content. Send an email to the webmaster of the site, requesting a reciprocal link. Be willing to have their link placed on your web site as well. Try to get at MINIMUM two or three good link-backs every week. This is actually harder than it sounds: If you get a 1% or 2% response rate to all the requests you send out, you'll be doing well. 
  
  Google and other important search engines will rank your site based on link popularity or page rank, as well as content on your pages. Link popularity is increased by the number and quality of links pointing to a specific page. Having links pointing to your web site from any other relevant, high-ranking, popular pages (not your own web site) can greatly increase your site's link popularity rating. Additionally, having a link pointing to your site from the popular search Directories such as Yahoo, LookSmart and the Open Directory Project (DMOZ) can greatly increase your site's link popularity and thus help to improve its chance for obtaining a higher listing in Google and other important search engines.
  
  How many sites should you try to get to carry your link on them? The importance of link popularity to your ranking in search engines cannot be overstated. It is HUGELY important. For best popularity ranking, that number is going to vary depending upon your competition. If the top-ranking site has 250,000 other related/relevant sites linked back to it, you'd better plan on getting at least a few thousand links (preferably several times that) if you want to be able to compete at all. If your competition is that tough, your better bet would be to NOT compete with the most-searched/most poupluar key phrases or keywords. Instead find and target "niche" keyphrases that are less competetive but will still bring people to your site. Keep in mind, even if you are using niche keyphrases, you WILL still need good link popularity, but may not need quite as many links.
  
  Do be careful though: You don't want to be linked from any sites, or include links on your own web site, that will negatively influence your own web site's credibility. Also if you sell a product, obviously you don't want to send people to a competitor's web site to buy it.