How Search Engines work
All search engines use automated robots commonly known as ‘bots’, ‘crawlers’ or ‘spiders’ to visit web pages and follow links to other web pages. The links can be ’internal’, visiting other pages on the same website, or ‘external’, visiting other websites. Once on a page the ‘bot’ deciphers the code on that page and stores selected pieces of code in a database, this is referred to as ‘indexing’. The search engines use algorithms to calculate page ranking, each search engine has it’s own algorithms and they are constantly updated.
What ‘bots’ don’t like.
‘Bots’ are not human and do not read a web page in the way that a person does. Lots of things that we see on a web page are totally invisible to a ‘bot’, this is important because it is the ‘bot’ that determines your page ranking.
Some items that can cause problems for search engines:
• Euphemisms and uncommon language: Content that is not written in clear language that people use to search can cause a problem. For example a ‘pre-enjoyed or pre-loved vehicle’ rather than a ‘used car’. International differences in spelling can also be difficult foe reach engines, such as , “color” vs. “colour.
• Content that is not text: Search engines are getting better at reading non-HTML text, however content in rich media format is still difficult for search engines to parse. This includes text in Flash files, images, photos, video, audio, and JavaScript.
• Online forms: Search engines find it difficult to complete online forms (such as a login), and thus any content contained behind them may remain hidden.
• Poor link structures: Not only do search engine uses the link structure to visit other pages but the number of links to a page determine that page’s importance. If a website’s link structure isn’t understandable to a search engine, it may not reach all of a website’s content; or, if it is crawled, the content may be deemed unimportant by the engine’s index.
• Unrelated location targeting and mixed messages: If your website appears to be targeting people in Kent but is written in a foreign language then that may cause the search engine problems. If your website appears to be about ‘home decorating’ but your content is about ‘selling cars’.
• Frames and iframes: Frames allow you to turn a Web page into a Web application:
Fixed Content: Different sections can be fixed position in the page.
Independent scrolling: Different sections can scroll without scrolling the entire page.
Mix and match: Content can be loaded from various sources.
Dynamic: Content can be changed without reloading the entire page.
Frames have largely been replaced with iframes and although technically, links in both frames and iframes are crawlable, they both present structural issues for search engines in terms of organization and following and are therefore probably best avoided.
Compromise
Like most things in life, it is all about compromise; obviously you are not going to stop using online forms, pictures, and video on your website just because it is not easy for search engines to recognize them. Your website should be designed for people, but with search engine limitations in mind. If you decide to have a Flash intro page with your keywords imbedded in animation, it may look spectacular but the search engines will almost certainly ignore it. The good news is that the search engines are continually improving they see websites
