Concertina pagination - getting large scale websites indexed

Posted on November 28, 2008
Filed Under I'm lovin it |

As many of you will know getting large numbers of pages into the index of search engines is a science on its own.


image by hoveringdog

It’s clear that cross linking from other pages of the project helps massively. Not only with indexation but with ranking of the pages as well. SEO black-holes like Wikipedia are the living proof of the effectiveness of - what you can call excessive cross-linking.

The other obvious helper to high volume indexation is back-link power. A nice back-link from the yahoo.com homepage will do wonders to the deep-crawl frequency of your site and the number of pages listed.

Ideally you will get back-links not only to you homepage but on any level in the hierarchy of your site.

Time is an additional factor in this equation. It takes time to get millions of pages into the index of search engines.

Now the above is very well known common knowledge among SEO people and even the below is no ground breaking paradigm shifting revelation - but at the same time its a very effective method that’s easy to implement but still rarely used amongst sites with huge numbers of content pages and multiple levels of categories.

The method I’m talking about is a non constant based pagination. It is often referred to as concertina.

And the method is as simple as effective. When in most cases the pagination will look like this.

And the pages will show 1 to 10 on the first page 11 to 20 on the second and so on and so forth. This scheme will stay the same even if you have tens of thousands of pages.

Due to the fact however that few sites have back-links from the yahoo.com leads to the fact that search engine crawler will get “tired” after a while. This means that the crawler will only get to a certain depth of pages on your site.

To help with this it is much more effective to implement a pagination that looks like this.

or this

For the first couple of pages its identical to the example shown above. But after page ten you will get page 20 then 50 then 100 and so on.

By using this method you make sure that the crawler hits pages further down the hierarchy quicker and this does lead to a significant increase of pages indexed.

Here are a couple of sites that use various types of concertina paging for you to have a look.

  • Challenges and opportunities of a large scale social search product
  • Fixing a Dripping Faucet - Or Why You Should Participate in Yahoo! Answers
  • Webmasterworld - Pubcon 2006 Las Vegas - 1st day panels list
  • Yahoo! Answers - No1 UK Website - hitwise
  • Comments

    2 Responses to “Concertina pagination - getting large scale websites indexed”

    1. stefield on December 3rd, 2008 12:41 pm

      Concertina is like sooo yesteryear dood.

      It’s all about “bell curve” pagination with judicious use of nofollow nowadays!

      The digital point e.g aint ideal. You’d want to plump for “next” rather than “last”.

    2. Frank Fuchs on December 3rd, 2008 5:10 pm

      haha,
      there you have it. Stephen that’s just what Yahoo! lost when you and Dan left us. All the innovative spirits gone and all that’s left is yesterdays news. However still better concertina than crappy out of the box.
      Thanks for pointing this out.

      Say hi to Dan
      cheers

    Leave a Reply




    *
    To prove you're a person (not a spam script), type the security text shown in the picture. Click here to regenerate some new text.
    Click to hear an audio file of the anti-spam word