At TOKY, we’ve built a couple sites for clients who wanted a site search. Rather than re-invent the wheel we’ve tried using Yahoo’s search (this was pre BOSS) as well as Google’s Custom Search Engine (CSE). We have had problems with both search applications. In particular, the results would be different from page to page. For instance, when searching an item it may tell me that there are 300 total results. But then three pages into those results it would tell me that there were really only 37 total search results.

This would make it quite difficult to include total search results for a search on the page. It also prevented us from providing pagination as a method of browsing the results. Instead we can only provide Previous Page / Next Page style navigation of the results.

I contacted Yahoo, and they explained that they were working on it. The issue was mainly because the search would hit a different server with each search call.

I tried contacting Google, but never had much luck getting an answer.

However, after testing CSE a bit more, I noticed that this issue was consistently happening on certain searches and at certain pages within the result set. Therefore, the issue couldn’t have been the same as it was with Yahoo’s search app.

I started playing with every parameter that can be customized when calling CSE. I found one that seems to fix the problem: filter. Here is what the API says about filter:

“Optional. The filter parameter activates or deactivates the automatic filtering of Google search results. See the Automatic Filtering section of this document for more information about Google’s search results filters.

The default value for the filter parameter is 1, which indicates that the feature is enabled.

Note: By default, Google applies filtering to all search results to improve the quality of those results.”

I turned this filter off and search results were consistent from page to page. Google may recommend that the filter is enabled, however, in the majority of my tests the first couple pages of results were the same with it on or off. If the filter was turned on, I would still encounter the issue where 3-5 pages into the results the total number of search results would drop dramatically.

If the main difference is that I get extra pages of search results, I’ll take it. I’d much prefer that over the chance of changing the total search results from page to page or even worse, using pagination and allowing the user to go a page of search results that doesn’t exist.