A couple of months ago, Kieskeurig, perhaps The Netherlands' best known consumer (price comparison) site, outsourced part of its' Domino infrastructure maintenance to e-office bv.

For you that are not familiar with kieskeurig.nl: this sitename has a very familiar ring to many Dutch internet users. In The Netherlands, "I checked it on kieskeurig" sounds as familiar as "I googled it": everyone will understand what you mean.

We interviewed Henri Hoetink, founder, owner and CEO of Kieskeurig, to talk about the site www.kieskeurig.nl and the company. And of course we discussed the technical aspects involved in running such a high profile site: kieskeurig.nl, with it's 125.000 unique daily visitors, is exploring the edges of what the Domino server can do. How many hits can a Domino server actually have?


Henri, could you tell us something about your company, Kieskeurig?

"We founded Kieskeurig in 1999, when I was still an e-office employee. Our goal was, and is: providing the consumer with reliable, independent product information and prices. We have grown to become the biggest, best known and most complete consumer information site in The Netherlands."

How many unique visitors come to kieskeurig.nl, on an average day?

"Usually we get more than 125.000 visitors, and together they generate a little under a million pageviews. Most visitors hit the site around lunchtime, and in the evening hours between 8 and 11 PM. As for data traffic, our site generates an average of 3,5 GB per hour."

What can you tell us about the content of kieskeurig.nl?

"The site contains some 370.000 product reviews, 60.000 product specifications, and more than 1 million shop prices. Keep in mind that we're collecting lots of shop prices in real time, for example those of book and cd stores. Those prices are not included in that 1 million figure."

Kieskeurig.nl is running on Lotus Notes/Domino. Why?

"We started out on Domino because I did a lot of Domino work when I was employed at e-office. Domino is a great platform for a document based approach. The integrated Notes backend and Domino http frontend together create an environment where you can integrate processes, and work on a website together, with your whole organization."

You just named some impressive user visit and pageview statistics. What kind of Domino environment does one need to serve so many users?

"Kieskeurig.nl consists of 11 web servers. Seven of them run Lotus Domino; four of them run Microsoft IIS, mostly for serving pictures. And finally, there's one logging and analysis server. All these servers run on Windows Server, and combined they use 21 CPU's, most of them Intel Xeon processors."

Is the data on your site static, or dynamic? Are there a lot of changes to the data?

"Absolutely. Daily, we spider over 250 web stores for their prices. All changes in products and prices at these stores are collected, and applied to the data on our site. As you can imagine, that's a lot of daily changes. On top of that, every day a lot of data is added manually, like product specifications and consumer product reviews."

Uptime is an important factor for a website. How is kieskeurig.nl doing in this regard?

"Usually, we get between somewhere between 99.8 and 100% uptime. For individual servers this is lower, of course, but by utilizing load balancing on the network routers, we've minimized the impact on the overall uptime of the site".

Running a site like kieskeurig.nl is a demanding task for a Domino server, and for your programming too. What can you tell us about that?

"Programming for a site like kieskeurig.nl is absolutely different from programming a normal Domino based website. For instance, a year ago we programmed a nice new feature into the site. After testing the agent code thoroughly, we brought in online. After fifteen seconds, the first webserver crashed! Again fifteen seconds later, the second one went down. And so on, until the site became unavailable. It turned out that the high number of concurrent users made the new agent so memory hungry, that it crashed the server within 15 seconds."

"We see something similar when code needs too much time to execute. If that happens. the number of waiting users will rise sharply, and user after user will get an "agent did not complete within time limit" message, or an HTTP error 500. In cases like this, it's extremely important to identify bottlenecks in the system, and look for the optimal balance. This goes for hardware as well as software and code."

"For configurable items, we mainly use profile documents. Other than that, a lot of things are hardcoded. This has some advantages, despite the fact that I was taught otherwise at e-office. Hardcoded stuff is often faster, and the risks are smaller. Profile documents can be deleted accidentally, replication and safe conflicts can occur, or the view that contains the profile documents can become unusable.
Another thing: DbLookups and @DbColumns are extremely rare on pages that can be requested by visitors."

"Inventive? Yes, I think we are. We are on the edges of what can be done with Domino. Being inventive is necessary for us."

The Good, the bad and the Ugly. First, what's good about Domino?

"It's possible to create nice applications, fast. And, almost everything is possible, with the Lotus programming languages LotusScript and the @Formula language. If that doesn't do it, you can always use Java."

What about the Bad, and the Ugly?

"Domino could use a boost in performance. Overall, it performs well, but there's certain moments when it doesn't. Our biggest problem is that Domino servertasks and web visitors can get in the way of one another.
A certain database can be replicated, and have it's view indexes and Full Text index updated at the same moment. If that happens, any action in the database will take five times longer. And, that database is probably used by dozens of web visitors, firing off queries and full text searches at that same moment. When that happens, it seems that Domino doesn't know what to handle first, and everything stalls. A server may perform badly because of this, for several minutes, or even up to an hour. I would very much like to have a notes.ini setting to avoid this kind of behaviour."

"Adding more servers to a cluster does not help much in this regard, as all changes are replicated to all servers almost instantly. This means that the replication and indexing happens on all those cluster servers simultaneously."

"But once the replication and indexing is done, the site performs fast as usual."

In Notes/Domino 7, you can use a relational DB2 database as an optional backend, although this is not yet fully supported by IBM. Are you investigating this new backend? Will you use it?

"We are investigating it, but right now the performance of DB2 as a backend to Domino is not satisfactory. But we are keeping a close eye on it."

How is your relationship with IBM? Are you in contact with them?

"Barely. We have an open support call, by the way, but that's about it."

Your servers are all running on the Windows server platform. Are you satisfied with that?

"Yes and no. Windows server is a stable operating system. But there's a limit on Windows server. Applications can use a maximum of 2 GB RAM, and could definitely become a problem for us. We're not looking for a replacement operating system right now, but that doesn't mean we never will."

*a Dutch version of this interview appeared on http://www.domino-weblog.nl earlier this week.
**the contents of this interview are copyrighted by e-office bv and kieskeurig.nl.

Blocked Response!07/06/2009 07:57:45 AM

This response from IP Address 124.125.175.126 was blocked by the owner of this blog.

---