title: Choosing an Operating System and Hardware

=head1 Is it important?

You can invest a lot of time and money into server tuning and code
rewriting according the guidelines you have just learned, but your code's
performance will be really bad if you do not take into account the
hardware demands, and do not wisely choose the operating system suited
to your needs. While the tips below apply to any webserver, they are
aimed at administrators of mod_perl-enabled webservers.

=head1 Choosing an Operating System

First let's talk about Operating Systems (OS). While I am personally a
Linux devotee, I do not want to start yet another OS war. Assuming
this, I will try to define what you should be looking for, then when
you know what you want from your OS, go find it. Visit the Web
sites of operating systems you are interested in.  You can gauge
user's opinions by searching relevant discussions in newsgroup and
mailing list archives such as Deja - http://deja.com and eGroups -
http://egroups.com . I will leave this fan research to the reader. But I
would use Linux or something from the *BSD family.

=head2 Stability and Robustness

Probably the most desired features in an OS are stability and
robustness. You are in an Internet business, which does not have
normal working hours, like many conventional businesses you know about
(9am to 5pm). You are open 24 hours a day. You cannot afford to be
off-line, for your customers will go shop at another service like
yours, unless you have a monopoly :) . If the OS of your choice
crashes every day or so, I would throw it away, after doing a little
investigation, for there might be a reason for a system crash. Like a
runaway server that eats up all the memory and disk space, so you
cannot blame the OS for that. Generally, people who use the OS for
some time can tell you a lot about its stability.

=head2 Memory Management

You want an OS with a good memory management, some OSes are well known
as memory hogs. The same code can use twice as much memory on one OS
compared to another. If the size of the mod_perl process is 10Mb and
you have tens of these running, it definitely adds up!

=head2 Memory Leakages

Some OSes and/or their libraries (like C runtime libraries) suffer from
memory leaks. We cannot afford such systems, for we already
know that a single mod_perl process sometimes serves thousands of
requests before it terminates. So if a leak occurs on every request,
our memory demands will be huge. Of course our code can be the cause
of the memory leaks as well (check out the C<Apache::Leak>
module). Certainly, we can lower the number of requests to be served
over the process' life, but that can degrade performance.

=head2 Sharing Memory

We want an OS with good memory sharing capabilities. As we have
learnt, if we preload the modules and scripts at server startup, they
are shared between the spawned children, at least for a part of a
process' life span, since memory pages can become "dirty" and cease to
be shared. This feature can save us a lot of memory!

=head2 Cost and Support

If we are in a big business we probably do not mind paying another
$1000 for some fancy OS with bundled support.  But if our resources
are low, we will look for cheaper and free OSs. Free does not mean
bad, it can be quite the opposite as we all either know from our own
experience or have read about in the news. Free OSs could have and do
have the best support we can find. It is very easy to understand -
most of the people are not rich and will try to use a cheaper or free
OS first if it does the work for them. Since it really fits their
needs, many people keep using it and eventually know it well enough to
be able to provide support for others in trouble.  Why would they do
this for free? For the spirit of the first days of the Internet, when
there was no commercial Internet and people helped each other, because
someone helped them in first place. I was there, I was touched by that
spirit and I will do anything to keep that spirit alive.

But, let's get back to our world. We are living in material world, and
our bosses pay us to keep the systems running. So if we feel that we
cannot provide the support ourselves and we do not trust the available
free resources, we must pay for an OS backed by a company, and blame
them for any problem. Our boss wants to be able to sue someone if the
project has a problem caused by the external product that is being
used in the project.  If we buy a product and the company selling it,
claims support, we have someone to sue. We do not have someone to
sue other than getting ourself fired if we go with Open Source and
it fails.

Also remember that if you spend less or zero money on OS and Software,
you will be able to buy better and stronger hardware.

=head2 Discontinued products

You have invested a lot of time and money into developing some
proprietary software that is bundled with the OS you were developing
under. Like writing a mod_perl handler that takes advantage of some
proprietary features of the OS and will not run on any other
OS. Things are under control, the performance is great and you sing
with happiness. But... one day the company who wrote our beloved OS
goes bankrupt, which is not unlikely to happen nowadays, or they
produce a newer incompatible version and they will not support the old
version our code runs under (and this happens regularly). You are
stuck with their last masterpiece and no support! What you are going
to do then? Invest more into porting the software to another OS...

Everyone can be hit by this mini-disaster, so it is better to check
the background of the company when making your choice, but still you
never know what will happen tomorrow. The OSs in this hazard group
are completely developed by single companies. Free OSs are probably
less susceptible to this, for development is distributed between many
companies and developers, so if a person who developed a really
important part of the kernel lost interest in continuing, someone else
will pick the falling flag and carry on.  Of course if tomorrow some
better project showed up, developers might migrate there and finally
drop the development: but in practice people are given support on older
versions and helped to migrate to current versions.

=head2 OS Releases

Actively developed OSs generally try to keep pace with the latest
technology developments, and continually optimize the kernel and other
parts of the OS to become better and faster. Nowadays, Internet and
networking in general are the hottest targets for system
developers. Sometimes a simple OS upgrade to the latest stable
version, can save you an expensive hardware upgrade. Also, remember
that when you buy new hardware, chances are that the latest software
will make the most of it. Since the existing software (drivers) might
support the brand new product because of its backwards compatibility
with previous products of the same family, it might not reap all the
benefits of the new features. It means that you could spend much less
money for almost the same functionality if you were to buy a previous
model of the same product.

=head1 Choosing Hardware

Since I am not fond of the idea of updating this section every time a
new processor or memory type comes out, I will only hint about what
you should look for and suggest that sometimes the most expensive
machine is not the one which provides the best performance.

Your demands are based on many aspects and components. Let's discuss
some of them.

In discussion course we use terms that may be unfamiliar to some readers: 

=over 4

=item *

Clustering - a group of machines connected together to perform one big
or many small computational tasks in a reasonable time. Clustering is also
used to provide 'fail-over' where if one machine fails its work is
transparently transferred to another without any interruption of service.

=item *

Load balancing - users can remember only a name of one of your
machines - namely of your server, but it cannot stand the heavy load,
so you use a clustering approach, distributing the load over a number
of machines. The central server, the one users access when they type
the name of the service, works as a dispatcher, by redirecting
requests to the rest of the machines, sometimes it also collects the
results and returns them to the users. One of the advantages is that
you can take one of the machines down for a repair or upgrade, and
your service will still work - the main server will not dispatch the
requests to the machine that was taken down. I will just say that
there are many load balancing techniques. (See L<High-Availability
Linux Project|download/High_Availability_Linux_Project> for more info.)

=item *

NIC - Network Interface Card.

=item *

RAM - Random Access Memory

=item *

RAID - META

=back

=head2 Expected site traffic

If you are building a fan site, but want to amaze your friends with a
mod_perl guest book, an old 486 machine could do it. If you are into a
serious business, it is very important to build a scalable server, so
if your service is successful and becomes popular, your server's
traffic doubles every few days, you should be ready to add more
resources dynamically. While we can define the webserver scalability
more precisely, the important thing is to make sure that you can add
more power to your webserver(s) without investing additional money
into a software development (almost, you will need a software to
connect your servers if you add more of them). It means that you
should choose a hardware/OS that can talk to other machines and become
a part of the cluster.

From the other hand if you prepare for a lot of traffic and buy a
monster to do the work for you, what happens if your service does not
prove to be as successful as you thought it would be. Then you've
spent too much money and meanwhile there were a new faster processors
and other hardware components have been released, so you lose again.

Wisdom and prophecy , that's all it takes :)

=head2 Cash

Everybody knows that Internet is a cash hole, what you throw in,
hardly comes back. This is not always true, but there is a lot of
wisdom in these words. While you have to invest money to build a
decent service, it can be cheaper! You can spend as much as 10 times
more money on a strong new machine, but get only a 10% improvement in
performance. Remember that a four year old processor is still very
powerful.

If you really need a lot of power do not think about a single strong
machine (unless you have money to throw away), think about clustering
and load balancing. You can probably buy 10 times more older but very
cheap machines and have a 8 times more power, than purchasing only one
single new machine. Why is that? Because as I mentioned before
generally the performance improvement is marginal while the price is
much higher. Because 10 machines will do faster disk I/O, than one
single machine, even if the disk is much faster. Yes, you have more
administration overhead, but there is a chance you will have it
anyway, for in a short time the machine you have just invested in will
not stand the load anyway and you will have to purchase more and think
about how to implement load balancing and file system distribution.

Why I am so convinced? Facts! Look at the most used services on the
Internet: search engines, email servers and the like -- most of them
are using a clustering approach. While you may not always notice this,
because they hide the real implementation behind proxy servers.

=head2 Internet Connection

You have the best hardware you can get, but the service is still
crawling. Make sure you have a fast Internet connection. Not as fast
as your ISP claims it to be, but fast as it should be. The ISP might
have a very good connection to the Internet, but puts many clients on
the same line. If these are heavy clients, your traffic will have to
share the same line and the throughput will decline. Think about a
dedicated connection and make sure it is truly dedicated. Trust the
ISP but check it!

The idea of having a connection to B<The Internet> is a little
misleading. Many Web hosting and co-location companies have large
amounts of bandwidth, but still have poor connectivity. The public
exchanges, such as MAE-East and MAE-West, frequently become
overloaded, yet many ISPs depend on these exchanges.

Private peering means that providers can exchange traffic much
quicker.

Also, if your Web site is of global interest, check that the ISP
has good global connectivity. If the Web site is going to be visited
mostly by people in a certain country or region, your server should
probably be located there.

And bad connectivity can directly influence your machine's
performance. Here is a story, one of the developers told on the
mod_perl mailing list:

  What relationship has 10% packet loss on one upstream provider got
  to do with machine memory ?

  Yes.. a lot. For a nightmare week, the box was located downstream of
  a provider who was struggling with some serious bandwidth problems
  of his own... people were connecting to the site via this link, and
  packet loss was such that retransmits and tcp stalls were keeping
  httpd heavies around for much longer than normal.. instead of
  blasting out the data at high or even modem speeds, they would be
  stuck at 1k/sec or stalled out...  people would press stop and
  refresh, httpds would take 300 seconds to timeout on writes to
  no-one.. it was a nightmare.  Those problems didn't go away till I
  moved the box to a place closer to some decent backbones.

  Note that with a proxy, this only keeps a lightweight httpd tied up,
  assuming the page is small enough to fit in the buffers.  If you are
  a busy internet site you always have some slow clients.  This is a
  difficult thing to simulate in benchmark testing, though.


=head2 I/O performance

If your service is I/O bound (does a lot of read/write operations to
disk, remember that relational databases are sitting on disk as well),
you need a very fast disk. So you should not spend money on Video card
and monitor (monochrome card and 14" B&W are perfectly adequate for a
server -- you will probably be telnetted or ssh-ed in most of the
time), but rather look for disks with the best price/performance
ratio. Of course, ask around and avoid disks that have a reputation
for headcrashes and other disasters.

With money in hand you should think about getting a RAID system. RAID
is generally a box with many HDs. It is capable of reading and writing
data much faster, and is protected against disk failures. It does this
by duplicating the same data over a number of disks, so if one fails,
the RAID controller detects it and the data is still correct on the
duplicated disks. You must think about RAID or similar systems if you
have an enormous data set to serve. (What is an enormous data set
nowadays? Gigabytes, terabytes?).

Ok, we have a fast disk, what's next? You need a fast disk
controller. So either you should use the one embedded on your
motherboard or you should plug a controller card if the one you have
onboard is not good enough.

=head2 Memory

How much RAM (Randomly Accessed Memory) do you need? Nowadays, chances
are you will hear: "Memory is cheap, the more you buy the better". But
how much is enough? The answer is pretty straightforward: "You do not
want your machine to swap". When the CPU needs to write something into
memory, but notices that it is already full, it takes the least
frequently used memory pages and swaps them out. Swapping out means
writing the data to disk.  Another process then references some of its
own data, which happens to be on one of the pages that has just been
swapped out. The CPU, ever obliging, swaps it back in again, probably
swapping out some other data that will be needed very shortly by
another process. Carried to the extreme, the CPU and disk start to
thrash hopelessly in circles, without getting any real work done.  The
less RAM there is, the more often this scenario arises. Worse, you can
exhaust swap space as well, and then the troubles really start...

How do you make a decision? You know the highest rate at which your
server expects to serve pages and how long it takes to do so. Now you
can calculate how many server processes you need. Knowing the maximum
size any of your servers can get, you know how much memory you
need. You probably need less memory than you have calculated if your
OS supports memory sharing and you know how to make best use of this
feature (preloading the modules and scripts at server startup). Do not
forget that other essential system processes need memory as well, so
you should plan not only for the web server, but also take into
account the other players. Remember that requests can be queued, so
you can afford to let your client wait for a few moments until a
server is available to serve it; your numbers will be more correct,
since you generally do not have the highest load, but you should be
ready to bear the peaks. So you need to reserve at least 20% of free
memory for peak situations. Many sites have crashed a few moments
after a big scoop about them was posted and an unexpected number of
requests suddenly came in. (This is called the Slashdot effect, which
was born at http://slashdot.org ) If you are about to announce
something cool, be aware of the possible consequences.

=head2 Bottlenecks

The most important thing to understand is that you might use the most
expensive components, but still get bad performance. Why? Let me
introduce an annoying word: bottleneck.

A machine is an aggregate of many big and small components. Each one
of them may be a bottleneck. If you have a fast processor but a small
amount of RAM (memory), the processor will be under-utilized waiting
for the kernel to swap the memory pages in and out, because memory is
too small to hold the most used ones. If you have a lot of memory and
a fast processor and a fast disk, but a slow controller - the
performance will still be bad, and you will have wasted money.

Use a fast NIC (Network Interface Card) that does not create a
bottleneck. If it is slow, the whole service is slow. This is the most
important component, since webservers are much more network-bound than
disk-bound!

=head2 Conclusion

To use your money optimally you have to understand the hardware very
well, so you will know what to pick. Otherwise, you should hire a
knowledgeable hardware consultants and employ them on a regular basis,
since your needs will probably change as time goes by and your
hardware will likewise be forced to adapt as well.

=cut



