DHCP

Dynamic Host Configuration Protocol (DHCP) servers are used to automate the process of providing an IP configuration to devices in the network. These servers respond to broadcast-based requests for a configuration by offering an IP address, subnet mask, and default gateway to the DHCP client. While these options provide basic network connectivity, many other options can also be provided, such as the IP address of a Trivial File Transfer Protocol (TFTP) server that IP phones can contact to download a configuration file. Here are some of the components that should be maximized to ensure good performance:

CPU Processing power is not one of the critical resources on a DHCP server. Ensure you have the recommended speed for the operating system with the DHCP service installed as indicated by the operating system documentation.

Memory RAM is one of the critical components on the DHCP server. You should provide plenty of RAM. After deployment you may discover you need to add more if you find the server’s usage of memory to be exceeding its capacity.

Disk Disk is the second critical component. If you intend to host a large number of scopes (scopes are ranges of IP addresses) on the server, keep in mind that each time you add a scope you need more disk space.

NIC Although this is not a component that’s critical to the operation of DHCP, the NIC could become a bottleneck in the system if it is not capable enough or if there are not enough NICs. Use at least 1 Gbps NICs and consider the option of using multiple NICs.

DNS

Domain Name System (DNS) servers resolve device and domain names (website names) to IP addresses, and vice versa. They make it possible to connect to either without knowing the IP address of the device or the server hosting the website. Clients are configured with the IP address of a DNS server (usually through DHCP) and make requests of the server using queries. The organization’s DNS server will be configured to perform the lookup of IP addresses for which it has no entry in its database by making requests of the DNS servers on the Internet, which are organized in a hierarchy that allows these servers to more efficiently provide the answer. When the DNS servers have completed their lookup, they return the IP address to the client so the client can make a direct connection using the IP address. Here are some of the components that should be maximized to ensure good performance:

RAM Memory is the most important resource because the entire DNS zone file will be loaded into memory. As you add zones, or as a zone gets larger, you should add memory. If you expect the server to be answering large numbers of queries concurrently, then that will require additional memory as well.

Disk Disk is not a critical component, but it does need to be large enough to hold all of the zones as well.

NIC The NIC is not a critical resource unless it becomes a bottleneck during times when there is an overwhelming number of concurrent queries. If you find that to be the case often, add more NICs.

CPU CPU is not a critical resource, but if you expect a heavy query load then you should get the fastest processor the budget allows. The CPU can become a bottleneck just as the NIC can in times of high workload.

NTP

Network Time Protocol (NTP) servers are used as a time source by the devices in the network. When the devices are configured with the address of an NTP server, they will periodically check in with the server to ensure their time is correct. This service ensures that log entries that are time stamped can be properly interpreted and that digital certificates, which depend heavily on time, continue to function correctly. NTP servers do not typically have high resource requirements, and this service is a good candidate for adding to a server already performing another role.

Operating System Updates

All operating system vendors periodically offer updates for their systems. These updates sometimes fix bugs or issues, and sometimes they add functionality. In some cases they close security loopholes that have been discovered. Regardless of the reason for their issuance, you should obtain, test, and deploy all of these updates. Many vendors like Microsoft makes that simple by including tools such as Windows Update that allow you to automate the process. If the operating system does not provide such a utility, then you should take it upon yourself to seek it out. In Exercise 3.1 you will schedule updates using the Windows Update tool in Windows Server 2012 R2.

Application Updates

Many servers run applications and services that require updating from time to time. The server might be running a database program or email software or any of the services required for the server to perform the roles described in the earlier section “Server Roles and Requirements.” Although keeping up with these updates may not be as simple as operating system updates (which in many cases can be automated), you should develop a system to keep aware of these updates when they are available. You might get on the mailing list of the application vendor so that you can be informed when updates are available. If all else fails, make it a point to visit their website from time to time and check for these updates.

Security Software Updates

These are among the most critical updates we will cover in this section. These updates include those for antivirus and antimalware. This software should be regularly updated (with particular interest paid to the definition files). While steps and options to configure these updates are unique to each product, with respect to the built-in Windows tools, such as Windows Defender, those definitions will be updated along with other updates you receive from Windows Update. Most vendors of security software provide a mechanism to download and apply these updates automatically. You should take advantage of such a mechanism if it is available. Although this software is important for users’ machines, it is even more essential for servers, which are frequent targets of attacks.

Firmware Updates

Any software that is built into a hardware device is called firmware. Firmware is typically in flash ROM and can be updated as newer versions become available. An example of firmware is the software in a laser printer that controls it and allows you to interact with it at the console (usually through a limited menu of options). Firmware updates are often released by hardware vendors. You should routinely check their websites for updates that you need to download and install.

Device Driver Updates

Device drivers are files that allow devices to communicate with the operating system. Called drivers for short, they’re used for interacting with printers, monitors, network cards, sound cards, and just about every type of hardware attached to the server. One of the most common problems associated with drivers is not having the current version—as problems are fixed, the drivers are updated. Typically driver updates don’t come into the picture until something stops working. When that is the case, you should consider the possibility that a driver update for the device might solve the issue before wasting a lot of time troubleshooting.

Compatibility Lists

When deploying a new server or when adding devices and applications to a server, it can be highly beneficial to ensure compatibility between the new addition and the server before spending money. Vendors of both software and hardware create compatibility lists that can be used to ensure that a potential piece of software or hardware will work with the server. These compatibility lists fall into three categories, as covered in the following sections.

Testing and Validation

We’ve just finished discussing all kinds of updates that you might apply to a server. Because servers are so important to the function of the network, you must treat updates for them with a dose of caution and skepticism. It is impossible for vendors to anticipate every scenario in which the server might be operating and thus impossible for them to provide assurance that the update won’t break something (like a mission-critical operation). With the potential exception of security updates, you should always test the updates and validate their compatibility with your set of hardware and software.

These tests should occur in a test network and not on production servers. One of the benefits of having a virtualization environment is the ability to take images of your servers and test the updates on those images in a virtual environment. Once the updates have been validated as compatible, you can deploy them to the live servers.

Scheduled Downtime

An SLA might specify that taking any infrastructure equipment down for maintenance requires prior notification and that the event may only occur during the evening or on the weekend. This type of event is called scheduled downtime. It is important that when you schedule this event, you allow enough time in the window to accomplish what you need to, and you should always have a plan in case it becomes obvious that the maintenance cannot be completed in the scheduled downtime.

Unscheduled Downtime

Unscheduled downtime is the nightmare of any administrator. This is when a device is down or malfunctioning outside of a scheduled maintenance window. Some SLAs might include an unscheduled downtime clause that penalizes the support team in some way after the amount of unscheduled downtime exceeds a predetermined level. If the support is being provided to a customer, there might be a financial penalty. If the SLA applies to another department in the same organization, the penalty might be in a different form.

Impact Analysis

Another typical issue requirement in most SLAs is that any schedule downtime should be communicated to all users and that the communication should include the specific parties that will be affected by the downtime and exactly what the impact will be. For example, a notification email describing the scheduled downtime might include the following:

• Departments affected
• Specific applications or service impacted
• Expected length of downtime

Client Notification

SLAs should also specify events that require client notification and the time period in which that must occur. It will cover all scheduled downtime but should also address the time period in which notifications should be sent out when unscheduled events occur. It might also discuss parameters for responding to client requests.

MTTR

One of the metrics that is used in planning both SLAs and IT operations in general is mean time to repair (MTTR). This value describes the average length of time it takes a vendor to repair a device or component. By building MTTR into the SLA, IT can assure that the time taken to repair a component or device will not be a factor that causes them to violate the SLA requirements. Sometimes MTTR is considered to be from the point at which the failure is first discovered to the point at which the equipment returns to operation. In other cases it is a measure of the elapsed time between the point where repairs actually begin and the point at which the equipment returns to operation. It is important that there is a clear understanding by all parties with regard to when the clock starts and ends when calculating MTTR.

Comparison Against Performance Baseline

If you have ever tried to look up a particular performance metric on the Internet to find out what a “normal” value is, you may be quite frustrated that you can’t get a straight answer. This is because you have to define what is normal for your network. This is done by creating a set of performance metrics during normal operations when all is working well. While earlier in this chapter we listed some rough guidelines about certain metrics, that is all they are: guidelines.

Only until you have created this set of values called a performance baseline can you say what is normal for your network. Creating the set of values should be done during regular operations, not during either times of unusual workload or times of very little workload. After this you can compare the performance metrics you generate later with this baseline. Let’s look at the four main resources.

CPU Utilization

When monitoring CPU, the specific counters you use depends on the server role. Consult the vendor’s documentation for information on those counters and what they mean to the performance of the service or application. Common counters monitored by server administrators are

Processor\% Processor Time The percentage of time the CPU spends executing a non-idle thread. This should not be over 85 percent on a sustained basis.

Processor\% User Time Represents the percentage of time the CPU spends in user mode, which means it is doing work for an application. If this value is higher than the baseline you captured during normal operation, the service or application is dominating the CPU.

Processor\% Interrupt Time The percentage of time the CPU receives and services hardware interrupts during specific sample intervals. If this is over 15 percent, there could be a hardware issue.

SystemProcessor Queue Length The number of threads (which are smaller pieces of an overall operation) in the processor queue. If this value is over 2 times the number of CPUs, the server is not keeping up with the workload.

Memory Utilization

As with CPUs, different server roles place different demands on the memory, so there may be specific counters of interest you can learn by consulting the vendor documentation. Common counters monitored by server administrators are

Memory\% Committed Bytes in Use The amount of virtual memory in use. If this is over 80 percent, you need more memory.

MemoryAvailable Mbytes The amount of physical memory (in megabytes) currently available. If this is less than 5 percent, you need more memory.

MemoryFree System Page Table Entries The number of entries in the page table not currently in use by the system. If the number is less than 5,000, there may be a memory leak (memory leaks occur when an application is issued memory that is not returned to the system. Over time this drains the server of memory.).

MemoryPool Non-Paged Bytes The size, in bytes, of the non-paged pool, which contains objects that cannot be paged to the disk. If the value is greater than 175 MB, you may have a memory leak (an application is not releasing its allocated memory when it is done).

MemoryPool Paged Bytes The size, in bytes, of the paged pool, which contains objects that can be paged to disk. (If this value is greater than 250 MB, there may be a memory leak.)

MemoryPages per Second The rate at which pages are written to and read from the disk during paging. If the value is greater than 1,000 as a result of excessive paging, there may be a memory leak.

Network Utilization

As you learned in the section “Server Roles and Requirements,” the NIC can become a bottleneck in the system if it cannot keep up with the traffic. Some common counters monitored by server administrators are

Network InterfaceBytes Total/Sec The percentage of bandwidth of which the NIC is capable that is currently being used. If this value is more than 70 percent of the bandwidth of the interface, the interface is saturated or not keeping up.

Network InterfaceOutput Queue Length The number of packets in the output queue. If this value is over 2, the NIC is not keeping up with the workload.

Disk Utilization

On several of the server roles we discussed, disk was the critical resource. When monitoring the disk subsystem, you must consider two issues: the speed with which the disk is being accessed and the capacity you have available. Let’s examine those metrics in more detail.

Storage Capacity

The second metric of interest when designing a storage solution is capacity. When planning and managing storage capacity, consider the following questions:

• What do you presently need? Remember that this includes not only the total amount of data you have to store but also the cost to the system for fault tolerance. For example, if data is located on a RAID 1 or mirrored drive, you need twice as much space for the data. Moreover, if the data is located on a RAID 5 array, your needs will depend on the number of drives in the array. As the number of drives in the array go up, the amount of space required for the parity information goes down as a percentage of the space in the drive. For example, with three drives you are losing 33 percent of the space for parity, but if you add another drive, that goes down to 25 percent of the space used for parity. Add another, and it goes down to 20 percent.
• How fasts are the needs growing? Remember that your needs are not static. They are changing at all times and probably growing. You can calculate this growth rate manually or you can use capacity-planning tools. If you have different classes of storage, you may want to make this calculation per class rather than overall because the growth rates may be significantly different for different classes.
• Are there major expansions on the table? If you are aware that major additions are occurring, you should reflect that in your growth rate. For example, if you know you adding a datacenter requiring a certain amount of capacity, that capacity should be added to the growth rate.

In many cases you find yourself in a crunch for space at a time when capacity cannot be added. In such cases, you should be aware of techniques for maximizing the space you have. Some of those approaches are as follows:

Using Disk Deduplication Tools In many cases, data is located on the same drive. Deduplication tools remove this redundancy while still making the data available at any location it was previously located by pointing to the single location. This process can be done after the data is on the drive or it can be implemented inline, which means data being written to the drive is examined first to see if it already exists on the drive. These two processes are shown in Figure 3.2.

Archiving Older Data You can also choose to have the older data moved off main storage to backup tapes or media to free up space.

Processes and Services Monitoring

On a more granular basis, you can separate the workload created by specific applications and services on a server. You do this by monitoring performance counters specific to the service or application. Often this exercise is undertaken when a server is experiencing a high workload and you need to determine the source of this workload. If you need to identify the performance counter that applies to an application or service, the best place to go for that information is the application or operating system vendor.

Log Monitoring

When you discover that a service or application is using more resources than normal, you may want to investigate the logs related to that service or application. Most server operating systems create these logs, or you may decide that it is easier to invest in a log monitoring system that can grab these logs off multiple servers and make them available to you in a central interface of some sort. This can be done in Windows by creating what are called event subscriptions. This simply means that one server subscribes to access a log on another server and make it available in the console of the first server. This is also called log forwarding. This concept is shown in Figure 3.3.

Check System Health Indicators

First there are some indicator mechanisms that will be provided by the vendor of the hardware and the software that can give you an early warning that something is amiss or is about to go bad. You should always react to these indicators in the same way you would react to a warning that your car is overheating, because they typically don’t come into play until the situation is getting serious.

LEDs

Most network devices, including servers, have LEDs on them that indicate certain things. The LED diagram for a Sun Blade X6250 Server Module is shown in Figure 3.4. The purpose of the LEDS and buttons is as follows:

• #1: LED that helps you identify which system you are working on
• #2: Indicates whether the server module is ready to be removed from the chassis
• #3: Service Action Required
• #4: Power/OK

Amount of memory has changed

LCD Messages

Many hardware devices like servers also have small LCD screens on the front that may be a source of messages that are helpful during troubleshooting. The Dell PowerEdge has the panel shown in Figure 3.5.

Not only is this panel used to make configurations, but it also displays error messages. When it does this, it changes the backlight color from blue to amber so that you notice it quicker. The types of alerts covered are as follows:

• Cable and board presence
• Temperature
• Voltages
• Fans
• Processors
• Memory
• Power supplies
• BIOS
• Hard drives

IBM calls their system light path diagnostics. This is a system consisting of an LED panel on the front and a series of LEDs inside the box near various components. One of the LEDs is the Information LED, and when it indicated there is an error, you can open the box and LEDs inside the box might be lit near the problem component, as shown in Figure 3.6.

Replace Failed Components

Inevitably, parts will go bad and you will have to replace them. Common components replacements are covered in this section.

Preventive Maintenance

Even when the servers are humming along happily with no issues, there are maintenance tasks that, if performed regularly, will increase the life of your servers and ward off avoidable issues. Two of these will go a long way toward avoiding overheating issues.

Proper Shutdown Procedures

When shutting down the server for maintenance, make sure you follow proper shutdown procedures. There are two types of reboots:

Soft Reboot Better for the system than a hard reboot, a soft reboot allows the proper shutdown of all running applications and services, yet requires that you be logged in as either administrator or root and that the server be in a responsive state. It is also good to know that since power is not completely removed, memory registers are not cleared.

Hard Reboot A hard reboot is not good for the system and equivalent to turning off the power and turning it back on. However, in cases where the server is unresponsive, a hard reboot may be the only option.

Always use a soft reboot whenever possible. A hard reboot does not give the server an opportunity to properly shut down all running applications and services.

Clustering

Clustering is the process of combining multiple physical or virtual servers together in an arrangement called a cluster, in which the servers work together to service the same workload or application. This may be done to increase performance or to ensure continued access to the service if a server goes down, or its goal could be both. A server cluster is generally recommended for servers running applications that have long-running in-memory state or frequently updated data. Typical uses for server clusters include file servers, print servers, database servers, and messaging servers. Clustering can be implemented in one of two ways: active/active or active/passive.

Active/Active

In an active/active cluster, both or all servers are actively servicing the workload. That doesn’t necessarily mean they are running the same applications at all times, but only that they are capable of taking over the workload of an application running on another cluster member if that member goes down. For example, in Figure 3.7 the server on the left is running two applications prior to failing, whereas the server on the right is running only one of the three applications. After the failure, the server on the right takes over the work of all three applications. With this arrangement you must ensure that the remaining server can handle the total workload.

Active/Passive

In an active/passive cluster, at least one of the servers in the cluster is not actively working but simply sitting idle until a failure occurs. This is shown in Figure 3.8. This arrangement comes at the cost of having a server sitting idle until a failure occurs. It has the benefit of providing more assurance that the workload will continue to be serviced with the same level of resources if the servers are alike.

Load Balancing

A second form of fault tolerance that focuses more on providing high availability of a resource is load balancing. In load balancing a frontend device or service receives work requests and allocates the requests to a number of backend servers. This type of fault tolerance is recommended for applications that do not have long-running in-memory state or frequently updated data. Web servers are good candidates for load balancing. The load balancing device or service can use several methods to allocate the work. Let’s look at the most common allocation method, round robin, and also talk about a key function in the load balancing system, the heartbeat.

Round Robin

In a round robin allocation system, the load balancer allocates work requests to each server sequentially, resulting in each getting an equal number of requests. As shown in Figure 3.9, this could mean that a user may make two requests that actually go to two different servers.

Heartbeat

A heartbeat connection is a connection between servers in a load balancing scenario across which the servers send a signal (called a heartbeat) used to determine when the other server is down. If one server goes down, the other will service the entire workload. The two servers are typically identical (content-wise), and this is used often as a failover technique. It may be a direct physical connection, as shown in Figure 3.10, or it might be done over the network.

Licensing

When you purchase software, you are purchasing the right to use it. At any point in time you may face a software audit from one of the major vendors. When this occurs, you will need to be able to provide written proof that you possess a number of licenses for a particular product that is equal or greater than the number of installations in your network. When that time comes, will you be able to locate these? They should be kept in a safe place where you can put your hand on them at a moment’s notice. That does not mean the records can’t be digital, but they must be available.

Labeling

Labeling servers, workstations, printers, ports on infrastructure devices (routers and switches), and other items is another form of asset documentation that often doesn’t receive enough attention. Not only does this make your day-to-day duties easier, it makes the process of maintaining accurate records simpler and supports a proper asset management plan. When periodic inventories are taken (you are doing that, right?), having these items labeled makes the process so much quicker. This goes for cables in the server room as well.

Warranty

Warranty information should be readily available to you when equipment breaks. You should never spend time or money repairing items that are still under warranty. It should not take you hours to locate this information when that time comes. Keep this paperwork or its digital equivalent close at hand in the same way you would the licensing information.

Life-Cycle Management

Managing assets becomes easier if you understand that an asset goes through various stages called its life cycle. Consequently, life-cycle management comprises the activities that we undertake with respect to the asset at various points in the life cycle. Let’s examine these stages.

Inventory

As I mentioned at the start of this section, asset management includes knowing what you have. You can’t know something is missing until you take an inventory, so you should take inventory on a regular basis. So what type of information is useful to record? You may choose to record more, but the following items should always be included:

Make The manufacturer of the device should be recorded as well as the name they give the device.

Model The exact model number should be recorded in full, leaving nothing out. Sometimes those dangling letters at the end of the model number are there to indicate how this model differs from another, or they could indicate a feature, so record the entire number.

Serial Number The serial number of the device should be recorded. This is a number that will be important to you with respect to the warranty and service support. You should be able to put your hands on this number quickly.

Asset Tag If your organization places asset tags on devices, it probably means you have your own internal numbering or other identification system in place. Record that number and any other pertinent information that the organization deems important enough to place on the asset tag, such as region and building.

Service Manuals

All service manuals that arrive with new hardware should be kept. They are invaluable sources of information related to the use and maintenance of your devices. They also contain contact information that may make it easier to locate help at a critical time. Many manuals have troubleshooting flowcharts that may turn a 4-hour solution into a 30-minute one. If a paper copy has not been retained, you can usually obtain these service manuals online at the vendor website.

Network Diagrams

All network diagrams should be kept in both hard copy and digital format. Moreover, these diagrams must be closely integrated with the change management process. The change management policy (covered later in this section) should specifically call for the updating of the diagram at the conclusion of any change made to the network that impacts the diagram and should emphasize that no change procedure is considered complete unless this update has occurred.

Architecture Diagrams

Any diagrams created to depict the architecture of a software program or group of programs should be kept. When the original developers are no longer with the company, they are invaluable to those left behind to understand the workings of the software. There may multiple layers of this documentation. Some may only focus on a single piece of software whereas others may depict how the software fits into the overall business process of the company. An example of such a diagram, called an enterprise architecture diagram, is shown in Figure 3.11.

Dataflow Diagrams

While some of your network diagrams will focus on the physical pieces of the network, others will be focused on the flow of data in the network in the process of doing business. So these may depict a workflow and how information involved in a single transaction or business process goes from one logical component in the network to another. An example of a dataflow diagram for an order system is shown in Figure 3.12.

Recovery Documentation

If your organization has a disaster recovery plan, that plan should call for recovery documentation. It should outline, in detail, the order with which devices should be recovered in the event of a disaster that causes either complete or partial destruction of the facility. It should also cover the steps to be taken in lesser events as well, such as a power outage, the theft of a device, or the failure of a device, and data recovery procedures as well.

Baseline Documentation

Earlier you learned the importance of creating performance baselines for major systems. The baselines will be used as a comparison point to readings taken at regular intervals. This means that the baseline data will need to be available at all times. A plan should be in place to consider at regular intervals whether major changes in the network may require new baselines to be taken.

Change Management Policies

As a part of the overall security policy, the change management policy will outline the steps involved in suggesting, considering, planning, executing, and documenting any change in the server configuration. No change should be made without it undergoing this process. The policy should be available at times for consultation when questions arise about making configuration changes.

Service-Level Agreement

Service-level agreements (SLAs) can be made with customers or with departments within the organization. These documents describe the type of service to be provided, in what format, in what time frame, and at what cost. Some SLAs are executed with vendors that provide service to the organization. These documents should be readily available when disagreements arise and clarifications need to be made.

Server Configuration

The exact configuration of every server should be fully recorded and updated any time a change is made. The following is information that should be included:

General

• Server name
• Server location
• Function or purpose of the server
• Software running on the server, including operating system, programs, and services

Hardware

• Hardware components, including the make and model of each part of the system

Configuration Information

• Event logging settings
• Services that are running
• Configuration of any security lockdown tool or setting
• Configuration and settings of software running on the server

Data

• Types of data stored on the server
• Owners of the data stored on the server
• Sensitivity of data stored on the server
• Data that should be backed up, along with its location
• Users or groups with access to data stored on the server
• Administrators, with a list of rights of each
• Authentication process and protocols for users of data on the server
• Authentication process and protocols used for authentication
• Data encryption requirements
• Authentication encryption requirements

Users

• Account settings
• List of users accessing data from remote locations and type of media they access data through, such as the Internet or a private network
• List of administrators administrating the server from remote locations and type of media they access the server through

Security

• Intrusion detection and prevention method used on the server
• Latest patch to operating system and each service running
• Individuals with physical access to the area the server is in and the type of access, such as key or card access
• Emergency recovery disk and date of last update
• Disaster recovery plan and location of backup data