by Torben Buck Sørensen, Senior Software Architect
Distribution Server: In this brief document, I’ll discuss the problem arising when 1000 clients try to download a 4 GB software package from the same server at once, and what it takes to make it tolerable for the network administrator.
Prerequisites
In order to be able to follow the instructions in this document, you’ll need CapaInstaller 5.8 or newer. The component that is used is the BAADS (Base Agent As Distribution Server), which is a role that the ordinary Base Agent can have when configured to, in the System Plugin of the CapaInstaller Console.
The problem at hand
In its purest form, the horror scenario is to have 1000 clients simultaneously try to download the same 4 GB software package from the Front-end Server. This will not only halt the download of software packages, but it will also affect the normal use of the network that is not software distribution related, for all other users in the organization.
The Front-end Server is the main anchor point for client computers into the CapaInstaller installation. Normal operations will also be hit for clients, not related to the 4TB transfer. The problem that we want to solve is to let the Front-end Server handle unit enrollments, inventory reporting, and package installations without clogging up the network adapter with unnecessary file transfers. In order to accomplish that, we want the clients to collect and share files among them.
Be aware that not all files are being shared among clients, because there is an overhead in keeping files organized and published. For that reason, only files with specific file extensions are shared. The default list of file extensions is based on the assumption that they represent ‘larger’ files where the benefit of file sharing, in saved bandwidth (and server CPU), surpasses that of the overhead of keeping track of the files (client CPU). The list defaults to; *.kit, *.iso, *.msi, *.zip, *.exe, *.cab.
The ‘no Management’ scenario
This is the worst-case scenario where every client downloads the package from the Front-end Server.
As illustrated by the color of the clients, every file origin from a separate file transfer (No, client6 and client9 do not have the same color), letting the Front-end Server foot the bandwidth/CPU bill. This will influence other operations on the Front-end.
Activating File sharing
By letting the clients be aware of each other, it is possible, once the file to be downloaded has been identified by the Front-end Server, to find it on a nearby client. As illustrated by color, client1 got the file from client0. This goes for client3/5 and client9/6/7 as well.
This will spread out the use of bandwidth and free up the Front-end Server to better handle other tasks. This, however, has a few drawbacks of its own. There is a limit to how much a single client can serve other clients without being affected by the extra load itself. The default value for Base Agents sharing files is two simultaneous transfers.
Another thing to consider is that a client that is in search of a file may not find it on any nearby peer. There can be various reasons for that.
- It could be the first client on that subnet to download the file
- Since clients have a limit on the number of peers to be aware of, the file may not be in its vicinity. The default is 30
- One or some of the nearby peers are in progress of downloading the file but have not advertised the file because the download and MD5 verification has not yet completed.
Using a Distribution Server
By using a Distribution Server, a client, with a valid MD5 checksum but with no file to be found matching that checksum (in neither the file cache nor on nearby peers) will request the file from the Distribution Server.
The Distribution Server will request file information (MD5 checksum) from the Front-end Server and then determine if it already holds the requested file in its file cache. If it does, it will let the client download the file right away. If the Distribution Server does not hold the file, it will download it and subsequently let the client download the file.
If the Distribution Server (DS0) is actively downloading a file on behalf of client0 (referred to as the remote transfer), and client5 requests DS0 for the same file, DS0 will not start a new separate download from the Front-end Server but rather put client5 I queue. Client5 will then be aware of both the progress of the remote transfer and the fact that it is number two in the queue. (along with the Max Requesters property of the DS).
Once the remote transfer has completed, each queued up client will have three pieces of information.
- The DS now has the file that I want to download
- The DS has a limit of simultaneous file transfers (Max Requesters that defaults to 5)
- I have a number in the queue that may or may not exceed the limit.
If the number in the queue does not exceed the limit, the client will download the file right away. If the client, on the other hand, is number 7 in the queue with a limit of 5, it will observe the DS property ‘Retry Delay’. This value (that defaults to 90 seconds) represents the time that the client will wait for other clients to complete their file transfer. When done waiting for the ‘Retry Delay’, the client will go back to the process of discovery. This will either result in finding the file on one or more peers or a new file request to the DS.
Other benefits of using a Distribution Server
Prior to CapaInstaller 5.8, the Distribution Server was simply an SMB share on a server where files were copied from without deliberations of reuse. Furthermore, files copied from the Distribution Server were selected by path & filename, meaning that a file updated on the Front-end could be out-of-sync with the copy on the Distribution Server if File Sync had not yet completed. This synchronization is performed by the Synchronization service (CiSync) that requires an SQL connection to be signaled to start a synchronization job.
The new Base Agent-based Distribution Server does not rely on an SQL connection as it only communicates with the Front-end Server (Internal or public). The Front-end Server delivers Configuration to the DS, which can include a schedule for recurring file preloads that update the DS with the bulk of data that makes it possible to serve all clients.
Furthermore, files are selected by MD5 checksum, rather than path and filename. This ensures that if a file on the Front-end Server is updated (and consequently gets a new MD5 checksum), it will represent an entirely new file on the DS, resulting in a new download from the Front-end Server to the DS. The obvious benefit of this is that clients cannot get a file from a Distribution Server that is ‘too old’.
How to choose the Distribution Server mode
The Distribution Server operates in one of three modes.
Fetch
Fetch mode is the simple ‘deploy-and-forget’ approach. When a BA requests a file, the file can either be present on the DS or not be present. If present, it will be served to the BA, if not, it will first be downloaded from the FE and then served to the BA. The DS can keep track of multiple BA’s requesting the same file simultaneously, thus avoid downloading the same file in two similar requests at once. When a client (BA) requests a file that has been changed on the Management Server, it also means that the file being requested has a changed MD5 checksum. Therefore, the new/updated file is downloaded (fetched) from the FE dynamically.
FetchAndPreload (default)
The FetchAndPreload mode acts in the same way that the Fetch mode does, except that it can also preload the redistributable files (Redist) and packages (ComputerJobs and UserJobs) from the Management Server. This also replaces the functionality of the Synchronization Service (CiSync). The Preload can be done in three ways
- By applying a preload-schedule that will update the DS on a regular basis, i.e., every night between 02.00 and 04.00
- On-demand (Update now) as an action on a specific DS in the System plugin of the CapaInstaller console
- On-demand (Update now) as an action on a specific software package in the Configuration Management plugin of the CapaInstaller console.
Preload
The Preload mode supplies the preload functionality described in the FetchAndPreload mode (above) but will NOT fetch new files on demand when/if a file has been updated on the Management Server. This mode exists in order to comply with strict bandwidth restrictions between geographical locations where heavy use of bandwidth over WAN is prohibited. This mode also resembles that of the old CiSync approach except that when a new/updated file that does not reside on the DS is requested, the DS denies the file request from the BA. This will result in the software package changing from ‘Installing’ to ‘Cancelled’, meaning that the client will retry the package installation on the next agent run until a preload has been completed or the requested file is made available on a nearby peer (BA).
How to deploy a new Distribution Server
The Distribution Server (DS) can be deployed to any Base Agent (BA) running version 5.8 or above. All management of the DS is done in the System plugin of the CapaInstaller Console 
To deploy a DS, go to the System Plugin and under the Management Points, right-click the Management Server under which the DS should be located and select ‘Deploy Server’.
Then select; ‘Deply Distribution Server to a Windows-based computer/server’.
In the Base Agent field, type in the name of an existing Base Agent that is linked to a Management Server on the same physical server. It does not have to be in the same Management Point as the Management Server under which the new DS is about to be deployed.
Select the Service mode that fits your requirements.
If you selected either ‘FetchAndPreload’ or ‘Preload’ you could now set a preload schedule, but it is not required. Enter a (or keep the proposed) Cache location. Select the maximum size for the cache. You can switch to a relative size, by clicking ‘GB’ and selecting ‘%’. The same goes for the Minimum size to keep free on the disk.
Finally, select the maximum concurrent requesters, and click ‘Next’.
Then enter one or more IP ranges/MAC addresses to determine which client computers you want to connect to this Distribution Server and click Next.
Name the server, enter a (meaningful) Change description and click Next.
Then click the newly deployed Distribution Server. In the List view to the right, you can see how much CPU and memory that the DS is using. The values are updated every 10 seconds (every 2 seconds while preloading).
How to upgrade an old Distribution Server (to BAADS)
The HTTP approach of the DS58 does not use the Synchronization Service (CiSync) to update files from the Management Server. Instead, by using either the Fetch or FetchAndPreload modes, it can deliver files that are binarily equal to those stored on the Management Server. This is a huge step forward because an updated (or new) software package is available on the fly. This means that an outdated software package never will be handed over to a client, the moment it is replaced on the Management Server. By utilizing a Front-end Server in the DMZ that is available as a public URL for the client computers and the DS58, there is also no need for a direct SQL connection in a remote location (via the CiSync and the Data Connection Service).
How to upgrade
To begin with, the DS57 must have a Base Agent (BA) version 5.8 or newer installed and the HWInventory package must be run so that the Hardware inventory (in the Configuration Management plugin) can show the Base Agent version of 5.8.x.x. or newer.
Then, in the System plugin, right-click the DS57. The popup-menu will show the ‘Upgrade to Base Agent Distribution Server…’. If a 5.8 BaseAgent is not detected, the menu item will be grayed out, and the tooltip will inform you of the BaseAgent requirements.
When you have the adequate Base Agent installed on the DS57, the menu item will be enabled. Click it.
The Base Agent name is already filled out, and you can then fill out the rest of the Distribution Server properties for the new DS58. (See above).
