Red Leader

A Red Leader is a role which is responsible for managing content transfers over a WAN. One Red Leader is assigned for each subnet. This is a 100% dynamic, automatic process which may change at any time during a download in order to optimize performance.

The figure below shows a StifleR environment of 3 subnets, grouped into a Network Group and a single Location, each with Leaders assigned.

Figure 8 shows a typical StifleR standard setup with Red and Blue Leaders assigned per subnet. Each Red Leader will download the source data and simultaneously share this with its subnet peers.

Red Leader Assignment

The Red Leader assignment is completely dynamic and may switch several times during downloads in order to ensure the most efficient usage of WAN and LAN infrastructure.

Typically, the first machine in a subnet that connects to the StifleR Server will be assigned as the Red Leader. That Red Leader then remains as Red Leader until a more suitable candidate connects in which case the new Red Leader is assigned and the old Red Leader gracefully retires into network obscurity.

Red Leader Traffic Flow

The following visualization depicts the flow between Server and Client.

Figure above shows traffic flow between StifleR server and clients. (NB. All traffic is UDP based and as such may be lost in transit) StifleR is designed to handle such interruption. *A client will De-Nominate itself when it has been unable to Check In to the Server within the threshold timeout

When Is a New Red Leader Re-assigned?

StifleR checks and compares the following for each client as part of the assignment process:

• If the current Red Leader has not responded to the Server within the configured threshold value

  • This typically occurs where there is a communication breakdown between a nominated Red Leader and the StifleR Server. The Server sends a message to appoint the Red Leader and the Red Leader must confirm the appointment within the threshold time. In the case a client cannot be assigned for a Network Group a warning will be recorded in the Windows Event Log and the Server will move on to the next best nominee

• If the Current Red Leader has been marked with the “NotLeaderMaterial” flag set to True:

  • It may be undesirable for certain clients to become Red Leaders and in such cases this can be configured on the client (by setting the NotLeaderMaterial setting value to 1 (in the StifleR.ClientApp.exe.config file) or scripted on the Server by setting the NotLeaderMaterial WMI property on the Client instance

  • A Client with a higher priority job starts

    • The higher priority job Client will become Red Leader

• A Client with the same priority but a higher BranchCache version connects to the Server

  • Higher BranchCache version Clients will be a preferred Red Leader as a V2 capable system Is able to utilize Deduplication during the download and can also provide a V1 hash to Windows 7 clients

• If a client connects that has downloaded more than the other clients over a configured threshold (10% more)

  • If the newly connected client holds more than 10% of the download in it’s cache than the current Red Leader it will become the preferred nominee and be appointed Red Leader

• If we have more data transferred (10% more) and priority is same or higher and BranchCache version is same or higher but uptime is higher

  • As most machines at the start of a job ends up within the same range, this is the most common rule that applies. Most Red Leaders have a high uptime

Generic Red Leader Selection Example

1. Machine1 connects and the StifleR server detects that no other clients are online for that segment. The client is not marked with NotLeaderMaterial so it will be selected as Red Leader. Machine1 is not currently transferring any active jobs. (Windows Service Mode)

2. Machine4 connects on the same segment. It is transferring a Low priority job and will be assigned as Red Leader. Server Assigns Machine4 to be Leader and retires Machine1

3. Machine5 connects and is transferring the same job as Machine4. Machine5 has a higher uptime, which will rank it higher than Machine4, but it has also transferred more or less as much as Machine4. Machine4 stays as Red Leader but Machine5 gets a BITS policy that stops it from actively transferring any large amount of data. Machine4 gets the high bandwidth transfer throttle value defined for that subnet

4. Machine6 comes online and is transferring a high priority job. Machine6 becomes the new Red Leader and the downloads for Machine4 and 5 is then stopped as both receive a low BITS policy value

5. Machine7 comes online and is a higher BranchCache version (Windows 10=V2) than the others (Windows 7=V1). As Machine7 is also downloading this high priority job, even though it has transferred less than Machine6 it is assigned Red Leader

6. After Machine6 and 7 complete their downloads Machine 5 becomes Red Leader

7. As soon as Machine4 and 5 complete their downloads Machine7 becomes the Red Leader as it has the highest BranchCache version

8. No client is actively transferring any data and Machine7 stays the Red Leader

Red Leader Selection Process High Level – Visualized

The following section shows a high level overview of how a Red Leader is used to transfer data and bandwidth throttling.

First off, for example a BITS job is started:

Figure above shows three clients in the same subnet starting the same BITS download at the same time. (Assume same operating system version). Each client connects to the content provider via BITS and checks in to the StifleR server via Stifler Client

Figure above shows three clients in the same subnet, The server chooses the best candidate to be Red Leader and communicates to it directly that it has been appointed Red Leader. Client1 is assigned Red Leader.

When the BITS job starts the StifleR clients send up statistics to the StifleR server. This provides the server with the information needed to select the best candidate to be Red Leader. In this case Client 1. As Client 1 downloads it will share content with the peer clients on the LAN. At the same time the peer clients 2 and 3 are automatically configured to throttle down their WAN BITS download speed leaving maximum bandwidth to the Content Server available for the Red Leader.

Once a Leader has been selected the StifleR server starts to send regular configuration messages to the Leader every x (configurable) seconds. The new leader uses these messages to set the (centrally configured) maximum bandwidth through an installed (Microsoft signed) filter driver. The message also includes flags that change other features such as InterVLAN transfers, BranchCache V1 content generation etc.

Figure above shows that Client 1 has gone AWOL. The server will pick up on this and automatically assign a new Red Leader for the subnet

Each Leader reports up to the StifleR server at a regular interval (tracked as LastCheckIn Value). In the event that this communication stops for a period the StifleR server will automatically mark the failed Red Leader as NotLeaderMaterial and assign a new Red Leader to take over the WAN download.

Figure above shows that Client 2 is assigned Red Leader. As soon as Client 2 starts downloading data this is picked up by Client 3 which then sources the data from Client 2 using BranchCache.

Seamlessly the other peers will swap sources to continue the download from the new Red Leader.