DRBD-Cookbook

How to create your own cluster solution, without SAN or NAS!

Joerg Christian Seubert

1. Edition

Contents

1  Introduction
1.1  Syntax of that book
1.2  Built-in bugs
1.3  Hostnames
2  Installation
2.1  Software
2.2  Requirements
3  Preliminary considerations
3.1  Disk drive – physically vs. LVM
3.2  Filesystem on the disk device
3.2.1  UFS / ext2
3.2.2  ext3 / ext4
3.2.3  xfs
3.2.4  BtrFS
3.2.5  OCFS2
3.2.6  Conclusion
4  Configuration-Basics on a two-node-cluster-array
4.1  SSH-Configuration
4.1.1  SSH-Key-Types
4.1.2  Recipe: Generate and distribute SSH keys
4.2  Configuration of the locale Disk devices
4.2.1  Recipe: Create LVM volume via YaST
4.2.2  Recipe: Create a LVM-Volume via shell
4.3  DRBD configuration
4.3.1  Performance
4.3.2  /etc/drbd.conf
4.3.3  /etc/drbd.d/global_common.conf
4.3.4  Resourceconfiguration
4.3.5  Resource Description:
4.3.6  Alternative notation
4.3.7  Ports
4.3.8  Recipe: Command sequence for the basic configuration
5  Data transmission in the backbone LAN
5.1  Recipe: Backbone LAN in the DRBD configuration
6  Multi-node cluster
6.1  Stacking-Device
6.1.1  Recipe: Commissioning a stacking device with DRBD8 under SLES11 SP4
6.1.2  Recipe: Commissioning a stacking device with DRBD9 under OpenSuSE 15.1
6.2  RAID1 over at least three nodes
7  Hardened cluster
7.1  Detecting the state of the firewall
7.2  Firewallzones
7.3  Build a new firewall-service
7.4  Build a new zone
7.5  Basics of a two-stage hardening concept
7.5.1  Hardening with the firewall
7.5.2  Hardening with the Secure-Shell-Options
8  Increase / decrease the size of the DRBD device
8.1  Unofficial method
8.1.1  Concept
8.1.2  Command sequence
8.2  The official way
9  Program your own cluster solution
9.1  Configuration file
9.1.1  Content of the configuration file
9.2  Virtual IP-address
9.2.1  Script explanation for my_virt_ip.pl:
9.3  Make database switchable
9.3.1  Script explanation for my_dev_switch.pl:
9.4  Communication between cluster nodes - the Horcher
9.4.1  Script explanation for my_horcher.pl:
9.5  Controlscript
9.5.1  Script explanation for my_control.pl:
9.6  Service control scripts for systemd
9.6.1  Explanation for mycluster_horcher.service:
9.6.2  Explanation for mycluster_control.service:
9.7  Initialization script for the cluster controller
9.7.1  Explanation for my_service.pl:
9.8  Maintenance
9.9  General information about the scripts
10  Include DRBD in Veritas Cluster
10.1  Include DRBD as Veritas-Agent
10.2  Increase or decrease a DRBD-Device, which is included in the Veritas-Cluster
11  DRBD and Docker
11.1  Preparation and first start of the container
11.2  Work with the container
12  Win-DRBD
13  SSH-Configuration on SLE 15 / OpenSuSE Leap 15.x
14  Creating a LVM-Volume-Group
14.1  YaST
14.2  Shell
15  Stop, start, enable and disable services - a little tutorial
15.1  Copying servicefiles
15.2  Use services via YaST
15.3  Operate services with systemctl
16  Sources and disclaimer
16.1  Sources
16.1.1  Internet
16.1.2  Books
16.2  Disclaimer:
17  About…
17.1  …the book…
17.2  …the author…
17.3  Legal notice:

1  Introduction

If you want to build a cluster, sooner or later you face the problem that the data must be usable on all participating servers. This problem can be solved by transporting the data once per minute from the active cluster node to the passive cluster node.
But what, if this "copy job" takes longer than one minute?
In this case, you either have the situation that the copy jobs overtake each other and never end, because the cluster node in question does nothing else but ’copy’ - and ’nothing else’ means ’nothing else’ - or the data is outdated every time.
Neither makes sense and is not desirable.
If you have the additional situation that all cluster nodes must not only read the data but also write it, ’practical copying’ no longer makes any sense at all. Usually, this problem is solved by using a SAN or NAS.
For a data center, where there are usually more than two machines running at 24 * 7 uptime, it may not be a problem to run one more machine per cluster group - this can be a ’disk pot’, known as a true SAN (storage area network), or it can be a network file server, known as a NAS (network-attached-storage).
However, small businesses and home users face the problem of having to pay for a SAN or NAS.
That’s where the DRBD - Distributed Replicated Block Device - product from LinBit (www.linbit.com) comes in.
DRBD gives you the ability to connect two or more cluster nodes together without using a SAN or NAS as a data device. DRBD runs like a local RAID controller creating a mirror device (RAID 1) - but with "local disks" connected by a LAN.
You can also use this variant in a large data center if your cluster needs to be independent of a SAN or NAS. For example, you can think of a monitoring server that monitors the SAN or NAS and has to run highly available, especially when the SAN or NAS is not running.
This cookbook teaches the basics of a DRBD active-passive cluster, extended by further possibilities (three-node cluster, backbone LAN, deployment of DRBD on a Veritas cluster, creation of an own cluster via PERL, cluster configuration via hardware systems and many more) and demonstrates the procedures in the form of ’listings’.
All examples are based on a test configuration with OpenSuSE Leap 15.1 (except 6.1.1) and can - with the necessary background knowledge - also be implemented in other Linux distributions.
In the text number 6.1.1 the listing is done with SLES 11 SP 4 to show the commands and screen outputs of DRBD version 8 compared to DRBD version 9 because there are some differences.
For using DRBD on Windows-Servers use chapter 12.

1.1  Syntax of that book

To distinguish keyboard inputs and on-screen outputs from the explanations, the commands and on-screen outputs are displayed as follows:

Listing 1.1:
example of a session

hostname:~ # echo "This is an example!" 
This is an example!

In the scripts, the individual lines are numbered consecutively and the individual lines are briefly explained in tabular form in the text following the respective listing.
This means that the commands of the "recipes" can be entered on the shell as shown in the examples. The screen output should also be, as shown. The disclaimer (16.2) is explicitly pointed out here, because your systems do not have to match my systems.

1.2  Built-in bugs

In the course of creating this book, I made various mistakes while working out the recipes, which, after careful consideration, I simply took over into the recipe.
The reason for this is that these mistakes can also happen to you, during operation.
In the context of the respective cooking recipe, I then corrected these mistakes again - also to show you how to save the situation, and which factors - not clearly visible at first - had an influence on the respective error situation.
In this way, you can learn from my mistakes to avoid or solve similar mistakes in your systems.

1.3  Hostnames

In an old Siemens-Nixdorf-UNIX-manual, the configuration was explained using hostnamens like Jupiter and Saturn.
Because the dwarf planet pair Pluto and Charon (Charon is the greatest companion of the dwarf planet Pluto) have their common center of gravity, around which they circle, outside of their respective counterparts, these names seemed to me to be suitable to represent a cluster function. Consequently, the second largest moon of Pluto, Nix forms the third host in the three-cluster-node array.

2  Installation

2.1  Software

The DRBD software is provided for the Server or Enterprise editions starting with the following Linux distributions and is updated accordingly (as of summer 2020):

Red Hat Enterprise Linux (RHEL), versions 6, 7 and 8

SUSE Linux Enterprise Server (SLES), versions 11 SP4, 12 and 15

Debian GNU/Linux, 8 (jessie), and 9 (stretch)

Ubuntu Server Edition LTS 14.04 (Trusty Tahr), LTS 16.04 (Xenial Xerus), and LTS 18.04 (Bionic Beaver)

In addition, OpenSuSE provides the DRBD packages from version Leap 42.1.
When using the command zypper, it looks like this (the output lines habe been shortened because the type is package in all cases):

Listing 2.1:
zypper search drbd

pluto:~ # zypper search drbd 
Loading repository data... 
Reading installed packages... 

S | Name                      | Summary                                                    | 
--+---------------------------+------------------------------------------------------------+- 
  | drbd                      | Linux driver for the "Distributed␣Replicated␣Block␣Device" | 
  | drbd-formula              | DRBD deployment salt formula                               | 
  | drbd-kmp-default          | Kernel driver                                              | 
  | drbd-kmp-preempt          | Kernel driver                                              | 
  | drbd-utils                | Distributed Replicated Block Device                        | 
  | drbdmanage                | DRBD distributed resource management utility               | 
  | yast2-drbd                | YaST2 - DRBD Configuration                                 | 
pluto:~ #

2.2  Requirements

"The system have to run!“

Admittedly, this sentence sounds rather stupid with regard to the minimum requirements of an installation - especially in a technical book. However, the fact is that the DRBD software does not have any special minimum requirements for the equipment of the cluster nodes, which is due to the fact that the DRBD function is integrated into the Linux kernel.
Equip your cluster nodes to meet your requirements and ensure that the high availability application runs properly on the deployed platform. With regard to synchronization, there are a few more notes.
To create this book, I installed two virtual machines on a laptop that had the "fabulous" memory size of 4 GB and a quad processor running at 2.16 GHz.
This might be enough for a workstation, laptop or desktop, but for a server or host this computer is a bit tight.
The two “VMs” on this laptop each have 1 CPU and 1 GB RAM.
In our case, the LAN connection is established by a single LAN adapter with a speed of 10 Mb/s - sufficient for home use, for a server …well ….
As I said, for a server environment that has a little more work to do than present "It works" via apache, this hardware configuration would be considered lean. But to show that it basically works, this configuration is still sufficient.
Depending on the size of the disk partitions you want to include in this RAID 1, you should consider setting up a separate LAN for disk synchronization (backend LAN). However, you should be careful about the speed of synchronization, otherwise your computers will be busy with disk synchronization only. But more about that later.
I also don’t want to write a novel about the minimum configuration of hosts here, others have done that before me. I also realize that some users consider 10 Mb/s to be clearly too slow for their home network.
I want to show that DRBD works even with absolutely minimal equipment. Which brings us back to cost savings, especially for small businesses.

3  Preliminary considerations

Before we take a closer look at the basic configuration of a two-node cluster, there are some basic considerations. If you have already made your selection or have special requirements, you can safely skip this chapter - but you do so at your own risk.
I myself am not one for reading through endless introductions, and I know colleagues who read the introductions very carefully and then didn’t know what to do when it came time to implement them.
The important thing for me in the preliminary considerations is to avoid unnecessary work, so that you don’t have any downtime at the end of a test run or even in a running, productive cluster.
And nothing is more deadly to a cluster than not being available.
That’s why the old do-it-yourself motto applies here, too:

Measure first, then cut!

3.1  Disk drive – physically vs. LVM

So, let’s first take a look at how the disk drive should be "designed".
Let’s first take a "physical disk device", i.e. an additional disk partition, in addition to the "classical" partitions like swap, root (/) and /home.
This solution has the advantage that there is no additional "virtualization layer" holding up processing operations, which could lead to performance degradation in case of doubt.
The disadvantage is that a subsequent increase or decrease in size, can only be carried out with increased effort if hardware actually has to be replaced.
Using Logical Volume Manager, or LVM for short, gains more headroom, but adds a virtualization layer on very tight systems, which can lead to the aforementioned performance degradation.
Both types of disks work with DRBD!
In the systems I have set up, I generally use Logical Volume Manager because the advantage of adding disks after the fact outweighs the disadvantage of performance degradation.

3.2  Filesystem on the disk device

In principle, a DRBD could also be used as a RAW device. Whether and which file system "runs" on the DRBD device does not really matter. Nevertheless, I would like to take a closer look at the different working methods of the file systems used to help you decide. All file systems have their specific advantages and disadvantages based on the way they work. For perhaps understandable reasons, I won’t go into more detail about tree structures or the like at this point. If you are interested in these specific points, you should consult the relevant technical literature or www.wikipedia.com.

3.2.1  UFS / ext2

The good old ’UNIX File System’ - because that’s what UFS stands for - was developed in the early 1980s and was the standard file system for all UNIX derivatives until the early 1990s. Today, however, it is only used in isolated cases.
However, the basic concept was passed on to the following file system generations:

all data is stored in blocks on the hard drive and

to get to a data block, the address of the memory block is stored in an area called "superblock", which is accessed first by the operating system.

In this way a tree structure is obtained, because each stored file is assigned a specific "inode number".
If a search is made for a specific file within the file system, the entire file tree must always be searched, which can take a comparatively long time for larger file trees with many substructures.
The "second extended file system" (ext2) essentially adopts this structure, but so-called "plugins" - i.e. extensions - can be added to handle fragmentation, compression and recovery of deleted data.

3.2.2  ext3 / ext4

The ext3 and ext4 file systems have evolved from the ext2 with the addition of a so-called journal and the ability to change the size of the file system while the file system is in use.
In a journaling file system, all changes are recorded in a special memory area called journal before the actual write to the selected block takes place. This makes it easier to reconstruct the writes if, for example, the system crashes or the power goes out during the write operation.
Another point of improvement of ext3 resp. ext4 over ext2 was the increase in file system partitions from 16 TB to 32 TB for ext3 and 1 EB (= exabyte) for ext4. Such device sizes could not have been imagined when the UFS was developed.
In addition, there are the extensions regarding the number of files and directories as well as the size of the individual files, which was still limited to 2 TB for ext2, could be between 16 GB and 2 TB for ext3, and is finally only limited by the size of the disk partition for ext4.

3.2.3  xfs

The file system xfs, originally developed by Silicon Graphics (SGI) exclusively for the in-house UNIX system "IRIX", is one of the oldest file systems. But just because something is getting on in years doesn’t mean it has to be "bad". It sets standards with maximum values of 16 EB per file system, a maximum number of 263 files and a size per file of 8 EB.
It also has significant advantages over ext3 and BtrFS, especially in terms of speed.
Some time ago, I had a case where about 100 GB needed to be copied from one host to another. The source filesystem was a BtrFS and the copy ran - to save LAN resources - over a TAR that was compressed on the source machine, pushed through an SSH tunnel and decompressed again on the target machine.
This work took a little over an hour - probably because the file system had many subdirectories.
After the work on the source file system was finished, among other things it was enlarged to 200 GB, I spontaneously decided to use xfs as the new file system.
The recovery time was 20 minutes!
Needless to say, I have been a self-confessed fan of this file system since that time, especially because the throughput has been confirmed in normal operation.

3.2.4  BtrFS

The BtrFS - spelled out B-Tree file system and not "Better FS" or even "Butter FS" - follows a completely different approach than the file systems available in the Linux environment so far. It is based partly on the considerations of the ZFS, which was developed about seven years before also by Sun Microsystems (in the meantime merged into ORACLE). It has built-in RAID, volume management, checksum-based protection against data transfer errors, and uses copy-on-write. copy-on-write is a method where a copy is not real until it is changed by one of the parties. As long as all parties involved have not changed their copy, it is sufficient to save the original once - in the respective file system. The integrated RAID system distinguishes between occupied and free data blocks, so when reconstructing a failed RAID volume, only the occupied disk space needs to be mirrored, which saves an enormous amount of time. In addition, this RAID works with larger data blocks than is the case with classic RAID methods. In a RAID1, there is no mirroring of all data blocks of a data carrier - regardless of whether they are occupied or not - but only the occupied blocks are distributed to all available data carriers. In this way, a RAID1 can be formed from an odd number of disks with different capacities without losing storage space. The B-tree structure - after which the file system is named - comes from the central concept of xfs. BtrFS is now used by SuSE as the file system of the future, while RedHat announced in August 2017 that it would discontinue long-term support for BtrFS in RHEL. Whereas so far it has remained with this announcement and for example fedora¹ uses BtrFS quite automatically unless something else is chosen during installation.
In addition to the experience described above, the following considerations should be made with regard to using BtrFS on production servers in conjunction with DRBD:

Most Linux servers purchased as 19-inch devices are equipped with a hardware RAID controller. This means that the RAID functionality of BtrFS is not needed here, since the hard disks connected to the RAID array have the same capacity anyway, otherwise the hardware RAID controller will not work properly or the storage space cannot be used.

The copy-on-write functionality explained above adds an additional virtualization layer that is already done by distributing the data to multiple cluster nodes with DRBD. However, a BtrFS array across multiple cluster nodes is not supported.

In certain cases, using BtrFS on a DRBD device makes perfect sense if you don’t want to miss out on the multiple functionalities of BtrFS.

An example is the possibility to create a snapshot of the file system. You should, however, be careful before automatically creating such snapshots, as this quickly consumes disk space that you might need for other purposes.

3.2.5  OCFS2

The Oracle Cluster File System 2 is a file system that enables simultaneous access from several cluster nodes to a disk device in a cluster array (concurrent access), developed by Oracle for open source clusters. The coordination works via the Distributed Lock Manager.
The required packages are included as of SLES 11 SP3.

3.2.6  Conclusion

You should also think very carefully about which file system you want to use during the basic considerations for setting up a cluster. If you have made the wrong decision, it is only possible to change the file system with increased effort. In this book I show a "recipe" (cf. text number 8 - without increasing the LVM volume) how you can help yourself here. However, you must be aware that this can only be achieved with a downtime of the cluster array.
If you are comparing the file systems shown here and do not need the special features that OCFS2 or BtrFS bring, xfs is the method of choice.

4  Configuration-Basics on a two-node-cluster-array

As a basis for all further steps, let’s first consider the configuration for an active-passive cluster consisting of two cluster nodes. Apache is used as the cluster service.
A classic example!

4.1  SSH-Configuration

Before we can take care of the actual configuration of the DRBD, we have to make sure that the existing cluster nodes can "chat" with each other undisturbed, which also makes our work a little easier.

4.1.1  SSH-Key-Types

It should be common knowledge that the capabilities of the remote shell (rsh) can be covered by the much more secure shell (ssh). The Secure Shell works by exchanging "keys" that are used like a fingerprint of the particular server and the particular user. If the key matches, the requests of the respective "foreign" server are allowed, if the key doesn’t match, they are rejected. The encryption algorithms RSA, DSA, ECDSA and Ed25519 can be used here, which one you prefer is up to you.
For using an encryption algorithm other than RSA, there is a little something to keep in mind with SLE/OpenSuSE 15, which I will go into in more detail in note 13.

4.1.2  Recipe: Generate and distribute SSH keys

In order for our cluster nodes pluto and charon to be able to "talk" to each other later, it is necessary to provide passwordless communication at root level.
So let’s first create an SSH RSA key on cluster node 1 (pluto) with the identifier root:

Listing 4.1:
generate a SSH key

pluto:~ # ssh-keygen -t rsa 
Generating public/private rsa key pair. 
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa. 
Your public key has been saved in /root/.ssh/id_rsa.pub. 
The key fingerprint is: 
SHA256:******************************************* root@pluto 
The key’s randomart image is: 
+---[RSA 2048]----+ 
~ snip ~ 
+----[SHA256]-----+ 
pluto:~ #

The input for the passphrase is logically omitted because the two servers are supposed to communicate without a password.
As you will understand, I am not publishing the key of my root-id here, so the corresponding areas in the screen output are slightly different. Your screen output should look similar, but not exactly the same.
For control:

Listing 4.2:
generate a SSH key - control

pluto:~ # cd /root/.ssh 
pluto:~/.ssh # ll 
total 8 
-rw------- 1 root root 1373 Dec 31 19:25 id_rsa 
-rw-r--r-- 1 root root  600 Dec 31 19:25 id_rsa.pub 
pluto:~/.ssh #

The bottom line is that we used the ssh-keygen -t rsa command to create two files that contain the non-public key (id_rsa) and the public key (id_rsa.pub) once. The option "’-t"’ represents the key type, i.e. "’rsa"’, "’dsa"’, "’ecdsa"’ or "’ed25519"’.
We now need to make this public key known to our second cluster node.
Classical:

Listing 4.3:
Copying SSH-Key and put into the authorized_keys file

pluto:~/.ssh # scp id_rsa.pub charon:/root/.ssh/pluto_root.pub 
The authenticity of host ’charon (192.168.178.201)’ can’t be established. 
ECDSA key fingerprint is SHA256:*******************************************. 
Are you sure you want to continue connecting (yes/no)? yes 
Warning: Permanently added ’charon,192.168.178.201’ (RSA) to the list of known hosts. 
Password: 
pluto:~/.ssh # scp id_rsa.pub charon:/root/.ssh/pluto_root.pub 
Password: 
id_rsa.pub                                    100%  600   350.5KB/s   00:00 
pluto:~/.ssh # ssh charon ’cat /root/.ssh/pluto_root.pub >> /root/.ssh/authorized_keys’ 
Password: 
pluto:~/.ssh # ssh charon ’uname -n’ 
charon

…or with command:

Listing 4.4:
Copying SSH-Key - alternate

pluto:~/.ssh # ssh-copy-id -i ./id_rsa.pub root@charon 
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "./id_rsa.pub" 
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed 
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -{}- if you are prompted now it is to install the new keys 
Password: 
Number of key(s) added: 1 
Now try logging into the machine, with:   "ssh␣’root@charon’" 
and check to make sure that only the key(s) you wanted were added. 
pluto:~/.ssh # ssh root@charon "uname -n" 
charon

Of course, in order for us to have a real dialog between cluster nodes, these steps must be performed on all cluster nodes, for all cluster nodes.
So the same procedure again from the other side:

Listing 4.5:
Exchange SSH keys

charon:~/.ssh # ssh-keygen -t rsa 
Generating public/private rsa key pair. 
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa. 
Your public key has been saved in /root/.ssh/id_rsa.pub. 
The key fingerprint is: 
SHA256:******************************************* root@pluto 
The key’s randomart image is: 
+---[RSA 2048]----+ 
~ snip ~ 
+----[SHA256]-----+

…or …

Listing 4.6:
Exchange SSH-Keys - alternate

charon:~/.ssh # ssh-copy-id -i ./id_rsa.pub root@pluto 
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ./id_rsa.pub 
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed 
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys 
Password: 
Number of key(s) added: 1 
Now try logging into the machine, with:   ssh ’root@pluto’ 
and check to make sure that only the key(s) you wanted were added. 
charon:~/.ssh # ssh root@pluto ’uname -n’ 
pluto 
charon:~/.ssh #

4.2  Configuration of the locale Disk devices

Let’s look at the current disk partitioning of the pluto server, for simplicity on SuSE via YaST.
. To get the overview from figure 4.1, we type yast2 in the shell and select the partitions section in the system folder.

Figure 4.1:
YaST2 on server pluto - basical overview

Figure 4.2:
YaST2 on server pluto - partition overview

As shown in Figure 4.2, a volume group named vg00 has already been created during the installation and several partitions have been created under it.
Since there are still 22 GB available in this volume group, we create our DRBD partition here - first via YaST.
I have shown how to basically create a volume group in the attachment 14.

4.2.1  Recipe: Create LVM volume via YaST

Because we are already in YaST, we can simply continue at this point and edit this partition (cf. 4.3).

Figure 4.3:
YaST2 at server pluto - create and name logical volume

We want to mirror the Apache web server data from one cluster node to its partner, so we name the logical volume: apache (cf. 4.3).

Figure 4.4:
YaST2 at server pluto - create logical volume and define size

For the disk size in our example, 5 GB should be sufficient (cf. 4.4).

Figure 4.5:
YaST2 at server pluto - create logical volume - define partitiontype

This menu (4.5) is to specify how the created partition will be used in the future. Since we need to set up the DRBD device first and then install a file system on it, the partition remains in its raw state for now (raw-volume 4.6).

Figure 4.6:
YaST2 at server pluto - create logical volume - not formating / not mounting

Accordingly, there is no mount.

Figure 4.7:
YaST2 at server pluto - create logical volume - final

To save the work, the menu item continue must now be selected (4.7).
System-wide, this partition can now be accessed under "/dev/vg00/apache", which is discussed in more detail in the DRBD configuration.

4.2.2  Recipe: Create a LVM-Volume via shell

With the command lvm the shell user has all the options that he also has via YaST.
Using "lvm vgdisplay" you get an overview of the created volume groups:

Listing 4.7:
vgdisplay

charon:~ # lvm vgdisplay 
        --- Volume group ----- 
  VG Name            vg00 
  System ID 
  Format            lvm2 
  Metadata Areas      1 
  Metadata Sequence No5 
  VG Access          read/write 
  VG Status          resizable 
  MAX LV            0 
  Cur LV            4 
  Open LV            4 
  Max PV            0 
  Cur PV            1 
  Act PV            1 
  VG Size            72.99 GiB 
  PE Size            4.00 MiB 
  Total PE          18685 
  Alloc PE / Size      12800 / 50.00 GiB 
  Free PE / Size      5885 / 22.99 GiB 
  VG UUID            eQ9tnQ-7hWv-7DPv-LzFO-PGMc-fn6W-MW8Yfb 
charon:~ #

As on the partner system pluto, a volume group vg00 with 4 volumes was already created during installation, leaving 22.99 GB of space.
In this space, we now create a logical volume named apache with the size of 5 GB.
To learn how to create a volume group using the shell, see the appendix 14.

Listing 4.8:
lvcreate

charon:~ # lvcreate --name apache --size 5GB vg00 
Logical volume "apache" created.

To get an overview here there is also a display command:

Listing 4.9:
lvdisplay

charon:~ # lvm lvdisplay 
        --- Logical volume --- 
  LV Path                  /dev/vg00/apache 
  LV Name                  apache 
  VG Name                  vg00 
  LV UUID                  YGtshV-MtDK-hpTP-mdu5-z84g-cRev-WRqxAV 
  LV Write Access            read/write 
  LV Creation host, time      charon, 2020-01-05 21:25:35 +0100 
  LV Status                available 
  # open                  0 
  LV Size                  5.00 GiB 
  Current LE              1280 
  Segments                1 
  Allocation              inherit 
  Read ahead sectors        auto 
  - currently set to        1024 
  Block device            254:4

The important value for us in the further course is the LV Path /dev/vg00/apache, which can be used to address the drive in the system as a device.
Because the command lvm lvdisplay prints all logical volumes of a volume group, I took the liberty to reduce this output to the level that prints the data we need.

4.3  DRBD configuration

DRBD has a central file directory where the configuration files are stored. These configuration files must be identical on all cluster nodes involved.
It makes sense to edit the configuration files on one node and then distribute them to the rest of the cluster nodes.
"The rest of the cluster nodes?" I hear you ask.
Yes - of course, because DRBD is not limited to a two-node cluster. But more on that later! (See text number ??!)

4.3.1  Performance

Before I go into more detail about the individual configuration parameters at this point, I would like to say a few words about server performance:4.3.5:
The text number 4.3.5 describes the different synchronization variants (protocols).
There are ways to influence the "performance" of this synchronization, but I strongly advise against it!
The synchronization performance is - from scratch - tuned to your system. If you interfere at this point, it can happen that you slow down your system and the server is then only busy with the synchronization!

4.3.2  /etc/drbd.conf

The central configuration file is the /etc/drbd.conf file. Here, the keyword include links the actual configuration files to the DRBD service.
The file should look something like the following:

Listing 4.10:
/etc/drbd.conf

pluto:~ # cat /etc/drbd.conf 
# You can find an example in  /usr/share/doc/drbd/drbd.conf.example 
include "drbd.d/global_common.conf"; 
include "drbd.d/*.res";

The examplefile is to find here

/usr/share/doc/packages/drbd-utils/drbd.conf.example.

The include "drbd.d/*.res" entry automatically includes all files that are in the /etc/drbd.d/ subdirectory and have the filename extension .res.
This configuration can be beneficial as it simplifies the work.
However, if you are still in the experimentation phase and want to use only certain configuration files, for example, it is advisable to comment out this line and enter only the files that should be loaded.

4.3.3  /etc/drbd.d/global_common.conf

The file /etc/drbd.d/global_common.conf sets common values that apply to all DRBD resources.
Because it would be too in-depth to go into the individual points of this file at the moment, we will just assume at this point that it exists and proceed as follows:

"Never change an running system!"

4.3.4  Resourceconfiguration

First, let’s think about what we want our resource to be called. Because we want to cluster an Apache website here, let’s just name our resource "apache" and create the following file:

/etc/drbd.d/apache.res:

Listing 4.11:
/etc/drbd.d/apache.res

pluto:/etc/drbd.d # cat apache.res 
resource apache { 
        protocol      C; 
        startup { 
                wfc-timeout          15; 
                degr-wfc-timeout      60; 
                become-primary-on      pluto; 
        } 
        disk { 
                on-io-error    pass_on; 
        } 
        on pluto { 
                meta-disk      internal; 
                disk        /dev/vg00/apache; 
                device      /dev/drbd0; 
                address      192.168.178.200:7751; 
        } 
        on charon { 
                meta-disk      internal; 
                disk        /dev/vg00/apache; 
                device      /dev/drbd0; 
                address      192.168.178.201:7752; 
        } 
}

Because the individual items have to be described in more detail, the description follows on the following pages. (Note 4.3.5)

4.3.5  Resource Description:

Keyword

Description

resource

This expression is used to detect or address the drbd resource in the respective commands.

The name should be "speaking" as well.

protocol

There are three "protocol types": A, B and C

"Protocol A"

Protocol A stands for "Asynchronous replication protocol", i.e. a write operation is completed on the primary system when the physical write operation on the local disk device is successfully completed, and the replication data packet is created on the local TCP send buffer on this node. The packet is just not sent!

→ In the event of a failure, this can lead to data inconsistencies because not all data may have arrived on the secondary server yet! On the other hand, the write process is "completed" very quickly and the device can execute the next command.

"Protocol B"

describes the so-called "memory synchronous (semi-synchronous) replication protocol". The local write operation on the primary server is considered complete when the physical write operation on the local disk has finished and the replication data packet has reached the secondary node.

→ This variant is slightly slower than the A variant, but there is usually no data loss in case of failover.

"Protocol C"

stands for ’Synchronous replication protocol’. Here, security comes first! Local writes on the primary node are considered complete when the physical write on the local device and on the secondary node device, has been successfully completed.

→ Latency is the slowest compared to "’A"’ and "’B"’, but data loss is completely eliminated.

device

specifies the name by which the DRBD drive is designated in the system – for example, for mounting.

disk

defines the name of the raw device – look at "create volumes"

meta-disk-internal

Specifies that the metadata must be present in the DRBD device.

wfc-timeout / degr-wfc-timeout

The wait-for-connection timeout is the time, in seconds, that is waited for the other side when the system boots, which means that the DRBD boot script is paused for so long. The value is set to 15 seconds by default.

The degraded-wait-for-connection timeout describes the amount of time a cluster node marked as degraded will be waited for. The cluster node is marked as degraded when the other side can no longer be reached. When the specified time has elapsed, the other site is marked as unreachable.

become-primary-on

On which node should the device be primary started?

on-io-error pass-on / detach

In case of an input/output error, further action is left to the file system (pass-on) or the disk device is disconnected (detach). Up to and including version 8.3 pass-on was the default, as of 8.4 detach has become the default.

on $nodename

The values set in this section refer to the cluster node $nodename - in our examples pluto, charon or nix.

meta-disk

The default value here is internal because the create-md stores the management data on the DRBD device.

disk

Name of the "physical device"’.

device

Defines the device name of the DRBD device in the running system, usually "/dev/drbd" followed by a decimal number, to show that DRBD is used.

address

Defines the IP-Address and the port needed for access.

Table 4.1: Resource Description

4.3.6  Alternative notation

The following is an alternative notation of the configuration file.
The advantage of this notation is that all disk-specific configurations are defined only once. Furthermore, this notation is much more effective for a triple cluster than the notation shown above.

Listing 4.12:
/etc/drbd.d/apache_alternate

pluto:/etc/drbd.d # cat apache_alternate.res 
resource apache_alternate { 
        protocol      C; 
        device      /dev/drbd1; 
        disk        /dev/vg00/apache_alternate; 
        meta-disk      internal; 
        startup { 
                wfc-timeout          15; 
                degr-wfc-timeout      60; 
                become-primary-on      charon; 
        } 
        disk { 
                on-io-error    pass_on; 
        } 
        on pluto { 
                address      192.168.178.200:7758; 
                node-id      0; 
        } 
        on charon { 
                address      192.168.178.201:7768; 
                node-id      1; 
        } 
        connection-mesh { 
                hosts        pluto charon ; 
                net { 
                        use-rleno; 
                } 
        } 
}

Extension in the alternative solution are very simple, as the "on", "connection-mesh" and "hosts" sections are simply extended to include another server.
Moreover, the constant information is mentioned only once, while in the older solution the same entries have to be made multiple times, which is error-prone.
Especially for larger cluster arrays, the alternative solution is the better way to work.
Because we are running both DRBD devices on our test environment, in the alternative solution, we run the charon node as the primary node and address the device in the system as /dev/drbd1.

4.3.7  Ports

Basically, the ports used are free to choose - which of course means that no ports may be used that are already occupied by other services. In numbers this means that the selected ports must not be smaller than 1024 and not larger than 65536, because the lower ports are already occupied by other services and DRBD communication does not work with higher port numbers.
However, you should always use a separate port for each DRBD-RAID combination. This rule was already in effect in version 8, so it is easier to track errors.
When combining SLES 15 SP1 with DRBD version 9 the following error message is issued if you do not follow this rule:

Listing 4.13:
Error message - double ports

pluto:/etc/drbd.d # drb

¹The "open variant" of RHEL.

¹Quote from Gil Grissom, CSI: Las Vegas

¹The tool to check a remote host for open ports is nmap. The output is clear and unambiguous. nmap is also available for Windows with the GUI Zenmap (https://nmap.org/zenmap).

¹In other words: "Excuse me?"

¹german / franconian for "eavesdropper" or "listener"