
CryptoServer Device Driver for Linux
====================================

Content:
(1) Host Driver for Linux kernel 
  - Prerequisites
  - Compilation
  - Installation
  - Test
  - Module Parameters

(2) Virtualization 
  - Enable Virtual Functions
  - Non-SRIOV Mode
  - Hybrid Mode
  
(3) Network Interface (Ethernet over PCIe)
  - Device
  - Host


(1) Host Driver for Linux kernel 
================================

Prerequisites
-------------

To compile the driver for the HSM device:
- the Linux headers for your active kernel must have been installed.
  Nowadays, every major distribution has a package for the headers.
  E.g. Debian 64-bit has a meta-package named "linux-headers-amd64". It is called "kernel-devel".
- root privileges on the machine are required.
- gcc is required

Kernel Modul Signing
--------------------

If required prepare your system for automatic kernel module signing (see: https://www.kernel.org/doc/html/v5.19/admin-guide/module-signing.html):

1. Create file: 'x509.genkey' with content:
   
   [ req ]
   default_bits = 4096
   distinguished_name = req_distinguished_name
   prompt = no
   string_mask = utf8only
   x509_extensions = myexts

   [ req_distinguished_name ]
   CN = Modules

   [ myexts ]
   basicConstraints=critical,CA:FALSE
   keyUsage=digitalSignature
   subjectKeyIdentifier=hash
   authorityKeyIdentifier=keyid

2. Generate key:
   openssl req -new -nodes -utf8 -sha512 -days 36500 -batch -x509 -config x509.genkey -outform DER -out signing_key.x509 -keyout signing_key.pem
   
3. Copy key and certificate to '/usr/src/linux-headers-<version>/certs', e.g.:
   cp signing_key.pem /usr/src/linux-headers-5.4.0-104-generic/certs
   cp signing_key.x509 /usr/src/linux-headers-5.4.0-104-generic/certs


Compilation
-----------

To compile the HSM Device driver:
1. Copy the HSM Device driver folder from the product CD (/driver/CryptoServer/linux) to your home directory.
2. Open a shell and navigate to where you copied the HSM driver folder.
3. Compile the module by executing 'make'. To build the module for a different kernel than the currently
   running one, execute 'make KERNELDIR=<kernel-source>' (KERNELDIR should point to a kernel source folder).
   When omitted, KERNELDIR="/lib/modules/$(shell uname -r)/build" is used.
   A simple 'make' should be sufficient if the module is being built natively on the machine on which it will
   be used.

Installation
------------

DEBIAN:
1. Install module by executing '[sudo] make install' (as root/super user)
2. The module will be automatically loaded if an HSM card is detected during boot. To force it to be loaded 
   create the file /etc/modules-load.d/cryptoserver.conf with the line 'cryptoserver' (debian 9 and newer).
3. Reboot the machine.

SLES:
1. Install module by executing '[sudo] make install' (as root/super user)
2. Change the value of 'allow_unsupported_modules' to 1 in /etc/modprobe.d/10-unsupported-modules.conf.
3. The module will be automatically loaded if an HSM card is detected during boot. To force it to be loaded 
   create the file /etc/modules-load.d/cryptoserver.conf with the line 'cryptoserver' (SLES 12 and newer).
4. Reboot the machine.

RHEL:
1. Install module by executing '[sudo] make install' (as root/super user)
2. The module will be automatically loaded if an HSM card is detected during boot. To force it to be loaded 
   create the file /etc/modules-load.d/cryptoserver.conf with the line 'cryptoserver' (RHEL 7 and newer).
3. Reboot the machine.

Test
----

1. Check that the kernel module is running: 
   > lsmod | grep cryptoserver
2. Check that the device node has been created: 
   > ls /dev/cs2.0
3. In case of an error check the kernel ring buffer:
   > [sudo] dmesg | grep :cs

Module Parameters
-----------------

LogLevel        log level for kernel messages (global):
                0:none, 1:error, 2:warning, 3:info [default], 4:trace, 5:xtrace, 6:debug
                
ErrorReporting  PCIe AER, bit field (global): 
                1:device, 2:bridge [default:0]
                
MsiMode         override automatic detection (CSE,Se2):
                0:legacy, 1:MSI/MSI-X interrupts
                
ShutdownMode    type of shutdown (CSe,Se2):
                0: node, 1:suspend (immediate shutdown), 2:halt (graceful shutdown, stop fw modules)

DeviceMask      channels (functions) to be enabled in non-SRIOV mode (CSAR model >= 8):
                bit mask [default: 0x00000000]

DeviceFlags     feature selector (CSAR model >= 8):
                bit field: 1:calc&check additional CRC, 2:activate virtual network interface



(2) Virtualization
==================

The CryptoServer supports 1 physical and up to 28 virtual functions that are associated with the basic physical function.
Each function, physical or virtual, is associated with a unique channel pair and a cHSM on the device.
Basically virtual functions can be used in the same way as physical functions. 
The type of function can be distinguished by the device ID:

    - physcial: C071
    - virtual:  C072

Note:
    - the PCIe slot has to support SR-IOV to basically access virtual function and additionally ARI to access more than 7 virtual functions.
    - the function (channel) numbers 1...3 are reserved for further use and are not available. 
    
    
Enable Virtual Functions
------------------------

After the host driver has been loaded only the physical function is available:
> lspci -d 168a:
01:00.0 Network and computing encryption device: Device 168a:c071

In order to enable the virtual functions the user has to issue the following commands:
> echo 28 > /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs

Please adapt the number of virtual devices to be enabled and the 'slot:bus:device.function' numbers to your needs.

Afterwards all functions will be listed as PCI device:
> lspci -d 168a:
01:00.0 Network and computing encryption device: Device 168a:c071
01:03.4 Network and computing encryption device: Device 168a:c072
01:03.5 Network and computing encryption device: Device 168a:c072
01:03.6 Network and computing encryption device: Device 168a:c072
01:03.7 Network and computing encryption device: Device 168a:c072
...

Please note that the device IDs of virtual functions ('c072') differ from physical functions ('c071').

The driver automatically starts (probes) the virtual functions. 
An overview of all devices can be output with the following command:
> cat  /proc/driver/cryptoserver
DriverVersion    5.17.3
LogLevel         [3] Info
  
 card slot           model device  alias      state    type
----------------------------------------------------------------------------
 0    0000:01:00.0   8     cs2.0   cs2.0.0    enabled  physfn (28/28 vf used)
 0    0000:01:00.4   8     cs2.1   cs2.0.4    enabled  virtfn -> cs2.0
 0    0000:01:00.5   8     cs2.2   cs2.0.5    enabled  virtfn -> cs2.0
 0    0000:01:00.6   8     cs2.3   cs2.0.6    enabled  virtfn -> cs2.0
 0    0000:01:00.7   8     cs2.4   cs2.0.7    enabled  virtfn -> cs2.0
 0    0000:01:00.8   8     cs2.5   cs2.0.8    enabled  virtfn -> cs2.0
 0    0000:01:00.9   8     cs2.6   cs2.0.9    enabled  virtfn -> cs2.0 
...
 0    0000:01:00.31  8     cs2.28  cs2.0.31   enabled  virtfn -> cs2.0
 
 
Non-SRIOV Mode
--------------

If the PCIe-slot doesn't support SR-IOV or is limitted to 8 functions, the driver also offers a so called 'non-SRIOV mode'. 
Therefore the driver has to be started with the module parameter 'DeviceMask=<bit mask>. 
Every bit in the mask represents one channel (0...31) to be used. 
A channel used in non-SRIOV mode can no longer be used via SRIOV.
> modprobe cryptoserver DeviceMask=0xFFFFFFFF
 
> cat  /proc/driver/cryptoserver
DriverVersion    5.17.3
LogLevel         [3] Info
 
 card slot           model device  alias      state    type
----------------------------------------------------------------------------
 0    0000:01:00.0   8     cs2.0   cs2.0.0    enabled  physfn (0/28 vf used)
 0    0000:01:00.0   8     cs2.1   cs2.0.1    enabled  child  -> cs2.0
 0    0000:01:00.0   8     cs2.2   cs2.0.2    enabled  child  -> cs2.0
 0    0000:01:00.0   8     cs2.3   cs2.0.3    enabled  child  -> cs2.0
 0    0000:01:00.0   8     cs2.4   cs2.0.4    enabled  child  -> cs2.0
 0    0000:01:00.0   8     cs2.5   cs2.0.5    enabled  child  -> cs2.0
 0    0000:01:00.0   8     cs2.6   cs2.0.6    enabled  child  -> cs2.0
 0    0000:01:00.0   8     cs2.7   cs2.0.7    enabled  child  -> cs2.0
 0    0000:01:00.0   8     cs2.8   cs2.0.8    enabled  child  -> cs2.0
...
 0    0000:01:00.0   8     cs2.31  cs2.0.31   enabled  child  -> cs2.0
 
Hybrid Mode
-----------

Both, non-SRIOV mode and SRIOV mode can also be combined:
> modprobe cryptoserver DeviceMask=0xF
 
> echo 28 > /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs 
 
> cat  /proc/driver/cryptoserver
DriverVersion    5.17.3
LogLevel         [3] Info
  
 card slot           model device  alias      state    type
----------------------------------------------------------------------------
 0    0000:01:00.0   8     cs2.0   cs2.0.0    enabled  physfn (4/28 vf used)
 0    0000:01:00.0   8     cs2.1   cs2.0.1    enabled  child  -> cs2.0
 0    0000:01:00.0   8     cs2.2   cs2.0.2    enabled  child  -> cs2.0
 0    0000:01:00.0   8     cs2.3   cs2.0.3    enabled  child  -> cs2.0
 0    0000:01:00.4   8     cs2.4   cs2.0.4    enabled  virtfn -> cs2.0
 0    0000:01:00.5   8     cs2.5   cs2.0.5    enabled  virtfn -> cs2.0
 0    0000:01:00.6   8     cs2.6   cs2.0.6    enabled  virtfn -> cs2.0
 0    0000:01:00.7   8     cs2.7   cs2.0.7    enabled  virtfn -> cs2.0
...



(3) Network Interface (Ethernet over PCIe)
==========================================

As of CSAR, model 8, the CryptoServer supports a virtual network interface. Therefore channel 0 (physical function) is used.

Device
------

On the device it can be activated with the module parameter 'EnableNetdev=1' (default: 1). 

The network interface parameters have to be configured in '/etc/network/interfaces', e.g.:
auto csar_net0
iface csar_net0 inet static
        address 192.168.1.1
        netmask 255.255.255.0

The proper function can be checked with:
> dmesg | grep csar | tail
...
[694889.264957] :csar_mailbox_task: csar_net0 is up
[694889.269615] IPv6: ADDRCONF(NETDEV_CHANGE): csar_net0: link becomes ready
 
> cat /proc/driver/csar-pcie.0
...
Statistics
 irq_cnt         130 195 1 [0 8478218]
 netstats        rx: packets:195      bytes:20352
                 tx: packets:130      bytes:17417

Host
----

On the host it can be activated with the module parameter 'DeviceFlags=2' (default: off).

The network settings on the host must correspond to those on the device,e.g.:
auto csar_net0
iface csar_net0 inet static
        address 192.168.1.2
        netmask 255.255.255.0

The proper function can be checked with:
> dmesg | grep cryptoserver | tail
...
[592677.934612] cryptoserver: cs8_mailbox_task[0]: csar_net0 is up
[592706.532409] cryptoserver: cs8_netdev_open: csar_net0 was opened
 
> cat /proc/driver/cs2.0
...
Netstats         packets    bytes
 rx              475        118408
 tx              696        58775
 
> ssh 192.168.1.1
... 