> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Ceph Installation ## 9.1 Ceph Introduction Ceph is a storage software designed for distributed computing. The official Ceph documentation can be found at [http://docs.ceph.com/en/latest/start/intro/](http://docs.ceph.com/en/latest/start/intro/). Integration of Ceph with BCM is an option, and it is discussed in this chapter. NVIDIA Base Command Manager version 10 comes with the Quincy (Ceph 17 series) release of Ceph. Ceph Quincy is properly supported for the RHEL8 and Rocky Linux 8 distributions. The current chapter discusses * The concepts and required hardware for Ceph (section [9.1](#91-ceph-introduction)) * Ceph installation with `cm-ceph-setup` (section [9.2](#92-ceph-installation-with-cm-ceph-setup)) * Ceph installation with Base View (section [9.3](#93-installation-of-ceph-from-base-view)) * Ceph settings checks, and management with BCM (section [9.4](#94-checking-and-getting-familiar-with-ceph-items-after-cm-ceph-setup)) ### 9.1.1 Ceph Object And Block Storage Ceph is a distributed storage software. It is based on an object store layer called RADOS (Reliable Autonomic Distributed Object Store), which consists of Ceph components called OSDs (Object Storage Daemons) and MONs (Monitoring Servers). These components feature heavily in Ceph. OSDs deal with storing the objects to a device, while MONs deal with mapping the cluster. OSDs and MONs, together carry out object storage and block storage within the object store layer. The Ceph Manager daemon (MGR) runs alongside monitor daemons, to provide additional monitoring and interfaces to external monitoring and management systems. The stack diagram of figure [9.1](#fig-ceph-concepts) illustrates these concepts. ![Ceph Concepts](/base-command-manager/product-manuals/_fern-img/2cf4ced5e14c1518793ddcfae5580c098b0b7bf589601c76b557bd31e70ee02b.webp) *Figure 9.1: Ceph Concepts* On top of the object store layer are 3 kinds of access layers: 1. **Block device access:** RADOS Block Device (RBD) access can be carried out in two slightly different ways: 1. via a Linux kernel module based interface to RADOS. The module presents itself as a block device to the machine running that kernel. The machine can then use the RADOS storage, that is typically provided elsewhere. 2. via the `librbd` library, used by virtual machines based on `qemu` or `KVM`. A block device that uses the library on the virtual machine then accesses the RADOS storage, which is typically located elsewhere. 2. **Gateway API access:** RADOS Gateway (RADOS GW) access provides an HTTP REST gateway to RADOS. Applications can talk to RADOS GW to access object storage in a high level manner, instead of talking to RADOS directly at a lower level. The RADOS GW API is compatible with the APIs of Swift and Amazon S3. 3. **Ceph Filesystem access:** CephFS provides a filesystem access layer. A component called MDS (Metadata Server) is used to manage the filesystem with RADOS. MDS is used in addition to the OSD and MON components used by the block and object storage forms when CephFS talks to RADOS. ### 9.1.2 Ceph Storage Backends OSDs have a choice of two storage backends for managing their data. These are BlueStore and FileStore. #### 9.1.2.1 BlueStore BlueStore is a special-purpose storage backend designed specifically for managing data on disk for Ceph OSD workloads. It is the default, and recommended, backend for Ceph from version 13.2.*x* onward. BlueStore consumes raw block devices or partitions. In contrast to the legacy FileStore approach, BlueStore avoids any intervening layers of abstraction that may limit performance or add complexity. BlueStore does however, by its design, in contrast to FileStore, require at least an extra volume on the node the OSD runs on. #### 9.1.2.2 FileStore FileStore is the legacy approach to storing objects in Ceph. It relies on a standard file system, which is normally XFS. FileStore is well-tested and widely used in production. However it does suffer from many performance deficiencies due to its overall design and reliance on a traditional file system for storing object data. Though it is technically possible to store Ceph data alongside other data when using FileStore, it is preferred that dedicated block devices (disks) are used. BCM supports both storage backends. In the following sections the storage backends are described in some more detail. ### 9.1.3 Ceph Software Considerations Before Use #### 9.1.3.1 Recommended Filesystem For Legacy FileStore BlueStore is the recommended storage backend for Ceph. BlueStore requires dedicated block devices (disks) that are fully managed by `ceph-volume`. The legacy FileStore backend, on the other hand, stores the data directly on a regular file system. If using FileStore, then recommended file system is XFS, due to its stability, ability to handle extreme storage sizes, and its intrinsic ability to deal with the significant sizes of the extended attributes required by Ceph. The nodes that run OSDs are typically regular nodes. Within the nodes, the storage devices used by FileStore OSDs automatically have their filesystems configured to be of the XFS type during the installation of Ceph with BCM. #### 9.1.3.2 Use Of `datanode` For The Protection Of Ceph Data OSD nodes store the actual data contents of the Ceph cluster. Ceph Monitor nodes also store some data content that is essential for the operation of the Ceph cluster. The devices of these nodes that store such content need protection from being wiped during the reprovisioning that takes place during a reboot of regular nodes. The recommended way to protect storage devices from being wiped is to set the `datanode` property of their node to `yes` ([here](/admin/provisioning#5444-full-install-confirmation-via-datanode-setting)). The `datanode` property is automatically set for Monitor and OSD nodes during installation of Ceph with BCM. #### 9.1.3.3 Use Of Slurm On OSD Nodes Ceph can be quite demanding of the network and I/O. Running Slurm jobs on an OSD node is therefore not recommended. In addition, if Slurm roles are to be assigned to nodes that have OSD roles, then the default ports 6817 and 6818 used by Slurm can conflict with the default range 6800-7300 used by the Ceph OSD daemons. If there is a need to run Slurm on an OSD node then it is necessary to arrange it so that the ports used do not conflict with each other. During installation, a warning is given when this conflict is present. ### 9.1.4 Hardware For Ceph Use **An absolute minimum installation:** can be carried out on two nodes, where: * 1 node, the head node, runs one Ceph Monitor and the first OSD. * 1 node, the regular node, runs the second OSD. This is however not recommended, or even supported by BCM. Reasons for why this is not recommended are: * If the Ceph monitor crashes, and there is no other Ceph monitor running, then Ceph cannot function, and data could be lost. * The first OSD on the head node requires its own Ceph-compatible filesystem. If that filesystem is not provided, then Ceph on the cluster will run, but in a degraded state. * Running a monitor service on the same host as an OSD may impair performance due to fsync issues with the kernel. Using such a system to try to get familiar with how Ceph behaves in a production environment with BCM is unlikely to be worthwhile. **A more useful minimum:** if there is a node to spare, then it is possible to install Ceph over 3 nodes as follows: * 1 node, the head node, runs one Ceph Monitor. * 1 node, the regular node, runs the first OSD. * 1 more node, also a regular node, runs the second OSD. In this case the OSD pool default size should be set to 2 in the Global OSD Settings (figure [9.9](#fig-urwid-cephinstall-global-osd-settings)). Although useful for some testing purposes, this is again not a production system, due to the possible loss of data as well as loss of service if the single Ceph Monitor has issues. This can therefore also not be regarded as a good test cluster. **For production use:** a redundant number of Ceph Monitor servers is recommended. This is because Ceph Monitors are crucial to Ceph operations. Since the number of Ceph Monitoring servers must be odd, then at least 3 Ceph Monitor servers, with each on a separate node, are recommended for production purposes. The recommended minimum of nodes for production purposes is then 5: * 2 regular nodes running OSDs. * 2 regular nodes running Ceph Monitors. * 1 head node running a Ceph Monitor. **Drives usable by Ceph:** Ceph OSDs can use any type of disk that presents itself as a block device in Linux. This means that a variety of drives can be used. ## 9.2 Ceph Installation With `cm-ceph-setup` Ceph installation for BCM can be carried out in a TUI session using the `cm-ceph-setup` utility. The utility installs `x86_64` Ceph for RHEL 8 and Rocky 8 only, and is part of the `cm-setup` package that comes with BCM. If the Ceph packages are not already installed, then the utility is able to install them for the head and regular nodes, assuming the repositories are accessible, and that the package manager priorities are at their defaults. ### 9.2.1 Ceph Installation: The Configuration Stage The `cm-ceph-setup` utility can be run as root from the head node to open up a TUI screen (figure [9.2](#fig-urwid-cephinstall-welcome)): ![Ceph Installation Welcome](/base-command-manager/product-manuals/_fern-img/9d5f62816c0d3d9151a46bd1242217896e905135d7edd39e39431f8002368887.webp) *Figure 9.2: Ceph Installation Welcome* Here the administrator may choose to * Deploy Ceph * Uninstall Ceph if it is already installed. **Ceph public network selection:** If the deploy option is chosen, then a screen opens up that allows the selection of the Ceph network used to connect Monitor, OSD and client nodes (figure [9.3](#fig-urwid-cephinstall-general-cluster-settings)): ![Ceph Installation: Public Network Selection](/base-command-manager/product-manuals/_fern-img/b91115b15db016e8bcdb3baf06a0074b44cac90f9b412db70575c46ee3e0fd7f.webp) *Figure 9.3: Ceph Installation: Public Network Selection* For a cluster that is configured in a standard default BCM Type 1 architecture, the network that is chosen is `internalnet`. In Ceph terminology this is called the *public*, or *front-side*, network. This should not be confused with the informal terminology a BCM administrator may sometimes use for a Type 1 architecture, where `externalnet` is sometimes called the public network. Network architecture types for cluster are discussed in [section 3.3.9 of the *Installation Manual*](/installation/installing#339-network-topology). **Ceph cluster network selection:** The next screen allows the Ceph *cluster*, or *back-side*, network to be selected (figure [9.4](#fig-urwid-cephinstall-cluster-network-settings)): ![Ceph Installation: Ceph Cluster Network Selection](/base-command-manager/product-manuals/_fern-img/b7eab35025aa2e21c6ed22105f0e08cdf233f7a2b101cfe7996f02c2250f9ce0.webp) *Figure 9.4: Ceph Installation: Ceph Cluster Network Selection* The OSDs use this network to rebalance storage. In a Type 1 architecture this is also typically `internalnet`. Ceph documentation suggests just using a single public network. In the Type 1 architecture case this is achieved by using the same network, `internalnet`, for the Ceph cluster network as for the Ceph public network. **Ceph Monitor role assignment to categories:** The `OK` button in figure [9.3](#fig-urwid-cephinstall-general-cluster-settings) then brings up a screen that allows the Ceph Monitor role to be assigned to zero or more categories (figure [9.5](#fig-urwid-cephinstall-monitors-setup)). ![Ceph Installation: Monitors Assignment To Categories](/base-command-manager/product-manuals/_fern-img/fb8a0aeed7da23b94bf944099273e6acddeb3ac0decef823e9e8cd9407186786.webp) *Figure 9.5: Ceph Installation: Monitors Assignment To Categories* **Ceph Monitor role assignment to nodes:** The next screen is similar, and allows Ceph Monitor roles to be assigned to the remaining individual nodes (figure [9.6](#fig-urwid-cephinstall-monitors-edit)): ![Ceph Installation: Monitors Assignment To Nodes](/base-command-manager/product-manuals/_fern-img/0f5d41b8fa0eb5fb4df67cdf34ef60dd62ba40f5d34ddf94a1f8e43025b653fd.webp) *Figure 9.6: Ceph Installation: Monitors Assignment To Nodes* **Ceph OSD role assignment to categories:** OSD Roles can then be assigned to zero or more categories (figure [9.7](#fig-urwid-cephinstall-osds-setup)): ![Ceph Installation: OSDs Assignment To Categories](/base-command-manager/product-manuals/_fern-img/02984f0bb692734d5c1fca062f463445a5e7c1aefdaa90a578c38acfac1806cc.webp) *Figure 9.7: Ceph Installation: OSDs Assignment To Categories* **Ceph OSD role assignment to nodes:** OSD roles can then be assigned individually to the remaining nodes (figure [9.8](#fig-urwid-cephinstall-osd-node-assign)): ![Ceph Installation: OSDs Assignment To Nodes](/base-command-manager/product-manuals/_fern-img/d50bd644e475f4bdac45cf2ca17cb2c4ff10d20e233f30556e2cfd8cc954a623.webp) *Figure 9.8: Ceph Installation: OSDs Assignment To Nodes* **Global Ceph OSD settings:** After OSD role assignment is completed, the next screen displayed is the Global Ceph OSD settings screen, (figure [9.9](#fig-urwid-cephinstall-global-osd-settings)) which allows the OSD pool default size to be set. The OSD pool default size is the default number of replicas for objects in the pool. It should be less than or equal to the number of OSD nodes. If unsure the administrator can just leave it at the default value. ![Ceph Installation OSD Global Settings: OSD Pool Default Size](/base-command-manager/product-manuals/_fern-img/41151a30ba9ce95cdfcf96290f2d24258c37810904bc5037aa3cb5595a847ee9.webp) *Figure 9.9: Ceph Installation OSD Global Settings: OSD Pool Default Size* **BlueStore device settings:** The next screen is the BlueStore configuration screen, which requires that block devices be specifed for the OSDs (figure [9.10](#fig-urwid-cephinstall-select-bluestore-devices-empty)). ![Ceph Installation: Block Devices For BlueStore](/base-command-manager/product-manuals/_fern-img/2a23aad295fe965d8e0a019fc9847e6f2dc4fc4d4fc9307426221bf174d4ea91.webp) *Figure 9.10: Ceph Installation: Block Devices For BlueStore* Typically, the administrator would have prepared the nodes that will be taking care of file storage with one or more block devices for BlueStore to use. If these are not there during deployment, then deployment will fail. **Ceph removal of OSD pools:** The next screen asks if Ceph is to be allowed to remove OSD pools (figure [9.11](#fig-urwid-cephinstall-osd-pools-removal)): ![Ceph Installation: Option To Have Ceph Allowed To Remove OSD Pools](/base-command-manager/product-manuals/_fern-img/794a55e4f1ba64aa89110a888bf344ee8ddb95bbda02a755e7ff7c0bded2ea67.webp) *Figure 9.11: Ceph Installation: Option To Have Ceph Allowed To Remove OSD Pools* **Cep dashboard screen:** The Ceph dashboard settings can then be specified: ![Ceph Installation: Dashboard Configuration](/base-command-manager/product-manuals/_fern-img/fd16d3cf04f212466d1aad75c6ab82e32038cb0390159a918cd07660ffc20062.webp) *Figure 9.12: Ceph Installation: Dashboard Configuration* The Ceph dashboard is a web frontend to manage and monitor Ceph cluster resources. **Summary screen:** The summary screen (figure [9.13](#fig-urwid-cephinstall-summary-save-configuration-options)) allows the configuration to be viewed, saved, or deployed in various combinations. ![Ceph Installation: Save Configuration Options](/base-command-manager/product-manuals/_fern-img/1ce6878f89486bd0205691aafe416ab5463daa0fd46a3d31e694506063db2812.webp) *Figure 9.13: Ceph Installation: Save Configuration Options* If a save option is chosen, then by default, the configuration is saved to `/root/cm-ceph-setup.conf`. To deploy, the administrator should choose the `Save config & deploy` option: ### 9.2.2 Ceph Installation: The Deployment Stage **Deployment session output:** If deployment is carried out, then the TUI screen ends, and session output similar to the following appears: ```bash Executing 35 stages ################### Starting execution for 'Ceph Setup' - ceph ## Progress: 0 #### stage: ceph: Networks is available and allowed types. Connecting to CMDaemon ## Progress: 2 #### stage: ceph: Monitor categories and nodes is available. ## Progress: 5 #### stage: ceph: OSD categories and nodes is available. ## Progress: 8 #### stage: ceph: Default pool size is correct for selected amount of nodes ## Progress: 11 #### stage: ceph: Check and normalize device paths to '/dev/' form. ## Progress: 14 #### stage: ceph: BlueStore configurations is correct and points to devices ## Progress: 17 #### stage: ceph: FileStore configurations is technically correct. ## Progress: 20 #### stage: ceph: Collection Nodes Online ## Progress: 22 #### stage: ceph: Get Software Image Paths ## Progress: 25 #### stage: ceph: Collection Package Manager Repos Add ## Progress: 28 #### stage: ceph: Collection Package Manager Repos Enable ## Progress: 31 #### stage: ceph: Collection Packages Installer ## Progress: 37 #### stage: ceph: Mark block devices "ClearedOnNextBoot" and "restart_required" ## Progress: 42 #### stage: ceph: Collection Nodes Reboot All nodes to be rebooted: mon002, mon003, osd001, osd002, mon001 Node has been rebooted mon002 Node has been rebooted mon003 Node has been rebooted osd001 Node has been rebooted osd002 Node has been rebooted mon001 Press ctrl+c to abort waiting and continue with deployment Waiting for nodes to start reboot Going to wait up to 30 minutes for the nodes to come back up. Waiting for 5 nodes to come back up Waiting for 5 nodes to come back up Waiting for 3 nodes to come back up All 5 nodes came back up. ## Progress: 45 #### stage: ceph: Mark monitors nodes as DataNode ## Progress: 48 #### stage: ceph: Mark osd nodes as DataNode ## Progress: 54 #### stage: ceph: Create Ceph cluster object ## Progress: 57 #### stage: ceph: Bootstrap Monitors ## Progress: 60 #### stage: ceph: Assign Monitor Role Assigning CephMonitorRole role ## Progress: 62 #### stage: ceph: Wait Monitors Majority Up ## Progress: 65 #### stage: ceph: Assign Ceph Mgr Role Assigning CephMGRRole role ## Progress: 68 #### stage: ceph: Mark OSD nodes with FileStore: restart_required ## Progress: 71 #### stage: ceph: Assign Ceph OSD Role Assigning CephOSDRole role ## Progress: 74 #### stage: ceph: Wait Osd Id File Store Assigned ## Progress: 80 #### stage: ceph: Configure Prometheus module Prometheus interface set up correctly ## Progress: 82 #### stage: ceph: Load Dashboard module ## Progress: 85 #### stage: ceph: Open Shorewall Port On Headnode Opening port 8443 in Shorewall for ceph dashboard rev. proxy Restarting shorewall ## Progress: 88 #### stage: ceph: Assign Generic Role ## Progress: 91 #### stage: ceph: Make the Dashboard visible to Base View ## Progress: 94 #### stage: ceph: Check Installation Ceph monitors started Ceph-manager started All services started ## Progress: 97 #### stage: ceph: Wait for CEPH OSD is full functional (all OSDs up) ## Progress: 100 Took: 09:21 min. Progress: 100/100 ################### Finished execution for 'Ceph Setup', status: completed Ceph Setup finished! [root@myhost ~]# ``` A log of the session is kept at `/var/log/cm-ceph-setup.log` as well as other relevant logs of installation process. ## 9.3 Installation Of Ceph From Base View Ceph can be installed from BCM in the following two ways: 1. Using the text-based `cm-ceph-setup` utility (section [9.2](#92-ceph-installation-with-cm-ceph-setup)). The utility is part of the `cm-setup` package. 2. Using the Base View Ceph Wizard Installer (this section). This is the recommended installation method. The Ceph Wizard Installer method can be accessed via the web browser as follows: * using the Base View navigation path `Storage > Ceph > Ceph wizard` * using the head node landing page (figure [2.1](/admin/clustermanagement#241-installing-the-cluster-management-gui-service)). A greytoned Ceph block with a ⊕ clickable link indicates that Ceph has not yet been installed by BCM. Clicking on that link then also leads to the Ceph Wizard Installer from within Base View. ### 9.3.1 Base View Ceph Install: Main Details Screen The Base View navigation path `Storage` > `Ceph` > `Ceph Wizard` brings the browser to the `Ceph main details` screen, (figure [9.14](#fig-bright-view-ceph-setup-wizard-general-ceph-cluster-settings-screen)), if Ceph has not yet been installed by BCM. This screen is beginning of the Ceph installation process, and the page displayed asks for details of the main Ceph configuration settings: ![Ceph Wizard Installation: General Cluster Settings](/base-command-manager/product-manuals/_fern-img/3c46d3f75b7fbdf2411acc75505b6411a29ced2d349a3e5179de3eda995b48a3.webp) *Figure 9.14: Ceph Wizard Installation: General Cluster Settings* The GUI screen of figure [9.14](#fig-bright-view-ceph-setup-wizard-general-ceph-cluster-settings-screen) is a combination of the following TUI screens: * the public network selection screen (figure [9.3](#fig-urwid-cephinstall-general-cluster-settings), [here](#fig-urwid-cephinstall-general-cluster-settings)) * the cluster network selection screen (figure [9.4](#fig-urwid-cephinstall-cluster-network-settings), [here](#fig-urwid-cephinstall-cluster-network-settings)) * the dashboard configuration screen (figure [9.12](#fig-urwid-cephinstall-dashboard), [here](#fig-urwid-cephinstall-dashboard)) The settings of the Base View screen are explained in the accompanying texts for figures [9.3](#fig-urwid-cephinstall-general-cluster-settings), [9.4](#fig-urwid-cephinstall-cluster-network-settings), and [9.12](#fig-urwid-cephinstall-dashboard). ### 9.3.2 Base View Ceph Install: Nodes Selection Screen The next screen is the Ceph Nodes selection screen (figure [9.15](#fig-bright-view-ceph-setup-wizard-ceph-nodes-selection-screen)). This allows items to be selected for use as Ceph Monitors and OSDs. The items to be selected can be categories or nodes: ![Ceph Wizard Installation: Ceph Nodes Selection Screen](/base-command-manager/product-manuals/_fern-img/580bb62a474126614bb112aa8a4024164e5dd19b4f15bf4bc7b2bcf9a9c030a8.webp) *Figure 9.15: Ceph Wizard Installation: Ceph Nodes Selection Screen* For every selected OSD category or node, the corresponding block devices need to be configured. Clicking on the small settings icon next to the checkbox opens the block device selection dialog (figure [9.16](#fig-bright-view-ceph-setup-wizard-bluestore-block-devices)): ![Ceph Wizard Installation: Ceph Block Devices Selection Screen](/base-command-manager/product-manuals/_fern-img/34d57074dc9fe5682bfac3277b5fa84a08126625ca37e83a93b927a8d1180441.webp) *Figure 9.16: Ceph Wizard Installation: Ceph Block Devices Selection Screen* Validation checks are carried out on the selection of monitoring nodes and OSDs is done, and on the specification of the OSD block devices. If the checks for these inputs are cleared, then the `Next` button becomes active. ### 9.3.3 Base View Ceph Install: Summary Screen The next screen is the `Summary` screen (figure [9.17](#fig-bright-view-ceph-setup-wizard-configuration-summary-screen)). This summarizes the choices that have been made. The `Show config` button displays the underlying raw YAML configuration in a popup window. ![Ceph Wizard Installation: Configuration Summary](/base-command-manager/product-manuals/_fern-img/5bff8498c516708f1dfbfb60fa4142b1ec38631ff7de412f646a2845797cb826.webp) *Figure 9.17: Ceph Wizard Installation: Configuration Summary* In figure [9.17](#fig-bright-view-ceph-setup-wizard-configuration-summary-screen) after the `Ready for deployment` checkbox is checked, the `Deploy` button proceeds with deploying Ceph according to the configuration specified in the wizard. ### 9.3.4 Base View Ceph Install: Deployment Screen During the deployment process, the progress is displayed (figure [9.18](#fig-bright-view-ceph-setup-wizard-deployment-progress)). ![Ceph Wizard Installation: Deployment Progress](/base-command-manager/product-manuals/_fern-img/d6422a60f35c4fb873f5ea8182e5e19e6745858ed87ad548e358996eb773946e.webp) *Figure 9.18: Ceph Wizard Installation: Deployment Progress* The event viewer in Base View also shows the changes taking place. When deployment is complete, the `Finish` button ends the wizard. The state of the deployed system can be checked as shown in section [9.4.1](#941-checking-on-ceph-and-ceph-related-files-from-the-shell). ## 9.4 Checking And Getting Familiar With Ceph Items After `cm-ceph-setup` ### 9.4.1 Checking On Ceph And Ceph-related Files From The Shell After deployment, the OSD and Monitor services take some time be created and to start up. When all is up and running, the status of a healthy system, according to the output of the `ceph -s` command, should look something like the following: **Example** ```bash [root@basecm10 ~]# ceph -s cluster: id: 86c81b39-e593-44bd-877b-234158123f0a health: HEALTH_OK services: mon: 1 daemons, quorum mon001 (age 8h) mgr: mon001(active, since 8h) osd: 4 osds: 4 up (since 8h), 4 in (since 8h) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 20 MiB used, 120 GiB / 120 GiB avail pgs: 1 active+clean ``` The `-h` option to `ceph` lists many options. Users of BCM should usually not need to use these, and should find it more convenient to use the Base View or `cmsh` front ends instead. #### 9.4.1.1 Generated YAML Configuration File A YAML configuration file, by default `cm-ceph-setup.conf`, is generated after a run by the `cm-ceph-setup` utility. #### 9.4.1.2 Using A YAML Configuration File The `-c` option to `cm-ceph-setup` allows an existing YAML configuration file to be used. **Example** ```bash [root@basecm10 ~]# cm-ceph-setup -c /root/cm-ceph-setup.conf ``` #### 9.4.1.3 A Sample YAML Configuration File **Example** ```bash ####################################################################### ## This config file should be used with cm-ceph-setup tool ## Example: ## cm-ceph-setup -c ## ## Generated by: ## cm-ceph-setup ## cluster-tools-8.2-112402_cm8.2_5c7d79cc2f ## cmdline: /cm/local/apps/cm-setup/bin/cm-ceph-setup ## Generate on host: ## basecm10 ## Date of generation: ## Tue Feb 12 17:43:22 2019 ## MD5 checksum of everything after the closing comment: ## 2fc239700ef9f639d5aeb7cb9103091a ## to compare: grep -v '^##' | md5sum ####################################################################### meta: command_line: /cm/local/apps/cm-setup/bin/cm-ceph-setup date: Tue Feb 12 17:43:22 2019 generated_with: Ceph Setup hostname: basecm10 package_name: cluster-tools-8.2-112402_cm8.2_5c7d79cc2f package_version: '112402' modules: ceph: dashboard: external_port: 8443 internal_port: 8444 password: ceph username: ceph head_node: external_repos: - https://openresty.org/package/centos/openresty.repo packages: - openresty monitors: allow_pool_delete: true categories: {} nodes: mon001: {} mon002: {} mon003: {} networks: cluster: '' public: '' osd: FileStore: journal_size: 5120 categories: {} nodes: osd001: BlueStore: configurations: osd0: device: /dev/vdc FileStore: configurations: {} shared_journal: device: '' osd002: BlueStore: configurations: osd0: device: /dev/vdc FileStore: configurations: {} shared_journal: device: '' pool_default_size: 2 packages: - ceph - cm-config-ceph-systemd prometheus: description: Prometheus Ceph plugin filter: Prometheus4CephFilter name: Prometheus4Ceph port: 9283 repos: - epel - Ceph - Ceph-noarch roles: ceph_dashboard_reverse_proxy: configurations: - content: ceph/templates/nginx.service kind: static name: service path: /usr/lib/systemd/system/ceph-dashboard-reverse-proxy.service - kind: template name: lua-script path: /cm/local/apps/ceph/dashboard/nginx/nginx.lua template: ceph/templates/lua.template - kind: template name: config path: /cm/local/apps/ceph/dashboard/nginx/nginx.conf template: ceph/templates/config.template env: ext_port: 8443 int_port: 8444 kind: generic nodes: - active packages: - openresty services: - ceph-dashboard-reverse-proxy ``` For legacy FileStore configurations, the partitioning of Ceph OSD storage devices is done using the disk setup functionality as described in section [3.12.3](/admin/configuring#3123-changing-disk-layouts). For BlueStore, the corresponding devices are listed in the `CephOSDBlueStoreConfig` of the `CephOSDRole` only, and no entries are added to the XML disk layout. #### 9.4.1.4 Installation Logs Installation logs to Ceph are kept at: ```bash /var/log/cm-ceph-setup.log ``` ### 9.4.2 Ceph Management With Base View And `cmsh` Only one instance of Ceph is supported at a time. Its name is `ceph`. #### 9.4.2.1 Ceph Overview From within `cmsh`, `ceph` mode can be accessed: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% ceph [basecm10->ceph]% ``` From within `ceph` mode, the `overview` command lists an overview of Ceph OSDs, MONs, and placement groups for the `ceph` instance: **Example** ```bash [basecm10->ceph]% overview ceph Parameter Value -------------------------------- ---------------------------- Status HEALTH_OK Number of OSDs 2 Number of OSDs up 2 Number of OSDs in 2 Number of mons 1 Number of placements groups 192 Placement groups data size 0B Placement groups used size 10.07GB Placement groups available size 9.91GB Placement groups total size 19.98GB ``` The Base View equivalent of the `overview` command is the Ceph `Overview` window, accessed via the navigation path `Storage` > `Ceph` > `Ceph Settings` > `Overview`. #### 9.4.2.2 Ceph General Property Settings Some of the major Ceph configuration parameters can be viewed and their values managed by CMDaemon from `ceph` mode. The `show` command shows parameters and their values for the `ceph` instance: **Example** ```bash [basecm10->ceph]% show ceph Parameter Value ------------------------------------ ------------------------------------------------ Admin keyring path /etc/ceph/ceph.client.admin.keyring Auto Adjust CRUSH Map no Bootstrapped yes Client admin key AQCI7klcHwWIBxAAMUSzd7eYYQiskZELGiaEaA== Cluster networks Config file path /etc/ceph/ceph.conf Creation time Thu, 24 Jan 2019 17:57:43 CET Extra config parameters osd journal size = 5120 Monitor daemon port 6789 Monitor key AQCI7klc2bSyAxAAa5p6p+ljWin75ucxR3gy+Q== Monitor keyring path /etc/ceph/ceph.mon.keyring Public networks Revision auth client required cephx yes auth cluster required cephx yes auth service required cephx yes filestore xattr use omap no fsid b4e9fd96-800d-4f66-87f1-79febb102ef5 mon allow pool delete yes mon max osd 10000 mon osd full ratio 0.950000 mon osd nearfull ratio 0.850000 name ceph osd pool default min size 0 osd pool default pg num 8 osd pool default pgp num 8 osd pool default size 2 rbd cache yes rbd cache max dirty 25165824 rbd cache max dirty age 1.000000 rbd cache size 33554432 rbd cache target dirty 16777216 rbd cache writethrough until flush yes rbd readahead disable after bytes 52428800 rbd readahead max bytes 524288 rbd readahead trigger requests 10 version 13.2.4 [basecm10->ceph]% ``` The Base View equivalent of these settings is in the `Settings` window, accessed via a navigation path of `Storage` > `Ceph` > `Ceph Settings` > `Overview` > `Settings`. **Ceph `extraconfigparameters` setting:** The `Extra config parameters` property of a `ceph` mode object can be used to customize the Ceph configuration file. The Ceph configuration file is typically in `/etc/ceph.conf`, and using `extraconfiparameters` settings, Ceph can be configured with changes that CMDaemon would otherwise not manage. After the changes have been set, CMDaemon manages them further. Thus, the following configuration section in the Ceph configuration file: ```bash [mds.2] host=rabbit ``` could be placed in the file via `cmsh` with: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% ceph [basecm10->ceph[ceph]]% append extraconfigparameters "[mds.2] host=rabbit" [basecm10->ceph*[ceph*]]% commit ``` If a section name, enclosed in square brackets, `[]`, is used, then the section is recognized at the start of an appended line by CMDaemon. If a section that is specified in the square brackets does not already exist in `/etc/ceph.conf`, then it will be created. The \`n\` is interpreted as a new line at its position. After the commit, the extra configuration parameter setting is maintained by the cluster manager. If the section already exists in `/etc/ceph.conf`, then the associated key=value pair is appended. For example, the following appends `host2=bunny` to an existing `mds.2` section: ```bash [basecm10->ceph[ceph]]% append extraconfigparameters "[mds.2] host2=bunny" [basecm10->ceph*[ceph*]]% commit ``` If no section name is used, then the key=value entry is appended to the \[global] section. ```bash [basecm10->ceph[ceph]]% append extraconfigparameters "osd journal size = 128" [basecm10->ceph*[ceph*]]% commit ``` The `/etc/ceph.conf` file has the changes written into it about a minute after the commit, and may then look like (some lines removed for clarity): ```bash [global] auth client required = cephx osd journal size=128 [mds.2] host=rabbit host2=bunny ``` As usual in `cmsh` operations (section [2.5.3](/admin/clustermanagement#253-working-with-objects)): * The `set` command clears `extraconfigparameters` before setting its value * The `removefrom` command operates as the opposite of the `append` command, by removing key=value pairs from the specified section. There are similar `extraconfigparameters` for Ceph OSD filesystem associations ([here](#osd-filesystem-association-extraconfigparameters-setting)) and for Ceph monitoring ([here](#ceph-monitoring-extraconfigparameters-setting)). #### 9.4.2.3 Ceph OSD Properties From within `ceph` mode, the `osdinfo` command for the Ceph instance displays the nodes that are providing OSDs along with their OSD IDs: **Example** ```bash [basecm10->ceph]% osdinfo ceph OSD id Node OSD name ------------ ---------------------- ------------ 0 node001 osd0 1 node002 osd0 ``` Within a device or category mode, the `roles` submode allows parameters of an assigned `cephosd` role to be configured and managed. **Example** ```bash [basecm10->device[node001]->roles]% show cephosd Parameter Value --------------------------- ------------------------------ Add services yes Name cephosd OSD configurations <1 in submode> Revision Type CephOSDRole ``` Within the `cephosd` role the templates for OSD filesystem configurations, `osdconfigurations`, can be set or modified: **Example** ```bash [basecm10->device[node001]->roles]% use cephosd [basecm10...[node001]->roles[cephosd]]% osdconfigurations [basecm10...osd]->osdconfigurations]% show osd0 Parameter Value -------------------------------- -------------------------------------- Automatically adjust weight off Extra config parameters Initial weight 0.1 Journal data /var/lib/ceph/osd/$cluster-$id/journal Journal size 0 MiB Name osd0 OSD data /var/lib/ceph/osd/$cluster-$id Production weight 1 Revision Type CephOSDLegacyConfig Weight adjust interval 5 Weight adjust rate 0.1 Weight interpretation scale ``` The Base View equivalent to access the preceding `cmsh` OSD configuration settings is via the role for a particular node or category. The navigation path that brings up these configuration options for node `node001` is, for example: `Devices` > `Physical Nodes` > `node001` > `Edit` > `Settings` > `Roles` > `cephosd` > `Edit` > `osd0` > `Edit` **OSD filesystem association `extraconfigparameters` setting:** Extra configuration parameters can be set for an OSD filesystem association such as `osd0` by setting values for its `extraconfigparameters` option. This is similar to how it can be done for Ceph general configuration ([here](#ceph-extraconfigparameters-setting)): **Example** ```bash [basecm10...osd]->osdconfigurations]% use osd0 [basecm10...osdconfigurations[osd0]]% show Parameter Value -------------------------------- -------------------------------------- ... Automatically adjust weight off Extra config parameters ... [basecm10...osdconfigurations[osd0]]% set extraconfigparameters "a=b" ... ``` #### 9.4.2.4 Ceph Monitoring Properties Similarly to Ceph OSD properties, the parameters of the `cephmonitor` role can be configured and managed from within the node or category that runs Ceph monitoring. **Example** ```bash [basecm10]% device use basecm10 [basecm10->device[basecm10]]% roles ; use cephmonitor [ceph->device[basecm10]->roles[cephmonitor]]% show Parameter Value -------------------------------- ------------------------------------ ... Extra config parameters Monitor data /var/lib/ceph/mon/$cluster-$hostname Name cephmonitor Revision Type CephMonitorRole ``` **Ceph monitoring `extraconfigparameters` setting:** Ceph monitoring can also have extra configurations set via the `extraconfigparameters` option, in a similar way to how it is done for Ceph general configuration ([here](#ceph-extraconfigparameters-setting)). The Base View equivalent to access the preceding `cmsh` Monitor configuration setting is via the role for a particular node or category. The navigation path that brings up these configuration options for node `node004` is, for example: `Devices` > `Physical Nodes` > `node004` > `Edit` > `Settings` > `Roles` > `cephmonitor` > `Edit` #### 9.4.2.5 Ceph `bootstrap` For completeness, the `bootstrap` command within `ceph` mode can be used by the administrator to initialize Ceph Monitors on specified nodes if they are not already initialized. Administrators are however not expected to use it, because they are expected to use the `cm-ceph-setup` installer utility when installing Ceph in the first place. The installer utility carries out the bootstrap initialization as part of its tasks. The `bootstrap` command is therefore only intended for use in the unusual case where the administrator would like to set up Ceph storage without using the `cm-ceph-setup` utility. ## 9.5 The Ceph Dashboard During Ceph installation, the Ceph Dashoard can also be installed * by ticking a checkbox in the screen of figure [9.12](#fig-urwid-cephinstall-dashboard) during the TUI `cm-ceph-setup` installation * by ticking a checkbox in the screen of figure [9.14](#fig-bright-view-ceph-setup-wizard-general-ceph-cluster-settings-screen) during the use of the Base View Ceph Wizard Installer After installation, the Dashboard can be accessed via the link in the Ceph block in the head node landing page (figure [2.1](/admin/clustermanagement#241-installing-the-cluster-management-gui-service), [here](/admin/clustermanagement#fig-bright-headnode-IP-address-landing-page)). If the Dashboard is installed with the default user and port setting, then it can be logged into as the user `ceph` via a URL of the form: `https://` \<*`host name or IP address`*>`:8443` On logging in, the Dashboard screen comes up (figure [9.19](#fig-ceph-dashboard)): ![Ceph Dashboard](/base-command-manager/product-manuals/_fern-img/7624d9fc8cb35b0c685e323a3ebf3eef37c6384566330355a585db7c70b4e13a.webp) *Figure 9.19: Ceph Dashboard*