{"id":28866,"date":"2023-07-24T15:34:12","date_gmt":"2023-07-24T13:34:12","guid":{"rendered":"https:\/\/blog.miniserver.it\/?p=28866"},"modified":"2023-07-25T09:59:27","modified_gmt":"2023-07-25T07:59:27","slug":"how-to-create-a-3-node-hyperconverged-cluster-with-ceph-cephfs","status":"publish","type":"post","link":"https:\/\/blog.miniserver.it\/en\/proxmox-ve\/how-to-create-a-3-node-hyperconverged-cluster-with-ceph-cephfs\/","title":{"rendered":"How to Create a 3 Node Hyperconverged Cluster with Ceph &#8211; CephFS"},"content":{"rendered":"<div class=\"fusion-fullwidth fullwidth-box fusion-builder-row-1 fusion-flex-container has-pattern-background has-mask-background nonhundred-percent-fullwidth non-hundred-percent-height-scrolling\" style=\"--awb-border-radius-top-left:0px;--awb-border-radius-top-right:0px;--awb-border-radius-bottom-right:0px;--awb-border-radius-bottom-left:0px;--awb-flex-wrap:wrap;\" ><div class=\"fusion-builder-row fusion-row fusion-flex-align-items-flex-start fusion-flex-content-wrap\" style=\"max-width:1123.2px;margin-left: calc(-4% \/ 2 );margin-right: calc(-4% \/ 2 );\"><div class=\"fusion-layout-column fusion_builder_column fusion-builder-column-0 fusion_builder_column_1_1 1_1 fusion-flex-column\" style=\"--awb-bg-size:cover;--awb-width-large:100%;--awb-margin-top-large:0px;--awb-spacing-right-large:1.92%;--awb-margin-bottom-large:20px;--awb-spacing-left-large:1.92%;--awb-width-medium:100%;--awb-order-medium:0;--awb-spacing-right-medium:1.92%;--awb-spacing-left-medium:1.92%;--awb-width-small:100%;--awb-order-small:0;--awb-spacing-right-small:1.92%;--awb-spacing-left-small:1.92%;\"><div class=\"fusion-column-wrapper fusion-column-has-shadow fusion-flex-justify-content-flex-start fusion-content-layout-column\"><div class=\"fusion-text fusion-text-1\"><p>In this article we will install and configure a Ceph Cluster environment with 3 hyperconverged nodes and then interface it, through CephFS with a Ceph Client.<\/p>\n<\/div><\/div><\/div><div class=\"fusion-layout-column fusion_builder_column fusion-builder-column-1 fusion_builder_column_1_1 1_1 fusion-flex-column\" style=\"--awb-padding-right:20px;--awb-padding-left:20px;--awb-bg-color:#f1f4f6;--awb-bg-color-hover:#f1f4f6;--awb-bg-blend:overlay;--awb-bg-size:cover;--awb-width-large:100%;--awb-margin-top-large:0px;--awb-spacing-right-large:1.92%;--awb-margin-bottom-large:0px;--awb-spacing-left-large:1.92%;--awb-width-medium:100%;--awb-spacing-right-medium:1.92%;--awb-spacing-left-medium:1.92%;--awb-width-small:100%;--awb-spacing-right-small:1.92%;--awb-spacing-left-small:1.92%;\"><div class=\"fusion-column-wrapper fusion-flex-justify-content-flex-start fusion-content-layout-column\"><div class=\"fusion-title title fusion-title-1 fusion-sep-none fusion-title-text fusion-title-size-two\" style=\"--awb-margin-top:20px;--awb-margin-bottom:20px;--awb-margin-top-small:10px;--awb-margin-right-small:0px;--awb-margin-bottom-small:10px;--awb-margin-left-small:0px;\"><h2 class=\"fusion-title-heading title-heading-left\" style=\"margin:0;\">How to create hyperconverged cluster with Ceph<\/h2><\/div><div class=\"awb-toc-el awb-toc-el--1\" data-awb-toc-id=\"1\" data-awb-toc-options=\"{&quot;allowed_heading_tags&quot;:{&quot;h3&quot;:0},&quot;ignore_headings&quot;:&quot;&quot;,&quot;ignore_headings_words&quot;:&quot;indice&quot;,&quot;enable_cache&quot;:&quot;yes&quot;,&quot;highlight_current_heading&quot;:&quot;no&quot;,&quot;hide_hidden_titles&quot;:&quot;yes&quot;,&quot;limit_container&quot;:&quot;post_content&quot;,&quot;select_custom_headings&quot;:&quot;&quot;,&quot;icon&quot;:&quot;fa-flag fas&quot;,&quot;counter_type&quot;:&quot;none&quot;}\" style=\"--awb-margin-bottom:20px;--awb-margin-left:30px;\"><div class=\"awb-toc-el__content\"><ul class=\"awb-toc-el__list awb-toc-el__list--0\"><li class=\"awb-toc-el__list-item\"><a class=\"awb-toc-el__item-anchor\" href=\"#toc_1_Installazione_di_CEPHADM_per_la_gestione_del\">1. Installing CEPHADM for cluster management<\/a><\/li><li class=\"awb-toc-el__list-item\"><a class=\"awb-toc-el__item-anchor\" href=\"#toc_2_Docker_installation\">2. Docker installation<\/a><\/li><li class=\"awb-toc-el__list-item\"><a class=\"awb-toc-el__item-anchor\" href=\"#toc_3_Creazione_della_CephFS\">3. CephFS creation<\/a><\/li><li class=\"awb-toc-el__list-item\"><a class=\"awb-toc-el__item-anchor\" href=\"#toc_4_Mount_da_un_Ceph_Client\">4. Ceph Client Mounting<\/a><\/li><li class=\"awb-toc-el__list-item\"><a class=\"awb-toc-el__item-anchor\" href=\"#toc_4_Ceph_Client\">4. Ceph Client<\/a><\/li><li class=\"awb-toc-el__list-item\"><a class=\"awb-toc-el__item-anchor\" href=\"#toc_6_Monitor_Restart\">6. Monitor Restart<\/a><\/li><\/ul><\/div><\/div><\/div><\/div><div class=\"fusion-layout-column fusion_builder_column fusion-builder-column-2 fusion_builder_column_1_1 1_1 fusion-flex-column\" style=\"--awb-bg-size:cover;--awb-width-large:100%;--awb-margin-top-large:30px;--awb-spacing-right-large:1.92%;--awb-margin-bottom-large:30px;--awb-spacing-left-large:1.92%;--awb-width-medium:100%;--awb-order-medium:0;--awb-spacing-right-medium:1.92%;--awb-spacing-left-medium:1.92%;--awb-width-small:100%;--awb-order-small:0;--awb-spacing-right-small:1.92%;--awb-spacing-left-small:1.92%;\"><div class=\"fusion-column-wrapper fusion-column-has-shadow fusion-flex-justify-content-flex-start fusion-content-layout-column\"><div class=\"fusion-text fusion-text-2\" style=\"--awb-margin-top:30px;\"><p>Suppose we have 3 machines available on which to install (node1, node2, node3)<\/p>\n<h3 id=\"toc_1_Installazione_di_CEPHADM_per_la_gestione_del\">1. Installing CEPHADM for cluster management<\/h3>\n<p>Run these commands to Install cephadm on node1.<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-1 > .CodeMirror, .fusion-syntax-highlighter-1 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-1 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-1 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-1 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_1\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_1\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_1\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">apt-get install ca-certificates curl gnupg\nCEPH_RELEASE=17.2.6\ncurl --silent --remote-name --location https:\/\/download.ceph.com\/rpm-${CEPH_RELEASE}\/el9\/noarch\/cephadm\nchmod +x cephadm\n.\/cephadm add-repo --release quincy\n.\/cephadm install<\/textarea><\/div><div class=\"fusion-text fusion-text-3\"><p>Install cephadmin as a &#8220;tool&#8221; on the system. This operation takes about 30minutes to complete.<\/p>\n<p>Once the installation is complete, we execute the ceph admin boostrap, using the IP of the host we are operating on<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-2 > .CodeMirror, .fusion-syntax-highlighter-2 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-2 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_2\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_2\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_2\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"1\" data-theme=\"default\">cephadm bootstrap --mon-ip 192.168.131.101<\/textarea><\/div><div class=\"fusion-text fusion-text-4\"><p>Now, cephadm will download the container images to create the Ceph cluster infrastructure and to send it running in our network 192.168.131.0\/24.<\/p>\n<\/div><div class=\"fusion-alert alert error alert-danger fusion-alert-left awb-alert-native-link-color\" role=\"alert\"><div class=\"fusion-alert-content-wrapper\"><span class=\"alert-icon\"><i class=\"awb-icon-exclamation-triangle\" aria-hidden=\"true\"><\/i><\/span><span class=\"fusion-alert-content\">Warning: we have not provided another network for the Cluster Ceph Network. Don&#8217;t do this procedure in production<\/span><\/div><\/div><div class=\"fusion-text fusion-text-5\"><p>At the end of the procedure, we see that the URL to access the cluster is:<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-28819\" src=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/URL-per-accedere-al-cluster.png\" alt=\"URL per accedere al cluster\" width=\"407\" height=\"68\" srcset=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/URL-per-accedere-al-cluster-200x33.png 200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/URL-per-accedere-al-cluster-400x67.png 400w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/URL-per-accedere-al-cluster.png 407w\" sizes=\"(max-width: 407px) 100vw, 407px\" \/><\/p>\n<p>&nbsp;<\/p>\n<\/div><div class=\"fusion-text fusion-text-6\"><p>The followng command is also provided<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-28822\" src=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/comando-ceph.png\" alt=\"comando ceph\" width=\"686\" height=\"49\" srcset=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/comando-ceph-200x14.png 200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/comando-ceph-400x29.png 400w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/comando-ceph-600x43.png 600w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/comando-ceph-669x49.png 669w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/comando-ceph.png 686w\" sizes=\"(max-width: 686px) 100vw, 686px\" \/><\/p>\n<p>&nbsp;<\/p>\n<\/div><div class=\"fusion-text fusion-text-7\"><p>The command will allow you to access the shell.<\/p>\n<p>The ceph GUI is very powerful and allows many operations but having access to the shell is very important, even in case of problems.<\/p>\n<p>Let&#8217;s get the ceph key<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-3 > .CodeMirror, .fusion-syntax-highlighter-3 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-3 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_3\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_3\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_3\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"1\" data-theme=\"default\">cat \/etc\/ceph\/ceph.pub<\/textarea><\/div><div class=\"fusion-text fusion-text-8\"><p>We then place it in the &#8220;authorized_keys&#8221; file of node2 and node3<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-4 > .CodeMirror, .fusion-syntax-highlighter-4 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-4 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_4\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_4\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_4\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"1\" data-theme=\"default\">mkdir .ssh\ncd .ssh\nnano authorized_keys\n\n## copiano la chiave nel file<\/textarea><\/div><div class=\"fusion-text fusion-text-9\"><p>This way ceph will have full control of the other two hosts<\/p>\n<p>Enter the GUI with the indicated URL<\/p>\n<\/div><div class=\"fusion-alert alert error alert-danger fusion-alert-left awb-alert-native-link-color\" role=\"alert\"><div class=\"fusion-alert-content-wrapper\"><span class=\"alert-icon\"><i class=\"awb-icon-exclamation-triangle\" aria-hidden=\"true\"><\/i><\/span><span class=\"fusion-alert-content\">use the IP of the VM or set the entry in the host file, In our case \u2014&gt; https:\/\/192.168.131.101:8443\/<\/span><\/div><\/div><div class=\"fusion-text fusion-text-10\"><p>Change the password as required. Then the process of creating the cluster starts. We thus land on the procedure for creating the Cluster Ceph<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-28828\" src=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/welcome-to-ceph-dashboard.png\" alt=\"welcome to ceph dashboard\" width=\"658\" height=\"404\" srcset=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/welcome-to-ceph-dashboard-200x123.png 200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/welcome-to-ceph-dashboard-400x246.png 400w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/welcome-to-ceph-dashboard-551x338.png 551w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/welcome-to-ceph-dashboard-600x368.png 600w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/welcome-to-ceph-dashboard.png 658w\" sizes=\"(max-width: 658px) 100vw, 658px\" \/><\/p>\n<p>&nbsp;<\/p>\n<\/div><div class=\"fusion-text fusion-text-11\"><p>Before proceeding, let&#8217;s install Docker on the other nodes.<\/p>\n<\/div><\/div><\/div><div class=\"fusion-layout-column fusion_builder_column fusion-builder-column-3 fusion_builder_column_1_1 1_1 fusion-flex-column\" style=\"--awb-bg-size:cover;--awb-width-large:100%;--awb-margin-top-large:0px;--awb-spacing-right-large:1.92%;--awb-margin-bottom-large:20px;--awb-spacing-left-large:1.92%;--awb-width-medium:100%;--awb-order-medium:0;--awb-spacing-right-medium:1.92%;--awb-spacing-left-medium:1.92%;--awb-width-small:100%;--awb-order-small:0;--awb-spacing-right-small:1.92%;--awb-spacing-left-small:1.92%;\"><div class=\"fusion-column-wrapper fusion-column-has-shadow fusion-flex-justify-content-flex-start fusion-content-layout-column\"><div class=\"fusion-text fusion-text-12\"><h3>2. Docker installation<\/h3>\n<p>SSH into the other two nodes and run the Docker installation procedure<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-5 > .CodeMirror, .fusion-syntax-highlighter-5 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-5 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-5 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-5 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_5\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_5\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_5\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">apt-get update\n\napt install curl\n\napt-get install ca-certificates curl gnupg\n\ninstall -m 0755 -d \/etc\/apt\/keyrings\n\ncurl -fsSL https:\/\/download.docker.com\/linux\/debian\/gpg | gpg --dearmor -o \/etc\/apt\/keyrings\/docker.gpg\n\nchmod a+r \/etc\/apt\/keyrings\/docker.gpg\n\necho \\\n  \"deb [arch=\"$(dpkg --print-architecture)\" signed-by=\/etc\/apt\/keyrings\/docker.gpg] https:\/\/download.docker.com\/linux\/debian \\\n  \"$(. \/etc\/os-release && echo \"$VERSION_CODENAME\")\" stable\" | \\\n   tee \/etc\/apt\/sources.list.d\/docker.list > \/dev\/null\n\napt-get update\n\napt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin<\/textarea><\/div><div class=\"fusion-text fusion-text-13\"><p>Once Docker is installed on both node2 and node3, continue with the installation from the GUI.<\/p>\n<p>Click on \u201cExpand Cluster\u201d and we find this screen<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-28831\" src=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster.png\" alt=\"ceph expand cluster\" width=\"1689\" height=\"503\" srcset=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster-200x60.png 200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster-400x119.png 400w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster-600x179.png 600w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster-768x229.png 768w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster-800x238.png 800w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster-1024x305.png 1024w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster-1135x338.png 1135w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster-1200x357.png 1200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster-1536x457.png 1536w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-expand-cluster.png 1689w\" sizes=\"(max-width: 1689px) 100vw, 1689px\" \/><\/p>\n<\/div><div class=\"fusion-text fusion-text-14\"><p>Let&#8217;s add node 2, with hostname ceph2 and IP 192.168.131.104<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-28833\" src=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-host.png\" alt=\"ceph add host\" width=\"901\" height=\"414\" srcset=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-host-200x92.png 200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-host-400x184.png 400w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-host-600x276.png 600w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-host-736x338.png 736w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-host-768x353.png 768w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-host-800x368.png 800w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-host.png 901w\" sizes=\"(max-width: 901px) 100vw, 901px\" \/><\/p>\n<\/div><div class=\"fusion-text fusion-text-15\"><p>We then do the same with node 3, with hostname ceph3<\/p>\n<\/div><div class=\"fusion-alert alert error alert-danger fusion-alert-left awb-alert-native-link-color\" role=\"alert\"><div class=\"fusion-alert-content-wrapper\"><span class=\"alert-icon\"><i class=\"awb-icon-exclamation-triangle\" aria-hidden=\"true\"><\/i><\/span><span class=\"fusion-alert-content\">This depends on the hostname you gave the machines. Change the name according to what you have.<\/span><\/div><\/div><div class=\"fusion-text fusion-text-16\"><p>We should see the following screen<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-28836\" src=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts.png\" alt=\"ceph add hosts\" width=\"1687\" height=\"629\" srcset=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts-200x75.png 200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts-400x149.png 400w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts-600x224.png 600w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts-768x286.png 768w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts-800x298.png 800w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts-907x338.png 907w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts-1024x382.png 1024w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts-1200x447.png 1200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts-1536x573.png 1536w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-add-hosts.png 1687w\" sizes=\"(max-width: 1687px) 100vw, 1687px\" \/><\/p>\n<\/div><div class=\"fusion-text fusion-text-17\"><p>Proceed with \u201cCreate OSD\u201d \u2014&gt; Advanced Mode \u2014&gt; +Add<\/p>\n<p>Wait for all disks to appear and select the filter so that they are all displayed<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-28838\" src=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-primary-devices.png\" alt=\"ceph primary devices\" width=\"1152\" height=\"548\" srcset=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-primary-devices-200x95.png 200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-primary-devices-400x190.png 400w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-primary-devices-600x285.png 600w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-primary-devices-711x338.png 711w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-primary-devices-768x365.png 768w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-primary-devices-800x381.png 800w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-primary-devices-1024x487.png 1024w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-primary-devices.png 1152w\" sizes=\"(max-width: 1152px) 100vw, 1152px\" \/><\/p>\n<\/div><div class=\"fusion-text fusion-text-18\"><p>Select Add<\/p>\n<p>Now let's add the services, going to Create Service \u2014&gt; Create and add the mds service on all the nodes<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-28840\" src=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-edit-service.png\" alt=\"ceph edit service\" width=\"824\" height=\"407\" srcset=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-edit-service-200x99.png 200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-edit-service-400x198.png 400w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-edit-service-600x296.png 600w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-edit-service-684x338.png 684w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-edit-service-768x379.png 768w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-edit-service-800x395.png 800w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-edit-service.png 824w\" sizes=\"(max-width: 824px) 100vw, 824px\" \/><\/p>\n<\/div><div class=\"fusion-text fusion-text-19\"><p>Now go to \u201cExpand Cluster\u201d<\/p>\n<p>will take care of creating all the services on all the nodes of the cluster and running .<\/p>\n<\/div><\/div><\/div><div class=\"fusion-layout-column fusion_builder_column fusion-builder-column-4 fusion_builder_column_1_1 1_1 fusion-flex-column\" style=\"--awb-bg-size:cover;--awb-width-large:100%;--awb-margin-top-large:0px;--awb-spacing-right-large:1.92%;--awb-margin-bottom-large:20px;--awb-spacing-left-large:1.92%;--awb-width-medium:100%;--awb-order-medium:0;--awb-spacing-right-medium:1.92%;--awb-spacing-left-medium:1.92%;--awb-width-small:100%;--awb-order-small:0;--awb-spacing-right-small:1.92%;--awb-spacing-left-small:1.92%;\"><div class=\"fusion-column-wrapper fusion-column-has-shadow fusion-flex-justify-content-flex-start fusion-content-layout-column\"><div class=\"fusion-text fusion-text-20\"><h3 id=\"toc_3_Creazione_della_CephFS\">3. CephFS creation<\/h3>\n<p><span style=\"background-color: var(--awb-bg-color-hover); color: var(--awb-text-color); font-family: var(--awb-text-font-family); font-size: var(--awb-font-size); font-style: var(--awb-text-font-style); font-weight: var(--awb-text-font-weight); letter-spacing: var(--awb-letter-spacing); text-align: var(--awb-content-alignment); text-transform: var(--awb-text-transform);\">Ceph File System (CephFS) is a POSIX-compliant file system that relies on object-oriented distributed Ceph storage, called RADOS, to provide highly available and high-performance storage for various applications.<\/span><\/p>\n<p>It supports traditional use cases such as shared directories, temporary workspace for high-performance computing, and shared storage for distributed workflows.<span style=\"color: var(--awb-text-color); font-family: var(--awb-text-font-family); font-size: var(--awb-font-size); font-style: var(--awb-text-font-style); font-weight: var(--awb-text-font-weight); letter-spacing: var(--awb-letter-spacing); text-align: var(--awb-content-alignment); text-transform: var(--awb-text-transform); background-color: var(--awb-bg-color-hover);\"><br \/>\n<\/span><\/p>\n<p>CephFS adopts a unique architecture where file metadata is stored separately from file data in RADOS. Metadata is served through a cluster of Metadata Servers (MDS) that can be scaled to handle high-speed metadata workloads.<\/p>\n<p>Clients directly access RADOS to read and write blocks of file data, allowing for linear scalability based on the size of the underlying RADOS object storage.<\/p>\n<p>Data access is coordinated by the MDS cluster, which acts as the authority for the distributed metadata cache maintained collaboratively between the clients and the MDSs themselves.<\/p>\n<p>Metadata mutations are efficiently written to a journal in RADOS by each MDS, without storing any metadata state locally.<\/p>\n<p>This model allows for consistent and fast collaboration between clients within a POSIX file system.<\/p>\n<p>CephFS is a scalable, distributed file system that provides real-time, simultaneous access to large amounts of data from multiple compute nodes.<\/p>\n<p>It is suitable for a variety of use cases that require distributed storage, parallel processing, and data reliability.<\/p>\n<p>We enter the Cephadm shell with:<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-6 > .CodeMirror, .fusion-syntax-highlighter-6 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-6 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-6 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-6 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_6\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_6\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_6\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">## questo comando (ovviamente) \u00e8 disponibile dove ho installato la cephadm\ncephadm shell<\/textarea><\/div><div class=\"fusion-text fusion-text-21\"><p>Let's first create two pools:<\/p>\n<p>The first pool will be the one for metadata<\/p>\n<p>The second pool will be the one for the actual data.<\/p>\n<p>We then execute the two commands for creating the pools:<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-7 > .CodeMirror, .fusion-syntax-highlighter-7 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-7 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-7 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-7 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_7\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_7\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_7\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">ceph osd pool create pool-storage\nceph osd pool create cephfs_metadata<\/textarea><\/div><div class=\"fusion-alert alert general alert-info fusion-alert-left awb-alert-native-link-color\" role=\"alert\"><div class=\"fusion-alert-content-wrapper\"><span class=\"alert-icon\"><i class=\"awb-icon-info-circle\" aria-hidden=\"true\"><\/i><\/span><span class=\"fusion-alert-content\">In a production environment, set the correct number of PGs as needed. The command with all options is:<\/span><\/div><\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-8 > .CodeMirror, .fusion-syntax-highlighter-8 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-8 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-8 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-8 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_8\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_8\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_8\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">ceph osd pool create {pool-name} [{pg-num} [{pgp-num}]] [replicated]\\         \n[crush-rule-name] [expected-num-objects]<\/textarea><\/div><div class=\"fusion-text fusion-text-22\"><p>Once the two pools have been created, let's create CephFS<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-9 > .CodeMirror, .fusion-syntax-highlighter-9 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-9 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-9 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-9 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_9\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_9\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_9\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">ceph fs new ceph_fs cephfs_metadata pool-storage\n## cephfs_metadata = pool creato solo per i  metadati\n## pool-storage = pool creato per i dati<\/textarea><\/div><div class=\"fusion-text fusion-text-23\"><p>Here are some useful commands for manipulating CephFS<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-10 > .CodeMirror, .fusion-syntax-highlighter-10 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-10 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-10 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-10 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_10\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_10\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_10\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"\" data-theme=\"default\">## lista tutte le CephFS create\nceph fs ls\n\n## rinomina la CephFS\nfs rename <file system name> <new file system name> [--yes-i-really-mean-it]\n\n## Mettere down il Cluster:\nfs set <fs_name> down true\n## Portarlo online\nfs set <fs_name> down false\n<\/textarea><\/div><div class=\"fusion-alert alert general alert-info fusion-alert-left awb-alert-native-link-color\" role=\"alert\"><div class=\"fusion-alert-content-wrapper\"><span class=\"alert-icon\"><i class=\"awb-icon-info-circle\" aria-hidden=\"true\"><\/i><\/span><span class=\"fusion-alert-content\">To eliminate a CephFS you must first deactivate all the MDSs present on the system Go to Cluster \u2014&gt; Services \u2014&gt; mds.mds Select the daemon and click STOP for each daemon Then execute:<\/span><\/div><\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-11 > .CodeMirror, .fusion-syntax-highlighter-11 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-11 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-11 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-11 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_11\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_11\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_11\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"\" data-theme=\"default\">ceph fs rm ceph_fs --yes-i-really-mean-it\n\n## ceph_fs = nome della CephFS che ho creato prima<\/textarea><\/div><div class=\"fusion-text fusion-text-24\"><p>This will NOT delete data within the pool, just CephFS<\/p>\n<\/div><div class=\"fusion-text fusion-text-25\"><h3 id=\"toc_4_Mount_da_un_Ceph_Client\">4. Ceph Client Mounting<\/h3>\n<p>Now that we have created CephFS, we want to make it usable on a Client machine (in this case a Debian)<\/p>\n<p>Let's move onto the machine with cephadm (node 1) and enter the shell<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-12 > .CodeMirror, .fusion-syntax-highlighter-12 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-12 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-12 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-12 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_12\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_12\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_12\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">cephadm shell<\/textarea><\/div><div class=\"fusion-text fusion-text-26\"><p>Let's create the admin user key, which is then used to mount from the ceph-client<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-13 > .CodeMirror, .fusion-syntax-highlighter-13 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-13 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-13 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-13 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_13\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_13\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_13\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">cd \/etc\/ceph\nceph auth get-key client.admin > admin.secret\ncat admin.secret<\/textarea><\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-14 > .CodeMirror, .fusion-syntax-highlighter-14 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-14 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-14 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-14 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_14\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_14\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_14\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">AQDMDotkM0IsGBAAoB\/GfBtrSXHnh2nh2FSeHA==root@ceph1:\/etc\/ceph#<\/textarea><\/div><div class=\"fusion-text fusion-text-27\"><p>Copy the output and use it in the client you want to mount the CephFS on<\/p>\n<\/div><div class=\"fusion-text fusion-text-28\"><h3>4. Ceph Client<\/h3>\n<p>Let's install Ceph Common first<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-15 > .CodeMirror, .fusion-syntax-highlighter-15 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-15 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_15\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_15\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_15\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"1\" data-theme=\"default\">apt install ceph-common<\/textarea><\/div><div class=\"fusion-text fusion-text-29\"><p>Once on the machine that will act as a ceph client, copy the key to the correct path<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-16 > .CodeMirror, .fusion-syntax-highlighter-16 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-16 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-16 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-16 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_16\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_16\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_16\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">mkdir \/etc\/ceph\nmkdir \/etc\/ceph\/admin\n\n## usiamo nano per copiare la key\nnano \/etc\/ceph\/admin.secret<\/textarea><\/div><div class=\"fusion-text fusion-text-30\"><p>Create a folder on the client, on which we will mount the CephFS<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-17 > .CodeMirror, .fusion-syntax-highlighter-17 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-17 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-17 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-17 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_17\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_17\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_17\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">mkdir \/etc\/mycephfs<\/textarea><\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-18 > .CodeMirror, .fusion-syntax-highlighter-18 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-18 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-18 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-18 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_18\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_18\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_18\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">mount -t ceph {ip-address-of-ceph-monitor}:\/ \/mnt\/mycephfs -o name=admin,secretfile=\/etc\/ceph\/admin.secret<\/textarea><\/div><div class=\"fusion-text fusion-text-31\"><p>In our case:<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-19 > .CodeMirror, .fusion-syntax-highlighter-19 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-19 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-19 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-19 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_19\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_19\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_19\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">mount -t ceph 192.168.131.101:\/ \/mnt\/mycephfs -o name=admin,secretfile=\/etc\/ceph\/admin.secret<\/textarea><\/div><div class=\"fusion-text fusion-text-32\"><p>Mount permanently, adding this line in the \/etc\/fstab file<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-20 > .CodeMirror, .fusion-syntax-highlighter-20 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-20 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-20 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-20 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_20\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_20\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_20\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\">192.168.131.101,192.168.131.130,192.168.131.132:\/ \/mnt\/mycephfs ceph name=admin,secretfile=\/etc\/ceph\/admin.secret,noatime,_netdev 0 0<\/textarea><\/div><div class=\"fusion-alert alert general alert-info fusion-alert-left awb-alert-native-link-color\" role=\"alert\"><div class=\"fusion-alert-content-wrapper\"><span class=\"alert-icon\"><i class=\"awb-icon-info-circle\" aria-hidden=\"true\"><\/i><\/span><span class=\"fusion-alert-content\">Be careful not to forget the \u201c_netdev\u201d option. This allows you not to attempt to mount the resource until the network interface is ready.<\/span><\/div><\/div><div class=\"fusion-text fusion-text-33\"><p>Once the CephFS resource is mounted, let's check on the Ceph dashboard.<\/p>\n<p>Under File Systems \u2014&gt; ceph_fs \u2014&gt; Clients<\/p>\n<p>We see that the client we just associated is present<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-28850\" src=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems.png\" alt=\"ceph file systems\" width=\"1885\" height=\"708\" srcset=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems-200x75.png 200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems-400x150.png 400w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems-600x225.png 600w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems-768x288.png 768w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems-800x300.png 800w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems-900x338.png 900w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems-1024x385.png 1024w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems-1200x451.png 1200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems-1536x577.png 1536w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-file-systems.png 1885w\" sizes=\"(max-width: 1885px) 100vw, 1885px\" \/><\/p>\n<\/div><div class=\"fusion-text fusion-text-34\"><p>From now on we can use our CephFS as we prefer (for example as a File Server).<\/p>\n<\/div><div class=\"fusion-text fusion-text-35\"><h3>6. Monitor Restart<\/h3>\n<p>We may need to restart the monitor, for example when we have latencies that present themselves with this error<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-28853\" src=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-errore.png\" alt=\"ceph errore\" width=\"982\" height=\"71\" srcset=\"https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-errore-200x14.png 200w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-errore-400x29.png 400w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-errore-600x43.png 600w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-errore-768x56.png 768w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-errore-800x58.png 800w, https:\/\/blog.miniserver.it\/wp-content\/uploads\/ceph-errore.png 982w\" sizes=\"(max-width: 982px) 100vw, 982px\" \/><\/p>\n<\/div><div class=\"fusion-text fusion-text-36\"><p>This error means that the Monitors are not synchronized.<\/p>\n<p>A useful command to restart the monitors all at once is the following:<\/p>\n<p>On the machine with cephadm (node 1) and enter the shell<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-21 > .CodeMirror, .fusion-syntax-highlighter-21 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-21 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_21\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_21\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_21\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"1\" data-theme=\"default\" data-mode=\"text\/x-sh\">cephadm shell<\/textarea><\/div><div class=\"fusion-text fusion-text-37\"><p>Then run:<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-22 > .CodeMirror, .fusion-syntax-highlighter-22 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-22 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-22 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-22 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:16px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_22\" style=\"font-size:16px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_22\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_22\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"1\" data-theme=\"default\" data-mode=\"text\/x-sh\">ceph orch restart mon\nScheduled to restart mon.ceph1 on host 'ceph1'\nScheduled to restart mon.ceph2 on host 'ceph2'\nScheduled to restart mon.ceph3 on host 'ceph3'<\/textarea><\/div><\/div><\/div><\/div><\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":11,"featured_media":28870,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[129],"tags":[],"class_list":["post-28866","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-proxmox-ve"],"_links":{"self":[{"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/posts\/28866","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/comments?post=28866"}],"version-history":[{"count":41,"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/posts\/28866\/revisions"}],"predecessor-version":[{"id":28911,"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/posts\/28866\/revisions\/28911"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/media\/28870"}],"wp:attachment":[{"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/media?parent=28866"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/categories?post=28866"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.miniserver.it\/en\/wp-json\/wp\/v2\/tags?post=28866"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}