<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Testo fumetto Carattere";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
span.StileMessaggioDiPostaElettronica17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.TestofumettoCarattere
{mso-style-name:"Testo fumetto Carattere";
mso-style-priority:99;
mso-style-link:"Testo fumetto";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 2.0cm 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=IT link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Hi,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>I try to test ceph 9.2 cluster.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>My lab have 1 mon and 2 osd with 4 disks each.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Only 1 osd server (with 4 disks) are online.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>The disks of second osd don’t go up …<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Some info about environment:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[ceph@OSD1 ~]$ sudo ceph osd tree<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -1 8.00000 root default<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -4 8.00000 datacenter dc1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -5 8.00000 room room1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -6 8.00000 row row1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -7 4.00000 rack rack1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -2 4.00000 host OSD1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 0 1.00000 osd.0 up 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 1 1.00000 osd.1 up 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 2 1.00000 osd.2 up 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 3 1.00000 osd.3 up 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -8 4.00000 rack rack2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -3 4.00000 host OSD2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 4 1.00000 osd.4 down 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 5 1.00000 osd.5 down 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 6 1.00000 osd.6 down 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 7 1.00000 osd.7 down 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>[ceph@OSD1 ceph-deploy]$ sudo ceph osd dump<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>epoch 411<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>fsid d17520de-0d1e-495b-90dc-f7044f7f165f<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>created 2015-11-14 06:56:36.017672<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>modified 2015-12-08 09:48:47.685050<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>flags nodown<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>pool 0 'rbd' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 53 flags hashpspool min_write_recency_for_promote 1 stripe_width 0<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>max_osd 9<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>osd.0 up in weight 1 up_from 394 up_thru 104 down_at 393 last_clean_interval [388,392) 192.168.64.129:6800/4599 192.168.62.129:6800/4599 192.168.62.129:6801/4599 192.168.64.129:6801/4599 exists,up 499a3624-b2ba-455d-b35a-31d628e1a353<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>osd.1 up in weight 1 up_from 396 up_thru 136 down_at 395 last_clean_interval [390,392) 192.168.64.129:6802/4718 192.168.62.129:6802/4718 192.168.62.129:6803/4718 192.168.64.129:6803/4718 exists,up d7933117-0056-4c3c-ac63-2ad300495e3f<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>osd.2 up in weight 1 up_from 400 up_thru 136 down_at 399 last_clean_interval [392,392) 192.168.64.129:6806/5109 192.168.62.129:6806/5109 192.168.62.129:6807/5109 192.168.64.129:6807/5109 exists,up 7d820897-8d49-4142-8c58-feda8bb04749<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>osd.3 up in weight 1 up_from 398 up_thru 136 down_at 397 last_clean_interval [386,392) 192.168.64.129:6804/4963 192.168.62.129:6804/4963 192.168.62.129:6805/4963 192.168.64.129:6805/4963 exists,up 96270d9d-ed95-40be-9ae4-7bf66aedd4d8<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>osd.4 down out weight 0 up_from 34 up_thru 53 down_at 58 last_clean_interval [0,0) 192.168.64.130:6800/3615 192.168.64.130:6801/3615 192.168.64.130:6802/3615 192.168.64.130:6803/3615 autoout,exists 6364d590-62fb-4348-b8fe-19b59cd2ceb3<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>osd.5 down out weight 0 up_from 145 up_thru 151 down_at 203 last_clean_interval [39,54) 192.168.64.130:6800/2784 192.168.62.130:6800/2784 192.168.62.130:6801/2784 192.168.64.130:6801/2784 autoout,exists aa51cdcc-ca9c-436b-b9fc-7bddaef3226d<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>osd.6 down out weight 0 up_from 44 up_thru 53 down_at 58 last_clean_interval [0,0) 192.168.64.130:6808/4975 192.168.64.130:6809/4975 192.168.64.130:6810/4975 192.168.64.130:6811/4975 autoout,exists 36672496-3346-446a-a617-94c8596e1da2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>osd.7 down out weight 0 up_from 155 up_thru 161 down_at 204 last_clean_interval [49,54) 192.168.64.130:6800/2434 192.168.62.130:6800/2434 192.168.62.130:6801/2434 192.168.64.130:6801/2434 autoout,exists 775065fa-8fa8-48ce-a4cc-b034a720fe93<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>All UUID are correct (the down is appear after upgrade).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Now I’m not able to create some osd with ceph-deploy. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>I have cancel all osd disk from cluster and deploy from zero.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Now I have some problem for service osd when start with systemd.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>[ceph@OSD2 ~]$ sudo systemctl start ceph-osd@4.service<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[ceph@OSD2 ~]$ sudo systemctl status ceph-osd@4.service -l<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>ceph-osd@4.service - Ceph object storage daemon<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> Loaded: loaded (/etc/systemd/system/ceph.target.wants/ceph-osd@4.service)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> Active: active (running) since Tue 2015-12-08 10:31:38 PST; 9s ago<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> Process: 6542 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> Main PID: 6599 (ceph-osd)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@4.service<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> ââ6599 /usr/bin/ceph-osd --cluster ceph --id 4 --setuser ceph --setgroup ceph -f<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Dec 08 10:31:38 OSD2.local ceph-osd-prestart.sh[6542]: create-or-move updated item name 'osd.4' weight 0.0098 at location {host=OSD2,root=default} to crush map<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Dec 08 10:31:38 OSD2.local systemd[1]: Started Ceph object storage daemon.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Dec 08 10:31:38 OSD2.local ceph-osd[6599]: starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4 /dev/sdb1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Dec 08 10:31:38 OSD2.local ceph-osd[6599]: 2015-12-08 10:31:38.702018 7f0a555fc900 -1 osd.4 411 log_to_monitors {default=true}<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[ceph@OSD2 ~]$<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> [ceph@OSD2 ~]$ sudo ceph osd tree<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -1 4.03918 root default<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -4 4.03918 datacenter dc1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -5 4.03918 room room1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -6 4.03918 row row1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -7 4.00000 rack rack1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -2 4.00000 host OSD1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 0 1.00000 osd.0 up 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 1 1.00000 osd.1 up 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 2 1.00000 osd.2 up 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 3 1.00000 osd.3 up 1.00000 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -8 0.03918 rack rack2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> -3 0.03918 host OSD2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 4 0.00980 osd.4 down 0 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 5 0.00980 osd.5 down 0 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 6 0.00980 osd.6 down 0 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> 7 0.00980 osd.7 down 0 1.00000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Script for start osd is:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[root@OSD2 ceph-4]# cat /etc/systemd/system/ceph.target.wants/ceph-osd\@4.service<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[Unit]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Description=Ceph object storage daemon<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>After=network-online.target local-fs.target<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Wants=network-online.target local-fs.target<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>PartOf=ceph.target<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>[Service]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>LimitNOFILE=1048576<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>LimitNPROC=1048576<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>EnvironmentFile=-/etc/sysconfig/ceph<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Environment=CLUSTER=ceph<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>ExecStart=/usr/bin/ceph-osd --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph -f<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>ExecReload=/bin/kill -HUP $MAINPID<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>[Install]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>WantedBy=ceph.target<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Some idea???<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>I’m crazy for this situation!!<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Thanks.<o:p></o:p></span></p><p class=MsoNormal><b><span lang=EN-US style='color:#1F497D;mso-fareast-language:IT'>AndreaIT</span></b><span lang=EN-US style='font-size:9.0pt;color:#1F497D;mso-fareast-language:IT'><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>