<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">This has probably nothing to do with systemd! CEPH finds his osd devices automatically via udev somewhere here: /usr/lib/udev/rules.d/95-ceph-osd.rules. Did you ever try to manually activate the disks like in the rules? Should be something like: for o in {0..3}; do i="\`echo \$o | tr 0123456789 abcdefghij\`" echo "... activating disk /dev/ceph${n}-hd\${i}1 on ceph$n" /usr/sbin/ceph-disk-activate /dev/ceph${n}-hd\${i}1 /usr/sbin/ceph-disk activate-journal /dev/ceph${n}-hd\${i}2 done As this has nothing to do with systemd we should not discuss this here further, but you might contact me directly via mail or go directly to the appropiate Ceph lists. Regards Joachim Am 08.12.2015 um 19:35 schrieb Andrea Annoè: </div> <blockquote cite="mid:8E92587C4632EA40A3928446F7895EFD0D810208F1@x-mbx.iks.local" type="cite"> <meta name="Generator" content="Microsoft Word 14 (filtered medium)"> <style></style> <div class="WordSection1"> Hi,<o:p></o:p> I try to test ceph 9.2 cluster.<o:p></o:p> <o:p> </o:p> My lab have 1 mon and 2 osd with 4 disks each.<o:p></o:p> <o:p> </o:p> Only 1 osd server (with 4 disks) are online.<o:p></o:p> The disks of second osd don’t go up …<o:p></o:p> <o:p> </o:p> Some info about environment:<o:p></o:p> [ceph@OSD1 ~]$ sudo ceph osd tree<o:p></o:p> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY<o:p></o:p> -1 8.00000 root default<o:p></o:p> -4 8.00000 datacenter dc1<o:p></o:p> -5 8.00000 room room1<o:p></o:p> -6 8.00000 row row1<o:p></o:p> -7 4.00000 rack rack1<o:p></o:p> -2 4.00000 host OSD1<o:p></o:p> 0 1.00000 osd.0 up 1.00000 1.00000<o:p></o:p> 1 1.00000 osd.1 up 1.00000 1.00000<o:p></o:p> 2 1.00000 osd.2 up 1.00000 1.00000<o:p></o:p> 3 1.00000 osd.3 up 1.00000 1.00000<o:p></o:p> -8 4.00000 rack rack2<o:p></o:p> -3 4.00000 host OSD2<o:p></o:p> 4 1.00000 osd.4 down 1.00000 1.00000<o:p></o:p> 5 1.00000 osd.5 down 1.00000 1.00000<o:p></o:p> 6 1.00000 osd.6 down 1.00000 1.00000<o:p></o:p> 7 1.00000 osd.7 down 1.00000 1.00000<o:p></o:p> <o:p> </o:p> [ceph@OSD1 ceph-deploy]$ sudo ceph osd dump<o:p></o:p> epoch 411<o:p></o:p> fsid d17520de-0d1e-495b-90dc-f7044f7f165f<o:p></o:p> created 2015-11-14 06:56:36.017672<o:p></o:p> modified 2015-12-08 09:48:47.685050<o:p></o:p> flags nodown<o:p></o:p> pool 0 'rbd' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 53 flags hashpspool min_write_recency_for_promote 1 stripe_width 0<o:p></o:p> max_osd 9<o:p></o:p> osd.0 up in weight 1 up_from 394 up_thru 104 down_at 393 last_clean_interval [388,392) 192.168.64.129:6800/4599 192.168.62.129:6800/4599 192.168.62.129:6801/4599 192.168.64.129:6801/4599 exists,up 499a3624-b2ba-455d-b35a-31d628e1a353<o:p></o:p> osd.1 up in weight 1 up_from 396 up_thru 136 down_at 395 last_clean_interval [390,392) 192.168.64.129:6802/4718 192.168.62.129:6802/4718 192.168.62.129:6803/4718 192.168.64.129:6803/4718 exists,up d7933117-0056-4c3c-ac63-2ad300495e3f<o:p></o:p> osd.2 up in weight 1 up_from 400 up_thru 136 down_at 399 last_clean_interval [392,392) 192.168.64.129:6806/5109 192.168.62.129:6806/5109 192.168.62.129:6807/5109 192.168.64.129:6807/5109 exists,up 7d820897-8d49-4142-8c58-feda8bb04749<o:p></o:p> osd.3 up in weight 1 up_from 398 up_thru 136 down_at 397 last_clean_interval [386,392) 192.168.64.129:6804/4963 192.168.62.129:6804/4963 192.168.62.129:6805/4963 192.168.64.129:6805/4963 exists,up 96270d9d-ed95-40be-9ae4-7bf66aedd4d8<o:p></o:p> osd.4 down out weight 0 up_from 34 up_thru 53 down_at 58 last_clean_interval [0,0) 192.168.64.130:6800/3615 192.168.64.130:6801/3615 192.168.64.130:6802/3615 192.168.64.130:6803/3615 autoout,exists 6364d590-62fb-4348-b8fe-19b59cd2ceb3<o:p></o:p> osd.5 down out weight 0 up_from 145 up_thru 151 down_at 203 last_clean_interval [39,54) 192.168.64.130:6800/2784 192.168.62.130:6800/2784 192.168.62.130:6801/2784 192.168.64.130:6801/2784 autoout,exists aa51cdcc-ca9c-436b-b9fc-7bddaef3226d<o:p></o:p> osd.6 down out weight 0 up_from 44 up_thru 53 down_at 58 last_clean_interval [0,0) 192.168.64.130:6808/4975 192.168.64.130:6809/4975 192.168.64.130:6810/4975 192.168.64.130:6811/4975 autoout,exists 36672496-3346-446a-a617-94c8596e1da2<o:p></o:p> osd.7 down out weight 0 up_from 155 up_thru 161 down_at 204 last_clean_interval [49,54) 192.168.64.130:6800/2434 192.168.62.130:6800/2434 192.168.62.130:6801/2434 192.168.64.130:6801/2434 autoout,exists 775065fa-8fa8-48ce-a4cc-b034a720fe93<o:p></o:p> <o:p> </o:p> All UUID are correct (the down is appear after upgrade).<o:p></o:p> Now I’m not able to create some osd with ceph-deploy. <o:p></o:p> I have cancel all osd disk from cluster and deploy from zero.<o:p></o:p> Now I have some problem for service osd when start with systemd.<o:p></o:p> <o:p> </o:p> [ceph@OSD2 ~]$ sudo systemctl start <a class="moz-txt-link-abbreviated" href="mailto:ceph-osd@4.service"> ceph-osd@4.service</a><o:p></o:p> [ceph@OSD2 ~]$ sudo systemctl status <a class="moz-txt-link-abbreviated" href="mailto:ceph-osd@4.service"> ceph-osd@4.service</a> -l<o:p></o:p> <a class="moz-txt-link-abbreviated" href="mailto:ceph-osd@4.service">ceph-osd@4.service</a> - Ceph object storage daemon<o:p></o:p> Loaded: loaded (/etc/systemd/system/ceph.target.wants/ceph-osd@4.service)<o:p></o:p> Active: active (running) since Tue 2015-12-08 10:31:38 PST; 9s ago<o:p></o:p> Process: 6542 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)<o:p></o:p> Main PID: 6599 (ceph-osd)<o:p></o:p> CGroup: /system.slice/system-ceph\<a class="moz-txt-link-abbreviated" href="mailto:x2dosd.slice/ceph-osd@4.service">x2dosd.slice/ceph-osd@4.service</a><o:p></o:p> ââ6599 /usr/bin/ceph-osd --cluster ceph --id 4 --setuser ceph --setgroup ceph -f<o:p></o:p> <o:p> </o:p> Dec 08 10:31:38 OSD2.local ceph-osd-prestart.sh[6542]: create-or-move updated item name 'osd.4' weight 0.0098 at location {host=OSD2,root=default} to crush map<o:p></o:p> Dec 08 10:31:38 OSD2.local systemd[1]: Started Ceph object storage daemon.<o:p></o:p> Dec 08 10:31:38 OSD2.local ceph-osd[6599]: starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4 /dev/sdb1<o:p></o:p> Dec 08 10:31:38 OSD2.local ceph-osd[6599]: 2015-12-08 10:31:38.702018 7f0a555fc900 -1 osd.4 411 log_to_monitors {default=true}<o:p></o:p> [ceph@OSD2 ~]$<o:p></o:p> [ceph@OSD2 ~]$ sudo ceph osd tree<o:p></o:p> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY<o:p></o:p> -1 4.03918 root default<o:p></o:p> -4 4.03918 datacenter dc1<o:p></o:p> -5 4.03918 room room1<o:p></o:p> -6 4.03918 row row1<o:p></o:p> -7 4.00000 rack rack1<o:p></o:p> -2 4.00000 host OSD1<o:p></o:p> 0 1.00000 osd.0 up 1.00000 1.00000<o:p></o:p> 1 1.00000 osd.1 up 1.00000 1.00000<o:p></o:p> 2 1.00000 osd.2 up 1.00000 1.00000<o:p></o:p> 3 1.00000 osd.3 up 1.00000 1.00000<o:p></o:p> -8 0.03918 rack rack2<o:p></o:p> -3 0.03918 host OSD2<o:p></o:p> 4 0.00980 osd.4 down 0 1.00000<o:p></o:p> 5 0.00980 osd.5 down 0 1.00000<o:p></o:p> 6 0.00980 osd.6 down 0 1.00000<o:p></o:p> 7 0.00980 osd.7 down 0 1.00000<o:p></o:p> <o:p> </o:p> Script for start osd is:<o:p></o:p> [root@OSD2 ceph-4]# cat /etc/systemd/system/ceph.target.wants/ceph-osd\@4.service<o:p></o:p> [Unit]<o:p></o:p> Description=Ceph object storage daemon<o:p></o:p> After=network-online.target local-fs.target<o:p></o:p> Wants=network-online.target local-fs.target<o:p></o:p> PartOf=ceph.target<o:p></o:p> <o:p> </o:p> [Service]<o:p></o:p> LimitNOFILE=1048576<o:p></o:p> LimitNPROC=1048576<o:p></o:p> EnvironmentFile=-/etc/sysconfig/ceph<o:p></o:p> Environment=CLUSTER=ceph<o:p></o:p> ExecStart=/usr/bin/ceph-osd --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph -f<o:p></o:p> ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i<o:p></o:p> ExecReload=/bin/kill -HUP $MAINPID<o:p></o:p> <o:p> </o:p> [Install]<o:p></o:p> WantedBy=ceph.target<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Some idea???<o:p></o:p> I’m crazy for this situation!!<o:p></o:p> <o:p> </o:p> Thanks.<o:p></o:p> AndreaIT<o:p></o:p> <o:p> </o:p> </div> <fieldset class="mimeAttachmentHeader"></fieldset> <pre wrap="">_______________________________________________ systemd-devel mailing list <a class="moz-txt-link-abbreviated" href="mailto:systemd-devel@lists.freedesktop.org">systemd-devel@lists.freedesktop.org</a> <a class="moz-txt-link-freetext" href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel">http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a> </pre> </blockquote> <pre class="moz-signature" cols="90">-- Joachim von Thadden Lead Technical Architect SEVEN PRINCIPLES AG Ernst-Dietrich-Platz 2 40882 Ratingen Mobil: +49 162 261 64 66 Tel: +49 2102 557 100 Fax: +49 2102 557 101 E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:joachim.von-thadden@7p-group.com">joachim.von-thadden@7p-group.com</a> Web: <a class="moz-txt-link-abbreviated" href="http://www.7p-group.com">www.7p-group.com</a> ________________________________ Aufsichtsrat: Prof. Dr. h.c. Hans Albert Aukes Vorstandsvorsitzender: Joseph Kronfli Handelsregister: HRB 30660 | USt-ID-Nr.: DE197820124 | Steuer-Nr.: 218/5734/1640 Sitz der Gesellschaft: Köln | Registriergericht: Amtsgericht Köln Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten Adressaten bestimmt. Wenn Sie nicht der vorgesehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, beachten Sie bitte, dass jede Form der Veröffentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-Mail unzulässig ist. Wir bitten Sie sofort den Absender zu informieren und die E-Mail zu löschen. The information contained in this e-mail is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you are not the intended recipient, any form of disclosure, reproduction, distribution or any action taken or refrained from in reliance on it, is prohibited and may be unlawful. Please notify the sender immediately and destroy this e-mail. </pre> </body> </html>