Shutting down machine via SSH does not return exit value

clamport asked:

We had an old version of Ubuntu (12.04) that used sysV. Due to LTS running out for it, we needed to switch to 16.04.

One of the major changes was the change from SysV to systemd. We noticed that when we made this change, all of our code that did any sort of ssh root@thisismyhostname "shutdown -h now" would hang, when previously it would initiate the shutdown, return EXIT_SUCCESS and close the connection. With systemd however, it seems that the connections remained open. I found the fix to that, which was that we needed to install and enable PAM and had to change from shutdown -h now to systemctl poweroff so that systemd would iterate through the connections and close them prior to shutting down, the functionality we want.

The problem that now happens is that probably 1/20th of the time, there seems to be a race condition that happens where the shutdown will get enough CPU to take the system down, which causes SSH to return 255 (the return code for connection closed) instead of returning EXIT_SUCCESS for the shutdown command run from SSH.

We (unfortunately) have more than one place in code that calls shutdown, and I feel like checking for either EXIT_SUCCESS or SSH’s 255 is somewhat of a hacky workaround. Does anyone know of a way to get Ubuntu 16.04 with systemd to return EXIT_SUCCESS after calling systemctl poweroff?

My answer:


Consider using systemctl‘s ability to send its own commands via ssh to remote hosts.

From the man page:

       -H, --host=
           Execute the operation remotely. Specify a hostname, or a username
           and hostname separated by "@", to connect to. The hostname may
           optionally be suffixed by a container name, separated by ":", which
           connects directly to a specific container on the specified host.
           This will use SSH to talk to the remote machine manager instance.
           Container names may be enumerated with machinectl -H HOST.

It works a lot like this:

# ssh-copy-id -i ~/.ssh/id_ed25519.pub 172.25.50.100

...

# systemctl -H 172.25.50.100 status
‚óŹ 172.25.50.100
    State: degraded
     Jobs: 0 queued
   Failed: 1 units
    Since: Thu 2017-03-02 01:56:07 EST; 17h ago
   CGroup: /
# systemctl -H 172.25.50.100 poweroff
# echo $?
0

View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.