How to wake FreeBSD 13.2 on usb console cable? (UPS and NUT are used)

[Gridsah]

I have a HPE microserver Gen10 server running FreeBSD13.2 and a APC bk650M2 ch UPS which offers a rj45 to usb cable to connect to my Gen10.

NUT 2.8.0 installed by pkg running on Gen10 works well:

- system poweroff works well when power is outage and battery.charge is lower than battery.charge.low
- upsc <nutdev1> shows the ups status well

After shutting down the server in power outage, Gen10 should be booted while I restore power supply just like my Synology NAS did. But It is not. Gen10 is not booted when I restore power supply. I can only boot it by press power button and the /var/log/messages shows nothing related to usb information.

I turn on the 'pre-wake on lan' on Gen10 BIOS. The WOL works and I can boot my Gen10. I cannot find any keywords like 'usb wake' 'usb boot' on BIOS and just find the power loss status: (when power back machine should be) power on, power off, last status. Besides, I changed the power loss status to power on, but It is not working.

I want the Gen10 being booted when the power back. But I don't know what I should do next.

(What other information should I provide?

 

[Phishfry]

system poweroff works well when power is outage

I changed the power loss status to power on, but It is not working.

"Always on" is usually what it says in the bios.
That is what controls this function.

The only thing I can think of is that it is not actually shutting down but suspending.
When power is re-applied suspend is not re-waking the machine.

What happens if you literally rip the cord and make sure the power is off?
Does the machine automatically start then?

This could be BMC related stuff too. Are you updated on the latest BIOS and BMC firmware?

BMC=Broadband Managment Controller
This acts at a layer underneath the BIOS and can control power.
I do not know if HP MicroServer has BMC but if it does it is worth investigating.

 

[Gridsah]

The only thing I can think of is that it is not actually shutting down but suspending.
When power is re-applied suspend is not re-waking the machine.

I am sure my Gen10 is shutdown, not suspending.
Because I configured SHUTDOWNCMD "/sbin/poweroff" in /usr/local/etc/nut/upsmon.conf and my Gen10 started to do self test while I press the power button (before what I did it was shutdown by upsmon daemon).

What happens if you literally rip the cord and make sure the power is off?
Does the machine automatically start then?

I changed 'power loss status' from 'last status' to 'power on'. It works fine:

1. Gen10 is shutdown by upsmon daemon
2. I pull the power plug (Gen10 to ups)
3. put the Gen10 plug back in ups and Gen10 start to do self test and then boot

But:

1. Gen10 is shutdown by upsmon daemon
2. I pull the power plug (ups to electric supply)
3. put the ups plug back in electric supply and nothing happens on Gen10

This could be BMC related stuff too. Are you updated on the latest BIOS and BMC firmware?

HPE Microserver Gen10 has no BMC (it is called iLO by HP), but Gen10 plus has one.

For now I wrote a script on my router to check if Gen10 is online. If not, wake it by WOL. Although this is also effective, it is a compromise approach. I am still want to find a way to let Gen10 boot from usb signal sent by ups.

 

[Andriy]

Apparently your UPS is either not properly configured or lacks some capabilities. A UPS is supposed to power-cycle a load or power-cycle itself (after a configured period of time) even if utility power returns "too quickly" (before the UPS shuts down).

 

[Gridsah]

Apparently your UPS is either not properly configured or lacks some capabilities. A UPS is supposed to power-cycle a load or power-cycle itself (after a configured period of time) even if utility power returns "too quickly" (before the UPS shuts down).

I have two identical UPS, one for my HPE Microserver Gen 10 and another one for my Synology NAS.

My Synology NAS works well. But Gen10 doesn't. Gen10 just does not booting automatically while the utility power is back, but Synology NAS does. I believe there is no problem with my UPS. There must be some places on FreeBSD system or Gen10 BIOS that I didn't configure correctly.

 

[smithi]

Apparently your UPS is either not properly configured or lacks some capabilities.

More likely the former.

A UPS is supposed to power-cycle a load or power-cycle itself (after a configured period of time) even if utility power returns "too quickly" (before the UPS shuts down).

Yes, and configured periods of time may be relevant here. Perhaps Gridsah will tell us UPS and upsmon settings?

One thing UPS has to guard against, after turning off the load's power due to battery getting below minimum remaining charge and after signaling the load, is to ignore too-short returns of power, as often occur in stormy conditions here.

So there must be a minimum period of time between UPS power return detection and switching on power to loads, that needs to be enough to ensure the load is convinced that it's truly 'off'.

Compared to WoL, very few systems support power on (from ACPI S4 or S5 state) by USB signal. Google e.g.
"wake on usb" bios ...

The USB cable is just for communication between (here) upsmon and the UPS, both for UPS config and it reporting status, including imminent power loss.

Usually the only restart signal is restoration of power, and yes, "always on" or "power on" is the right setting.

And rather than poweroff,
/sbin/shutdown -p now Message to log
may be useful in syslog.

 

[Gridsah]

Yes, and configured periods of time may be relevant here. Perhaps Gridsah will tell us UPS and upsmon settings?

Thanks for your analysis. Below are my configs. These configs works on Linux host with NUT older than v2.8.0 (ASUS B360M gaming plus + i3-8100). And I think these configs should also work on my Gen10 with FreeBSD system.

Besides, I am not sure if these configs are right now.

root@gen10:/usr/local/etc/nut # cat nut.conf | grep -v "^#"

MODE=netserver

root@gen10:/usr/local/etc/nut # cat ups.conf | grep -v "^#"

maxretry = 3

[nutdev1]
    driver = "usbhid-ups"
    port = "auto"
    vendorid = "051D"
    productid = "0002"
    product = "Back-UPS BK650M2-CH FW:294803G -292804G"
    serial = "xxxx"
    vendor = "American Power Conversion"
    ignorelb
    override.battery.runtime.low = -1
    override.battery.charge.low = 60

root@gen10:/usr/local/etc/nut # cat upsd.users | grep -v "^#"

[admin]
    password = adminpwd
    upsmon master

[slave1]
    password  = slave1pwd
    upsmon slave

root@gen10:/usr/local/etc/nut # cat upsmon.conf | grep -v "^#"

MONITOR nutdev1@localhost 1 admin adminpwd master

MINSUPPLIES 1

SHUTDOWNCMD "/sbin/poweroff"

POLLFREQ 5

POLLFREQALERT 5

HOSTSYNC 15

DEADTIME 15

POWERDOWNFLAG /etc/killpower

RBWARNTIME 43200

NOCOMMWARNTIME 300

FINALDELAY 5

 

[Gridsah]

One thing UPS has to guard against, after turning off the load's power due to battery getting below minimum remaining charge and after signaling the load, is to ignore too-short returns of power, as often occur in stormy conditions here.

So there must be a minimum period of time between UPS power return detection and switching on power to loads, that needs to be enough to ensure the load is convinced that it's truly 'off'.

I waited long enough while testing. And there are no harsh conditions such as storms. I pretty much provided an ideal testing environment with cautious.

Compared to WoL, very few systems support power on (from ACPI S4 or S5 state) by USB signal. Google e.g.
"wake on usb" bios ...

The USB cable is just for communication between (here) upsmon and the UPS, both for UPS config and it reporting status, including imminent power loss.

Yes, I have never seen such a configuration in the BIOS about something like 'wake on usb'.

And rather than poweroff,
/sbin/shutdown -p now Message to log
may be useful in syslog.

Where should I put these code? Replace SHUTDOWNCMD "/sbin/poweroff" with /sbin/shutdown -p now Message to log in upsmon.conf ?

 

[Gridsah]

Usually the only restart signal is restoration of power, and yes, "always on" or "power on" is the right setting.

Yes. I unplugged the UPS from the utility power. After FreeBSD shut down due to low UPS battery,

- I plug UPS to utility power back. Nothing happens.
- I unplugged my Gen10 from UPS, and then plug Gen10 to UPS back. Gen10 booted.

"always on" in BIOS works well.

 

[Andriy]

Try to do testing mentioned here:

6. Configuration notes

Specifically, see "Testing shutdowns".

 

[ralphbsz]

Yes. I unplugged the UPS from the utility power. After FreeBSD shut down due to low UPS battery,

- I plug UPS to utility power back. Nothing happens.
- I unplugged my Gen10 from UPS, and then plug Gen10 to UPS back. Gen10 booted.

"always on" in BIOS works well.

I think you found the race condition that is a problem in communication between UPS and computer that I haven't solved yet either (I'm using apcupsd, not nut):

• Utility power fails. UPS starts supplying power.
• UPS battery becomes empty enough to trigger shutdown. UPS signals computer that power will be lost soon. Computer begins shutdown, and completes shutdown.
• While the computer shuts down, the UPS is sitting on a delay timer, for example "shut yourself down in 1 minute after signaling the computer".
• During that delay time, and AFTER the computer has completed shutdown, utility power is restored. The UPS switches back to utility power.
• Because power to the computer was never interrupted, the computer does not reboot. Ever.

This is a classic case of a race condition in a distributed system. The reason is that there is a bit of "state" that is being stored in two places, namely "the UPS will run out of power". It is stored in the UPS, which correctly signals the computer when entering this state. It is also stored in the computer, by virtue of the computer shutting down and waiting for the power interruption to reboot. Unfortunately, that bit of state is updated without coordinating between the two entities involved, because the computer (being shut down) has no means to listen to state updates.

As Andriy said already, the correct fix would be for the UPS to deliver on its promise that power will be interrupted (a promise that is implicit when it signaled that power will be lost. I get the feeling that consumer-grade APC units are not even capable of doing that. My tests of exactly the same situation with a similar model (I think APC calls it ES-550) have found the same problem.

One of the issues that makes testing this difficult / tedious / impossible is that every test cycle takes 10 to 20 minutes. So what I do is this: I don't actually connect the computer to the UPS. Instead I use a small electric heater as a load on the UPS, and also connect a desk lamp to it, as an indicator. I configure the UPS to shut down when the battery reaches about 90% capacity, so I don't have to wait long. Then I simulate the computer power being turned off and on by pulling its plug. Like this I can avoid reboot cycles, and the tests are a little less painful.

 

[Gridsah]

Try to do testing mentioned here:

6. Configuration notes

Specifically, see "Testing shutdowns".

I tested shutdowns and got information below. But I didn't know what these commands actually did:

root@gen10:~ # upsdrvctl -t shutdown
Network UPS Tools - UPS driver controller 2.8.0
*** Testing mode: not calling exec/kill
   0.000000     [D2]
If you're not a NUT core developer, chances are that you're told to enable debugging
to see why a driver isn't working for you. We're sorry for the confusion, but this is
the 'upsdrvctl' wrapper, not the driver you're interested in.

Below you'll find one or more lines starting with 'exec:' followed by an absolute
path to the driver binary and some command line option. This is what the driver
starts and you need to copy and paste that line and append the debug flags to that
line (less the 'exec:' prefix).

Alternately, provide an additional '-d' (lower-case) parameter to 'upsdrvctl' to
pass its current debug level to the launched driver.

   0.000518     [D1] Shutdown UPS: nutdev1
   0.000525     [D2] exec:  /usr/local/libexec/nut/usbhid-ups -a nutdev1 -k

As Andriy said already, the correct fix would be for the UPS to deliver on its promise that power will be interrupted (a promise that is implicit when it signaled that power will be lost. I get the feeling that consumer-grade APC units are not even capable of doing that. My tests of exactly the same situation with a similar model (I think APC calls it ES-550) have found the same problem.

That reminds me something. I found an external circuit board on the main board of my Synology NAS. Even when the NAS is powered off, this small circuit board remains power on. When I bought my UPS, I noticed that Synology specially made adaptations for this UPS. Is this why Synology can work fine (automatically boot when utility power is back) on this UPS?

 

[Gridsah]

I think you found the race condition that is a problem in communication between UPS and computer that I haven't solved yet either (I'm using apcupsd, not nut):

So, maybe I need to unplug utility power to wait my Gen10 poweroff and then wait for enough time to see if my UPS shutdown? If my UPS does, plug the utility power back to see if my Gen10 boot automatically?

 

[VladiBG]

You need UPS with switched output outlets. Your backup ups doesn't have this functionality and cannot turn off and on it's outlet when the power is restored so even if you put your computer on always on / power on in the BIOS settings it cannot detect when the power came back to boot up again. Your option is to use ILO to manually power on the computer when the power is restored.

What are the features of the switched outlet groups on my SMT, SMX, or SRT Smart-UPS? | Schneider Electric Singapore

 

[Gridsah]

You need UPS with switched output outlets. Your backup ups doesn't have this functionality and cannot turn off and on it's outlet when the power is restored so even if you put your computer on always on / power on in the BIOS settings it cannot detect when the power came back to boot up again. Your option is to use ILO to manually power on the computer when the power is restored.

What are the features of the switched outlet groups on my SMT, SMX, or SRT Smart-UPS? | Schneider Electric Singapore

Thanks. From what the URL said, this is indeed the case. Synology works on it because they did extra adaptations.

Besides, HPE microserver Gen10 have no iLO, but Gen10 plus has a optional one.

Now I write WOL script on my router not plugged to UPS and execute it regularly. This works well.

 

[smithi]

Try to do testing mentioned here:

6. Configuration notes

Specifically, see "Testing shutdowns".

These are good docs, far better than when I setup upsd etc ~'09 for our FM community radio station's main server in the Rainbow Region, so named for its thunderstorms.

Near impossible to debug remotely, but I'm finding it hard to believe that special equipment is needed to get this working, though we used Eaton not APC UPS.

Reposting part of config:

root@gen10:/usr/local/etc/nut # cat ups.conf | grep -v "^#"

maxretry = 3

[nutdev1]
driver = "usbhid-ups"
port = "auto"
vendorid = "051D"
productid = "0002"
product = "Back-UPS BK650M2-CH FW:294803G -292804G"
serial = "xxxx"
vendor = "American Power Conversion"
ignorelb
override.battery.runtime.low = -1
override.battery.charge.low = 60

Wondering what 'ignorelb' and the following overrides accomplish?

No shame in using the WoL solution, but it shouldn't be necessary; I still feel there's likely a timing issue here that longer delays between test actions or better settings might resolve.

That said, I'm in no position to test it, so letting it go ...

 

[ralphbsz]

Near impossible to debug remotely, ...

And even in person, it's very tedious to debug. I've been having problems with my apsupsd installation at home (similar UPS model), and I need to find a free afternoon to debug it. But now that it's summer and we don't have frequent power outages, I'm just not motivated to work on it.

 

[Gridsah]

Wondering what 'ignorelb' and the following overrides accomplish?

Some UPS will immediately send a low-charge-state signal to NUT when it detects that the utility power disappears. NUT detects a low-charge-state signal and immediately triggers the shutdown operation.

After setting ignorelb, you also need to set 'battery.charge.low' and 'battery.runtime.low'. These variables define when the low-charge-state signal is triggered. '-1' indicates that this variable is not used.

The shutdown operation is triggered when the value of a variable meets one of the following conditions:

- Battery.charge < battery.charge.low
- Battery.runtime < battery.runtime.low

"override.battery.runtime.low = -1" means that NUT doesn't use 'battery.runtime.low' to determine if the battery is low.

"override.battery.charge.low = 60" means that when the UPS power is lower than 60%, the low-charge-state signal will be triggered, and NUT will shut down after receiving the signal.

 

[Gridsah]

And even in person, it's very tedious to debug. I've been having problems with my apsupsd installation at home (similar UPS model), and I need to find a free afternoon to debug it. But now that it's summer and we don't have frequent power outages, I'm just not motivated to work on it.

Absolutely right. I am working on it because I just purchased it. But now, I don't have much motivation to solve this problem. Anyway, WoL can also be used. ???