Python 100% CPU

[bhboyle]

Checklist

• I made sure that there are no existing issues or discussions - open issues,closed issues,open discussion, closed discussion - which may answer my question or that I could contribute my information to.
• I have read the Known Issues Wiki Page and my question / issue is not listed.
• I have provided the version of the software and the operating system in my post.
• If I have an issue with genmon not communicating with the controller, I have read and followed the serial troubleshooting guide.
• I have included the generator make and model in my submission.

Version

Genmon Version V1.19.03 Generator is a Generac 9KW

Have you submitted your log files from the About page? Note that outbound email must be working for logs to be submitted.

I have not submitted logs

What is/are your question(s)?

Hey folks I am not totaly sure this is even an issue. Everyting is working and data is flowing as exected and has been for six moths or so. My question is that on my instanse, (Debian VM on Proxmox) one of the two CPUs is pegged ay 100% continuously and stays there all the time even right after a reboot. The item that is pgging the CPU at 100% is a python somthing...

I dont know if this an issue or not as i dont recall if it was always this way.

Here is the top line of Top.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
429 root 20 0 860924 47372 15216 S 100.0 2.4 12:43.58 python

Just looking for some input.

CHeers.

Additional information

No response

[skipfire]

You could stop GenMon and see if that process goes away. ./genmon/startgenmon.sh stop

[bhboyle]

Thank you for that. I did as you suggested and it did bring the cpu usage down to almost zero. unpon restart, right back up to 100%

I installed this 6-8 months ago, is there an update procdure I should consider?

[skipfire]

You can update through the genmon UI on the about page, also I believe from the command line it is ./genmon/genmonmaint.sh -u

I do recommend following jgyates' suggestion first about getting the full command, it would help tell which of the multiple genmon threads may be causing it.

As a VM it is possible it's coming from the outside. If one of the other instances is consuming a lot of that CPU it may be blocking your genmon instance from having a command intended for that CPU from being processed in a timely fashion. Are you able to shut down all the other things on that virtual host temporarily and restart genmon so it's got a clean start, and then see if it spikes again? That can help determine if it is inside or outside your genmon VM.

Is the 100% CPU causing a problem or are you just seeing that it is pegged?

[jgyates]

you can also look at the process number that is causing the high load and then type this:

 ps -axo pid,cmd | grep [p]ython

It should print out the process number and the full python command line for each process.

Of course it is possible that the VM is just over loaded depedning on the VM settings and the host system requirements. If that is the case then you can enable the "Optimize for slower CPUs" setting on the Settings page in the web interface. This will reduce the frequency serial polling of modbus. This setting was introduced to keep slower raspberry pis (pi zero) from a high CPU load.

[bhboyle]

Here is the result of the above command.

8047 /home/bboyle/genmon/genenv/bin/python /home/bboyle/genmon/genmon.py -c /etc/genmon/
8058 /home/bboyle/genmon/genenv/bin/python /home/bboyle/genmon/genserv.py -c /etc/genmon/
8059 /home/bboyle/genmon/genenv/bin/python /home/bboyle/genmon/addon/genmqtt.py -c /etc/genmon/

The VM is a two cpu Debian instance with two gigs of ram.

Like I was saying, I dont recall it running like this before. I think it is only the last couple of days.

[jgyates]

In your first post the process ID was 429. This does not match the process IDs in the output of the above command. Have you restarted genmon? If so then run top again to get the PID of the process at 100%.

[bhboyle]

Yes I did stop in based on the suggestion at the start of this conversation to confirm if it was indeed Genmon.
Here is the update Top command output:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8047 root 20 0 860928 52492 15248 S 97.7 2.6 72:03.66 python

Here is theoutput of ps -axo pid,cmd | grep [p]ython
bboyle@Genmon:~/genmon$ ps -axo pid,cmd | grep [p]ython
8047 /home/bboyle/genmon/genenv/bin/python /home/bboyle/genmon/genmon.py -c /etc/genmon/
8058 /home/bboyle/genmon/genenv/bin/python /home/bboyle/genmon/genserv.py -c /etc/genmon/
8059 /home/bboyle/genmon/genenv/bin/python /home/bboyle/genmon/addon/genmqtt.py -c /etc/genmon/

Sorry for the confusion.

[jgyates]

ok. Genmon is process number 8047. See if the "Optimize for slower CPUs" setting makes a difference.

[bhboyle]

So I turned on the Optimize for slower CPU. and saw no change. So I restarted Genmon using the stop and start commands.

I waited two minutes to see if it would settle but it is still running at 100%
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
43578 root 20 0 860932 47720 15296 S 100.3 2.4 0:58.01 python

I see a couple of options to try here. I have a VM backup from the weekend that I could restore and see if that makes any differece. The second option is I see that I am behind a version or two so I could do an update.

Thoughts?

[bhboyle]

OH?! That sounds like a great idea. Is there a doc you can point me to ti impliment that?

[skipfire]

Switch is a root level element. After doing this when you visit the ESP32 device's webpage you will have a new line with a toggle. Did you assemble your own hardware or is this a purchased item (such as PintSize OpenGenSet)? Cable issues are a possibility when it's self-assembled, especially if using individual header jumper wires.

switch:
  - platform: restart
    name: "$display_name Restart"

[bhboyle]

Also... Im getting this in the Home Assistant logs from the ESP32 Bridge...

WARNING genmon-bridge @ 172.17.17.22: Connection error occurred: genmon-bridge @ 172.17.17.22: The connection dropped immediately after encrypted hello; Try enabling encryption on the device or turning off encryption on the client (ESPHome Logs 2024.12.4).

.22 is the Bridge .16 in the Genmon Server

[skipfire]

Probably just another symptom of the same thing. Hopefully rebooting it helps.

[bhboyle]

@skipfire

Thank you I will give that a try. I think I may have to go and have look at the ESP as it gives this error when trying to upload the updated firmware.

INFO Uploading /data/build/genmon-bridge/.pioenvs/genmon-bridge/firmware.bin (957152 bytes)
ERROR Error receiving acknowledge version: [Errno 104] Connection reset by peer

It is so odd as it has been running like a champ for many months now in all sorts of weather.

It is a DIY device but as I said it has been super stable. We got some crazy snow here the last few days perhaps some got inside the housing of the Genny.

Thank you so much for helping me to diagnose this issue. I will let you know what I find when I get inside the housing.