How to Fix 96% CPU Usage Caused by Swoole Port Conflicts on Ubuntu
1. Background
While maintaining an Ubuntu server, I noticed that the CPU usage suddenly increased to an unusually high level.
The server was equipped with an 8-core CPU. Under normal circumstances, CPU usage remained relatively low. However, during one inspection, the CPU usage had reached 96%.
At the same time, the server monitoring dashboard showed network traffic of approximately 127 Kb/s. There was no obvious spike in network activity.
Since multiple websites were running on this server, including projects built with ThinkPHP and Swoole, I needed to identify which process was responsible for the abnormal CPU consumption.
The investigation eventually revealed the root cause: Two different websites were running Swoole services that attempted to listen on the same IP address and port.
After correcting the port configuration, CPU usage dropped from 96% to just 2%.
This article documents the troubleshooting process and the solution.
2. Check Server Resource Usage
The first step was to use the Linux top command to inspect the server's current resource usage:
topThe process list provides useful information about CPU and memory consumption.
During this investigation, several processes caught my attention:
| Process | Description |
|---|---|
| swoole: manager process | Swoole manager process |
| swoole: hot update process | Swoole hot-update process |
| swoole: http server | Swoole HTTP server process |
| think swoole restart | ThinkPHP Swoole restart command |
| supervisord | Supervisor process manager |
| redis-server | Redis service |
| mysqld | MySQL database service |
The Swoole-related processes deserved further investigation.
However, it is important to understand that Swoole uses a multi-process architecture. Under normal circumstances, it runs multiple processes, such as Manager, Worker, and Task Worker processes.
Therefore, having multiple Swoole processes does not necessarily indicate a problem.
What really matters is whether there are duplicate service instances, repeated restart attempts, or multiple websites using the same listening port.
Several Swoole-related processes appeared in the process list, including the Manager process, HTTP server process, and hot-update process. These processes were selected for further investigation.
3. Investigate Swoole Processes
To check whether Swoole services were being started repeatedly, I used the following command:
ps -eo pid,ppid,lstart,etime,%cpu,%mem,cmd | grep -E 'swoole restart|swoole: manager|swoole: hot update|swoole: http server' | grep -v grep
This command displays the following information:
PID: Process ID.
PPID: Parent process ID.
lstart: Process start time.
etime: Total process running time.
%CPU: CPU usage.
%MEM: Memory usage.
CMD: Process startup command.
By examining the process start times, parent-child relationships, and startup commands, you can determine whether duplicate processes or abnormal restart behavior exist.
If Supervisor is used to manage Swoole services, it is also worth checking its configuration for duplicate startup tasks:
sudo grep -RniE 'swoole|think.*restart|autostart|autorestart' /etc/supervisor /www/server/panel 2>/dev/null | head -100
4. Identify the Root Cause: Two Websites Using the Same Swoole Listening Port
After further investigation, the problem was finally identified.
Two different websites were running their own Swoole services on the same server. However, both services were configured to listen on the same IP address and port.
In other words, two independent Swoole services were attempting to use the same network endpoint.
This configuration can cause port-binding conflicts, startup failures, or repeated restart attempts, depending on how the Swoole services are configured and managed.
If Supervisor or another process manager is responsible for keeping the services running, it may repeatedly attempt to restart a service when startup fails.
This can result in additional resource consumption and abnormal CPU usage.
Therefore, the issue was not caused by excessive website traffic. It was related to a conflict between the listening configurations of two Swoole services.
5. Solution: Assign a Different Port to Each Website
Once the root cause was identified, the solution was straightforward:
Each website must use a unique listening port for its Swoole service.
For example:
| Website | Swoole Listening Address | Port |
|---|---|---|
| Website A | 0.0.0.0 | 9501 |
| Website B | 0.0.0.0 | 9502 |
The actual port numbers can be adjusted according to your server configuration. The important thing is to ensure that independent services do not unintentionally use the same IP address and port combination.
After modifying the Swoole configuration, the Nginx reverse proxy configuration must also be updated so that each domain forwards requests to the correct Swoole service.
For example:
Website A:
location / {
proxy_pass http://127.0.0.1:9501;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
Website B:
location / {
proxy_pass http://127.0.0.1:9502;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
Note: These Nginx configurations are examples of reverse proxy settings. You may need to adjust them according to your actual project requirements, especially if WebSocket connections, HTTPS, or additional request headers are involved.
After making the changes, start or reload the corresponding Swoole services and verify that both websites are accessible.
6. Verify the Listening Ports
Use the ss command to check whether the two Swoole services are listening on their respective ports:
sudo ss -lntp | grep -E ':9501|:9502'
Under normal circumstances, the output should show two separate listening ports.
You can also use the following command to view all TCP listening ports on the server:
sudo ss -lntpThis allows you to inspect the listening addresses, port numbers, and associated processes.
7. Final Results
After adjusting the port configuration, I checked the server's resource usage again.
The results were as follows:
| Item | Before | After |
|---|---|---|
| CPU Cores | 8 | 8 |
| CPU Usage | 96% | 2% |
| Swoole Listening Configuration | Both websites used the same endpoint | Each website used a separate port |
| Website Status | Service conflict | Operating normally |
<font color="green">The CPU usage dropped from 96% to just 2%, and both websites returned to normal operation.</font>
This confirmed that the abnormal CPU usage was related to the conflicting Swoole listening configurations.
8. Lessons Learned
This troubleshooting experience highlights several important points.
1. Do Not Assume High CPU Usage Is Caused by Website Traffic
When server CPU usage reaches 90% or higher, it does not necessarily mean that website traffic is too high.
Abnormal resource consumption can also be caused by background processes, scheduled tasks, repeated service restarts, or incorrect process management configurations.
Start by identifying the processes consuming CPU resources using commands such as top and ps.
2. Plan Ports Carefully When Hosting Multiple Swoole Applications
If multiple Swoole projects are deployed on the same server, each project should have its own listening port.
For example:
CRMEB project: 9501
Hyperf project: 9502
Other Swoole projects: 9503
Avoid unintentionally configuring multiple independent services to listen on the same IP address and port.
3. Check Supervisor for Duplicate Startup Configurations
If Supervisor is used to manage Swoole services, make sure each website has the correct startup command and process management configuration.
Avoid configuring multiple Supervisor programs to manage the same Swoole instance.
4. Remember to Update Nginx After Changing Ports
When the Swoole listening port changes, the corresponding Nginx reverse proxy destination must also be updated.
Otherwise, the Swoole service may be running normally while the website remains inaccessible.
9. Conclusion
At first, the server's CPU usage reaching 96% looked like a typical server load problem.
However, by examining system processes and Swoole services step by step, I eventually discovered that two websites were using the same listening endpoint.
After assigning a separate port to each service, CPU usage quickly returned to 2%.
When deploying multiple ThinkPHP, CRMEB, or Hyperf applications using Swoole on the same server, it is important to pay attention not only to CPU, memory, and network traffic, but also to port allocation and process management configurations.
A small configuration conflict can sometimes cause unexpectedly high resource consumption.
Comments
Post a Comment