mirror of
https://github.com/fatedier/frp.git
synced 2026-05-15 16:15:49 -06:00
[GH-ISSUE #3493] FRP Client reconnect fails because the same process is holding lock on proxy name/port #2792
Labels
No labels
In Progress
WIP
WaitingForInfo
bug
doc
duplicate
easy
enhancement
future
help wanted
invalid
lifecycle/stale
need-issue-template
need-usage-help
no plan
proposal
pull-request
question
todo
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/frp#2792
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @pierre-pretorius on GitHub (Jun 22, 2023).
Original GitHub issue: https://github.com/fatedier/frp/issues/3493
Bug Description
Please look at the logs below. There are comments in between which explains the problem. A workaround to avoid the problem is to set a schedule to restart the FRPC process every day.
frpc Version
0.44.0
frps Version
0.45.0
System Architecture
linux/arm (client) and linux/amd64 (server)
Configurations
Server
[common]
bind_port = *******
authentication_method = token
token = ***************
dashboard_port = ****
dashboard_user = *********
dashboard_pwd = *********
log_file = /var/log/frps.log
log_level = warn
Client
[web]
type = tcp
local_port = 80
remote_port = 30000
use_encryption = true
use_compression = true
Logs
Client
The client runs successfully for many weeks, but then at some point starts saying "port unavailable" when it tries to reconnect.
I then force the port open on the server without doing anything on the client. The client error message then changes to "proxy name already in use":
I then restart the FRP client and it connects successfully. This seems to indicate the frpc client process had an open connection even though it was trying to reconnect.
Server
After running successfully a long time, the server eventually says port is unavailable when the client reconnects:
Then once we force the port open on the server:
Steps to reproduce
We can't easily reproduce it. We run many thousands of clients and there are always at least 10+ stuck in this "Reconnect while proxy or port in use" mode which doesn't resolve by itself.
Affected area
@fatedier commented on GitHub (Jun 25, 2023):
What specific operation was performed?
Could you please provide more detailed logs when frpc is disconnected?
@pierre-pretorius commented on GitHub (Jun 26, 2023):
We run this command on the FRPS server, using the IP address of the FRPC client:
I will investigate again and post a longer log.
@fatedier commented on GitHub (Jun 27, 2023):
By the way, if you can upgrade to the latest version v0.50.0, it would be better.
@fatedier commented on GitHub (Jun 27, 2023):
In addition, it is important to know why the connection between frpc and frps is being disconnected. Is it due to a machine malfunction on frps, network fluctuations, or other reasons? To track these issues, more detailed logs are needed.
@pierre-pretorius commented on GitHub (Jul 14, 2023):
We upgrade to v0.50.0 with config below and found the root cause.
After the server is running for a day or so, about 0.1% of the FRP clients are continuously trying to connect but fail:
On the server we can see that an nginx processes is still connected to the port 30000 of FRPS even though the FRP client is disconnected and is trying to re-establish connection. On server:
If we kill the nginx process, then the netstat connection above disappears and the FRP client stops looping and reconnects successfully. This means it's somehow possible for a connection to the remote_port on the server to stop the FRP client from connecting again after it drops. In our case the nginx proxy_pass directive is making the connection.
We found two possible workarounds (both work):
a) Configure your nginx to periodically shutdown and recycle workers. This has the unfortunate side effect of dropping existing web sockets. FRP clients will be in reconnect loop for a while until nginx worker is recycled.
b) Create a script that reads the FRPS log and detects a FRP client is continuously unable to connect, then it uses tcpkill to close any connection to the remote_port on the server. In our case this closes the nginx proxy_pass connection:
We currently use workaround (b) and abandoned workaround (a).
@fatedier commented on GitHub (Jul 14, 2023):
@pierre-pretorius Great.
However, I am still a bit confused. Normally, the TIME_WAIT connection that you find through netstat should not affect frps from re-listening on port 30000. Can you filter not only the connections of nginx but also all connections on port 30000?
@github-actions[bot] commented on GitHub (Aug 21, 2023):
Issues go stale after 30d of inactivity. Stale issues rot after an additional 7d of inactivity and eventually close.