github-starred/frp
2
0
Fork 0
mirror of https://github.com/fatedier/frp.git synced 2026-05-15 16:15:49 -06:00
Code Issues 40 Projects Packages Wiki Activity Actions 1

[GH-ISSUE #5090] read from workConn for sudp error: EOF; all visitors/proxies "stuck" #3994

New issue
Closed
opened 2026-05-05 14:32:22 -06:00 by gitea-mirror · 8 comments
gitea-mirror commented 2026-05-05 14:32:22 -06:00
Owner
Copy link

Originally created by @ByteCrunch on GitHub (Dec 9, 2025).
Original GitHub issue: https://github.com/fatedier/frp/issues/5090

Bug Description

We have issues for multiple frpc instances at different locations using different internet connections.

The symptoms are that all the proxies/visitors are broken after the following output in the frpc logs:

[visitor/stcp.go:112] [c3265bd915ac1b76] [prdkeycloak] get newVisitorConnRespMsg error: EOF
[proxy/sudp.go:143] [c3265bd915ac1b76] [dns] read from workConn for sudp error: EOF
[proxy/sudp.go:160] [c3265bd915ac1b76] [dns] writer goroutine for sudp work connection closed

In the frps logs at the same time, all proxies are closed:

proxy/proxy.go:204] [5d25de26cc7b4706] [prdeksd] get a user connection [117.240.119.149:54188]
[proxy/proxy.go:204] [5d25de26cc7b4706] [prdkeycloak] get a user connection [117.240.119.149:54188]
[proxy/proxy.go:115] [c3265bd915ac1b76] [zzdynprt-50084] proxy closing
[proxy/proxy.go:115] [c3265bd915ac1b76] [zzdynprt-49783] proxy closing
[...]
[proxy/proxy.go:201] [c3265bd915ac1b76] [zzdynprt-50084] listener is closed: listener closed
[...]

In the frps metrics the client count is decreased by 1 in this case.

Restarting frpc always recovers the connections until the same symptoms will show up again after some hours or days.

frpc never recovers, there are also no further log lines or attempts to reconnect to the server. Basically frpc is stuck until restarted.

As a mitigation we have implemented a wrapper which, when frpc log lines like "read from workConn for sudp error: EOF" appear, will check the general internet connectivity and a service reachable via frp itself and then will restart the frpc process.

We always see that there is no general internet outage as at the same time. Just the services provided via frp are not reachable.

frpc Version

v0.65.0

frps Version

v0.64.0

System Architecture

linux/amd64

Configurations

frpc.toml

serverAddr = "x.x.x.x"
serverPort = 9000
transport.tcpMux = true # multiplex connections

loginFailExit = false

# webserver needed for hot reload feature
webServer.addr = "127.0.0.1"
webServer.port = 7400
webServer.user = "admin"
webServer.password = "admin"

auth.method = "token"
auth.token = "..."

log.level = "debug"

[[proxies]]
name = "dns"
type = "sudp"
secretKey = "..."
localIP = "172.16.153.1"
localPort = 53

[[visitors]]
name = "prdkeycloak"
type = "stcp"
secretKey = "..."
serverName = "prdkeycloak"
bindAddr = "172.16.199.1"
bindPort = 5443

[...]

Logs

2025-11-27 11:13:11 [info     ] performing periodic checks                                                       filename=main.py func_name=periodic_check lineno=78
2025-11-27 11:13:12 [info     ] system health: {'cpu_usage': '0.0%', 'memory_usage': '17.5%', 'disk_usage': '13.6%', 'open_socket_count': 254} filename=main.py func_name=periodic_check lineno=79
2025-11-27 11:13:15 [info     ] connectivity check: internet (https://www.google.com) succeeded.               filename=main.py func_name=check_connectivity lineno=47
2025-11-27 11:13:20 [warning  ] connectivity check: visitor service (https://172.16.199.1:5443) failed: <urlopen error _ssl.c:993: The handshake operation timed out> filename=main.py func_name=check_connectivity lineno=53
2025-11-27 11:13:20 [info     ] [FRPC] 2025-11-27 11:13:15.310 [D] [visitor/stcp.go:86] [c3265bd915ac1b76] [prdkeycloak] get a new stcp user connection
 filename=main.py func_name=handle_frpc_output lineno=94
2025-11-27 11:13:21 [info     ] [FRPC] 2025-11-27 11:13:21.081 [W] [visitor/stcp.go:112] [c3265bd915ac1b76] [prdkeycloak] get newVisitorConnRespMsg error: EOF
2025-11-27 11:13:21.082 [W] [proxy/sudp.go:143] [c3265bd915ac1b76] [dns] read from workConn for sudp error: EOF
2025-11-27 11:13:21.082 [I] [proxy/sudp.go:160] [c3265bd915ac1b76] [dns] writer goroutine for sudp work connection closed
 filename=main.py func_name=handle_frpc_output lineno=94
2025-11-27 11:13:21 [warning  ] Potential issue detected in frpc logs, performing connectivity checks            filename=main.py func_name=handle_frpc_output lineno=132
2025-11-27 11:13:22 [info     ] system health: {'cpu_usage': '0.0%', 'memory_usage': '17.5%', 'disk_usage': '13.6%', 'open_socket_count': 253} filename=main.py func_name=handle_frpc_output lineno=133
2025-11-27 11:13:22 [warning  ] connectivity check: visitor service (https://172.16.199.1:5443) failed: <urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1010)> filename=main.py func_name=check_connectivity lineno=53
2025-11-27 11:13:22 [info     ] restarting frpc process                                                          filename=main.py func_name=restart_frpc lineno=186
2025-11-27 11:13:22 [error    ] frpc process exited: (CalledProcessError(1, 'frpc process terminated due to failed connectivity check'),) filename=main.py func_name=start lineno=162
2025-11-27 11:13:22 [info     ] Restarting frpc in 5 seconds                                                     filename=main.py func_name=restart_message_and_wait lineno=179
2025-11-27 11:13:27 [info     ] Starting SecretsFS                                                               filename=main.py func_name=start lineno=147
2025-11-27 11:13:27 [info     ] Starting frpc                                                                    filename=main.py func_name=start lineno=153
2025-11-27 11:13:27 [info     ] frpc process running                                                             pid=69 filename=main.py func_name=start lineno=157
2025-11-27 11:13:27 [info     ] [FRPC] 2025-11-27 11:13:27.262 [I] [sub/root.go:149] start frpc service for config file [/opt/frp/etc/frpc.toml]
 filename=main.py func_name=handle_frpc_output lineno=94
2025-11-27 11:13:27 [info     ] [FRPC] 2025-11-27 11:13:27.263 [I] [client/service.go:325] try to connect to server...
 filename=main.py func_name=handle_frpc_output lineno=94
2025-11-27 11:13:27 [info     ] frpc is trying to connect to its FRP server, checking internet connectivity      filename=main.py func_name=handle_frpc_output lineno=99
2025-11-27 11:13:30 [info     ] connectivity check: internet (https://www.google.com) succeeded.               filename=main.py func_name=check_connectivity lineno=47
2025-11-27 11:13:30 [info     ] [FRPC] 2025-11-27 11:13:27.263 [I] [client/service.go:204] admin server listen on 127.0.0.1:7400
2025-11-27 11:13:27.572 [I] [client/service.go:317] [80eca8832f0346c8] login to server success, get run id [80eca8832f0346c8]

Steps to reproduce

Unfortunately we haven't found a way to reproduce the error.
It occurs sporadically after hours or days of using frp tunneling without issues. But it only happens when actual traffic is handled by frp - not when frpc is only connected to a frps without actually using any proxies/visitors.

Affected area

  • Docs
  • Installation
  • Performance and Scalability
  • Security
  • User Experience
  • Test and Release
  • Developer Infrastructure
  • Client Plugin
  • Server Plugin
  • Extensions
  • Others
Originally created by @ByteCrunch on GitHub (Dec 9, 2025). Original GitHub issue: https://github.com/fatedier/frp/issues/5090 ### Bug Description We have issues for multiple frpc instances at different locations using different internet connections. The symptoms are that all the proxies/visitors are broken after the following output in the frpc logs: ``` [visitor/stcp.go:112] [c3265bd915ac1b76] [prdkeycloak] get newVisitorConnRespMsg error: EOF [proxy/sudp.go:143] [c3265bd915ac1b76] [dns] read from workConn for sudp error: EOF [proxy/sudp.go:160] [c3265bd915ac1b76] [dns] writer goroutine for sudp work connection closed ``` In the frps logs at the same time, all proxies are closed: ``` proxy/proxy.go:204] [5d25de26cc7b4706] [prdeksd] get a user connection [117.240.119.149:54188] [proxy/proxy.go:204] [5d25de26cc7b4706] [prdkeycloak] get a user connection [117.240.119.149:54188] [proxy/proxy.go:115] [c3265bd915ac1b76] [zzdynprt-50084] proxy closing [proxy/proxy.go:115] [c3265bd915ac1b76] [zzdynprt-49783] proxy closing [...] [proxy/proxy.go:201] [c3265bd915ac1b76] [zzdynprt-50084] listener is closed: listener closed [...] ``` In the frps metrics the client count is decreased by 1 in this case. Restarting frpc always recovers the connections until the same symptoms will show up again after some hours or days. frpc never recovers, there are also no further log lines or attempts to reconnect to the server. Basically frpc is stuck until restarted. As a mitigation we have implemented a wrapper which, when frpc log lines like "read from workConn for sudp error: EOF" appear, will check the general internet connectivity and a service reachable via frp itself and then will restart the frpc process. We always see that there is no general internet outage as at the same time. Just the services provided via frp are not reachable. ### frpc Version v0.65.0 ### frps Version v0.64.0 ### System Architecture linux/amd64 ### Configurations frpc.toml ``` serverAddr = "x.x.x.x" serverPort = 9000 transport.tcpMux = true # multiplex connections loginFailExit = false # webserver needed for hot reload feature webServer.addr = "127.0.0.1" webServer.port = 7400 webServer.user = "admin" webServer.password = "admin" auth.method = "token" auth.token = "..." log.level = "debug" [[proxies]] name = "dns" type = "sudp" secretKey = "..." localIP = "172.16.153.1" localPort = 53 [[visitors]] name = "prdkeycloak" type = "stcp" secretKey = "..." serverName = "prdkeycloak" bindAddr = "172.16.199.1" bindPort = 5443 [...] ``` ### Logs ``` 2025-11-27 11:13:11 [info ] performing periodic checks filename=main.py func_name=periodic_check lineno=78 2025-11-27 11:13:12 [info ] system health: {'cpu_usage': '0.0%', 'memory_usage': '17.5%', 'disk_usage': '13.6%', 'open_socket_count': 254} filename=main.py func_name=periodic_check lineno=79 2025-11-27 11:13:15 [info ] connectivity check: internet (https://www.google.com) succeeded. filename=main.py func_name=check_connectivity lineno=47 2025-11-27 11:13:20 [warning ] connectivity check: visitor service (https://172.16.199.1:5443) failed: <urlopen error _ssl.c:993: The handshake operation timed out> filename=main.py func_name=check_connectivity lineno=53 2025-11-27 11:13:20 [info ] [FRPC] 2025-11-27 11:13:15.310 [D] [visitor/stcp.go:86] [c3265bd915ac1b76] [prdkeycloak] get a new stcp user connection filename=main.py func_name=handle_frpc_output lineno=94 2025-11-27 11:13:21 [info ] [FRPC] 2025-11-27 11:13:21.081 [W] [visitor/stcp.go:112] [c3265bd915ac1b76] [prdkeycloak] get newVisitorConnRespMsg error: EOF 2025-11-27 11:13:21.082 [W] [proxy/sudp.go:143] [c3265bd915ac1b76] [dns] read from workConn for sudp error: EOF 2025-11-27 11:13:21.082 [I] [proxy/sudp.go:160] [c3265bd915ac1b76] [dns] writer goroutine for sudp work connection closed filename=main.py func_name=handle_frpc_output lineno=94 2025-11-27 11:13:21 [warning ] Potential issue detected in frpc logs, performing connectivity checks filename=main.py func_name=handle_frpc_output lineno=132 2025-11-27 11:13:22 [info ] system health: {'cpu_usage': '0.0%', 'memory_usage': '17.5%', 'disk_usage': '13.6%', 'open_socket_count': 253} filename=main.py func_name=handle_frpc_output lineno=133 2025-11-27 11:13:22 [warning ] connectivity check: visitor service (https://172.16.199.1:5443) failed: <urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1010)> filename=main.py func_name=check_connectivity lineno=53 2025-11-27 11:13:22 [info ] restarting frpc process filename=main.py func_name=restart_frpc lineno=186 2025-11-27 11:13:22 [error ] frpc process exited: (CalledProcessError(1, 'frpc process terminated due to failed connectivity check'),) filename=main.py func_name=start lineno=162 2025-11-27 11:13:22 [info ] Restarting frpc in 5 seconds filename=main.py func_name=restart_message_and_wait lineno=179 2025-11-27 11:13:27 [info ] Starting SecretsFS filename=main.py func_name=start lineno=147 2025-11-27 11:13:27 [info ] Starting frpc filename=main.py func_name=start lineno=153 2025-11-27 11:13:27 [info ] frpc process running pid=69 filename=main.py func_name=start lineno=157 2025-11-27 11:13:27 [info ] [FRPC] 2025-11-27 11:13:27.262 [I] [sub/root.go:149] start frpc service for config file [/opt/frp/etc/frpc.toml] filename=main.py func_name=handle_frpc_output lineno=94 2025-11-27 11:13:27 [info ] [FRPC] 2025-11-27 11:13:27.263 [I] [client/service.go:325] try to connect to server... filename=main.py func_name=handle_frpc_output lineno=94 2025-11-27 11:13:27 [info ] frpc is trying to connect to its FRP server, checking internet connectivity filename=main.py func_name=handle_frpc_output lineno=99 2025-11-27 11:13:30 [info ] connectivity check: internet (https://www.google.com) succeeded. filename=main.py func_name=check_connectivity lineno=47 2025-11-27 11:13:30 [info ] [FRPC] 2025-11-27 11:13:27.263 [I] [client/service.go:204] admin server listen on 127.0.0.1:7400 2025-11-27 11:13:27.572 [I] [client/service.go:317] [80eca8832f0346c8] login to server success, get run id [80eca8832f0346c8] ``` ### Steps to reproduce Unfortunately we haven't found a way to reproduce the error. It occurs sporadically after hours or days of using frp tunneling without issues. But it only happens when actual traffic is handled by frp - not when frpc is only connected to a frps without actually using any proxies/visitors. ### Affected area - [ ] Docs - [ ] Installation - [x] Performance and Scalability - [ ] Security - [ ] User Experience - [ ] Test and Release - [ ] Developer Infrastructure - [ ] Client Plugin - [ ] Server Plugin - [ ] Extensions - [ ] Others
gitea-mirror 2026-05-05 14:32:22 -06:00
gitea-mirror commented 2026-05-05 14:32:25 -06:00
Author
Owner
Copy link

@fatedier commented on GitHub (Dec 9, 2025):

You can also upgrade frps to version v0.65 and observe it for a while. I remember that version updated the underlying yamux library and fixed a possibly related issue.

<!-- gh-comment-id:3631837355 --> @fatedier commented on GitHub (Dec 9, 2025): You can also upgrade **frps** to version **v0.65** and observe it for a while. I remember that version updated the underlying **yamux** library and fixed a possibly related issue.
gitea-mirror commented 2026-05-05 14:32:26 -06:00
Author
Owner
Copy link

@kangnan15 commented on GitHub (Dec 31, 2025):

You can also upgrade frps to version v0.65 and observe it for a while. I remember that version updated the underlying yamux library and fixed a possibly related issue.

When restarting frps, frpc will encounter the following error message
read from workConn for udp error: EOF
writer goroutine for udp work connection closed

<!-- gh-comment-id:3702696449 --> @kangnan15 commented on GitHub (Dec 31, 2025): > You can also upgrade **frps** to version **v0.65** and observe it for a while. I remember that version updated the underlying **yamux** library and fixed a possibly related issue. When restarting frps, frpc will encounter the following error message read from workConn for udp error: EOF writer goroutine for udp work connection closed
gitea-mirror commented 2026-05-05 14:32:27 -06:00
Author
Owner
Copy link

@ByteCrunch commented on GitHub (Jan 2, 2026):

Happy new year,

we have updated the first instances of frps to v0.65 resulting in all components being on the latest version and we still see occurences of this issue.
In order to gain more insights, we have compiled frpc with debug symbols and added GDB backtracing to our wrapper which now executes the following when the issue occurs before restarting the frpc process:

source /opt/go/src/runtime/runtime-gdb.py
info goroutines
goroutine all bt
quit

GDB log when issue occured:
gdb.txt

In parallel we were trying to reproduce the issue by introducing artificial bad network conditions with toxyproxy (cutting the connection to frps, introducing packet loss, high latency, limit bandwidth) for a small frpc/frps setup with 2 simple services. However, in all these scenarios, frpc sucessfully reconnected after disabling the toxyproxy "toxics" and we did not observe the behavior of frpc being "stuck".

<!-- gh-comment-id:3704769587 --> @ByteCrunch commented on GitHub (Jan 2, 2026): Happy new year, we have updated the first instances of frps to v0.65 resulting in all components being on the latest version and we still see occurences of this issue. In order to gain more insights, we have compiled frpc with debug symbols and added GDB backtracing to our wrapper which now executes the following when the issue occurs before restarting the frpc process: ``` source /opt/go/src/runtime/runtime-gdb.py info goroutines goroutine all bt quit ``` GDB log when issue occured: [gdb.txt](https://github.com/user-attachments/files/24405669/gdb.txt) In parallel we were trying to reproduce the issue by introducing artificial bad network conditions with toxyproxy (cutting the connection to frps, introducing packet loss, high latency, limit bandwidth) for a small frpc/frps setup with 2 simple services. However, in all these scenarios, frpc sucessfully reconnected after disabling the toxyproxy "toxics" and we did not observe the behavior of frpc being "stuck".
gitea-mirror commented 2026-05-05 14:32:28 -06:00
Author
Owner
Copy link

@fatedier commented on GitHub (Jan 4, 2026):

Thanks a lot for providing the detailed debugging information — it’s very helpful for narrowing down the root cause. Based on what you shared, PR https://github.com/fatedier/frp/pull/5083
looks like it may fix this issue. I’m planning to release a new version including this change in the near future.

<!-- gh-comment-id:3707584391 --> @fatedier commented on GitHub (Jan 4, 2026): Thanks a lot for providing the detailed debugging information — it’s very helpful for narrowing down the root cause. Based on what you shared, PR https://github.com/fatedier/frp/pull/5083 looks like it may fix this issue. I’m planning to release a new version including this change in the near future.
gitea-mirror commented 2026-05-05 14:32:29 -06:00
Author
Owner
Copy link

@fatedier commented on GitHub (Jan 4, 2026):

You can try v0.66.0 now.

<!-- gh-comment-id:3708109920 --> @fatedier commented on GitHub (Jan 4, 2026): You can try v0.66.0 now.
gitea-mirror commented 2026-05-05 14:32:29 -06:00
Author
Owner
Copy link

@ByteCrunch commented on GitHub (Jan 7, 2026):

Thanks a lot for the quick release!
We started deployment of the new frpc / frps versions which will take some time to be rolled out and will provide feedback when we have more experience with these.

<!-- gh-comment-id:3717818278 --> @ByteCrunch commented on GitHub (Jan 7, 2026): Thanks a lot for the quick release! We started deployment of the new frpc / frps versions which will take some time to be rolled out and will provide feedback when we have more experience with these.
gitea-mirror commented 2026-05-05 14:32:30 -06:00
Author
Owner
Copy link

@ByteCrunch commented on GitHub (Jan 15, 2026):

Just to give you some interim feedback:
We rolled out v0.66.0 for a couple of systems and so far we don't observe the freezing behavior when issue with the internet connection occur. frpc is able to gracefully reconnect when the network conditions are healthy again. So it is really promising that our issue is fixed. We will continue with the roll out and further observe the frpc behavior.

<!-- gh-comment-id:3755625413 --> @ByteCrunch commented on GitHub (Jan 15, 2026): Just to give you some interim feedback: We rolled out v0.66.0 for a couple of systems and so far we don't observe the freezing behavior when issue with the internet connection occur. frpc is able to gracefully reconnect when the network conditions are healthy again. So it is really promising that our issue is fixed. We will continue with the roll out and further observe the frpc behavior.
gitea-mirror commented 2026-05-05 14:32:31 -06:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jan 30, 2026):

Issues go stale after 14d of inactivity. Stale issues rot after an additional 3d of inactivity and eventually close.

<!-- gh-comment-id:3821177147 --> @github-actions[bot] commented on GitHub (Jan 30, 2026): Issues go stale after 14d of inactivity. Stale issues rot after an additional 3d of inactivity and eventually close.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
dev
new
master
copilot/review-pull-request-5198
v0.68.1
v0.68.0
v0.67.0
v0.66.0
v0.65.0
v0.64.0
v0.63.0
v0.62.1
v0.62.0
v0.61.2
v0.61.1
v0.61.0
v0.60.0
v0.59.0
v0.58.1
v0.58.0
v0.57.0
v0.56.0
v0.55.1
v0.55.0
v0.54.0
v0.53.2
v0.53.1
v0.53.0
v0.52.3
v0.52.2
v0.52.1
v0.52.0
v0.51.3
v0.51.2
v0.51.1
v0.51.0
v0.50.0
v0.49.0
v0.48.0
v0.47.0
v0.46.1
v0.46.0
v0.45.0
v0.44.0
v0.43.0
v0.42.0
v0.41.0
v0.40.0
v0.39.1
v0.39.0
v0.38.0
v0.37.1
v0.37.0
v0.36.2
v0.36.1
v0.36.0
v0.35.1
v0.35.0
v0.34.3
v0.34.2
v0.34.1
v0.34.0
v0.33.0
v0.32.1
v0.32.0
v0.31.2
v0.31.1
v0.31.0
v0.30.0
v0.29.1
v0.29.0
v0.28.2
v0.28.1
v0.28.0
v0.27.1
v0.27.0
v0.26.0
v0.25.3
v0.25.2
v0.25.1
v0.25.0
v0.24.1
v0.24.0
v0.23.3
v0.23.2
v0.23.1
v0.23.0
v0.22.0
v0.21.0
v0.20.0
v0.19.1
v0.19.0
v0.18.0
v0.17.0
v0.16.1
v0.16.0
v0.15.1
v0.15.0
v0.14.1
v0.14.0
v0.13.0
v0.12.0
v0.11.0
v0.10.0
v0.9.3
v0.9.2
v0.9.1
v0.9.0
v0.8.1
v0.8.0
v0.7.0
v0.6.0
v0.5.0
v0.3.0
v0.2.0
v0.1.0
Labels
Clear labels   
In Progress

   
WIP

   
WaitingForInfo

   
bug

   
doc

   
duplicate

   
easy

   
enhancement

   
future

   
help wanted

   
invalid

   
lifecycle/stale

   
need-issue-template

   
need-usage-help

   
no plan

   
proposal

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
todo
No labels
In Progress
WIP
WaitingForInfo
bug
doc
duplicate
easy
enhancement
future
help wanted
invalid
lifecycle/stale
need-issue-template
need-usage-help
no plan
proposal
pull-request
question
todo
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/frp#3994
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?