github-starred/frp
2
0
Fork 0
mirror of https://github.com/fatedier/frp.git synced 2026-05-15 16:15:49 -06:00
Code Issues 40 Projects Packages Wiki Activity Actions 1

[GH-ISSUE #2895] [Feature Request] 用golang v1.16.7以上版本编译frpc会随机出现短时间cpu利用率100%问题 #2307

New issue
Closed
opened 2026-05-05 13:29:14 -06:00 by gitea-mirror · 11 comments
gitea-mirror commented 2026-05-05 13:29:14 -06:00
Owner
Copy link

Originally created by @gmd20 on GitHub (Apr 15, 2022).
Original GitHub issue: https://github.com/fatedier/frp/issues/2895

Describe the feature request

有个服务器上发现frpc会不定时出现cpu100%的问题，持续时间比较短，可能几秒或者几十秒后自动恢复正常。
感觉和 frps的连接的稳定性有关系，可能连接断开重连过就会出问题？

我一开始是用golang v1.18编译的，出现问题后搜索到timer相关的一个bug了(https://github.com/golang/go/issues/51654 ).
后面把golang降级回v1.16.6重新编译frpc，同样的运行环境就没有再出现问题了。

这个bug官方还没有修复，不知道能否在frpc代码这边做一下规避了，不然是支持不了golang v1.17和v1.18

下面这个是出现问题时的top和perf命令的输出：

出现问题时top命令看cpu利用率单核100%
PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND^[[m
20875 20853 root     S     698m 18.4   3 24.9 frpc -c /etc/frpc.ini
 6234  2864 root     R    57088  1.4   0 24.9 perf record -p 20875 sleep 5

Load average: 0.22 0.18 0.15 1/233 6275
PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND^[[m
20875 20853 root     S     698m 18.4   3 22.5 frpc -c /etc/frpc.ini
 6234  2864 root     S    59408  1.5   0  0.8 perf record -p 20875 sleep 5

下面是2次出现问题时，用perf 分析frpc进程的cpu利用率
Samples: 17K of event 'cycles:ppp', Event count (approx.): 10951031887
Overhead  Command  Shared Object      Symbol
  31.93%  frpc     frpc               [.] runtime.stealWork
  13.16%  frpc     frpc               [.] runtime.netpoll
   5.55%  frpc     frpc               [.] runtime.findrunnable
   4.47%  frpc     [kernel.kallsyms]  [k] sysret_check
   4.09%  frpc     [vdso]             [.] __vdso_clock_gettime
   3.99%  frpc     frpc               [.] runtime.checkTimers
   3.88%  frpc     frpc               [.] runtime.epollwait.abi0
   2.80%  frpc     [kernel.kallsyms]  [k] _raw_spin_lock_irqsave
   2.52%  frpc     frpc               [.] runtime.checkTimersNoP
   2.20%  frpc     [kernel.kallsyms]  [k] system_call_after_swapgs
   2.17%  frpc     [kernel.kallsyms]  [k] fget_light
   2.13%  frpc     [kernel.kallsyms]  [k] _raw_spin_unlock_irqrestore
   2.00%  frpc     frpc               [.] runtime.pidleget
   1.95%  frpc     frpc               [.] runtime.unlock2
   1.93%  frpc     frpc               [.] runtime.nanotime1.abi0
   1.88%  frpc     frpc               [.] runtime.lock2
   1.63%  frpc     [kernel.kallsyms]  [k] sys_epoll_wait
   1.63%  frpc     [kernel.kallsyms]  [k] ep_poll
   1.44%  frpc     frpc               [.] runtime.pidleput
   1.18%  frpc     [kernel.kallsyms]  [k] fput
   1.16%  frpc     [kernel.kallsyms]  [k] sys_epoll_pwait
   0.96%  frpc     [kernel.kallsyms]  [k] system_call
   0.89%  frpc     frpc               [.] runtime.checkRunqsNoP
   0.56%  frpc     frpc               [.] runtime.releasep
   0.43%  frpc     frpc               [.] runtime.wirep
   0.37%  frpc     frpc               [.] runtime.(*mcache).prepareForSweep
   0.28%  frpc     [kernel.kallsyms]  [k] system_call_fastpath
   0.25%  frpc     frpc               [.] runtime.acquirep
   0.22%  frpc     frpc               [.] runtime.checkIdleGCNoP

Samples: 20K of event 'cycles:ppp', Event count (approx.): 12786384439
Overhead  Command  Shared Object      Symbol
  32.70%  frpc     frpc               [.] runtime.stealWork
  12.51%  frpc     frpc               [.] runtime.netpoll
   5.36%  frpc     frpc               [.] runtime.findrunnable
   4.51%  frpc     [kernel.kallsyms]  [k] sysret_check
   4.30%  frpc     frpc               [.] runtime.epollwait.abi0
   4.08%  frpc     frpc               [.] runtime.checkTimers
   4.01%  frpc     [vdso]             [.] __vdso_clock_gettime
   3.03%  frpc     [kernel.kallsyms]  [k] _raw_spin_lock_irqsave
   2.41%  frpc     frpc               [.] runtime.checkTimersNoP
   2.34%  frpc     [kernel.kallsyms]  [k] system_call_after_swapgs
   2.15%  frpc     [kernel.kallsyms]  [k] _raw_spin_unlock_irqrestore
   2.09%  frpc     [kernel.kallsyms]  [k] fget_light
   2.04%  frpc     frpc               [.] runtime.unlock2
   1.83%  frpc     [kernel.kallsyms]  [k] sys_epoll_wait
   1.78%  frpc     frpc               [.] runtime.pidleget
   1.71%  frpc     frpc               [.] runtime.nanotime1.abi0
   1.52%  frpc     frpc               [.] runtime.lock2
   1.52%  frpc     [kernel.kallsyms]  [k] ep_poll
   1.35%  frpc     [kernel.kallsyms]  [k] sys_epoll_pwait
   1.34%  frpc     frpc               [.] runtime.pidleput
   1.02%  frpc     [kernel.kallsyms]  [k] fput
   0.96%  frpc     [kernel.kallsyms]  [k] system_call
   0.93%  frpc     frpc               [.] runtime.checkRunqsNoP
   0.64%  frpc     frpc               [.] runtime.releasep
   0.42%  frpc     frpc               [.] runtime.(*mcache).prepareForSweep
   0.32%  frpc     frpc               [.] runtime.wirep
   0.29%  frpc     [kernel.kallsyms]  [k] system_call_fastpath
   0.24%  frpc     frpc               [.] runtime.acquirep
   0.20%  frpc     frpc               [.] runtime.checkIdleGCNoP
   0.20%  frpc     frpc               [.] runtime.updateTimerPMask
   0.15%  frpc     frpc               [.] runtime.runqget
   0.07%  frpc     [kernel.kallsyms]  [k] __x86_indirect_thunk_rax
   0.05%  frpc     [kernel.kallsyms]  [k] update_wall_time
   0.04%  frpc     [kernel.kallsyms]  [k] _raw_qspin_lock
   0.04%  frpc     [kernel.kallsyms]  [k] __netif_receive_skb_core
   0.03%  frpc     [kernel.kallsyms]  [k] task_tick_fair
   0.03%  frpc     [kernel.kallsyms]  [k] rcu_check_callbacks

Describe alternatives you've considered

No response

Affected area

  • Docs
  • Installation
  • Performance and Scalability
  • Security
  • User Experience
  • Test and Release
  • Developer Infrastructure
  • Client Plugin
  • Server Plugin
  • Extensions
  • Others
Originally created by @gmd20 on GitHub (Apr 15, 2022). Original GitHub issue: https://github.com/fatedier/frp/issues/2895 ### Describe the feature request 有个服务器上发现frpc会不定时出现cpu100%的问题，持续时间比较短，可能几秒或者几十秒后自动恢复正常。 感觉和 frps的连接的稳定性有关系，可能连接断开重连过就会出问题？ 我一开始是用golang v1.18编译的，出现问题后搜索到timer相关的一个bug了(https://github.com/golang/go/issues/51654 ). 后面把golang降级回v1.16.6重新编译frpc，同样的运行环境就没有再出现问题了。 这个bug官方还没有修复，不知道能否在frpc代码这边做一下规避了，不然是支持不了golang v1.17和v1.18 下面这个是出现问题时的top和perf命令的输出： ```text 出现问题时top命令看cpu利用率单核100% PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND^[[m 20875 20853 root S 698m 18.4 3 24.9 frpc -c /etc/frpc.ini 6234 2864 root R 57088 1.4 0 24.9 perf record -p 20875 sleep 5 Load average: 0.22 0.18 0.15 1/233 6275 PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND^[[m 20875 20853 root S 698m 18.4 3 22.5 frpc -c /etc/frpc.ini 6234 2864 root S 59408 1.5 0 0.8 perf record -p 20875 sleep 5 下面是2次出现问题时，用perf 分析frpc进程的cpu利用率 Samples: 17K of event 'cycles:ppp', Event count (approx.): 10951031887 Overhead Command Shared Object Symbol 31.93% frpc frpc [.] runtime.stealWork 13.16% frpc frpc [.] runtime.netpoll 5.55% frpc frpc [.] runtime.findrunnable 4.47% frpc [kernel.kallsyms] [k] sysret_check 4.09% frpc [vdso] [.] __vdso_clock_gettime 3.99% frpc frpc [.] runtime.checkTimers 3.88% frpc frpc [.] runtime.epollwait.abi0 2.80% frpc [kernel.kallsyms] [k] _raw_spin_lock_irqsave 2.52% frpc frpc [.] runtime.checkTimersNoP 2.20% frpc [kernel.kallsyms] [k] system_call_after_swapgs 2.17% frpc [kernel.kallsyms] [k] fget_light 2.13% frpc [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore 2.00% frpc frpc [.] runtime.pidleget 1.95% frpc frpc [.] runtime.unlock2 1.93% frpc frpc [.] runtime.nanotime1.abi0 1.88% frpc frpc [.] runtime.lock2 1.63% frpc [kernel.kallsyms] [k] sys_epoll_wait 1.63% frpc [kernel.kallsyms] [k] ep_poll 1.44% frpc frpc [.] runtime.pidleput 1.18% frpc [kernel.kallsyms] [k] fput 1.16% frpc [kernel.kallsyms] [k] sys_epoll_pwait 0.96% frpc [kernel.kallsyms] [k] system_call 0.89% frpc frpc [.] runtime.checkRunqsNoP 0.56% frpc frpc [.] runtime.releasep 0.43% frpc frpc [.] runtime.wirep 0.37% frpc frpc [.] runtime.(*mcache).prepareForSweep 0.28% frpc [kernel.kallsyms] [k] system_call_fastpath 0.25% frpc frpc [.] runtime.acquirep 0.22% frpc frpc [.] runtime.checkIdleGCNoP Samples: 20K of event 'cycles:ppp', Event count (approx.): 12786384439 Overhead Command Shared Object Symbol 32.70% frpc frpc [.] runtime.stealWork 12.51% frpc frpc [.] runtime.netpoll 5.36% frpc frpc [.] runtime.findrunnable 4.51% frpc [kernel.kallsyms] [k] sysret_check 4.30% frpc frpc [.] runtime.epollwait.abi0 4.08% frpc frpc [.] runtime.checkTimers 4.01% frpc [vdso] [.] __vdso_clock_gettime 3.03% frpc [kernel.kallsyms] [k] _raw_spin_lock_irqsave 2.41% frpc frpc [.] runtime.checkTimersNoP 2.34% frpc [kernel.kallsyms] [k] system_call_after_swapgs 2.15% frpc [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore 2.09% frpc [kernel.kallsyms] [k] fget_light 2.04% frpc frpc [.] runtime.unlock2 1.83% frpc [kernel.kallsyms] [k] sys_epoll_wait 1.78% frpc frpc [.] runtime.pidleget 1.71% frpc frpc [.] runtime.nanotime1.abi0 1.52% frpc frpc [.] runtime.lock2 1.52% frpc [kernel.kallsyms] [k] ep_poll 1.35% frpc [kernel.kallsyms] [k] sys_epoll_pwait 1.34% frpc frpc [.] runtime.pidleput 1.02% frpc [kernel.kallsyms] [k] fput 0.96% frpc [kernel.kallsyms] [k] system_call 0.93% frpc frpc [.] runtime.checkRunqsNoP 0.64% frpc frpc [.] runtime.releasep 0.42% frpc frpc [.] runtime.(*mcache).prepareForSweep 0.32% frpc frpc [.] runtime.wirep 0.29% frpc [kernel.kallsyms] [k] system_call_fastpath 0.24% frpc frpc [.] runtime.acquirep 0.20% frpc frpc [.] runtime.checkIdleGCNoP 0.20% frpc frpc [.] runtime.updateTimerPMask 0.15% frpc frpc [.] runtime.runqget 0.07% frpc [kernel.kallsyms] [k] __x86_indirect_thunk_rax 0.05% frpc [kernel.kallsyms] [k] update_wall_time 0.04% frpc [kernel.kallsyms] [k] _raw_qspin_lock 0.04% frpc [kernel.kallsyms] [k] __netif_receive_skb_core 0.03% frpc [kernel.kallsyms] [k] task_tick_fair 0.03% frpc [kernel.kallsyms] [k] rcu_check_callbacks ``` ### Describe alternatives you've considered _No response_ ### Affected area - [ ] Docs - [ ] Installation - [X] Performance and Scalability - [ ] Security - [ ] User Experience - [ ] Test and Release - [ ] Developer Infrastructure - [ ] Client Plugin - [ ] Server Plugin - [ ] Extensions - [ ] Others
gitea-mirror 2026-05-05 13:29:14 -06:00
  • closed this issue
  • added the
    bug
    todo
         labels
gitea-mirror commented 2026-05-05 13:29:17 -06:00
Author
Owner
Copy link

@fatedier commented on GitHub (Apr 17, 2022):

我这边缺乏复现的环境，等官方修复吧，到时候也麻烦帮忙测试下。

<!-- gh-comment-id:1100880027 --> @fatedier commented on GitHub (Apr 17, 2022): 我这边缺乏复现的环境，等官方修复吧，到时候也麻烦帮忙测试下。
gitea-mirror commented 2026-05-05 13:29:18 -06:00
Author
Owner
Copy link

@gmd20 commented on GitHub (Apr 18, 2022):

我这边缺乏复现的环境，等官方修复吧，到时候也麻烦帮忙测试下。

好，看他们帖子里说的可能跟在不同的线程上都修改网络连接的deadline timer有关系。反正我有几台服务器在不同省份网络环境不一样，大部分都会遇到这个问题（一天中可能会出现几次cpu告警），有的办公网网络稳定的就不会遇到问题。 出现问题后全部替换到1.16.6编译的版本就没有再遇到问题了。等官方有更新出来了我到时再看看吧。

<!-- gh-comment-id:1101019852 --> @gmd20 commented on GitHub (Apr 18, 2022): > 我这边缺乏复现的环境，等官方修复吧，到时候也麻烦帮忙测试下。 好，看他们帖子里说的可能跟在不同的线程上都修改网络连接的deadline timer有关系。反正我有几台服务器在不同省份网络环境不一样，大部分都会遇到这个问题（一天中可能会出现几次cpu告警），有的办公网网络稳定的就不会遇到问题。 出现问题后全部替换到1.16.6编译的版本就没有再遇到问题了。等官方有更新出来了我到时再看看吧。
gitea-mirror commented 2026-05-05 13:29:18 -06:00
Author
Owner
Copy link

@fatedier commented on GitHub (Apr 18, 2022):

你可以试着将 GOMAXPROCS 设置为 1，再测试一下看看，因为 frpc 一般来说不需要耗费太多的 cpu，单核一般也就足够了。

<!-- gh-comment-id:1101036148 --> @fatedier commented on GitHub (Apr 18, 2022): 你可以试着将 GOMAXPROCS 设置为 1，再测试一下看看，因为 frpc 一般来说不需要耗费太多的 cpu，单核一般也就足够了。
gitea-mirror commented 2026-05-05 13:29:19 -06:00
Author
Owner
Copy link

@gmd20 commented on GitHub (Apr 19, 2022):

你可以试着将 GOMAXPROCS 设置为 1，再测试一下看看，因为 frpc 一般来说不需要耗费太多的 cpu，单核一般也就足够了。

测试了，golang v1.18.1 还是有问题，设置 GOMAXPROCS=1 后避免这个问题了。

<!-- gh-comment-id:1101968671 --> @gmd20 commented on GitHub (Apr 19, 2022): > 你可以试着将 GOMAXPROCS 设置为 1，再测试一下看看，因为 frpc 一般来说不需要耗费太多的 cpu，单核一般也就足够了。 测试了，golang v1.18.1 还是有问题，设置 GOMAXPROCS=1 后避免这个问题了。
gitea-mirror commented 2026-05-05 13:29:20 -06:00
Author
Owner
Copy link

@hongweipeng commented on GitHub (Apr 29, 2022):

我这边也出现了，我用的是 release 页面下载编译后的文件的

<!-- gh-comment-id:1113125036 --> @hongweipeng commented on GitHub (Apr 29, 2022): 我这边也出现了，我用的是 release 页面下载编译后的文件的
gitea-mirror commented 2026-05-05 13:29:20 -06:00
Author
Owner
Copy link

@hongweipeng commented on GitHub (Apr 29, 2022):

图片

<!-- gh-comment-id:1113286174 --> @hongweipeng commented on GitHub (Apr 29, 2022): ![图片](https://user-images.githubusercontent.com/7546325/165949378-1a25b97d-a8c2-44be-b19b-b42415516bc4.png)
gitea-mirror commented 2026-05-05 13:29:21 -06:00
Author
Owner
Copy link

@HuaWang135608 commented on GitHub (Jul 5, 2022):

确实，我昨天也看到有一个服务器CPU占用100%了

<!-- gh-comment-id:1174539701 --> @HuaWang135608 commented on GitHub (Jul 5, 2022): 确实，我昨天也看到有一个服务器CPU占用100%了
gitea-mirror commented 2026-05-05 13:29:21 -06:00
Author
Owner
Copy link

@fatedier commented on GitHub (Jul 14, 2022):

c006b7ac27
看起来对应的修复已被合入。

待社区上游发布新的 release 后会重新发布一个 frp 的版本。

<!-- gh-comment-id:1183912154 --> @fatedier commented on GitHub (Jul 14, 2022): https://github.com/golang/go/commit/c006b7ac2765252f397dec40fef610a3c17d956d 看起来对应的修复已被合入。 待社区上游发布新的 release 后会重新发布一个 frp 的版本。
gitea-mirror commented 2026-05-05 13:29:21 -06:00
Author
Owner
Copy link

@befen commented on GitHub (Nov 25, 2022):

@gmd20 @fatedier
怎么设置 GOMAXPROCS 为 1

<!-- gh-comment-id:1326931314 --> @befen commented on GitHub (Nov 25, 2022): @gmd20 @fatedier 怎么设置 GOMAXPROCS 为 1
gitea-mirror commented 2026-05-05 13:29:22 -06:00
Author
Owner
Copy link

@fatedier commented on GitHub (Nov 25, 2022):

最新的 release 版本所使用的 Go 版本应该已经修复此问题。

<!-- gh-comment-id:1326964695 --> @fatedier commented on GitHub (Nov 25, 2022): 最新的 release 版本所使用的 Go 版本应该已经修复此问题。
gitea-mirror commented 2026-05-05 13:29:22 -06:00
Author
Owner
Copy link

@befen commented on GitHub (Nov 25, 2022):

@fatedier 好的，谢谢！

<!-- gh-comment-id:1326970837 --> @befen commented on GitHub (Nov 25, 2022): @fatedier 好的，谢谢！
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
dev
new
master
copilot/review-pull-request-5198
v0.68.1
v0.68.0
v0.67.0
v0.66.0
v0.65.0
v0.64.0
v0.63.0
v0.62.1
v0.62.0
v0.61.2
v0.61.1
v0.61.0
v0.60.0
v0.59.0
v0.58.1
v0.58.0
v0.57.0
v0.56.0
v0.55.1
v0.55.0
v0.54.0
v0.53.2
v0.53.1
v0.53.0
v0.52.3
v0.52.2
v0.52.1
v0.52.0
v0.51.3
v0.51.2
v0.51.1
v0.51.0
v0.50.0
v0.49.0
v0.48.0
v0.47.0
v0.46.1
v0.46.0
v0.45.0
v0.44.0
v0.43.0
v0.42.0
v0.41.0
v0.40.0
v0.39.1
v0.39.0
v0.38.0
v0.37.1
v0.37.0
v0.36.2
v0.36.1
v0.36.0
v0.35.1
v0.35.0
v0.34.3
v0.34.2
v0.34.1
v0.34.0
v0.33.0
v0.32.1
v0.32.0
v0.31.2
v0.31.1
v0.31.0
v0.30.0
v0.29.1
v0.29.0
v0.28.2
v0.28.1
v0.28.0
v0.27.1
v0.27.0
v0.26.0
v0.25.3
v0.25.2
v0.25.1
v0.25.0
v0.24.1
v0.24.0
v0.23.3
v0.23.2
v0.23.1
v0.23.0
v0.22.0
v0.21.0
v0.20.0
v0.19.1
v0.19.0
v0.18.0
v0.17.0
v0.16.1
v0.16.0
v0.15.1
v0.15.0
v0.14.1
v0.14.0
v0.13.0
v0.12.0
v0.11.0
v0.10.0
v0.9.3
v0.9.2
v0.9.1
v0.9.0
v0.8.1
v0.8.0
v0.7.0
v0.6.0
v0.5.0
v0.3.0
v0.2.0
v0.1.0
Labels
Clear labels   
In Progress

   
WIP

   
WaitingForInfo

   
bug

   
doc

   
duplicate

   
easy

   
enhancement

   
future

   
help wanted

   
invalid

   
lifecycle/stale

   
need-issue-template

   
need-usage-help

   
no plan

   
proposal

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
todo
No labels
In Progress
WIP
WaitingForInfo
bug
doc
duplicate
easy
enhancement
future
help wanted
invalid
lifecycle/stale
need-issue-template
need-usage-help
no plan
proposal
pull-request
question
todo
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/frp#2307
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?