mirror of
https://github.com/netblue30/firejail.git
synced 2026-05-15 14:16:14 -06:00
[GH-ISSUE #5293] Slowdown with latest kernels #2947
Labels
No labels
LTS merge
LTS merge
bug
bug
converted-to-discussion
doc-todo
documentation
duplicate
enhancement
file-transfer
firecfg
firejail-in-firejail
firetools
graphics
help wanted
information_old
installation
invalid
modif
moved
needinfo
networking
notabug
notourbug
old-version
overlayfs
packaging
profile-request
pull-request
question
question_old
removal
runtime-permissions
sandbox-ipc
security
stale
wiki
wiki
wontfix
wordpress
workaround
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/firejail#2947
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @NetSysFire on GitHub (Aug 4, 2022).
Original GitHub issue: https://github.com/netblue30/firejail/issues/5293
Description
With the newest kernels, at least on Arch Linux, which ships 5.18.15-arch1-1, the time to initialize the sandbox (
Child process initialized in ...) skyrockets. Arch Linux's kernel patches are minimal enough that I think they are not the issue here.Steps to Reproduce
firejail ls.To be clear here: Right now it takes around 900 ms to init, on a prior boot without updating or touching anything, 35 ms. On slightly earlier kernel versions I sometimes even had an unusable 10 seconds.
I need to do proper tests, but it looks like systemd sandboxing, which also makes use of the same kernel features, may also be affected. I noticed a slowdown there, too but I did not measure it. I only took note of this during the start and shutdown sequence, and it was worse in the latter.
However, when I had the unusable 10 seconds delay, only firejailed stuff was affected by it. All systemd service units, which do not use firejail but their own sandboxing, initialized without delay, resulting in a normal system startup.
I did run firejail with
--debugand the slowdown was very visible when it was setting up filesystems.Expected behavior
No slowdown.
Actual behavior
Slowdown.
Behavior without a profile
n/a
Additional context
Ping @glitsj16, I discussed this on IRC with them and they said they also noticed slowdowns. It may very well be a kernel regression, aka a "notourbug".
The bug is so inconsistent it is driving me insane, so I'd love some more pairs of eyes looking over this.
Environment
Checklist
/usr/bin/vlc) "fixes" it).https://github.com/netblue30/firejail/issues/1139)browser-allow-drm yes/browser-disable-u2f noinfirejail.configto allow DRM/U2F in browsers.--profile=PROFILENAMEto set the right profile. (Only relevant for AppImages)Log
n/a
@ghost commented on GitHub (Aug 4, 2022):
After hearing about this by accidental presence on #archlinux IRC I installed several kernels available on Arch Linux to dig deeper into this slowdown issue:
There's indeed a rather significant difference in sandbox initialization time depending on kernel in use, as observed by the OP. I cannot (yet) determine what could be causing any of this but I'm keeping these extra kernels installed to check how things evolve on future releases. Even when its not our bug it's always nice to know how firejail users in the field may notice these fluctuations, and how we could respond to potential reports.
@reinerh commented on GitHub (Aug 4, 2022):
I don't notice any slowdown on Debian (5.18.5),
firejail lsinitialization always takes about 50ms.@NetSysFire commented on GitHub (Aug 4, 2022):
Hmm. I wonder if this could be hardware specific then. My CPU is a ryzen 1600af. I wonder if the mitigations could affect the sandbox init time.
@reinerh commented on GitHub (Aug 4, 2022):
I just updated my kernel to 5.18.14 and noticed a very small slowdown; it consistently takes about 60ms now.
Edit: CPU: Intel Core i7-7820HQ
@NetSysFire commented on GitHub (Aug 4, 2022):
Alright, I recorded
firejail --debugwith script. These files are not human-readable. To replay them, usescriptreplay, it is bundled in util-linux and therefore usually included in every installation.The exact command you need is:
scriptreplay -T timing.log -B io.logio.log
timing.log
@reinerh commented on GitHub (Aug 4, 2022):
Can you maybe try recording it with
perf(needs debug symbols)? For example:(might need to be run as root)
If that works there are also tools to generate flamegraphs etc., see https://www.brendangregg.com/perf.html
For me it seems to spend a lot of time in
fs_blacklist, and there inglobbingandexpand_macros.@NetSysFire commented on GitHub (Aug 18, 2022):
It fluctuates quite a bit for me. I have built firejail from master with debug symbols. Right now it is at 86 ms which is fine. I will re-test after a reboot which also applies the kernel update. It appears that a reboot randomizes the time it takes for the sandbox to initialize, its super weird.
After that I will see if this either another kernel upgrade or the upgrade to the unstable version has helped, if not I will provide the perf stuff.
@ghost commented on GitHub (Aug 18, 2022):
There's a nice tool in arch linux repo https://archlinux.org/packages/community/x86_64/cargo-flamegraph/. Pretty handy to create flamegraphs directly from perf output.
@NetSysFire commented on GitHub (Aug 18, 2022):
Alright. I got 16k ms after a reboot and kernel upgrade.
perf.data.gz
Had to gzip because of github.
@ghost commented on GitHub (Aug 18, 2022):
This sounds increasingly more plausible, especially when taken in context with the observation shared on IRC that the 16k ms sowdown is also noticeable on starting systemd sandboxed units.
I just finished another round of tests on the kernels noted in https://github.com/netblue30/firejail/issues/5293#issuecomment-1205415123. On an Intel® Core™2 Duo CPU T6600 @ 2.20GHz × 2 there's hardly any difference between running with/without
mitigations=offand I'm seeing 'normal' sandbox initialization of ~70ms. Alas I never measured this before this issue was reported, but FWIW I can't say I notice any recent regressions.Realizing this is a big ask, but would any of you be willing to run the same tests with
mitigations=offto rule those out?@reinerh commented on GitHub (Aug 18, 2022):
According to the perf trace it seems to take >95% of the time doing syscalls (read).
@NetSysFire commented on GitHub (Aug 18, 2022):
I am a noob at interpreting perf traces, but
93.88%--get_dominating_idlooks really suspicious, especially since it appears to be used several times and it is the only thing which takes this amount of time.Sure, I can do that. Applying the kernel parameter will take a reboot and since the time used for initializing is very inconsistent, I will report back if it still happens after I reboot. So expect another comment from me tomorrow or so.
@NetSysFire commented on GitHub (Aug 21, 2022):
Took a bit longer than I planned because I did not get a slowdown immediately, but it is currently at 5k ms even with mitigations=off
@NetSysFire commented on GitHub (May 10, 2024):
Still happening by the way. glitsj16 said on IRC I should bump this.
@ghost commented on GitHub (May 10, 2024):
Wondering if this might be related to https://github.com/netblue30/firejail/pull/6307. Couldn't determine that (yet) as OP needed to attend other things on IRC.
@NetSysFire commented on GitHub (May 11, 2024):
Looks to be unrelated. I freshly built firejail from -git while I had that persistent delay again just now, according to
time firejail lsit takes 6s of cpu time.@ghost commented on GitHub (May 12, 2024):
Recap of the latest revisit (with recent 6.8.x kernels) on IRC #firejail:
disable-common.incTL;DR looks like a kernel issue. Not very likely we can do much about it.
@NetSysFire commented on GitHub (May 12, 2024):
I updated my BIOS, yes the firmware, the BIOS, however you want to call it, and the issue got immediately better. This is very cursed indeed that the firmware affects a random syscall. However, since this issue is very random I am unsure if this is really fixed. Give me some more time and I'll see if this still happens.
Hardware, for anyone interested:
@NetSysFire commented on GitHub (May 23, 2024):
So far no more slowdowns. I still cant believe this issue got fixed by a BIOS update of all things. I should have applied more holy water.