Changes:
* Use `scan-build make` instead of running `scan-build` inside of `make`
(this appears to be necessary for the output synchronization to work)
* Use `-j "$(nproc)"` and `-Orecurse` for the main `make` step in all
jobs (including where this step is currently not parallel)
The main drawback of using parallel make (`-j`) is that the output of
different jobs may be printed interspersed, which makes the output
harder to read and less stable across multiple executions.
Example:
job1: line1
job1: line2
job2: line1
job3: line1
job1: line3
Using `-Orecurse` should fix this by ensuring that the output of all
jobs is still printed sequentially in the order that the jobs were
executed (that is, as if `-j` was not used), even if the jobs themselves
are executed in parallel.
This should ensure that the main `make` step in each job runs its
targets in parallel and has a stable output at the same time, making it
easier to compare the logs of the same job across different CI runs.
Note: The `-O` flag is specific to GNU make and was added in version 4.0
(2013-10-09).
Related commits:
* 500d8f2d6 ("ci: run make in parallel where applicable", 2023-08-14) /
PR #5960
* 1f6400bd8 ("build: sync scan-build target with CI", 2024-02-24) /
PR #6222
Fixes the following error in `debian_ci`[1]:
Running with gitlab-runner 18.7.0~pre.433.g3a5f2314 (3a5f2314)
on green-1.saas-linux-small-amd64.runners-manager.gitlab.com/default JLgUopmMV, system ID: s_deaa2ca09de7
Preparing the "docker+machine" executor
Using Docker executor with image registry.salsa.debian.org/salsa-ci-team/ci-image-git-buildpackage:latest ...
Using effective pull policy of [always] for container registry.salsa.debian.org/salsa-ci-team/ci-image-git-buildpackage:latest
Pulling docker image registry.salsa.debian.org/salsa-ci-team/ci-image-git-buildpackage:latest ...
Using docker image sha256:d3583847cf2ed2f794f97a09510fd08a07c483a37718e81fe01d5576d4aca1a7 for registry.salsa.debian.org/salsa-ci-team/ci-image-git-buildpackage:latest with digest registry.salsa.debian.org/salsa-ci-team/ci-image-git-buildpackage@sha256:1416995f1e5984f414b083806cc591223d176678a9252de990b38b887981efc3 ...
[...]
Executing "step_script" stage of the job script
Using effective pull policy of [always] for container registry.salsa.debian.org/salsa-ci-team/ci-image-git-buildpackage:latest
Using docker image sha256:d3583847cf2ed2f794f97a09510fd08a07c483a37718e81fe01d5576d4aca1a7 for registry.salsa.debian.org/salsa-ci-team/ci-image-git-buildpackage:latest with digest registry.salsa.debian.org/salsa-ci-team/ci-image-git-buildpackage@sha256:1416995f1e5984f414b083806cc591223d176678a9252de990b38b887981efc3 ...
$ git checkout -B ci_build "$CI_COMMIT_SHA"
Switched to a new branch 'ci_build'
$ gitlab-ci-enable-sid
$ gitlab-ci-enable-experimental
[...]
$ apt-get install --no-install-recommends -qy gawk
[...]
$ gitlab-ci-git-buildpackage
[...]
make master branch current for gbp
Switched to a new branch 'master'
+ '[' -z '' ']'
+ gbp buildpackage -uc -us --git-pristine-tar --git-debian-branch=master
/usr/bin/gitlab-ci-git-buildpackage: line 66: gbp: command not found
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1
[1] https://gitlab.com/Firejail/firejail_ci/-/jobs/12593768999
Instead of Debian 10, use Debian 11 in `build_debian_package`.
It currently fails to update the package index[1]:
Executing "step_script" stage of the job script 00:01
Using effective pull policy of [always] for container debian:buster
Using docker image sha256:69530eaa9e7e18d0aad40c38b75a22b40c6ebdc374c059bd5f2eb07042caa50a for debian:buster with digest debian@sha256:58ce6f1271ae1c8a2006ff7d3e54e9874d839f573d8009c20154ad0f2fb0a225 ...
$ apt-get update -qy
Ign:1 http://deb.debian.org/debian buster InRelease
Ign:2 http://deb.debian.org/debian-security buster/updates InRelease
Ign:3 http://deb.debian.org/debian buster-updates InRelease
Err:4 http://deb.debian.org/debian buster Release
404 Not Found [IP: 151.101.2.132 80]
Err:5 http://deb.debian.org/debian-security buster/updates Release
404 Not Found [IP: 151.101.2.132 80]
Err:6 http://deb.debian.org/debian buster-updates Release
404 Not Found [IP: 151.101.2.132 80]
Reading package lists...
E: The repository 'http://deb.debian.org/debian buster Release' does not have a Release file.
E: The repository 'http://deb.debian.org/debian-security buster/updates Release' does not have a Release file.
E: The repository 'http://deb.debian.org/debian buster-updates Release' does not have a Release file.
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1
Also, note that LTS support for Debian 10 ended on 2024-06-30, while
Debian 11 has LTS support until 2026-08-31[2].
See also commit 7b8ce3acf ("ci: upgrade debian:stretch to
debian:buster", 2023-04-25) / issue #5818.
[1] https://gitlab.com/Firejail/firejail_ci/-/jobs/10737624219
[2] https://wiki.debian.org/LTS
To make it consistent with the standard GNU make targets.
From the manual of GNU Make (version 4.4.1-2):
> 'installcheck'
> Perform installation tests (if any). The user must build and
> install the program before running the tests. You should not
> assume that '$(bindir)' is in the search path.
Commands used to search and replace:
$ git grep -Ilz print-version |
xargs -0 -I '{}' sh -c "printf '%s\n' \"\$(sed \
's/print-version/installcheck/g' '{}')\" >'{}'"
$ git grep -Ilz 'print version' .github/workflows |
xargs -0 -I '{}' sh -c "printf '%s\n' \"\$(sed \
's/print version/make installcheck/g' '{}')\" >'{}'"
Added on commit c9531d95e ("build: add print-version target and use in
CI", 2024-02-22) / #6230.
Build log[1]:
$ make
[...]
make -C src/libtrace/
make[1]: Entering directory '/builds/Firejail/firejail_ci/src/libtrace'
gcc -ggdb -O2 -DVERSION='"0.9.73"' [...] -c ../../src/libtrace/libtrace.c -o ../../src/libtrace/libtrace.o
../../src/libtrace/libtrace.c:451:59: warning: 'struct stat64' declared inside parameter list will not be visible outside of this definition or declaration
451 | typedef int (*orig_stat64_t)(const char *pathname, struct stat64 *statbuf);
| ^~~~~~
../../src/libtrace/libtrace.c:453:41: warning: 'struct stat64' declared inside parameter list will not be visible outside of this definition or declaration
453 | int stat64(const char *pathname, struct stat64 *statbuf) {
| ^~~~~~
../../src/libtrace/libtrace.c: In function 'stat64':
../../src/libtrace/libtrace.c:457:40: error: passing argument 2 of 'orig_stat64' from incompatible pointer type [-Wincompatible-pointer-types]
457 | int rv = orig_stat64(pathname, statbuf);
| ^~~~~~~
| |
| struct stat64 *
../../src/libtrace/libtrace.c:457:40: note: expected 'struct stat64 *' but argument is of type 'struct stat64 *'
../../src/libtrace/libtrace.c: At top level:
../../src/libtrace/libtrace.c:476:60: warning: 'struct stat64' declared inside parameter list will not be visible outside of this definition or declaration
476 | typedef int (*orig_lstat64_t)(const char *pathname, struct stat64 *statbuf);
| ^~~~~~
../../src/libtrace/libtrace.c:478:42: warning: 'struct stat64' declared inside parameter list will not be visible outside of this definition or declaration
478 | int lstat64(const char *pathname, struct stat64 *statbuf) {
| ^~~~~~
../../src/libtrace/libtrace.c: In function 'lstat64':
../../src/libtrace/libtrace.c:482:41: error: passing argument 2 of 'orig_lstat64' from incompatible pointer type [-Wincompatible-pointer-types]
482 | int rv = orig_lstat64(pathname, statbuf);
| ^~~~~~~
| |
| struct stat64 *
../../src/libtrace/libtrace.c:482:41: note: expected 'struct stat64 *' but argument is of type 'struct stat64 *'
make[1]: Leaving directory '/builds/Firejail/firejail_ci/src/libtrace'
Environment: gcc (14.2.0-r4), musl-dev (1.2.5-r8) on Alpine Linux v3.21.
Relates to #6610.
[1] https://gitlab.com/Firejail/firejail_ci/-/jobs/8853165146
For consistency and to make it clearer where jobs differ (for example,
to see where `--enable-analyzer` is used).
Changes:
* Always use --prefix=/usr and --enable-fatal-warnings (except in the
Alpine job due to current warnings; see #6224)
* Use the same argument order
Note: mkdeb.sh and platform/rpm/mkrpm.sh already pass `--prefix=/usr` to
./configure.
So that they fail early instead of letting them run indefinitely when
there are problems with the CI infrastructure.
Use 5 minutes for the jobs that usually complete in under a minute
(check-profiles and codespell) and 10 minutes for the rest (most jobs
usually take 1-3 minutes).
General changes:
* Use a single -q on update, as the output is not too long
* Use a single -q on install, to show all packages at once
GitLab-specific changes:
* Use `DEBIAN_FRONTEND=noninteractive` to reduce noise
* Use --no-install-recommends to avoid installing unnecessary packages
* Filter out uninteresting lines on install
Note: `DEBIAN_FRONTEND` does not appear to be needed in the default
GitHub runner container and not many packages are currently being
downloaded/installed in them, so do the above changes only jobs that use
custom Docker images.
There already exists a workflow dedicated to profile checks:
* .github/workflows/profile-checks.yml.
Keep the build and lint jobs separate to make it easier to spot build vs
lint failures in CI.
See also commit c3b42dbd2 ("ci: disable sort.py on gitlab jobs as well",
2023-04-10).
Example log of it failing:
$ ./configure
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/tmp/build':
configure: error: C compiler cannot create executables
See `config.log' for more details
To make debugging easier.
Use a separate shell script instead of just a make target to ensure that
it can safely run before ./configure and without having make installed.
For increased readability.
Note: `>` basically turns each newline into a space while `|` keeps
newlines as is. Both remove leading indentation.
Note2: On jobs using `apt-get install`, this commit moves package names
to their own line, to make it easier to compare which packages are being
installed across such jobs.
This amends the following commits:
* 370b9db39 ("gitlab pipeline fixes", 2020-04-21)
* 42fdea77f ("gitlab", 2020-04-21)
Added on commit 2e14c1a1d ("Adding sort.py to GitLab CI (#2973)",
2019-09-21).
Currently, the author name is used as the author email in git and vice
versa.
Changes:
* Split the commands to make them easier to read
* Put the name command first to match the usual git usage
* Fix swapped name/email arguments
* Use the DEBFULLNAME / DEBEMAIL environment variables instead of
GitLab-specific ones
Added on commit 4cf51b591 ("extend gitlab-ci configuration to run Debian
CI tests", 2019-06-21).
Split them into separate steps to make it clearer in the logs which
command causes a job to fail when it does.
Note that they are already spli in the GitHub workflows:
* .github/workflows/build.yml
* .github/workflows/build-extra.yml
This is currently causing the following jobs to fail on GitLab[1]:
* build_ubuntu_package
* build_fedora_package
See commit df6ea884f ("merges, disable sort.py in profile checks
temporarely, two more private-etc profiles", 2023-02-14).
[1] https://gitlab.com/Firejail/firejail_ci/-/pipelines/832588068
The official .deb package is always built with apparmor support, so use
`--enable-apparmor` in mkdeb.sh and remove the "deb-apparmor" target in
order to reduce redundancy.
Note that custom configure options may be specified by calling
./mkdeb.sh directly.
For example, to build the .deb package without apparmor support, instead
of running `make deb`, the following commands can be used:
make dist
./mkdeb.sh --disable-apparmor
Also, change the `build_apparmor` GitLab CI job into
`build_no_apparmor`, which is intended to check that building without
apparmor still works.
Note: This commit makes the resulting .deb package not have an
"-apparmor" suffix (see `EXTRA_VERSION` in mkdeb.sh), to avoid
redundancy (as having apparmor support becomes the default).
Misc: This is a follow-up to #5654.
Relates to #5154#5176#5547.
The "deb" target depends on the "dist" target, which creates an archive
from DISTFILES.
The arguments to ./configure are misleading, as they do not affect the
archive that is used by `make deb`. That is the case because the
configure output files (config.mk and config.sh) are not copied into the
dist archive, only their input files (config.mk.in and config.sh.in).
In order to affect the .deb package, the configure arguments have to be
passed to mkdeb.sh, which then forwards them to ./configure itself.
Note: This does not apply to the rpm-based jobs, as `make rpms` uses the
files directly rather than using the dist archive.
Relates to #5154.
Currently, mkdeb.sh (which is used to make a .deb package) runs
./configure with hardcoded options (some of which are automatically
detected based on configure-time variables). To work around the
hardcoding, contrib/fj-mkdeb.py is used to add additional options by
rewriting the actual call to ./configure on mkdeb.sh. For example, the
following invocation adds --disable-firetunnel to mkdeb.sh:
$ ./configure && ./contrib/fj-mkdeb.py --disable-firetunnel
To avoid depending on another script and to avoid re-generating
mkdeb.sh, just let the latter pass the remaining arguments (the first
one is an optional package filename suffix) to ./configure directly.
Example:
$ make distclean && ./configure && make dist &&
./mkdeb.sh "" --disable-firetunnel
Additionally, change contrib/fj-mkdeb.py to do roughly the same as the
above example, by simply forwarding the arguments that it receives to
./mkdeb.sh (which then forwards them to ./configure). Also, remove the
--only-fix-mkdeb option, since the script does not change mkdeb.sh
anymore. With these changes, the script's usage (other than when using
--only-fix-mkdeb) should remain the same.
Note: To clean the generated files and then make a .deb package using
the default configuration, the invocation is still the same:
$ make distclean && ./configure && make deb
Note2: Running ./configure in the above examples is only needed for
generating Makefile/mkdeb.sh from Makefile.in/mkdeb.sh.in after running
distclean, so that running `make` / `./mkdeb.sh` afterwards works.
Should fully fix#772.
Relates to #1205#3414#5148.
The "build_apparmor" job was added on commit 342e71cd8 ("Add
deb-apparmor build to Gitlab CI", 2019-01-26). It would call
`./mkdeb-apparmor.sh`, which would run `./configure --enable-apparmor`
directly, adding `-lapparmor` to `EXTRA_LDFLAGS` and thus passing it to
the linker.
Later, commit 87e7b3139 ("Configure Debian package with AA and SELinux
options", 2020-05-13) / PR #3414 merged mkdeb.sh and mkdeb-apparmor.sh
into mkdeb.sh.in, which does not always pass `--enable-apparmor` to
./configure directly. Instead, it adds `--enable-apparmor` depending on
whether the `$HAVE_APPARMOR` environment variable is set, which would be
done by a previous run of ./configure with `--enable-apparmor`. Since
on "build_apparmor" ./configure is not run the first time with
`--enable-apparmor`, neither is it on the second time and thus
`-lapparmor` is never passed to the linker. This commit adds
`--enable-apparmor` to the first ./configure run on the ci job, so that
it gets passed to the one being executed on mkdeb.sh as well.
To ensure that firejail was actually built with support for it.
Note: This commit intentionally fails on GitLab CI to demonstrate that
the above is currently not the case.
Likely caused by commit 500a56efd ("more on nettrace", 2022-01-07).
From the build log of "debian_ci" for the above commit[1]:
make[1]: Leaving directory '/builds/Firejail/firejail_ci'
dh_fixperms -Nfirejail
debian/rules override_dh_missing
make[1]: Entering directory '/builds/Firejail/firejail_ci'
dh_missing -pfirejail --fail-missing
dh_missing: warning: etc/firejail/hostnames exists in debian/tmp but is not installed to anywhere
dh_missing: error: missing files, aborting
[1] https://gitlab.com/Firejail/firejail_ci/-/jobs/1952432676
