Commit graph

877 commits

Author SHA1 Message Date
Evan Maddock
6daca3ea17
common: Convert yconvert.py to Python 3
Signed-off-by: Evan Maddock <maddock.evan@vivaldi.net>
2024-04-02 15:29:55 -04:00
Evan Maddock
16d20bf4c8
common: Convert yauto.py to Python 3
Signed-off-by: Evan Maddock <maddock.evan@vivaldi.net>
2024-04-02 15:03:51 -04:00
Evan Maddock
9766cdf73c
common: Convert pbump.py to Python 3
Signed-off-by: Evan Maddock <maddock.evan@vivaldi.net>
2024-04-02 14:06:18 -04:00
Fabio Forni
1631b73fe4 common/Scripts/dloader.py: Port to Python 3 + Black format 2024-04-02 19:45:12 +02:00
Rune Morling
efc83c3119
ci/package_checks: Improve linter feedback and bump check (#2038)
**Summary**

This PR:

- Improves the output format when not running in CI.
- Shows this output when being called via a hook.
- Extends the package bump check so that pspec files must always be
bumped.

Resolves #1888.

**Test Plan**

Run CI checks locally using `go-task check` in a fairly dirty working
directory:


![image](817d8e39-a269-44d2-8e19-80d75e4698a2)

Try to commit a changed pspec file:

```
$ gotopkg nano
$ go-task build
$ git add pspec_x86_64.xml
$ git commit
ERR packages/n/nano/pspec_x86_64.xml:1: Package release is not incremented by 1
Package checks failed: <snip>/packages/common/CI/package_checks.py packages/n/nano/pspec_x86_64.xml
```

**Checklist**

- [x] ~~Package was built and tested against unstable~~
2024-04-02 19:08:28 +02:00
Silke Hofstra
46590963b1 worklog.py: Add html output format 2024-04-01 18:05:34 +02:00
Silke Hofstra
05ff11feca ci/package_checks: Show an error when the pspec has not been bumped 2024-03-31 18:44:21 +02:00
Silke Hofstra
0f28053733 ci/package_checks: Create and use classes for package.yml and pspec.xml 2024-03-31 18:44:21 +02:00
Silke Hofstra
fae0cf3877 hooks/pre-commit: Show output from package checks
**Summary**

Show the output from the package checks so the committer is aware that there are warnings or
informational messages for the package.
2024-03-31 18:44:21 +02:00
Silke Hofstra
cd86c954e5 ci/package_checks: Improve output when called outside of CI
**Summary**

Configure a decent logger and log results when not running in the CI.
2024-03-31 18:44:21 +02:00
Evan Maddock
d6bec7ef41 common: Update ISO packages list
Signed-off-by: Evan Maddock <maddock.evan@vivaldi.net>
2024-03-31 01:26:00 +01:00
Silke Hofstra
d06ef6f45f worklog.py: Improve help text 2024-03-21 22:23:10 +01:00
Silke Hofstra
e6024af4ad worklog.py: Add emoji, build time and comment to the worklog build output
Improve the worklog build output with:

- An emoji indicating the job status.
- A color indicating the job status.
- The build duration when the job is finished.
- The comment given to the bulid.
2024-03-21 22:22:43 +01:00
Silke Hofstra
4b002f5c20 worklog.py: add -F/--follow option that follows the log
**Summary**

Add an option that prints build log entries as soon as they are available.
This works similarly to the familiar options in `tail`.
2024-03-20 21:12:02 +01:00
Silke Hofstra
77b4ac49c5 worklog: Improve output of update subcommand
Change the output of the `update` subcommand with:

- A Markdown format that can be used for publishing.
- Multiple authors (one for each commit).
- The GitHub author that made the change.
2024-03-07 21:10:24 +01:00
Joey Riches
792b0d78d4 common/Scripts: Allow no args usage of cpesearch()
**Summary**
- When no args are passed to cpesearch() it'll use the current
  directory name. Useful in combination with gotopkg().
2024-02-29 17:52:57 +00:00
Silke Hofstra
bb245a4777
worklog.py: Add a -s/--sort option that sorts output (#1724)
**Summary**

Without specifying the -s/--sort option, worklog.py will output the
latest build or commit last.

This enables the ability to trivially keep an eye on large PR (re)builds
with something like:

    while true; do
        date
        ./worklog.py --format md builds 2024-02-24 now |tail -n5
        echo "----"
        sleep 30
    done

**Test Plan**

Test the output of worklog.py with and without -s/--sort options.

**Checklist**

- [ ] Package was built and tested against unstable
2024-02-25 16:03:04 +01:00
Rune Morling
57f85a9f4a
get_unique_updates Initial commit (#1622)
**Summary**

Script which can take the markdown output of worklog.py and
 output one entry per package using the highest version

**Test Plan**

- Examined output sent to stdout and output written to file to ensure
data is correct
- Ran automated test from common/perl

```bash
❯ prove -v get_unique_updates/t/*.t                                                                                                             
get_unique_updates/t/get_unique_updates.t .. 
# Seeded srand with seed '20240214' from local date.
1..6
ok 1 - Module loads correctly.
ok 2 - Markdown file passes file check
ok 3 - Text file fails file check
ok 4 - Entry for package with 2 versions has highest value
ok 5 - Entry for package with 3 versions has highest version
test looking for /home/tracey/packaging/packages/common/perl/get_unique_updates/t/../t/data/markdown_output.md
ok 6 - Output file is written and has expected number of packages
ok
All tests successful.
Files=1, Tests=6,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.06 cusr  0.01 csys =  0.08 CPU)
Result: PASS
```


**Checklist**

- [ ] Package was built and tested against unstable N/A
2024-02-25 14:16:12 +01:00
Tracey Clark
4203cc9ef3 Update module load order. Add vstags to gitignore 2024-02-24 22:08:42 -06:00
Tracey Clark
0aa2de3433 Add missing module to check 2024-02-24 21:45:33 -06:00
Tracey Clark
1c5bf8a6b7 Remove test module 2024-02-24 21:42:14 -06:00
Rune Morling
a300869c18
worklog.py: Add a -s/--sort option that sorts output
Without specifying the -s/--sort option, worklog.py will output the
latest build or commit last.

This enables the ability to trivially keep an eye on large PR (re)builds
with something like:

    while true; do
        date
        ./worklog.py --format md builds 2024-02-24 now |tail -n5
        echo "----"
        sleep 30
    done

Signed-off-by: Rune Morling <ermo@serpentos.com>
2024-02-24 18:35:31 +01:00
Tracey
d776236087
Update README.md
Updated notes about yabi to reflect that it is no longer used. Removed information on filing issues for it.
2024-02-22 16:06:04 -06:00
Tracey Clark
b2eabc4ca4 Rename README so GitHub can display it nicely 2024-02-21 20:47:12 -06:00
Tracey Clark
c677d0792d Add module check to ensure dependencies met 2024-02-21 20:43:13 -06:00
Tracey Clark
4ec1c00e13 get_unique_updates Initial commit
Script which can take the markdown output of worklog.py and
 output one entry per package using the highest version
2024-02-18 16:56:19 -06:00
Rune Morling
f501790397
helpers*sh: Update/add cpesearch for bash/fish/zsh
Now supports showing usage if '-h' and '--help' are supplied as arguments

Signed-off-by: Rune Morling <ermo@serpentos.com>
2024-01-31 21:31:03 +01:00
Rune Morling
924ad56611
cpesearch: Rename from cvefuzzysearch per request
**Summary**

Signed-off-by: Rune Morling <ermo@serpentos.com>
2024-01-31 20:19:26 +01:00
Rune Morling
1593c59ba7
helpers*sh: Add a primitive cvefuzzysearch function
This can be useful when looking up the correct CPE $PRODUCT:$VENDOR pair
for monitoring.yml purpurses and for manually searching for known CVEs.

This commit only adds a PoC bash script helper.

Signed-off-by: Rune Morling <ermo@serpentos.com>
2024-01-30 17:47:59 +01:00
Joey Riches
b73fc44307 common: Fix typo in custom local repo template 2024-01-24 19:03:34 +00:00
Rune Morling
eb18e1b921
helpers.*sh: Rename function to localrepo_reindex
This fixes an issue where lightdm-wrapper would choke on the previous
function name (reindex-localrepo) when logging into Budgie.

Reported and tested working by @yusi1.

Signed-off-by: Rune Morling <ermo@serpentos.com>
2024-01-22 02:24:31 +01:00
Rune Morling
32b4caeafd
helpers.*sh: Add reindex-localrepo convenience function
**Summary**
We have no convenient way to re-index our local repo.

Since it is quite a long command, why not add a convenience function to
do it correctly that is maintained by Solus Staff?

This commit does just that for all helper scripts.

Signed-off-by: Rune Morling <ermo@serpentos.com>
2024-01-20 18:56:19 +01:00
Silke Hofstra
a5f2eccabe common: Fix worklog script for commits without scope 2024-01-20 16:14:35 +01:00
Evan Maddock
c69f2e98cc
common: Update ISO package list
Signed-off-by: Evan Maddock <maddock.evan@vivaldi.net>
2023-12-15 20:02:56 -05:00
Silke Hofstra
1878105829 ci/package_checks: fix check for frozen packages
Fix the package check triggering on packages not in the ISO.
2023-12-16 00:19:52 +01:00
Rune Morling
493bb834e7
ci/package_checks: fix dependency order for pkgconfig32 (#961)
**Summary**

Fix `pkgconfig32` entries not being sorted in a separate section by
sorting them *before* `pkgconfig` (as that is currently most common).

Resolves #921

**Test Plan**

- Run checks against `packages/a/avahi/package.yml`, see no dependency
order violation.
- Run checks against `packages/c/cups/package.yml`, see dependency order
violation (it has `pkgconfig` before `pkgconfig32`).

**Checklist**

- [x] ~~Package was built and tested against unstable~~ n/a
2023-12-14 08:09:34 +01:00
Rune Morling
2d4962b13d
ci/package_checks: fix SPDX license check for WITH/AND/OR (#962)
**Summary**

SPDX license identifiers may be a 'simple' or 'compound' expression,
meaning that the following formats are also allowed:

- A document reference instead of license identifier.
- `+` after the license identifier.
- License identifier followed `WITH` and an exception identifier.
- The above split by `AND` or `OR`.
- The above contained between `()`.

All but the first case should now be valid according to the package
checks. The first case is still invalid, as we want license identifiers
and not document references in our packages.

See: https://spdx.github.io/spdx-spec/v2.3/SPDX-license-expressions/

Resolves #956

**Test Plan**

Manually run the package checks against all packages with `WITH/AND/OR`
in the license:

```console
> common/CI/package_checks.py (rg 'license *:.* (WITH|AND|OR) ' -l |rg yml)
Checking files: packages/w/wxsqlite3/package.yml, packages/w/wxwidgets/package.yml, packages/w/wxPython/package.yml, packages/w/wxwidgets30/package.yml, packages/o/openscenegraph/package.yml, packages/l/libabigail/package.yml, packages/o/openjdk-17/package.yml, packages/o/openjdk-11/package.yml, packages/o/opencascade/package.yml, packages/o/opam/package.yml, packages/j/jtreg/package.yml
Found 0 result(s), 0 error(s)
```

Manually invalidate one of the licenses and run the check again:

```console
./common/CI/package_checks.py (rg 'license *:.* (WITH|AND|OR) ' -l |rg yml)
Checking files: packages/w/wxsqlite3/package.yml, packages/w/wxwidgets30/package.yml, packages/w/wxwidgets/package.yml, packages/w/wxPython/package.yml, packages/o/openscenegraph/package.yml, packages/o/openjdk-11/package.yml, packages/o/openjdk-17/package.yml, packages/l/libabigail/package.yml, packages/o/opencascade/package.yml, packages/o/opam/package.yml, packages/j/jtreg/package.yml
Found 1 result(s), 0 error(s)
::warning file=packages/l/libabigail/package.yml,line=7::invalid license identifier: 'Apache-2.0 WITH LLVM-exceptionx'
```

**Checklist**

- [x] ~~Package was built and tested against unstable~~ n/a
2023-12-14 08:08:59 +01:00
Rune Morling
fa92f3d362
ci/package_checks: add check for package version being a string (#963)
**Summary**

Ensure that package versions are always strings and not numbers, as
might happen with versions like '1.2' and '20230102'.

**Test Plan**

Manually run check against repo:

```shell
$ ./common/CI/package_checks.py **/package.yml
Checking files: <snip>
Found 234 result(s), 234 error(s)
::error file=packages/a/a2jmidid/package.yml,line=2::Package version is not a string
::error file=packages/a/accounts-qml-module/package.yml,line=2::Package version is not a string
<etc>
```

**Checklist**

- [x] ~~Package was built and tested against unstable~~ n/a
2023-12-14 08:08:02 +01:00
Joey Riches
f3c9ab8905 common/Scripts: return 1 instead of exit 1 on err in helper.sh
So it doesn't exit the shell.
2023-12-13 18:42:45 +00:00
Evan Maddock
962d983d06
CI/package_checks: Revert freeze period
I don't know what the proper time format is, but it isn't this.

Signed-off-by: Evan Maddock <maddock.evan@vivaldi.net>
2023-12-12 15:28:46 -05:00
Evan Maddock
cc84aa1c56
ci/package_checks: Set current freeze period
Signed-off-by: Evan Maddock <maddock.evan@vivaldi.net>
2023-12-12 13:29:18 -05:00
Evan Maddock
a57ba34d77
common: Update ISO package list
Signed-off-by: Evan Maddock <maddock.evan@vivaldi.net>
2023-12-12 13:03:44 -05:00
Silke Hofstra
06ce2aa892 ci/package_checks: add check for frozen packages
Add a simple configuration file for the CI checks,
that includes dates for a package freeze.

Packages updates for packages in `common/iso_packages.txt` result in
either a notice outside of freeze periods, or a warning in freeze periods.
2023-12-06 14:31:44 +01:00
Silke Hofstra
34c248c38b common: add list of packages included in all ISOs 2023-12-06 14:13:39 +01:00
Rune Morling
68406a8c42
common/Scripts/helpers.sh: Document bash completion
A little is better than nothing?

Signed-off-by: Rune Morling <ermo@serpentos.com>
2023-12-05 01:08:27 +01:00
Silke Hofstra
20abd1fa3f common: shellcheck Bash helper script 2023-12-04 23:10:46 +01:00
Silke Hofstra
5e72dd0589 ci/package_checks: add check for package version being a string
Ensure that package versions are always strings and not numbers,
as might happen with versions like '1.2' and '20230102'.
2023-12-03 15:39:25 +01:00
Silke Hofstra
aa3b66e09c ci/package_checks: fix SPDX license check for WITH/AND/OR
SPDX license identifiers may be a 'simple' or 'compound' expression,
meaning that the following formats are also allowed:

- A document reference instead of license identifier.
- `+` after the license identifier.
- License identifier followed `WITH` and an exception identifier.
- The above split by `AND` or `OR`.
- The above contained between `()`.

All but the first case should now be valid according to the package checks.
The first case is still invalid, as we want license identifiers and not document references in our packages.

See: https://spdx.github.io/spdx-spec/v2.3/SPDX-license-expressions/

Resolves #956
2023-12-03 12:56:43 +01:00
Silke Hofstra
135aa60d44 ci/package_checks: fix dependency order for pkgconfig32
Fix `pkgconfig32` entries not being sorted in a separate section
by sorting them *before* `pkgconfig` (as that is currently most common).

Resolves #921
2023-12-03 12:34:25 +01:00
Silke Hofstra
a240fb46cb ci/package_checks: add check for system.base/devel dependencies 2023-11-27 20:38:03 +01:00