Once you're done writing you app package, you'll want to check that everything works correctly. At first, you can manually try to install your app on some test server of your own :
sudo yunohost app install ./path/to/mycustomapp_ynh
Testing everything manually often becomes tedious ;). The YunoHost maintains several tools to automatically and somewhat "objectively" analyze and tests our 400+ apps catalog.
The package linter performs a static analysis of your app to check a bunch of recommended practice.
It is pretty straightforward to run considering that you should only need Python installed.
Package check is a more elaborate software that will tests many scenarios for you app such as:
Package check will then summarize the results and compute a quality level ranging from 0 to 8.
To run its tests, package check uses a LXC container to manipulate the package in a clean environment without any previous installations. Installing LXC/LXD is unfortunately not that straighforward (it kinda depends on your OS/distribution), though we documented how to do so in the README.
Package check interfaces with your app's
tests.toml which allows to control a few things about which tests are run - though it is usually pretty empty as many things can be deduced from the app's manifest.
More info about the syntax here are also available in the README
Package check will compute a quality level ranging from 0 to 8.
Apps with level equal or lower than 4 are considered "bad quality" and YunoHost will discourage people from installing such apps.
While this definition may vary with time, the current definition as of February 2023 is roughly:
The YunoHost project also developed an interface called
yunorunner which interfaces with
package_check, handles a job queue, and automatically add jobs to the queue using some triggers.
The two major ones are:
package_check yourself, and has the advantage of the results being automatically public, which facilitates collective debugging.
Members of the YunoHost-Apps organization can trigger jobs on the dev CI directly from a pull request simply by commenting something like
!testme (cf for example here). A .png summary of the tests will be automatically displayed once the job completes (and you can click the link to see the entire job execution and debug it).
Constrain 1 : Gitlab-CI or other similar solutions are mostly based around Docker, while we use LXC. In particular, we do want to reuse LXC snapshots of successful install during other tests (upgrade, backup/restore, ..) rather than reinstalling the app from scratch everytime, which drastically reduces the test time. We could do so using Gitlab artifacts, but such artifacts are automatically made public which is not convenient because they contain a full filesystem and their only use it to speed up the test process. Moreover, in the Gitlab-CI paradigm, jobs are not running on the same machine and they would need to download the snapshot which can be lengthy. The other mechanism, caching, is explicitly advertised as not reliable in Gitlab's-CI doc. What would be helpful would be some non-public artifact system (see similar discussion here)
Constrain 2 : Our test workflow is somewhat complex and we have 400+ apps to maintain. It's not acceptable to have to duplicate/maintain the CI logic in app each repository, so we need some sort of repo that handles the CI logic, while still being to supersed some behavior from the app repo itself. Gitlab-CI has some syntax which can allow this, but this remains quite laborious.
Constrain 3 : Having a common job queue/dashboard UI accross all app. In the Gitlab-CI paradigm, each repository has its own job queue/dashboard UI, but this is not really convenient.
Found errors? Think you can improve this documentation? Simply click the Edit link at the top of the page, and then the icon on Github to suggest changes.