Yunohost app list migration
Alex from Yunohost told us (@ilja and I) that the release of Yunohost 3.8 would make some breaking changes on how the package lists are managed. In short, it won't be possible to manage a custom app list from the web interface or from the cli. The custom app lists that are currently installed on cubes will be removed.
Here is what Alex told us:
The thing is that there's a big refactoring to the app list management in 3.8 ... Mostly to integrate app categories (which required a change in the way we fetch app list) and because I got fed up of the underlying code that was a total mess.
Also the "multiple app catalog" seemed like a big YAGNI after complaining so many times about it ended up removing it because nobody seemed to use it. It's still available but requires to manually add a line in a .yml file in /etc/yunohost/.
Yunohost is also more and more discouraging user to install "bad quality" apps (c.f. the level key in the json app catalog). In particular anything below level 4 is considered bad quality. I think that's already the case in 3.7 ?
Anyway, as said a lot of things would be simplified (well, requires a bit of work but I can help) if :
- neutrinet_ynh goes into the "official" app catalog
- but that also means that neutrinet will be tested on the app CI, and it should be level 4 to be easily installable in the app list (otherwise users gotta ask to see 'all apps and not just decent quality' ones)
- we can run some tests about the package on our dev ci to see what level it goes up to
- but I can already say that our package_linter.py will block level 4 currently. There are some errors that are easy to fix. Warnings won't block level 4 but ideally they should still be taken care of as much as possible ...
- it would help if the main branch you work on is "master" (or an alias can be created to stable, idk)
- once everything is done, you're still independant in the management of the app (nothing to push on apps.json), the only thing that changes is that the CI may warn you about failing tests
(hopefully this makes some sense)
It followed some questions from our part.
What exactly do you mean with the "multiple app catalog"?
The "Custom applications list"
Because of the breaking changes required by 3.8 to have app categories, the format of the apps.json changes.
And because it was assumed that nobody was using custom app list anymore (e.g. La Brique was doing so in the past but not anymore) the 3.8 will agressively resets the app list to just keep Yunohost's default app list ... Now that I think about it and that I found somebody using that system, I could try to implement some compatibility mode for older app lists format but meh...
dev list for testing purpose. What is the workflow for testing within the official app list?We started using a
The usual practice is to tell people to install or upgrade using the testing branch. C.f. an example here for nextcloud : https://github.com/YunoHost-apps/nextcloud_ynh#developers-infos (notice the /tree/testing at the end). Looking at the code in the core, same thing should also work for non-github repo (even though the url goes nowhevere, it's just a quick-and-dirty parsing)
As you know, we are using a Gitlab repo, with a mirror to Github. Could we keep working on Gitlab, or is Github required to work with the list?
No worries, any git[whatever] is okay as long as repo can be cloned. There already are a couple of apps hosted on framagit or other forges in the list
We were thinking about using the CI tool provided by Ynh, so that's good :) Do we need to write tests somehow? How is it working?
I'll start a test on the dev CI to see how that looks. (Btw you can probably ask Maniack for an access too, just need an SSH key, c.f. https://yunohost.org/#/packaging_apps_ci )
I think by default it magically finds some dummy default values to use ... But usually project set up a "check_process" file in the repo. The syntax is a bit funky but for example you can look at : https://github.com/YunoHost-Apps/anarchism_ynh/blob/testing/check_process (or any other app that has this file). I see that neutrinet_ynh only ask for a domain + path so maybe not even needed ? Full doc is here : https://github.com/YunoHost/package_check#syntax-of-check_process
For when is planned the 3.8? In other words, how much time do we have?
There are many fixes to do, at least another testing iteration to validate everything, so would say a good 2~3 weeks.
Is it still possible to add a custom app list?
Adding a custom list now would look like:
echo "- id: neutrinet url: https://.../neutrinet_ynh_list" >> /etc/yunohost/apps_catalog.yml
(assuming the actual json is at https://.../neutrinet_ynh_list/v2/apps/json)
yunohost tools update --apps to refresh the app catalogs cache.
If you wish to build and maintain your own app list, you can have a look at: https://github.com/YunoHost/apps/
Basically you "just" need a cron job that calls rebuild.sh, itself calling list_builder.py
Maybe just a few things to tweak for everything to be smooth on your side
What will happen with the instances that already have the applist?
The applists will be gone completely, which means you'll have to figure out a way to get the applist back (or better, the new applist in).
The big question is how to properly handle the upgrade so that everything stays consistent... We could also just have some ad-hoc piece of code in Yunohost that says "if there's the neutrinet list, replace it with the new neutrinet list"
Is it possible to just change our applist to the newer format?
Note that you need to keep serving both the old version and the new version (it's what we already do as you can notice in https://app.yunohost.org/ ... apps.json is the current version (pre-3.8) and default/v2/apps,json is the new version (post 3.8)
In fact you could generate the new version easily by adding a top-level "apps:" key and an empty "categories:" key ... which is easier on the short-term. But on the long-term it might be worth to setup a real list_builder.py + cron job if we do improvements or other migrations in the future...
Also there's in fact a yunohost app to setup the app list system (yup, appception) : https://github.com/YunoHost-Apps/app_ynh_core but I recently yolo-commited without testing, but that can be used as partial doc :/
The question is whether it's easier/better for us to move to the official app list, or still keep our own.
We formulated some concerns to Alex:
- On not allowing custom app-lists: Although we understand not wanting people to think badly of ynh based on some bad apps that aren't even really part of the project, and we even think that's a valid concern! But this is also a step towards centralization and we dislike that on principle alone... Ynh is already too depended on central services (github, but also things like ip.yunohost.org...), removing the possibility for custom applists just adds to that I think. But that's maybe not relevant here...
- On timing and on the breaking change itself: There are ynh instances out there with this app list, these should not break. For us this wouldn't be too much of a problem, there aren't big changes planned for the app (we recently did a whole rewrite, but things should be more stable now)
- On the removal of custom app lists already installed: We COULD add the list back from the app, but if people upgrade ynh before they can upgrade the app then that's a problem...
- On the CI tool: The app needs an installed and working VPN, so the CI tool might fail (which doesn't mean the app is uninstallable)
We asked on Mattermost what should we do. The current consensus is (but this could still change ofc):
The saner approach would be to move to the Yunohost app list. Maintaining a whole list for only one app is quite overkill... Later, if we feel like it, we could upgrade our old list to make it compliant with the new app list system and move back in our own app list.