Recalbox development (add emulator filesystem)

Since https://github.com/recalbox/recalbox-buildroot/pull/652, the process to generate the es_systems.cfg + the share_init/roms folder is now automated if you follow this guide.
A new super package has been created : recalbox-romfs. This package will generate in the $(TARGET_DIR) the /recalbox/share_init/roms directory structure + files inside, as well as the system entry for ES. Let's look through the whole process in details !
We'll use 2 examples : a single emulator one (PPSSPP), and a multi core one (SNES), and will conclude with a mix : several emulators with several cores for 1 system.
​
Write your emulator package
​
That is mandatory, needless to say. This guide works the same for libretro cores. PPSSPP : nothing particular, it's a single single standalone emulator SNES : we have, at the moment i'm writing this wiki, 4 cores (catsfc, pocketsnes, snes9x_next, snes9x) for 1 emulator (retroarch)
​
Use the helper script : empack.py
​
To make things easier, you can (and should) use scripts/linux.empack.py. Here is the help:
usage: empack.py [-h] -s SYSTEM -e EXTENSIONS [-f FULLNAME] [-p PLATFORM]
                 [-t THEME]
                 packageDetails [packageDetails ...]
empack must be run from the top of your repo as it will check the required tree structure below.
Switches between brackets are not mandatory. If you don't specify them, they will default to the system name.
Here is a small table showing the long name of each switch:
Short name
Long name
-s
--system
-e
--extension
-f
--fullname
-p
--platform
-t
--theme
​
The easy case
The psp recalbox-romfs was generated using : ./scripts/linux/empack.py -s psp -e '.iso .ISO .cso .CSO' -f 'Sony Playstation Portable' -p psp -t psp BR2_PACKAGE_PPSSPP Let's break it down
-s psp that's the ES system name, giving the name to the roms folder and the ES system name
-e '.iso .ISO .cso .CSO' are the extensions the emulator handles
-f 'Sony Playstation Portable' is the full name of the system
-p psp is the platform name (that's an ES specific thing for scraping)
-t psp is the name of the theme to be applied in ES
BR2_PACKAGE_PPSSPP is the buildroot variable of the PPSSPP package
Here is the output:
                system: psp
                extensions: .iso .ISO .cso .CSO
                fullname: Sony Playstation Portable
                platform: psp
                theme: psp
                singleMode: True
                single emulator package: BR2_PACKAGE_PPSSPP
                multi emulator packages: {}
                ------
                list of BR package variables: ['BR2_PACKAGE_PPSSPP']
​
== Creating new package dir structure: OK !
== Writing package/recalbox-romfs/recalbox-romfs-psp/recalbox-romfs-psp.mk : OK !
== Writing the Config.in:  OK!
== Copy the previous fsoverlay of this system if it exists: No overlay, creating default files instead: OK !
​
Now you will have to edit :
​
  * package/recalbox-romfs/recalbox-romfs/recalbox-romfs.mk :
    Mind the tabulation. The shell may have added spaces instead
​
# System: psp
ifeq ($(BR2_PACKAGE_PPSSPP),y)
    RECALBOX_ROMFS_DEPENDENCIES += recalbox-romfs-psp
endif
​
  * Add a source to package/recalbox-romfs/Config.in :
 source package/recalbox-romfs/recalbox-romfs-psp/Config.in
​
  * Add dependencies to package/recalbox-romfs/recalbox-romfs/Config.in :
​
      || BR2_PACKAGE_RECALBOX_ROMFS_PSP
​
  * Add to the emulators/cores Config.in:
​
    select BR2_PACKAGE_RECALBOX_ROMFS_PSP
You should now do what you're asked to, and do not forget to edit your emulator's Config.in to make sure it selects its recalbox-romfs-. This dependancy is extremly important so that there is no action made in the menuconfig.
​
Now lets look in depth the recalbox-romfs-psp package:
the package is located in package/recalbox-romfs/recalbox-romfs-psp
empack also created the roms/psp subfolder there, with some simple readme's you should edit. If you want to add some free roms, you should put them there.
the xml part will be generated with the .mk file when buildroot will make recalbox-romfs
Once Buildroot will make recalbox-romfs-psp here are the results of the package :
$ ls -lR output/build/recalbox-romfs-psp
output/build/recalbox-romfs-psp:
total 8
-rw-r--r-- 1 subs subs  410 janv. 25 17:59 psp.xml
drwxr-xr-x 3 subs subs 4096 janv. 25 17:59 roms
​
output/build/recalbox-romfs-psp/roms:
total 4
drwxr-xr-x 2 subs subs 4096 janv. 25 17:59 psp
​
output/build/recalbox-romfs-psp/roms/psp:
total 8
-rw-r--r-- 1 subs subs 128 janv. 25 17:59 _lisezmoi.txt
-rw-r--r-- 1 subs subs 137 janv. 25 17:59 _readme.txt
$ cat output/build/recalbox-romfs-psp/psp.xml
<system>
     <fullname>Sony Playstation Portable</fullname>
     <name>psp</name>
     <path>/recalbox/share/roms/psp</path>
     <extension>.iso .ISO .cso .CSO</extension>
     <command>python /usr/lib/python2.7/site-packages/configgen/emulatorlauncher.pyc %CONTROLLERSCONFIG% -system %SYSTEM% -rom %ROM% -emulator %EMULATOR% -core %CORE% -ratio %RATIO%</command>
     <platform>psp</platform>
     <theme>psp</theme>
 </system>
See ? the roms folder was copied, and the package generated the required part for es_systems.cfg. Do you understand now the meaning of each switch of empack ? :smile:
​
A more complex case
We mentionned earlier snes as it is a more complex case. Its package was generated with ./scripts/linux/empack.py -s snes -e '.smc .sfc .SMC .SFC .zip .ZIP .mgd .MGD' -f 'Super Nintendo Entertainment System' -p snes -t snes libretro:catsfc:BR2_PACKAGE_LIBRETRO_CATSFC libretro:pocketsnes:BR2_PACKAGE_LIBRETRO_POCKETSNES libretro:snes9x_next:BR2_PACKAGE_LIBRETRO_SNES9X_NEXT libretro:snes9x:BR2_PACKAGE_LIBRETRO_SNES9X  Here is the output:
                system: snes
                extensions: .smc .sfc .SMC .SFC .zip .ZIP .mgd .MGD
                fullname: Super Nintendo Entertainment System
                platform: snes
                theme: snes
                singleMode: False
                single emulator package: 
                multi emulator packages: {'libretro': {'snes9x_next': 'BR2_PACKAGE_LIBRETRO_SNES9X_NEXT', 'catsfc': 'BR2_PACKAGE_LIBRETRO_CATSFC', 'snes9x': 'BR2_PACKAGE_LIBRETRO_SNES9X', 'pocketsnes': 'BR2_PACKAGE_LIBRETRO_POCKETSNES'}}
                ------
                list of BR package variables: ['BR2_PACKAGE_LIBRETRO_CATSFC', 'BR2_PACKAGE_LIBRETRO_POCKETSNES', 'BR2_PACKAGE_LIBRETRO_SNES9X_NEXT', 'BR2_PACKAGE_LIBRETRO_SNES9X']
​
== Writing the Config.in:  OK!
== Creating new package dir structure: package/recalbox-romfs/recalbox-romfs-snes/roms/snes already exists ... Are you sure of what you're doing ? Exiting ...
Adding emulator snes libretro
== Writing package/recalbox-romfs/recalbox-romfs-snes/recalbox-romfs-snes.mk : OK !
== Copy the previous fsoverlay of this system if it exists: No overlay, creating default files instead: OK !
​
Now you will have to edit :
​
  * package/recalbox-romfs/recalbox-romfs/recalbox-romfs.mk :
    Mind the tabulation. The shell may have added spaces instead
​
# System: snes
ifneq ($(BR2_PACKAGE_LIBRETRO_CATSFC)$(BR2_PACKAGE_LIBRETRO_POCKETSNES)$(BR2_PACKAGE_LIBRETRO_SNES9X_NEXT)$(BR2_PACKAGE_LIBRETRO_SNES9X),)
    RECALBOX_ROMFS_DEPENDENCIES += recalbox-romfs-snes
endif
​
  * Add a source to package/recalbox-romfs/Config.in :
 source package/recalbox-romfs/recalbox-romfs-snes/Config.in
​
  * Add dependencies to package/recalbox-romfs/recalbox-romfs/Config.in :
​
      || BR2_PACKAGE_RECALBOX_ROMFS_SNES
​
  * Add to the emulators/cores Config.in:
​
    select BR2_PACKAGE_RECALBOX_ROMFS_SNES
Let's not go through every parameters with command line switches, but focus on what really changed.
For example libretro:pocketsnes:BR2_PACKAGE_LIBRETRO_POCKETSNES means "emulator libretro has a core called pocketsnes and this core's buildroot variable is BR2_PACKAGE_LIBRETRO_POCKETSNES". In fact, if you know how the es_systems.cfg looks like, it's pretty easy.
Mind the syntax : emulator:core:variable each field being seperated by a colon :
Once empack finished its job, it made the recalbox-romfs-snes package so that libretro can handle 4 different cores. Here is the result of the package
$ ls -lR output/build/recalbox-romfs-psp
output/build/recalbox-romfs-psp:
total 8
-rw-r--r-- 1 subs subs  410 janv. 25 17:59 psp.xml
drwxr-xr-x 3 subs subs 4096 janv. 25 17:59 roms
​
output/build/recalbox-romfs-psp/roms:
total 4
drwxr-xr-x 2 subs subs 4096 janv. 25 17:59 psp
​
output/build/recalbox-romfs-psp/roms/psp:
total 8
-rw-r--r-- 1 subs subs 128 janv. 25 17:59 _lisezmoi.txt
-rw-r--r-- 1 subs subs 137 janv. 25 17:59 _readme.txt
subs@recalbuntu:~/git/recalbox-build-pi3$ ls -lR output/build/recalbox-romfs-snes
output/build/recalbox-romfs-snes:
total 8
drwxr-xr-x 3 subs subs 4096 janv. 25 17:59 roms
-rw-r--r-- 1 subs subs  619 janv. 25 17:59 snes.xml
​
output/build/recalbox-romfs-snes/roms:
total 4
drwxr-xr-x 2 subs subs 4096 janv. 25 17:59 snes
​
output/build/recalbox-romfs-snes/roms/snes:
total 264
-rw-r--r-- 1 subs subs 262144 janv. 25 17:59 DonkeyKongClassic (Shiru).smc
-rw-r--r-- 1 subs subs    120 janv. 25 17:59 _lisezmoi.txt
-rw-r--r-- 1 subs subs    120 janv. 25 17:59 _readme.txt
$ cat output/build/recalbox-romfs-snes/snes.xml
<system>
     <fullname>Super Nintendo Entertainment System</fullname>
     <name>snes</name>
     <path>/recalbox/share/roms/snes</path>
     <extension>.smc .sfc .SMC .SFC .zip .ZIP .mgd .MGD</extension>
     <command>python /usr/lib/python2.7/site-packages/configgen/emulatorlauncher.pyc %CONTROLLERSCONFIG% -system %SYSTEM% -rom %ROM% -emulator %EMULATOR% -core %CORE% -ratio %RATIO%</command>
     <platform>snes</platform>
     <theme>snes</theme>
     <emulators>
        <emulator name="libretro">
            <cores>
                <core>snes9x_next</core>
                <core>catsfc</core>
                <core>pocketsnes</core>
                <core>snes9x</core>
            </cores>
        </emulator>
    </emulators>
 </system>
See how the cores where assembled together inside the emulator ?
​
empack 9th Dan level
Now here is a more complex example of what empack can handle. Mame has 2 emulators, one has 3 cores : ./scripts/linux/empack.py -s mame -e '.zip .ZIP' -f 'Mame' -p mame -t mame libretro:mame078:BR2_PACKAGE_LIBRETRO_MAME2003 libretro:imame4all:BR2_PACKAGE_LIBRETRO_IMAME advancemame:advancemame:BR2_PACKAGE_ADVANCEMAME 
So here, advncemame being a standalone emulator, we have to use the emulator/core syntax, otherwise empack can't handle a mix of multiple cores for 1 emulator + 1 standalone. But this has no impact for ES. You could eventually edit the .mk to simplify the xml generation for advancemame, but this is definitely not mandatory
​
Putting it all together
​
Let's now talk about the interactions between emulator packages, recalbox-romfs- and the main recalbox-romfs package.
​
When you ran empack, you were asked ot do a few edits in at least 4 different files. What you did:
recalbox-romfs will depend on recalbox-romfs- only if one of the cores/emulators are compiled
the general Config.in will display the selection screen of the recalbox-romfs system packages
add another dependency for the config menu between the main recalbox-roms and the recalbox-romfs-
make sure the emulator/cores that are necessary for the system select the system recalbox-romfs- package
This way, defconfigs don't need to be modified for emulators, Buildroot follows the dependencies links and builds the expected packages.
​
At the end, you'll find:
ex_systems.cfg in output/target/recalbox/share_init/system/.emulationstation
the roms folder in output/target/recalbox/share_init/roms
​
Pro tips
​
read any package made by empack, you will see the full syntax used to generate the .mk
say you want tos add a mame2010 core to the mame system : find the command line that generated the current recalbox-romfs-mame (see previous tip) and add a new parameter. Despite empack won't erase an existing package (just delete it), empack will generate a new package with your new core. Edits are shorter as you just need to do the emodifications linked to your new core