TeX Live: the new multi fmtutil
Some time ago I posted a few articles on internals of TeX Live, which might be of interest for the general user, too: TeX Live internals and Internals of TeX Live 2: multi-updmap. With TeX Live 2014 frozen and the pretest for 2015 about to start, I thought I give a write up of the one big internal change, namely that fmtutil has gone “multi” in the same way updmap did this 2 years ago.
Where it all began…
In the very beginning of the comuting history, when memory was scarce, computers slow, and Knuth went forth to create the most advanced type-setting system, he devised a way to speed things up and at the same time preserve space: format dumps. I will not go into details what they are and how they are made, but you can think of them as dumps of the state of the program (TeX, Metafont,…) after the (painful) initialization, that can be easily and quickly loaded and used as new starting point.
Of course, when there was only one TeX program and one Metafont program, managing these format dumps was a rather easy task, but over time many things have changed: different programs, different formats, various additions for internationalization. Up to the point that nowadays very few people actually know what is going on when “formats are rebuild” or how these pieces play together.
Comes into play .. fmtutil
What is fmtutil
Written long time ago by Thomas Esser for he teTeX, this script and its accompanying configuration file allows to define formats in a line based configuration file, and rebuild them in various ways. This script has served the TeX community for many many years. The head of the shell script mentiones as first change in 2001, but I am sure that the actual script is much older.
To give you an idea how a typical line in the configuration file looks like, let us look at the format definition for LaTeX as we do have it in TeX Live:
latex pdftex language.dat -translate-file=cp227.tcx *latex.ini
It tells us that the LaTeX format should be build with the pdftex engine, that the command line options for the dump process are -translate-file=cp227.tcx *latex.ini, and that (indirectly via latex.ini) the file language.dat is used for hyphenation pattern definitions.
We can tell fmtutil (the program) to rebuild the format as follows (we use fmtutil-sys here out of various reasons, mostly that we don’t want people to run this command in real and then have a bound to be broken installation 😉
$ fmtutil-sys --byfmt latex
fmtutil: running `pdftex -ini -jobname=latex -progname=latex -translate-file=cp227.tcx *latex.ini' ...
This is pdfTeX, Version 3.14159265-2.6-1.40.15 (TeX Live 2014) (INITEX)
...[lots of output here]
Transcript written on latex.log.
fmtutil: /home/norbert/tl/2014/texmf-var/web2c/pdftex/latex.fmt installed.
fmtutil: No errors, exiting successfully.
$
Other possible options to rebuild formats are --all, --by-engine ENGINE etc, but see later for details.
Format of the configuration file
Returning to the definition of LaTeX, the format of a configuration line consists of four fields:
NAME ENGINE HYPHENFILE CMDLINE-ARGS
Where NAME is the name of the format, as it will be normally called when executed as program. Examples are latex, pdflatex, lollipop.
The second field ENGINE gives the executable, the engine in TeX jargon, that is used to create and load the dump. Typical engines are tex, pdftex, mf, or platex.
The third fields specifies the hyphenation patterns. This file is normally not loaded directly, but via the initialization file given in the fourth argument. It is kept here so that updates to formats based on hyphenation patterns can be executed. Typical values are language.dat (for LaTeX like formats), language.def (for eTeX like formats), and language.def.lua (for luatex based formats).
The most complex field makes up the rest of the line, the command line options to pass to the engine. Furthermore, it is important that the last part of the command line arguments is the name of the file that is to be run through the initialization engine. Typical examples of things appearing in the command line arguments are translation directives for multilingualization (-translate-file=cp227.tcx), switches to turn on extra functionality (-etex), and of course the ini-file, often prefixed with a *, which also turns on extended capabilities (*pdflatex.ini).
Operation mode
The new operation mode, introduced in TeX Live 2015, now reads all fmtutil.cnf files found, i.e., all the files given by
kpsewhich -all fmtutil.cnf
in contrast to the former operation mode of only reading the first found.
Reasons to introduce this mode are various, and similar (or the same) to what I stated when switching to multi-updmap:
- Configuration of available formats can be put into the tree where the format files are located.
In particular, additions and adaptions to formats will automatically survive a new installation if they are located in TEXMFLOCAL‘s fmtutil.cnf, which will be read together with the new ones. - Users without write permission to the system installation can still have their own formats, and do not need to have a copy of the system’s fmtutil.cnf.
Using all fmtutil.cnf files means that the following files – if they exist – are used (under the assumption of a standard TeX Live installation, where
System mode (fmtutil-sys)
| TEXMFSYSCONFIG | $ROOT/YYYY/texmf-config/web2c/fmtutil.cnf |
| TEXMFSYSVAR | $ROOT/YYYY/texmf-var/web2c/fmtutil.cnf |
| TEXMFLOCAL | $ROOT/texmf-local/web2c/fmtutil.cnf |
| TEXMFDIST | $ROOT/YYYY/texmf-dist/web2c/fmtutil.cnf |
User mode (fmtutil)
| TEXMFCONFIG | $HOME/.texliveYYY/texmf-config/web2c/fmtutil.cnf |
| TEXMFVAR | $HOME/.texliveYYYY/texmf-var/web2c/fmtutil.cnf |
| TEXMFHOME | $HOME/texmf/web2c/fmtutil.cnf |
| TEXMFSYSCONFIG | $ROOT/YYYY/texmf-config/web2c/fmtutil.cnf |
| TEXMFSYSVAR | $ROOT/YYYY/texmf-var/web2c/fmtutil.cnf |
| TEXMFLOCAL | $ROOT/texmf-local/web2c/fmtutil.cnf |
| TEXMFDIST | $ROOT/YYYY/texmf-dist/web2c/fmtutil.cnf |
Adding and overriding formats
By layering these fmtutil.cnf files on top, system administrators and users can add or override formats to their needs. Adding a local (system-local) format can easily be done by adding a line in the (otherwise empty) fmtutil.cnf file in TEXMFLOCAL/web2c, and running fmtutil-sys. In the same way a user can add his personal format (although we are not sure whether there is someone out there who wants to do that!).
Disabling, or overriding a format can be achieved in a very similar way, by either re-defining it – here the keys are that both the NAME and ENGINE part need to agree with the ones from the format to be overridden. Disabling can also be done, if necessary, by prefixing an override line (or the same line) with #! in a higher priority configuration file.
As an example, assume I want the mltex format to be build with a specifically crafted ini-file my-mltex.ini. What I need to do is add the following line to TEXMFLOCAL‘s fmtutil.cnf
mltex pdftex - -translate-file=cp227.tcx -mltex my-mltex.ini
After that, the mltex format will be built with my own my-mltex.ini file.
In the same way, to disable it completely, I could add the line:
#! mltex pdftex - -translate-file=cp227.tcx -mltex mltex.ini
and future runs to fmtutil will ignore this format.
Enabling and disabling formats can also be done via the command line options --enablefmt and --disablefmt, but new formats need to be defined by hand. In case of usage of --enablefmt and --disablefmt, fmtutil decides as follows where to save the changes:
If config files are given on the command line, then the first one given will be used to save any changes from --enablefmt or --disablefmt. If the config files are taken from kpsewhich output, then the algorithm is more complex:
- If
$TEXMFCONFIG/web2c/fmtutil.cnfor$TEXMFHOME/web2c/fmtutil.cnfappears in the list of used files, then the one listed first bykpsewhich --all(equivalently, the one returned bykpsewhich fmtutil.cnf), is used. - If neither of the above two are present and changes are made, a new config file is created in
$TEXMFCONFIG/web2c/fmtutil.cnf.
In general, the idea is that if a given config file is not writable, a higher-level one can be used. That way, the distribution’s settings can be overridden for system-wide using TEXMFLOCAL, and then system settings can be overridden again for a particular using using TEXMFHOME.
Running fmtutil as user (instead of system administrator)
If a user (any) runs fmtutil (not fmtutil-sys) once, local copies of the format dumps are created, shadowing the system files. After an update of the formats on the system, the user would still run his own old formats. To update them, he has to re-run fmtutil, but no further configuration is necessary. Before TL2015 it was necessary to update also the fmtutil.cnf file of the user.
Recommendations
For system administrators or anyone who used the former fmtutil-local.cnf way of configurating the fmtutil.cnf file, please convert to the new format. The only thing changed is the format of the disable lines:
old: #!formatname
new: #! formatname engine ....
For single user installations we recommend installation as normal user, putting additional formats into TEXMFLOCAL, and always run the -sys variants.
fmtutil invocation
We conclude with a list of ways to invoce the fmtutil program. In principle the invocation options are the same as the ones from the original shell script, with some changes:
--cnffile FILEreadFILEinstead offmtutil.cnf, can be given multiple times, in which case all the files are used. The following options adjust the behaviour of the program:--fmtdir DIRECTORY--no-engine-subdirdon’t use engine-specific subdir of the fmtdir--no-error-if-no-formatexit successfully if no format is selected--no-error-if-no-engine=ENGINE1,ENGINE2,...exit successfully even if the required engine is missing, if it is included in the list.--quietbe silent--test(not implemented, just for compatibility)--dolinks(not implemented, just for compatibility)--force(not implemented, just for compatibility)
They need to be followed by one valid command from the following list:
--allrecreate all format files--missingcreate all missing format files--refreshrecreate only existing format files--byengine ENGINENAME(re)create formats usingENGINENAME--byfmt FORMATNAME(re)create format forFORMATNAME--byhyphen HYPHENFILE(re)create formats that depend onHYPHENFILE--enablefmt FORMATNAMEenable formatname in config file--disablefmt FORMATNAMEdisable formatname in config file--listcfglist (enabled and disabled) configurations, filtered to available formats--catcfgoutput the content of the config file--showhyphen FORMATNAMEprint name of hyphenfile for formatFORMATNAME--versionshow version information and exit--helpshow this message and exit
That’s it. I hope that this will help the one or other system administrator, at least my own Debian packaging will become a bit simpler!
Enjoy!
Hello,
since updating from TeXLive 2015 to 2016, the packages mylatex and mylatexformat do not work any more. Instead, when adding certain fonts (such as Charter), there is an error message
! Undefined control sequence.
\mdseries@tt
However, the document compiles fine otherwise. Is this related to fmtutil? Neither fmtutil –all nor fmtutil-sys –all changed anything though.
What did fail? Generation of the format or using the generated format file?