Continuous integration testing of TeX Live sources

The TeX Live sources consists in total of around 15000 files and 8.7M lines (see git stats). It integrates several upstream projects, including big libraries like FreeType, Cairo, and Poppler. Changes come in from a variety of sources: external libraries, TeX specific projects (LuaTeX, pdfTeX etc), as well as our own adaptions and changes/patches to upstream sources. Since quite some time I wanted to have a continuous integration (CI) testing, but since our main repository is based on Subversion, the usual (easy, or the one I know) route via Github and one of the CI testing providers, didn’t come to my mind – until last week.

Over the weekend I have set up CI testing for our TeX Live sources by using the following ingredients: git-svn for checkout, Github for hosting, Travis-CI for testing, and a cron job that does the connection. To be more specific:

• git-svn I use git-svn to check out only the source part of the (otherwise far to big) subversion repository onto my server. This is similar to the git-svn checkout of the whole of TeX Live as I reported here, but contains only the source part.
• Github The git-svn checkout is pushed to the project TeX-Live/texlive-source on Github.
• Travis-CI The CI testing is done in the TeX-Live/texlive-source project on Travis-CI (who are offering free services for open source projects, thanks!)

Although this sounds easy, there are a few stumbling blocks: First of all, the .travis.yml file is not contained in the main subversion repository. So adding it to the master tree that is managed via git-svn is not working, because the history is rewritten (git svn rebase). My solution was to create a separate branch travis-ci which adds only the .travis.yml file and merge master.

Travis-CI by default tests all branches, and does not test those not containing a .travis.yml, but to be sure I added an except clause stating that the master branch should not be tested. This way other developers can try different branches, too. The full .travis.yml can be checked on Github, here is the current status:

# .travis.yml for texlive-source CI building
# Norbert Preining
# Public Domain

language: c

branches:
  except:
  - master

before_script:
  - find . -name \*.info -exec touch '{}' \;

before_install:
  - sudo apt-get -qq update
  - sudo apt-get install -y libfontconfig-dev libx11-dev libxmu-dev libxaw7-dev

script: ./Build

What remains is stitching these things together by adding a cron job that regularly does git svn rebase on the master branch, merges the master branch into travis-ci branch, and pushes everything to Github. The current cron job is here:

#!/bin/bash
# cron job for updating texlive-source and pushing it to github for ci
set -e

TLSOURCE=/home/norbert/texlive-source.git
GIT="git --no-pager"

quiet_git() {
    stdout=$(tempfile)
    stderr=$(tempfile)

    if ! $GIT "$@" $stdout 2>$stderr; then
	echo "STDOUT of git command:"
	cat $stdout
	echo "************"
        cat $stderr >&2
        rm -f $stdout $stderr
        exit 1
    fi

    rm -f $stdout $stderr
}

cd $TLSOURCE
quiet_git checkout master
quiet_git svn rebase
quiet_git checkout travis-ci
# don't use [skip ci] here because we only built the 
# last commit, which would stop building
quiet_git merge master -m "merging master"
quiet_git push --all

With this setup we can CI testing of our changes in the TeX Live sources, and in the future maybe some developers will use separate branches to get testing there, too.

Enjoy.