test-master

test-master is a local-only test setup for the elmo build infrastructure.

Take-Aways

This document is a follow-up to the a10n test setup. You'll see how to:

Configure master-ball/test-master
Set up the master
Set up the slave
Run the bots
Watch the results

There are going to be several shell terminals and virtualenvs throughout this page. In each command, the shell prompt indicates the activated virtualenv, if so, and the directory you should be in. That is, a

ls

is a ls in the master-ball directory with the @master virtualenv activated.

Requirements

All the requirements to set up test-master should already be there once you've set up the a10n test environment.

Install and Configure test-master

For the general master setup, you want to follow the instructions for l10n-master, but you'll put your local.py into the test-master directory.

Your test-master/local.py should look somewhat like this:

from base_settings import *
DATABASES = {
    'default': {} # you know this stuff
}
REPOSITORY_BASE = '/my/home/stage/repos/'
buildbase = '/my/home/stage/builds/'
ES_COMPARE_HOST = 'localhost:9200'
ES_COMPARE_INDEX = 'elmo-comparisons'

SECRET_KEY = 'Do Not Tell Me'

The REPOSITORY_BASE points to the repos subdirectory in your test environment, and buildbase is where buildbot will store logs and other build data. We'll use the same directories when in settings/mounts_local.py when setting up elmo.

Master setup

As you're in a new shell now, and we'll want to create a virtualenv for master now..

virtualenv @master
. @master/bin/activate
pip install -r requirements.txt

For the stuff that follows, you should generally have that on. You'll see by the (@master) prefix in the code snippets which lines are in the env for sure.

We're working in test-master , so let's go there, and create the master.

cd test-master
../scripts/buildbot create-master .

You only need to do that once. Next we need the local.py with the following contents, plus your common database setup:

from base_settings import *
DATABASES = {
    'default': {} # you know this stuff
}
REPOSITORY_BASE = '/my/home/stage/repos/'
buildbase = '/my/home/stage/builds/

SECRET_KEY = 'Do Not Tell Me'

That's all. Next you want to set up a buildbot slave, with slave-ball. You want to set that up with name and password from slaves.json.

Elasticsearch Index

You need to have an elasticsearch instance running. The documentation here assumes that you're running it on `localhost` and the standard `9200` port.

For the initial setup, you need to create an Index, where elasticsearch (ES in short) stores its data.

To do so, head over to your elmo install, activate the virtualenv there, and run

./manage.py createcompareindex

Slave setup

The slave setup is the very same for test-master and l10n-master, so pick that up quickly from the slave-ball doc.

Running the bots

The first time you run the bots, you may want to run the get-pushes daemon before the build bots. Usually, it's the last bot to start, though.
This is to get the initial push to en-US into the database without triggering a build for it, which is closer to what happens in reality.

There are 8 services involved in running the dashboard, the elmo webserver, the mercurial web server, elasticsearch, the buildbot master and slave, rabbitmq, a10n hg, and the twistd get-pushes daemon. You always want to have your mercurial server up. Same goes for elasticsearch and rabbitmq. The elmo server can be up or not, it doesn't interfere with the other bots. It's just good to be there to check that the rest reports as expected. The four remaining bots and services are the actual build system, and the starting order of those has actual impact on what happens.

elmo

Often you want to run the elmo webserver, too. If you're using a virtualenv for elmo, use that. We'll assume the default port here and going forward, 8000.

./manage.py runserver

rabbitmq

On most systems, rabbitmq will be automatically run as part of the boot process.

elasticsearch

Find the installation path to elasticsearch, and run

./bin/elasticsearch

mercurial web server

Before you start the build automation, be sure that the mercurial webserver is running.

hg serve --webdir-conf=webdir.conf -p 8001

The master and slave both need each other to get any work done. The order of startup isn't really important, as the slave continuously try to connect to the master. When they don't reach the master, they back up, though. The quickest startup time is to start the master first, and then the slave.

buildbot master

In the master-ball directory, do

./scripts/buildbot start test-master
tail -f test-master/twistd.log

where test-master is the path to the test-master directory, not the name or something else.

You'll already see a bit of work in the log, as the master gets itself ready to get crackin'. But for the actual work, it needs the slave(s).

buildbot slave

In the virtualenv of slave-ball and that directory, run

./scripts/buildbot start slave
tail -f slave/twistd.log

You want to make sure that the bots do the right thing up to here, in particular that the startup builds are successful.

Next up is the combo of a10n hg and get-pushes. get-pushes is a daemon that polls the hg server, and adds information about new pushes into rabbitmq. That queue is worked through by a10n hg, which pulls the working repos, and fills in detailed information about new mercurial changesets into the elmo database. You probably want to do this before you start the buildbot master, just because it's more realistic to have repos first, and then builds.

a10n hg

This is run in the a10n directory

./scripts/a10n hg

get-pushes

This is also run in the a10n directory

twistd -n get-pushes

Now you're looking at the log file of the get-pushes daemon. If you're running it for the first time, you should find a push on mozilla , and then cycle every few seconds.

Watch the results

You should have 6 shells with logs for each part of the automation now:

the django log for elmo
the hg output of hg serve (noisy, that's cool)
the tail of twistd.log for get-pushes (this makes hg noisy)
the output of a10n hg, processing pushes as they come
the tail of twistd.log for test-master
the tail of twistd.log for the slave connected to test-master

Expectations on elmo

On elmo, the homepage should show 4 active l10n repositories, and the pushes view over all repositories should show one push on the mozilla repository.

Whenever you start the buildmaster, it will trigger a build per tree to get more data from the l10n.ini files on hg, and as you started all bots, you should see one build on the waterfall, for the tree-builder Builder. If that box isn't green, your setup is broken. From there, you get to the actual build display, showing one step configured fx . You should be able to surf to it's stdio log, and see the build properties. In particular the locales and tree build properties should match your setup.

Play

Next up is play. Let's be a localizer and push our changes. The staging script already pre-filled all working copies with data, which we can now just push. While we're doing that, watching the TBPL (bad name) is entertaining. (That page only refreshes every 5 minutes, tweak delay in a js shell for now.)

Let's go to our working directories, l10n/ab for a start, and push. Be sure to have your virtualenv activated again.

cd workdir/l10n/ab
hg push

That will tell you that it's inserting that changeset to the pushlog. That's the local one per repository that our mercurial hooks do.

Next, the get-pushes daemon will find that, and add it to the overall elmo pushes database. You can verify that on the pushes page.

Then, the test-master will pick that up, and schedule a compare-locales build for it, which then runs and does the work on the slave.

As the build starts (and when it's ended), it will show up on the build reporting page.

Now that we have a real active tree, you'll see the fx tree show up on the homepage, linking to the dashboard of the tree. Which gives us a chance to verify that the rich compare-locales output works as well, if you click on the C link there.

Next steps

Next up, you can push the remaining localizations, and also do an edit to the l10n files in mozilla , check that in and push, to see how the build system behaves if en-US changes. Tip: It should update all locales.

If you just want to test build automation, you can fake changes, too.

./scripts/buildbot sendchange --master=localhost:9876 -u me \
    --branch=l10n --property=locale:de browser/file.properties

The options for sendchange mean, resp.

the port of the master, localhost:9876
your name, pick what you want
the branch. For a standalone repository like mozilla this is mozilla. For l10n repositories, this is the parent. So for a change that affected /l10n/de you pass l10n
If this is an l10n repository, pass the leafname of the repo. I.e., for a change to /l10n/de pass locale:de. The name and value of the property are separated by a colon.
One or more files. Depending on the files you pass and l10n.ini, different builds will be triggered.

Also visit the upstream documentation for sendchange.

Shutting down

Enough learning, let's shut the bots down in a safe order:

First, stop taking new pushes into our system, and kill

get-pushes

If you're running it interactively, just Crtl-C. If you had it daemonize, you'll need to kill the process.

kill `cat twistd.pid`

Make sure that master and slave are idle, i.e., the logs only show pings, and then stop master and slave. The order doesn't matter much at this point.

slave

./scripts/buildbot stop slave

master

./scripts/buildbot stop test-master