escape(window): blog

hg-git part i

aki@escapewindow.com — Mon, 06 May 2013 22:01:09 GMT

I was about a week into my m-c hg-git conversion^* when gps blogged about faster hg-git.

I was about two weeks into the initial conversion when I tried the latest upstream hg-git, with gps' patches. That conversion finished in less than a day, while the initial conversion was still chugging along on a different core. However, the git shas differed from our existing pre-cvs conversion repositories. This was due to different behavior when faced with a bogus email address.

Once I backed out the corresponding email munging commits, and landed Ehsan's timezone fix for a separate issue that halted the conversion process, we're now back to a converted m-c with the same shas.

At this point I killed the still-running initial conversion.

I'm still working on automating the cvs prepending and pulling in other repos/branches, but this is a good sanity-check point.

^* Note, I had pulled both mozilla-central and mozilla-b2g18 into the same working directory as a proof-of-concept workflow test before converting. In hindsight this was a terrible decision, as it more than doubled the estimated conversion time.

mozharness-based desktop unittests are live

aki@escapewindow.com — Thu, 21 Feb 2013 02:51:15 GMT

We are now running Firefox desktop unittests through mozharness for Firefox-22-based branches (mozilla-central, mozilla-inbound, try, and project branches).

Jordan Lund started this work during his internship last year, and had the mozharness scripts and configs code-complete by the end of his internship. I took over afterwards, but my efforts were sporadic, between high-priority mobile and b2g work.

This was a multi-phase operation:

We enabled them on Cedar
- This required platform-specific fixes, since we were working against a subset of platforms in staging.
- Cedar unittests were essentially the same as, say, mozilla-central, other than having the logic in mozharness instead of hardcoded in buildbot.
At this point, the majority of the tests were green, but a number were perma-orange or red or missing or had intermittent issues not present in buildbot-based unittests. So we went through and identified+fixed the various problems per-suite or per-platform, whether that was missing debug tests or barfing on leaks.
We also cleaned up issues that made these jobs hard to sheriff, e.g. missing automated retries or ugly log summaries.
Whenever P0 interrupts hit (regularly), we had to put this work on the back burner, and determine what changes had made the buildbot-based unittests diverge from the mozharness-based unittests when we revisited. This back-and-forth added months to the schedule, but allowed us to respond to higher priority tasks more quickly.
Once all the blocker bugs were resolved, I split the rollout into three parts:
1. Get mozilla-tests/config.py to not warn in python mode (pep8). This wasn't critical, but helped avoid spurious conflicts later. Easy peasy. (comment 8 through comment 10)
2. Split out the mobile tests into their own config file. This helped me limit the scope of my config.py rewrites to desktop only. (comment 11 through comment 28)
3. The final set of patches made mozharness-based unittests default for Firefox desktop, with exceptions for aurora, beta, release, esr, and b2g18*. (comment 29 through comment 60)
I ran through a number of spot tests in staging to limit risk for rollout; I also waited until after merge day so the unittests could ride the trains. So far the only fallout is some bustage in debug b2g emulator tests on b2g18 and b2g18_v1_0_1 due to missing symbols (now fixed; we also have bugs to enable symbols filed).

Making large sweeping changes in infrastructure while

responding to constant priority interrupts (often for weeks or months at a time), and
avoiding bustage

is a long game. But we're playing for keeps.

Thanks to Jordan for the initial work around this, and A-Team, sheriffs, and RelEng for their help + patience.

mozharness 0.6

aki@escapewindow.com — Fri, 01 Feb 2013 04:20:52 GMT

" I certainly don't want to wait another 5 months before the next arbitrary version bump."

-- me, 11 months ago.

What's happened in the past 11 months? In no particular order,

completed Android Native support,
Fennec armv6+noion support,
broadened VirtualenvMixin functionality (setup.py support, pip freeze, PyWin32 support, --find-links, ...),
further mozbase support / integration,
developer-runnable Android multilocale build support,
Marionette tests on Firefox desktop,
mock (mock_mozilla) support,
tooltool support,
retry support,
Boot2Gecko (FirefoxOS) device builds (panda, unagi, unagi_eng, unagi_stable, otoro) (all multilocale),
Boot2Gecko desktop builds (all multilocale),
Boot2Gecko emulator tests: marionette-webapi, mochitests, reftests, crashtests, xpcshell,
mozpool support for Boot2Gecko + Android Pandas,
Boot2Gecko Panda Gaia UI tests,
Android signing on demand,
TBPL parser compatibility improvements,
per-locale status in mobile_l10n.py,
mozharness Firefox desktop unittests (ready to roll out),
mozharness Firefox desktop talos support (baking on Cedar),
proof-of-concept jetperf,
proof-of-concept remote (mobile device) talos support (needs work),
further Peptest support (which has since been disabled),
MPL 2.0,
and a mozharness production branch, which will let us run CI tests against default before merging, to avoid production bustage.

So, yeah. Some minor version bumps are larger than others. We've gotten all this done thanks to great efforts by the A-Team and RelEng.

B2G automation is largely mozharness-based; mobile is maybe half-mozharness and Firefox desktop is still largely buildbot-based. But, without actually crunching numbers, I would guess that rolling out mozharness Firefox desktop unittests would push us somewhere near the halfway point. Soon.

I'm glad to see the project continuing to gain traction, and we've been seeing concrete evidence of the theoretical benefits I've been touting since the beginning.

This time I'm not going to make any predictions about when I'll bump the arbitrary version next, but I'll schedule something in Things.

Building Android multilocale locally

aki@escapewindow.com — Thu, 31 May 2012 20:00:49 GMT

Tl;dr: it's now easier to run multilocale builds locally. I'd love feedback in terms of whether this is useful for you, and if there are changes that would make this script better for your workflow.

I updated the wiki as well.

what & why

There are only three ways to get a multilocale build currently:

triggering a nightly build off of a multilocale-enabled branch,
pinging me directly and asking for a multilocale build, or
building multilocale locally. Up until now, that involved a combination of digging through logs, asking for help, and reverse-engineering the multilocale script with a lot of trial and error.

The latter was so difficult because I hadn't spent the time to make it friendly to run locally. Now that bug 753545 has landed, it should now be relatively easy.

The config file is written to [hopefully] be easily editable and usable.

running the script

</p>

Create a directory to run in. It's simplest to create a new directory, although you can use an existing directory with mozilla-central (or other tree) already checked out.
```
mkdir multilocale
cd multilocale
```

Clone mozharness and copy the standalone config file for easy editing+use.

hg clone http://hg.mozilla.org/build/mozharness
cp mozharness/configs/multi_locale/standalone_mozilla-central.py myconfig.py

Customize your config file.
By default it's going to clone mozilla-central into a directory called mozilla-central.
It also assumes your objdir will be named objdir-droid and that will be specified in your mozconfig, which will be in a file named mozconfig in this directory.
This should hopefully be easy to edit. Holler if it's confusing.
Afterwards, you can verify it's still valid python by typing
```
python myconfig.py
```
which should exit silently.
If your config file still says that your mozconfig is in os.path.join(os.getcwd(), "mozconfig"), copy your mozconfig into this directory and make sure it's named mozconfig.
Make sure mozilla-central (or whatever repo you specified) is in this directory.
You can either clone it manually, or use the script's pull-build-source action:
```
mozharness/scripts/multil10n.py --cfg myconfig.py --pull-build-source
```
Run the script.
By default, this will:
- pull the locale source,
- copy your mozconfig into the tree and build mozilla-central,
- package the en-US apk,
- backup your objdir to a -bak directory (e.g. objdir-droid-bak) via rsync,
- restore your objdir via rsync (no changes),
- add the locales to your objdir,
- and package the multilocale build.
(See the actions section below for more advanced usage.)
```
mozharness/scripts/multil10n.py --cfg myconfig.py
```

developing + debugging

The above instructions are sufficient for when you want a single multilocale build.

If you're changing m-c code or otherwise want multiple builds, you probably want to follow this workflow:

After running the multilocale steps, you want to restore your objdir to en-US. At least until someone figures out how to clobber the multilocale-specific bits, rsyncing from a clean backup directory seemed like the most expedient solution.
```
mozharness/scripts/multil10n.py --cfg myconfig.py --restore-objdir
```
Then make your code changes.
Then run the default actions again. If this seems like it's doing more than it needs to, you're correct. See the actions section below for more advanced usage.
```
mozharness/scripts/multil10n.py --cfg myconfig.py
```

actions

(I covered actions to some degree here as well, but the below specifically covers this script + config file.)

what actions are available?

You can see the default actions, defined in the config file, by:

mozharness/scripts/multil10n.py --cfg myconfig.py --list-actions

This will give you this output:

Actions available: clobber, pull-build-source, pull-locale-source, build, package-en-US, upload-en-US, backup-objdir, restore-objdir, add-locales, package-multi, upload-multi Default actions: pull-locale-source, build, package-en-US, backup-objdir, restore-objdir, add-locales, package-multi

what do these actions do?

I'd like to have more information about each action built into each script. Here's an overview of multil10n.py's actions:

clobber - This is effectively no-op as long as work_dir is set to '.'. Otherwise it nukes the entire work_dir.
pull-build-source - This clones the repos defined in repos.
pull-locale-source - This clones the repos defined in l10n_repos, as well as the l10n repositories defined in the locales file (specified here)
build - This copies the mozconfig path specified here into the source tree, and runs make -f client.mk build.
package-en-US - This runs make package.
upload-en-US - To be written. It will call make upload.
backup-objdir - This rsyncs your objdir to a backup directory, since there's no current solution for clobbering the multilocale bits in your objdir.
restore-objdir - This rsyncs the backup directory into your objdir. If you want to re-run the build or re-add the locales without a lot of old cruft lying around, I recommend doing this first.
add-locales - This runs compare-locales and adds the locale strings + manifests into your objdir.
package-multi - This runs make package for your multilocale apk.
upload-multi - To be written. It will call make upload.

how do I run a single action?

You can run any single action by itself by prepending -- in front of the action name (making it a command line option). For instance, to only run the restore-objdir action,

mozharness/scripts/multil10n.py --cfg myconfig.py --restore-objdir

how do I skip an action?

You can run the default set of actions, minus an action, by prepending --no- to the action(s) you want to skip. To run the default actions without pulling the locale source,

mozharness/scripts/multil10n.py --cfg myconfig.py --no-pull-locale-source

how do I run several actions at once?

Running the script with no action-specific command line options will iterate over the list of default actions.

You can also have a combination of --ACTION or --no-ACTION to add an action to or remove an action from the default list. For example,

mozharness/scripts/multil10n.py --cfg myconfig.py --no-pull-locale-source --no-build --no-package-en-US --no-backup-objdir

and

mozharness/scripts/multil10n.py --cfg myconfig.py --restore-objdir --add-locales --package-multi

are equivalent.

You also have the option of editing your default_actions in myconfig.py if that makes things easier.

mozharness presentation

aki@escapewindow.com — Thu, 19 Apr 2012 01:40:22 GMT

For those interested, I gave an overview of mozharness for the Automation Team today.

I mentioned that we're not yet at that point of critical mass where the accumulated knowledge and shared code help make everything go faster. But we're getting there.

The presentation "slides" are available here.

mozharness 0.5

aki@escapewindow.com — Wed, 29 Feb 2012 01:35:37 GMT

A lot's happened since mozharness 0.4 landed in late September. We:

added a bunch of Android native support.

enabled Android single locale repacks with nightly update snippet generation
ported Android signing with passphrase verification, signature verification, single locale support, and update snippet creation.

rolled peptest out to production on Try, across all desktop test platforms.
improved mozharness virtualenv support, with real-life mozbase usage in peptest.
fixed actions-in-config-files.
separated the output parser from ShellMixin.run_command(), so we can

parse output from subprocess or get_output_from_command(),
eventually add context lines to output parsing, and
potentially split serial tasks into multiple parallel jobs with their own log parsing.

added chunking support to split up jobs across machines.
added query_exe() and which() support to specify or find executables.
added add_failure(), query_failure(), and summarize_success_count() support to track granular status across a list of tasks.
added a BuildbotMixin and ReleaseMixin to tie into our existing buildbot infrastructure/configs.
precompiled the error_list regexes.
made various OSMixin and ShellMixin method improvements.
added a setup.py.
added a pyflakes call and Debian/Ubuntu support to unit.sh.
moved mozilla-specific modules into mozharness.mozilla.*.
retired Maemo scripts (Maemo tier 3).

This feels like an as-good-as-any time to arbitrarily increment the arbitrary version number: mozharness 0.5.

I think the best part of this release is how more people got involved; I can feel the momentum building. I certainly don't want to wait another 5 months before the next arbitrary version bump.

mobile/ is now mobile/xul/

aki@escapewindow.com — Thu, 17 Nov 2011 16:47:17 GMT

Today we moved mobile to mobile/xul in mozilla-central, in preparation for landing the native UI code in mobile/android.
Mozilla-aurora, mozilla-beta, and mozilla-release are currently unaffected.

Branch	Android platform	Mobile source location
mozilla-central	android	mobile/android
mozilla-central	android-xul	mobile/xul
mozilla-aurora	linux-android	mobile
mozilla-beta	linux-android	mobile
mozilla-release	linux-android	mobile

NOTE: mobile/android has not yet landed. The builds are switched over and will need to be hidden. I'm doing that as I see them, but would appreciate any help in this matter.

The mobile team has said that mobile/android is at least a few days away from landing.

Attention project branch owners! If you have Android builds on your project branch:

You will need to merge from mozilla-central for the 'android-xul' builds to go green.
The 'android' builds will burn until mobile/android lands and you re-merge from mozilla-central. These builds need to be hidden.

Attention trychooser users! You probably want the 'android-xul' platform. The mozconfig situation will be a little convoluted for now, so it would be best to land a mozconfig-extra-android-xul at the top of your push to make sure you get the right settings.

Attention l10n users! Due to the move of mobile/locales, we have broken mobile localization on mozilla-central. We will be fixing this and finding a single locale repack solution ASAP.

device_talosrunner.py: a first peek

aki@escapewindow.com — Sat, 05 Nov 2011 02:03:32 GMT

[what is it?]

device_talosrunner.py is a work-in-progress mozharness script that sets up talos and runs it against a device using either the sut or adb protocols.

It's designed to:

allow anyone with a [supported and rooted] Android device to run talos without building the tree;
allow someone with access to a staging/production tegra to run a production-like talos run without setting up buildbot;
at some point, become the official way we run talos on Android in production.

This is similar to, and parallel to, the work on make talos-remote. However, aiui, that is developer-oriented, whereas device_talosrunner.py is targeted towards someone who wants to test pre-built apks (as well as replacing our current production code).

I have used this script to run talos on both a tegra over sut, and an Asus Eee Pad Transformer using adb-over-ip, successfully.

[how do i use it?]

You need:

python (2.5 - 2.7.x)
virtualenv
hg, to clone the various repos
a copy of my github mozharness repo, on the branch 'talosrunner';
a rooted, supported Android device that is either running the sutagent, or is attach{ed,able} via adb. The adb calls require busybox.

Once you have those things,

Create a config file. I've got a wip tablet config that I use for my Asus Eee Pad Transformer, and a wip tegra config that I use for a staging tegra.
If you're using adb-over-usb, the device needs to be connected to your desktop/laptop. If you're using adb-over-ip or sut, specifying the device_ip should be sufficient.
If this is the first time you're running device_talosrunner.py, you need a virtualenv. Run
mozharness/scripts/device_talosrunner.py --cfg users/aki/tablet1.py --create-virtualenv
(replacing the config file path with the path to your config file). This will create a python virtualenv in ./venv and install PyYAML, a Talos dependency, into it.
Run
mozharness/scripts/device_talosrunner.py --cfg users/aki/tablet1.py
If the start_python_webserver option is set, the script will start a webserver and point the device at it. Otherwise it'll point the device at whatever talos_webserver is set.

When this script is polished and mature, I definitely want people to be comfortable using it on their personal phones and tablets, but it's still rough around the edges.

This will uninstall Fennec and possibly reboot your device, so please be careful if you're trying this on a device/profile that you haven't backed up.

[what's left to do?]

I've only tested a handful of talos suites against two devices; for this to be a production replacement, it needs to be able to run reliably against a large pool of devices.
This relies on a talos patch in bug 688604 for the python webserver; I most likely need to use mozhttpd.
I've hit issues with both m-c and birch builds, and will most likely need to make changes for the native UI.
I have an idea which would allow us to find the next available and working tegra, which may reduce red and purple Tegra results; this needs writing and testing. More on that later.

mozharness: actions

aki@escapewindow.com — Wed, 12 Oct 2011 07:11:46 GMT

that allows production infrastructure and individual developers to use the same scripts.

What are actions?

Actions are discrete logical steps to a script, similar to [simple] makefile targets.

Since certain actions, like upload or notify might not be useful to certain consumers of a script, it's helpful to be able to turn those off.

Versatile, reusable configs are key to sharing scripts between automation and users with different workflows. Actions are another key ingredient.

What are default actions?

default_actions is a subset of all_actions, and specifies the list of actions to run if no action-specific command line options are specified.

Using the configtest.py script as an example:

deathduck:~/src/clean/mozharness [19:09:59] (default)
621$ scripts/configtest.py --list-actions
Actions available: list-config-files, test-json-configs, test-python-configs
Default actions: test-json-configs, test-python-configs

These are specified here.

It's now possible to specify your own default_actions in your config file, so if you want to run a script with a different set of default actions, you can do so without copy/pasting a ton of command line arguments.

How is the list of actions-to-run generated?

I have that documented here.

Using configtest.py as an example again:

# Only run the list-config-files action
configtest.py --list-config-files

# Run the list-config-files and test-json-configs actions
configtest.py --list-config-files --test-json-configs
# or
configtest.py --action list-config-files,test-json-configs
# or
configtest.py --action list-config-files --action test-json-configs

# Run list-config-files in addition to the default actions
configtest.py --add-action list-config-files

# Run the default actions, but not test-python-configs
configtest.py --no-test-python-configs

How are the actions run in the script?

That's handled by BaseScript.run():

Change all dashes in the action name to underscores; that's the method to call
If the action is not in the list of actions to run, skip + go to the next action.
if preflight_METHODNAME exists, run that before running METHODNAME.
Call METHODNAME.
If postflight_METHODNAME exists, run that after running METHODNAME.

Mozharness home

mozharness: logging

aki@escapewindow.com — Wed, 12 Oct 2011 01:30:02 GMT

with full logging

Why so many log files?

This is the MultiFileLogger, which creates a log file per log level. We could just as easily default to the SimpleFileLogger or write a new class, e.g. a MultiNetworkLogger.

deathduck:~/src/clean/mozharness [17:39:57] (default)
608$ ls -l logs
total 24
drwxr-xr-x   8 asasaki  staff   272B Oct 11 17:18 ./
drwxr-xr-x  15 asasaki  staff   510B Oct 11 17:18 ../
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_critical.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_error.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_fatal.log
-rw-r--r--   1 asasaki  staff   5.1K Oct 11 17:18 log_info.log
-rw-r--r--   1 asasaki  staff   3.6K Oct 11 17:18 log_raw.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_warning.log

Because each log file is a superset of all levels above it, you can easily glance at the warning or error log file and determine whether the script detected any warnings or errors.
(debug > info > warning > error > critical > fatal, where '>' means is-a-superset-of)

These are all standard logging module levels except for FATAL, which I added to indicate a critical error that should also exit the script.

What do Mozharness logs look like?

Here's a snippet from running scripts/configtest.py (without arguments):

17:18:00     INFO - Testing /src/clean/mozharness/scripts/../configs/deb_repos/staging_release_mozilla-release.py.
17:18:00     INFO - Good.
17:18:00     INFO - Testing /src/clean/mozharness/scripts/../configs/test/test.py.
17:18:00     INFO - Good.
17:18:00     INFO - 5 of 5 python config files were good.
17:18:00     INFO - #####
17:18:00     INFO - ##### ConfigTest summary:
17:18:00     INFO - #####
17:18:00     INFO - 17 of 17 json config files were good.
17:18:00     INFO - 5 of 5 python config files were good.

The timestamp and log level are prepended to most log files and the console output. By default, we also create a raw log without that information, which is potentially more scrape-friendly.

How does Mozharness detect errors?

The core error detection is through the ShellMixin.run_command error_list.

Generic error lists are defined in errors.py. These are ordered lists of substrings and/or regular expressions, which, if matched against a log line, determines its log level. When we hit a match, we exit the loop.

Two very important features I want to add here:

context lines:
If we set 'context_lines': (10, 3), I want ten lines up and three lines down to also be marked at this log level.
This requires buffering lines before outputting to the log file and/or console.
This allows for more useful contents in the error log. Rather than only highlighting, say, make: *** [all] Error 2, you can also include some lines of context above and/or below that line.
explanations, to translate obscure error messages into "this probably means [insert human readable message here]". More here.

Mozharness home

mozharness: config

aki@escapewindow.com — Wed, 12 Oct 2011 00:09:35 GMT

Mozharness is a configuration-driven script harness

How is configuration specified in Mozharness?

A script can define an optional initial config, as a dictionary or a config file.
Next, if we specify --config-file FILE on the command line, that file's contents will be added to self.config.
Finally, command line options are added to self.config.

If the same config key is specified in multiple locations, the command line takes precedence over the config file, which takes precedence over the initial config [file].
(command line options > config file > initial config > initial config file)

How does a script add command line options?

The command line options are all optparse-based.

Since there's a global list of command line arguments for all mozharness scripts, I'm currently passing in an additional list from scripts to add to that global list.

What does a config file look like?

Here's a json config file, and a WIP python config file.

We can currently specify anything here. I have a config file well-formedness checker, but I'd love to test for misspellings or missing/extraneous config options as well.

How do we set the config to read-only?

This is handled by locking the ReadOnlyDict.

I cover why here.

What is the volatile config?

After locking the read-only config, mozharness dumps a copy of the running config in a file called localconfig.json, which can be reused or inspected later.

If you specify set of actions to run, that information will currently be saved with the rest of your config. The next time someone runs the script with that config file, that set of non-default actions might be unexpected and undesirable.

Previously, I discarded action config before dumping that file. In the talosrunner branch, I'm now moving action-specific config to self.config['volatile_config']; we'll have the choice whether to re-use that or ignore it in future runs.

If I ever fully support --dry-run / --noop, that will likely be another candidate for a volatile config option.

mozharness home

mozharness 0.4

aki@escapewindow.com — Fri, 30 Sep 2011 22:33:50 GMT

I landed mozharness 0.4 today (2091ae4c21ec).

Some notable points:

documentation;
python virtualenv creation;
new MercurialVCS and sourcetool, based on lib/python/util/hg and hgtool (bug 650882);
more mixins to avoid duplication of code;
other than bug 690911, mozharness unit tests now pass on windows and linux.

These changes are largely behind the scenes.
However, work is already under way on talosrunner (bug 640890 and bug 650887); more on that in a bit.

(What is mozharness?)

6.0b2: the first combined desktop/mobile release

aki@escapewindow.com — Thu, 14 Jul 2011 03:21:27 GMT

Firefox 6.0b2 is the first combined release: Firefox desktop and Firefox for Mobile running in the same automation, run by the same release engineer.

I remember talking about this with John O'Duinn in pie-in-the-sky brainstorming sessions back in Building K a few years ago, back before Fennec 1.0. It seemed so far away, then. It's a bit surreal seeing it run now.

A large part of the team chipped in on this one, but Rail did most of the heavy lifting. Thank you Rail!

mobile-browser is dead; long live mozilla-central/mobile !

aki@escapewindow.com — Thu, 07 Apr 2011 00:55:39 GMT

The mobile-browser repo is now a subdirectory of mozilla-central.

Mozilla-central-based project branches will need to re-sync from m-c to go green.
Outstanding patches will need to be applied accordingly.
If there are any outstanding un-ack'ed issues as a result of this merge, please ping me and file a mozilla.org::Release Engineering bug.

On Try, we can still build older, non-mobile-merge builds (e.g., a Fennec 4.0-based patch) by using the mobile-repo and mobile-rev special files as described here.

mozharness 0.3

aki@escapewindow.com — Tue, 05 Apr 2011 10:45:51 GMT

I've been too busy to announce the landing of mozharness 0.3 on March 5.

I'm no longer completely embarrassed by the code. (I consider that the most important feature of v 0.3.) Further details are here: bug 628571, comment 1

This is still a work in progress, as evidenced by both the version number and my lengthy mozharness todo list.

The list of problems which mozharness can help solve in the near future is growing, even as the harness evolves and stabilizes.

For those at the Mozilla All Hands this week, I'll be discussing mozharness with the A-Team (schedule TBD; I'm thinking Thursday, 11am PDT). If others wish to join, or set up alternate meetings, please comment or email <aki AT mozilla DOT com>

(what is mozharness?
mozharness source)

tl;dr 4.0.x branching

aki@escapewindow.com — Thu, 17 Mar 2011 09:00:11 GMT

In case John's branching blog post was tl;dr:

Mozilla branch	Mobile branch	Firefox desktop version	Firefox mobile (Fennec) version
mozilla-central	mobile-browser ^[1]	Firefox Next	Fennec Next
mozilla-2.0	N/A	Firefox 4.0[.x]	N/A
mozilla-2.1	mobile-2.0	N/A	Fennec 4.0[.x]

[1]: We're still debating whether we're going to merge mobile-browser in.

And for l10n:

Firefox desktop version	Firefox mobile (Fennec) version	L10n repo directory
Firefox Next	Fennec Next	l10n-central
Firefox 4.0[.x]	Fennec 4.0[.x]	l10n-mozilla-2.0 ^[2]

[2]: We're using the same set of l10n-mozilla-2.0 repos for both Firefox 4.0.x and Fennec 4.0.x.

dispelling some mozharness fallacies

aki@escapewindow.com — Thu, 18 Nov 2010 12:19:45 GMT

My previous blog post, what is mozharness?, appears to have caused some dismay.
I'm writing this to answer Axel's invitation to respond, and to clear up some misconceptions about mozharness.

[assertions]

Respect for Mozilla l10n.

This conversation is larger than just localization. However, the above thread brings up l10n specifically, and the first two mozharness scripts involve l10n, so here are my high-level thoughts:

I remain highly impressed with the state of l10n at Mozilla: the sheer number of locales, the contributions of volunteer localizers, the sim-shipping of localized releases with en-US releases. Never have I seen localization done anywhere near so efficiently and well, anywhere in software. I believe strongly that this is key to Firefox's success. And Axel is a significant part of that.

I also believe that however good our localization story may be, there's definitely room for improvement.
We seem to disagree as to how to improve that story for the moment.

Mozharness is imperfect software with time-tested concepts.

I will be the first to admit that mozharness is imperfect. I'm self-taught. Python is a relatively new language to me. I wrote mozharness to solve complex problems with tight deadlines. Mozharness, as it exists today, is essentially non-feature-complete beta software.

(With that in mind, I will be speaking about what mozharness could be, in forward-thinking statements. Saying "Nothing prevents us from doing ____" does not mean it'll take zero development/testing/roll out time. I am speaking about technical feasibility only.)

However, the concepts behind mozharness are lessons I've learned over the years. Usually the hard way.
I'm open to changing mozharness' specific implementation details, but I strongly believe the concepts themselves are right.
It falls on my shoulders to communicate those clearly.

Buildbot was not written to micromanage slaves.
The above statement sums up my first conversation with Brian Warner, where we vehemently agreed that too much complex logic had been relegated to Mozilla's buildbot masters.
Even if one ignores his original intent, I assert that moving complex logic from an overloaded master to its relatively unloaded slaves will be
1. more efficient
2. more scalable
3. more portable
I'll revisit this, and the other assertions, in the next three sections.

[fallacy #1: mozharness will make {builds,tests,repacks} less granular]

This can be split into two concerns: granularity of status, and granularity of easy-to-replicate steps. This is going to be a long section; I still haven't fully convinced everyone on my own team about these points.

Mozharness can sum up status at the end of jobs.
I don't want to spend this entire post talking about abstractions and what mozharness could be if this nebulous vision in my head somehow sees the light of day. So here's a concrete example that exists today.
signdebs.py outputs a summary at the end of every log:
19:29:25 INFO - ##### 19:29:25 INFO - ##### MaemoDebSigner summary: 19:29:25 INFO - ##### 19:29:25 ERROR - Can't download http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-maemo5-gtk-l10n/he/fennec_4.0~b3~20101117005726_armel.deb; skipping he on fremantle 19:29:25 ERROR - Can't download http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-maemo5-gtk-l10n/ja/fennec_4.0~b3~20101117005726_armel.deb; skipping ja on fremantle 19:29:25 ERROR - Can't download http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-maemo5-gtk-l10n/ja-JP-mac/fennec_4.0~b3~20101117005726_armel.deb; skipping ja-JP-mac on fremantle 19:29:25 ERROR - Can't download http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-maemo5-qt-l10n/he/fennec_4.0~b3~20101117010937_armel.deb; skipping he on fremantle-qt 19:29:25 ERROR - Can't download http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-maemo5-qt-l10n/ja/fennec_4.0~b3~20101117010937_armel.deb; skipping ja on fremantle-qt 19:29:25 ERROR - Can't download http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-maemo5-qt-l10n/ja-JP-mac/fennec_4.0~b3~20101117010937_armel.deb; skipping ja-JP-mac on fremantle-qt 19:29:25 INFO - Uploaded multi on fremantle successfully. 19:29:25 INFO - Uploaded en-US on fremantle successfully. 19:29:25 INFO - Uploaded ar on fremantle successfully. 19:29:25 INFO - Uploaded be on fremantle successfully. 19:29:25 INFO - Uploaded ca on fremantle successfully. 19:29:25 INFO - Uploaded cs on fremantle successfully. ... snip 76 more lines of "Uploaded ___ on ___ succesfully."
I'd love for this to be prettier and take fewer lines, like "82/88 deb repos uploaded successfully!" with more verbose information about the ones that failed. But again, deadlines, and pretty statuses were not a hard requirement.

What's keeping us from emailing this summary somewhere? Nothing.

What's keeping us from updating a database or hitting a cgi with this status? Nothing.

What's keeping us from sending out a Pulse message with this info? Probably the fact that I know very very little about how to send a Pulse message. Other than that, probably nothing.

We'll want to make those updates conditional, if "notify-pulse" in self.actions:, for example, so staging or development runs don't attempt to send production status messages. But if it can be done in a script, it can be done at the end of any mozharness script.

Mozharness can update status during jobs.
What's keeping us from updating status in the middle of a for locale in locales: for loop, for example? NOTHING.
Though it is more expensive. If you want to hit a cgi, for example, mid-for-loop, then that cgi needs to be up and available the entirety of every script run (or the script needs to fail gracefully, or have retry logic, or queueing logic, or whatever.) You need to decide when the cost of faster status updates is worth the effort, since post-processing tends to be less expensive.
There is nothing preventing us from doing so, however, if the need outweighs the cost of updating status throughout the script.
If it can be done in a script, it can be done inside of any mozharness for loop.

Mozharness parses its own logs during runCommand() calls already.
The runCommand() method uses pre-defined error lists to determine whether specific lines in the log are errors.
These are far from complete, and need to be fleshed out further, but the framework is there.
Recently, I thought we should add a summary (or somesuch) key to that list of dictionaries. We could, say, add a substr of "Assertion failure: !(addr & GC_CELL_MASK)" with a custom level of intermittent_orange (somewhere between info and warning?) and a summary of "Intermittent orange: This looks like bug 583554!"
As long as we're in conjecture-land, we can combine this with a post- or mid-job status update that can populate an intermittent orange database with the specific details of this job.
Awesome or not awesome? You vote.

Mozharness can create buildbot-parseable status.
This point's reverse could be a top level fallacy in itself.
There are two approaches here.
If it's a hard requirement that we lose none of the "granularity" of the existing buildbot steps, nothing prevents us from creating an action list that encapsulates each buildbot action.
Then you can script.py --only-set-props-builddir, for example. A buildbot addStep(['python', 'script.py', '--only-%s' % stepName]) for every single thrilling step in the one-hundred-one steps here.
Can you? Yes. Do you want to? I would argue no.
Do developers really care if buildbot step #73 dies with python exception ____? Or do they only really care if compilation fails on file X at line Y (link to hg annotate with appropriate finger pointing here)?
(What if we tied in a ping in #developers or an email message for the suspected culprit in the notify action? Not free, in terms of mozharness development time, but is it doable in scripts? Seems like it, to me.)
Do localizers really care if buildbot step #45 dies with compare-locales exception ___? Or do they just want a description of which strings are missing, or which XML files need updating, with a link to a wiki page on how to fix those things?
The second approach could involve writing a buildbot-property parseable file during or at the end of the mozharness script, and adding a buildbot addStep(SetProperty("cat filename")) afterwards to set buildbot-statusdb-parseable buildbot properties.

Mozharness could create a step-summary log.
As I explained in my previous blog post, I lean very heavily towards more verbosity than less, though you could --log-level error (or w/e) to ignore most of this verbosity.
In a --multi-log run, I could add a step-level log. Basically, any calls to the BaseScript wrapper methods could also write their equivalent to a log.
(Huh??? English, do you speak it?)
If we cared enough about this, we could create yet another log file. The BaseScript.chdir() method could output cd DIRNAME to this log. And the BaseScript.runCommand() method could write its command line to this log file, and so on. This log file would approach being an executable shell script.
I say approach since there aren't always going to be universal scriptable equivalents. But this would be an improvement over the status quo.

High-level granularity is not always desirable.
So you're a developer or a localizer. You've painstakingly set up scratchbox to the point where it works (congratulations!!). You now want to attempt a Maemo multilocale build without checking into the tree and either requesting or waiting for a nightly. What steps do you follow?
You could drill down into the 101 buildbot steps, each with python-list-format command lines, and full env dumps per step. Granular, no? Run each one, with appropriate env settings, and then it should work! Maybe! Have fun!
Or you could take a [yet to be written, but next in line] mozharness script, a config file that's oriented towards standalone developers (you may have to edit paths), and an example command line, and run it. It should hopefully either give you a descriptive error message (and ultra-verbose logs), or a usable multilocale deb file.
Which would you prefer? (Don't let me influence your decision.)

[fallacy #2: a script means complex undecipherable command lines]

(Phew! That last section was a doozy, wasn't it? This section will be shorter, I promise!!)

This section also has a couple components, though much shorter.

First, Ben's work in bug 608004 is not mozharness.
Is it a step in that direction? Sure, it's moving logic out of buildbot towards slave-side python scripts. Does that mean it's mozharness? No. Are we considering merging tools/lib/python code into mozharness? Yes. But it's not reality yet.
I haven't looked at his patch; I've been bogged down trying to fix the Android Tegras. And writing this lengthy blog post. But it is not mozharness.
Could it be ported to mozharness? In my eyes, yes, easily, without even looking at the code. I've thought about writing this script in mozharness myself in my "spare time". But again, I haven't looked at the patch.
Could it be reliant on long command line arguments? Sure, that's probably the case outside of mozharness. But that's not what I'm discussing in this blog post.
The only two scripts in mozharness, as of this writing, are for Maemo deb signing and Android multilocale.
Second, in mozharness, practically every option that's specifiable via commandline is specifiable in a config file.
What does this mean? You certainly could specify a massive command line that challenges ARG_MAX every single time. Or, if you find yourself doing this often, you could save all of that in a config file (json only, currently, but we can add .py or other filetype support relatively easily) and just run path/to/script.py --config-file path/to/config/file .
In fact, right now any mozharness script will look, by default, for ./localconfig.json and use that if no --config-file is specified. I'm debating whether this might actually be harmful in production systems, but can you really get much simpler than path/to/script.py? Or python path/to/script.py if your #! support is broken.

[fallacy #3: mozharness will replace all of buildbot]

Is mozharness powerful enough to replace all of the complex buildbotcustom.process.factory logic in buildbot? Absolutely.

Is that my short term goal? No. I have real bugs to fix. Blockers for shipping real product. Replacing working code with a rewrite-without-urgent-need is at the bottom of my todo list.

The reasons I wrote however much of mozharness I did include:

RelEng has been considering ways to move build logic to slave-side scripts, and this is my proof-of-concept
I've been trying to solve real problems with real deadlines. Like MultiNightlyL10n being the only real blocker to moving the mobile build infrastructure from the crufty buildbot-0.7 branch to the supported and shiny default 0.8.x branch.
I do secretly hope that the community will buy into this, to the point where I can afford to spend the time to do this. Because, if I haven't made it clear by this point, I believe in this. But if there's no immediate goal or community buy-in, that's a huge task to tackle.

I mean, the barrier of entry to buildbot is... high. First, install buildbot! Then, navigate our buildbot-configs and buildbotcustom repos (easy!), set up your master, then set up a slave that points to the master, then somehow use the appropriate one of the six buildbot methods to trigger a build/test/repack that you want, and debug from there.

Or, check out this repo, potentially modify this config file that's tailored to your use case, and run this script. You'll either get a [hopefully] useful error message, or your <select ... > <option ... >(while i'm promising the moon, select one)</option> <option ... >multilocale build</option> <option ... >l10n repack</option> <option ... >standalone talos performance results</option> <option ... >orange intermittent test results</option> <option ... >pgo build</option> <option ... >WHATEVER A SCRIPT CAN DO</option> <option ... >THAT WE FEEL IS WORTH THE TIME TO WRITE</option> <option ... ></option> </select> with verbose logs and python source to tweak if you want to delve into this shit.

But am I going to volunteer to port all of that stuff if people aren't into it? Fuck no. I will argue this to the ground, evidently, because I've thought about this stuff for years and years. This blog post may end up being my own personal ten fucking days rubicon, with its forward-thinking year's worth of "we could do this!" statements. But I still feel like I haven't touched on everything I've thought about over the years. And I'm defending my not-yet-fully-formed mozharness against allegations that it's going to be harmful for some reason.

Even if we did port all of factory.py to mozharness, buildbot's ability to queue and manage multiple build slave pools is a level above and beyond mozharness.

......... If this section seems less coherent and well-thought out compared to the previous sections,

tired
late
bourbon

Ok. It's late. I'm getting punchy. My profanity-to-signal ratio is rising sharply. I've written a diatribe that is probably the longest post on a script harness EVAR. And I have a nine-fucking-o-clock meeting I have to be up for. And coherent.

I'm stopping this post right meow.

[EDIT]: (let's just pretend dreamwidth/eljay didn't munge my select/option tags, shall we?)

what is mozharness?

aki@escapewindow.com — Mon, 15 Nov 2010 19:31:27 GMT

I've been relatively quiet about mozharness since it's still a work in progress. It seems to be gaining some traction, however, so here's an attempt at a one line description of what mozharness is, or at least will be:

Mozharness is a configuration-driven script harness with full logging that allows production infrastructure and individual developers to use the same scripts.

I'm sure I'll want to amend that at some point, but that works for now. Delving a bit deeper into that statement, still at a high level:

[i: Configuration-driven]

At a previous job, I had to wrangle 70,000+ lines of intertangled spaghetti perl. The original config file you gave the system often had very little to do with the workflow and end result, since hardcodes and runtime configuration shifts were liberally applied throughout the codebase. Debugging a shifting config setting which might be set to four different values at different points in the script can be extremely frustrating. As a direct result, my replacement code was strictly, strictly configuration-driven. Mozharness follows this path.

After parsing the script-level config defaults, the optional configuration file, and command line arguments, mozharness dumps the script's running configuration in a reusable config file format (currently json), and locks the running configuration.

To quote myself from here,

Any time you're trying to get around this lock, you're probably doing things wrong. It's strict, it's possibly a pain, it forces you to think about configuration first and config-driven behavior later. I try to think of ways around it as well, and bend the rules sometimes. But all in all, in my entire career, I've never been unhappy to know exactly what to expect from a script if I look at the config file + command line options. I can't say the same about mid-script configuration changes.

The dumped configuration file can be uploaded along with the logs and resulting binaries (if appropriate) for later perusal, debugging, and/or reuse.

[ii: Log everything]

When debugging a script, verbose and complete logs can very much be your friend. A paraphrased conversation from a previous job:

Boss' boss: Do we have full logs for all of that?

Me: No.

Boss: Yes.

Me: No we don't.

Boss: Yes we do.

Me: We do not! I can point out a hundred places in our code where we don't. Even if we did, we chdir, chmod, chown, rename, rmtree, system(), `backtick`, file open(), and a shit ton of other things that happen or fail silently, or get output to the console but not the logs. We. Do. Not. Log. Everything.

Boss' boss: That's hilarious. Did you guys rehearse this?

Soon afterwards I wrote my own replacement functions that did the same things, but with more error checking and full logging. With timestamps, so we could tell when a particular step was suddenly taking a lot longer than it normally did. And error parsing, with separate log files for fatal/critical/error/warning/info/debug (each a subset of the next), for easily finding the information you need. Or way more information than you need.

Mozharness follows that path, but with 100% less perl.

[iii: Modular/portable scripts]

There are so many benefits to having scripts that can be run by various people in different roles and different machines.

it kills machine-specific hardcodes (these would go into config files)
it requires documentation and some user-friendliness
it forces the script to be modular

By modular, here, I mean able to switch discrete portions of the script on or off depending on context.

For instance, a script that clobbers, pulls source, builds, packages, uploads the bits somewhere, and notifies people might insert status into a database and email tinderbox and a developer email list as part of the "notify" action.

A developer using the same script might not want to upload the build anywhere or notify anyone but herself at the end of it. To effectively share the script, those actions need to be switched on or off at will, and configurable.

Similarly, someone writing and debugging the packaging or upload portion of the script would be best served not having to start from square 1 every test run. Being able to run *only* the packaging or upload step would speed that up immensely. (Or re-running an intermitent orange test over and over without reinstalling, for example.)

As far as sharing production infrastructure scripts with developers, specifically, there are also these wins:

developers can test with the same scripts that production infra uses, removing one variable
developers can potentially provide useful patches to production infra, if they're so inclined
it helps prevent Release Engineering's automation from being a black box

This sharing of scripts has been a long term goal of mine elsewhere, but I haven't seen it fully successful and widespread as of yet. I haven't given up on it.

I think those three points provide a fairly good overview of mozharness.

The source for mozharness is here. Currently there are only scripts for creating Maemo deb repos and dealing with Android multilocale builds; this list will grow in the not too distant future.

Android multilocale builds

aki@escapewindow.com — Sat, 23 Oct 2010 21:52:26 GMT

Android multilocale builds have landed, thanks to Coop, Mwu, Armenzg's previous work, and a little thing I call mozharness. (More on that in a later post.)

These are active on mozilla-central nightly builds only.
They are currently using the Maemo multilocale locale list.
bug 596004 is currently open to determine the set of android-specific locales to include. I imagine this will result in a mobile-browser/locales/android-locales file.

Index of /pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-r7

Name	Last modified	Size

Parent Directory		-
en-US/	23-Oct-2010 13:45	-
fennec-4.0b2pre.multi.eabi-arm.apk	23-Oct-2010 13:47	14M
fennec-4.0b2pre.multi.eabi-arm.checksums	23-Oct-2010 13:47	714
fennec-4.0b2pre.multi.eabi-arm.tests.zip	23-Oct-2010 13:47	38M
fennec-4.0b2pre.multi.eabi-arm.txt	23-Oct-2010 13:47	28
gecko-unsigned-unaligned.apk	23-Oct-2010 13:47	14M

Index of /pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-r7/en-US

Name	Last modified	Size

Parent Directory		-
fennec-4.0b2pre.en-US.eabi-arm.apk	23-Oct-2010 13:45	12M
fennec-4.0b2pre.en-US.eabi-arm.checksums	23-Oct-2010 13:45	916
fennec-4.0b2pre.en-US.eabi-arm.tests.zip	23-Oct-2010 13:45	38M
fennec-4.0b2pre.en-US.eabi-arm.txt	23-Oct-2010 13:45	28
gecko-unsigned-unaligned.apk	23-Oct-2010 13:45	12M

Where are the mobile builds?

aki@escapewindow.com — Wed, 21 Jul 2010 03:08:32 GMT

We've been thinking about changing our mobile upload logic for a while now.

"mobile-trunk" and "mobile-1.9.2" were effective enough abstractions, but forced us to translate those back to branch names in automation,
uploading all platforms to the same directory resulted in same-named files being overwritten by subsequent builds (e.g. the windows l10n repack overwriting the linux l10n repack's xpi files), and
old versions of the same files would hang around in the latest nightly directories after a version bump. Or, in the case of the Maemo deb files, every build would upload a differently-named deb.

The first two issues should be solved now that bug 579097 has landed. Instead of stuffing everything into latest-mobile-trunk/, we're using a branch-platform naming structure.

This structure should also allow us to solve issue #3 elegantly. We're currently copying new files into a directory with existing files, with the assumption that they'll overwrite each other. This, of course, breaks when the filename changes. And adding per-file cleanup logic tends to be complex or messy or both.
If, instead, the latest-* directories were softlinks to the latest dated directory (e.g. latest-mozilla-central-maemo4 -> 2010/07/2010-07-20-13-mozilla-central-maemo4/), there would be no old files to clean up. I'm a big fan of directory-level upload logic, because of things like this.
This is on my radar; not written yet. Tracked in bug 580495.

So where are the builds? Still in the nightly directory. But the layout's a bit different.

For trunk (mozilla-central), look here:

Android latest-mozilla-central-android-r7/

Maemo 5 GTK (Fremantle) latest-mozilla-central-maemo5-gtk/

Maemo 5 QT (Fremantle-QT) latest-mozilla-central-maemo5-qt/

Maemo 4 (Chinook) latest-mozilla-central-maemo4/

Linux Desktop latest-mozilla-central-linux/

MacOSX Desktop latest-mozilla-central-macosx/

Windows Desktop latest-mozilla-central-win32/

For 1.1.x nightlies (mozilla-1.9.2), look here:

Maemo 5 GTK (Fremantle) latest-mozilla-1.9.2-maemo5-gtk/

Maemo 4 (Chinook) latest-mozilla-1.9.2-maemo4/

Linux Desktop latest-mozilla-1.9.2-linux/

MacOSX Desktop latest-mozilla-1.9.2-macosx/

Windows Desktop latest-mozilla-1.9.2-win32/

Depend builds are still in the misnamed tinderbox-builds directory, and the Maemo nightly update repositories are still in the same place.

new Maemo Fennec nightly .install files

aki@escapewindow.com — Wed, 23 Jun 2010 05:23:07 GMT

Some of you may have noticed that the Maemo Fennec nightly deb repositories now have multiple .install files:

Index of /pub/mozilla.org/mobile/repos/trunk_multi

Name Last modified Size Description

Parent Directory -

dists/ 22-Jun-2010 12:46 -

trunk_multi_nightly_chinook.install 22-Jun-2010 12:43 306

trunk_multi_nightly_fremantle.install 22-Jun-2010 12:44 312

trunk_multi_nightly_fremantle_qt.install 22-Jun-2010 12:46 321

Name	Last modified	Size	Description

Parent Directory		-
dists/	22-Jun-2010 12:46	-
trunk_multi_nightly_chinook.install	22-Jun-2010 12:43	306
trunk_multi_nightly_fremantle.install	22-Jun-2010 12:44	312
trunk_multi_nightly_fremantle_qt.install	22-Jun-2010 12:46	321

This is due to work done in bug 563767 (enable Maemo5 qt debs) and bug 563081 (deb repo updates with both chinook and fremantle).

Quick terminology table:

Device	OS Version	Maemo codename
N810	Maemo 4	Chinook
N900	Maemo 5	Fremantle

(The QT builds are a variant of the Maemo 5 builds, using the QT toolkit instead of GTK. If you don't know what this is, ignore it :)

These repos should be updated nightly with their corresponding builds. I have seen issues and confusion stemming from having multiple Fennec repos on a single device, so I tend to remove all Mozilla repos from the App Manager, uninstall Fennec+Mozilla Runtime, then click on the appropriate .install file.

Additional instructions available here.

(More QT repos will be available once bug 563770 (enable Maemo 5 QT l10n repacks) starts showing up in tonight's nightly builds.)

[EDIT]: I've renamed the .install files in bug 574104.

Pooling, part 3: the pooling model on multiple branches

aki@escapewindow.com — Tue, 03 Nov 2009 20:58:08 GMT

(Continuing the blogging blitz: here is pooling, part 3.)

The build pool consists of a number of identical build machines that can handle all builds of a certain category, across branches.

Builds on checkin

Pooling lends itself to building on checkin: each checkin triggers a full set of builds.

This gives much more granular information about each checkin: does it build on every platform? Does it pass all tests? This saves many hours of "who broke the build" digging. As a result, the tree should stay open longer.

The tradeoff is wait times. During peak traffic, checkins can trigger more build requests than there are available build slaves. As builds begin to queue, new build requests sit idle for longer and longer before build slaves are available to handle those requests.

You can combat wait times via queue collapsing: Once builds queue, the master can combine multiple checkins in the next build. However, this negatively affects granular per-checkin information.

Another solution to wait times is adding more build slaves to the pool.

Dynamic allocation

As long as there are available build slaves, the pool is dynamically allocated to where it's needed. If one branch is especially busy, more build slaves can be temporarily allocated to that branch. Or if the debug build takes twice as long, more build slaves can be allocated to keep it from falling behind.

(At Mozilla, this happens within Buildbot and requires no manual intervention beyond the initial configuration.)

This is in direct contrast to the tinderbox model, where busier branches or longer builds would always mean more changes per build.

Dynamic allocation adds a certain amount of fault tolerance. In the tinderbox model, a single machine going down could cause tree closure. In the pooling model, a number of build machines in the pool could fall over, and the builds would continue at a slower rate.

The main drawback to dynamic allocation is that an extremely long build or an overly busy branch can starve the other builds/branches of available build machines.

Self-testing process

In the tinderbox model, one of the weaknesses was machine setup documentation. This can be assuaged with strict change management and VM cloning, but there's no real ongoing test to verify that the documentation is up to date.

Since pooled slaves jump from build to build and from branch to branch, it's easier to detect whether breakage is build slave- or code/branch- specific. This isn't perfect, especially with heisenbugs, but it's definitely an improvement.

In addition, every time you set up a new build slave, that tests the documentation and process. This happens much, much more often than spinning up new tinderboxes in the tinderbox model.

Spinning up a new branch or build

Since the pool of slaves can handle any existing branch or build, it's relatively easy to spin up a new, compatible branch or build type. It's even possible to do so by merely updating the master config files, with none of the "spin up N new tinderbox machines" work.

However, new branches and build types do add more load to the pool; it's important to keep capacity and wait times in mind. As the full set of builds show, it's easy to lose track of just how much your build pool is responsible for.

Still, I think it's clear that this is a big Win for pooling, as the number of active branches and builds at Mozilla are as high as I've seen anywhere.

The tyranny of the single config

It's very, very powerful to have a single configuration that works for all builds across all branches. However, this is also a very strict limitation.

In the tinderbox model, a change could be made to a single machine without affecting the stability of any other builds or branches. Once that one build goes green, you're golden.

In the pooling model, the change needs to propagate across the entire pool, and it affects all builds across all branches. As the number of branches and build types grow, the testing matrix for config changes grows as well.

And at some point, new, incompatible requirements rear their ugly head -- maybe an incompatible new toolchain that can't coexist with the previous one, or a whole new platform. At that point, you need to create a new pool. And ramping that up from zero can be a time consuming process.

I hope the above helps illustrate the pooling model and some of its benefits and drawbacks.

We don't just have a single build pool here, however; we have multiple, and the number is growing. This was partially by design, and partially to deal with growing pains as we scale larger and larger.

I'll illustrate where we are today in the next segment: split pools.

Pooling, part 2: the tinderbox model on multiple branches

aki@escapewindow.com — Tue, 27 Oct 2009 19:32:28 GMT

(Continuing the blogging blitz: here is pooling, part 2.
This illustrates how the Tinderbox model can quickly become a headache to maintain on multiple branches, and what problems the pooling model is trying to solve.)

Since each column is its own build machine, if trunk has 12 columns (and you want to have the same coverage on the new branch), you need to spin up 12 new tinderbox machines with similar configurations for the new branch.

Let's reexamine the benefits and drawbacks of the Tinderbox model, with multiple branches in mind:

[i] Anyone can spin up a new builder.

If anyone wants to start working on a new project, platform, or branch, they can run their own tinderbox and send the results to the tinderbox server on their own schedule. This meant that developers could have the coverage they wished, and community members could add ports for the platforms they cared about.

After these ran for a while, they were often "donated" to the Release team to maintain.

This worked fairly well, but donated tinderboxen often came undocumented, resulting in maintenance headaches down the road. Many, many machines were labeled "Don't Touch!" because no one knew if any changes would break anything, and no one knew how to rebuild them if anything catastrophic happened.

[ii] It's relatively simple to make changes to a single build.

If a particular branch needs a different toolchain or setting, it's not difficult to set up that branch's build machine with that. In fact, when we wanted to, say, change compilers on a single branch, we usually spun up a new build machine with that new compiler, and ran it in parallel with the old one until it was reliably green.

These inconsistencies also made it difficult to determine why changes worked on one branch but not another. Was it the new compiler? Or a hidden environment variable? Were the patch/service pack levels the same? Did it matter that one tinderbox was running win2k when the other ran NT?

[iii] Consistent wait times [for a single build].

No matter how many checkins happen on any (or all) branches, wait times stay consistent.

On the other hand, if a flurry of checkins happen on trunk, and the branches lie idle, all of those changes are picked up by the trunk builders. The branch builders continue building the latest revision on those idle branches or lie idle.

The drawbacks stay the same, although amplified with each additional machine and build type to administer and maintain.

I wasn't at Mozilla at the time, but as I understand it, a little more than two years ago, the tree would regularly be held closed whenever a single build machine went down -- unscheduled downtimes on a fairly consistent basis. In addition to the tree closures required to figure out who broke the build.

These were among the reasons for the move to Buildbot pooling, which I'll cover in part 3.

Pooling, part 1: the tinderbox model on a single branch

aki@escapewindow.com — Tue, 27 Oct 2009 18:51:51 GMT

(Continuing the blogging blitz: here is pooling, part 1.
This illustrates how builds were set up at one point in Mozilla and Netscape's past, mainly to contrast with how they're set up currently.)

(from here)

There were many variations* of the old Tinderbox model of continuous integration. The basic concept involved a single machine running a single type of build (e.g., Win32 Opt Depend) on a 24/7 basis; when the previous build finished, the next build would start.

When we needed more build types (e.g. adding MacOSX coverage), we added more machines, one for each new build type. They would each be represented by their own column on the Tinderbox page, color coded green for success, red for failure, orange for test failed.

There are inherent benefits to such a model:

[i] Anyone can spin up a new builder.

This is partially due to the delivery of logs via mail (and later, in Tinderbox 2, via ftp), but also because each machine and tinderbox client is standalone. Anyone with a spare machine can spin up a Tinderbox builder.

[ii] It's relatively simple to make changes to a single build.

Need a new compiler? A different SDK? A whole new toolchain? Track down the machine running that build and make those changes, and you're done.

(You documented that, right?)

[iii] Consistent wait times [for a single build].

The maximum wait time for one build type to pick up your change is a little less than one full build cycle (if you happen to check in immediately after a build cycle starts, you need to wait for the next cycle). If a full build takes one hour, the longest end-to-end time is a little less than 2 hours. This is true whether one person checked in or five hundred people checked in.

(Later, people started running two of the same build and staggering them so that the longest wait time was a little less than 1/2 a full build cycle.)

Any drawbacks?

[i] The tree has many single points of failure.

Most of these build machines are unique. If something happens to one machine, that column goes perma-red or drops from the waterfall. If it's measuring something critical (and most of them are), that means tree closure.

[ii] It's easy to lose track of build [script|machine] changes.

It is simple to make changes to the build toolchain, scripts, or environment on individual tinderboxen. Unfortunately, it's also simple to make those changes without properly documenting or checking in those changes. It's only a matter of time before this becomes a problem.

Missing or faulty documentation might only be discovered after massive hardware failure, long after the people responsible for those changes have moved on. If you're unfortunate enough to not have a recent clone or full backup of that machine, you may be looking at a possible multi-day tree closure.

This also affects spinning up a new build machine or making changes to an existing one. If there are settings you're unaware of, troubleshooting the problem can eat up valuable time.

[iii] It's hard to track down who broke the tree.

Since each build cycle can pick up multiple checkins, it can be difficult to tell which checkin broke a particular build or test. This can become a protracted session of finger pointing, involving multiple developers and the reliability of the build machine(s) in question.

This was exacerbated by the old CVS problem of figuring out which build actually picked up your checkin. Also, the fact that each build machine (tinderbox column) has different-length cycles means that builds start at different times, and each build picks up different combinations of new checkins. Those can each break in new and exciting ways, for different reasons.

Don't get me wrong; I have a fondness for Tinderbox that it seems few people share. But I can be objective about its strengths and weaknesses, and one of its weaknesses is that it doesn't scale very well. At least not Scale with a capital Scale. (And scaling is a major factor in our decisions today.)

I'll illustrate that a bit more in the next segment: the tinderbox model on multiple branches.

^* (We did have depend tinderboxen that spit out clobber or release builds at certain times of day or when a certain file was touched. We also had machines that cycled through several different build types -- these exceptions tended to occur on side projects that had fewer developers or less hardware. But for the most part, it was a single machine for a single build type.)

Blogging Blitz: Pooling Intro

aki@escapewindow.com — Tue, 27 Oct 2009 01:08:23 GMT

As part of RelEng's Blogging Blitz, I'm going to write a bit about [build slave] pooling concepts, differentiating between the old Tinderbox model and the Buildbot pool-of-slaves model.

The topics covered will be:

The tinderbox model on a single branch.
The tinderbox model on multiple branches.
The pooling model on multiple branches.
Split pools.
Some new approaches.

Boss' boss:	Do we have full logs for all of that?
Me:	No.
Boss:	Yes.
Me:	No we don't.
Boss:	Yes we do.
Me:	We do not! I can point out a hundred places in our code where we don't. Even if we did, we chdir, chmod, chown, rename, rmtree, system(), `backtick`, file open(), and a shit ton of other things that happen or fail silently, or get output to the console but not the logs. We. Do. Not. Log. Everything.
Boss' boss:	That's hilarious. Did you guys rehearse this?

Android	latest-mozilla-central-android-r7/
Maemo 5 GTK (Fremantle)	latest-mozilla-central-maemo5-gtk/
Maemo 5 QT (Fremantle-QT)	latest-mozilla-central-maemo5-qt/
Maemo 4 (Chinook)	latest-mozilla-central-maemo4/
Linux Desktop	latest-mozilla-central-linux/
MacOSX Desktop	latest-mozilla-central-macosx/
Windows Desktop	latest-mozilla-central-win32/

Maemo 5 GTK (Fremantle)	latest-mozilla-1.9.2-maemo5-gtk/
Maemo 4 (Chinook)	latest-mozilla-1.9.2-maemo4/
Linux Desktop	latest-mozilla-1.9.2-linux/
MacOSX Desktop	latest-mozilla-1.9.2-macosx/
Windows Desktop	latest-mozilla-1.9.2-win32/