escape(window): blog

hg-git part i

2013-05-06T22:01:09Z

I was about a week into my m-c hg-git conversion^* when gps blogged about faster hg-git.

I was about two weeks into the initial conversion when I tried the latest upstream hg-git, with gps' patches. That conversion finished in less than a day, while the initial conversion was still chugging along on a different core. However, the git shas differed from our existing pre-cvs conversion repositories. This was due to different behavior when faced with a bogus email address.

Once I backed out the corresponding email munging commits, and landed Ehsan's timezone fix for a separate issue that halted the conversion process, we're now back to a converted m-c with the same shas.

At this point I killed the still-running initial conversion.

I'm still working on automating the cvs prepending and pulling in other repos/branches, but this is a good sanity-check point.

^* Note, I had pulled both mozilla-central and mozilla-b2g18 into the same working directory as a proof-of-concept workflow test before converting. In hindsight this was a terrible decision, as it more than doubled the estimated conversion time.

mozharness-based desktop unittests are live

2013-02-21T02:51:15Z

We are now running Firefox desktop unittests through mozharness for Firefox-22-based branches (mozilla-central, mozilla-inbound, try, and project branches).

Jordan Lund started this work during his internship last year, and had the mozharness scripts and configs code-complete by the end of his internship. I took over afterwards, but my efforts were sporadic, between high-priority mobile and b2g work.

This was a multi-phase operation:

We enabled them on Cedar
- This required platform-specific fixes, since we were working against a subset of platforms in staging.
- Cedar unittests were essentially the same as, say, mozilla-central, other than having the logic in mozharness instead of hardcoded in buildbot.
At this point, the majority of the tests were green, but a number were perma-orange or red or missing or had intermittent issues not present in buildbot-based unittests. So we went through and identified+fixed the various problems per-suite or per-platform, whether that was missing debug tests or barfing on leaks.
We also cleaned up issues that made these jobs hard to sheriff, e.g. missing automated retries or ugly log summaries.
Whenever P0 interrupts hit (regularly), we had to put this work on the back burner, and determine what changes had made the buildbot-based unittests diverge from the mozharness-based unittests when we revisited. This back-and-forth added months to the schedule, but allowed us to respond to higher priority tasks more quickly.
Once all the blocker bugs were resolved, I split the rollout into three parts:
1. Get mozilla-tests/config.py to not warn in python mode (pep8). This wasn't critical, but helped avoid spurious conflicts later. Easy peasy. (comment 8 through comment 10)
2. Split out the mobile tests into their own config file. This helped me limit the scope of my config.py rewrites to desktop only. (comment 11 through comment 28)
3. The final set of patches made mozharness-based unittests default for Firefox desktop, with exceptions for aurora, beta, release, esr, and b2g18*. (comment 29 through comment 60)
I ran through a number of spot tests in staging to limit risk for rollout; I also waited until after merge day so the unittests could ride the trains. So far the only fallout is some bustage in debug b2g emulator tests on b2g18 and b2g18_v1_0_1 due to missing symbols (now fixed; we also have bugs to enable symbols filed).

Making large sweeping changes in infrastructure while

responding to constant priority interrupts (often for weeks or months at a time), and
avoiding bustage

is a long game. But we're playing for keeps.

Thanks to Jordan for the initial work around this, and A-Team, sheriffs, and RelEng for their help + patience.

mozharness 0.6

2013-02-01T04:20:52Z

" I certainly don't want to wait another 5 months before the next arbitrary version bump."

-- me, 11 months ago.

What's happened in the past 11 months? In no particular order,

completed Android Native support,
Fennec armv6+noion support,
broadened VirtualenvMixin functionality (setup.py support, pip freeze, PyWin32 support, --find-links, ...),
further mozbase support / integration,
developer-runnable Android multilocale build support,
Marionette tests on Firefox desktop,
mock (mock_mozilla) support,
tooltool support,
retry support,
Boot2Gecko (FirefoxOS) device builds (panda, unagi, unagi_eng, unagi_stable, otoro) (all multilocale),
Boot2Gecko desktop builds (all multilocale),
Boot2Gecko emulator tests: marionette-webapi, mochitests, reftests, crashtests, xpcshell,
mozpool support for Boot2Gecko + Android Pandas,
Boot2Gecko Panda Gaia UI tests,
Android signing on demand,
TBPL parser compatibility improvements,
per-locale status in mobile_l10n.py,
mozharness Firefox desktop unittests (ready to roll out),
mozharness Firefox desktop talos support (baking on Cedar),
proof-of-concept jetperf,
proof-of-concept remote (mobile device) talos support (needs work),
further Peptest support (which has since been disabled),
MPL 2.0,
and a mozharness production branch, which will let us run CI tests against default before merging, to avoid production bustage.

So, yeah. Some minor version bumps are larger than others. We've gotten all this done thanks to great efforts by the A-Team and RelEng.

B2G automation is largely mozharness-based; mobile is maybe half-mozharness and Firefox desktop is still largely buildbot-based. But, without actually crunching numbers, I would guess that rolling out mozharness Firefox desktop unittests would push us somewhere near the halfway point. Soon.

I'm glad to see the project continuing to gain traction, and we've been seeing concrete evidence of the theoretical benefits I've been touting since the beginning.

This time I'm not going to make any predictions about when I'll bump the arbitrary version next, but I'll schedule something in Things.

x: back to arusha, then home. 13-14 oct 2012

2012-10-25T05:12:39Z

On Saturday morning we took a very quick game drive, staying close to camp. It was more of a farewell round than anything, a good close to the week. On the way to the airstrip it seemed like animal after animal came out to say goodbye.

We flew back to Arusha, where we visited a souvenir shop and art gallery. Then to a day room near the airport, where we showered and re-packed our luggage for the international flight home. And I managed to get rid of the rice in the sandbag that I really didn't use that much all trip.

I managed to steal some pictures of Arusha this time.

Some geeky camera and travel thoughts:

I love my new camera.
The 100-400mm isn't as sharp, or fast, or as sexy as a super telephoto prime. But it's faster to deploy, easier to carry, and less unwieldy in cramped quarters. Faster at grabbing difficult shots in limited time. At least until I get some practice time in with the big guns.
Next time, if I go on another more photo-oriented safari, I'm thinking a super telephoto prime, a mid-range telephoto zoom, and a wide lens like a 35mm f/1.4 for documenting the trip. And give IR another shot.
Here's a thought. Since I used vacuum sealed bags filled with air for padding, maybe I could protect my monopod with... water wings?
While I didn't use my sandbags much (plural; one stayed in my duffel all trip), I used the heck out of my tiny tabletop Manfrotto tripod. Maybe because I was going with a lighter long lens.
I used the heck out of the 2x extender too.
I kind of missed my 7D. With its 1.6 crop factor, my effective 800mm f/11 would have become an effective 1280mm f/11, which would have helped with the rhinos, leopards, and serval cat.
There wasn't any need for me to change money.
Laundry was super cheap at Ngorongoro Sopa Lodge (about a buck an item). And Sayari camp was all-inclusive.
I should have tried longer exposures and lower ISOs on the leopard cub.
I think I'm close, photography wise. Not yet good enough to be pro, but promising enough to think I can if I dedicate myself. Keep at it. And buy more gear ;-) ;-)

The rest of the photos are here.

Previously:

ix: the perfect day, part ii. 12 oct 2012

2012-10-25T04:26:07Z

Back at camp, a zebra baby and mum were eating grass and nursing just outside our tent. This was a good omen for the evening.

We started our evening safari in great spirits, wondering how we could possibly top the cheetah sighting earlier. (I secretly wanted to go back.)

Then we came across the newborn zebra, still wearing its afterbirth. The perfect finale to an amazing trip.

(The bright flashes are me getting worried about depth of field + focus, and stepping down the aperture.)

An ibis posed in the water for us afterwards.

Our final campfire, with drinks, and final dinner, were sentimental, with speeches. Tim, a National Geographic employee, had his photos running in a slideshow -- mine are unabashedly wildlife-centric; his actually told the story of the entire trip quite eloquently, giving me ideas for future trips.

This was the perfect day.

The rest of the photos are here.

Previously:

viii: the perfect day, part i. 12 oct 2012

2012-10-25T04:09:47Z

We started early the next morning, and caught the leopard mum at their home rock.

(tight crop)

We saw wildebeest on the move.

Our driver got stuck twice: once in deep muddy ruts, and once as we were crossing a stream. That one required a rope, another car, and people leaning on the side of the car to keep it from rolling over into the water.

After a breakfast in the field, we got a lot of pictures of an old male lion lounging in the shade, then an entire pride of lions snoozing: one here, one there, four there, two there.

Then we saw a gathering of cars, and drove over post haste. Cheetahs.

</e>

The cubs followed their mom a distance as she stared at some distant gazelle? and I did everything I could to prepare to capture her running. It wasn't to be, however; we left before anything happened.

Still, I'm pretty stoked we saw 4 cheetahs. That's over 1% of the regional population.

We headed back for camp to rest for the afternoon, and planned to head back out in the evening for the second half of our perfect day.

The rest of the photos are here.

Previously:

vii: northern serengeti. sayari camp. 11 oct 2012

2012-10-25T03:37:32Z

On Thursday, we took a quick morning safari before heading to the Seronera airstrip, where we took two Cessnas to northern Serengeti.

Our new vehicles were fully open on the sides but covered on top. I found it a bit more limiting for photography but managed to do pretty ok anyway.

We were greeted by the Mara river, and giraffes, and baby warthogs.

Sayari camp was beautiful, and welcoming, and we needed escorts to walk us to and from our tents at night, since animals regularly walked through camp.

We went back out, braving the rain.

We drove up to the Kenya border.

At some point, I took this photo, and when I got back I photoshopped it as best as I was able. I'm mixed. I think I'm going to stick with the subtler infrared filters and less photoshop.

At the end of the day, as we were hydroplaning a bit over muddy roads (dusty or muddy, nothing in between!) we caught sight of a leopard cub. Our first. At a great distance, in the dark.

The rest of the photos are here.

Previously:

vi: olduvai gorge. seronera. serengeti. 10 oct 2012

2012-10-25T02:52:07Z

In the morning, we left early to get to Olduvai Gorge before the heat, and because Louise Leakey had a plane to catch. We saw the original site and spent some time in the museum. I decided to start using my largely-ignored-til-now full-spectrum/infrared camera at this point, and was pretty happy with the results.

We entered Serengeti National Park and saw a pride of lions sleeping in the shade.

We stopped in Seronera for lunch, where there were hyrax waiting to eat any scraps we dropped. One of them peed in Marc's coffee as he cut my smuggled leftover birthday cake from the night before. :)

While photographing dwarf mongoose, I dropped my 5d2 and broke my 715nm infrared filter. I still had a 590nm infrared filter so I tried that for the rest of the trip, but I'm just not as into the results. The colors are gaudier and I don't have the chops to take advantage of the super color in photoshop. I have a new 715nm filter on its way.

We saw some lion cubs crossing the road with their mother, and watched the pride for a while.

When we heard of something exciting in the distance, we took off, full speed, and decided the hotel could wait a while. We sped along for a while until we realized we were heading right into a storm, and headed back to the hotel. (I later found we were trying for a leopard, which would have been our first.)

On the way to the hotel, we found an elephant in the road, drinking rainwater.

We checked into the Serengeti Sopa Lodge, again with the customary wet washcloth and fresh fruit juice welcome. The view outside our balcony was gorgeous, and included some monkeys on the balcony next door.

The rest of the photos are here.

Previously:

v: ngorongoro crater. best birthday ever. 9 oct 2012

2012-10-24T06:01:47Z

On Tuesday we woke early, to enter the park as soon as it opened. The first animal we saw was a baby jackal, which I took as a good omen.

Soon afterwards, we were driving full speed over the dusty road, catching air whenever we hit a particularly big bump. Marc said this was the rarest of mammals we would see all trip: the rhino, which was down to 27? in the region due to poaching.

(I didn't expect to use my 2x extender nearly so much, but it turned out to be very handy. These guys are still small, even at 800mm.)

We saw a couple zebras bickering.

We ate breakfast by the water. Hippos in the water, zebras and cape buffalo drinking at the watering hole. Little birds stayed close as we ate, knowing someone would drop something tasty.

After breakfast, we saw the wildebeest running, as if from something, and then we saw it -- a large male, awake.

Not too far off, we saw a large cluster of vehicles, a sure sign there was something exciting to see. And there was: a cape buffalo / pride of lions standoff.

I could have stayed and watched all day.

Further along, we saw 181 grey crowned cranes by the water -- as I understand it, the majority of the regional population, and a larger group than had been sighted together to date this year.

To end the safari, we saw a serval cat hunting, and I managed to capture it as it pounced.

We had the afternoon off, during which I exhaustedly tried to nap. I ended up skipping dinner to go to bed early, which meant I missed my surprise birthday celebration. I'm fine with that; it was the best birthday ever.

The rest of the photos are here.

Previously:

iv: maasai. ngorongoro crater. 8 oct 2012

2012-10-24T05:10:50Z

On Monday morning, we visited a Maasai chief. He answered questions about his culture: how all cows belong to the Maasai; how they herd the cows away when wildebeest are giving birth, since the afterbirth is poisonous to the cows; polygamy; how they traditionally stained their furs red but now bought red-dyed clothing.

After lunch we drove to Ngorongoro Crater, where, because of the protected conservation area, there were lots of animals. Grey crowned cranes; jackals; hyenas.

We drove up to a hippo pool and saw them up close. None of this struggling-at-800mm like the day before.

A lion was sleeping in the distance, our first big cat.

Further along, we saw two lions sleeping next to the road, lazy after mating. They rolled over, and I managed to get a picture when the male lifted his head, just for a moment.

We arrived at Ngorongoro Sopa Lodge just as the sun was setting.

At dinner, the lights went out, then the servers came out with a lit torch, singing in Swahili; I guessed this was their birthday ritual.

The rest of the photos are here.

Previously:

iii: lake manyara. plantation lodge. 7 oct 2012

2012-10-24T02:08:25Z

Our second safari started in the morning, which made for a cooler drive and more active animals. This was back at Lake Manyara, but the difference between this morning and the previous afternoon was stark.

We saw a baboon grooming a blue monkey, which Marc Baker (our guide) hadn't ever seen or heard of before. Turns out someone has seen the reverse, also at Lake Manyara. Perhaps this is an interspecies friendship or romance.

We watched a family of baboons in the trees. Ammy, our driver, told us that you can watch baboons and elephants all day; they're always doing something interesting. Lions tend to sleep.

We saw our first hippos way off in the distance, snoozing. Along with wildebeest, zebra, a warthog. We were all excited by this, but had this happened a few days later we wouldn't have even blinked. The trip really did improve day by day.

We stopped when we saw an elephant (with both tusks this time) eating seeds by the side of the road. We were so close, and he let us watch and photograph him for a long long time.

Some more photos from the morning, including a baby baboon and a peeing giraffe:

In the afternoon we walked around the coffee plantation, which was nice, since we didn't get much exercise the rest of the week. The whole "don't leave the vehicle or you'll get eaten" rule kind of put a damper on that. Daniel from the Lion Project, who we met later, jumped rope for exercise.

The rest of the photos are here.

Previously:

ii: meetup. first safari - lake manyara. plantation lodge. 6 oct 2012

2012-10-23T06:54:36Z

We met our travel companions on the morning of the sixth. As always, I left the people-photographing to others, so I'm hoping we do end up sharing photos. We all got along pretty well.

We headed out for Lake Manyara, which we reached in the afternoon. Our Land Cruisers had pop tops, but without the miniature roofs, which allowed for greater freedom.

I think of that first outing as getting our feet wet; a chance to see what it would be like, without overwhelming us. I definitely had too much gear along with me; by the end of the trip I left my Kiboko at camp and was a lot comfier.

Our new digs were pretty swank. Evidently we were spoiled -- others had small rooms, but for some reason we had the huge palatial room.

That one's the bathroom. For the bath. And the shower. The toilet's in the next room. There was also an upstairs with a divan that didn't see any use.

Our dining room.

The rest of the photos are here.

Previously:

i: departure. arusha coffee lodge. 3-5 oct, 2012

i: departure. arusha coffee lodge. 3-5 oct, 2012

2012-10-23T06:22:25Z

Months of anticipation and preparation: camera research; safari blogs; a trip to Safari West; vaccinations. Considering each item for its utility and weight, to stay below the 33 pound limit; packing and repacking.

I flew to Tanzania on October 3, meeting my father in a long transfer line at Schiphol as we changed planes. My previous strategy of downloading a season of television on my phone worked well, again: this time it was Firefly. That, and Sid Meier's Pirates.

After some twenty-plus hours of travel, we set foot in Africa for the first time. Albert had been sent to pick me up at Kilimanjaro, but thought there would only be one passenger, which worried us, but it turned out to be fine. He checked his papers in the car and we were off to Arusha.

The roads were strewn with speed bumps: three small, one large, three small. My standing theory: these replaced speed limit signs and laws. The houses by the side of the road were largely dark shapes of rusted corrugated iron, some ready to fall over with a strong huff and puff.

I kind of wish I had documented the actual trip a bit more, rather than focusing so strongly on wildlife, and to a lesser degree, landscapes. The hotels were all beautiful and comfortable and guilt-inducingly luxurious. The food was tasty. But I had my focus, and something had to take a back seat. I recuperated in the evenings rather than pore over photos and write up blog posts; these scribblings are all from memory.

Albert taught us Swahili phrases as we drove, and I dozed off for a bit. We were greeted at the hotel by wet cloths, fresh fruit juice, and iced coffee. The room had a canopy-like mosquito net.

We had arrived a day earlier, and spent the next day relaxing. We asked about taking a walk outside the plantation, and were warned that it would be dangerous for us.

Yuki: Danger from animals? Or humans?

Guard: Humans.

We stayed inside.

My father wanted to see a big cat of some sort. (Later this turned into a major hankering for a leopard sighting.) I wanted to see, and more importantly, photograph, a cheetah kill.

Building Android multilocale locally

2012-05-31T20:00:49Z

Tl;dr: it's now easier to run multilocale builds locally. I'd love feedback in terms of whether this is useful for you, and if there are changes that would make this script better for your workflow.

I updated the wiki as well.

what & why

There are only three ways to get a multilocale build currently:

triggering a nightly build off of a multilocale-enabled branch,
pinging me directly and asking for a multilocale build, or
building multilocale locally. Up until now, that involved a combination of digging through logs, asking for help, and reverse-engineering the multilocale script with a lot of trial and error.

The latter was so difficult because I hadn't spent the time to make it friendly to run locally. Now that bug 753545 has landed, it should now be relatively easy.

The config file is written to [hopefully] be easily editable and usable.

running the script

</p>

Create a directory to run in. It's simplest to create a new directory, although you can use an existing directory with mozilla-central (or other tree) already checked out.
```
mkdir multilocale
cd multilocale
```

Clone mozharness and copy the standalone config file for easy editing+use.

hg clone http://hg.mozilla.org/build/mozharness
cp mozharness/configs/multi_locale/standalone_mozilla-central.py myconfig.py

Customize your config file.
By default it's going to clone mozilla-central into a directory called mozilla-central.
It also assumes your objdir will be named objdir-droid and that will be specified in your mozconfig, which will be in a file named mozconfig in this directory.
This should hopefully be easy to edit. Holler if it's confusing.
Afterwards, you can verify it's still valid python by typing
```
python myconfig.py
```
which should exit silently.
If your config file still says that your mozconfig is in os.path.join(os.getcwd(), "mozconfig"), copy your mozconfig into this directory and make sure it's named mozconfig.
Make sure mozilla-central (or whatever repo you specified) is in this directory.
You can either clone it manually, or use the script's pull-build-source action:
```
mozharness/scripts/multil10n.py --cfg myconfig.py --pull-build-source
```
Run the script.
By default, this will:
- pull the locale source,
- copy your mozconfig into the tree and build mozilla-central,
- package the en-US apk,
- backup your objdir to a -bak directory (e.g. objdir-droid-bak) via rsync,
- restore your objdir via rsync (no changes),
- add the locales to your objdir,
- and package the multilocale build.
(See the actions section below for more advanced usage.)
```
mozharness/scripts/multil10n.py --cfg myconfig.py
```

developing + debugging

The above instructions are sufficient for when you want a single multilocale build.

If you're changing m-c code or otherwise want multiple builds, you probably want to follow this workflow:

After running the multilocale steps, you want to restore your objdir to en-US. At least until someone figures out how to clobber the multilocale-specific bits, rsyncing from a clean backup directory seemed like the most expedient solution.
```
mozharness/scripts/multil10n.py --cfg myconfig.py --restore-objdir
```
Then make your code changes.
Then run the default actions again. If this seems like it's doing more than it needs to, you're correct. See the actions section below for more advanced usage.
```
mozharness/scripts/multil10n.py --cfg myconfig.py
```

actions

(I covered actions to some degree here as well, but the below specifically covers this script + config file.)

what actions are available?

You can see the default actions, defined in the config file, by:

mozharness/scripts/multil10n.py --cfg myconfig.py --list-actions

This will give you this output:

Actions available: clobber, pull-build-source, pull-locale-source, build, package-en-US, upload-en-US, backup-objdir, restore-objdir, add-locales, package-multi, upload-multi Default actions: pull-locale-source, build, package-en-US, backup-objdir, restore-objdir, add-locales, package-multi

what do these actions do?

I'd like to have more information about each action built into each script. Here's an overview of multil10n.py's actions:

clobber - This is effectively no-op as long as work_dir is set to '.'. Otherwise it nukes the entire work_dir.
pull-build-source - This clones the repos defined in repos.
pull-locale-source - This clones the repos defined in l10n_repos, as well as the l10n repositories defined in the locales file (specified here)
build - This copies the mozconfig path specified here into the source tree, and runs make -f client.mk build.
package-en-US - This runs make package.
upload-en-US - To be written. It will call make upload.
backup-objdir - This rsyncs your objdir to a backup directory, since there's no current solution for clobbering the multilocale bits in your objdir.
restore-objdir - This rsyncs the backup directory into your objdir. If you want to re-run the build or re-add the locales without a lot of old cruft lying around, I recommend doing this first.
add-locales - This runs compare-locales and adds the locale strings + manifests into your objdir.
package-multi - This runs make package for your multilocale apk.
upload-multi - To be written. It will call make upload.

how do I run a single action?

You can run any single action by itself by prepending -- in front of the action name (making it a command line option). For instance, to only run the restore-objdir action,

mozharness/scripts/multil10n.py --cfg myconfig.py --restore-objdir

how do I skip an action?

You can run the default set of actions, minus an action, by prepending --no- to the action(s) you want to skip. To run the default actions without pulling the locale source,

mozharness/scripts/multil10n.py --cfg myconfig.py --no-pull-locale-source

how do I run several actions at once?

Running the script with no action-specific command line options will iterate over the list of default actions.

You can also have a combination of --ACTION or --no-ACTION to add an action to or remove an action from the default list. For example,

mozharness/scripts/multil10n.py --cfg myconfig.py --no-pull-locale-source --no-build --no-package-en-US --no-backup-objdir

and

mozharness/scripts/multil10n.py --cfg myconfig.py --restore-objdir --add-locales --package-multi

are equivalent.

You also have the option of editing your default_actions in myconfig.py if that makes things easier.

mozharness presentation

2012-04-19T01:40:22Z

For those interested, I gave an overview of mozharness for the Automation Team today.

I mentioned that we're not yet at that point of critical mass where the accumulated knowledge and shared code help make everything go faster. But we're getting there.

The presentation "slides" are available here.

mozharness 0.5

2012-02-29T01:35:37Z

A lot's happened since mozharness 0.4 landed in late September. We:

added a bunch of Android native support.

enabled Android single locale repacks with nightly update snippet generation
ported Android signing with passphrase verification, signature verification, single locale support, and update snippet creation.

rolled peptest out to production on Try, across all desktop test platforms.
improved mozharness virtualenv support, with real-life mozbase usage in peptest.
fixed actions-in-config-files.
separated the output parser from ShellMixin.run_command(), so we can

parse output from subprocess or get_output_from_command(),
eventually add context lines to output parsing, and
potentially split serial tasks into multiple parallel jobs with their own log parsing.

added chunking support to split up jobs across machines.
added query_exe() and which() support to specify or find executables.
added add_failure(), query_failure(), and summarize_success_count() support to track granular status across a list of tasks.
added a BuildbotMixin and ReleaseMixin to tie into our existing buildbot infrastructure/configs.
precompiled the error_list regexes.
made various OSMixin and ShellMixin method improvements.
added a setup.py.
added a pyflakes call and Debian/Ubuntu support to unit.sh.
moved mozilla-specific modules into mozharness.mozilla.*.
retired Maemo scripts (Maemo tier 3).

This feels like an as-good-as-any time to arbitrarily increment the arbitrary version number: mozharness 0.5.

I think the best part of this release is how more people got involved; I can feel the momentum building. I certainly don't want to wait another 5 months before the next arbitrary version bump.

mobile/ is now mobile/xul/

2011-11-17T16:47:17Z

Today we moved mobile to mobile/xul in mozilla-central, in preparation for landing the native UI code in mobile/android.
Mozilla-aurora, mozilla-beta, and mozilla-release are currently unaffected.

Branch	Android platform	Mobile source location
mozilla-central	android	mobile/android
mozilla-central	android-xul	mobile/xul
mozilla-aurora	linux-android	mobile
mozilla-beta	linux-android	mobile
mozilla-release	linux-android	mobile

NOTE: mobile/android has not yet landed. The builds are switched over and will need to be hidden. I'm doing that as I see them, but would appreciate any help in this matter.

The mobile team has said that mobile/android is at least a few days away from landing.

Attention project branch owners! If you have Android builds on your project branch:

You will need to merge from mozilla-central for the 'android-xul' builds to go green.
The 'android' builds will burn until mobile/android lands and you re-merge from mozilla-central. These builds need to be hidden.

Attention trychooser users! You probably want the 'android-xul' platform. The mozconfig situation will be a little convoluted for now, so it would be best to land a mozconfig-extra-android-xul at the top of your push to make sure you get the right settings.

Attention l10n users! Due to the move of mobile/locales, we have broken mobile localization on mozilla-central. We will be fixing this and finding a single locale repack solution ASAP.

device_talosrunner.py: a first peek

2011-11-05T02:03:32Z

[what is it?]

device_talosrunner.py is a work-in-progress mozharness script that sets up talos and runs it against a device using either the sut or adb protocols.

It's designed to:

allow anyone with a [supported and rooted] Android device to run talos without building the tree;
allow someone with access to a staging/production tegra to run a production-like talos run without setting up buildbot;
at some point, become the official way we run talos on Android in production.

This is similar to, and parallel to, the work on make talos-remote. However, aiui, that is developer-oriented, whereas device_talosrunner.py is targeted towards someone who wants to test pre-built apks (as well as replacing our current production code).

I have used this script to run talos on both a tegra over sut, and an Asus Eee Pad Transformer using adb-over-ip, successfully.

[how do i use it?]

You need:

python (2.5 - 2.7.x)
virtualenv
hg, to clone the various repos
a copy of my github mozharness repo, on the branch 'talosrunner';
a rooted, supported Android device that is either running the sutagent, or is attach{ed,able} via adb. The adb calls require busybox.

Once you have those things,

Create a config file. I've got a wip tablet config that I use for my Asus Eee Pad Transformer, and a wip tegra config that I use for a staging tegra.
If you're using adb-over-usb, the device needs to be connected to your desktop/laptop. If you're using adb-over-ip or sut, specifying the device_ip should be sufficient.
If this is the first time you're running device_talosrunner.py, you need a virtualenv. Run
mozharness/scripts/device_talosrunner.py --cfg users/aki/tablet1.py --create-virtualenv
(replacing the config file path with the path to your config file). This will create a python virtualenv in ./venv and install PyYAML, a Talos dependency, into it.
Run
mozharness/scripts/device_talosrunner.py --cfg users/aki/tablet1.py
If the start_python_webserver option is set, the script will start a webserver and point the device at it. Otherwise it'll point the device at whatever talos_webserver is set.

When this script is polished and mature, I definitely want people to be comfortable using it on their personal phones and tablets, but it's still rough around the edges.

This will uninstall Fennec and possibly reboot your device, so please be careful if you're trying this on a device/profile that you haven't backed up.

[what's left to do?]

I've only tested a handful of talos suites against two devices; for this to be a production replacement, it needs to be able to run reliably against a large pool of devices.
This relies on a talos patch in bug 688604 for the python webserver; I most likely need to use mozhttpd.
I've hit issues with both m-c and birch builds, and will most likely need to make changes for the native UI.
I have an idea which would allow us to find the next available and working tegra, which may reduce red and purple Tegra results; this needs writing and testing. More on that later.

Galápagos: Saturday, October 22 - Tuesday October 25: GPS -> GYE -> JFK -> SFO

2011-11-01T08:55:19Z

[Endeavor -> Baltra airport]

I woke early Saturday, since I needed to put my checked luggage outside the door of my cabin before breakfast. And since the crew had two hours between our vacating our cabins and the next week's passengers arriving to change the sheets. I decided to give them a bit more time.

The ship has a two week agenda. We saw the south and east islands; the next week's group would see the young, western islands. Lava. Isabela and Fernandina. Flightless cormorants. Between this and the dry/wet season split, I feel like I've seen about a quarter of what there is to see, and I didn't see everything this time around, either.

They played Lindblad Expedition videos as we waited to leave after breakfast... and I want to go on them all. The Antarctic, yes. Seeing the polar bear pressing its face against the ship's porthole to look inside (and hearing Birgit talk about that, the other night) definitely whetted my appetite for the Arctic, before it's gone. The Amazon seemed fun. Baja is close but similarly matches desert and ocean. We didn't get to the Costa Rica portion; it was time to depart.

Our last zodiac ride was a bit sad. I gave Gaby a hug at the dock; all the naturalists were great but I had been with her for some of the most memorable walks.

gaby

a blurry gaby, fernando, aura, celso

sofia, juan carlos, aura

I spotted the last two Giftmas items I needed for the nieces -- bought 'em and then headed into the airport, where there was free wifi. Ahhhh.

[Guayaquil]

Back at Guayaquil, where we were all a bit frazzled. I wanted to go on the city tour despite only having an hour between arriving at the hotel and leaving -- essentially only a half hour in the hotel room to wash up and go out again.

I did enjoy it though, the boardwalk and the colorful old town. Tempered by the sight of malnourished kids trying to sell candy and the knowledge that hailing a taxi on the street was asking to get robbed.

It was nice traveling with only my point and shoot and iPhone camera, after lugging two DSLRs around all week.

Got back, worked out, showered. Headed down to the bar, where many of us were gathered before dinner; it was like the ship, except no soothing announcement from Cindy.

I ate with the nineteen -- those who had gone on the Machu Picchu extension trip before the Galapagos cruise. Now I'm wondering why I hadn't gone, as well; it sounded fun and the group was very tight. I think I was worried about needing more clothes and more camera gear, which was valid. I think limiting myself on this first solo trip might have been good as well.

The prawns were frickin HUGE and seriously tasty.

Afterwards I joined Kristie and Maureen back at the bar and got tipsy on pricy booze and stayed up past my bedtime. It was fun though.

[JFK]

The smooth travels so far? GYE -> JFK not so much.

Getting called in to have my luggage opened + inspected at GYE, passing by drug sniffing dogs; not being able to communicate well with the Spanish-speaking security staff;
the screaming babies on the plane all the way to JFK, and I mean screaming. all. the. way;
noticing that the airline had failed to give me a US customs form after waiting in line, forcing me to double back, find a form, fill it out, and wait in line again;
US customs stopping me to check my bags again at JFK

all added up to a relieved Aki once I was at my hotel shortly after 11pm.

Staying in Manhattan for a day seems odd, especially after a trip to the Galápagos. I think it was a good end to my vacation, though. A full day to myself, guided only by my iPhone, Yelp, and vague "hm, I want to [eat|see|do] ____".

a bowl of ramen;
Adorama. I got to B&H just when they closed, but that's definitely on my list for next time;
stopped by a Barnes&Noble to pick up The Beak of the Finch, which Alan was trying to read on the plane. It's a fascinating read so far, and I'm only a few chapters in. Much better/more thorough than the Daphne Major documentary we watched, though the documentary whetted my appetite, surely;
stopped by Guitar Center on a whim;
a slice of New York pizza;
Zuccotti Park and #ows;
a quick walk by the 9/11 tribute grounds;
pastrami sammich;
just a general feel for the city that you can only get by walking through it. Bus tours have a different feel.

Google maps tells me a direct route from my hotel to Zuccotti Park was ~4miles. I'm guessing I walked 9-10 miles that day.

I definitely want to spend more time there at some point.

[JFK -> SFO]

The flight home was uneventful. I'm still not all unpacked, nearly a week later.

I'm so glad I brought:

my cotton carrier; this allowed me to lug around my 100-400mm with ease.
both my dslrs, though I wish I had brought my 24-105mm instead of my 17-40mm, given how often I was photographing at the 40mm end of that lens. I was using my 7D + 100-400mm between 80-90% of the time. still, being able to switch from telephoto to wide angle near instantaneously helped me capture a lot of shots I wouldn't have gotten otherwise.
my point and shoot and underwater housing were useful, though other passengers were getting better results with smaller cheaper underwater cameras.

The bulk of my camera accessories (filters, mini tripod, 1.4x extender) were all unused. Speed and not having to fiddle with things are so important when you have limited time on the islands, and when you're traveling with a group and need to keep up.

... So many great memories. I'd recommend this trip in a second. Already thinking about where I want to go next.

Galápagos: Friday, October 21: Genovesa

2011-11-01T07:52:57Z

[Prince Philip's Steps]

Genovesa was the bird island. The steps led up a cliff wall; at the top was a bunch of leafless trees (dry season) where red-footed boobies nested. Red-footed boobies can, despite their webbed feet, grasp tree branches. The branches give them cover against other birds (frigates and hawks iirc).

red feet, blue beaks

I definitely got my practice in photographing birds in flight this day.

We spent a lot of time looking for owls, and finally spotted one, far in the distance. I managed to get a number of shots at the far end of my 100-400mm, which was the highlight of the morning.

[Snorkeling]

The snorkeling was decent. Our group managed to find Helen's sunglasses that she dropped in the water on the way down from Prince Philip's steps. I think my underwater point-and-shoot skills were the best of the trip, though. Part of it was getting rid of the orange filter; the rest was knowing the settings that worked for me and knowing to get close.

Practice makes perfect, I think.

[Departure briefing]

Afterwards we had our departure briefing, which was sad. I left a tip for the crew+naturalists and donated to the Galápagos Lindblad fund.

[Regatta]

There was a bit of a tradition -- the passengers against the crew, in a kayak race around a zodiac and back. Cindy insisted the night before that the crew never win, but despite our best efforts, the crew came away with first prize. However, there were prizes for the first crew team and the first passenger team, and lots of amusement all around.

[Darwin Bay beach trail]

Finally, we had a wet landing on the beach, the one wet landing where you needed shoes at all, really. I wasn't looking forward to it since I had rubbed my feet raw in my flip-flops the very first day at Cerro Brujo, but I changed into my hiking boots on the beach and all was good.

Birds were nesting all around and flying overhead; more bird-on-the-wing photography! I spent a decent amount of time trying to catch the sunlight behind the white feathers of the swallow-tailed gull.

After hiking over the lava rocks to see the view, we saw gulls mating.

(slideshow)

Galápagos: Thursday, October 20: Bartolomé & Sombrero Chino

2011-11-01T03:49:34Z

We woke early, again, to hike to the top of Bartolomé Islet, to see and photograph the iconic Galápagos view.

The snorkeling off the beach of Bartolomé might have been the best of the entire trip. A couple penguins jumped in the water with us, and I saw 3 different white tip reef sharks, one almost close enough to touch.

fish

shark

I'm still very much a bad underwater photographer, but had fun doing it. Some of the others are getting some great shots underwater. I think I'd throw all mine out if they came out like that on land.

After lunch, on the way to Sombrero Chino, we passed by an island with an old crater, and in passing caught a maybe 30 second glimpse of flamingos in the lagoon inside that crater. I managed to get a few not-too-bad-for-being-very-distant shots. It's kind of awesome how they were able to tell us practically to the minute when that view would appear.

I missed the 2nd round of snorkeling, but caught the zodiac ride near sunset, where we saw sea lions playing near the shore. We circled several times, waiting for penguins, but nothing. As it grew dark and we prepared to turn around, a penguin swam up to the lava rocks; we converged; it climbed up and we all were taking a ton of pictures. Then, as we watched, a bunch more penguins swam up and congregated on the rocks. Very, very special.

I cranked the ISO on these last two, as it got dark, so they look brighter than the earlier ones... Can you tell I don't do a lot of editing?

Dinner was an outdoor barbeque where Cindy, our expedition leader, told us we needed to visit South Georgia... Falklands, South Georgia, and the Antarctic peninsula is a full 3 weeks. "Galapagos on steroids" is her description of it.

A bunch more people were out looking for activity in the water; word spreads fast. Wasn't quite the same; headed inside and made the mistake of checking some work email.

(slideshow)

Galápagos: Wednesday, October 19: Santa Cruz (Cerro Dragon + Daphne Major)

2011-10-31T05:23:34Z

In the morning we went ashore at Cerro Dragon to see land iguanas -- we saw a few, but the highlight was probably both the arid landscape and the flamingos.

Later in the morning we went snorkeling by the cliffside to see invertebrates -- I saw coral, sea urchins, stars, a sea snake, and a sleeping sea lion -- others saw a manta ray, two octopi, and a swimming sea lion that passed right by them. Slightly jealous, but I'll live :)

After lunch Celso gave a presentation on Darwin. I didn't go kayaking, again, but I did go on a zodiac ride... and got some pictures, notably some distorted pictures of a ray.

The "What Darwin never saw" documentary was cool -- essentially, the Galápagos are well suited for noticeable evolution because there are multiple islands that have different climates and ecosystems that favor different traits, and there are fairly regular events where a large percentage of populations (finches, in this case) die out, favoring the traits that are best adapted for that island's ecology.

Champagne on the bow, though it got dark before we could circumnavigate Daphne Major. After dinner, however, while I was talking with Betty about which five photos I should submit to the slideshow, Helen called us out to the deck -- there were sea lions and sharks in the water, drawn to the light. We hung out for a long while out there, watching the dark forms of sea lions hunt and the forms of sharks swimming through the water, some turtles too. That was cool.

(Slideshow)

Galápagos: Tuesday, October 18: Santa Cruz (Puerto Ayora)

2011-10-31T03:40:23Z

We went ashore at Puerto Ayora, the economic hub of the islands, where we went to the Charles Darwin Research Station to see their tortoise breeding program. So many cute tortoises!

There are tortoise species specific to each island, evolved to match the food supply. At the CDRS they breed the tortoises and bring them back to each island when they're old enough to survive on their own.

We got to see Lonesome George, who may be the last Pinta tortoise.

They also had some Galápagos land iguanas.

A few of the naturalists were able to visit their families, which was good to see. The walk to town was fun; I picked up some presents for the family. Then we took a bus to the highlands.

The sugar cane mill was a nice diversion. I quite enjoyed lunch at a highlands restaurant called Altair. They had an owl nesting in a building in their backyard; when I went to photograph it, everyone handed me their camera to take a picture. That was fun, though I learned I don't have a clue how to navigate around manual mode on a Nikon.

The pit craters weren't particularly interesting to me, since I'm not a huge botanist, and the photo opportunities were slim. The walk through the tortoise migratory route was pretty amazing, though, since there were dozens of tortoises, in the wild, along the way.

During that walk we also walked into a lava tube. We also spotted a rare vermillion flycatcher.

We picked up some high schoolers who had won a school contest, and get to ride with us on the ship. They had never been on a boat and have never snorkeled; I learned second hand that most of the locals aren't allowed into most of the national park areas at all (97% of the islands). I do think that showing them what they're preserving will help local conservation efforts.

(slideshow)

Galápagos: Monday, October 17: Floreana

2011-10-31T00:00:37Z

Up at 5:45, 15 minutes before the optional wakeup call for the wet landing at Point Cormorant.

This was a sea turtle nesting spot. The nesting mostly happens at night, so we caught sight of a sea turtle in the surf, but none on land. The salt water lagoon was low on water, which meant no shrimp, which meant no flamingos.

dry season

I skipped the quick Zodiac ride around the islands, but went for the deep water snorkeling at Champion Islet.

When we jumped overboard, the water was cool, again, but what first struck me was the hundreds and hundreds of fish swimming maybe ten feet below us.

We swam along the rock wall to our right, with the ocean floor going deep below us to the left, and saw many fish, some starfish, and thousands of spiny sea urchins. The highlight was probably the diving and somersaulting young sea lion; Celso, our naturalist, somersaulted with it for a little while, which was fun to watch. I wish I had thought of getting video of that.

(Here, again, unfamiliarity with the camera bit me; I took some video of some fish, or thought I did. Afterwards I realized I had *started* recording when I thought I was *stopping* the recording, so my two underwater videos were of odd angles of nothing particularly interesting, as I swam around looking for more fish. Good thing it was only my underwater point-n-shoot that this was true for.)

After lunch, a nap sounded more appealing than kayaking + zodiac ride + the post office barrel, so I dozed off for the afternoon. Later I found that they had spotted penguins [!] though not terribly close and against the black lava background. Hopefully I'll be able to catch some penguins later in the week. (*me, reading this now: [!] :)

Birgit's post-dinner crittercam presentation was pretty cool. We saw clips of emperor penguins hunting, sharks hunting, seals and sea lions capturing and eating octopi, whales feeding, sea turtles looking for mates, and even a cat catching and eating rats, all from their points of view.

Tomorrow looks amazing, at Santa Cruz -- the Tortoise Rearing Center, the Charles Darwin Research Center, a sugar cane mill, a highlands forest, and some local shops during a full day out all sound wonderful.

(slideshow)

Galápagos: Sunday, October 16: Española

2011-10-30T23:36:57Z

[Dawn]

It's near impossible to drag me out of bed to check email and get to work. Yesterday I woke at 4:30 to fly to the Galápagos, and today I woke at 5:50 to see the dawn across the ocean. I think morning-person-ness is all in the context.

Speaking of, in the I'm-a-bit-surprised-but-it-makes-sense-really-department: at the equator, sunrise is a few minutes off of 6am; sunset is a few minutes off of 6pm, and when it sets it sets fast. And it's like that year-round. If you're looking to bask in hours and hours of sunlight, you're better off going toward the poles during the summer.

(It was overcast and there was no magnificent sunrise to be seen; I went back to bed.)

((My writing style is choppy & mixed tense & inconsistent; this is probably because I wrote the bulk of this on the ship, edited, and added more at home when prepping pictures.))

[Española]

Española is the oldest of the Galápagos islands, which are built from a volcanic hotspot in the northwest and slowly travel southeast via tectonic movement. The southeast currents hammer the island, which is slowly dying; when it's eroded below sea level, it will be gone forever.

Española is the breeding/nesting ground for albatrosses, blue-footed boobies, and sea lions.

[Deep water snorkeling]

Breakfast, then snorkel gear (L fits me for a shorty wetsuit) then deep water snorkeling. Cindy, our expedition leader, made this sound a bit daunting. (If you weren't sure about anything, you should practice off the beach, since otherwise you might be sitting in the Zodiac the whole time and ruining things for your buddy.)

But really, it was easy and fine and felt great going in -- water a bit cold but not too bad, especially with the wetsuit.

I was predictably awful with my unfamiliar underwater camera. I had some ideas of how to improve it the next time, and considered leaving it behind the next time too.

Sofia, the naturalist for the first snorkel group, swam with a real underwater video camera. She followed schools of fish closely and was fun to watch; very graceful in the water. Seems like a fun job :)

I didn't know it at the time, but the snorkeling would only get better from here.

I had a chance to go back to the beach afterwards, but I chose to shower and dress in a bit more of a relaxed fashion.

lobster carapace

[Punta Suarez]

This was a dry landing, and a 1.5-2mi hike over rocky terrain. It wasn't so hard, but a lot of the others had trouble with it or took the short route.

I loved it.

It was sea lion pup season; lots of babies and pregnant mothers.

It was also baby albatross season. Blue footed boobies, endemic mockingbirds, a Galapagos hawk, crabs, marine iguanas, frigate birds, albatrosses, yellow warblers.

We saw an albatross mating dance.

We saw a pair of blue footed boobies nesting a pebble, which they will do if they've found a good nesting site but don't have an egg yet.

(I really love this shot)

We saw a blowhole -- a narrow gap between rocks that the surf shot into and up with a burst of spray, maybe 20, 30 feet in the air.

One of the other travelers had a 500mm lens, trying to get shots of birds on the wing, but I think I held my own with my 100-400mm.

Sea lions and blue footed boobies were directly on the path a lot of the way. I kind of want to see the wet season now, but the dry season is great for fish and birds.

After dinner one of the naturalists, Fernando, talked about how to take better pictures. I knew most of it but it was good for a review, and a lot of people asked me for help.

(slideshow; lots more pics)