TL;DR Talos runs performance tests on Firefox e10s on mozilla-central, not yet on try-server. OS X still doesn’t work. e10s reaults are similar, with notable scroll performance improvement on Windows and Linux, and notable WebGL regression on Windows.

Electrolysis, or e10s, is a Firefox project whose goal is to spread the work of browsing the web over multiple processes. The main initial goal is to separate the UI from web content and reduce negative effects one could have over the other.

e10s is already enabled by default on Firefox Nightly builds, and tabs which run on a different process than the UI are marked with an underline at the tab’s title.

While currently the e10s team’s main focus is correctness more than performance (one bug list and another), we can start collecting performance data and understand roughly where we stand.

jmaher, wlach and myself worked to make Talos run well in e10s Firefox and provide meaningful results. The Talos harness and tests now run well on Windows and Linux, while OS X should be handled shortly (bug 1124728). Session restore tests are still not working with e10s (bug 1098357).

Talos e10s tests run by default on m-c pushes, though Treeherder still hides the e10s results (they can be unhidden from the top right corner of the Treeherder job page).

To compare e10s Talos results with non-e10s we use compare.py, a script which is available in the Talos repository. We’ve improved it recently to make such comparisons more useful. It’s also possible to use the compare-talos web tool.

Here are some numbers on Windows 7 and Ubuntu 32 comparing e10s to non-e10s Talos results of a recent build using compare.py (the output below has been made more readable but the numbers have not been modified).

At the beginning of each line:

• A plus + means that e10s is better.
• A minus - means that e10s is worse.

The change % value simply compare the numbers on both sides. For most tests raw numbers are lower-is-better and therefore a negative percentage means that e10s is better. Tests where higher-is-better are marked with an asterix * near the percentage value (and for these values positive percentage means that e10s is better).

Descriptions of all Talos tests and what their numbers mean.

$ python compare.py --compare-e10s --rev 42afc7ef5ccb --pgo --verbose --branch Firefox --platform Win7 --master-revision 42afc7ef5ccb

Windows 7          [ non-e10s ]             [  e10s   ]
                   [ results  ]   change %  [ results ]

-   tresize               15.1   [  +1.7%]      15.4
-   kraken              1529.3   [  +3.9%]    1589.3
+   v8_7               17798.4   [  +1.6%]*  18080.1
+   dromaeo_css         5815.2   [  +3.7%]*   6033.2
-   dromaeo_dom         1310.6   [  -0.5%]*   1304.5
+   a11yr                178.7   [  -0.2%]     178.5
++  ts_paint             797.7   [ -47.8%]     416.3
+   tpaint               155.3   [  -4.2%]     148.8
++  tsvgr_opacity        228.2   [ -56.5%]      99.2
-   tp5o                 225.4   [  +5.3%]     237.3
+   tart                   8.6   [  -1.0%]       8.5
+   tcanvasmark         5696.9   [  +0.6%]*   5732.0
++  tsvgx                199.1   [ -24.7%]     149.8
+   tscrollx               3.0   [  -0.2%]       3.0
--- glterrain              5.1   [+268.9%]      18.9
+   cart                  53.5   [  -1.2%]      52.8
++  tp5o_scroll            3.4   [ -13.0%]       3.0


$ python compare.py --compare-e10s --rev 42afc7ef5ccb --pgo --verbose --branch Firefox --platform Linux --master-revision 42afc7ef5ccb

Ubuntu 32          [ non-e10s ]             [  e10s   ]
                   [ results  ]    change   [ results ]

++  tresize               17.2   [ -25.1%]      12.9
-   kraken              1571.8   [  +2.2%]    1606.6
+   v8_7               19309.3   [  +0.5%]*  19399.8
+   dromaeo_css         5646.3   [  +3.9%]*   5866.8
+   dromaeo_dom         1129.1   [  +3.9%]*   1173.0
-   a11yr                241.5   [  +5.0%]     253.5
++  ts_paint             876.3   [ -50.6%]     432.6
-   tpaint               197.4   [  +5.2%]     207.6
++  tsvgr_opacity        218.3   [ -60.6%]      86.0
--  tp5o                 269.2   [ +21.8%]     328.0
--  tart                   6.2   [ +13.9%]       7.1
--  tcanvasmark         8153.4   [ -15.6%]*   6877.7
--  tsvgx                580.8   [ +10.2%]     639.7
++  tscrollx               9.1   [ -16.5%]       7.6
+   glterrain             22.6   [  -1.4%]      22.3
-   cart                  42.0   [  +6.5%]      44.7
++  tp5o_scroll            8.8   [ -12.4%]       7.7

For the most part, the Talos scores are comparable with a few improvements and a few regressions - most of them relatively small. Windows e10s results fare a bit better than Linux results.

Overall, that’s a great starting point for e10s!

A noticeable improvement on both platforms is tp5o-scroll. This test scrolls the top-50 Alexa pages and measures how fast it can iterate with vsync disabled (ASAP mode).

A noticeable regression on Windows is WebGL (glterrain) - Firefox with e10s performs roughly 3x slower than non-e10s Firefox - bug 1028859 (bug 1144906 should also help for Windows).

A supposed notable improvement is of the tsvg-opacity test, however, this test is sometimes too sensitive to underlying platform changes (regardless of e10s), and we should probably keep an eye on it (yet again, e.g. bug 1027481).

We don’t have bugs filed yet for most Talos e10s regressions since we don’t have systems in place to alert us of them, and it’s still not trivial for developers to obtain e10s test results (e10s doesn’t run on try-server yet, and on m-c it also doesn’t run on every batch of pushes). See bug 1144120.

Snappiness is something that both the performance team and the e10s team care deeply about, and so we’ll be working closely together when it comes time to focus on making multi-process Firefox zippy.

Thanks to vladan and mconley for their valuable comments.