TL;DR Talos runs performance tests on Firefox e10s on mozilla-central, not yet on try-server. OS X still doesn’t work. e10s reaults are similar, with notable scroll performance improvement on Windows and Linux, and notable WebGL regression on Windows.
Electrolysis, or e10s, is a Firefox project whose goal is to spread the work of browsing the web over multiple processes. The main initial goal is to separate the UI from web content and reduce negative effects one could have over the other.
e10s is already enabled by default on Firefox Nightly builds, and tabs which run on a different process than the UI are marked with an underline at the tab’s title.
jmaher, wlach and myself worked to make Talos run well in e10s Firefox and provide meaningful results. The Talos harness and tests now run well on Windows and Linux, while OS X should be handled shortly (bug 1124728). Session restore tests are still not working with e10s (bug 1098357).
Talos e10s tests run by default on m-c pushes, though Treeherder still hides the e10s results (they can be unhidden from the top right corner of the Treeherder job page).
To compare e10s Talos results with non-e10s we use
compare.py, a script which is available in the Talos repository. We’ve improved it recently to make such comparisons more useful. It’s also possible to use the compare-talos web tool.
Here are some numbers on Windows 7 and Ubuntu 32 comparing e10s to non-e10s Talos results of a recent build using
compare.py (the output below has been made more readable but the numbers have not been modified).
At the beginning of each line:
- A plus
+means that e10s is better.
- A minus
-means that e10s is worse.
change % value simply compare the numbers on both sides. For most tests raw numbers are lower-is-better and therefore a negative percentage means that e10s is better. Tests where higher-is-better are marked with an asterix
* near the percentage value (and for these values positive percentage means that e10s is better).
Descriptions of all Talos tests and what their numbers mean.
$ python compare.py --compare-e10s --rev 42afc7ef5ccb --pgo --verbose --branch Firefox --platform Win7 --master-revision 42afc7ef5ccb Windows 7 [ non-e10s ] [ e10s ] [ results ] change % [ results ] - tresize 15.1 [ +1.7%] 15.4 - kraken 1529.3 [ +3.9%] 1589.3 + v8_7 17798.4 [ +1.6%]* 18080.1 + dromaeo_css 5815.2 [ +3.7%]* 6033.2 - dromaeo_dom 1310.6 [ -0.5%]* 1304.5 + a11yr 178.7 [ -0.2%] 178.5 ++ ts_paint 797.7 [ -47.8%] 416.3 + tpaint 155.3 [ -4.2%] 148.8 ++ tsvgr_opacity 228.2 [ -56.5%] 99.2 - tp5o 225.4 [ +5.3%] 237.3 + tart 8.6 [ -1.0%] 8.5 + tcanvasmark 5696.9 [ +0.6%]* 5732.0 ++ tsvgx 199.1 [ -24.7%] 149.8 + tscrollx 3.0 [ -0.2%] 3.0 --- glterrain 5.1 [+268.9%] 18.9 + cart 53.5 [ -1.2%] 52.8 ++ tp5o_scroll 3.4 [ -13.0%] 3.0 $ python compare.py --compare-e10s --rev 42afc7ef5ccb --pgo --verbose --branch Firefox --platform Linux --master-revision 42afc7ef5ccb Ubuntu 32 [ non-e10s ] [ e10s ] [ results ] change [ results ] ++ tresize 17.2 [ -25.1%] 12.9 - kraken 1571.8 [ +2.2%] 1606.6 + v8_7 19309.3 [ +0.5%]* 19399.8 + dromaeo_css 5646.3 [ +3.9%]* 5866.8 + dromaeo_dom 1129.1 [ +3.9%]* 1173.0 - a11yr 241.5 [ +5.0%] 253.5 ++ ts_paint 876.3 [ -50.6%] 432.6 - tpaint 197.4 [ +5.2%] 207.6 ++ tsvgr_opacity 218.3 [ -60.6%] 86.0 -- tp5o 269.2 [ +21.8%] 328.0 -- tart 6.2 [ +13.9%] 7.1 -- tcanvasmark 8153.4 [ -15.6%]* 6877.7 -- tsvgx 580.8 [ +10.2%] 639.7 ++ tscrollx 9.1 [ -16.5%] 7.6 + glterrain 22.6 [ -1.4%] 22.3 - cart 42.0 [ +6.5%] 44.7 ++ tp5o_scroll 8.8 [ -12.4%] 7.7
For the most part, the Talos scores are comparable with a few improvements and a few regressions - most of them relatively small. Windows e10s results fare a bit better than Linux results.
Overall, that’s a great starting point for e10s!
A noticeable improvement on both platforms is tp5o-scroll. This test scrolls the top-50 Alexa pages and measures how fast it can iterate with vsync disabled (ASAP mode).
A supposed notable improvement is of the tsvg-opacity test, however, this test is sometimes too sensitive to underlying platform changes (regardless of e10s), and we should probably keep an eye on it (yet again, e.g. bug 1027481).
We don’t have bugs filed yet for most Talos e10s regressions since we don’t have systems in place to alert us of them, and it’s still not trivial for developers to obtain e10s test results (e10s doesn’t run on try-server yet, and on m-c it also doesn’t run on every batch of pushes). See bug 1144120.
Snappiness is something that both the performance team and the e10s team care deeply about, and so we’ll be working closely together when it comes time to focus on making multi-process Firefox zippy.
Thanks to vladan and mconley for their valuable comments.