• In general, Fennec scrolls worse than Chrome, especially on non trivial pages. But there are other issues too. We filed the following bugs:
  • Bug 1217415 - Fennec page navigation is slower than Chrome on some sites (e.g. Wikipedia, ebay)
  • Bug 1217372 - Fennec has text input lag in autocomplete boxes (google, bing) which Chrome doesn’t.
  • Bug 1217370 - On fast scroll swipes, sometimes the momentum is less than expected (and less than Chrome).
  • Bug 1217364 - Inconsistent scroll progression (momentum) without user inputs.
  • Bug 1217366 - Visible low resolution rendering, especially for fast swipes.

Hopefully, at least some of the scroll issues would be resolved or at the very least improved once Fennec gets async-pan-and-zoom (APZ). APZ is expected to land soon on Fennec nightly 44 before it becomes Aurora (on Firefox-desktop - APZ is already enabled by default on nightly builds).

Content-perf observations

Also on the subject of content-perf, we’ve recently filed quite a few bugs for desktop Firefox which were observed throughout our experiments. Vladan also blogged about it here and more recently here.

However, the bugs which we filed usually relate to specific cases or issues, but don’t expose non-issues, i.e. cases where Firefox is similar or better than other browsers, and they also don’t expose the scope of the experiments and the big picture in order to better assess the weight of the existing issues, nor do they expose general observations which were made.

To address this, we created a content-performance observations page. This page summarizes all the experiments which we performed, including their scope, procedures and observations, for both Desktop Firefox and fennec.

Let us know if you have any feedback on existing experiments and results, or suggestions for more experiments on either Desktop or Android, especially if there are no existing bugs where such discussion could take place.

Electrolysis, or e10s, is a Firefox project whose goal is to spread the work of browsing the web over multiple processes. The main initial goal is to separate the UI from web content and reduce negative effects one could have over the other.

e10s is already enabled by default on Firefox Nightly builds, and tabs which run on a different process than the UI are marked with an underline at the tab’s title.

While currently the e10s team’s main focus is correctness more than performance (one bug list and another), we can start collecting performance data and understand roughly where we stand.

jmaher, wlach and myself worked to make Talos run well in e10s Firefox and provide meaningful results. The Talos harness and tests now run well on Windows and Linux, while OS X should be handled shortly (bug 1124728). Session restore tests are still not working with e10s (bug 1098357).

Talos e10s tests run by default on m-c pushes, though Treeherder still hides the e10s results (they can be unhidden from the top right corner of the Treeherder job page).

To compare e10s Talos results with non-e10s we use compare.py, a script which is available in the Talos repository. We’ve improved it recently to make such comparisons more useful. It’s also possible to use the compare-talos web tool.

Here are some numbers on Windows 7 and Ubuntu 32 comparing e10s to non-e10s Talos results of a recent build using compare.py (the output below has been made more readable but the numbers have not been modified).

At the beginning of each line:

• A plus + means that e10s is better.
• A minus - means that e10s is worse.

The change % value simply compare the numbers on both sides. For most tests raw numbers are lower-is-better and therefore a negative percentage means that e10s is better. Tests where higher-is-better are marked with an asterix * near the percentage value (and for these values positive percentage means that e10s is better).

Descriptions of all Talos tests and what their numbers mean.

$ python compare.py --compare-e10s --rev 42afc7ef5ccb --pgo --verbose --branch Firefox --platform Win7 --master-revision 42afc7ef5ccb

Windows 7          [ non-e10s ]             [  e10s   ]
                   [ results  ]   change %  [ results ]

-   tresize               15.1   [  +1.7%]      15.4
-   kraken              1529.3   [  +3.9%]    1589.3
+   v8_7               17798.4   [  +1.6%]*  18080.1
+   dromaeo_css         5815.2   [  +3.7%]*   6033.2
-   dromaeo_dom         1310.6   [  -0.5%]*   1304.5
+   a11yr                178.7   [  -0.2%]     178.5
++  ts_paint             797.7   [ -47.8%]     416.3
+   tpaint               155.3   [  -4.2%]     148.8
++  tsvgr_opacity        228.2   [ -56.5%]      99.2
-   tp5o                 225.4   [  +5.3%]     237.3
+   tart                   8.6   [  -1.0%]       8.5
+   tcanvasmark         5696.9   [  +0.6%]*   5732.0
++  tsvgx                199.1   [ -24.7%]     149.8
+   tscrollx               3.0   [  -0.2%]       3.0
--- glterrain              5.1   [+268.9%]      18.9
+   cart                  53.5   [  -1.2%]      52.8
++  tp5o_scroll            3.4   [ -13.0%]       3.0


$ python compare.py --compare-e10s --rev 42afc7ef5ccb --pgo --verbose --branch Firefox --platform Linux --master-revision 42afc7ef5ccb

Ubuntu 32          [ non-e10s ]             [  e10s   ]
                   [ results  ]    change   [ results ]

++  tresize               17.2   [ -25.1%]      12.9
-   kraken              1571.8   [  +2.2%]    1606.6
+   v8_7               19309.3   [  +0.5%]*  19399.8
+   dromaeo_css         5646.3   [  +3.9%]*   5866.8
+   dromaeo_dom         1129.1   [  +3.9%]*   1173.0
-   a11yr                241.5   [  +5.0%]     253.5
++  ts_paint             876.3   [ -50.6%]     432.6
-   tpaint               197.4   [  +5.2%]     207.6
++  tsvgr_opacity        218.3   [ -60.6%]      86.0
--  tp5o                 269.2   [ +21.8%]     328.0
--  tart                   6.2   [ +13.9%]       7.1
--  tcanvasmark         8153.4   [ -15.6%]*   6877.7
--  tsvgx                580.8   [ +10.2%]     639.7
++  tscrollx               9.1   [ -16.5%]       7.6
+   glterrain             22.6   [  -1.4%]      22.3
-   cart                  42.0   [  +6.5%]      44.7
++  tp5o_scroll            8.8   [ -12.4%]       7.7

For the most part, the Talos scores are comparable with a few improvements and a few regressions - most of them relatively small. Windows e10s results fare a bit better than Linux results.

Overall, that’s a great starting point for e10s!

A noticeable improvement on both platforms is tp5o-scroll. This test scrolls the top-50 Alexa pages and measures how fast it can iterate with vsync disabled (ASAP mode).

A noticeable regression on Windows is WebGL (glterrain) - Firefox with e10s performs roughly 3x slower than non-e10s Firefox - bug 1028859 (bug 1144906 should also help for Windows).

A supposed notable improvement is of the tsvg-opacity test, however, this test is sometimes too sensitive to underlying platform changes (regardless of e10s), and we should probably keep an eye on it (yet again, e.g. bug 1027481).

We don’t have bugs filed yet for most Talos e10s regressions since we don’t have systems in place to alert us of them, and it’s still not trivial for developers to obtain e10s test results (e10s doesn’t run on try-server yet, and on m-c it also doesn’t run on every batch of pushes). See bug 1144120.

Snappiness is something that both the performance team and the e10s team care deeply about, and so we’ll be working closely together when it comes time to focus on making multi-process Firefox zippy.

Thanks to vladan and mconley for their valuable comments.

Now, activeTicks is quite simple - it counts the duration a user has been interacting with Firefox. It’s part of the Firefox-Health-Report (FHR), but since it would be super useful to correlate this value with other telemetry values (such as number of hangs, etc), and since Telemetry and FHR live on different Clouds, we wanted to have a copy of this number also at telemetry. Fair enough.

I found the activeTicks code and played with it a bit to understand how it works. It turns out to simply count the number of “time units” (5s) at which the user moves the mouse over Firefox or otherwise interacted with it. There’s only one place at the code which counts this value, and it’s at services/datareporting/sessions.jsm. Great - this appears to be a simple task then.

While not strictly related to our story, I also found out that on Windows systems - it incorrectly kept counting also when the mouse pointer hovers Firefox even if the user is away from the computer. Georg Fritzsche, who maintains the data reporting modules took it and then also handled the request to keep both the “old” and the “fixed” values reported in parallel for a while - to better understand the relation between them. That story is on bug 1107779 and bug 1107782.

Back to our story.

Now, Telemetry mostly holds histograms - for which it has a decent API, but this activeTicks value is a plain scalar which doesn’t really fit nicely into a histogram. Telemetry also happens to have a “Simple Measurements” section which holds various values - most of them are scalars (such as various startup times, number of maximum concurrent threads, etc), but also some JSON strings which are compound UI related stuff, etc.

You can view the simple measurements values if you enter about:telemetry at the URL bar of Firefox and expand the SimpleMeasurements section.

Unfortunately, simple measurements is a bit of a hack. It doesn’t have an API, and each module which reports a simple measurement value also implements the interfacing with it. Also unfortunate - interfacing with it from the code which counts activeTicks was not straight forward.

So I looked a bit more on types of telemetry values and then I noticed Count Histograms. The docs were a bit sparse, but it was being tested and seemed to work well when I experimented with it, it was designed to hold a single value, and best of all - has an API which fits activeTicks perfectly: myHistogram.add(). Simple counting is exactly what we need! Did I mention it works?

• Patch v1: bullet-proof single line of code which increments a telemetry count whenever activeTicks is incremented for FHR.

Bump v1: while a count histogram has an almost-perfect semantic fit and a working API for our needs, apparently the telemetry dashboard currently doesn’t display count histograms as nicely or as usefully as it does for simple measurements.

Since we’re in a bit of a hurry with this, and since fixing the telemetry dashboard is likely to be a longer path, after some discussion we decided to use simple measurements after all - a fair request considering the factors.

But due to the lack of simple measurements API - instead of using the API right where activeTicks is incremented, it was suggested to collect the value where the telemetry values were collected - similar to some of the other simple measurements values.

However, since this part of the code doesn’t have an obvious interface to the SessionRecorder module (an object which exposes the activeTicks value and is an instance which the datareporting service - DRS - holds), it was suggested to simply take the preference value which counts activeTicks - since the only thing which incrementing activeTicks does is updating this pref.

While a bit hacky, it’s still quite safe, and it was suggested that such approach would still get approved. Fair enough.

• Patch v2: relatively safe implementation which - when collecting the simple measurements telemetry values - also collects the activeTicks preference value if FHR is enabled.

Bump v2: the module owner isn’t happy to “bypass the FHR API”, but since such API didn’t exist, it was suggested to add an API. A fair request considering that it’s only a single function which exposes the SessionRecorder instance of the datareporting service.

• Patch v3:
  • add an API to the DRS which exposes the SessionRecorder instance - if it has one (it may not have one if a mostly-unused preference disables FHR).
  • still relatively safe: if the DRS has a SessionRecorder instance, use its activeTicks getter (which always existed), else, report an activeTicks value of -1.

Bump v3: while the patch was quite safe, it was preferred to also have some “basic-coverage test” for this new mechanism. A very fair request.

Unfortunately, the owner was unavailable for clarifications, but nevertheless, I evaluated what might fail, concluded that only the API could fail, and wrote a new xpcshell test which covers several cases of this new API.

• Bump v4. Apparently I misinterpreted the request, and it was preferred to have more of a unit/black-box test where the telemetry payload is tested to contain -1 or actual activeTicks value - according to various cases, and it was suggested that we could add that to an existing test which already tests the telemetry payload - test_TelemetryPing.js . A fair request - black-box tests are good, and enhancing an existing test sounds reasonable.

• Bump v5: Turns out that while the code itself is quite safe, it’s not trivial to create a case which ends up with activeTicks as -1 since it only happens when the DRS is present but FHR is completely disabled - which doesn’t happen at the code by default.

Still, it was deemed important to test this -1 case, and after several failed attempts to initialize the DRS later at the test (in order to check the payload before the DRS instantiates SessionRecorder - which should end up with activeTicks==-1), we came up with a “solution”: Let’s initialize the datareporting service twice!

First we’ll set this rarely-used preference to disable FHR, init the DRS, get a telemetry payload and check that it has activeTicks==-1, then restore the FHR pref, re-initialize the DRS (which luckily doesn’t do much - but does instantiates SessionRecorder according to this pref) - and then continue the test as usual.

And the patch lands.

• Bump v6: and gets backed out for failure on some Android systems - which apparently are not compiled with the datareporting service and as a result part of the new test which uses it throws an exception. Whoops. My bad.

• Patch v7: Fix it, do a try push with xpcshell tests on every platform I could select on trychooser - we’re green! re-land.

Bump v8: a day goes by, and it turns out the new test fails for Thunderbird, and the try-push didn’t cover Thunderbird. Apparently Thunderbird does have the datareporting service, but it doesn’t have the rarely-used FHR preference, and so when we try to read the pref initially such that we can restore its value later - it throws an exception.

Clone and build Thunderbird. Run this single test to check out what happens there.

The test doesn’t run. Apparently running xpcshell tests with Thunderbird works a bit differently than Firefox.

Search for docs, find docs, try again. Doesn’t work - I can’t run only this single test. I did manage to run all the xpcshell tests, but after about 15 mins where they were still running (and didn’t get to “my” test yet) I decided this is not practical and aborted.

Asked around, didn’t get too far, started experimenting with test invocations, found out how to invoke-a-Thunderbird-xpcshell-test-which-resides-at-the-mozilla-central-repository-copy-which-Thunderbird-depends-on, updated the wiki with this missing info (need to run from within the Thunderbird obj dir but with the test path relative to the m-c repo rather than relative to the Thunderbird repo).

Back to the test. Added some code to also test the case where the FHR preference doesn’t exist and expect activeTicks==-1 on this case. Run the test. Fails. It seems that while the preference doesn’t exist, the DRS still instantiates SessionRecorder, and activeTicks actually ends up as working - but the test expected it to have -1 since the FHR pref doesn’t exist.

Examined the DRS code - it reads the preference with a fallback to true, so it ends up with SessionRecorder and working activeTicks also when the pref doesn’t exist.

Hmm…

Falling back to true when the FHR-enabled pref doesn’t exist is a bit of a dodgy logic (even if it shouldn’t happen in practice), and I don’t like to duplicate logic, and especially not dodgy ones. But I need this logic at the test in order to know what kind of activeTicks value to expect on every platform which the test runs on.

I could expose this logic as yet another API of the datareporting service, and only use it at the test, but then who knows who might use it later and for what.

Or I could change the logic to be more reasonable and don’t instantiate SessionRecorder if the pref doesn’t exist, but I wouldn’t dare do that since I have no clue which other parts of the code rely on this behaviour. I need a solution…

• Patch v8: duplicate that logic at the test, add a comment on both places where this logic is implemented, and add a small safeguard at the test to cover some of these assumptions which the test now makes.

This seems to work locally for both Thunderbird and Firefox for desktop.

Another try-push on all platforms with xpcshell tests to make sure everything still works everywhere.

Start this blog posts while the tests are running. They just turned out all green, ask a review for the new patch. Yay!

• Summary (for now):
  • Where we started:
    • A single bullet proof line of code which increases a telemetry count whenever FHR increases activeTicks.
  • Where we are following several very reasonable incremental requests, and some days later:
    • A fairly safe new API for the datareporting service to expose SessionRecorder.
    • A fairly safe code which uses this API when telemetry pings are collected.
    • A fairly delicate and hacky test which:
      • Initializes the datareporting service twice.
      • Duplicates dodgy internal logic which the actual code doesn’t care about.
      • Has logic for combinations of the DRS and FHR-enabled pref which the actual code doesn’t care about, and probably nothing cares about - except for this test.
      • Mini test-self-test to validate some of these assumptions.

It’s probably a safe bet to say that the test will break before the code which it tests breaks. This can’t be a good thing..

There are probably few junctions where we could have taken a different path and possibly end up with better/simpler/safer/smaller code, and yet, this is where it stands now.

Moral? Not sure. All the requests were reasonable and each step kinda makes sense on its own. Maybe we should have done the right thing (TM) by keeping the first patch and fixing the telemetry dashboard instead? ;)

Easy to say in retrospective though, when we see how it evolved and which thing led to what.

But hey, we did end up with some more useful info at the Thunderbird-xpcshell-tests wiki page!

Edit - a day later and I decided to make it right. Gone are the double init and too much logic. The final patch is much cleaner, though still didn’t land. Tomorrow hopefully!

Talos tsvg and tscroll are about to be replaced with tsvgx and tscrollx, respectively (bug 897054). The main difference is that the “x” tests stress Firefox by iterating animations as fast as possible, AKA “ASAP mode”.

The old tests were performing some animation and then report overall (or average per frame) duration. However, they were using intervals which were not stressing Firefox at all, making the results almost meaningless WRT svg/scroll performance, and rather mostly sensitive to timing changes.

Stressing Firefox exposed some issues such as paint starvation (bug 880036).

Hopefully the new tests will have better correlation to our performance. Joel Maher did (and still does) a lot of work on the Talos side of things with these new tests (a l-o-t!)

There’s a more technical explanation on this dev.platform post.

TART - Tab Animation Regression Test

After previous work which went into improving tab animations in Firefox, It’s time to put it into Talos. TART is implemented as an addon which works either stand alone or from within Talos. It measures frames intervals during different animation cases, and it works equally well on mozilla-central and on the UX branch.

TART uses ASAP mode to iterate animation with unlimited frame rate, thus able to expose differences even if under normal conditions it would have been limited to 60Hz.

Joel Maher did most of the talos side of things, and currently the addon awaits review.

• TART tests animation on 3 main cases:
  • “simple” - open/close of about:blank.
  • “icon” - open/close of a blank html page with favicon and a long title.
  • “newtab” - open with and without newtab preload.

“simple” is tested at DPI scaling of 1.0, “icon” at scaling of 1.0 and 2.0, and “newtab” at default scaling.

• Each animation produces 3 result values:
  • “all” - Overall average interval.
  • “half” - Average interval over 50% of the designated duration - from the end of the animation backwards.
  • “error” - The difference between the designated duration to the actual one.

Typically, “half” is the value which corresponds to the stable part of the animation once it’s hopefully settled. Most tab animations have first frame (or few) which are longer than the rest, so the “all” value, while not meaningless, doesn’t tell the whole story.

• Some issues which had to be resolved while working on TART:
  • Inactive tabs are throttled, so to get accurate timing (intervals, end event), the test is executed from the Chrome window (rather than from the tab itself).
  • At DPI scaling of 2.0 at the talos window size, Australis can’t open a second tab without shrinking the first (thus measures animation of more than one tab). Solution: keep the TART tab pinned.
  • On OS X, “ASAP mode” doesn’t work with OMTC. Solution for now: disable OMTC for TART.
  • On ASAP mode, mostly on OSX, paints can be starved (bug 880036). Solution: use a temporary pref to prevent this (bug 884955).
  • At DPI scaling of 2.0, Australis can only have 3 tabs before it starts overflowing the tabstrip (e.g. bug 891450). This made it very hard to test multi-tab shrink/expand in a comparable fashion on m-c and Australis, and eventually this test was dropped for TART v1.

You can watch TART at bug 848358. Other than a review, it also awaits bug 888899 which should enable “ASAP” (stress) mode on OS X as well.

Absolute scroll smoothness on windows

I was asked by Taras to look into scroll smoothness on windows. Apparently, even after vlad’s great timing improvement, the timers filter removal and making vsync on windows work, Firefox still almost never scrolls 100% smoothly on windows.

I did some extensive research into scroll performance of Firefox and other browsers on various platforms, and posted the results at bug 894128. You’ll also find there a a bookmarklet to test scroll performance on any browser.

One interesting observation is that it’s very hard to programatically detect if the scroll is absolutely smooth. Frame intervals apparently don’t tell the whole story (though they are not meaningless), and neither their standard deviation. To judge if the scroll is 100% smooth, it has to be assessed visually.

• Some interesting observations from this research:
  • On OS X, even with 50% perf per core (and half the cores as well) compared to a fast Windows system, Firefox scrolls much smoother than on Windows, even with OMTC off.
  • Firefox may occasionally degrade scroll for several frames for no apparent reason (the content is homogenous). GC/CC?
  • OMTC typically doubles the throughput on OS X. On windows OMTC is not yet mature enough to test this.
  • Very low intervals variance is not required for 100% smooth scroll (though IE and Chrome mostly manage very low variance compared to Firefox).
  • Looks like Chrome and IE use something like APZC during scroll. IE is especially exceptional and is able to practically never drop a frame even on extremely low-end HW (Atom z2760), and also almost never show blanks during scroll.

Jet has already shown interest in looking into it. Hopefully some day Firefox will scroll as smooth as it can get…

Vsync

Vsync has finally landed on Windows Nightlies not long ago. This means that Firefox will now synchronize its paints with the actual refresh rate of the monitor (if available), which is essential for properly smooth animations. This will work (and is enabled by default) on Windows Vista and later with DWM enabled (“Aero” themes). This also works with WebGL content such as Epic Citadel (demo).

Without vsync, Firefox uses 60hz refresh rate by default, which works relatively decently with 60hz displays, but fails badly at displaying properly smooth animations on monitors with other refresh rates (50hz on quite a few laptop displays, 100hz monitors, etc).

However, the current implementation synchronizes with the main display only, so if using a multi-monitor setup, Firefox windows on secondary monitors might not gain from this.

You can control the refresh rate manually by modifying layout.frame_rate at about:config (and restart Firefox). The default value is -1 which means “Use vsync if available or 60hz otherwise”. Any other integer value > 0 will force that specific refresh rate.

Naturally, if Firefox can’t animate specific content quickly enough to keep up with the designated rate, then the actual rate will be lower.

about:newtab

about:newtab is the default page when opening a new empty tab - with the thumbnails of recently visited pages. It was affecting tab animations badly, to the point of displaying as little as a 3 frames slideshow on low-end systems while animating the tabstrip after pressing the + button to add a tab.

Tim Taubert from the Firefox team has done some extensive work recently on several bugs which fall into two categories:

• Enable newtab preload by default. Newtab preload means that Firefox prepares the newtab page in the background, such that when opening a new tab it’s already rendered and therefore could be quickly displayed with very little processing. It has been working for a while but disabled by default due to accessibility issues. Tim removed this blocker, as well as simplified the preload mechanism and made sure that background preload doesn’t happen during animation. This landed about a week ago and newtab preload is now enabled by default on nightlies.

• Prevent unnecessary reflows of the newtab page. Reflow is the process of re-calculating the layout of the page, e.g. after window resize, and it’s relatively CPU intensive. Tim discovered that newtab causes unnecessary reflows on several cases, some of them related to preload. He fixed them and also added an automatic test to detect if new reflows are accidentally introduced over time.

The result of all this work is that animating a new tab with the newtab page is now almost as smooth as opening a blank tab, with few further potential improvements to be worked on soon. Well done!

Talos

Bug 590422 (remove timers averaging filter) landed more than two months ago and should greatly improve timers accuracy, but it caused few regressions which are still being worked on. Talos tscroll was already modified as a result - which landed about 2 weeks ago, and recently tsvg regressions from this bug were examined as well.

As it turned out, tsvg was rendering SVG animations at 100-200ms intervals, and measuring/reporting the overall runtime as representative of SVG performance. At 4 of the 7 animation tests within the tsvg suite, more than 95% of the time was spent inside setTimeout with the browser completely idle. This resulted in tests which in practice represent the accuracy of timers more than anything else.

To make sure the overall runtime represents actual SVG performance, we should iterate the animation frames as fast as possible. We could do this with setTimeout(loop, 0), but since Firefox renders to screen at 60hz, and it’s usually smart enough to delay reflow calculations until it’s absolutely necessary, we could end up fully processing only one in few frames when the SVG iterations are quicker than 60Hz, and at tsvg they’re much quicker than that.

To make sure each frame is fully processed on each iteration, we should therefore use requestAnimationFrame (fondly nicked rAF) - which makes sure that once we’re back at the callback, whatever layout changes which we introduced were already fully processed and flushed by Firefox. However, rAF iterates at 60hz by default (or vsync rate), which is usually not fast enough to stress our SVG tests. The solution is to set layout.frame_rate to a high enough value (during these tests) which causes Firefox to iterate and fully-process each frame as fast as possible. Joel Maher helps a lot with testing and deploying the new tsvg tests, and we hope to land it soon.

Making tsvg iterate ASAP also brought us back to tscroll. The recent change of tscroll was to use rAF instead of setTimeout for scroll iterations. However, while it represents scroll smoothness under normal conditions, it doesn’t detect small regressions - since Firefox isn’t actually being stressed and the scroll doesn’t limit our iteration frequency. But stressing the browser with ASAP iteration will lose the real-world simulation of normal refresh rates. So we don’t have a single approach to satisfy both of these needs.

We’re still considering this, but it seems talos will eventually run tscroll twice: once @60hz to make sure we can keep up at real-world scenarios, and once with ASAP iterations to detect performance regressions at much higher resolutions.

Paint starvation

While working on tsvg with ASAP iterations, we noticed that Firefox sometimes appears frozen for the entire animation (typically less than 1 second) - it only shows the last animation frame. While this happened when rAF was set to iterate ASAP, we discovered that there are scenarios under perfectly normal conditions which would exhibit exactly the same symptom.

This is now bug 880036. It shows is that if, for some reason, Firefox can’t iterate frames as fast as it wants to (60hz by default), then it may spend too much time on these iteration and prevent other event types from being handled on time. The most noticeable one is the paint event, since it makes Firefox appear frozen, but possibly other events are starved as well.

A patch which seems to fix the issue was already posted, but since this freeze appears to ‘fix’ itself after about 2s, we want to first understand this unfreeze mechanism, and possibly fix it instead of posting a patch to cover up for another bug.

The official python server is there. The instructions on this page are for setting up a small 3rd party weave (Firefox sync) PHP server which is compatible with current Firefox 21 and nightlies as far as I can tell, and which I started using on my own systems as a replacement for the Mozilla sync server.

I’m not associated with this server in any way, and I’m not a security expert. Use at your own risk.

The short version - with sqlite:

1. Have a web server with HTTPS and php5+sqlite (if self-signed cert, make sure to permanently accept it before Firefox sync setup).
2. Create a directory for weave (e.g. /var/www/weave), and put the files from this repository. Make sure the web server has write access to it (e.g. sudo chown www-data /var/www/weave).
3. Browse to https://your.domain.com/weave/index.php (sqlite selected by default), click OK.
4. The server is now ready and operational. Sync URL is https://your.domain.com/weave/index.php/ (note the trailing ‘/’).
5. Once an account was created after setting up sync from Firefox, you can disable further new accounts registrations at settings.php (new pairings with existing accounts will still work).
6. If using sqlite, make sure that weave_db is not accessible from outside (e.g. using .htaccess).
7. If required, to reset the server and delete the accounts: delete weave_db and settings.php at the weave directory (and go to step 3).
8. Migration to a new server: Unlink all devices (Tools>Options>Sync>Unlink), setup sync on one device with the new server, pair the other devices normally. If you don’t wish to share all items (Addons/Passwords/etc), make sure to click “Sync Options” at the setup/pair dialogs since it resets to default more than expected.

The long version

And other gotchas as encountered with Ubuntu 12.04 LTS. All notes on weave security were gathered from irc.mozilla.org#sync, and I hope I got them right. Thanks to telliott, rnewman and witheld for their help and patience.

1. Have a web server with HTTPS and php5+sqlite support.
   • HTTPS is not mandatory for Firefox sync, but highly recommended. Sync sends the account password in plain text to the server. This password can NOT be used to decrypt the synced data at the DB, but it does allow an attacker to login to this sync account and make modifications such as delete elements at the DB for this account. HTTPS prevents password snooping and further attacks on the server/DB.
   • Using self-signed certificate (snakeoil) will work for sync. However:
     • Any Firefox client which uses this server for sync should first permanently approve the certificate (e.g. by browsing to https://your.domain.com and adding the exception), preferably also verifying that the signature matches the server’s.
     • Instructions for setup of Apache2 + HTTPS with self-signed certificate.
     • Self-signed certificate for sync apparently doesn’t always work on Firefox for Android.
   • If required, add sqlite support to php. E.g. sudo apt-get install php5-sqlite.
2. Create a directory for weave (e.g. /var/www/weave), and put the files from this repo
   • By default, if using sqlite, the DB will be created at the same directory with file name weave_db, and therefore the web-server will need write access to that dir. Also, the automatic setup will create a settings.php file at the same directory, which also requires write access (e.g. sudo chown www-data /var/www/weave).
   • If using manual setup (manually creating settings.php): If setting the sqlite DB file path elsewhere, or if using mysql, there’s no need for that dir to have write access (but still need write access to the sqlite DB file if using sqlite).
3. Browse to https://your.domain.com/weave/index.php , select sqlite and press OK (the other option is mysql)
   • Once OK is clicked, a new file settings.php will be created which allows some configuration (disable new accounts registration, sqlite db file path, mysql credentials).
   • If sqlite was selected, the DB file will be created at the same directory as weave_db.
4. You can now setup Sync in firefox: choose a custom server and enter the URL (note the trailing ‘/’): https://your.domain.com/weave/index.php/
   • To pair new devices, follow the normal procedure (no need to manually enter the custom URL on new devices - just use the 12 chars code and it’ll work).
5. Note that the server is now public and anyone can create a new account on it by setting up Firefox sync with this server.
   • Edit settings.php if you wish to disable further new accounts registration (this doesn’t prevent pairing new devices on existing accounts).
6. If using sqlite with default DB file path, then the DB file will be accessible via the web (https://your.domain.com/weave/weave_db). While the data at the DB is encrypted (and the account login password cannot be used to decrypt the data), it’s still highly recommended to make it inaccessible from the outside.
   • I used .htaccess to prevent access to everything but index.php at that directory, which works:

1
2
3
4
5
6
7

Order Deny,Allow
Deny from all
Allow from 127.0.0.1
<Files index.php>
    Order Allow,Deny
    Allow from all
</Files>

• If your Apache server doesn’t seem to read the .htaccess file, you might need to enable it, e.g. by adding to /etc/apache2/httpd.conf (empty by default):

1
2
3

<Directory /var/www/weave>
AllowOverride All
</Directory>

And restarting apache. E.g. sudo service apache2 restart

• Alternatively (haven’t tried myself): move weave_db to a dir outside of wwwroot and edit settings.php accordingly.
• I haven’t tried using mysql with this server.

1. If required, to reset the server and delete the accounts
   • Delete weave_db and settings.php at the weave directory (and go to step 3).
2. Migration to a new server
   • Unlink all devices (Tools>Options>Sync>Unlink).
   • Setup sync on one device with the new server.
   • Pair the other devices normally.
   • If you don’t wish to share all items (Addons/Passwords/etc), make sure to click “Sync Options” at the setup/pair dialogs since it resets to default more than expected.

Good luck!

Vsync from a thread

Following the first experimental vsync implementation on Windows (using GetVBlankInfo which is non-blocking and synchronous), I’ve been experimenting with another implementation which vlad suggested: Run a thread which uses WaitForVBlank (blocking until next vblank) and post the timing event to the main thread.

The conclusions are here. In a nutshell: WaitForVBlank works nicely (when it works), but also blocks the main thread, and while some circumvention is possible (using sleep), it’s still far from ideal. Thanks goes (yet again) to Bas for helpful tips on Windows APIs and integration into the Layer manager.

Since OMTC might integrate WaitForVBlank into its pipe more naturally, it was decided to land the original approach - which is very lite and reasonably working, especially when using the main/single monitor - and reconsider the other approach with OMTC.

Expect vsync on Windows nightlies soon.

New tab previews hurts tab animation

That’s bug 843853. I decided to put it on hold for now, as it appears that not enough resources can be directed towards it. It would have been nice, however, to see a more practical list of priorities - which more carefully takes resources into account. Hopefully, we’ll have such list soon.

Timer filter removal regressions

Bug 590422 (remove averaging filter for timers - which probably hurts timing more than it helps) landed some time ago, but still rears its head. The most recent is lost frames while decoding video.

Apparently what happens is that the filter removal also hurts average timer intervals when high resolution timers are not enabled, and video decoding uses timers without making sure they’re in “high-res mode”. The fix was quick everyone’s happy.

This also brought a new discussion about automatically enabling high-res timers according to some heuristics on the requested timeouts (and possibly also the request source). While it sounds useful, the main argument against it is that high-res timers have slightly higher power draw, and that it could be abused by content pages which use setTimeout(0) while what they actually want is “soon enough”, and therefore get much more frequent callbacks than required.

This discussion is still ongoing, you’re welcome to contribute more factors to consider.

Tabstrip animation project goals

There has been some discussion on refining the project goals, to make them both useful from a user perspective (rather than mostly synthetic bar) and also reasonably achievable. We feel comfortable with this definition, which, in a nutshell, is:

• Have Smooth enough (50FPS or more) animation during the last part of the animation.
• Make sure this happens on some concrete cases we care about (such that newtab open, close a tab and get into a gmail tab, etc).

What’s next?

• Land Windows vsync.

• Get a clear priorities list for newtab slowdown.

• Implement a regression test which could measure our discrete tab animation goals (tab animation telemetry is useful, but has lower resolution than we’d like), and give each a score. Still not sure if as a talos test, but right now talos appears to be the proper place for it.

The previous post introduced the Tabstrip animation project, and some work which has been done so far. This post reviews some more recent related progress.

Vsync

During the Paris snappy work week, Bas and I discussed animation smoothness, and specifically the default 60hz refresh rate which Firefox currently uses. We’ve conducted a very unscientific poll among some attendees at the meeting, and found out that 4 out of 9 laptops had 50Hz monitor refresh rate. The rest had 60. For those with 60Hz, the current refresh system is acceptable, even if not perfect (actually … Often, developers choose 60 Hz as the refresh rate, not knowing that the enumerated refresh rate from the monitor is approximately 60,000 / 1,001 Hz …), but those with 50Hz monitors will always get a very noticeable jitter during any animation.

So Bas wrote a patch to expose some vblank timing info on windows via widget, and I updated the refresh driver to use this info. The result is decent vsync synchronization on monitors with all refresh rates (you can test the slightly outdated Windows try build, which hasn’t landed).

However, by using the vblank timing info and targeting a timer just past it, we’re forced to “lose” some 2-3ms in order to always hit the correct vsync phase - by targeting a bit later than the actual future vblank signal, to make sure we account for the inherent inaccuracies of our timers system (integer ms interval, callback delays/too-early, etc).

Vladimir didn’t like this approach, and so now we’re trying a different one with a thread which blocks on actual vblank signal, and then posts an event when we’ve hit it (no patch yet).

However, this approach is also not perfect. As bas noted, the earlier approach allows easier association with different windows/monitors, possibly easier to integrate with fallback to the current code path, and also probably allows easier disabling of vsync on cases where we can’t animate fast enough, or even if gamers want to reduce input/output lag to the minimum possible.

The verdict is not out on this yet, but you can follow the windows vsync implementation and the global vsync tracking bug.

New tab page

Tim taubert has been working on improving new tab page rendering, which often competes with tab open animation on resources, and may degrade it, especially on slow systems.

While at it, we got distracted by a patch which regressed animation smoothness after changing its trigger from asynchronous (setTimeout 0) to synchronous. The main arguments were that setTimeout delays the animation start more than we should, and therefore synchronous is the way to go. On the other hand, setTimeout causes a relatively small delay, and could result in considerably smoother animation, since its callback is typically (but not always) handled after the entire current event queue is handled.

Eventually, we settled on using requestAnimationFrame which is asynchronous and will not delay the first animation frame, but may still be handled earlier than setTimeout, especially on slow systems. The numbers approve this theory (15 animation frames using setTimeout, 7 frames after changing to synchronous, and 12 frames after switching to rAF).

Looking back at this, however, this was a distraction and somewhat premature discussion. The reason being that these measurements were taken with about:blank page, and not our typical newtab page (which is still too slow to actually measure improved animation when using it, on slow systems).

Note to self: Identify distractions and premature discussions before they consume too much time.

Back to new tab rendering. Trying different semi-random approaches didn’t get us too far:

E.g. replacing XUL flexbox with CSS3 - which gave some improvement but added complexity, we’ve considered delaying the tab animation until after newtab rendering settles, or delaying newtab rendering until after the animation is done (or even disabling tab animation on slow systems, especially now that we have tab animation smoothness measuring infrastructure).

We should probably get back to basics and profile the interaction between tab animation and newtab page rendering (which is not so easy, since a lot of stuff is happening when a tab is animating concurrently with new tab page rendering, but we should still do that), get some real real numbers such that we could point some real fingers ;)

Yet another note: While the newtab rendering is apparently high priority for both the performance team and the fx-team, in a hindsight, it’s got too little progress. Maybe we have too many high priority items for the resources we can afford, and we need to get back to re-prioritizing the high priority items?

Lightweight Themes (LWT, previously: Personas)

MattN and mconley are implementing Lightweight themes support for Australis.

As part of their ongoing effort to keep performance in check, they’ve performed animation measurements, comparing performance on various systems and getting some real numbers. They’re summurizing their results on this spreadsheet.

Their current results with LTW is that tab animation is on par with Australis (at the above spreadsheet, columns AL-AP, rows 46-58), which itself is almost on par with current theme, which is a great achievement, considering Australis is much more complex.

Keep up the great work! :)

Here’s a summary of recent events.

Tabstrip animation project

As part of the recent Performance team changes (1, 2), I’m now the tech lead for the tabstrip animation project. It’ll require some additional coordinations and regular blog posts, but otherwise, work continues as usual.

The goal of the tabstrip animation project:

Make tabstrip animations as smooth and as snappy as possible, both visually and perceptually, by identifying and removing bottlenecks, deficiencies, and perception issues. The ultimate goal here is 60 FPS on a recent Atom CPU on all platforms, with minimal delays.

So far, this work touched several different subjects:

• Infrastructure and telemetry for tabstrip animation smoothness measurements. This infrastructure already proved very useful with the various subjects below. In general, good measurements allow much more concrete goals, feedback and regression detections. We should strive to use measurements wherever possible, while making sure the measurements are a good representation for the kind of performance which interests us.

• Timing related improvements. Timing is the essence of animation. It’s important for tabstrip animations and for Firefox in general to be able to present each frame at exactly the right time, or else the animation won’t be as smooth as it could be. This is an ongoing effort within Mozilla.

Following Vladimir’s very useful timers change which recently arrived to the release channel, there were few more issues remaining, mainly timers averaging delay removal which was a long standing bug which also affected tab animation and finally resolved recently, vsync synchronization (present frames in sync with the monitor refresh rate) which greatly affects perception of smoothness and is a sore issue since Firefox is now lagging behind Chrome and IE on this (preliminary Windows vsync build/patch), and finally, OMTC which aims at offloading some rendering/animation related tasks off the main thread, and which needs to integrate well with other timing related work.

• Identifying and removal of relevant graphic bottlenecks. Tabstrip animation (and animation in general) is limited by rendering speed. Any improvements here translate to possibly faster and smoother animation, especially on slow computers. We’ve already improved D2D accelerated tab animation by about 25% by improving the gradients cache system, and we still have similar potential improvements with SVG raster caching, and possibly other graphic bottlenecks.

• Theme optimizations. During the development of Australis theme, we’ve noticed that its tabs animate slower than the current theme. This was followed up by great ongoing effort and attention from Mike Conley (good post) and Matthew Noorenberghe to measure and improve tab animation speed with Australis. One of the results of this effort was replacing SVG usage with images, but we’d still like to get back to SVG, as it’s more elegant and it scales better for different sizes/DPI.

• Browser issues which affect tab animation. Even if the timing is great, there are no major GFX bottlenecks, and the theme is light to render, tab animation is still affected by implementations within the browser. E.g. the way it’s triggered (sync or async, delayed, etc), pages which render concurrently with animation and compete with it on resources (e.g. the new tab page) and possibly other similar issues. Tim Taubert from the Firefox team has been working on these issues.

• Regression detection. We’ve already identified an unoptimal talos scroll test during the timer filter removal process, and posted an improvement. We should also add a similar tab animation regression test. In general, it’s important that talos tests are useful, accurate, and are good representation of actual regressions which we care about. Nathan is the performance team representative for talos issues, and I’ll help as much as I can.

I’ll be posting regular progress reports on this subject.

Talos tests are pretty simple - they perform some task within a browser, then report a completion duration (or a comma-separated list of those) via the tpRecordTime javascript talos call. Talos then logs and processes those values, tries to ignore the noisy bits, and comes up with an average and StdDev values (it also deploys a similar process over several runs of the same test) - which can then be compared to other runs of the same test on different versions/platforms of Firefox.

Run and develop talos tests - without talos

While updating tscroll talos test, I found that running the test in a browser outside of the talos framework could be useful. The page could be reloaded to re-run the test, allowing quicker iterations of the code. While this isn’t as sterile as the talos run environment, it’s still useful during development. This required substituting tpRecordTime with my own, and also displaying various statistics on the collected data.

I’m not the not the only one who deployed such tactics, and one can find commented out alerts and statistics calculation (which are not reported to talos) on various talos tests.

So we had the idea of separating this debug stuff from tscroll, such that one can include this talos-debug.js file from any test, and then simply load the page in a browser and get a report of the recorded values together with some statistics.

This file adds a window.talosDebug object which provides some statistical functions (sum, average, median and stddev) and a tpRecordTime method which also displays the stats. It can be controlled by few properties (such as displayData - if set to true, will also display the raw collected data). It also provides automatic detection of the “warmup period” for the data set (see next).

If a global tpRecordTime doesn’t exist, then it maps it to window.talosDebug.tpRecordTime. It’s safe to include this file in any talos test, which will allow running the test in a browser, while it shouldn’t affect the results when running within the talos framework.

Bug 849558 is about adding talos-debug.js to the talos repository. If you touch the talos tests, consider using it instead of rolling out your own. Suggestions are welcome.

Automated tests and noise removal

The talos tscroll test records average frame interval of different real-world-like content types scroll. It showed some regression when I tried to remove a timer compensation mechanism which was actually hurting timimg on most cases these days. When I looked at the tscroll source code, I thought that this isn’t a real regression, and instead, we might be able to improve tscroll.

While modifying tscroll to be less susceptible to timing changes (and also report all the raw intervals instead of the average only), I noticed that typically the first few frame intervals during scroll are more noisy than the rest.

While this isn’t surprising - the browser, layers, etc need to build-up and settle-down, hopefully quickly - it did bring up the question of how to handle this.

Since this is obviously a transitional effect, we probably want to exclude this warm-up period when measuring scroll smoothness. I manually estimated the number of first intervals we should ignore by observing the values. It appeared that ignoring the first 5 values would do the trick for most of the tsctoll sub-tests, and that’s what we agreed to do.

For fun, I also wrote some auto-warmup-detection code, and incorporated it into talos-debug.js. It actually works surprisingly consistently and accurately for the noise patterns (is that an oxymoron??) of scroll intervals, coming up, for instance, with results such as this (notice how stddev goes from 10.8% over all the values, down to 2.8% after ignoring the first 2 out of 42 intervals):

Count: 44
Average: 16.38
Median: 16.88
StdDev: 1.77 (10.83% of average)

Warmup auto-detected: 2
After ignoring first 2 items:
Count: 42
Average: 16.70
Median: 16.89
StdDev: 0.47 (2.79% of average)

Recorded:
5.68, 13.59, [warmed-up:], 17.26, 16.93, 15.95, 17.05, 16.98, 16.08, 16.94, 16.91, 16.39, 16.72, 17.03, 16.03, 16.96, 17.72, 16.03, 17.04, 17.07, 15.92, 16.99, 17.09, 15.94, 16.99, 16.99, 16.01, 16.99, 17.08, 15.96, 16.84, 17.21, 16.11, 16.86, 16.92, 16.21, 16.86, 16.88, 16.19, 16.88, 16.97, 16.21, 16.74, 17.11, 16.21

Stddev from item NN to last:
1.77, 0.66, [warmed-up:], 0.47, 0.46, 0.47, 0.46, 0.46, 0.46, 0.46, 0.46, 0.47, 0.47, 0.48, 0.49, 0.48, 0.49, 0.45, 0.44, 0.44, 0.45, 0.43, 0.43, 0.43, 0.41, 0.42, 0.42, 0.40, 0.41, 0.41, 0.38, 0.39, 0.37, 0.35, 0.37, 0.37, 0.37, 0.38, 0.40, 0.39, 0.42, 0.44, 0.45, 0.64, 0.00

Looking at this issue with a wider perspective, the subject of warmup period (and noise in general) is common to automated testing. It definitely affects stability of the results, and mostly hurts StdDev - the results appear noisier than they actually are after everything settles down.

Talos uses a similar strategy (warning - year old post). It runs each test few times, then removes the noisiest result, which is typically the first one.

However, as was noted at some of the comments on that post, the warmup period does carry its own weight and probably shouldn’t be ignored completely.

Maybe if we deploy an automatic warmup detection system similar to the one described above (after it’s tuned and tested with a much wider types of results than just the frame intervals during scroll), we could have a double win - reduce the noise level of the results, AND record the warmup period required for each test. We could then watch this value over time, and if - for a specific test - the warmup period goes up, then it’s probably a regression, and vice verse.

Turns out that it’s quite hard to provide a consistent scroll experience on the various platforms which Firefox supports, due to many laptop and touchpad manufacturers, different drivers, OS configurations etc. We could have provided some (or a lot of) configuration UI, but the best solution would be to get it as good as possible out of the box.

And to do that, we need data - and you might be able to help.

On my system (Asus laptop, Elantech touchpad, Windows 7), the touchpad does produce pixel-accurate scroll in Firefox, but doesn’t send its events as true pixel data (WM_GESTURES windows event). Instead, it sends mouse wheel events, but very rapidly and with fractional line values (it’s a legitimate approach with WM_MOUSEWHEEL). This results in Firefox processing it through the “mouse wheel path”, even when the frequency and value of the events allow pixel-accurate positioning of the content.

It was interesting to compare how this kind of event is handled on other browsers, so I wrote a cross-browser testcase which measures just that. The best way to use it is to scroll (e.g. with two fingers) consistently from top to bottom of the touch pad. After each such scroll, the page displays related events values, and the overall scrolled distance (pageYOffset). Reset and repeat this few times, until you can get consistent results. Then, try it also with other browsers.

On my system, if Firefox scroll x pixels for one top-to-bottom two-fingers swipe, then, for the same swipe:
- Chrome scrolls 1.7x
- IE scrolls 6x
- Opera scrolls 13x

Quite a difference! With the numbers (consistently) all over the place. For me, Opera scrolls way too fast (does it treat every fractional mouse wheel event as a full line request?), Firefox scrolls slower than I’d like. Chrome could also be faster, and IE feels a bit too fast for my taste. Obviously, there’s no real right or wrong here, as it’s a very subjective matter.

How do the browsers compare on your system? (Linux? OS X? Windows?) Is Firefox considerably slower than the rest? Does it feel too slow to you? If you have a bugzilla account, you can post directly at the bug, otherwise, you’re welcome to leave a comment here.

Welcome aboard.

Since I joined the performance team at Mozilla not long ago, I’ve been working on animation smoothness, starting with tab animation, but touching related subjects as well.

In this space, I hope to provide interesting progress notes on this and related subjects.

Stay tuned.