Converting an old overlay-based Firefox extension into a restartless addon

Converting an old overlay-based Mozilla Firefox extension of non-trivial complexity into a restartless addon used to be impractical. When restartless addon support was added in Firefox, quite frankly, it was a partially implemented mess. Bugs have been filed and fixed over the years, sometimes taking forever to see any progress, some not at all. I took a stab at converting Flagfox into a restartless addon around Firefox 10.0 and found it to still be impractical at the time. My second attempt was last month, and I’ve got a Flagfox beta up and running that is restartless (and extractionless). There aren’t really that many good sources of information on how to make this transition online, and it’s easy to think it’s still not practical. What follows is an attempt to remedy this a little bit.

Update:
An updated version of this guide is available on MDN, and will be updated as needed.

Contents:

Step 0: Requirements
Step 1: No more resource:// URIs
Step 2: No more input streams
Step 3: Manually handle default preferences
Step 4: No more internal JAR files
Step 5: No more XUL overlays
Step 6: bootstrap.js
Step 7: Bypass cache when loading properties files
Put it all together
References

This is a rough step-by-step guide that goes through all of the things I had to do to get Flagfox 5.0b1 up and running as a restartless addon… though, rewritten into a more logical to follow order. Flagfox is a single programmer project, so I had to learn this pretty much on my own, thus it’s always possible I’ve just missed something obvious. If I did, please point it out to me. I’d also like to update MDN with better info at some point as well.

Step 0: Requirements

First off, what kind of addon are we talking about here? Well, XUL overlays and windows, JSM files, chrome & resource mappings with localization, default preferences, but no XPCOM components of your own. Some of that will have to be replaced and the rest will need to be loaded differently.

Next, what’s the minimum version of Firefox this is viable in? (preferably an ESR) My answer to that question is Firefox 17 ESR or later (or anything else Gecko 17+, such as SeaMonkey 2.14+). It might be doable with something less but I’ve lost track of what got fixed when and some of what I’ll recommend using will require Gecko 15, 16, or possibly 17. This is two ESRs back, which should be plenty. Using the current Firefox 24 ESR, Firefox 26 stable version, or Firefox 29 Nightly is generally a better idea if given the option, but some users take forever to upgrade.

There will be no usage of the SDK/Jetpack or any other external libraries here. Everything will use APIs available in Firefox 17+ or code provided here.

Step 1: No more resource:// URIs

If you load one of Mozilla’s internal JSM files, say for example, Services.jsm, you’ll do so via privileged JavaScript code like this:

Components.utils.import("resource://gre/modules/Services.jsm");

From here on out, I’m just going to assume you’ve imported Services.jsm somewhere at the top of whatever file you’re in and will be using it in all code examples. The examples will also assume that you know how to properly add instructions to your addon’s chrome.manifest to add resource, chrome, locale, & etc. mappings, so that you can access your files with custom paths such as:

resource://myAddon/filename.ext
chrome://myAddon/content/filename.ext

You too can stick a resource mapping into your chrome.manifest for your addon and load your own JSM from resource:// URIs. It’s a great way to modularize your code that’s been available since Firefox 3. Unfortunately, this is still not usable in restartless addons, which looks bad, but only because Mozilla is still using resource:// URIs internally and in examples for no good reason. This will be the first of a series of APIs I will mention that started out implemented without a basic obviously-needed feature that would get implemented later without everyone noticing. You can use chrome:// URIs with “Components.utils.import()” just fine; in fact you’ve been able to since Firefox 4. However, because it was implemented first for only file:// and resource:// but not chrome://, everyone who learned of this new feature learned that you had to load JSM from resource:// URIs and just stuck with that forever. It does still work if you don’t have restartlessness to worry about, though the protocol (or scheme, or whatever term you prefer) really should be avoided at this point. The resource:// protocol actually bleeds into content which allows webpages to detect installed addons using the protocol, which is not particularly fantastic. (just the static file contents, not any loaded script/data)

Step 1a: Load your JSM from chrome://

To anyone reading this that is already confused, no, this has nothing to do with the Google Chrome web browser. Google did something annoying and lazy when they wrote their browser: they didn’t bother to name it. The word “chrome” is an old technical term for the parts of a web browser other than the web page it is showing. The idea is that you have the web content on screen and then the bits around it: the “chrome”. (I didn’t make up the stupid jargon) Google naming their browser Chrome (upper-case ‘C’) was roughly equivalent to as if Ford were to name their next vehicle the Ford Car. All of the usage of the word “chrome” and the “chrome://” protocol here predates Google Chrome and has nothing to do with it.

Now with that preface out of the way, this part is easy: drop support for Firefox 3.x if you haven’t already, move your JSM files to wherever you’ve got your chrome mapping to for your XUL overlay and/or windows, import your files from that new chrome mapped path instead of the old resource one, and remove your “resource” line from your chrome.manifest file. It’s probably a good idea to do this even if you aren’t going fully restartless / extractionless due to the previously mentioned exposure to content of resource mappings.

Also, drop support for Firefox 4 through 9 while you’re at it. Prior to Firefox 10, the chrome.manifest file you rely on wasn’t loaded automatically for restartless addons. Hacks were required, and probably a bad idea.

Step 1b: Audit any remaining resource:// URI usage

If you’re not me, then maybe you’re lucky and didn’t need resource:// URIs for anything else. You’ll be able to skip to “step 3”. If not, see if you still can’t do things any other way. As with JSMs, a chrome:// URI may be more appropriate. If you want to also make your addon extractionless then you may need “step 2” if you’re loading files with nsIFileInputStream or something similar, or a jar: URI might work. If not, a file:// URI might be fine for you. Restartless addons can easily get a URI for their install location on startup, so you should look into what you can do with that.

Step 2: No more input streams

As mentioned above, you might not need this bit. I did, and the docs on the topic suck enough for me to think others might benefit from what I learned here, so here’s the gist.

How to get and load the data of of your addon’s files using the Addon Manager API:

// This is the OLD way of getting one of your files
const myAddonID = ...;  // Just store a constant with your ID
Components.utils.import("resource://gre/modules/AddonManager.jsm");
AddonManager.getAddonByID(myAddonID,function(addon) {
    var file = Services.io.newURI("resource://myAddon/filename.ext",null,null)
                          .QueryInterface(Components.interfaces.nsIFileURL)
                          .file;
    var stream = Components.classes["@mozilla.org/network/file-input-stream;1"]
                           .createInstance(Components.interfaces.nsIFileInputStream)
                           .QueryInterface(Components.interfaces.nsISeekableStream);
    stream.init(file, 0x01, 0444, 0);  // read-only, read by owner/group/others, normal behavior
    /* do stuff */
});

That bit of code is paraphrased and kind of crap, but it should work. (though, please run in terror from the usage of an octal integer literal; this is the standard way to do it, which I’m sure is in docs and used everywhere, but octal literals are “evil” and being phased out, which actually makes them more “evil” because they’re going to break; use ES5 strict mode to break it on your own terms so you can fix it) In my case, I also created a nsIBinaryInputStream instance and read files in bytes (e.g. 32-bit integers, or fun stuff like 48-bit integers). Not ideal, but it worked and performed more than sufficiently well. All of that code above is no longer viable unless you can hack together a way to get a file:// URI to the same file, and that’s only if you don’t also go extractionless (which you should).

Ok, great; how do you replace that? The answer to that question is to load your file from a chrome:// URI using XMLHttpRequest. You may now have another question: wait, what does this have to do with XML or HTTP? The answer to that question is, of course, nothing. XMLHttpRequest is an API created by Microsoft, adopted by Mozilla and other vendors, and hacked into a Swiss Army knife of file loading. You can use it in a web page to fetch a file from your server and you can use it in your addon to fetch a local file from your addon install. The name is a vestigial structure that just makes things confusing. I had to ask someone to make sure this was the “Correct” and best way to do things; apparently it is. It’s available in the global for a window, but in JSM you’ll need to fetch it from an interface:

const XMLHttpRequest = Components.Constructor("@mozilla.org/xmlextras/xmlhttprequest;1", "nsIXMLHttpRequest");

Here’s how to load a file using it:

function loadFile(url,type,returnresult)
{
    var request = new XMLHttpRequest();
    request.open("GET", url, true);  // async=true
    request.responseType = type;
    request.onerror = function(event) {
        logErrorMessage("Error attempting to load: " + url);
        returnresult(null);
    };
    request.onload = function(event) {
        if (request.response)
            returnresult(request.response);
        else
            request.onerror(event);
    };
    request.send();
}
loadFile("chrome://myAddon/content/filename.ext",dataType,function(data) {
    /* do stuff with data */
});

If your file is text, use “text” as your data type. If you’re getting JSON this way make sure to explicitly set the type as “text” if you intend to parse it yourself. Even though it says that the default type is “text”, Firefox will attempt to autodetect and fail, resulting in an error message in the console. This doesn’t seem to break anything, but it is easily avoidable by being explicit with the type. MDN says you can set the type to “json” instead, if you prefer to have it parse things for you.

If your file is not text or JSON, then you’re going to want to read binary data. The new way to do this is to use JavaScript typed arrays. Specify “arraybuffer” as your data type to get one from your XMLHttpRequest. To access that data you’re going to need a data view to look at your typed array with. Data that’s homogeneous might get away with using something like Uint32Array or one of the other standard typed array views, but it’s probably a bad idea. The basic typed array views are not endian-safe. This is incredibly stupid. You’d think such an important new JavaScript feature made available for web content and chrome alike would at least have a way to set and keep track of endianness, but no, it doesn’t. Don’t use any of the basic typed arrays for any data you did not earlier write into them in the same program session. Also, they kind of suck if your data isn’t all of the exact same type (which it probably isn’t).

The solution to read arbitrary binary data, of various sizes, in an endian-safe way, is to use DataView. The other typed array stuff is viable in Firefox 4+. This wasn’t added until Firefox 15. That’s not just Mozilla’s fault; the spec was just missing a sane binary view for a while. Again, this is an API implemented with an obvious hole in the spec. If you were using nsIBinaryInputStream or anything similar, figuring out DataView will be fairly straightforward. Just read the docs and it’s pretty simple. It will probably be notably faster than whatever you were doing before.

Oh, and supposedly XMLHttpRequest can glitch when used in JSM under versions of Firefox less than 16. This was a footnote on MDC somewhere, so don’t quote me on that. However, I’ve already said that all of this should be taken as requiring Firefox 17+ already.

Step 3: Manually handle default preferences

Normal addons load default preferences from a standardized file automatically. Restartless addons don’t (for no good reason; probably the worst offender in the “they just didn’t bother” list). This part is fairly easy to implement yourself, at least. I’ll just paste you some of my handling functions to borrow:

function getGenericPref(branch,prefName)
{
    switch (branch.getPrefType(prefName))
    {
        default:
        case 0:   return undefined;                      // PREF_INVALID
        case 32:  return getUCharPref(prefName,branch);  // PREF_STRING
        case 64:  return branch.getIntPref(prefName);    // PREF_INT
        case 128: return branch.getBoolPref(prefName);   // PREF_BOOL
    }
}
function setGenericPref(branch,prefName,prefValue)
{
    switch (typeof prefValue)
    {
      case "string":
          setUCharPref(prefName,prefValue,branch);
          return;
      case "number":
          branch.setIntPref(prefName,prefValue);
          return;
      case "boolean":
          branch.setBoolPref(prefName,prefValue);
          return;
    }
}
function setDefaultPref(prefName,prefValue)
{
    var defaultBranch = Services.prefs.getDefaultBranch(null);
    setGenericPref(defaultBranch,prefName,prefValue);
}
function getUCharPref(prefName,branch)  // Unicode getCharPref
{
    branch = branch ? branch : Services.prefs;
    return branch.getComplexValue(prefName, Components.interfaces.nsISupportsString).data;
}
function setUCharPref(prefName,text,branch)  // Unicode setCharPref
{
    var string = Components.classes["@mozilla.org/supports-string;1"]
                           .createInstance(Components.interfaces.nsISupportsString);
    string.data = text;
    branch = branch ? branch : Services.prefs;
    branch.setComplexValue(prefName, Components.interfaces.nsISupportsString, string);
}

Just grab the above, move your default preferences file to your chrome mapping, and then do the following line once during your addon’s startup:

Services.scriptloader.loadSubScript("chrome://myAddon/content/defaultprefs.js", {pref:setDefaultPref} );

That’s it. Once you’ve got the machinery to load and save preferences without having to jump through the various pref type hoops the actual preferences API sends you through, loading the actual preferences file is one line. I’d generally still recommend using the type specific functions for each pref individually, but to load the defaults just use the generic functions above and it’s quite simple. The other generic functions are provided above in case you need them. (it’d be nice if the preferences interface just did this basic stuff, but it doesn’t, and its plain text handling doesn’t work with Unicode properly)

Step 4: No more internal JAR files

You know how I’ve been mentioning extractionless addons every once in a while thus far? Well, you should probably consider switching to be extractionless when you go restartless. An old-style addon installer is packaged something like this:

myAddon.xpi file (glorified ZIP)
└─ chrome.manifest
└─ install.rdf
└─ chrome folder
  └─ myAddon folder
    └─ content.jar file
      └─ content folder (most files go here)
      └─ locale folder (your locale files go here)

In versions of Firefox prior to 4.0 (Gecko 2.0), the XPI would be extracted into a folder in your profile’s extensions folder. In current versions it stays unextracted as an XPI. If you were using input streams you already had to deal with this because they weren’t an option without extraction. Opting-out to extractionlessness is done via the “unpack” flag in install.rdf.

Why the internal JAR? Well, two reasons:

1. Prior to extractionless addons, all of your files got extracted. Putting them in one single JAR file made all your stuff load in one file read, which was faster. Extractionless XPIs are bascially a standardization of this idea.
2. XPI files are glorified ZIPs, and ZIP compression is horrible. Doing an uncompressed internal JAR (aka, another ZIP) acts like a poor-man’s solid archive and significantly boosts the overall compression ratio of the XPI, resulting in smaller installers and updates.

So, it’s pretty much internal JAR or extractionless XPI. Well, you can’t use an internal JAR anymore. Firefox aggressively caches addon data a bit too much. Your restartless addon won’t actually re-load some types of files if they are in a JAR and the addon is updated without a restart. The big culprits are JSM files and locale files (namely property files), though in some situations this is true for dynamically loaded image files too. You’re still going to have to manually clear the chrome cache on addon shutdown to work around this, but that doesn’t seem to be enough with an internal JAR. So, time to switch to extractionless, too. See here for the list of stuff you can’t have in addition to no resource:// URIs or file:// URIs to files inside your XPI.

If you actually can’t find a way to go fully extractionless, you could hack together some combination of internal JAR(s) and extracted files. I previously had Flagfox 4.x doing this; the JSM files were extracted while everything else stayed in the JAR under the chrome mapping. It can be done.

However, you really should go extractionless. I’ve had a few users reporting some bizarre Flagfox error messages on the support forums that could only be triggered by franken-installs of Flagfox 4.2.x. The errors indicate files from two different versions trying to load and breaking. (current JSM with old IPDB) I don’t know what has been corrupting these few people’s Firefox profiles and resulting in botched Flagfox installs. I do know that going extractionless means that a Flagfox 5+ install is one file, which is kinda hard to screw up. Extractionless addon support appears to be useful for reliability, for whatever reason.

Step 5: No more XUL overlays

Ok, now we’re getting into some more drastic changes. You won’t be able to use your chrome.manifest to load XUL overlays anymore with a restartless addon. You could look into dynamically loading and unloading your overlay, however dynamically manipulating the DOM of your XUL window is the more straightforward route (at least in my case).

Figure out what XUL elements you need to create for your addon to add your interface, where it needs to go into a XUL window, and how to do it. Docs: document.getElementByID(), document.createElement(), Element reference, Node reference (DOM elements are also nodes).

You’ll need to write two functions. One to take a XUL window object and then create and add your elements, and then another to find your elements and remove them from the window object. The former will need to be run on addon startup and the later on addon shutdown. Until you get your bootstrap.js running you should use a basic overlay onto the XUL window with an event listener for “load” to catch overlay load and then run your manual UI construction function.

Step 6: bootstrap.js

Wow, finally; some restartless addon stuff. Personally, I ended up doing a lot of this out of order and having to hack my way through to figure out what parts needed doing. You can learn from me and do this part after the previous steps are done.

A bootstrap.js file in the root of your XPI (next to your chrome.manifest and install.rdf) will be the heart of your restartless addon. Think of it as main.c, but for JavaScript based Firefox restartless addons. A basic bootstrap.js file:

Components.utils.import("resource://gre/modules/Services.jsm");
function startup(data,reason) {
    Components.utils.import("chrome://myAddon/content/myModule.jsm");
    myModule.startup();  // Do whatever initial startup stuff you need to do

    forEachOpenWindow(loadIntoWindow);
    Services.wm.addListener(WindowListener);
}
function shutdown(data,reason) {
    if (reason == APP_SHUTDOWN)
        return;

    forEachOpenWindow(unloadFromWindow);
    Services.wm.removeListener(WindowListener);

    myModule.shutdown();  // Do whatever shutdown stuff you need to do on addon disable

    Components.utils.unload("chrome://myAddon/content/myModule.jsm");  // Same URL as above

    // HACK WARNING: The Addon Manager does not properly clear all addon related caches on update;
    //               in order to fully update images and locales, their caches need clearing here
    Services.obs.notifyObservers(null, "chrome-flush-caches", null);
}
function install(data,reason) { }
function uninstall(data,reason) { }
function loadIntoWindow(window) {
/* call/move your UI construction function here */
}
function unloadFromWindow(window) {
/* call/move your UI tear down function here */
}
function forEachOpenWindow(todo)  // Apply a function to all open browser windows
{
    var windows = Services.wm.getEnumerator("navigator:browser");
    while (windows.hasMoreElements())
        todo(windows.getNext().QueryInterface(Components.interfaces.nsIDOMWindow));
}
var WindowListener =
{
    onOpenWindow: function(xulWindow)
    {
        var window = xulWindow.QueryInterface(Components.interfaces.nsIInterfaceRequestor)
                              .getInterface(Components.interfaces.nsIDOMWindow);
        function onWindowLoad()
        {
            window.removeEventListener("load",onWindowLoad);
            if (window.document.documentElement.getAttribute("windowtype") == "navigator:browser")
                loadIntoWindow(window);
        }
        window.addEventListener("load",onWindowLoad);
    },
    onCloseWindow: function(xulWindow) { },
    onWindowTitleChange: function(xulWindow, newTitle) { }
};

As mentioned above, Components.utils.unload() won’t work properly unless the JSM file it is unloading is not in a JAR.

For tearing down and cleaning up on a per-window basis, there is another route you can take. Instead of directly calling your tear down function, make your unloadFromWindow() something like this:

function unloadFromWindow(window)
{
    var event = window.document.createEvent("Event");
    event.initEvent("myAddonName-unload",false,false);
    window.dispatchEvent(event);
}

In each window you can then register on startup to listen for your custom “myAddonName-unload” event and just tear down and clean up when that event or a regular “unload” event comes in.

Step 7: Bypass cache when loading properties files

The above will get you a working addon that will install without a Firefox restart. It will even get you a working addon that will update without a Firefox restart… usually. When Mozilla implemented restartless addon support they didn’t do the sensible thing and write a damned test extension using all the features one would expect. Some parts work only if you don’t look too closely. Localization is one of them. As mentioned in the previous section, you’ll need to clear the chrome caches on addon shutdown, namely for chrome images and properties files. Doing this will get an update’s new properties file to load, however sometimes this will instead produce an error on next property access. It just doesn’t seem that it can reliably clear the cache correctly, for whatever reason. String changes seem to be fine, however the addition or removal of strings can sometimes produce this error. It’s not reliably reproducible, but it does happen. Yes, this is a pain in the ass.

The suggestion that seems to work is to use a hack to bypass the string bundle cache. You should still be caching a reference to your string bundle on addon startup, preferably using XPCOMUtils.jsm to lazily load the file. Here’s how I am now doing things:

Components.utils.import("resource://gre/modules/Services.jsm");
Components.utils.import("resource://gre/modules/XPCOMUtils.jsm");
XPCOMUtils.defineLazyGetter(this, "strings", function() { return loadPropertiesFile("chrome://myAddon/locale/mystrings.properties"); });
function loadPropertiesFile(path)
{
    /* HACK: The string bundle cache is cleared on addon shutdown, however it doesn't appear to do so reliably.
       Errors can erratically happen on next load of the same file in certain instances. (at minimum, when strings are added/removed)
       The apparently accepted solution to reliably load new versions is to always create bundles with a unique URL so as to bypass the cache.
       This is accomplished by passing a random number in a parameter after a '?'. (this random ID is otherwise ignored)
       The loaded string bundle is still cached on startup by Flagfox and should still be cleared out of the cache on addon shutdown.
       This just bypasses the built-in cache for repeated loads of the same path so that a newly installed update loads cleanly. */
    return Services.strings.createBundle(path + "?" + Math.random());
}

Just do “strings.GetStringFromName(stringID)” as you normally would. The lazy getter magic will cause the file to be automatically loaded the first time it is needed, after which point a reference to the loaded string bundle will be stored in “strings” for future accesses. You still need to clear the cache on addon shutdown, however it will now also load cleanly on addon updates. (the old file should still be cleared)

Put it all together

That should be all the pieces. Your chrome.manifest will have just chrome and locale (and possibly skin) mappings in it now. No resource mappings or chrome overlays. The new entry point for your addon is via bootstrap.js:startup() rather than a “load” handler in a XUL overlay.

Your localization handling should be unaffected by your transition to a restartless/extractionless addon so long as you properly clear the chrome cache on addon shutdown and load your properties files using the method listed above. Your property files and DTD files loaded from chrome:// URIs should work just as before. This is all assuming a minimum version of Firefox 17+ (or other Gecko 17+ application) which you should remember to state explicitly in your install.rdf.

Just remember that whatever you start you also need to have the ability to undo. In order for your addon to update without a restart it needs to be able to shutdown/disable cleanly.

Also note that once you do get this all up and running, your users will still have to restart Firefox once to install your first restartless update. While your new addon may not need a restart to install, if you’re updating from an old version that is not restartless then it will need a restart to uninstall that first.

This post ended up getting quite large and with a fair bit of detail and code. If there’s anything wrong here, please, by all means, point it out to me and I will correct it. I’m tired of only finding years-old out-of-date articles on this topic online, so I hope this will be found and be useful to some people.

Here are a few of the resources I used to learn this stuff:

• Mozilla Developer Network – Bootstrapped extensions (lots of contributors; updated regularly)
• Dave Townsend (Firefox Developer) – Playing with windows in restartless bootstrapped extensions (2011-01-19)
• Wladimir Palant (author of Adblock Plus) – How many hacks does it take to make your extension install without a restart? (obsolete; 2010-09-10)
• Relevant still-open bugs on Bugzilla:
  • Wladimir Palant – Allow bootstrapped add-ons to have default preferences (bug 564675)
  • Wladimir Palant – String bundle cache should be cleared if a restartless extension updates (bug 719376)
  • Dave Garrett (me) – Allow resource protocol mappings in chrome.manifest files loaded dynamically (bug 675372)
• Also, thanks to iann on the Mozilla Addons Forum for confirming that “XMLHttpRequest is a perfectly standard way to access *anything* with a URL”.
• Feel free to take a peek at the source for Flagfox (new restartless beta is here) or my other addon, Config Descriptions (the later of which is a much simpler restartless extension I wrote a while ago)

List of edits made to this post thus far:

1. Added more links to docs on MDN
2. Fixed styling of code example blocks to have a horizontal scrollbars
3. Link to bug I filed about AddonManager.getAddonByID() being unreliable in the past
4. Drop third argument from usage of addEventListener/removeEventListener in example as it is no longer needed for Firefox 6+. I had already cleaned up my main files to do this, but had missed a spot in my bootstrap.js.
5. Note that there still may be issues related to property files not loading properly on update.
6. Add a seventh step to change the way properties files are loaded to bypass the string bundle cache to avoid errors on addon update. (it’s what Adblock Plus does to work around locale file loading issues)
7. Fix typo in bootstrap.js code example. The second “startup()” function was obviously supposed to be a “shutdown()”. Whoops. :p