Flagfox 6.0 Beta 3 with Firefox 57+ support

The Flagfox 6.0b1 post is here, and the 6.0b2 post is here. Prior posts on the general topic of this endeavor are here and here. Per standard beta software recommendations, general backup info is here.

Flagfox 6.0 Beta 3 has been posted to AMO’s beta channel for people to test. Everyone who is already signed up for the Flagfox beta channel should automatically get the update.

This time, more fixes, including more fixed known issues and more fixed previously-unknown issues. Also, I got JS actions working properly, so they’re enabled now. There’s some changes to startup, so I definitely want to get people using this new version so I can make sure it works better for everyone.

Note: The number of things that only work properly in Firefox 57+ (and unfixable in Firefox 56) keeps growing. I’m considering making the minimum supported version Firefox 57 instead of 56, for the final release. I’ll at least keep it at 56 for the remainder of the beta test, however.

Flagfox 6.0b3 changelog:

  • Rewrote IPDB startup sequence a bit to fix various startup issues
  • Fixed useless context menus loading for options page
  • Relaxed pasteable text requirement for options paste menu item (Firefox 57+ only)
  • Fix “show” checkbox being reset on custom action edit
  • Finished implementation of JS actions
  • Improved default Page Metadata JS action to list everything it can find (with improved formatting)

The changelog for 6.0 beta 2 is here.

The changelog for 6.0 beta 1 is here.

For the list of stuff that’s broken because Mozilla/Google’s addon API kinda sucks, read here.

Flagfox 6.0b3 known issues:

  • The new page element action menus do not yet support actions that use IP addresses (WIP; implementation TBD)
  • A few RTL issues (always expected; Arabic and Hebrew users are probably used to seeing this in all software)
  • Options edit action dialog autocomplete and help popup are disabled (cut for time; will reimplement later)
  • Options page is kinda ugly, due to having to port to HTML and CSS being horrible (and me sucking at it)
  • Keyboard focus handling in options subdialogs is not working correctly
  • Icon/tooltip sometimes doesn’t update correctly on non-existent domain error pages
  • All page notifications for TLD/IP country mismatches have been disabled until I can make it work better

Bug reports / feedback can be posted in a comment to this post or to the thread on the support forum. You can also email me or send me a tweet or whatever. Comments for old beta posts here will be locked; please post to the new one, so we can keep track of what’s applicable for what version.

Lastly, THANK YOU. There is no deluge of beta testers, but just a handful of people are needed to be a real help. Even if you don’t personally hit anything really bad, posting a comment with any info you think might be helpful would be quite welcome.

Advertisement

Flagfox 6.0 Beta 2 with Firefox 57+ support

The Flagfox 6.0b1 post is here. Prior posts on the general topic of this endeavor are here and here. Per standard beta software recommendations, general backup info is here.

Flagfox 6.0 Beta 2 has been posted to AMO’s beta channel for people to test. Everyone who is already signed up for the Flagfox beta channel should automatically get the update.

This update has a bunch of fixes, including 5 things listed in the prior posts’s “known issues” as well as a fix for the proxy problem reported by a beta tester in the comments there. I’ve also got some other fixes, including one that could cause some notable issues if you try to import certain types of actions lists (more than one at once). There’s also 3 new default actions (not in-menu by default, but available via options) and one dropped one, though it was only dropped because it’s now equivalent to another listed one (W3C’s newer validator got promoted to their default, so I don’t need two entries anymore). Also, better favicon handling and icons for those new HTTP security/protocol feature detection tags in the flag icon tooltip. The detailed changelogs for betas to this point are listed below, along with the current known issue lists.

Flagfox 6.0b2 changelog:

  • Fixed proxy detection to work properly with sloppy undocumented behavior of Mozilla’s WebExt webRequest API (Firefox 57+ only)
  • Fixed actions importer to always import action lists with names, using host name or placeholder, if needed
  • Fixed a bunch of little bugs in the new options page (with selection handling, drag/drop, & clone selected action)
  • Implemented manual paste menu item using new API and unified paste handling
  • Added an insecure HTTP tooltip tag, mutually exclusive with HSTS & HTTP/2 tags
  • Improved HTTP protocol/security tooltip tags with simple, easy-to-notice icons (Unicode; all in color, in Nightly)
  • Added ability to handle default action favicons on non-standard paths; all should show, now (improvement over 5.x)
  • Dropped obsolete Validate.nu default action, as it’s now the same as the main W3C Validator
  • Added some new default actions: Is It Up? (alternative status checker), Built With, & Mozilla Security Observatory
  • Added hooks to options for clone selected action (if any) on normal copy/cut, via main menu or hotkey (e.g. ctrl+c)

The changelog for 6.0 beta 1 is here.

For the list of stuff that’s broken because Mozilla/Google’s addon API kinda sucks, read here.

Flagfox 6.0b2 known issues:

  • JS actions are currently not available due WebExt API restrictions; I can probably hack around this
  • The new page element action menus do not yet support actions that use IP addresses (WIP; implementation TBD)
  • There’s sometimes a IPDB recursion error on first startup, though it (usually) recovers with nothing breaking
  • A few RTL issues (always expected; Arabic and Hebrew users are probably used to seeing this in all software)
  • Options edit action dialog autocomplete and help popup are disabled (cut for time; will reimplement later)
  • Options page is kinda ugly, due to having to port to HTML and CSS being horrible (and me sucking at it)
  • Keyboard focus handling in options subdialogs is not working correctly
  • Icon/tooltip sometimes doesn’t update correctly on non-existent domain error pages
  • All page notifications for TLD/IP country mismatches have been disabled until I can make it work better

Bug reports / feedback can be posted in a comment to this post or to the thread on the support forum. You can also email me or send me a tweet or whatever. Comments for old beta posts here will be locked; please post to the new one, so we can keep track of what’s applicable for what version.

Lastly, THANK YOU. There is no deluge of beta testers, but just a handful of people are needed to be a real help. Even if you don’t personally hit anything really bad, posting a comment with any info you think might be helpful would be quite welcome.

Flagfox 6.0 Beta 1 with Firefox 57+ support

Dear lord that took forever… my initial estimate morphed from dozens of hours to a couple hundred hours to… more… but I’ve got something worth posting now. I spent half of the last week rewriting some important stuff because I found yet another Firefox bug in the WebExtension API, this time a regression from 56 (yep, they managed to break some WebExt stuff on exactly the Firefox 57 update). My running todo list of Mozilla bugs to file is currently longer than my changelog. My previous posts on this topic are here and here.

Flagfox 6.0 Beta 1 has been posted to AMO’s beta channel for people to test

As with all software development builds, a backup of your relevant data is generally recommended. Though, seeing as I was already using the WebExt API for storage in Flagfox 5.2 (for transitioning data, seeing as the WebExt API has NO method to access old prefs), you’re probably fine (though, I did change the action tab/window open setting format, but that’s a really minor thing). Flagfox 5.2+ data is in the “browser-extension-data” subfolder of your Firefox profile. Addon data is in there by addon ID, for which Flagfox has the lovely very-old-style ID of “{1018e4d6-728f-4b20-ad56-37578a4de76b}”. (full Firefox profile backup instructions are here)

Flagfox 6.0b1 changelog:

  • First pure WebExtension API version of Flagfox; Firefox 57+ support added (min version Firefox 56)
  • Ported Flagfox options from XUL window to HTML tab (along with all other dialogs), per WebExt requirement
  • Improved IPDB to now run in its own dedicated JS Worker thread
  • Started using ES6 promises everywhere (some new uses because it’s better; many compelled by WebExt API)
  • Faster flag icon updates using new WebExt API (pages loaded/cached prior to install/update must be refreshed)
  • Added simple indicator tags for HSTS and HTTP/2 to flag icon tooltips
  • Added Flagfox actions context menus for elements within a page and background tabs (IP-related stuff not yet implemented)
  • Improved keyboard support in the options page
  • Added options page support for paste to import actions from text, via main menu or hotkey (e.g. ctrl+v)
  • Fixed some bugs with actions import/drag/drop in options under recent versions of Firefox
  • Added Linux support for opening actions in a background window (not that anyone likely cares)
  • Implemented custom locale properties file parsing/loading, because screw WebExtension’s bloated I18N
  • Implemented universal automatic English fallback for locales (Flagfox 5 had some; Flagfox 6 has it for everything)
  • Now stripping redundant manual English fallback strings on build, to save a bit of space
  • Added automatic locale file unload after disuse to reduce memory use when no longer needed
  • Significantly improved location/host/ip caching (which is now required, due to WebExt APIs)
  • Implemented (partial) replacements for various features Mozilla abandoned with the WebExt transition (e.g. eTLD)
  • Removed old grey question mark icon; replaced its last remaining use with globe icon
  • Various default action changes (Netcraft & IntoDNS HTTPS; Wave A11Y dropped; WOT demoted from default menu)
  • Geotool action hotkey changed from Ctrl+Alt+G to Alt+Shift+G due to the very limited WebExtension hotkey API
  • New default action hotclicks: “Check Server Status” on double-click and “Google Cache” on triple-click
  • IPv4 & IPv6 address location database updates for December 2017

Note that some new stuff should be considered to be “experimental”, for now (aka it might not work perfectly yet). The new features were actually fairly quick to implement, given the APIs available. The new menus are the main thing you can point to as an actual improvement that’s a direct result of the WebExtension API (out of, like, two things). The HTTP info tags took like 10 minutes. 🙂

Broken stuff I can’t fix, due to WebExtension API limitations:

  • Icon is smaller and squarified, per WebExt API’s apparent requirements (not documented properly)
  • Icon menu has default toolbar menu items in it, even if N/A; no way to disable; nightly is better with this
  • Icon menu limited to 6 top-level items, per arbitrary WebExt API limitation; others in auto-submenu
  • Icon menu item favicons (if enabled) have no default; WebExt API has no way to set a default/onerror
  • All menu generation is done more often than prior versions due to primitive WebExt API (no show event)
  • Icon tooltip doesn’t have columns or decent styling, as the WebExt API here is primitive (plain text only)
  • Icon middle-click settings no longer work, due to limited WebExt API (existing settings left alone, but hidden)
  • Hotkey customization is no longer available, due to very limited WebExt API (no user-set hotkey support, at all)
  • The WebExt notifications API outputs to the OS notifications system, which may get truncated text (likely to replace)
  • There is no way to select “Don’t show this again” for notifications (custom buttons not supported in Firefox)
  • Firefox wipes all WebExtension data on uninstall, without user notification or prompting (affects everyone)

Known issues (aka broken stuff I should be able to fix):

  • JS actions are currently not available due WebExt API restrictions; I can probably hack around this
  • The new page element action menus do not yet support actions that use IP addresses (WIP; implementation TBD)
  • There’s sometimes a IPDB recursion error on first startup, though it recovers with nothing breaking
  • A few RTL issues (always expected; Arabic and Hebrew users are probably used to seeing this in all software)
  • Options page action drag/drop to reorder always snaps to the bottom if you drag out of the box
  • Options edit action dialog autocomplete and help popup are disabled (cut for time; will reimplement later)
  • Options page is kinda ugly, due to having to port to HTML and CSS being horrible (and me sucking at it)
  • Options page add item hover menu option for pasting doesn’t work (may or may not be possible with the API)
  • Keyboard focus handling in options subdialogs is not working correctly
  • Icon/tooltip sometimes doesn’t update correctly on non-existent domain error pages
  • All page notifications for TLD/IP country mismatches have been disabled until I can make it work better
  • The browser console will end up with lots of log messages from l10n.js due to leaving a setting fairly low for testing. The setting has been turned up a bit and the log messages will be shut up for future versions.
  • New default action hotclicks aren’t activated on startup; reload Flagfox or set/unset a action option to work-around (sigh… found literally minutes after posting this)

Another known issue: I need to look into making the content area on this blog wider to fit this stuff better. (Edit: fixed via theme change)

Also note that this is the first time I’m using Mozilla’s beta channel since they switched to mandatory addon signing. It’ll probably work fine, but I don’t actually know yet.

Bug reports / feedback can be posted in a comment to this post or to the thread on the support forum. You can also email me or send me a tweet or whatever.

Reports of my demise have been greatly exaggerated

Mozilla just released Firefox 57; they’re calling it Firefox Quantum. It’s got some new stuff under the hood, but the big news that’s relevant here is that they’ve banned all of the addon APIs that have existed in Firefox and other Mozilla software for over a decade in favor of their new WebExtensions API (that’s based on Google Chrome’s extension API), and the transition is painful. Just looking at some of the most used Firefox addons, Adblock Plus (#1) and Easy Screenshot (#4) only released their rewrites a week before the deadline. (Adblock Plus got some features cut in its new version, as well) Video DownloadHelper (#3) released their revision a matter of hours into the Firefox 57 release day. NoScript (#5) is currently without a WebExtension version (as is Giorgio’s other popular addon, Flashgot (#20)), and Firebug (#6) has been been officially discontinued as of the Firefox 57 release. I see another 7 in the top 40 that aren’t compatible, for a total of 11 (including Flagfox). Most of the popular stuff has been rewritten for the new API, but most of the popular stuff has a team of developers, and are probably not doing this sitting on their couch.

I planned to have Flagfox 6.0 ready for release before Firefox 57 released and Mozilla commenced the add-on-apocalypse, but sadly, that didn’t happen. After many technical, personal, and health related delays, I made the time to work on rewriting Flagfox as a WebExtension, and whilst I was correct in my estimates on what part would be the hardest, I vastly underestimated how time consuming it would be. It’s hundreds of hours of work, and I do need to sleep, from time to time. I at least prioritized getting Flagfox 5.2 released a few weeks ago, which uses the WebExtension API for the preferences storage, if available, so at least when I do release the new update, people won’t lose their settings and saved actions. I only learned a week or so before releasing 5.2 that the WebExtension API has no method to import old data from “legacy” extensions, whatsoever. Any other addon that is in the same boat as me, but didn’t put out a hybrid-WebExtension release to port over their user storage, will end up with all of their users’ data getting lost without a way to fix it from within the addon (unless, of course, it’s an online service, in which case they’re lucky enough to not rely entirely on Firefox to not ditch their stuff). Oh, and while we’re on that subject, don’t uninstall any addon that’s using the WebExtension API for its preferences/data unless you’ve either backed up your stuff or want it gone. Firefox deletes your saved data from (hybrid-)WebExtension on uninstall without even telling you, let alone prompting.

As to one of the big technical delays:
<CSS rant>
I hate CSS with the passion of a thousand burning suns. It’s 20 years of crap piled upon crap; people make jokes about how awful it is for a reason, and they’re downplaying things. I had to rewrite the Flagfox options GUI in HTML in a tab instead of its original XUL, and my opinion of XUL is no longer nearly as low as it was before. Just putting together something with a box with dynamically generated stuff in it, and not having it fall off the bottom of the screen for no reason took a while. Fun fact: in CSS, you can use the “vh” unit to set something as a percentage of the vertical page height. Set it to 100vh… and it falls off the bottom and you have to scroll. Turns out it always counts in a potential horizontal scrollbar; there is no good fix that isn’t just fudging it. I could continue ranting about how setting a width/height to 100% doesn’t do anything unless you also set that on all of its containers, and also the html and body tags. Some sizing things don’t work unless you set a minimum, even if it’s zero. Only way I could get the secondary dialogs to not be cropped with scrollbars was to use an ancient JS hack to force resize them after creation; I can’t figure out how to properly do it with CSS, and neither can anyone else, it seems. Learning this crap is not only slow due to the fact that CSS seems to have been designed by an alien from non-euclidean space, but it’s such a mess that when you search for “how do I do ___ in CSS?”, you get a deluge of hits, all of which have many answers, many of which differ, and you could try them all and still not have it work. Reading the actual documentation isn’t that much better, seeing as there’s 20 years of crap in this sprawling mess of a spec.
</CSS rant>

I have got that working now, though. The new version may actually work better than the old, even if it might be a little ugly at the moment. I’m currently dealing with some async hell in getting the WebExtension APIs hooked up to do the actual backend work properly. It took a while to learn JS promises, but unlike CSS, I actually figured it out and understand it enough to properly use it, at least to the degree that I need to. Everything else I have left looks to be straightforward, even if it’ll take a while to learn each individual new API, because nothing is really simple anymore.

TL;DR: Flagfox doesn’t support Firefox 57 yet, but I’m working fairly steadily towards getting it ready. I think I may chose to go a bit slower, however, as sleeping every other night is not healthy. (that’s not a joke) Flagfox is a single developer hobby project, so having some patience would be greatly appreciated. If you wish to avoid the Firefox 57 update for the time being, you can just install Firefox 52 ESR (extended support release), which is the other version Mozilla puts out. It’s going to keep getting security and stability updates for another half a year or so, and “legacy” addons work with it still (if not better, as some new bugs crept in later versions that broke some things). It’s officially for “large organizations” who don’t like frequent feature changes and things breaking, but regular users who also want this can use it just fine.

Oh, and in case you’re wondering how Mozilla thought this was going to go, I figured out why they released a major redesign of the Mozilla Add-ons site very recently. You’ll notice that all addons’ pages no longer show the text of their reviews anymore. I’m pretty sure they knew damned well that there was going to be a lot of pissed users, even plenty for addons that did make the jump, because plenty of addons had to cut features to get things working with the new limited API. There’s plenty of other features Mozilla dropped from their site, as well, though at least for now you can switch back to the old version via a link hidden in the footer. A big one that annoys me and has no real workaround is that email notifications of reviews all say “none” for the message now, so if a user posts a bug report in a review and a moderator deletes it before I read it, I’ll never read it. (not that it matters at the moment) Oh, and their server was dead for this morning, which isn’t a great sign, either. I’d be posting (a link to) this on the Flagfox addon page if it weren’t for the fact that I’ve got 30-some-odd languages that I’d have to edit the text for, and the new version of their site dropped the developer comments section where I’d normally put this sort of stuff.

Anyway, this post is too long. The new version will be done eventually. I don’t need you to send me more messages like somehow I don’t know the situation. Please be patient, and try to treat other people on the Internet like people.

(comments disabled because I don’t have the time to deal with them right now, and I’ve already got enough communication channels to worry about)

Things I’ve learned from the survey thus far

I have a survey link in the Flagfox icon menu for the current version to get some stats and info about Flagfox actions usage, sync, and some of the services involved here. All submissions are anonymous and I will not be posting raw data, but here are a few interesting things I have learned from this experiment thus far. Note that this survey is not otherwise advertised, so this is a survey of more advanced users. (which is intentional)
Continue reading

Issues with Flagfox 5.0 under certain themes

Apparently the default Mac OS X Firefox theme is squishing the icons a little bit. The different OSes have varying theme characteristics in the current version of Firefox. This is a pain in the ass for development, especially because I don’t own a Mac. I had about 200 beta testers for a month or so, but nobody noticed the subtle issue. I’m looking into a way to fix it at the moment.

Some people also do not like the light flag border when using a dark theme. I tested this before the release and thought it looked OK, but I don’t use dark themes personally. The 5.0.1 update has an updated border that will use a dark border under dark themes, so it will be less noticeable.

Also previously noted is that if you use a lightweight theme (wallpaper/skin/Persona) under Linux (or possibly some other OSes, but not Windows) then Firefox applies an unfocused effect to your interface. If you click the address bar and the flag icon gets lighter, it’s because of your theme. Under Windows this only affects the background and not the icon. There is no way to override this effect for just Flagfox; it affects everything, including search plugin icons. I was well aware of this effect before the release, as I am on Linux myself, however I don’t personally have an issue with it. If you do, you can disable it with the following userChrome.css:

#urlbar:-moz-lwtheme:not([focused="true"]),
.searchbar-textbox:-moz-lwtheme:not([focused="true"]) {
    opacity: 1 !important;
}

Update:
A fix for the border issue has been released in Flagfox 5.0.1.