diff --git a/doc/updateJSONSpec.md b/doc/updateJSONSpec.md new file mode 100644 index 000000000000..061629fd763c --- /dev/null +++ b/doc/updateJSONSpec.md @@ -0,0 +1,160 @@ +Previous Commentary +=================== + +This document used to be hosted as a Gist, and there was some commentary on the document +on the relevant gist page. That commentary can be [found here](https://gist.github.com/redwire/2e1d8377ea58e43edb40). + +update.json and update.json.sig +=============================== + +The file `update.json` provides information about the currently deployed HTTPS Everywhere +ruleset library. The file contains the same kind of information as is stored in the +`update.rdf` file used to tell the HTTPS Everywhere extension where to fetch and how to +verify updates to the whole extension. + +The utility script `https_everywhere/utils/ruleset_update_manifest.py` automates most of +the process of creating `update.json`, and is simply run with the command: + + python utils/ruleset_update_manifest.py + +Signing the `update.json` file requires the use of NSS tools and pk1sign, which should +come with the NSS tools. They can be installed on Debian-based Gnu+Linux systems via: + + sudo apt-get install libnss3-tools + +A signing key can be created using the following commands: + + # Make the NSS keyring directory + mkdir nssdb + # Initialize the directory to be used as a keyring + certutil -N -d nssdb + # Generate a 2048-bit self-signed object signing certificate nicknamed "httpse" + certutil -S -n httpse -g 2048 -d nssdb -t "p,p,u" -s "CN=EFF" -x + +Finally, issuing the following command will output both the signature of `update.json` +and the public key used to verify it, both base64 encoded. + + pk1sign -i update.json -k httpse -d nssdb + +Fetching +======== + +Retrieving updates for only a part of an extension is not possible using the builtin +extension update mechanisms supported by Firefox. However, fetching files using standard +XMLHTTPRequests from within the extension is trivial to accomplish and does not require trying +to force the browser to prematurely check for updates or apply changes only to parts of the +extension. +Thus, a simple XHR will be used to fetch `update.json` from eff.org. + +A signature over the raw bytes of `update.json` will be served as `update.json.sig` from a +separate hard-coded URL. + +The extension should check for ruleset updates when the browser is started, and then again +every 10800000 milliseconds (3 hours). +If an attempt to fetch or verify an update fails, the extension should request `update.json` +again every 300000 + R milliseconds, where R is a random number between 0 and 300000. +The extension should attempt this a maximum of 6 times until it is +able to fetch `update.json` and verify the contents before defaulting back to waiting 3 hours. + +Every time the extension finds that the data provided by `update.json` to be inauthentic, +either as a result of the hash of the database file not matching or the signature not verifying, +the extension must send a POST request to a hardcoded failure-reporting url containing the data +in the `update.json` file that it tried and failed to verify. The format for failure reports +has yet to be determined. + +Verification and Version Checking +================================= + +In order to compute the hash of the database file, the nsICryptoHash class will be used. +https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsICryptoHash +SHA1 is currently being used as the hashing algorithm. + +In order to verify the signature over `update.json` file contents, which is the content of `update.json.sig`, +the nsIDataSignatureVerifier class will be used. +https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIDataSignatureVerifier + +The `version` will be compared to the currently installed ruleset library version using nsIVersionComparator. +https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIVersionComparator +Any HTTPS Everywhere extension with version X.Y.Z can ONLY accept ruleset versions of the form X.Y.Z.N, which +is to say that the extension will only accept ruleset versions that are subversions of itself. + +The branch field can be checked and referenced against a value stored in user preferences. +A stable release of the extension will have a `branch` preference set to "stable", +and a development release will set the option to "development" or another specified constant. + +Updating +======== + +For an early implementation of the ruleset update mechanism, the extension will update its +ruleset library (stored as a SQLite database) by fetching a SQLite database file from eff.org +containing the new ruleset library, and replace its local database file. +Replacing the local rulesets library will only occur after validating `update.json` +(pseudocode below). + +Manifest +======== + +The following is the format for a simple `update.json` containing information about a single +ruleset library to be used by all target applications. + + {"branch": + ,"changes": + ,"date": + ,"hash": + ,"hashfn": + ,"source": + ,"version": } + +The meanings of each field are explained as follows: + +* `branch` is a machine-readable target extension branch, e.g "stable", "developer", "willeatyourpc", etc. The extension SHOULD check that this field matches the extension's branch preference and MAY refuse to accept updates it doesn't like, e.g. stable extension release MAY refuse to update to developer rulesets. +* `date` is a formatted date string detailing the date that the release to the ruleset was released, and will be set +automatically by the `ruleset_update_manifest.py` utility. The format for the date is "dd-mm-yyyy". For example, "08-06-2014". +* `changes` is a short human-readable description of what has been changed, perhaps a list of the biggest or most requested fixes +* `hash` is a hash of the database file's content, encoded in hex. The hash function to use is specified by `hashfn`. +* `hashfn` is the name of the hash function used to compute the hash of the database file and MUST be one of md5, sha1, sha256, sha384, or sha512. +* `source` is the URL from which the most recently released database file is to be fetched. The URL must address a valid eff.org location and must not be overwritten by the extension. +* `version` is the ruleset release version, which is a subversion of the extension release version and MUST be checked by the updater to determine whether the advertised release is newer than the currently installed ruleset library. A ruleset release with a version number such as "3.5.1.2" must ONLY be accepted by HTTPS-Everywhere version 3.5.1, and so on. + +Pseudocode of update procedure +============================== + +The following is a high-level description of how the update mechanism is expected to function. Specifically, it details the order in which important authenticity tests are carried out + + // Called at startup of HTTPS-E and every three hours afterwards. + function checkAndApplyUpdate() do + updateURL := preferences.get("updateURL") + updateSigURL := preferences.get("updateSigURL") + branchName := preferences.get("branchName") + currentVersion := preferences.get("rulesetVersion") + # tryFetch tries to request data up to 6 times in randomly padded intervals + # See the "Fetching" section + updateJSON := tryFetch(updateURL) + updateData := JSON.parse(upateJSON) + shouldUpdate := updateJSON != null and\ # Were able to receive data + updateData.version > currentVersion and\ + isSubversion(updateData.version, EXTENSION_VERSION) and\ + branchName == updateData.branch + inauthentic := true + if shouldUpdate then + signature := tryFetch(updateSigURL) + if isValidSignature(signature, updateJSON, PUBLIC_SIGNING_KEY) then + databaseSource := tryFetch(updateData.source) + if databaseSource == null then + return + endif + dbHash := hashFunctions[updateData.hashfn](databaseSource) + if dbHash == updateData.hash then + dbFile := openFile(RULESET_DB_FILE) + write(databaseSource, dbFile) + HTTPSRules.init() + inauthentic := false + endif + endif + endif + if inauthentic then + # tryPost works like tryFetch, but also sends data with a POST request + tryPost(REPORT_INAUTHENTIC_UPDATE_URL, updateData) + endif + endfunction + diff --git a/https-everywhere-tests/test/test-rsupdate-verify.js b/https-everywhere-tests/test/test-rsupdate-verify.js new file mode 100644 index 000000000000..64e93956aa70 --- /dev/null +++ b/https-everywhere-tests/test/test-rsupdate-verify.js @@ -0,0 +1,101 @@ +// Test the signature verification of the sha256 hash of update.json + +const { Cc, Ci, Cu } = require('chrome'); +const { atob, btoa} = Cu.import('resource://gre/modules/Services.jsm', {}); + +const PUBKEY = ""+ + "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvrJKpqX5"+ + "kbiSX6DEKNcFO8U/Kw3jn1z+im+hSwCHNCXUzJ1IkwtRe2QkK66g"+ + "1kCGnQ9rxnaOnLRSi8DK6Yypobm0paG71/WtHyRQzKDASPbVhy0U"+ + "Men/3sGBOIOlT1JbZskHxVdEBJfb7YOr+a1BSgaIsrbaI7n9tmrT"+ + "ysC5ECN5i5ETFQz0Hni7iqKWUB/a2dfDu0U4VDsJHIt1PKsduIJG"+ + "aACT7+CZuaw3Jvc/utCOh0tgMHXxtxrezRu56ouVcttsQSuVQ56g"+ + "xSwnASElECsmUgs6ci+ts4LMDrF8l/J1t778lIPQb2jf3QrNxsgV"+ + "LKFSePJ2bwONkTSaj48I2wIDAQAB"; + +const UPDATE_JSON = '' + + '{"branch": "development"\n'+ + ',"changes": "Still trying to get signature verification working"\n'+ + ',"date": "04-08-2014"\n'+ + ',"hash": "8476a3638e2e95b6381aba4367e8c1c0f361bdaed501cd2f6f91b74e4545"\n'+ + ',"hashfn": "sha256"\n'+ + ',"source": "http://0.0.0.0:8000/data/rulesets.sqlite"\n'+ + ',"version": "5.0.0.1"}\n'; + +const UPDATE_JSON_SIG = ""+ + "MIIBFDANBgkqhkiG9w0BAQUFAAOCAQEAWi6AThn723ovBCri3iuxu"+ + "9g2mQGvKrWzeOBPW/2kc8U0DwVMn/v17ZtjWJuFnFyCpWFT88/gnC"+ + "a+QV3+fupw4D9RwkFvdyKAp8ok7jGdrDl78jEmpqbqlc9Oz7WXDFr"+ + "5OpxSjkd4jn4Pks0ImhZejuoWE1dLg7qLQhJ3gYzAT4IRKVVzBTkA"+ + "zHHxKvuHzWpD3Q7pI/GkqUcygyJdVwYv6LQesnmdMzwIpyfgf/6ac"+ + "znoQj3LsNn0W1Q6so64ng7aWx+2gFZp/f9Y8I5llWHaO55wEgKOBZ"+ + "4n7OOvTLn0jyfbwkySzjaXmFgE1be6TmZ/dXggaI0w0uG5L2QIxfOhhg=="; + +function hashSHA256(data) { + let converter = Cc['@mozilla.org/intl/scriptableunicodeconverter'] + .createInstance(Ci.nsIScriptableUnicodeConverter); + let hashing = Cc['@mozilla.org/security/hash;1'] + .createInstance(Ci.nsICryptoHash); + function toHexString(charCode) { + return ('0' + charCode.toString(16)).slice(-2); + } + hashing.init(hashing.SHA256); + converter.charset = 'UTF-8'; + let result = {}; + let converted = converter.convertToByteArray(data, result); + hashing.update(converted, converted.length); + let hashed = hashing.finish(false); + return [toHexString(hashed.charCodeAt(i)) for (i in hashed)].join(''); +} + +function validUpdateData(updateHash, signature) { + return Cc['@mozilla.org/security/datasignatureverifier;1'] + .createInstance(Ci.nsIDataSignatureVerifier) + .verifyData(updateHash, signature, PUBKEY); +} + +exports['test binary-base64 encoding'] = function(assert) { + assert.strictEqual('hello', atob(btoa('hello')), + 'Test that binary/base64 encoding works.'); +}; + +/* This test is just meant to make sure that the object was parsed into JSON + * properly and that the attributes of the object created can be read. + */ +exports['test update JSON parsing'] = function(assert) { + let updateObj = JSON.parse(UPDATE_JSON); + assert.equal(updateObj.hash, + '8476a3638e2e95b6381aba4367e8c1c0f361bdaed501cd2f6f91b74e4545', + 'Test that the data was parsed into JSON properly'); +}; + +exports['test update JSON signature validity'] = function(assert) { + let hashed = hashSHA256(UPDATE_JSON); + console.log("###############################################################"); + console.log(hashed); + console.log(hashed.charCodeAt(hashed.length - 1)); + console.log("###############################################################"); + let verifier = Cc['@mozilla.org/security/datasignatureverifier;1'] + .createInstance(Ci.nsIDataSignatureVerifier); + assert.equal(hashed, + 'e05c92fcb9ed93344d8f5e9b358e64f7e0ab13eb8cd3a6fce2581f1d8cc73832', + 'Test that the update.json data hashed to the right value'); + assert.equal(typeof verifier, 'object', 'Test verifier creation success'); + assert.ok(verifier.verifyData( + UPDATE_JSON, UPDATE_JSON_SIG, PUBKEY), + //hashed, UPDATE_JSON_SIG, PUBKEY), + 'Test that the update.json raw data is authentic'); +}; + +exports['test ruleset version comparison'] = function(assert) { + let vcmp = Cc['@mozilla.org/xpcom/version-comparator;1'] + .createInstance(Ci.nsIVersionComparator); + assert.ok(vcmp.compare('3.5.2', '3.5.2') === 0, + 'Test that equal version numbers are confirmed equal by version comparator'); + assert.ok(vcmp.compare('4.0development.17', '3.5.3') > 0, + 'Test that 4.0development.17 > 3.5.3'); + assert.ok(vcmp.compare('3.5.3.1', '3.5.3.2') < 0, + 'Test that ruleset version 3.5.3.2 > 3.5.3.1'); +}; + +require('sdk/test').run(exports); diff --git a/src/chrome/content/code/HTTPSRules.js b/src/chrome/content/code/HTTPSRules.js index 62a72fbe7bd6..de36d93742f9 100644 --- a/src/chrome/content/code/HTTPSRules.js +++ b/src/chrome/content/code/HTTPSRules.js @@ -389,7 +389,16 @@ const HTTPSRules = { this.scanRulefiles(rulefiles); // Initialize database connection. - var dbFile = new FileUtils.File(RuleWriter.chromeToPath("chrome://https-everywhere/content/rulesets.sqlite")); + // Use the rulesets database file downloaded in an update if one exists. + var updatedPath = HTTPSEverywhere.instance.UPDATED_RULESET_DBFILE_PATH(); + var updatedDBFile = Cc['@mozilla.org/file/local;1'].createInstance(Ci.nsILocalFile); + updatedDBFile.initWithPath(updatedPath); + if (updatedDBFile.exists()) { + var dbFile = new FileUtils.File(updatedPath); + } else { + var dbFile = new FileUtils.File(RuleWriter.chromeToPath( + "chrome://https-everywhere/content/rulesets.sqlite")); + } var rulesetDBConn = Services.storage.openDatabase(dbFile); this.queryForRuleset = rulesetDBConn.createStatement( "select contents from rulesets where id = :id"); diff --git a/src/chrome/content/code/RulesetUpdater.js b/src/chrome/content/code/RulesetUpdater.js new file mode 100644 index 000000000000..633e31ded179 --- /dev/null +++ b/src/chrome/content/code/RulesetUpdater.js @@ -0,0 +1,287 @@ +/* Secure ruleset udpate mechanism + * The contents of this file exist to provide HTTPS Everywhere with a secure mechanism + * for updating the extension's database of rulesets. + * This "module" handles the tasks of fetching an update.json[1] manifest and: + * 1) determining whether an update to the ruleset library has been released, and + * 2) verifies that the update is authentic before applying the new ruleset. + * + * [1] The format and specification of the update.json file is detailed within a + * Github gist, at https://gist.github.com/redwire/2e1d8377ea58e43edb40 + * + * The file https-everywhere/utils/ruleset_update_manifest.py exists to automate + * part of the process of creating the update.json manifest data. + */ + +// TODO +// Set this value. +/* Hardcoded public key used to verify the signature over the update data */ +const RULESET_UPDATE_KEY = ''; + +/* installed ruleset version preference key */ +const RULESET_VERSION_PREF = 'extensions.https_everywhere.ruleset_version'; + +/* key for the preference that holds the url to fetch update.json and update.json.sig from */ +const RSUPDATE_URL_PREF = 'extensions.https_everywhere.ruleset_update_url'; +const RSUPDATE_SIG_URL_PREF = 'extensions.https_everywhere.ruleset_update_signature_url'; + +/* path to the temporary download location of new ruleset database files */ +const TMP_RULESET_DBFILE_PATH = OS.Path.join( + OS.Constants.Path.tmpDir, + nonceForTmpFile(32) + "new_rulesets.sqlite"); + +/* maximum number of attempts to fetch ruleset updates */ +const MAX_RSUPDATE_FETCHES = 6; + +const _prefs = CC["@mozilla.org/preferences-service;1"] + .getService(CI.nsIPrefService).getBranch(""); + +function nonceForTmpFile(numBytes) { + var rng = Cc['@mozilla.org/security/random-generator;1'] + .createInstance(Ci.nsIRandomGenerator); + var buffer = new Array(numBytes); + rng.generateRandomBytes(numBytes, buffer); + var s = [("0" + b.toString(16)).slice(-2) for each (b in bytes)].join(""); + return s; +} + +/******************************************************************************* + *** Design Explanation * + * HTTPSEverywhere cannot be referred to within the constructor of a more * + * typical object definition, because doing so would create a circular * + * dependency between HTTPSEverywhere's and RulesetUpdater's constructors. * + * Using an anonymous function, I create a new scope within which helper * + * functions can be defined and invoked in a natural order. * + ******************************************************************************* + */ + +const RulesetUpdater = { // BEGIN OBJECT DEFINITION +fetch_update : (function() { // BEGIN MODULE + +/* Initiates the check for ruleset updates. + */ +function fetchUpdate() { + https_everywhereLog(INFO, "Calling fetchUpdate"); + var manifestSrc = _prefs.getCharPref(RSUPDATE_URL_PREF); + HTTPSEverywhere.instance.try_request(MAX_RSUPDATE_FETCHES, 'GET', manifestSrc, + function(responseText) { + https_everywhereLog(INFO, "Successfully fetched update.json file data"); + conditionallyApplyUpdate(responseText); + }); +} + +/* Get the branch name of the installed extension. + * Assumes that, if "development" is in the version string, the branch is "development", + * otherwise, that it is "stable". + * + * @param version - The version string of the extension. + * @return string - The branch name of the extension ("development" or "stable"). + */ +function extensionBranchName(version) { + if (version.indexOf("development") >= 0) { + return "development"; + } + return "stable"; +} + +/* Ensures that the ruleset update described by the downloaded update.json file is + * appropriate for this version of the extension and is new, verifies update.json's + * authenticity, and fetches the database file containing the updated rulesets. + * + * @param update - The string contents of the update.json file. + */ +function conditionallyApplyUpdate(update) { + https_everywhereLog(INFO, "Got update data:"); + https_everywhereLog(INFO, update); + var updateObj = JSON.parse(update); + AddonManager.getAddonByID("https-everywhere@eff.org", function (addon) { + var extVersion = addon.version; + var extBranch = extensionBranchName(extVersion); + var rulesetVersion = _prefs.getCharPref(RULESET_VERSION_PREF); + https_everywhereLog(INFO, "Inside call to conditionallyApplyUpdate"); + if (!checkVersionRequirements(extVersion, rulesetVersion, updateObj.version)) { + https_everywhereLog(NOTE, 'Downloaded an either incompatible ruleset library or not a new one.'); + return; + } + if (updateObj.branch !== extBranch) { + https_everywhereLog(NOTE, 'Downloaded a ruleset update for the incorrect branch.'); + return; + } + var sigFileSrc = _prefs.getCharPref(RSUPDATE_SIG_URL_PREF); + HTTPSEverywhere.instance.try_request(MAX_RSUPDATE_FETCHES, 'GET', sigFileSrc, function(signature) { + signature = signature.trim(); + https_everywhereLog(INFO, "Successfully fetched update.json.sig file data"); + if (verifyUpdateSignature(update, signature)) { + https_everywhereLog(INFO, "Ruleset update data signature verified successfully"); + fetchVerifyAndApplyDBFile(updateObj.source, updateObj.version, updateObj.hashfn, updateObj.hash); + } else { + https_everywhereLog(WARN, 'Validation of the update signature provided failed.'); + // TODO + // Ping the verification-failure-reporting URL + } + }); + }); +} + +/* Attempts to verify the provided signature over updateStr using + * the hardcoded RULESET_UPDATE_KEY public key. + * + * @param updateStr - The string contents of the downloaded update.json file. + * @param signature - The string contents of update.json.sig. + * @return bool - true if the update.json content is authentic else false. + */ +function verifyUpdateSignature(updateStr, signature) { + var verifier = Cc['@mozilla.org/security/datasignatureverifier;1'] + .createInstance(Ci.nsIDataSignatureVerifier); + https_everywhereLog(INFO, "Created instance of nsIDAtaSignatureVerifier"); + return verifier.verifyData(updateStr, signature, RULESET_UPDATE_KEY); +} + + +/* Checks that the ruleset version to download is greater than the current ruleset library + * version and is a subversion of the extension version. + * + * @param extVersion - The version of the extension (eg: "5.0.0") + * @param rsVersion - The current ruleset version (eg: "5.0.0.1") + * @param newVersion - The version of the new ruleset supplied by update.json. + * @return bool - true if the new ruleset version is applicable else false. + */ +function checkVersionRequirements(extVersion, rsVersion, newVersion) { + var verCompare = Cc['@mozilla.org/xpcom/version-comparator;1'] + .getService(Ci.nsIVersionComparator); + https_everywhereLog(INFO, "Checking version requirements with extension version " + extVersion + + " and ruleset version " + rsVersion); + var newRulesetExtVer = newVersion.slice(0, newVersion.lastIndexOf('.')); + var sameExtVer = verCompare.compare(extVersion, newRulesetExtVer) === 0; + var newRSVer = verCompare.compare(newVersion, rsVersion) > 0; + return sameExtVer && newRSVer; +} + +/* Compute the hash of the contents of a binary file's contents. + * + * @param path - The string path to the file whose contents are to be hashed. + * @param length - The number of bytes of data in the file to hash. + * @param hashfn - The name of the hash function to use (eg: "sha256"). + * @return string - The hex-encoded hash of the file's contents. + */ +function hashBinaryFile(path, length, hashfn) { + var READONLY = 0x01; + var READ_PERMISSIONS = 0444; + var NOFLAGS = 0; + var f = Cc['@mozilla.org/file/local;1'].createInstance(Ci.nsILocalFile); + var istream = Cc['@mozilla.org/network/file-input-stream;1'] + .createInstance(Ci.nsIFileInputStream); + var binaryIn = Cc['@mozilla.org/binaryinputstream;1'].createInstance(Ci.nsIBinaryInputStream); + var hashing = Cc['@mozilla.org/security/hash;1'].createInstance(Ci.nsICryptoHash); + switch (hashfn) { + case 'md5': hashing.init(hashing.MD5); break; + case 'sha1': hashing.init(hashing.SHA1); break; + case 'sha256': hashing.init(hashing.SHA256); break; + case 'sha384': hashing.init(hashing.SHA384); break; + case 'sha512': hashing.init(hashing.SHA512); break; + default: return ''; + } + f.initWithPath(path); + istream.init(f, READONLY, READ_PERMISSIONS, NOFLAGS); + binaryIn.setInputStream(istream); + hashing.updateFromStream(binaryIn, length); + var hash = hashing.finish(false); // Get binary data back + istream.close(); + function toHexStr(charCode) { + return ('0' + charCode.toString(16)).slice(-2); + } + return [toHexStr(hash.charCodeAt(i)) for (i in hash)].join(''); +} + +/* Fetches the contents of the new ruleset database file, ensures that the contents' hash + * matches what update.json says it should be, and then makes the call to apply the new + * rulesets database. + * + * @param url - The URL from which to fetch the new rulesets database file. + * @param version - The ruleset version (eg: 5.0.0.1). + * @param hashfn - The name of the hash function to use when hashingthe database file. + * @param hash - The hash provided by update.json. + */ +function fetchVerifyAndApplyDBFile(url, version, hashfn, hash) { + var DEFAULT_PERMISSIONS = -1; + var NOFLAGS = 0; + (function recur(max_times) { + https_everywhereLog(INFO, "Making request to get database file at " + url); + var xhr = Cc['@mozilla.org/xmlextras/xmlhttprequest;1'].createInstance(Ci.nsIXMLHttpRequest); + xhr.open("GET", url, true); + xhr.responseType = 'arraybuffer'; + xhr.onload = function(evt) { + var arrayBuffer = xhr.response; + if (arrayBuffer) { + var byteArray = new Uint8Array(arrayBuffer); + https_everywhereLog(INFO, "byteArray has length " + byteArray.length); + var file = Cc['@mozilla.org/file/local;1'].createInstance(Ci.nsILocalFile); + var outstream = Cc['@mozilla.org/network/file-output-stream;1'] + .createInstance(Ci.nsIFileOutputStream); + var binout = Cc['@mozilla.org/binaryoutputstream;1'].createInstance(Ci.nsIBinaryOutputStream); + file.initWithPath(TMP_RULESET_DBFILE_PATH); + outstream.init(file, DEFAULT_PERMISSIONS, DEFAULT_PERMISSIONS, NOFLAGS); + binout.setOutputStream(outstream); + binout.writeByteArray(byteArray, byteArray.length); + outstream.close(); + var dbHash = hashBinaryFile(TMP_RULESET_DBFILE_PATH, byteArray.length, hashfn); + https_everywhereLog(INFO, "dbhash = " + dbHash); + if (dbHash === hash) { + https_everywhereLog(INFO, + 'Hash of database file downloaded matches the hash provided by update.json'); + applyNewRuleset(version); + } else { + https_everywhereLog(INFO, 'Hash of database file did not match the one in update.json'); + if (max_times > 0) { // Strict limit test + recur(max_times - 1); + } + // TODO: Ping EFF URL to report authenticity verification failure + } + } else { + https_everywhereLog(INFO, 'Did not download any database data'); + if (max_times > 0) { // Strict limit test + recur(max_times - 1); + } + // TODO: Ping EFF URL to report download failure + } + }; + xhr.send(null); + })(MAX_RSUPDATE_FETCHES); +} + +/* Moves the downloaded rulesets database into a permanent location and reinitializes + * HTTPSRules to use the rulesets. + * + * @param version - The ruleset version. + */ +function applyNewRuleset(version) { + var DIRECTORY_TYPE = 1; + var DIRECTORY_PERMISSIONS = 0777; + https_everywhereLog(INFO, 'In applyNewRuleset'); + var updatedPath = HTTPSEverywhere.instance.UPDATED_RULESET_DBFILE_PATH(); + var permFile = Cc['@mozilla.org/file/local;1'].createInstance(Ci.nsILocalFile); + var tempFile = Cc['@mozilla.org/file/local;1'].createInstance(Ci.nsILocalFile); + permFile.initWithPath(updatedPath); + tempFile.initWithPath(TMP_RULESET_DBFILE_PATH); + https_everywhereLog(INFO, 'Initialized local database files'); + var permParent = permFile.parent; + if (permFile.exists()) { + permFile.remove(false); + https_everywhereLog(INFO, 'Removed existing updated database file'); + } else if (!permParent.exists()) { + permParent.create(DIRECTORY_TYPE, DIRECTORY_PERMISSIONS); + https_everywhereLog(INFO, 'Created directory for downloaded ruleset database files'); + } + tempFile.moveTo( + permParent, + OS.Path.basename(updatedPath)); + https_everywhereLog(INFO, 'Copied new database file to permanent location'); + HTTPSRules.init(); + _prefs.setCharPref(RULESET_VERSION_PREF, version); + https_everywhereLog(INFO, 'Reinitialized HTTPSRules with new database'); +} + +// Export only fetchUpdate +return fetchUpdate; + +})() // END MODULE +}; // END OBJECT DEFINITION diff --git a/src/chrome/content/toolbar_button.js b/src/chrome/content/toolbar_button.js index 2549dae9f35b..baede5c4fbf1 100644 --- a/src/chrome/content/toolbar_button.js +++ b/src/chrome/content/toolbar_button.js @@ -31,6 +31,11 @@ if (!httpsEverywhere) { var httpsEverywhere = {}; } * such as when the toolbar is disabled. */ httpsEverywhere.toolbarButton = { + + /** + * Name of the preference for enabling/disabling the ruleset updater. + */ + RULESET_UPDATER_PREF: "extensions.https_everywhere.ruleset_updater.enabled", /** * Name of preference for determining whether to show ruleset counter. @@ -58,6 +63,12 @@ httpsEverywhere.toolbarButton = { var tb = httpsEverywhere.toolbarButton; + // Now that HTTPSE has been initialized, start the ruleset updater if enabled. + var rulesetUpdaterEnabled = Services.prefs.getBoolPref(tb.RULESET_UPDATER_PREF); + if (rulesetUpdaterEnabled) { + HTTPSEverywhere.start_ruleset_updater(); + } + // make sure the checkbox for showing counter is properly set var showCounter = tb.shouldShowCounter(); var counterItem = document.getElementById('https-everywhere-counter-item'); @@ -71,6 +82,10 @@ httpsEverywhere.toolbarButton = { toolbarbutton.setAttribute('http_nowhere', showHttpNowhere ? 'true' : 'false'); + // make sure UI for ruleset updater toggle is properly set + var rulesetUpdaterItem = document.getElementById('ruleset-updater-item'); + rulesetUpdaterItem.setAttribute('checked', rulesetUpdaterEnabled.toString()); + // make sure UI is set depending on whether HTTPS-E is enabled toggleEnabledUI(); @@ -228,7 +243,6 @@ httpsEverywhere.toolbarButton = { function https_everywhere_load() { window.removeEventListener('load', https_everywhere_load, true); - // on first run, put the context menu in the addons bar try { var first_run; try { @@ -306,6 +320,16 @@ function show_applicable_list(menupopup) { } +function toggleRulesetUpdater() { + var rulesetUpdaterItem = document.getElementById('ruleset-updater-item'); + var checked = rulesetUpdaterItem.getAttribute('checked'); + if (checked === 'true' || checked === true) { + HTTPSEverywhere.start_ruleset_updater(); + } else { + HTTPSEverywhere.cancel_ruleset_updater(); + } +} + function toggle_rule(rule_id) { // toggle the rule state HTTPSEverywhere.https_rules.rulesetsByID[rule_id].toggle(); diff --git a/src/chrome/content/toolbar_button.xul b/src/chrome/content/toolbar_button.xul index a4fe612acf5c..efc1915c0d0e 100644 --- a/src/chrome/content/toolbar_button.xul +++ b/src/chrome/content/toolbar_button.xul @@ -46,6 +46,9 @@ + + diff --git a/src/components/https-everywhere.js b/src/components/https-everywhere.js index 513c14155a94..ca946ee86d28 100644 --- a/src/components/https-everywhere.js +++ b/src/components/https-everywhere.js @@ -33,6 +33,8 @@ const Cr = Components.results; Cu.import("resource://gre/modules/Services.jsm"); Cu.import("resource://gre/modules/FileUtils.jsm"); +Cu.import("resource://gre/modules/osfile.jsm"); +Cu.import("resource://gre/modules/AddonManager.jsm"); const CP_SHOULDPROCESS = 4; @@ -42,8 +44,14 @@ const SERVICE_NAME = "Encrypts your communications with a number of major websit const LLVAR = "LogLevel"; +const MIN_REATTEMPT_REQ_INTERVAL = 300000; +const RULESET_FETCH_INTERVAL_PREF = "ruleset_updater.interval"; +const UPDATED_RULESET_DBFILE_PATH = OS.Path.join(OS.Constants.Path.profileDir, + "HTTPSEverywhereRulesetUpdates", + "rulesets.sqlite"); + const IOS = CC["@mozilla.org/network/io-service;1"].getService(CI.nsIIOService); -const OS = CC['@mozilla.org/observer-service;1'].getService(CI.nsIObserverService); +const ObsServ = CC['@mozilla.org/observer-service;1'].getService(CI.nsIObserverService); const LOADER = CC["@mozilla.org/moz/jssubscript-loader;1"].getService(CI.mozIJSSubScriptLoader); const _INCLUDED = {}; @@ -142,7 +150,7 @@ function xpcom_checkInterfaces(iid,iids,ex) { throw ex; } -INCLUDE('ChannelReplacement', 'IOUtil', 'HTTPSRules', 'HTTPS', 'Thread', 'ApplicableList'); +INCLUDE('ChannelReplacement', 'IOUtil', 'HTTPSRules', 'HTTPS', 'Thread', 'ApplicableList', 'RulesetUpdater'); Components.utils.import("resource://gre/modules/XPCOMUtils.jsm"); @@ -178,7 +186,7 @@ function StorageController(command) { function HTTPSEverywhere() { // Set up logging in each component: - HTTPS.log = HTTPSRules.log = RuleWriter.log = this.log = https_everywhereLog; + HTTPS.log = HTTPSRules.log = RuleWriter.log = RulesetUpdater.log = this.log = https_everywhereLog; this.log = https_everywhereLog; this.wrappedJSObject = this; @@ -187,11 +195,20 @@ function HTTPSEverywhere() { // should probably be refactored this.INCLUDE=INCLUDE; this.ApplicableList = ApplicableList; + this.ruleset_updater = RulesetUpdater; this.browser_initialised = false; // the browser is completely loaded this.prefs = this.get_prefs(); this.rule_toggle_prefs = this.get_prefs(PREFBRANCH_RULE_TOGGLE); + this.rsupdate_fetch_timer = null; // nsITimer object fetching ruleset updates + + // Wrap useful constants in a method so that they can be accessed from any component + // with a reference to the HTTPSEverywhere object without directly exposing the variables. + this.MIN_REATTEMPT_REQ_INTERVAL = function() { return MIN_REATTEMPT_REQ_INTERVAL; }; + this.RULESET_UPDATE_CHECK_INTERVAL = function() { return this.prefs.getIntPref(RULESET_FETCH_INTERVAL_PREF); }; + this.UPDATED_RULESET_DBFILE_PATH = function() { return UPDATED_RULESET_DBFILE_PATH; }; + this.httpNowhereEnabled = this.prefs.getBoolPref("http_nowhere.enabled"); // We need to use observers instead of categories for FF3.0 for these: @@ -581,10 +598,10 @@ HTTPSEverywhere.prototype = { this.log(DBUG, "Got profile-after-change"); if(this.prefs.getBoolPref("globalEnabled")){ - OS.addObserver(this, "cookie-changed", false); - OS.addObserver(this, "http-on-modify-request", false); - OS.addObserver(this, "http-on-examine-merged-response", false); - OS.addObserver(this, "http-on-examine-response", false); + ObsServ.addObserver(this, "cookie-changed", false); + ObsServ.addObserver(this, "http-on-modify-request", false); + ObsServ.addObserver(this, "http-on-examine-merged-response", false); + ObsServ.addObserver(this, "http-on-examine-response", false); var dls = CC['@mozilla.org/docloaderservice;1'] .getService(CI.nsIWebProgress); @@ -780,6 +797,69 @@ HTTPSEverywhere.prototype = { return o_branch; }, + /* Try to make an XHR to the specified URL with a given method (GET/POST/...), + * and call the onSuccess function if the request succeeds. + * The function will attempt at most maxCalls requests. + */ + try_request: function(maxCalls, method, url, onSuccess) { + var xhr = Cc['@mozilla.org/xmlextras/xmlhttprequest;1'] + .createInstance(Ci.nsIXMLHttpRequest); + xhr.open(method, url, true); + xhr.onreadystatechange = function() { + if (xhr.readyState === 4) { // Complete + if (xhr.status === 200) { // OK + onSuccess(xhr.responseText); + } else { + // TODO + // Include code to ping a verification-error-reporting URL + if (maxCalls > 0) { + var timePadding = (1000000 * Math.random()) % 300000; + var timer = CC["@mozilla.org/timer;1"].createInstance(CI.nsITimer); + timer.initWithCallback( + function() { this.try_request(maxCalls - 1, method, url, onSuccess); }, + MIN_REATTEMPT_REQ_INTERVAL + timePadding, + timer.TYPE_ONE_SHOT); + } + } + } + }; + xhr.send(); + }, + + /* Start the ruleset updater up. Makes the first update retrieval call + * and starts the interval-based repeating update tests. + */ + start_ruleset_updater: function() { + if (this.rsupdate_fetch_timer !== null) { + this.log(INFO, 'Cannot restart ruleset updater because it is already running'); + return false; // False -> Already running + } + this.rsupdate_fetch_timer = CC["@mozilla.org/timer;1"].createInstance(CI.nsITimer); + this.rsupdate_fetch_timer.initWithCallback( + function() { + this.log(INFO, 'Retrieving ruleset update information'); + this.ruleset_updater.fetch_update(); + }, + this.prefs.getIntPref(RULESET_FETCH_INTERVAL_PREF), + this.rsupdate_fetch_timer.TYPE_REPEATING_SLACK); + this.ruleset_updater.fetch_update(); + this.log(INFO, 'First ruleset update retrieval started'); + return true; // True -> Started successfully + }, + + /* Stop the ruleset update fetching mechanism if it's running. + */ + cancel_ruleset_updater: function() { + if (this.rsupdate_fetch_timer === null) { + this.log(INFO, 'Ruleset updater could not be cancelled because it is not running'); + return false; // False -> Not running + } + this.rsupdate_fetch_timer.cancel(); + this.rsupdate_fetch_timer = null; + this.log(INFO, 'Ruleset updater was cancelled successfully'); + return true; // True -> Cancelled successfully + }, + chrome_opener: function(uri, args) { // we don't use window.open, because we need to work around TorButton's // state control @@ -806,10 +886,10 @@ HTTPSEverywhere.prototype = { this.obsService.removeObserver(this, "profile-before-change"); this.obsService.removeObserver(this, "profile-after-change"); this.obsService.removeObserver(this, "sessionstore-windows-restored"); - OS.removeObserver(this, "cookie-changed"); - OS.removeObserver(this, "http-on-modify-request"); - OS.removeObserver(this, "http-on-examine-merged-response"); - OS.removeObserver(this, "http-on-examine-response"); + ObsServ.removeObserver(this, "cookie-changed"); + ObsServ.removeObserver(this, "http-on-modify-request"); + ObsServ.removeObserver(this, "http-on-examine-merged-response"); + ObsServ.removeObserver(this, "http-on-examine-response"); var catman = CC["@mozilla.org/categorymanager;1"] .getService(CI.nsICategoryManager); @@ -829,10 +909,10 @@ HTTPSEverywhere.prototype = { this.obsService.addObserver(this, "profile-before-change", false); this.obsService.addObserver(this, "profile-after-change", false); this.obsService.addObserver(this, "sessionstore-windows-restored", false); - OS.addObserver(this, "cookie-changed", false); - OS.addObserver(this, "http-on-modify-request", false); - OS.addObserver(this, "http-on-examine-merged-response", false); - OS.addObserver(this, "http-on-examine-response", false); + ObsServ.addObserver(this, "cookie-changed", false); + ObsServ.addObserver(this, "http-on-modify-request", false); + ObsServ.addObserver(this, "http-on-examine-merged-response", false); + ObsServ.addObserver(this, "http-on-examine-response", false); var dls = CC['@mozilla.org/docloaderservice;1'] .getService(CI.nsIWebProgress); @@ -922,5 +1002,4 @@ if (XPCOMUtils.generateNSGetFactory) var NSGetFactory = XPCOMUtils.generateNSGetFactory([HTTPSEverywhere]); else var NSGetModule = XPCOMUtils.generateNSGetModule([HTTPSEverywhere]); - /* vim: set tabstop=4 expandtab: */ diff --git a/src/defaults/preferences/preferences.js b/src/defaults/preferences/preferences.js index b75634e2845d..09644da02b90 100644 --- a/src/defaults/preferences/preferences.js +++ b/src/defaults/preferences/preferences.js @@ -15,6 +15,17 @@ pref("extensions.https_everywhere.performance_tests", false); // enable rulesets that trigger mixed content blocking pref("extensions.https_everywhere.enable_mixed_rulesets", false); +// Preferences required by the ruleset updater +pref("extensions.https_everywhere.branch_name", "development"); +pref("extensions.https_everywhere.release_version", "5.0.0"); +pref("extensions.https_everywhere.ruleset_version", "5.0.0.0"); +// TODO +// Replace testing URLs with real ones +pref("extensions.https_everywhere.ruleset_update_url", ""); +pref("extensions.https_everywhere.ruleset_update_signature_url", ""); +pref("extensions.https_everywhere.ruleset_updater.enabled", true); +pref("extensions.https_everywhere.ruleset_updater.interval", 10800000); // Milliseconds + // HTTP Nowhere preferences pref("extensions.https_everywhere.http_nowhere.enabled", false); pref("extensions.https_everywhere.http_nowhere.orig.ocsp.required", false); diff --git a/utils/ruleset_update_manifest.py b/utils/ruleset_update_manifest.py new file mode 100644 index 000000000000..a800a113ff94 --- /dev/null +++ b/utils/ruleset_update_manifest.py @@ -0,0 +1,119 @@ +#! /usr/bin/env python + +"""Creates a ruleset update JSON manifest file according to the format +specified in https-everywhere/doc/updateJSONSpec.md +Commentary can be read on the HTTPS Everywhere mailing list: +https://lists.eff.org/pipermail/https-everywhere/2014-May/002069.html +""" + +import hashlib +import time +import json +import sys +import re + +# The lowest version of Python expected to be supported +MIN_PYTHON_VER = (2, 6) +PYTHON_VERSION_3 = (3, 0) + +# The time format for the date field +TIME_FORMAT = "%d-%m-%Y" # dd-mm-yyyy + +# A dictionary mapping the names of hash functions supported by both +# Python's hashlib and the Gecko XPCOM interface nsICryptoHash. +HASH_FUNCTIONS = { + "md5" : hashlib.md5, + "sha1" : hashlib.sha1, + "sha256" : hashlib.sha256, + "sha384" : hashlib.sha384, + "sha512" : hashlib.sha512 +} + +# Dictionary of field names for the JSON object to build that need to be +# supplied by a human. Maps the field name to a short description. +UPDATE_FIELDS = { + "branch": "the ruleset branch", + "changes": "a short description of recent changes", + "version": "a subversion of the target extension", + "source": "a valid eff.org URL pointing to the database file", + "hashfn": "the hash function to use (md5/sha1/sha256/sha384/sha512)" +} + +# Python 3's `input` returns a string the way python 2's raw_input does. +# Type checking will be done at each step regardless, so we will just use the input function +# to collect the input as a string and try to convert it. +if sys.version_info >= MIN_PYTHON_VER and sys.version_info < PYTHON_VERSION_3: + input = raw_input +elif sys.version_info < MIN_PYTHON_VER: + raise 'Versions of python older than %s are not supported.' %('.'.join(MIN_PYTHON_VER)) + +def hex_str(data): + """ Convert data into a hex string """ + pad = lambda c: ('0' + c) if len(str(c)) == 1 else c + return ''.join([pad(hex(ord(char))[2:]) for char in data]) + +def formatted_time(): + """ Return the date in a nice, human-readable format """ + return time.strftime(TIME_FORMAT, time.gmtime()) + +def valid_eff_url(updateObj): + """ Make sure the supplied source points to an eff.org URL """ + # This regex works the same way in Javascript. + regex = "^https:\/\/(www\.)?eff\.org\/[\w\-_\/]+\.sqlite$" + matched = re.match(regex, updateObj['source']) is not None + if not matched: + print("### Source is not a valid eff.org URL") + return matched + +def valid_branch_name(updateObj): + """ Test to make sure the branch name is recognized """ + valid_branches = ["development", "stable"] + matched = any([updateObj['branch'] == branch for branch in valid_branches]) + if not matched: + print("### Branch is neither of " + '/'.join(valid_branches)) + return matched + +def valid_hash_function(updateObj): + """ Test to make sure the hash function is supported """ + matched = updateObj['hashfn'] in HASH_FUNCTIONS.keys() + if not matched: + print("### " + updateObj['hashfn'] + " is not a supported hash function.") + return matched + +# A list of sanity-testing functions used to verify that the values provided to +# build the update object with make sense. +SANITY_CHECKS = [valid_eff_url, valid_branch_name, valid_hash_function] + +update = {} +print("Please supply the necessary fields to build update.json") +for field in UPDATE_FIELDS.keys(): + update[field] = input(field + ', ' + UPDATE_FIELDS[field] + ": ") +update['date'] = formatted_time() +update['hash'] = None +done = not all(map(lambda test: test(update), SANITY_CHECKS)) +if done: # Any test failed + print("Since one or more of the sanity checks failed, the JSON data will not be written.") + sys.exit(1) +hash_fn = HASH_FUNCTIONS[update['hashfn']] +while update['hash'] is None: + dbfile_path = input("Enter the path to the database file on disk: ") + try: + hashed_data = hash_fn(open(dbfile_path, 'r').read()).digest() + update['hash'] = hex_str(hashed_data) + except IOError: + print("Could not compute the hash of the contents of " + dbfile_path) + update['hash'] = None +while not done: + file_name = input("Where should the JSON contents be stored? ") + try: + json_data = json.dumps(update, sort_keys=True).replace(', ', '\n,') + open(file_name, 'w').write(json_data + '\n') + print("The update contents have been successfully written.") + done = True + except IOError: + print("Could not open " + file_name + " for writing.") + except TypeError: + print("Something really strange happened and the update data could not be serialized!") + print("Please get in touch with a maintainer of HTTPS-Everywhere to have the issue investigated.") + done = True # Lie, because breaking out of (otherwise infinite) while loops isn't "pythonic" +print("Exiting...") diff --git a/utils/testing/rsupdate_fetching/fileserver.sh b/utils/testing/rsupdate_fetching/fileserver.sh new file mode 100755 index 000000000000..f9bbc3b9f00e --- /dev/null +++ b/utils/testing/rsupdate_fetching/fileserver.sh @@ -0,0 +1,9 @@ +#! /usr/bin/env bash + +version=`python --version` + +if [[ $version < "Python 3" ]]; then + python -m SimpleHTTPServer +else + python -m http.server +fi