=== modified file 'MANIFEST.in' --- MANIFEST.in 2014-01-30 16:56:57 +0000 +++ MANIFEST.in 2015-05-20 14:55:53 +0000 @@ -1,5 +1,5 @@ include *.py MANIFEST.in -global-include *.txt *.rst *.json *.ini *.gpg *.pem *.service *.in *.conf *.cfg +global-include *.txt *.rst *.json *.ini *.gpg *.pem *.service *.in *.conf *.cfg *.sh prune build prune dist prune .tox === modified file 'NEWS.rst' --- NEWS.rst 2014-09-26 14:36:34 +0000 +++ NEWS.rst 2015-05-20 14:55:53 +0000 @@ -2,7 +2,78 @@ NEWS for system-image updater ============================= -2.5 (2014-XX-XX) +3.0 (2015-05-08) +================ + * Support a built-in PyCURL-based downloader in addition to the traditional + ubuntu-download-manager (over D-BUS) downloader. Auto-detects which + downloader to use based on whether udm is available on the system bus, + pycurl is importable, and the setting of the SYSTEMIMAGE_PYCURL environment + variable. Initial contribution by Michael Vogt. (LP: #1374459) + * Support alternative machine-id files as fall backs if the D-Bus file does + not exist. Specifically, add systemd's /etc/machine-id to the list. + Initial contribution by Michael Vogt. (LP: #1384859) + * Support multiple configuration files, as in a `config.d` directory. Now, + configuration files are named `NN_whatever.ini` where "NN" must be a + numeric prefix. Files are loaded in sorted numeric order, with later files + overriding newer files. Support for both the `client.ini` and + `channel.ini` files has been removed. (LP: #1373467) + * The `[system]build_file` variable has been removed. Build number + information now must come from the `.ini` files, and last update date + comes from the newest `.ini` file loaded. + * The `-C` command line option now takes a path to the configuration + directory. + * Reworked the checking and downloading locks/flags to so that they will work + better with configuration reloading. (LP: #1412698) + * Support for the `/etc/ubuntu-build` file has been removed. The build + number now comes from the configuration files. (LP: #1377312) + * Move the `archive-master.tar.xz` file to `/usr/share/system-image` for + better FHS compliance. (LP: #1377184) + * Since devices do not always reboot to apply changes, the `[hooks]update` + variable has been renamed to `[hooks]apply`. (LP: #1381538) + * For testing purposes only, `system-image-cli` now supports an + undocumented command line switch `--skip-gpg-verification`. Originally + given by Jani Monoses. (LP: #1333414) + * A new D-Bus signal `Applied(bool)` is added, which is returned in + response to the `ApplyUpdate()` asynchronous method call. For devices + which do not need to reboot in order to apply the update, this is the only + signal you will get. If your device needs to reboot you will also receive + the `Rebooting(bool)` command as with earlier versions. The semantics of + the flag argument are the same in both cases, as are the race timing issues + inherent in these signals. See the `system-image-dbus(8)` manpage for + details. (LP: #1417176) + * As part of LP: #1417176, the `--no-reboot` switch for + `system-image-cli(1)` has been deprecated. Use `--no-apply` instead + (`-g` is still the shortcut). + * Support production factory resets. `system-image-cli --production-reset` + and a new D-Bus API method `ProductionReset()` are added. Given by Ricardo + Salveti. (LP: #1419027) + * A new key, `target_version_detail` has been added to the dictionary + returned by the `.Information()` D-Bus method. (LP: #1399687) + * The `User-Agent` HTTP header now also includes device and channel names. + (LP: #1387719) + * Added `--progress` flag to `system-image-cli` for specifying methods for + reporting progress. Current available values are: `dots` (compatible with + system-image 2.5), `logfile` (compatible with system-image 2.5's + `--verbose` flag), and `json` for JSON records on stdout. (LP: #1423622) + * Support for the `SYSTEMIMAGE_DBUS_DAEMON_HUP_SLEEP_SECONDS` environment + variable has been removed. + * Fix `system-image-cli --list-channels`. (LP: #1448153) + +2.5.1 (2014-10-21) +================== + * Make phased upgrade percentage calculation idempotent for each tuple of + (channel, target-build-number, machine-id). Also, modify the candidate + upgrade path selection process such that if the lowest scored candidate + path has a phased percentage greater than the device's percentage, the + candidate will be ignored, and the next lowest scored candidate will be + checked until either a winner is found or no candidates are left, in which + case the device is deemed to be up-to-date. (LP: #1383539) + * `system-image-cli -p/--percentage` is added to allow command line override + of the device's phased percentage. + * `system-image-cli --dry-run` now also displays the phase percentage of the + winning candidate upgrade path. + +2.5 (2014-09-29) ================ * Remove the previously deprecated `system-image-cli --dbus` command line switch. (LP: #1369717) === modified file 'PKG-INFO' --- PKG-INFO 2014-09-26 14:36:34 +0000 +++ PKG-INFO 2015-05-20 14:55:53 +0000 @@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: system-image -Version: 2.5 +Version: 3.0 Summary: Ubuntu System Image Based Upgrades Home-page: UNKNOWN Author: Barry Warsaw === modified file 'cli-manpage.rst' --- cli-manpage.rst 2014-09-17 13:41:31 +0000 +++ cli-manpage.rst 2015-05-20 14:55:53 +0000 @@ -7,9 +7,9 @@ ------------------------------------------------ :Author: Barry Warsaw -:Date: 2014-09-16 -:Copyright: 2013-2014 Canonical Ltd. -:Version: 2.4 +:Date: 2015-01-15 +:Copyright: 2013-2015 Canonical Ltd. +:Version: 3.0 :Manual section: 1 @@ -68,10 +68,17 @@ -n, --dry-run Calculate and print the upgrade path, but do not download or apply it. - ---no-reboot - Downloads all files and prepares for a reboot into recovery, but doesn't - actually issue the reboot. + *New in system-image 2.5.1: output displays the target phase percentage* + +-p VALUE, --percentage VALUE + For testing purposes, force a device specific phase percentage. The value + must be an integer between 0 and 100. *New in system-image 2.5.1* + +-g, --no-apply + Downloads all files and prepares for, but does not actually apply the + update. On devices which require a reboot to apply the update, no reboot + is performed. *New in system-image 3.0: --no-reboot is renamed to + --no-apply* -v, --verbose Increase the logging verbosity. With one ``-v``, logging goes to the @@ -79,10 +86,13 @@ enabled. With two ``-v`` (or ``-vv``), logging both to the console and to the log file are output at ``DEBUG`` level. --C FILE, --config FILE - Use the given configuration file, otherwise use the default. The program - will optionally also read a ``channel.ini`` file in the same directory as - ``FILE``. +-C DIR, --config DIR + Use the given configuration directory, otherwise use the system default. + The program will read all the files in this directory that begin with a + number, followed by an underscore, and ending in ``.ini`` + (e.g. ``03_myconfig.ini``). The files are read in sorted numerical order + from lowest prefix number to highest, with later configuration files able + to override any variable in any section. --factory-reset Wipes the data partition and issues a reboot into recovery. This @@ -91,6 +101,13 @@ --show-settings Show all the key/value pairs in the settings database. +--progress [dots|logfile|json] + Report progress in various ways. `dots` prints some dots every once in a + while to stderr; this mimic what was available in system-image 2.5. + `logfile` prints messages at debug level to the system-image log file, and + is also available in 2.5 (via the `--verbose` flag). `json` prints JSON + records to stdout. *New in system-image 3.0* + --get KEY Print the value for the given key in the settings database. If the key is missing, a default value is printed. May be given multiple times. @@ -107,15 +124,11 @@ FILES ===== -/etc/system-image/client.ini - Default configuration file. - -/etc/system-image/channel.ini - Optional configuration file overrides (for the ``[service]`` section - only). +/etc/system-image/[0-9]+*.ini + Default configuration files. SEE ALSO ======== -client.ini(5), system-image-dbus(8) +system-image.ini(5), system-image-dbus(8) === added file 'coverage-curl.ini' --- coverage-curl.ini 1970-01-01 00:00:00 +0000 +++ coverage-curl.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,21 @@ +[run] +branch = true +parallel = true +omit = + setup* + systemimage/data/* + systemimage/docs/* + systemimage/testing/* + systemimage/tests/* + systemimage/udm.py + /usr/lib/* + .tox/coverage-curl/lib/python3.4/distutils/* + .tox/coverage-curl/lib/python3.4/site-packages/pkg_resources* + .tox/coverage-udm/lib/python3.4/distutils/* + .tox/coverage-udm/lib/python3.4/site-packages/pkg_resources* + +[paths] +source = + systemimage + .tox/coverage-curl/lib/python*/site-packages/systemimage + .tox/coverage-udm/lib/python*/site-packages/systemimage === added file 'coverage-udm.ini' --- coverage-udm.ini 1970-01-01 00:00:00 +0000 +++ coverage-udm.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,21 @@ +[run] +branch = true +parallel = true +omit = + setup* + systemimage/data/* + systemimage/docs/* + systemimage/testing/* + systemimage/tests/* + systemimage/curl.py + /usr/lib/* + .tox/coverage-curl/lib/python3.4/distutils/* + .tox/coverage-curl/lib/python3.4/site-packages/pkg_resources* + .tox/coverage-udm/lib/python3.4/distutils/* + .tox/coverage-udm/lib/python3.4/site-packages/pkg_resources* + +[paths] +source = + systemimage + .tox/coverage-curl/lib/python*/site-packages/systemimage + .tox/coverage-udm/lib/python*/site-packages/systemimage === removed file 'coverage.ini' --- coverage.ini 2014-09-17 02:58:58 +0000 +++ coverage.ini 1970-01-01 00:00:00 +0000 @@ -1,15 +0,0 @@ -[run] -branch = true -parallel = true -omit = - setup* - systemimage/data/* - systemimage/docs/* - systemimage/testing/* - systemimage/tests/* - /usr/lib/* - -[paths] -source = - systemimage - .tox/coverage/lib/python*/site-packages/systemimage === modified file 'dbus-manpage.rst' --- dbus-manpage.rst 2014-09-26 14:36:34 +0000 +++ dbus-manpage.rst 2015-05-20 14:55:53 +0000 @@ -7,9 +7,9 @@ ----------------------------------------- :Author: Barry Warsaw -:Date: 2014-07-15 -:Copyright: 2013-2014 Canonical Ltd. -:Version: 2.3 +:Date: 2015-01-15 +:Copyright: 2013-2015 Canonical Ltd. +:Version: 3.0 :Manual section: 8 @@ -42,10 +42,13 @@ enabled. With two ``-v`` (or ``-vv``), logging both to the console and to the log file are output at ``DEBUG`` level. --C FILE, --config FILE - Use the given configuration file, otherwise use the default. The program - will optionally also read a ``channel.ini`` file in the same directory as - ``FILE``. +-C DIR, --config DIR + Use the given configuration directory, otherwise use the system default. + The program will read all the files in this directory that begin with a + number, followed by an underscore, and ending in ``.ini`` + (e.g. ``03_myconfig.ini``). The files are read in sorted numerical order + from lowest prefix number to highest, with later configuration files able + to override any variable in any section. D-BUS API @@ -94,11 +97,12 @@ ``ApplyUpdate()`` This is an **asynchronous** call used to apply a previously downloaded - update and initiate a reboot to apply the update. It is a no-op if no new - update has been downloaded. Just before the device reboots, a - ``Rebooting`` signal is sent, although the timing of this signal both - being sent and received depends on how quickly the device is shut down for - reboot. + update. After the update has been applied, an ``Applied`` signal is + sent. Some devices require a reboot in order to apply the update, and + such devices may also issue a ``Rebooting`` signal. However, on devices + which require a reboot, the timing and emission of both the ``Applied`` + and ``Rebooting`` signals are in a race condition with system shutdown, + and may not occur. ``CancelUpdate()`` This is a **synchronous** call to cancel any update check or download in @@ -143,12 +147,21 @@ * *version_detail* - A string containing a comma-separated list of key-value pairs providing additional component version details, e.g. "ubuntu=123,mako=456,custom=789". + * *target_version_detail* - Like *version_detail* but contains the + information from the server. If an update is known to be available, + this will be taken from ``index.json`` file's image specification, for + the image that the upgrade will leave the device at. If no update is + available this will be identical to *version_detail*. If no + `CheckForUpdate()` as been previously performed, then the + *target_version_detail* will be the empty string. * *last_check_date* - The last time a ``CheckForUpdate()`` call was performed. *New in system-image 2.3* - *New in system-image 2.5: target_build_number* + *New in system-image 2.5: target_build_number was added.* + + *New in system-image 3.0: target_version_detail was added.* ``FactoryReset()`` This is a **synchronous** call which wipes the data partition and issue a @@ -157,6 +170,13 @@ *New in system-image 2.3*. +``ProductionReset()`` + This is a **synchronous** call which wipes the data partition, sets a flag + for factory wipe (used in production), and issue a reboot to recovery. + A ``Rebooting`` signal may be sent, depending on timing. + + *New in system-image 3.0*. + ``SetSetting(key, value)`` This is a **synchronous** call to write or update a setting. ``key`` and ``value`` are strings. While any key/value pair may be set, some keys @@ -265,15 +285,20 @@ * **last_reason** - A string containing the reason for why this updated failed. +``Applied(status)`` + Sent in response to an ``ApplyUpdate()`` call. See the timing caveats for + that method. **New in system-image 3.0** + + * **status** - A boolean indicating whether an update has been applied or + not. + ``Rebooting(status)`` - Sent just before the device reboots. Because the system is in the process - of being rebooted, clients may or may not receive this signal. + On devices which require a reboot in order to apply an update, this signal + may be sent in response to an ``ApplyUpdate()`` call. See the timing + caveats for that method. - * **status** - A boolean indicating whether the application of the update - is successful or not. Generally, when status is true you won't ever - receive the signal because the device will be rebooting. When status is - false it means the application of the update or reboot failed for some - reason. + * **status** - A boolean indicating whether the device has initiated a + reboot sequence or not. ``SettingChanged(key, value)`` Sent when a setting is changed. This signal is not sent if the new value @@ -286,7 +311,7 @@ Additional API details ---------------------- -The ``SetSettings()`` call takes a key string and a value string. The +The ``SetSetting()`` call takes a key string and a value string. The following keys are predefined. * *min_battery* - The minimum battery strength which will allow downloads @@ -311,12 +336,8 @@ FILES ===== -/etc/system-image/client.ini - Default configuration file. - -/etc/system-image/channel.ini - Optional configuration file overrides (for the ``[service]`` section - only). +/etc/system-image/[0-9]+*.ini + Default configuration files. /etc/dbus-1/system.d/com.canonical.SystemImage.conf DBus service permissions file. @@ -328,6 +349,7 @@ SEE ALSO ======== -client.ini(5), system-image-cli(1) +system-image.ini(5), system-image-cli(1) + .. _`ISO 8601`: http://en.wikipedia.org/wiki/ISO_8601 === modified file 'debian/changelog' --- debian/changelog 2014-09-29 19:02:48 +0000 +++ debian/changelog 2015-05-20 14:55:53 +0000 @@ -1,3 +1,80 @@ +system-image (3.0-0ubuntu2) UNRELEASED; urgency=medium + + * New upstream release. + - LP: #1374459 - Support a built-in PyCURL-based downloader in + addition to the traditional ubuntu-download-manager (over D-BUS) + downloader. Auto-detects which downloader to use based on whether + udm is available on the system bus, pycurl is importable, and the + setting of the SYSTEMIMAGE_PYCURL environment variable. Initial + contribution by Michael Vogt. + - LP: #1384859 - Support alternative machine-id files as fall backs if + the D-Bus file does not exist. Specifically, add systemd's + /etc/machine-id to the list. Initial contribution by Michael Vogt. + - LP: #1373467 - Support multiple configuration files, as in a + `config.d` directory. Now, configuration files are named + `NN_whatever.ini` where "NN" must be a numeric prefix. Files are + loaded in sorted numeric order, with later files overriding newer + files. Support for both the `client.ini` and `channel.ini` files has + been removed. + - The `[system]build_file` variable has been removed. Build number + information now must come from the `.ini` files, and last update + date comes from the newest `.ini` file loaded. + - The `-C` command line option now takes a path to the configuration + directory. + - LP: #1412698 - Reworked the checking and downloading locks/flags to + so that they will work better with configuration reloading. + - LP: #1377312 - Support for the `/etc/ubuntu-build` file has been + removed. The build number now comes from the configuration files. + - LP: #1377184 - Move the `archive-master.tar.xz` file to + `/usr/share/system-image` for better FHS compliance. + - LP: #1381538 - Since devices do not always reboot to apply changes, + the `[hooks]update` variable has been renamed to `[hooks]apply`. + - LP: #1333414 - For testing purposes only, `system-image-cli` now + supports an undocumented command line switch + `--skip-gpg-verification`. Originally given by Jani Monoses. + - LP: #1417176 - A new D-Bus signal `Applied(bool)` is added, which is + returned in response to the `ApplyUpdate()` asynchronous method + call. For devices which do not need to reboot in order to apply the + update, this is the only signal you will get. If your device needs + to reboot you will also receive the `Rebooting(bool)` command as + with earlier versions. The semantics of the flag argument are the + same in both cases, as are the race timing issues inherent in these + signals. See the `system-image-dbus(8)` manpage for details. + - As part of LP: #1417176, the `--no-reboot` switch for + `system-image-cli(1)` has been deprecated. Use `--no-apply` instead + (`-g` is still the shortcut). + - LP: #1419027 - Support production factory resets. `system-image-cli + --production-reset` and a new D-Bus API method `ProductionReset()` + are added. Given by Ricardo Salveti. + - LP: #1399687 - A new key, `target_version_detail` has been added to + the dictionary returned by the `.Information()` D-Bus method. + - LP: #1387719 - The `User-Agent` HTTP header now also includes device + and channel names. + - LP: #1423622 - Added `--progress` flag to `system-image-cli` for + specifying methods for reporting progress. Current available values + are: `dots` (compatible with system-image 2.5), `logfile` + (compatible with system-image 2.5's `--verbose` flag), and `json` + for JSON records on stdout. + - Support for the `SYSTEMIMAGE_DBUS_DAEMON_HUP_SLEEP_SECONDS` environment + variable has been removed. + - LP: #1448153 - Fix `system-image-cli --list-channels`. + * d/rules: + - Run both the cURL and UDM based tests. + - Run tests with more verbosity. + - Install the archive-master keyring files to /usr/share instead of + /etc for better FHS compliance. (LP: #1377184) + * d/control: + - Add python3-pycurl to Build-Depends. + - Bump Standards-Version to 3.9.6 with no other changes necessary. + - system-image-common now depends on + `ubuntu-download-manager | python3-pycurl` so that UDM doesn't need to + be pulled in for snappy. (LP: #1431696) + * d/tests/control: Disable DEP-8 "smoketests" which try to access + external resources. This is now prohibited by policy for + pocket-promotion tests. (LP: #1457070) + + -- Barry Warsaw Wed, 20 May 2015 10:46:17 -0400 + system-image (2.5-0ubuntu1) utopic; urgency=medium [ Barry Warsaw ] === modified file 'debian/control' --- debian/control 2014-09-09 17:27:17 +0000 +++ debian/control 2015-05-20 14:55:53 +0000 @@ -16,10 +16,11 @@ python3-nose2, python3-pkg-resources, python3-psutil, + python3-pycurl, python3-setuptools, python3-xdg, ubuntu-download-manager -Standards-Version: 3.9.5 +Standards-Version: 3.9.6 XS-Testsuite: autopkgtest Vcs-Bzr: https://code.launchpad.net/~ubuntu-managed-branches/ubuntu-system-image/system-image Vcs-Browser: http://bazaar.launchpad.net/~ubuntu-managed-branches/ubuntu-system-image/system-image/files @@ -48,7 +49,7 @@ python3-gnupg, python3-pkg-resources, python3-xdg, - ubuntu-download-manager, + ubuntu-download-manager | python3-pycurl, ${misc:Depends}, ${python3:Depends} Description: Ubuntu system image updater === modified file 'debian/rules' --- debian/rules 2014-09-09 17:27:17 +0000 +++ debian/rules 2015-05-20 14:55:53 +0000 @@ -9,11 +9,18 @@ dh $@ --with python3 --buildsystem=pybuild override_dh_auto_test: - unset http_proxy; unset https_proxy; \ - export SYSTEMIMAGE_REACTOR_TIMEOUT=1200; \ - export SYSTEMIMAGE_DBUS_DAEMON_HUP_SLEEP_SECONDS=2; \ - PYBUILD_SYSTEM=custom \ - PYBUILD_TEST_ARGS="{interpreter} -m nose2 -v" dh_auto_test + export http_proxy= ; \ + export https_proxy= ; \ + export SYSTEMIMAGE_REACTOR_TIMEOUT=1200; \ + export SYSTEMIMAGE_PYCURL=1; \ + PYBUILD_SYSTEM=custom \ + PYBUILD_TEST_ARGS="{interpreter} -m nose2 -vv" dh_auto_test + export http_proxy= ; \ + export https_proxy= ; \ + export SYSTEMIMAGE_REACTOR_TIMEOUT=1200; \ + export SYSTEMIMAGE_PYCURL=0; \ + PYBUILD_SYSTEM=custom \ + PYBUILD_TEST_ARGS="{interpreter} -m nose2 -vv" dh_auto_test # pybuild can't yet handle Python 3 packages that don't start with "python3-". # See bug #751908 - In the meantime, this override isn't perfect, but it gets @@ -30,11 +37,9 @@ usr/lib/python3.?/dist-packages/systemimage/testing dh_install -p system-image-cli usr/bin/system-image-cli dh_install -p system-image-common \ - debian/archive-master.tar.xz etc/system-image - dh_install -p system-image-common \ - debian/archive-master.tar.xz.asc etc/system-image - dh_install -p system-image-common \ - systemimage/data/client.ini etc/system-image + debian/archive-master.tar.xz usr/share/system-image + dh_install -p system-image-common \ + debian/archive-master.tar.xz.asc usr/share/system-image dh_install -p system-image-dbus usr/bin/system-image-dbus usr/sbin dh_install -p system-image-dbus \ systemimage/data/com.canonical.SystemImage.service \ === renamed file 'debian/tests/client.ini.in' => 'debian/tests/00_default.ini.in' === modified file 'debian/tests/control' --- debian/tests/control 2014-07-23 22:51:19 +0000 +++ debian/tests/control 2015-05-20 14:55:53 +0000 @@ -1,11 +1,11 @@ -Tests: smoketest -Restrictions: isolation-container -Depends: system-image-common, system-image-cli, system-image-dbus, ubuntu-download-manager, dbus, dbus-x11, python3-psutil, python3-xdg +#Tests: smoketest +#Restrictions: isolation-container +#Depends: system-image-common, system-image-cli, system-image-dbus, ubuntu-download-manager, dbus, dbus-x11, python3-psutil, python3-xdg, python3-pycurl -Tests: smoketest-noreboot -Restrictions: isolation-container allow-stderr -Depends: system-image-common, system-image-cli, system-image-dbus, ubuntu-download-manager, dbus, dbus-x11, python3-psutil, python3-xdg +#Tests: smoketest-noreboot +#Restrictions: isolation-container allow-stderr +#Depends: system-image-common, system-image-cli, system-image-dbus, ubuntu-download-manager, dbus, dbus-x11, python3-psutil, python3-xdg, python3-pycurl Tests: dryrun Restrictions: allow-stderr -Depends: @, ubuntu-download-manager, dbus, dbus-x11, python3-psutil, python3-xdg, python3-setuptools, python3-nose2 +Depends: @, ubuntu-download-manager, dbus, dbus-x11, python3-psutil, python3-xdg, python3-setuptools, python3-nose2, python3-pycurl === modified file 'debian/tests/dryrun' --- debian/tests/dryrun 2014-07-18 16:32:44 +0000 +++ debian/tests/dryrun 2015-05-20 14:55:53 +0000 @@ -5,7 +5,7 @@ # require network access, so it is compatible with less isolated (but also # lighter weight) containers such as schroot. # -# Copyright (C) 2014 Canonical Ltd. +# Copyright (C) 2014-2015 Canonical Ltd. # Author: Barry Warsaw python3 -m nose2 -vv -P TestCLIMainDryRun === modified file 'debian/tests/prep.py' --- debian/tests/prep.py 2013-12-13 13:55:51 +0000 +++ debian/tests/prep.py 2015-05-20 14:55:53 +0000 @@ -1,5 +1,8 @@ #!/usr/bin/python3 +# Copyright (C) 2013-2015 Canonical Ltd. +# Author: Barry Warsaw + import os tmpdir = os.environ['ADTTMP'] @@ -8,15 +11,19 @@ os.makedirs(os.path.join(tmpdir, 'android'), exist_ok=True) os.makedirs(os.path.join(tmpdir, 'ubuntu'), exist_ok=True) +config_d = os.path.join(tmpdir, 'config.d') +os.makedirs(config_d, exist_ok=True) + substitutions = dict( TMPDIR=tmpdir, ARTIFACTS=artifacts, ) -with open('debian/tests/client.ini.in', encoding='utf-8') as fp: +with open('debian/tests/00_default.ini.in', encoding='utf-8') as fp: ini_template = fp.read() ini_contents = ini_template.format(**substitutions) -with open(os.path.join(tmpdir, 'client.ini'), 'w', encoding='utf-8') as fp: +default_ini = os.path.join(config_d, '00_default.ini') +with open(default_ini, 'w', encoding='utf-8') as fp: fp.write(ini_contents) === modified file 'debian/tests/smoketest' --- debian/tests/smoketest 2014-07-18 16:32:44 +0000 +++ debian/tests/smoketest 2015-05-20 14:55:53 +0000 @@ -5,9 +5,9 @@ # isolation-container restricted, so it requires an isolated test container # like QEMU. # -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw set -e python3 debian/tests/prep.py -system-image-cli -C $ADTTMP/client.ini -d mako -c devel -b 0 --dry-run +system-image-cli -C $ADTTMP/config.d -d mako -c devel -b 0 --dry-run === modified file 'debian/tests/smoketest-noreboot' --- debian/tests/smoketest-noreboot 2014-07-23 22:51:19 +0000 +++ debian/tests/smoketest-noreboot 2015-05-20 14:55:53 +0000 @@ -7,9 +7,9 @@ # # This is like smoketest except that it does a full download. # -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw set -e python3 debian/tests/prep.py -system-image-cli -C $ADTTMP/client.ini -d mako -c devel -b 0 --no-reboot -v +system-image-cli -C $ADTTMP/config.d -d mako -c devel -b 0 --no-reboot -v === modified file 'debian/tests/unittests' --- debian/tests/unittests 2013-12-13 13:55:51 +0000 +++ debian/tests/unittests 2015-05-20 14:55:53 +0000 @@ -2,7 +2,7 @@ # # autopkgtest check: Run tox against the built package. # -# Copyright (C) 2013 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw set -e === modified file 'ini-manpage.rst' --- ini-manpage.rst 2014-09-17 13:41:31 +0000 +++ ini-manpage.rst 2015-05-20 14:55:53 +0000 @@ -1,39 +1,43 @@ -========== -client.ini -========== - - ------------------------------------------------ -Ubuntu System Image Upgrader configuration file ------------------------------------------------ +================ +system-image.ini +================ + + +------------------------------------------------ +Ubuntu System Image Upgrader configuration files +------------------------------------------------ :Author: Barry Warsaw -:Date: 2014-09-11 -:Copyright: 2013-2014 Canonical Ltd. -:Version: 2.4 +:Date: 2015-01-15 +:Copyright: 2013-2015 Canonical Ltd. +:Version: 3.0 :Manual section: 5 DESCRIPTION =========== -``/etc/system-image/client.ini`` is the configuration file for the system -image upgrader. It is an ini-style configuration file with sections that -define the service to connect to, as well as local system resources. -Generally, the options never need to be changed. - -The system image upgrader will also optionally read a -``/etc/system-image/channel.ini`` file with the same format as ``client.ini``. -This file should only contain a ``[service]`` section for overriding in the -``client.ini`` file. All other sections are ignored. +``/etc/system-image/config.d`` is the default configuration directory for the +system image upgrader. It contains ini-style configuration files with +sections that define the service to connect to, as well as local system +resources. Generally, the options never need to be changed. + +The system image upgrader will read all files in this directory that start +with a numeric prefix, followed by an underscore, and then any alphanumeric +suffix, ending in ``.ini``. E.g. ``07_myconfig.ini``. + +The files are read in sorted numerical order, from lowest prefix number to +highest, with later configuration files able to override any variable in any +section. SYNTAX ====== -Sections are delimited by square brackets, e.g. ``[service]``. Variables -inside the service separate the variable name and value by a colon. Blank -lines and lines that start with a ``#`` are ignored. +Sections in the ``.ini`` files are delimited by square brackets, +e.g. ``[service]``. Variables inside the service separate the variable name +and value by a colon. Blank lines and lines that start with a ``#`` are +ignored. THE SERVICE SECTION @@ -82,10 +86,6 @@ This section contains the following variables: -build_file - The file on the local file system containing the system's current build - number. - tempdir The base temporary directory on the local file system. When any of the system-image processes run, a secure subdirectory inside `tempdir` will be @@ -181,9 +181,11 @@ The Python import path to the class implementing the upgrade scoring algorithm. -reboot - The Python import path to the class that implements the system reboot - command. +apply + The Python import path to the class that implements the mechanism for + applying the update. This often reboots the device. + + *New in system-image 3.0: ``reboot`` was renamed to ``apply``* THE DBUS SECTION @@ -204,6 +206,7 @@ system-image-cli(1) + [1]: https://wiki.ubuntu.com/ImageBasedUpgrades/Server [2]: https://wiki.ubuntu.com/ImageBasedUpgrades/GPG === modified file 'setup.cfg' --- setup.cfg 2014-09-26 14:36:34 +0000 +++ setup.cfg 2015-05-20 14:55:53 +0000 @@ -4,7 +4,7 @@ logging-filter = systemimage [egg_info] -tag_svn_revision = 0 tag_build = tag_date = 0 +tag_svn_revision = 0 === modified file 'setup.py' --- setup.py 2014-02-20 23:03:24 +0000 +++ setup.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify === modified file 'system_image.egg-info/PKG-INFO' --- system_image.egg-info/PKG-INFO 2014-09-26 14:36:34 +0000 +++ system_image.egg-info/PKG-INFO 2015-05-20 14:55:53 +0000 @@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: system-image -Version: 2.5 +Version: 3.0 Summary: Ubuntu System Image Based Upgrades Home-page: UNKNOWN Author: Barry Warsaw === modified file 'system_image.egg-info/SOURCES.txt' --- system_image.egg-info/SOURCES.txt 2014-09-26 14:36:34 +0000 +++ system_image.egg-info/SOURCES.txt 2015-05-20 14:55:53 +0000 @@ -2,7 +2,8 @@ NEWS.rst README.rst cli-manpage.rst -coverage.ini +coverage-curl.ini +coverage-udm.ini dbus-manpage.rst ini-manpage.rst setup.cfg @@ -17,10 +18,12 @@ system_image.egg-info/top_level.txt systemimage/__init__.py systemimage/api.py +systemimage/apply.py systemimage/bag.py systemimage/candidates.py systemimage/channel.py systemimage/config.py +systemimage/curl.py systemimage/dbus.py systemimage/device.py systemimage/download.py @@ -32,14 +35,13 @@ systemimage/logging.py systemimage/main.py systemimage/reactor.py -systemimage/reboot.py systemimage/scores.py systemimage/service.py systemimage/settings.py systemimage/state.py +systemimage/udm.py systemimage/version.txt systemimage/data/__init__.py -systemimage/data/client.ini systemimage/data/com.canonical.SystemImage.conf systemimage/data/com.canonical.SystemImage.service systemimage/docs/__init__.py @@ -70,77 +72,121 @@ systemimage/tests/test_settings.py systemimage/tests/test_state.py systemimage/tests/test_winner.py +systemimage/tests/data/00.ini +systemimage/tests/data/01.ini systemimage/tests/data/__init__.py +systemimage/tests/data/api.channels_01.json +systemimage/tests/data/api.index_01.json +systemimage/tests/data/api.index_02.json +systemimage/tests/data/api.index_03.json systemimage/tests/data/archive-master.gpg systemimage/tests/data/bad_cert.pem systemimage/tests/data/bad_key.pem +systemimage/tests/data/candidates.index_01.json +systemimage/tests/data/candidates.index_02.json +systemimage/tests/data/candidates.index_03.json +systemimage/tests/data/candidates.index_04.json +systemimage/tests/data/candidates.index_05.json +systemimage/tests/data/candidates.index_06.json +systemimage/tests/data/candidates.index_07.json +systemimage/tests/data/candidates.index_08.json +systemimage/tests/data/candidates.index_09.json +systemimage/tests/data/candidates.index_10.json +systemimage/tests/data/candidates.index_11.json +systemimage/tests/data/candidates.index_12.json +systemimage/tests/data/candidates.index_13.json systemimage/tests/data/cert.pem -systemimage/tests/data/channel_01.ini -systemimage/tests/data/channel_02.ini -systemimage/tests/data/channel_03.ini -systemimage/tests/data/channel_04.ini -systemimage/tests/data/channel_05.ini -systemimage/tests/data/channel_06.ini -systemimage/tests/data/channel_07.ini -systemimage/tests/data/channels_01.json -systemimage/tests/data/channels_02.json -systemimage/tests/data/channels_03.json -systemimage/tests/data/channels_04.json -systemimage/tests/data/channels_05.json -systemimage/tests/data/channels_06.json -systemimage/tests/data/channels_07.json -systemimage/tests/data/channels_08.json -systemimage/tests/data/channels_09.json -systemimage/tests/data/channels_10.json -systemimage/tests/data/channels_11.json +systemimage/tests/data/channel.channels_01.json +systemimage/tests/data/channel.channels_02.json +systemimage/tests/data/channel.channels_03.json +systemimage/tests/data/channel.channels_04.json +systemimage/tests/data/channel.channels_05.json systemimage/tests/data/com.canonical.SystemImage.service.in systemimage/tests/data/com.canonical.applications.Downloader.service.in -systemimage/tests/data/config_00.ini -systemimage/tests/data/config_01.ini -systemimage/tests/data/config_02.ini -systemimage/tests/data/config_03.ini -systemimage/tests/data/config_04.ini -systemimage/tests/data/config_05.ini -systemimage/tests/data/config_06.ini -systemimage/tests/data/config_07.ini -systemimage/tests/data/config_08.ini -systemimage/tests/data/config_09.ini -systemimage/tests/data/config_10.ini +systemimage/tests/data/config.config_01.ini +systemimage/tests/data/config.config_02.ini +systemimage/tests/data/config.config_03.ini +systemimage/tests/data/config.config_04.ini +systemimage/tests/data/config.config_05.ini +systemimage/tests/data/config.config_06.ini +systemimage/tests/data/config.config_07.ini +systemimage/tests/data/config.config_08.ini +systemimage/tests/data/config.config_09.ini +systemimage/tests/data/config.config_10.ini +systemimage/tests/data/config.config_11.ini systemimage/tests/data/dbus-system.conf.in +systemimage/tests/data/dbus.channels_01.json +systemimage/tests/data/dbus.index_01.json +systemimage/tests/data/dbus.index_02.json +systemimage/tests/data/dbus.index_03.json +systemimage/tests/data/dbus.index_04.json +systemimage/tests/data/dbus.index_05.json +systemimage/tests/data/dbus.index_06.json systemimage/tests/data/device-signing.gpg +systemimage/tests/data/download.index_01.json systemimage/tests/data/expired_cert.pem systemimage/tests/data/expired_key.pem +systemimage/tests/data/gpg.channels_01.json +systemimage/tests/data/helpers.config_01.ini +systemimage/tests/data/helpers.config_02.ini systemimage/tests/data/image-master.gpg systemimage/tests/data/image-signing.gpg -systemimage/tests/data/index_01.json -systemimage/tests/data/index_02.json -systemimage/tests/data/index_03.json -systemimage/tests/data/index_04.json -systemimage/tests/data/index_05.json -systemimage/tests/data/index_06.json -systemimage/tests/data/index_07.json -systemimage/tests/data/index_08.json -systemimage/tests/data/index_09.json -systemimage/tests/data/index_10.json -systemimage/tests/data/index_11.json -systemimage/tests/data/index_12.json -systemimage/tests/data/index_13.json -systemimage/tests/data/index_14.json -systemimage/tests/data/index_15.json -systemimage/tests/data/index_16.json -systemimage/tests/data/index_17.json -systemimage/tests/data/index_18.json -systemimage/tests/data/index_19.json -systemimage/tests/data/index_20.json -systemimage/tests/data/index_21.json -systemimage/tests/data/index_22.json -systemimage/tests/data/index_23.json -systemimage/tests/data/index_24.json -systemimage/tests/data/index_25.json +systemimage/tests/data/index.channels_01.json +systemimage/tests/data/index.channels_02.json +systemimage/tests/data/index.channels_03.json +systemimage/tests/data/index.channels_04.json +systemimage/tests/data/index.channels_05.json +systemimage/tests/data/index.index_01.json +systemimage/tests/data/index.index_02.json +systemimage/tests/data/index.index_03.json +systemimage/tests/data/index.index_04.json +systemimage/tests/data/index.index_05.json systemimage/tests/data/key.pem +systemimage/tests/data/main.channels_01.json +systemimage/tests/data/main.channels_02.json +systemimage/tests/data/main.channels_03.json +systemimage/tests/data/main.config_01.ini +systemimage/tests/data/main.config_02.ini +systemimage/tests/data/main.config_03.ini +systemimage/tests/data/main.config_04.ini +systemimage/tests/data/main.config_05.ini +systemimage/tests/data/main.config_07.ini +systemimage/tests/data/main.index_01.json +systemimage/tests/data/main.index_02.json +systemimage/tests/data/main.index_03.json +systemimage/tests/data/main.index_04.json +systemimage/tests/data/main.index_05.json systemimage/tests/data/master-secring.gpg systemimage/tests/data/nasty_cert.pem systemimage/tests/data/nasty_key.pem +systemimage/tests/data/scores.index_01.json +systemimage/tests/data/scores.index_02.json +systemimage/tests/data/scores.index_03.json +systemimage/tests/data/scores.index_04.json +systemimage/tests/data/scores.index_05.json +systemimage/tests/data/scores.index_06.json +systemimage/tests/data/scores.index_07.json systemimage/tests/data/spare.gpg -systemimage/tests/data/sprint_nexus7_index_01.json -tools/demo.ini \ No newline at end of file +systemimage/tests/data/state.channels_01.json +systemimage/tests/data/state.channels_02.json +systemimage/tests/data/state.channels_03.json +systemimage/tests/data/state.channels_04.json +systemimage/tests/data/state.channels_05.json +systemimage/tests/data/state.channels_06.json +systemimage/tests/data/state.channels_07.json +systemimage/tests/data/state.config_01.ini +systemimage/tests/data/state.config_02.ini +systemimage/tests/data/state.index_01.json +systemimage/tests/data/state.index_02.json +systemimage/tests/data/state.index_03.json +systemimage/tests/data/state.index_04.json +systemimage/tests/data/state.index_05.json +systemimage/tests/data/state.index_06.json +systemimage/tests/data/state.index_07.json +systemimage/tests/data/state.index_08.json +systemimage/tests/data/winner.channels_01.json +systemimage/tests/data/winner.channels_02.json +systemimage/tests/data/winner.index_01.json +systemimage/tests/data/winner.index_02.json +tools/demo.ini +tools/runme.sh \ No newline at end of file === modified file 'systemimage/api.py' --- systemimage/api.py 2014-09-17 13:41:31 +0000 +++ systemimage/api.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -24,10 +24,9 @@ import logging -from systemimage.helpers import last_update_date -from systemimage.reboot import factory_reset +from systemimage.apply import factory_reset, production_reset from systemimage.state import State -from unittest.mock import patch + log = logging.getLogger('systemimage') @@ -63,8 +62,12 @@ return '' @property - def last_update_date(self): - return last_update_date() + def version_detail(self): + try: + return self._winners[-1].version_detail + except IndexError: + # No winners. + return '' class Mediator: @@ -115,13 +118,20 @@ def download(self): """Download the available update.""" # We only want callback progress during the actual download. - with patch.object(self._state.downloader, 'callback', self._callback): - self._state.run_until('reboot') + old_callbacks = self._state.downloader.callbacks[:] + try: + self._state.downloader.callbacks = [self._callback] + self._state.run_until('apply') + finally: + self._state.downloader.callbacks = old_callbacks - def reboot(self): - """Issue the reboot.""" + def apply(self): + """Apply the update.""" # Transition through all remaining states. list(self._state) def factory_reset(self): factory_reset() + + def production_reset(self): + production_reset() === renamed file 'systemimage/reboot.py' => 'systemimage/apply.py' --- systemimage/reboot.py 2014-09-17 13:41:31 +0000 +++ systemimage/apply.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -16,9 +16,11 @@ """Reboot issuer.""" __all__ = [ - 'BaseReboot', + 'BaseApply', + 'Noop', 'Reboot', 'factory_reset', + 'production_reset', ] @@ -32,24 +34,34 @@ log = logging.getLogger('systemimage') -class BaseReboot: - """Common reboot actions.""" +class BaseApply: + """Common apply-the-update actions.""" - def reboot(self): # pragma: no cover + def apply(self): # pragma: no cover """Subclasses must override this.""" raise NotImplementedError -class Reboot(BaseReboot): - """Issue a standard reboot.""" +class Reboot(BaseApply): + """Apply the update by rebooting the device.""" - def reboot(self): + def apply(self): try: check_call('/sbin/reboot -f recovery'.split(), universal_newlines=True) except CalledProcessError as error: log.exception('reboot exit status: {}'.format(error.returncode)) raise + # This code may or may not run. We're racing against the system + # reboot procedure. + config.dbus_service.Rebooting(True) + + +class Noop(BaseApply): + """No-op apply, mostly for testing.""" + + def apply(self): + pass def factory_reset(): @@ -59,4 +71,15 @@ with atomic(command_file) as fp: print('format data', file=fp) log.info('Performing a factory reset') - config.hooks.reboot().reboot() + config.hooks.apply().apply() + + +def production_reset(): + """Perform a production reset.""" + command_file = os.path.join( + config.updater.cache_partition, 'ubuntu_command') + with atomic(command_file) as fp: + print('format data', file=fp) + print('enable factory_wipe', file=fp) + log.info('Performing a production factory reset') + config.hooks.apply().apply() === modified file 'systemimage/bag.py' --- systemimage/bag.py 2014-09-17 13:41:31 +0000 +++ systemimage/bag.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -42,6 +42,12 @@ class Bag: + # NOTE: This class's methods share a namespace with the possible + # configuration variable names in the various sections. Thus no variable + # in any section can be named `update`, `keys`, or `get`. They also can't + # be named like any of the non-public methods, but that's usually not a + # problem. Ideally, we'd name the methods part of the reserved namespace, + # but it seems like a low tech debt for now. def __init__(self, *, converters=None, **kws): self._converters = make_converter(converters) self.__original__ = {} === modified file 'systemimage/candidates.py' --- systemimage/candidates.py 2014-09-17 13:41:31 +0000 +++ systemimage/candidates.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify === modified file 'systemimage/channel.py' --- systemimage/channel.py 2014-09-17 13:41:31 +0000 +++ systemimage/channel.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify === modified file 'systemimage/config.py' --- systemimage/config.py 2014-09-17 13:41:31 +0000 +++ systemimage/config.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -17,7 +17,6 @@ __all__ = [ 'Configuration', - 'DISABLED', 'config', ] @@ -27,79 +26,180 @@ from configparser import ConfigParser from contextlib import ExitStack -from pkg_resources import resource_filename +from pathlib import Path from systemimage.bag import Bag from systemimage.helpers import ( - as_loglevel, as_object, as_timedelta, makedirs, temporary_directory) - - -DISABLED = object() + NO_PORT, as_loglevel, as_object, as_port, as_stripped, as_timedelta, + makedirs, temporary_directory) + + +SECTIONS = ('service', 'system', 'gpg', 'updater', 'hooks', 'dbus') +USER_AGENT = ('Ubuntu System Image Upgrade Client: ' + 'device={0.device};channel={0.channel};build={0.build_number}') def expand_path(path): return os.path.abspath(os.path.expanduser(path)) -def port_value_converter(value): - if value.lower() in ('disabled', 'disable'): - return DISABLED - result = int(value) - if result < 0: - raise ValueError(value) - return result - - -def device_converter(value): - return value.strip() +class SafeConfigParser(ConfigParser): + """Like ConfigParser, but with default empty sections. + + This makes the **style of loading keys/values into the Bag objects a + little cleaner since it doesn't have to worry about KeyErrors when a + configuration file doesn't contain a section, which is allowed. + """ + + def __init__(self, *args, **kws): + super().__init__(args, **kws) + for section in SECTIONS: + self[section] = {} class Configuration: - def __init__(self, ini_file=None): - # Defaults. - self.config_file = None - self.service = Bag() - self.system = Bag() - if ini_file is None: - ini_file = resource_filename('systemimage.data', 'client.ini') - self.load(ini_file) - self._override = False - # 2013-10-14 BAW This is a placeholder for rendezvous between the - # downloader and the D-Bus service. When running udner D-Bus and we - # get a `paused` signal from the download manager, we need this to - # plumb through an UpdatePaused signal to our clients. It rather - # sucks that we need a global for this, but I can't get the plumbing - # to work otherwise. This seems like the least horrible place to - # stash this global. + def __init__(self, directory=None): + self._set_defaults() + # Because the configuration object is a global singleton, it makes for + # a convenient place to stash information used by widely separate + # components. For example, this is a placeholder for rendezvous + # between the downloader and the D-Bus service. When running under + # D-Bus and we get a `paused` signal from the download manager, we need + # this to plumb through an UpdatePaused signal to our clients. It + # rather sucks that we need a global for this, but I can't get the + # plumbing to work otherwise. This seems like the least horrible place + # to stash this global. self.dbus_service = None - # Cache/overrides. + # This is used to plumb command line arguments from the main() to + # other parts of the system. + self.skip_gpg_verification = False + # Cache. self._device = None self._build_number = None + self.build_number_override = False self._channel = None + # This is used only to override the phased percentage via command line + # and the property setter. + self._phase_override = None self._tempdir = None + self.config_d = None + self.ini_files = [] + self.http_base = None + self.https_base = None + if directory is not None: + self.load(directory) + self._calculate_http_bases() self._resources = ExitStack() atexit.register(self._resources.close) - def load(self, path, *, override=False): - parser = ConfigParser() - files_read = parser.read(path) - if files_read != [path]: - raise FileNotFoundError(path) - self.config_file = path - self.service.update(converters=dict(http_port=port_value_converter, - https_port=port_value_converter, + def _set_defaults(self): + self.service = Bag( + base='system-image.ubuntu.com', + http_port=80, + https_port=443, + channel='daily', + build_number=0, + ) + self.system = Bag( + timeout=as_timedelta('1h'), + tempdir='/tmp', + logfile='/var/log/system-image/client.log', + loglevel=as_loglevel('info'), + settings_db='/var/lib/system-image/settings.db', + ) + self.gpg = Bag( + archive_master='/usr/share/system-image/archive-master.tar.xz', + image_master='/var/lib/system-image/keyrings/image-master.tar.xz', + image_signing= + '/var/lib/system-image/keyrings/image-signing.tar.xz', + device_signing= + '/var/lib/system-image/keyrings/device-signing.tar.xz', + ) + self.updater = Bag( + cache_partition='/android/cache/recovery', + data_partition='/var/lib/system-image', + ) + self.hooks = Bag( + device=as_object('systemimage.device.SystemProperty'), + scorer=as_object('systemimage.scores.WeightedScorer'), + apply=as_object('systemimage.apply.Reboot'), + ) + self.dbus = Bag( + lifetime=as_timedelta('10m'), + ) + + def _load_file(self, path): + parser = SafeConfigParser() + str_path = str(path) + parser.read(str_path) + self.ini_files.append(path) + self.service.update(converters=dict(http_port=as_port, + https_port=as_port, build_number=int, - device=device_converter, + device=as_stripped, ), - **parser['service']) - if (self.service.http_port is DISABLED and - self.service.https_port is DISABLED): + **parser['service']) + self.system.update(converters=dict(timeout=as_timedelta, + loglevel=as_loglevel, + settings_db=expand_path, + tempdir=expand_path), + **parser['system']) + self.gpg.update(**parser['gpg']) + self.updater.update(**parser['updater']) + self.hooks.update(converters=dict(device=as_object, + scorer=as_object, + apply=as_object), + **parser['hooks']) + self.dbus.update(converters=dict(lifetime=as_timedelta), + **parser['dbus']) + + def load(self, directory): + """Load up the configuration from a config.d directory.""" + # Look for all the files in the given directory with .ini or .cfg + # suffixes. The files must start with a number, and the files are + # loaded in numeric order. + if self.config_d is not None: + raise RuntimeError('Configuration already loaded; use .reload()') + self.config_d = directory + if not Path(directory).is_dir(): + raise TypeError( + '.load() requires a directory: {}'.format(directory)) + candidates = [] + for child in Path(directory).glob('*.ini'): + order, _, base = child.stem.partition('_') + # XXX 2014-10-03: The logging system isn't initialized when we get + # here, so we can't log that these files are being ignored. + if len(_) == 0: + continue + try: + serial = int(order) + except ValueError: + continue + candidates.append((serial, child)) + for serial, path in sorted(candidates): + self._load_file(path) + self._calculate_http_bases() + + def reload(self): + """Reload the configuration directory.""" + # Reset some cached attributes. + directory = self.config_d + self.ini_files = [] + self.config_d = None + self._build_number = None + # Now load the defaults, then reload the previous config.d directory. + self._set_defaults() + self.load(directory) + + def _calculate_http_bases(self): + if (self.service.http_port is NO_PORT and + self.service.https_port is NO_PORT): raise ValueError('Cannot disable both http and https ports') # Construct the HTTP and HTTPS base urls, which most applications will - # actually use. We do this in two steps, in order to support - # disabling one or the other (but not both) protocols. + # actually use. We do this in two steps, in order to support disabling + # one or the other (but not both) protocols. if self.service.http_port == 80: http_base = 'http://{}'.format(self.service.base) - elif self.service.http_port is DISABLED: + elif self.service.http_port is NO_PORT: http_base = None else: http_base = 'http://{}:{}'.format( @@ -107,7 +207,7 @@ # HTTPS. if self.service.https_port == 443: https_base = 'https://{}'.format(self.service.base) - elif self.service.https_port is DISABLED: + elif self.service.https_port is NO_PORT: https_base = None else: https_base = 'https://{}:{}'.format( @@ -119,45 +219,13 @@ if https_base is None: assert http_base is not None https_base = http_base - self.service['http_base'] = http_base - self.service['https_base'] = https_base - try: - self.system.update(converters=dict(timeout=as_timedelta, - build_file=expand_path, - loglevel=as_loglevel, - settings_db=expand_path, - tempdir=expand_path), - **parser['system']) - except KeyError: - # If we're overriding via a channel.ini file, it's okay if the - # [system] section is missing. However, the main configuration - # ini file must include all sections. - if not override: - raise - # Short-circuit, since we're loading a channel.ini file. - self._override = override - if override: - return - self.gpg = Bag(**parser['gpg']) - self.updater = Bag(**parser['updater']) - self.hooks = Bag(converters=dict(device=as_object, - scorer=as_object, - reboot=as_object), - **parser['hooks']) - self.dbus = Bag(converters=dict(lifetime=as_timedelta), - **parser['dbus']) + self.http_base = http_base + self.https_base = https_base @property def build_number(self): if self._build_number is None: - if self._override: - return self.service.build_number - else: - try: - with open(self.system.build_file, encoding='utf-8') as fp: - return int(fp.read().strip()) - except FileNotFoundError: - return 0 + self._build_number = self.service.build_number return self._build_number @build_number.setter @@ -166,24 +234,18 @@ raise ValueError( 'integer is required, got: {}'.format(type(value).__name__)) self._build_number = value + self.build_number_override = True @build_number.deleter def build_number(self): self._build_number = None @property - def build_number_cli(self): - return self._build_number - - @property def device(self): if self._device is None: # Start by looking for a [service]device setting. Use this if it # exists, otherwise fall back to calling the hook. self._device = getattr(self.service, 'device', None) - # The key could exist in the channel.ini file, but its value could - # be empty. That's semantically equivalent to a missing - # [service]device setting. if not self._device: self._device = self.hooks.device().get_device() return self._device @@ -203,6 +265,18 @@ self._channel = value @property + def phase_override(self): + return self._phase_override + + @phase_override.setter + def phase_override(self, value): + self._phase_override = max(0, min(100, int(value))) + + @phase_override.deleter + def phase_override(self): + self._phase_override = None + + @property def tempdir(self): if self._tempdir is None: makedirs(self.system.tempdir) @@ -211,21 +285,13 @@ dir=self.system.tempdir)) return self._tempdir - -# Define the global configuration object. Normal use can be as simple as: -# -# from systemimage.config import config -# build_file = config.system.build_file -# -# In the test suite though, the actual configuration object can be easily -# patched by doing something like this: -# -# test_config = Configuration(...) -# with unittest.mock.patch('config._config', test_config): -# run_test() -# -# and now every module which does the first code example will get build_file -# from the mocked Configuration instance. + @property + def user_agent(self): + return USER_AGENT.format(self) + + +# Define the global configuration object. We use a proxy here so that +# post-object creation loading will work. _config = Configuration() === added file 'systemimage/curl.py' --- systemimage/curl.py 1970-01-01 00:00:00 +0000 +++ systemimage/curl.py 2015-05-20 14:55:53 +0000 @@ -0,0 +1,275 @@ +# Copyright (C) 2014-2015 Canonical Ltd. +# Author: Barry Warsaw + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 3 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +"""Download files via PyCURL.""" + +__all__ = [ + 'CurlDownloadManager', + ] + + +import pycurl +import hashlib +import logging + +from contextlib import ExitStack +from gi.repository import GLib +from systemimage.config import config +from systemimage.download import Canceled, DownloadManagerBase + +log = logging.getLogger('systemimage') + + +# Some cURL defaults. XXX pull these out of the configuration file. +CONNECTION_TIMEOUT = 120 # seconds +LOW_SPEED_LIMIT = 10 +LOW_SPEED_TIME = 120 # seconds +MAX_REDIRECTS = 5 +MAX_TOTAL_CONNECTIONS = 4 +SELECT_TIMEOUT = 0.05 # 20fps + + +def _curl_debug(debug_type, debug_msg): # pragma: no cover + from systemimage.testing.helpers import debug + with debug(end='') as ddlog: + ddlog('PYCURL:', debug_type, debug_msg) + + +def make_testable(c): + # The test suite needs to make the PyCURL object accept the testing + # server's self signed certificate. It will mock this function. + pass + + +class SingleDownload: + def __init__(self, record): + self.url, self.destination, self.expected_checksum = record + self._checksum = None + self._fp = None + self._resources = ExitStack() + + def make_handle(self, *, HEAD): + # If we're doing GET, record some more information. + if not HEAD: + self._checksum = hashlib.sha256() + # Create the basic PyCURL object. + c = pycurl.Curl() + # Set the common options. + c.setopt(pycurl.URL, self.url) + c.setopt(pycurl.USERAGENT, config.user_agent) + # If we're doing a HEAD, then we don't want the body of the + # file. Otherwise, set things up to write the body data to the + # destination file. + if HEAD: + c.setopt(pycurl.NOBODY, 1) + else: + c.setopt(pycurl.WRITEDATA, self) + self._fp = self._resources.enter_context( + open(self.destination, 'wb')) + # Set some limits. XXX Pull these out of the configuration files. + c.setopt(pycurl.FOLLOWLOCATION, 1) + c.setopt(pycurl.MAXREDIRS, MAX_REDIRECTS) + c.setopt(pycurl.CONNECTTIMEOUT, CONNECTION_TIMEOUT) + # If the average transfer speed is below 10 bytes per second for 2 + # minutes, libcurl will consider the connection too slow and abort. + ## c.setopt(pycurl.LOW_SPEED_LIMIT, LOW_SPEED_LIMIT) + ## c.setopt(pycurl.LOW_SPEED_TIME, LOW_SPEED_TIME) + # Fail on error codes >= 400. + c.setopt(pycurl.FAILONERROR, 1) + # Switch off the libcurl progress meters. The multi that uses + # this handle will set the transfer info function. + c.setopt(pycurl.NOPROGRESS, 1) + # ssl: no need to set SSL_VERIFYPEER, SSL_VERIFYHOST, CAINFO + # they all use sensible defaults + # + # Enable debugging. + self._make_debuggable(c) + # For the test suite. + make_testable(c) + return c + + def _make_debuggable(self, c): + """Add some additional debugging options.""" + ## c.setopt(pycurl.VERBOSE, 1) + ## c.setopt(pycurl.DEBUGFUNCTION, _curl_debug) + pass + + def write(self, data): + """Update the checksum and write the data out to the file.""" + self._checksum.update(data) + self._fp.write(data) + # Returning None implies that all bytes were written + # successfully, so it's better to be explicit. + return None + + def close(self): + self._resources.close() + + @property + def checksum(self): + # If no checksum was expected, pretend none was gotten. This + # makes the verification step below a wee bit simpler. + if self.expected_checksum == '': + return '' + return self._checksum.hexdigest() + + +class CurlDownloadManager(DownloadManagerBase): + """The PyCURL based download manager.""" + + def __init__(self, callback=None): + super().__init__() + if callback is not None: + self.callbacks.append(callback) + self._pausables = [] + self._paused = False + + def _get_files(self, records, pausable): + # Start by doing a HEAD on all the URLs so that we can get the total + # target download size in bytes, at least as best as is possible. + with ExitStack() as resources: + handles = [] + multi = pycurl.CurlMulti() + multi.setopt( + pycurl.M_MAX_TOTAL_CONNECTIONS, MAX_TOTAL_CONNECTIONS) + for record in records: + download = SingleDownload(record) + resources.callback(download.close) + handle = download.make_handle(HEAD=True) + handles.append(handle) + multi.add_handle(handle) + # .add_handle() does not bump the reference count, so we + # need to keep the PyCURL object alive for the duration + # of this download. + resources.callback(multi.remove_handle, handle) + self._perform(multi, handles) + self.total = sum( + handle.getinfo(pycurl.CONTENT_LENGTH_DOWNLOAD) + for handle in handles) + # Now do a GET on all the URLs. This will write the data to the + # destination file and collect the checksums. + with ExitStack() as resources: + resources.callback(setattr, self, '_handles', None) + downloads = [] + multi = pycurl.CurlMulti() + multi.setopt( + pycurl.M_MAX_TOTAL_CONNECTIONS, MAX_TOTAL_CONNECTIONS) + for record in records: + download = SingleDownload(record) + downloads.append(download) + resources.callback(download.close) + handle = download.make_handle(HEAD=False) + self._pausables.append(handle) + multi.add_handle(handle) + # .add_handle() does not bump the reference count, so we + # need to keep the PyCURL object alive for the duration + # of this download. + resources.callback(multi.remove_handle, handle) + self._perform(multi, self._pausables) + # Verify internally calculated checksums. The API requires + # a FileNotFoundError to be raised when they don't match. + # Since it doesn't matter which one fails, log them all and + # raise the first one. + first_mismatch = None + for download in downloads: + if download.checksum != download.expected_checksum: + log.error('Checksum mismatch. got:{} != exp:{}: {}', + download.checksum, download.expected_checksum, + download.destination) + if first_mismatch is None: + first_mismatch = download + if first_mismatch is not None: + # For backward compatibility with ubuntu-download_manager. + raise FileNotFoundError('HASH ERROR: {}'.format( + first_mismatch.destination)) + self._pausables = [] + + def _do_once(self, multi, handles): + status, active_count = multi.perform() + if status == pycurl.E_CALL_MULTI_PERFORM: + # Call .perform() again before calling select. + return True + elif status != pycurl.E_OK: + # An error occurred in the multi, so be done with the + # whole thing. We can't get a description string out of + # PyCURL though. Just raise one of the urls. + log.error('CurlMulti() error: {}', status) + raise FileNotFoundError(handles[0].getinfo(pycurl.EFFECTIVE_URL)) + # The multi is okay, but it's possible there are errors pending on + # the individual downloads; check those now. + queued_count, ok_list, error_list = multi.info_read() + if len(error_list) > 0: + # It helps to have at least one URL in the FileNotFoundError. + first_url = None + log.error('Curl() errors encountered:') + for c, code, message in error_list: + url = c.getinfo(pycurl.EFFECTIVE_URL) + if first_url is None: + first_url = url + log.error(' {} ({}): {}', message, code, url) + raise FileNotFoundError('{}: {}'.format(message, first_url)) + # For compatibility with .io_add_watch(), we return False if we want + # to stop the callbacks, and True if we want to call back here again. + return active_count > 0 + + def _perform(self, multi, handles): + # While we're performing the cURL downloads, we need to periodically + # process D-Bus events, otherwise we won't be able to cancel downloads + # or handle other interruptive events. To do this, we grab the GLib + # main loop context and then ask it to do an iteration over its events + # once in a while. It turns out that even if we're not running a D-Bus + # main loop (i.e. during the in-process tests) periodically dispatching + # into GLib doesn't hurt, so just do it unconditionally. + self.received = 0 + context = GLib.main_context_default() + while True: + # Do the progress callback, but only if the current received size + # is different than the last one. Don't worry about in which + # direction it's different. + received = int( + sum(c.getinfo(pycurl.SIZE_DOWNLOAD) for c in handles)) + if received != self.received: + self._do_callback() + self.received = received + if not self._do_once(multi, handles): + break + multi.select(SELECT_TIMEOUT) + # Let D-Bus events get dispatched, but only block if downloads are + # paused. + while context.iteration(may_block=self._paused): + pass + if self._queued_cancel: + raise Canceled + # One last callback, unconditionally. + self.received = int( + sum(c.getinfo(pycurl.SIZE_DOWNLOAD) for c in handles)) + self._do_callback() + + def pause(self): + for c in self._pausables: + c.pause(pycurl.PAUSE_ALL) + self._paused = True + # 2014-10-20 BAW: We could plumb through the `service` object from + # service.py (the main entry point for system-image-dbus, but that's + # actually a bit of a pain, so do the expedient thing and grab the + # interface here. + percentage = (int(self.received / self.total * 100.0) + if self.total > 0 else 0) + config.dbus_service.UpdatePaused(percentage) + + def resume(self): + self._paused = False + for c in self._pausables: + c.pause(pycurl.PAUSE_CONT) === removed file 'systemimage/data/client.ini' --- systemimage/data/client.ini 2014-01-30 15:41:03 +0000 +++ systemimage/data/client.ini 1970-01-01 00:00:00 +0000 @@ -1,35 +0,0 @@ -# Default and example .ini configuration file. -# Edit this and put it in /etc/system-image/client.ini - -[service] -base: system-image.ubuntu.com -http_port: 80 -https_port: 443 -channel: daily -build_number: 0 - -[system] -timeout: 1h -build_file: /etc/ubuntu-build -tempdir: /tmp -logfile: /var/log/system-image/client.log -loglevel: info -settings_db: /var/lib/system-image/settings.db - -[gpg] -archive_master: /etc/system-image/archive-master.tar.xz -image_master: /var/lib/system-image/keyrings/image-master.tar.xz -image_signing: /var/lib/system-image/keyrings/image-signing.tar.xz -device_signing: /var/lib/system-image/keyrings/device-signing.tar.xz - -[updater] -cache_partition: /android/cache/recovery -data_partition: /var/lib/system-image - -[hooks] -device: systemimage.device.SystemProperty -scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot - -[dbus] -lifetime: 10m === modified file 'systemimage/dbus.py' --- systemimage/dbus.py 2014-09-26 14:36:34 +0000 +++ systemimage/dbus.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -94,14 +94,14 @@ def __init__(self, bus, object_path, loop): super().__init__(bus, object_path) - self._loop = loop + self.loop = loop self._api = Mediator(self._progress_callback) log.info('Mediator created {}', self._api) self._checking = Lock() + self._downloading = Lock() self._update = None - self._downloading = False self._paused = False - self._rebootable = False + self._applicable = False self._failure_count = 0 self._last_error = '' @@ -110,8 +110,15 @@ # Asynchronous method call. log.info('Enter _check_for_update()') self._update = self._api.check_for_update() + log.info('_check_for_update(): checking lock releasing') + try: + self._checking.release() + except RuntimeError: + log.info('_check_for_update(): checking lock already released') + else: + log.info('_check_for_update(): checking lock released') # Do we have an update and can we auto-download it? - downloading = False + delayed_download = False if self._update.is_available: settings = Settings() auto = settings.get('auto_download') @@ -119,14 +126,19 @@ if auto in ('1', '2'): # XXX When we have access to the download service, we can # check if we're on the wifi (auto == '1'). + delayed_download = True GLib.timeout_add(50, self._download) - downloading = True + # We have a timing issue. We can't lock the downloading lock here, + # otherwise when _download() starts running in ~50ms it will think a + # download is already in progress. But we want to send the UAS signal + # here and now, *and* indicate whether the download is about to happen. + # So just lie for now since in ~50ms the download will begin. self.UpdateAvailableStatus( self._update.is_available, - downloading, + delayed_download, self._update.version, self._update.size, - self._update.last_update_date, + last_update_date(), self._update.error) # Stop GLib from calling this method again. return False @@ -148,24 +160,24 @@ completes. The argument to that signal is a boolean indicating whether the update is available or not. """ - self._loop.keepalive() + self.loop.keepalive() # Check-and-acquire the lock. - log.info('test and acquire checking lock') + log.info('CheckForUpdate(): checking lock test and acquire') if not self._checking.acquire(blocking=False): + log.info('CheckForUpdate(): checking lock not acquired') # Check is already in progress, so there's nothing more to do. If # there's status available (i.e. we are in the auto-downloading # phase of the last CFU), then send the status. if self._update is not None: self.UpdateAvailableStatus( self._update.is_available, - self._downloading, + self._downloading.locked(), self._update.version, self._update.size, - self._update.last_update_date, + last_update_date(), "") - log.info('checking lock not acquired') return - log.info('checking lock acquired') + log.info('CheckForUpdate(): checking lock acquired') # We've now acquired the lock. Reset any failure or in-progress # state. Get a new mediator to reset any of its state. self._api = Mediator(self._progress_callback) @@ -176,7 +188,7 @@ # this method can return immediately. GLib.timeout_add(50, self._check_for_update) - @log_and_exit + #@log_and_exit def _progress_callback(self, received, total): # Plumb the progress through our own D-Bus API. Our API is defined as # signalling a percentage and an eta. We can calculate the percentage @@ -187,12 +199,12 @@ @log_and_exit def _download(self): - if self._downloading and self._paused: + if self._downloading.locked() and self._paused: self._api.resume() self._paused = False log.info('Download previously paused') return - if (self._downloading # Already in progress. + if (self._downloading.locked() # Already in progress. or self._update is None # Not yet checked. or not self._update.is_available # No update available. ): @@ -204,47 +216,33 @@ log.info('Update failures: {}; last error: {}', self._failure_count, self._last_error) return - self._downloading = True - log.info('Update is downloading') - try: - # Always start by sending a UpdateProgress(0, 0). This is - # enough to get the u/i's attention. - self.UpdateProgress(0, 0) - self._api.download() - except Exception: - log.exception('Download failed') - self._failure_count += 1 - # Set the last error string to the exception's class name. - exception, value = sys.exc_info()[:2] - # if there's no meaningful value, omit it. - value_str = str(value) - name = exception.__name__ - self._last_error = ('{}'.format(name) - if len(value_str) == 0 - else '{}: {}'.format(name, value)) - self.UpdateFailed(self._failure_count, self._last_error) - else: - log.info('Update downloaded') - self.UpdateDownloaded() - self._failure_count = 0 - self._last_error = '' - self._rebootable = True - self._downloading = False - log.info('releasing checking lock from _download()') - try: - self._checking.release() - except RuntimeError: - # 2014-09-11 BAW: We don't own the lock. There are several reasons - # why this can happen including: 1) the client canceled the - # download while it was in progress, and CancelUpdate() already - # released the lock; 2) the client called DownloadUpdate() without - # first calling CheckForUpdate(); 3) the client called DU() - # multiple times in a row but the update was already downloaded and - # all the file signatures have been verified. I can't think of - # reason why we shouldn't just ignore the double release, so - # that's what we do. See LP: #1365646. - pass - log.info('released checking lock from _download()') + log.info('_download(): downloading lock entering critical section') + with self._downloading: + log.info('Update is downloading') + try: + # Always start by sending a UpdateProgress(0, 0). This is + # enough to get the u/i's attention. + self.UpdateProgress(0, 0) + self._api.download() + except Exception: + log.exception('Download failed') + self._failure_count += 1 + # Set the last error string to the exception's class name. + exception, value = sys.exc_info()[:2] + # if there's no meaningful value, omit it. + value_str = str(value) + name = exception.__name__ + self._last_error = ('{}'.format(name) + if len(value_str) == 0 + else '{}: {}'.format(name, value)) + self.UpdateFailed(self._failure_count, self._last_error) + else: + log.info('Update downloaded') + self.UpdateDownloaded() + self._failure_count = 0 + self._last_error = '' + self._applicable = True + log.info('_download(): downloading lock finished critical section') # Stop GLib from calling this method again. return False @@ -257,15 +255,15 @@ """ # Arrange for the update to happen in a little while, so that this # method can return immediately. - self._loop.keepalive() + self.loop.keepalive() GLib.timeout_add(50, self._download) @log_and_exit @method('com.canonical.SystemImage', out_signature='s') def PauseDownload(self): """Pause a downloading update.""" - self._loop.keepalive() - if self._downloading: + self.loop.keepalive() + if self._downloading.locked(): self._api.pause() self._paused = True error_message = '' @@ -277,40 +275,34 @@ @method('com.canonical.SystemImage', out_signature='s') def CancelUpdate(self): """Cancel a download.""" - self._loop.keepalive() + self.loop.keepalive() # During the download, this will cause an UpdateFailed signal to be # issued, as part of the exception handling in _download(). If we're # not downloading, then no signal need be sent. There's no need to # send *another* signal when downloading, because we never will be # downloading by the time we get past this next call. self._api.cancel() - # If we're holding the checking lock, release it. - try: - log.info('releasing checking lock from CancelUpdate()') - self._checking.release() - log.info('released checking lock from CancelUpdate()') - except RuntimeError: - # We're not holding the lock. - pass # XXX 2013-08-22: If we can't cancel the current download, return the # reason in this string. return '' @log_and_exit def _apply_update(self): - self._loop.keepalive() - if not self._rebootable: + self.loop.keepalive() + if not self._applicable: command_file = os.path.join( config.updater.cache_partition, 'ubuntu_command') if not os.path.exists(command_file): - # Not enough has been downloaded to allow for a reboot. - self.Rebooting(False) + # Not enough has been downloaded to allow for the update to be + # applied. + self.Applied(False) return - self._api.reboot() - # This code may or may not run. We're racing against the system - # reboot procedure. - self._rebootable = False - self.Rebooting(True) + self._api.apply() + # This code may or may not run. On devices for which applying the + # update requires a system reboot, we're racing against that reboot + # procedure. + self._applicable = False + self.Applied(True) @log_and_exit @method('com.canonical.SystemImage') @@ -322,7 +314,7 @@ @log_and_exit @method('com.canonical.SystemImage', out_signature='isssa{ss}') def Info(self): - self._loop.keepalive() + self.loop.keepalive() return (config.build_number, config.device, config.channel, @@ -332,23 +324,27 @@ @log_and_exit @method('com.canonical.SystemImage', out_signature='a{ss}') def Information(self): - self._loop.keepalive() + self.loop.keepalive() settings = Settings() current_build_number = str(config.build_number) + version_detail = getattr(config.service, 'version_detail', '') response = dict( current_build_number=current_build_number, device_name=config.device, channel_name=config.channel, last_update_date=last_update_date(), - version_detail=getattr(config.service, 'version_detail', ''), + version_detail=version_detail, last_check_date=settings.get('last_check_date'), ) if self._update is None: response['target_build_number'] = '-1' + response['target_version_detail'] = '' elif not self._update.is_available: response['target_build_number'] = current_build_number + response['target_version_detail'] = version_detail else: response['target_build_number'] = str(self._update.version) + response['target_version_detail'] = self._update.version_detail return response @log_and_exit @@ -359,7 +355,7 @@ Some values are special, e.g. min_battery and auto_downloads. Implement these special semantics here. """ - self._loop.keepalive() + self.loop.keepalive() if key == 'min_battery': try: as_int = int(value) @@ -385,22 +381,24 @@ @method('com.canonical.SystemImage', in_signature='s', out_signature='s') def GetSetting(self, key): """Get a setting.""" - self._loop.keepalive() + self.loop.keepalive() return Settings().get(key) @log_and_exit @method('com.canonical.SystemImage') def FactoryReset(self): self._api.factory_reset() - # This code may or may not run. We're racing against the system - # reboot procedure. - self.Rebooting(True) + + @log_and_exit + @method('com.canonical.SystemImage') + def ProductionReset(self): + self._api.production_reset() @log_and_exit @method('com.canonical.SystemImage') def Exit(self): """Quit the daemon immediately.""" - self._loop.quit() + self.loop.quit() @log_and_exit @signal('com.canonical.SystemImage', signature='bbsiss') @@ -416,21 +414,21 @@ log.debug('EMIT UpdateAvailableStatus({}, {}, {}, {}, {}, {})', is_available, downloading, available_version, update_size, last_update_date, repr(error_reason)) - self._loop.keepalive() + self.loop.keepalive() - @log_and_exit + #@log_and_exit @signal('com.canonical.SystemImage', signature='id') def UpdateProgress(self, percentage, eta): """Download progress.""" log.debug('EMIT UpdateProgress({}, {})', percentage, eta) - self._loop.keepalive() + self.loop.keepalive() @log_and_exit @signal('com.canonical.SystemImage') def UpdateDownloaded(self): """The update has been successfully downloaded.""" log.debug('EMIT UpdateDownloaded()') - self._loop.keepalive() + self.loop.keepalive() @log_and_exit @signal('com.canonical.SystemImage', signature='is') @@ -438,21 +436,28 @@ """The update failed for some reason.""" log.debug('EMIT UpdateFailed({}, {})', consecutive_failure_count, repr(last_reason)) - self._loop.keepalive() + self.loop.keepalive() @log_and_exit @signal('com.canonical.SystemImage', signature='i') def UpdatePaused(self, percentage): """The download got paused.""" log.debug('EMIT UpdatePaused({})', percentage) - self._loop.keepalive() + self.loop.keepalive() @log_and_exit @signal('com.canonical.SystemImage', signature='ss') def SettingChanged(self, key, new_value): """A setting value has change.""" log.debug('EMIT SettingChanged({}, {})', repr(key), repr(new_value)) - self._loop.keepalive() + self.loop.keepalive() + + @log_and_exit + @signal('com.canonical.SystemImage', signature='b') + def Applied(self, status): + """The update has been applied.""" + log.debug('EMIT Applied({})', status) + self.loop.keepalive() @log_and_exit @signal('com.canonical.SystemImage', signature='b') === modified file 'systemimage/device.py' --- systemimage/device.py 2014-09-17 13:41:31 +0000 +++ systemimage/device.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify === modified file 'systemimage/docs/conf.py' --- systemimage/docs/conf.py 2014-02-20 23:03:24 +0000 +++ systemimage/docs/conf.py 2015-05-20 14:55:53 +0000 @@ -41,7 +41,7 @@ # General information about the project. project = u'Image Update Resolver' -copyright = u'2013-2014, Canonical Ltd.' +copyright = u'2013-2015, Canonical Ltd.' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the === modified file 'systemimage/download.py' --- systemimage/download.py 2014-09-17 13:41:31 +0000 +++ systemimage/download.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -17,9 +17,9 @@ __all__ = [ 'Canceled', - 'DBusDownloadManager', 'DuplicateDestinationError', 'Record', + 'get_download_manager', ] @@ -28,39 +28,18 @@ import logging from collections import namedtuple -from contextlib import suppress from io import StringIO from pprint import pformat -from systemimage.config import config -from systemimage.reactor import Reactor -from systemimage.settings import Settings - -# The systemimage.testing module will not be available on installed systems -# unless the system-image-dev binary package is installed, which is not usually -# the case. Disable _print() debugging in that case. -def _print(*args, **kws): - with suppress(ImportError): - # We must import this here to avoid circular imports. - from systemimage.testing.helpers import debug - with debug(check_flag=True) as ddlog: - ddlog(*args, **kws) - - -# Parameterized for testing purposes. -DOWNLOADER_INTERFACE = 'com.canonical.applications.Downloader' -MANAGER_INTERFACE = 'com.canonical.applications.DownloadManager' -OBJECT_NAME = 'com.canonical.applications.Downloader' -OBJECT_INTERFACE = 'com.canonical.applications.GroupDownload' -USER_AGENT = 'Ubuntu System Image Upgrade Client; Build {}' + +try: + import pycurl +except ImportError: # pragma: no cover + pycurl = None log = logging.getLogger('systemimage') -def _headers(): - return {'User-Agent': USER_AGENT.format(config.build_number)} - - class Canceled(Exception): """Raised when the download was canceled.""" @@ -89,78 +68,10 @@ url=url, destination=destination, checksum=checksum) -class DownloadReactor(Reactor): - def __init__(self, bus, callback=None, pausable=False): - super().__init__(bus) - self._callback = callback - self._pausable = pausable - self.error = None - self.canceled = False - self.received = 0 - self.total = 0 - self.react_to('canceled') - self.react_to('error') - self.react_to('finished') - self.react_to('paused') - self.react_to('progress') - self.react_to('resumed') - self.react_to('started') - - def _do_started(self, signal, path, started): - _print('STARTED:', started) - - def _do_finished(self, signal, path, local_paths): - _print('FINISHED:', local_paths) - self.quit() - - def _do_error(self, signal, path, error_message): - _print('ERROR:', error_message) - log.error(error_message) - self.error = error_message - self.quit() - - def _do_progress(self, signal, path, received, total): - self.received = received - self.total = total - _print('PROGRESS:', received, total) - if self._callback is not None: - # Be defensive, so yes, use a bare except. If an exception occurs - # in the callback, log it, but continue onward. - try: - self._callback(received, total) - except: - log.exception('Exception in progress callback') - - def _do_canceled(self, signal, path, canceled): - # Why would we get this signal if it *wasn't* canceled? Anyway, - # this'll be a D-Bus data type so converted it to a vanilla Python - # boolean. - _print('CANCELED:', canceled) - self.canceled = bool(canceled) - self.quit() - - def _do_paused(self, signal, path, paused): - _print('PAUSE:', paused, self._pausable) - send_paused = self._pausable and config.dbus_service is not None - if send_paused: # pragma: no branch - # We could plumb through the `service` object from service.py (the - # main entry point for system-image-dbus, but that's actually a - # bit of a pain, so do the expedient thing and grab the interface - # here. - percentage = (int(self.received / self.total * 100.0) - if self.total > 0 else 0) - config.dbus_service.UpdatePaused(percentage) - - def _do_resumed(self, signal, path, resumed): - _print('RESUME:', resumed) - # There currently is no UpdateResumed() signal. - - def _default(self, *args, **kws): - _print('SIGNAL:', args, kws) # pragma: no cover - - -class DBusDownloadManager: - def __init__(self, callback=None): +class DownloadManagerBase: + """Base class for all download managers.""" + + def __init__(self): """ :param callback: If given, a function that is called every so often during downloading. @@ -168,12 +79,79 @@ of bytes received so far, and the total amount of bytes to be downloaded. """ - self._iface = None + # This is a list of functions that are called every so often during + # downloading. Functions in this list take two arguments, the number + # of bytes received so far, and the total amount of bytes to be + # downloaded. + self.callbacks = [] + self.total = 0 + self.received = 0 self._queued_cancel = False - self.callback = callback def __repr__(self): # pragma: no cover - return ''.format(id(self)) + return '<{} at 0x{:x}>'.format(self.__class__.__name__, id(self)) + + def _get_download_records(self, downloads): + """Convert the downloads items to download records.""" + records = [item if isinstance(item, _RecordType) else Record(*item) + for item in downloads] + destinations = set(record.destination for record in records) + # Check for duplicate destinations, specifically for a local file path + # coming from two different sources. It's okay if there are duplicate + # destination records in the download request, but each of those must + # be specified by the same source url and have the same checksum. + # + # An easy quick check just asks if the set of destinations is smaller + # than the total number of requested downloads. It can't be larger. + # If it *is* smaller, then there are some duplicates, however the + # duplicates may be legitimate, so look at the details. + # + # Note though that we cannot pass duplicates destinations to udm, so we + # have to filter out legitimate duplicates. That's fine since they + # really are pointing to the same file, and will end up in the + # destination location. + if len(destinations) < len(downloads): + by_destination = dict() + unique_downloads = set() + for record in records: + by_destination.setdefault(record.destination, set()).add( + record) + unique_downloads.add(record) + duplicates = [] + for dst, seen in by_destination.items(): + if len(seen) > 1: + # Tuples will look better in the pretty-printed output. + duplicates.append( + (dst, sorted(tuple(dup) for dup in seen))) + if len(duplicates) > 0: + raise DuplicateDestinationError(sorted(duplicates)) + # Uniquify the downloads. + records = list(unique_downloads) + return records + + def _do_callback(self): + # Be defensive, so yes, use a bare except. If an exception occurs in + # the callback, log it, but continue onward. + for callback in self.callbacks: + try: + callback(self.received, self.total) + except: + log.exception('Exception in progress callback') + + def cancel(self): + """Cancel any current downloads.""" + self._queued_cancel = True + + def pause(self): + """Pause the download, but only if one is in progress.""" + pass # pragma: no cover + + def resume(self): + """Resume the download, but only if one is in progress.""" + pass # pragma: no cover + + def _get_files(self, records, pausable): + raise NotImplementedError # pragma: no cover def get_files(self, downloads, *, pausable=False): """Download a bunch of files concurrently. @@ -204,52 +182,16 @@ :raises: DuplicateDestinationError if more than one source url is downloaded to the same destination file. """ - assert self._iface is None if self._queued_cancel: # A cancel is queued, so don't actually download anything. raise Canceled if len(downloads) == 0: # Nothing to download. See LP: #1245597. return - # Convert the downloads items to download records. - records = [item if isinstance(item, _RecordType) else Record(*item) - for item in downloads] - destinations = set(record.destination for record in records) - # Check for duplicate destinations, specifically for a local file path - # coming from two different sources. It's okay if there are duplicate - # destination records in the download request, but each of those must - # be specified by the same source url and have the same checksum. - # - # An easy quick check just asks if the set of destinations is smaller - # than the total number of requested downloads. It can't be larger. - # If it *is* smaller, then there are some duplicates, however the - # duplicates may be legitimate, so look at the details. - # - # Note though that we cannot pass duplicates destinations to udm, - # so we have to filter out legitimate duplicates. That's fine since - # they really are pointing to the same file, and will end up in the - # destination location. - if len(destinations) < len(downloads): - by_destination = dict() - unique_downloads = set() - for record in records: - by_destination.setdefault(record.destination, set()).add( - record) - unique_downloads.add(record) - duplicates = [] - for dst, seen in by_destination.items(): - if len(seen) > 1: - # Tuples will look better in the pretty-printed output. - duplicates.append( - (dst, sorted(tuple(dup) for dup in seen))) - if len(duplicates) > 0: - raise DuplicateDestinationError(sorted(duplicates)) - # Uniquify the downloads. - records = list(unique_downloads) - bus = dbus.SystemBus() - service = bus.get_object(DOWNLOADER_INTERFACE, '/') - iface = dbus.Interface(service, MANAGER_INTERFACE) - # Better logging of the requested downloads. + records = self._get_download_records(downloads) + # Better logging of the requested downloads. However, we want the + # entire block of multiline log output to appear under a single + # timestamp. fp = StringIO() print('[0x{:x}] Requesting group download:'.format(id(self)), file=fp) for record in records: @@ -258,69 +200,38 @@ else: print('\t{} [{}] -> {}'.format(*record), file=fp) log.info('{}'.format(fp.getvalue())) - object_path = iface.createDownloadGroup( - records, - 'sha256', - False, # Don't allow GSM yet. - # https://bugs.freedesktop.org/show_bug.cgi?id=55594 - dbus.Dictionary(signature='sv'), - _headers()) - download = bus.get_object(OBJECT_NAME, object_path) - self._iface = dbus.Interface(download, OBJECT_INTERFACE) - # Are GSM downloads allowed? Yes, except if auto_download is set to 1 - # (i.e. wifi-only). - allow_gsm = Settings().get('auto_download') != '1' - DBusDownloadManager._set_gsm(self._iface, allow_gsm=allow_gsm) - # Start the download. - reactor = DownloadReactor(bus, self.callback, pausable) - reactor.schedule(self._iface.start) - log.info('[0x{:x}] Running group download reactor', id(self)) - reactor.run() - # This download is complete so the object path is no longer - # applicable. Setting this to None will cause subsequent cancels to - # be queued. - self._iface = None - log.info('[0x{:x}] Group download reactor done', id(self)) - if reactor.error is not None: - log.error('Reactor error: {}'.format(reactor.error)) - if reactor.canceled: - log.info('Reactor canceled') - # Report any other problems. - if reactor.error is not None: - raise FileNotFoundError(reactor.error) - if reactor.canceled: - raise Canceled - if reactor.timed_out: - raise TimeoutError - # For sanity. - for record in records: - assert os.path.exists(record.destination), ( - 'Missing destination: {}'.format(record)) - - @staticmethod - def _set_gsm(iface, *, allow_gsm): - # This is a separate method for easier testing via mocks. - iface.allowGSMDownload(allow_gsm) - - def cancel(self): - """Cancel any current downloads.""" - if self._iface is None: - # Since there's no download in progress right now, there's nothing - # to cancel. Setting this flag queues the cancel signal once the - # reactor starts running again. Yes, this is a bit weird, but if - # we don't do it this way, the caller will immediately get a - # Canceled exception, which isn't helpful because it's expecting - # one when the next download begins. - self._queued_cancel = True + self._get_files(records, pausable) + + +def get_download_manager(*args): + # We have to avoid circular imports since both download managers import + # various things from this module. + from systemimage.curl import CurlDownloadManager + from systemimage.udm import DOWNLOADER_INTERFACE, UDMDownloadManager + # Detect if we have ubuntu-download-manager. + # + # Use PyCURL based downloader if no udm is found, or if the environment + # variable is set. However, if we're told to use PyCURL and it's + # unavailable, throw an exception. + cls = None + use_pycurl = os.environ.get('SYSTEMIMAGE_PYCURL') + if use_pycurl is None: + # Auto-detect. For backward compatibility, use udm if it's available, + # otherwise use PyCURL. + try: + bus = dbus.SystemBus() + bus.get_object(DOWNLOADER_INTERFACE, '/') + udm_available = True + except dbus.exceptions.DBusException: + udm_available = False + if udm_available: + cls = UDMDownloadManager + elif pycurl is None: + raise ImportError('No module named {}'.format('pycurl')) else: - self._iface.cancel() - - def pause(self): - """Pause the download, but only if one is in progress.""" - if self._iface is not None: # pragma: no branch - self._iface.pause() - - def resume(self): - """Resume the download, but only if one is in progress.""" - if self._iface is not None: # pragma: no branch - self._iface.resume() + cls = CurlDownloadManager + else: + cls = (CurlDownloadManager + if use_pycurl.lower() in ('1', 'yes', 'true') + else UDMDownloadManager) + return cls(*args) === modified file 'systemimage/gpg.py' --- systemimage/gpg.py 2014-09-17 13:41:31 +0000 +++ systemimage/gpg.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -188,6 +188,11 @@ :type data_path: str :return: bool """ + # For testing on some systems that are connecting to test servers, GPG + # verification isn't possible. The s-i-cli supports a switch to + # disable all GPG checks. + if config.skip_gpg_verification: + return True with open(signature_path, 'rb') as sig_fp: verified = self._ctx.verify_file(sig_fp, data_path) # If the file is properly signed, we'll be able to get back a set of === modified file 'systemimage/helpers.py' --- systemimage/helpers.py 2014-09-17 13:41:31 +0000 +++ systemimage/helpers.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -20,6 +20,8 @@ 'MiB', 'as_loglevel', 'as_object', + 'as_port', + 'as_stripped', 'as_timedelta', 'atomic', 'calculate_signature', @@ -34,7 +36,6 @@ import os import re -import time import random import shutil import logging @@ -46,11 +47,12 @@ from importlib import import_module +UNIQUE_MACHINE_ID_FILES = ['/var/lib/dbus/machine-id', '/etc/machine-id'] LAST_UPDATE_FILE = '/userdata/.last_update' -UNIQUE_MACHINE_ID_FILE = '/var/lib/dbus/machine-id' DEFAULT_DIRMODE = 0o02700 MiB = 1 << 20 EMPTYSTRING = '' +NO_PORT = object() def calculate_signature(fp, hash_class=None): @@ -79,7 +81,7 @@ """Like os.remove() but don't complain if the file doesn't exist.""" try: os.remove(path) - except (FileNotFoundError, IsADirectoryError): + except (FileNotFoundError, IsADirectoryError, PermissionError): pass @@ -195,13 +197,26 @@ dbus = 'ERROR' main_level = getattr(logging, main, None) if main_level is None or not isinstance(main_level, int): - raise ValueError + raise ValueError(value) dbus_level = getattr(logging, dbus, None) if dbus_level is None or not isinstance(dbus_level, int): - raise ValueError + raise ValueError(value) return main_level, dbus_level +def as_port(value): + if value.lower() in ('disabled', 'disable'): + return NO_PORT + result = int(value) + if result < 0: + raise ValueError(value) + return result + + +def as_stripped(value): + return value.strip() + + @contextmanager def temporary_directory(*args, **kws): """A context manager that creates a temporary directory. @@ -227,30 +242,24 @@ def last_update_date(): """Return the last update date. - Taken from the mtime of the following files, in order: - - - /userdata/.last_update - - /etc/system-image/channel.ini - - /etc/ubuntu-build - - First existing path wins. + If /userdata/.last_update exists, we use this file's mtime. If it doesn't + exist, then we use the latest mtime of any of the files in + /etc/system-image/config.d/*.ini (or whatever directory was given with the + -C/--config option). """ # Avoid circular imports. from systemimage.config import config - channel_ini = os.path.join( - os.path.dirname(config.config_file), 'channel.ini') - ubuntu_build = config.system.build_file - for path in (LAST_UPDATE_FILE, channel_ini, ubuntu_build): - try: - # Local time, since we can't know the timezone. - timestamp = datetime.fromtimestamp(os.stat(path).st_mtime) - # Seconds resolution. - timestamp = timestamp.replace(microsecond=0) - return str(timestamp) - except (FileNotFoundError, PermissionError): - pass - else: - return 'Unknown' + try: + timestamp = datetime.fromtimestamp(os.stat(LAST_UPDATE_FILE).st_mtime) + except (FileNotFoundError, PermissionError): + # We fall back to the latest mtime of the config.d/*.ini files. + timestamps = sorted( + datetime.fromtimestamp(path.stat().st_mtime) + for path in config.ini_files) + if len(timestamps) == 0: + return 'Unknown' + timestamp = timestamps[-1] + return str(timestamp.replace(microsecond=0)) def version_detail(details_string=None): @@ -270,19 +279,20 @@ return details -_pp_cache = None - -def phased_percentage(*, reset=False): - global _pp_cache - if _pp_cache is None: - with open(UNIQUE_MACHINE_ID_FILE, 'rb') as fp: - data = fp.read() - now = str(time.time()).encode('us-ascii') - r = random.Random() - r.seed(data + now) - _pp_cache = r.randint(0, 100) - try: - return _pp_cache - finally: - if reset: - _pp_cache = None +def phased_percentage(channel, target): + # Avoid circular imports. + from systemimage.config import config + if config.phase_override is not None: + return config.phase_override + for path in UNIQUE_MACHINE_ID_FILES: + try: + with open(path, 'r', encoding='utf-8') as fp: + machine_id = fp.read().strip() + break # pragma: no branch + except FileNotFoundError: + pass + else: + raise RuntimeError('No machine-id file found') + r = random.Random() + r.seed('{}.{}.{}'.format(channel, target, machine_id)) + return r.randint(0, 100) === modified file 'systemimage/image.py' --- systemimage/image.py 2014-09-17 13:41:31 +0000 +++ systemimage/image.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -74,3 +74,7 @@ @property def phased_percentage(self): return self.__untranslated__.get('phased-percentage', 100) + + @property + def version_detail(self): + return self.__untranslated__.get('version_detail', '') === modified file 'systemimage/index.py' --- systemimage/index.py 2014-02-20 23:03:24 +0000 +++ systemimage/index.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -24,7 +24,6 @@ from datetime import datetime, timezone from systemimage.bag import Bag -from systemimage.helpers import phased_percentage from systemimage.image import Image @@ -49,7 +48,6 @@ global_ = Bag(generated_at=generated_at) # Parse the images. images = [] - percentage = phased_percentage() for image_data in mapping['images']: # Descriptions can be any of: # @@ -70,6 +68,5 @@ image = Image(files=bundles, descriptions=descriptions, **image_data) - if percentage <= image.phased_percentage: - images.append(image) + images.append(image) return cls(global_=global_, images=images) === modified file 'systemimage/keyring.py' --- systemimage/keyring.py 2014-02-20 23:03:24 +0000 +++ systemimage/keyring.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -30,7 +30,7 @@ from contextlib import ExitStack from datetime import datetime, timezone from systemimage.config import config -from systemimage.download import DBusDownloadManager +from systemimage.download import get_download_manager from systemimage.gpg import Context from systemimage.helpers import makedirs, safe_remove from urllib.parse import urljoin @@ -86,8 +86,8 @@ else: srcurl = urls ascurl = urls + '.asc' - tarxz_src = urljoin(config.service.https_base, srcurl) - ascxz_src = urljoin(config.service.https_base, ascurl) + tarxz_src = urljoin(config.https_base, srcurl) + ascxz_src = urljoin(config.https_base, ascurl) # Calculate the local paths to the temporary download files. The # blacklist goes to the data partition and all the other files go to the # cache partition. @@ -102,7 +102,7 @@ safe_remove(ascxz_dst) with ExitStack() as stack: # Let FileNotFoundError percolate up. - DBusDownloadManager().get_files([ + get_download_manager().get_files([ (tarxz_src, tarxz_dst), (ascxz_src, ascxz_dst), ]) === modified file 'systemimage/logging.py' --- systemimage/logging.py 2014-09-17 13:41:31 +0000 +++ systemimage/logging.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -80,7 +80,9 @@ def initialize(*, verbosity=0): """Initialize the loggers.""" main, dbus = config.system.loglevel - for name, loglevel in (('systemimage', main), ('systemimage.dbus', dbus)): + for name, loglevel in (('systemimage', main), + ('systemimage.dbus', dbus), + ('dbus.proxies', dbus)): level = { 0: logging.ERROR, 1: logging.INFO, === modified file 'systemimage/main.py' --- systemimage/main.py 2014-09-26 14:36:34 +0000 +++ systemimage/main.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -21,18 +21,19 @@ ] -import os import sys +import json import logging import argparse from dbus.mainloop.glib import DBusGMainLoop from pkg_resources import resource_string as resource_bytes +from systemimage.apply import factory_reset, production_reset from systemimage.candidates import delta_filter, full_filter from systemimage.config import config -from systemimage.helpers import last_update_date, makedirs, version_detail +from systemimage.helpers import ( + last_update_date, makedirs, phased_percentage, version_detail) from systemimage.logging import initialize -from systemimage.reboot import factory_reset from systemimage.settings import Settings from systemimage.state import State from textwrap import dedent @@ -41,12 +42,47 @@ __version__ = resource_bytes( 'systemimage', 'version.txt').decode('utf-8').strip() -DEFAULT_CONFIG_FILE = '/etc/system-image/client.ini' +DEFAULT_CONFIG_D = '/etc/system-image/config.d' COLON = ':' +LINE_LENGTH = 78 + + +class _DotsProgress: + def __init__(self): + self._dot_count = 0 + + def callback(self, received, total): + received = int(received) + total = int(total) + sys.stderr.write('.') + sys.stderr.flush() + self._dot_count += 1 + show_dots = self._dot_count % LINE_LENGTH == 0 + if show_dots or received >= total: # pragma: no cover + sys.stderr.write('\n') + sys.stderr.flush() + + +class _LogfileProgress: + def __init__(self, log): + self._log = log + + def callback(self, received, total): + self._log.debug('received: {} of {} bytes', received, total) + + +def _json_progress(received, total): + # For use with --progress=json output. LP: #1423622 + message = json.dumps(dict( + type='progress', + now=received, + total=total)) + sys.stdout.write(message) + sys.stdout.write('\n') + sys.stdout.flush() def main(): - global config parser = argparse.ArgumentParser( prog='system-image-cli', description='Ubuntu System Image Upgrader') @@ -54,10 +90,10 @@ action='version', version='system-image-cli {}'.format(__version__)) parser.add_argument('-C', '--config', - default=DEFAULT_CONFIG_FILE, action='store', - metavar='FILE', - help="""Use the given configuration file instead of - the default""") + default=DEFAULT_CONFIG_D, action='store', + metavar='DIRECTORY', + help="""Use the given configuration directory instead + of the default""") parser.add_argument('-b', '--build', default=None, action='store', help="""Override the current build number just @@ -76,12 +112,16 @@ full updates or only delta updates. The argument to this option must be either `full` or `delta`""") - parser.add_argument('-g', '--no-reboot', + parser.add_argument('-g', '--no-apply', default=False, action='store_true', help="""Download (i.e. "get") all the data files and prepare for updating, but don't actually reboot the device into recovery to apply the update""") + # Deprecated since si 3.0. + parser.add_argument('--no-reboot', + default=False, action='store_true', + help="""Deprecated; use -g/--no-apply""") parser.add_argument('-i', '--info', default=False, action='store_true', help="""Show some information about the current @@ -94,6 +134,15 @@ parser.add_argument('-v', '--verbose', default=0, action='count', help='Increase verbosity') + parser.add_argument('--progress', + default=[], action='append', + help="""Add a progress meter. Available meters are: + dots, logfile, and json. Multiple --progress + options are allowed.""") + parser.add_argument('-p', '--percentage', + default=None, action='store', + help="""Override the device's phased percentage value + during upgrade candidate calculation.""") parser.add_argument('--list-channels', default=False, action='store_true', help="""List all available channels, then exit""") @@ -102,6 +151,12 @@ help="""Perform a destructive factory reset and reboot. WARNING: this will wipe all user data on the device!""") + parser.add_argument('--production-reset', + default=False, action='store_true', + help="""Perform a destructive production reset + (similar to factory reset) and reboot. + WARNING: this will wipe all user data + on the device!""") parser.add_argument('--switch', default=None, action='store', metavar='CHANNEL', help="""Switch to the given channel. This is @@ -128,27 +183,36 @@ help="""Delete the key and its value. It is a no-op if the key does not exist. Multiple --del arguments can be given.""") + # Hidden system-image-cli only feature for testing purposes. LP: #1333414 + parser.add_argument('--skip-gpg-verification', + default=False, action='store_true', + help=argparse.SUPPRESS) args = parser.parse_args(sys.argv[1:]) try: config.load(args.config) - except FileNotFoundError as error: - parser.error('\nConfiguration file not found: {}'.format(error)) + except (TypeError, FileNotFoundError): + parser.error('\nConfiguration directory not found: {}'.format( + args.config)) assert 'parser.error() does not return' # pragma: no cover - # Load the optional channel.ini file, which must live next to the - # configuration file. It's okay if this file does not exist. - channel_ini = os.path.join(os.path.dirname(args.config), 'channel.ini') - try: - config.load(channel_ini, override=True) - except FileNotFoundError: - pass - - # Perform a factory reset. + + if args.skip_gpg_verification: + print("""\ +WARNING: All GPG signature verifications have been disabled. +Your upgrades are INSECURE.""", file=sys.stderr) + config.skip_gpg_verification = True + + # Perform factory and production resets. if args.factory_reset: factory_reset() # We should never get here, except possibly during the testing # process, so just return as normal. return 0 + if args.production_reset: + production_reset() + # We should never get here, except possibly during the testing + # process, so just return as normal. + return 0 # Handle all settings arguments. They are mutually exclusive. if sum(bool(arg) for arg in @@ -213,6 +277,8 @@ config.channel = args.channel if args.device is not None: config.device = args.device + if args.percentage is not None: + config.phase_override = args.percentage if args.info: alias = getattr(config.service, 'channel_target', None) @@ -245,11 +311,16 @@ print('version {}: {}'.format(key, details[key])) return 0 + DBusGMainLoop(set_as_default=True) + if args.list_channels: state = State() try: state.run_thru('get_channel') except Exception: + print('Exception occurred during channel search; ' + 'see log file for details', + file=sys.stderr) log.exception('system-image-cli exception') return 1 print('Available channels:') @@ -261,33 +332,26 @@ print(' {} (alias for: {})'.format(key, alias)) return 0 - # When verbosity is at 1, logging every progress signal from - # ubuntu-download-manager would be way too noisy. OTOH, not printing - # anything leads some folks to think the process is just hung, since it - # can take a long time to download all the data files. As a compromise, - # we'll output some dots to stderr at verbosity 1, but we won't log these - # dots since they would just be noise. This doesn't have to be perfect. - if args.verbose == 1: # pragma: no cover - dot_count = 0 - def callback(received, total): - nonlocal dot_count - sys.stderr.write('.') - sys.stderr.flush() - dot_count += 1 - if dot_count % 78 == 0 or received >= total: - sys.stderr.write('\n') - sys.stderr.flush() - else: - def callback(received, total): - log.debug('received: {} of {} bytes', received, total) - - DBusGMainLoop(set_as_default=True) state = State(candidate_filter=candidate_filter) - state.downloader.callback = callback + + for meter in args.progress: + if meter == 'dots': + state.downloader.callbacks.append(_DotsProgress().callback) + elif meter == 'json': + state.downloader.callbacks.append(_json_progress) + elif meter == 'logfile': + state.downloader.callbacks.append(_LogfileProgress(log).callback) + else: + parser.error('Unknown progress meter: {}'.format(meter)) + assert 'parser.error() does not return' # pragma: no cover + if args.dry_run: try: state.run_until('download_files') except Exception: + print('Exception occurred during dry-run; ' + 'see log file for details', + file=sys.stderr) log.exception('system-image-cli exception') return 1 # Say -c was given. This will fail. @@ -296,16 +360,20 @@ else: winning_path = [str(image.version) for image in state.winner] kws = dict(path=COLON.join(winning_path)) + target_build = state.winner[-1].version if state.channel_switch is None: # We're not switching channels due to an alias change. template = 'Upgrade path is {path}' + percentage = phased_percentage(config.channel, target_build) else: # This upgrade changes the channel that our alias is mapped # to, so include that information in the output. template = 'Upgrade path is {path} ({from} -> {to})' kws['from'], kws['to'] = state.channel_switch + percentage = phased_percentage(kws['to'], target_build) print(template.format(**kws)) - return + print('Target phase: {}%'.format(percentage)) + return 0 else: # Run the state machine to conclusion. Suppress all exceptions, but # note that the state machine will log them. If an exception occurs, @@ -313,13 +381,15 @@ log.info('running state machine [{}/{}]', config.channel, config.device) try: - if args.no_reboot: - state.run_until('reboot') + if args.no_apply or args.no_reboot: + state.run_until('apply') else: list(state) except KeyboardInterrupt: # pragma: no cover return 0 except Exception: + print('Exception occurred during update; see log file for details', + file=sys.stderr) log.exception('system-image-cli exception') return 1 else: === modified file 'systemimage/reactor.py' --- systemimage/reactor.py 2014-09-17 13:41:31 +0000 +++ systemimage/reactor.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -76,11 +76,14 @@ self._quitter = GLib.timeout_add_seconds( self._active_timeout, self._quit_with_error) - def react_to(self, signal): + def react_to(self, signal, object_path=None): signal_match = self._bus.add_signal_receiver( - self._handle_signal, signal_name=signal, + self._handle_signal, + signal_name=signal, + path=object_path, member_keyword='member', - path_keyword='path') + path_keyword='path', + ) self._signal_matches.append(signal_match) def schedule(self, method, milliseconds=50): === modified file 'systemimage/scores.py' --- systemimage/scores.py 2014-09-17 13:41:31 +0000 +++ systemimage/scores.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -26,19 +26,18 @@ import logging -from io import StringIO from itertools import count -from systemimage.helpers import MiB +from systemimage.helpers import MiB, phased_percentage + log = logging.getLogger('systemimage') - COLON = ':' class Scorer: """Abstract base class providing an API for candidate selection.""" - def choose(self, candidates): + def choose(self, candidates, channel): """Choose the candidate upgrade paths. Lowest score wins. @@ -47,10 +46,14 @@ the device from the current version to the latest version, sorted in order from oldest verson to newest. :type candidates: list of lists + :param channel: The channel being upgraded to. This is used in the + phased update calculate. + :type channel: str :return: The chosen path. :rtype: list """ if len(candidates) == 0: + log.debug('No candidates, so no winner') return [] # We want to zip together the score for each candidate path, plus the # candidate path, so that when we sort the sequence, we'll always get @@ -68,17 +71,31 @@ # Be sure that after all is said and done we return the list of Images # though! scores = sorted(zip(self.score(candidates), count(), candidates)) - fp = StringIO() - print('{} path scores (last one wins):'.format( - self.__class__.__name__), - file=fp) - for score, i, candidate in reversed(scores): - print('\t[{:4d}] -> {}'.format( + # Calculate the phase percentage for the device. Use the highest + # available build number as input into the random seed. + max_target_number = -1 + for score, i, path in scores: + # The last image will be the target image. + assert len(path) > 0, 'Empty upgrade candidate path?' + max_target_number = max(max_target_number, path[-1].version) + assert max_target_number != -1, 'No max target version?' + device_percentage = phased_percentage(channel, max_target_number) + log.debug('Device phased percentage: {}%'.format(device_percentage)) + log.debug('{} path scores:'.format(self.__class__.__name__)) + # Log the candidate paths, their scores, and their phases. + for score, i, path in reversed(scores): + log.debug('\t[{:4d}] -> {} ({}%)'.format( score, - COLON.join(str(image.version) for image in candidate)), - file=fp) - log.debug('{}'.format(fp.getvalue())) - return scores[0][2] + COLON.join(str(image.version) for image in path), + (path[-1].phased_percentage if len(path) > 0 else '--') + )) + for score, i, path in scores: + image_percentage = path[-1].phased_percentage + # An image percentage of 0 means that it's been pulled. + if image_percentage > 0 and device_percentage <= image_percentage: + return path + # No upgrade path. + return [] def score(self, candidates): # pragma: no cover """Like `choose()` except returns the candidate path scores. === modified file 'systemimage/service.py' --- systemimage/service.py 2014-09-17 13:41:31 +0000 +++ systemimage/service.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -20,7 +20,6 @@ ] -import os import sys import dbus import logging @@ -33,7 +32,8 @@ from systemimage.dbus import Loop from systemimage.helpers import makedirs from systemimage.logging import initialize -from systemimage.main import DEFAULT_CONFIG_FILE +from systemimage.main import DEFAULT_CONFIG_D + # --testing is only enabled when the systemimage.testing package is # available. This will be the case for the upstream source package, and when @@ -60,32 +60,28 @@ action='version', version='system-image-dbus {}'.format(__version__)) parser.add_argument('-C', '--config', - default=DEFAULT_CONFIG_FILE, action='store', - metavar='FILE', - help="""Use the given configuration file instead of - the default""") + default=DEFAULT_CONFIG_D, action='store', + metavar='DIRECTORY', + help="""Use the given configuration directory instead + of the default""") parser.add_argument('-v', '--verbose', default=0, action='count', help='Increase verbosity') # Hidden argument for special setup required by test environment. if instrument is not None: # pragma: no branch parser.add_argument('--testing', - default=False, action='store', + default=None, action='store', + help=argparse.SUPPRESS) + parser.add_argument('--self-signed-cert', + default=None, action='store', help=argparse.SUPPRESS) args = parser.parse_args(sys.argv[1:]) try: config.load(args.config) - except FileNotFoundError as error: - parser.error('\nConfiguration file not found: {}'.format(error)) + except TypeError as error: + parser.error('\nConfiguration directory not found: {}'.format(error)) assert 'parser.error() does not return' # pragma: no cover - # Load the optional channel.ini file, which must live next to the - # configuration file. It's okay if this file does not exist. - channel_ini = os.path.join(os.path.dirname(args.config), 'channel.ini') - try: - config.load(channel_ini, override=True) - except FileNotFoundError: - pass # Create the temporary directory if it doesn't exist. makedirs(config.system.tempdir) @@ -112,12 +108,13 @@ loop = Loop() testing_mode = getattr(args, 'testing', None) if testing_mode: - instrument(config, stack) + instrument(config, stack, args.self_signed_cert) config.dbus_service = get_service( testing_mode, system_bus, '/Service', loop) else: from systemimage.dbus import Service config.dbus_service = Service(system_bus, '/Service', loop) + try: loop.run() except KeyboardInterrupt: # pragma: no cover === modified file 'systemimage/settings.py' --- systemimage/settings.py 2014-09-17 13:41:31 +0000 +++ systemimage/settings.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify === modified file 'systemimage/state.py' --- systemimage/state.py 2014-09-17 13:41:31 +0000 +++ systemimage/state.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -35,7 +35,7 @@ from systemimage.candidates import get_candidates, iter_path from systemimage.channel import Channels from systemimage.config import config -from systemimage.download import DBusDownloadManager, Record +from systemimage.download import Record, get_download_manager from systemimage.gpg import Context, SignatureError from systemimage.helpers import ( atomic, calculate_signature, makedirs, safe_remove, temporary_directory) @@ -58,6 +58,9 @@ self.got = got self.expected = checksum + def __str__(self): # pragma: no cover + return 'got:{0.got} != exp:{0.expected}: {0.destination}'.format(self) + def _copy_if_missing(src, dstdir): dst_path = os.path.join(dstdir, os.path.basename(src)) @@ -110,7 +113,7 @@ self.files = [] self.channel_switch = None # Other public attributes. - self.downloader = DBusDownloadManager() + self.downloader = get_download_manager() self._next.append(self._cleanup) def __iter__(self): @@ -211,7 +214,7 @@ # I think it makes no sense to check the blacklist when we're # downloading a blacklist file. log.info('Looking for blacklist: {}'.format( - urljoin(config.service.https_base, url))) + urljoin(config.https_base, url))) get_keyring('blacklist', url, 'image-master') except SignatureError: log.exception('No signed blacklist found') @@ -247,7 +250,7 @@ url = 'gpg/blacklist.tar.xz' try: log.info('Looking for blacklist again: {}', - urljoin(config.service.https_base, url)) + urljoin(config.https_base, url)) get_keyring('blacklist', url, 'image-master') except FileNotFoundError: log.info('No blacklist found on second attempt') @@ -273,9 +276,9 @@ get_keyring( 'image-signing', 'gpg/image-signing.tar.xz', 'image-master', self.blacklist) - channels_url = urljoin(config.service.https_base, 'channels.json') + channels_url = urljoin(config.https_base, 'channels.json') channels_path = os.path.join(config.tempdir, 'channels.json') - asc_url = urljoin(config.service.https_base, 'channels.json.asc') + asc_url = urljoin(config.https_base, 'channels.json.asc') asc_path = os.path.join(config.tempdir, 'channels.json.asc') log.info('Looking for: {}', channels_url) with ExitStack() as stack: @@ -329,8 +332,8 @@ self._next.append(partial(self._get_index, device.index)) def _get_device_keyring(self, keyring): - keyring_url = urljoin(config.service.https_base, keyring.path) - asc_url = urljoin(config.service.https_base, keyring.signature) + keyring_url = urljoin(config.https_base, keyring.path) + asc_url = urljoin(config.https_base, keyring.signature) log.info('getting device keyring: {}', keyring_url) get_keyring( 'device-signing', (keyring_url, asc_url), 'image-signing', @@ -378,7 +381,7 @@ def _get_index(self, index): """Get and verify the index.json file.""" - index_url = urljoin(config.service.https_base, index) + index_url = urljoin(config.https_base, index) asc_url = index_url + '.asc' index_path = os.path.join(config.tempdir, 'index.json') asc_path = index_path + '.asc' @@ -410,7 +413,7 @@ # winner. Otherwise, trust the configured build number. channel = self.channels[config.channel] # channel_target is the channel we're on based on the alias mapping in - # our channel.ini file. channel_alias is the alias mapping in the + # our config files. channel_alias is the alias mapping in the # channel.json file, i.e. the channel an update will put us on. channel_target = getattr(config.service, 'channel_target', None) channel_alias = getattr(channel, 'alias', None) @@ -418,17 +421,23 @@ channel_target is None or channel_alias == channel_target): build_number = config.build_number - elif config.build_number_cli is not None: - # An explicit --build on the command line still takes precedence. - build_number = config.build_number_cli else: - # This is a channel switch caused by a new alias. - build_number = 0 + # This is a channel switch caused by a new alias. Unless the + # build number has been explicitly overridden on the command line + # via --build/-b, use build number 0 to force a full update. + build_number = (config.build_number + if config.build_number_override + else 0) self.channel_switch = (channel_target, channel_alias) candidates = get_candidates(self.index, build_number) + log.debug('Candidates from build# {}: {}'.format( + build_number, len(candidates))) if self._filter is not None: candidates = self._filter(candidates) - self.winner = config.hooks.scorer().choose(candidates) + self.winner = config.hooks.scorer().choose( + candidates, (channel_target + if channel_alias is None + else channel_alias)) # If there is no winning upgrade candidate, then there's nothing more # to do. We can skip everything between downloading the files and # doing the reboot. @@ -474,11 +483,11 @@ else: # Add the data file, which has a checksum. downloads.append(Record( - urljoin(config.service.http_base, filerec.path), + urljoin(config.http_base, filerec.path), dst, checksum)) # Add the signature file, which does not have a checksum. downloads.append(Record( - urljoin(config.service.http_base, filerec.signature), + urljoin(config.http_base, filerec.signature), asc)) signatures.append((dst, asc)) checksums.append((dst, checksum)) @@ -603,9 +612,9 @@ file=fp) # The filesystem must be unmounted. print('unmount system', file=fp) - self._next.append(self._reboot) + self._next.append(self._apply) - def _reboot(self): - log.info('rebooting') - config.hooks.reboot().reboot() + def _apply(self): + log.info('applying') + config.hooks.apply().apply() # Nothing more to do. === modified file 'systemimage/testing/controller.py' --- systemimage/testing/controller.py 2014-09-17 13:41:31 +0000 +++ systemimage/testing/controller.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -29,24 +29,31 @@ import psutil import subprocess +try: + import pycurl +except ImportError: + pycurl = None + from contextlib import ExitStack from distutils.spawn import find_executable from pkg_resources import resource_string as resource_bytes from systemimage.helpers import temporary_directory from systemimage.testing.helpers import ( - data_path, find_dbus_process, reset_envar) + data_path, find_dbus_process, makedirs, reset_envar, wait_for_service) +from unittest.mock import patch SPACE = ' ' -OVERRIDE = os.environ.get('SYSTEMIMAGE_DBUS_DAEMON_HUP_SLEEP_SECONDS') -HUP_SLEEP = (0 if OVERRIDE is None else int(OVERRIDE)) +DLSERVICE = os.environ.get( + 'SYSTEMIMAGE_DLSERVICE', + '/usr/bin/ubuntu-download-manager' + # For debugging the in-tree version of u-d-m. + #'/bin/sh $HOME/projects/phone/trunk/tools/runme.sh' + ) def start_system_image(controller): - bus = dbus.SystemBus() - service = bus.get_object('com.canonical.SystemImage', '/Service') - iface = dbus.Interface(service, 'com.canonical.SystemImage') - iface.Info() + wait_for_service(reload=False) process = find_dbus_process(controller.ini_path) if process is None: raise RuntimeError('Could not start system-image-dbus') @@ -78,12 +85,14 @@ def start_downloader(controller): - bus = dbus.SystemBus() - service = bus.get_object('com.canonical.applications.Downloader', '/') - iface = dbus.Interface( - service, 'com.canonical.applications.DownloadManager') + service = dbus.SystemBus().get_object('org.freedesktop.DBus', '/') + iface = dbus.Interface(service, 'org.freedesktop.DBus') + reply = 0 + while reply != 2: + reply = iface.StartServiceByName( + 'com.canonical.applications.Downloader', 0) + time.sleep(0.1) # Something innocuous. - iface.defaultThrottle() process = _find_udm_process() if process is None: raise RuntimeError('Could not start ubuntu-download-manager') @@ -105,24 +114,30 @@ process.wait(60) -DLSERVICE = '/usr/bin/ubuntu-download-manager' -# For debugging the in-tree version of u-d-m. -#DLSERVICE = '/bin/sh /home/barry/projects/phone/runme' - - SERVICES = [ ('com.canonical.SystemImage', - '{python} -m {self.MODULE} -C {self.ini_path} --testing {self.mode}', + '{python} -m {self.MODULE} -C {self.ini_path} ' + '{self.curl_cert} --testing {self.mode}', start_system_image, stop_system_image, ), - ('com.canonical.applications.Downloader', + ] + + +if pycurl is None: + USING_PYCURL = False +else: + USING_PYCURL = int(os.environ.get('SYSTEMIMAGE_PYCURL', '0')) + +if not USING_PYCURL: + SERVICES.append( + ('com.canonical.applications.Downloader', DLSERVICE + - ' {self.certs} -disable-timeout -stoppable -log-dir {self.tmpdir}', + ' {self.udm_certs} -disable-timeout -stoppable -log-dir {self.tmpdir}', start_downloader, stop_downloader, - ), - ] + ) + ) class Controller: @@ -131,17 +146,19 @@ MODULE = 'systemimage.testing.service' def __init__(self, logfile=None, loglevel='info'): + self.loglevel = loglevel # Non-public. self._stack = ExitStack() self._stoppers = [] # Public. self.tmpdir = self._stack.enter_context(temporary_directory()) self.config_path = os.path.join(self.tmpdir, 'dbus-system.conf') - self.ini_path = None self.serverdir = self._stack.enter_context(temporary_directory()) self.daemon_pid = None self.mode = 'live' - self.certs = '' + self.udm_certs = '' + self.curl_cert = '' + self.patcher = None # Set up the dbus-daemon system configuration file. path = data_path('dbus-system.conf.in') with open(path, 'r', encoding='utf-8') as fp: @@ -151,19 +168,27 @@ with open(self.config_path, 'w', encoding='utf-8') as fp: fp.write(config) # We need a client.ini file for the subprocess. - ini_tmpdir = self._stack.enter_context(temporary_directory()) - ini_vardir = self._stack.enter_context(temporary_directory()) - ini_logfile = (os.path.join(ini_tmpdir, 'client.log') - if logfile is None - else logfile) - self.ini_path = os.path.join(self.tmpdir, 'client.ini') + self.ini_tmpdir = self._stack.enter_context(temporary_directory()) + self.ini_vardir = self._stack.enter_context(temporary_directory()) + self.ini_logfile = (os.path.join(self.ini_tmpdir, 'client.log') + if logfile is None + else logfile) + self.ini_path = os.path.join(self.tmpdir, 'config.d') + makedirs(self.ini_path) + self._reset_configs() + + def _reset_configs(self): + for filename in os.listdir(self.ini_path): + if filename.endswith('.ini'): + os.remove(os.path.join(self.ini_path, filename)) template = resource_bytes( - 'systemimage.tests.data', 'config_03.ini').decode('utf-8') - with open(self.ini_path, 'w', encoding='utf-8') as fp: - print(template.format(tmpdir=ini_tmpdir, - vardir=ini_vardir, - logfile=ini_logfile, - loglevel=loglevel), + 'systemimage.tests.data', '01.ini').decode('utf-8') + defaults = os.path.join(self.ini_path, '00_defaults.ini') + with open(defaults, 'w', encoding='utf-8') as fp: + print(template.format(tmpdir=self.ini_tmpdir, + vardir=self.ini_vardir, + logfile=self.ini_logfile, + loglevel=self.loglevel), file=fp) def _configure_services(self): @@ -184,16 +209,41 @@ self._stoppers.append(stopper) # If the dbus-daemon is running, reload its configuration files. if self.daemon_pid is not None: - service = dbus.SystemBus().get_object('org.freedesktop.DBus', '/') - iface = dbus.Interface(service, 'org.freedesktop.DBus') - iface.ReloadConfig() - time.sleep(HUP_SLEEP) + wait_for_service() + + def _set_udm_certs(self, cert_pem, certificate_path): + self.udm_certs = ( + '' if cert_pem is None + else '-self-signed-certs ' + certificate_path) + + def _set_curl_certs(self, cert_pem, certificate_path): + # We have to set up the PyCURL downloader's self-signed certificate for + # the test in two ways. First, because we might be spawning the D-Bus + # service, we have to pass the path to the cert to that service... + self.curl_cert = ( + '' if cert_pem is None + else '--self-signed-cert ' + certificate_path) + # ...but the controller is also used to set the mode for foreground + # tests, such as test_download.py. Here we don't spawn any D-Bus + # processes, but we still have to mock make_testable() in curl.py so + # that the PyCURL object accepts the self-signed cert. + if self.patcher is not None: + self.patcher.stop() + self.patcher = None + if cert_pem is not None: + def self_sign(c): + c.setopt(pycurl.CAINFO, certificate_path) + self.patcher = patch('systemimage.curl.make_testable', self_sign) + self.patcher.start() def set_mode(self, *, cert_pem=None, service_mode=''): self.mode = service_mode - self.certs = ( - '' if cert_pem is None - else '-self-signed-certs ' + data_path(cert_pem)) + certificate_path = data_path(cert_pem) + if USING_PYCURL: + self._set_curl_certs(cert_pem, certificate_path) + else: + self._set_udm_certs(cert_pem, certificate_path) + self._reset_configs() self._configure_services() def _start(self): @@ -213,7 +263,7 @@ daemon_exe, #'/usr/lib/x86_64-linux-gnu/dbus-1.0/debug-build/bin/dbus-daemon', '--fork', - '--config-file=' + self.config_path, + '--config-file=' + str(self.config_path), # Return the address and pid on stdout. '--print-address=1', '--print-pid=1', === modified file 'systemimage/testing/dbus.py' --- systemimage/testing/dbus.py 2014-09-26 14:36:34 +0000 +++ systemimage/testing/dbus.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -23,6 +23,11 @@ import os +try: + import pycurl +except ImportError: + pycurl = None + from dbus.service import method, signal from gi.repository import GLib from systemimage.api import Mediator @@ -45,7 +50,7 @@ fp.write(SPACE.join(args[0]).strip()) -def instrument(config, stack): +def instrument(config, stack, cert_file): """Instrument the system for testing.""" # Ensure the destination directories exist. makedirs(config.updater.data_partition) @@ -54,9 +59,16 @@ # file which the testing parent process can open and read. safe_reboot = _ActionLog('reboot.log') stack.enter_context( - patch('systemimage.reboot.check_call', safe_reboot.write)) + patch('systemimage.apply.check_call', safe_reboot.write)) stack.enter_context( patch('systemimage.device.check_output', return_value='nexus7')) + # If available, patch the PyCURL downloader to accept self-signed + # certificates. + if pycurl is not None: + def self_sign(c): + c.setopt(pycurl.CAINFO, cert_file) + stack.enter_context( + patch('systemimage.curl.make_testable', self_sign)) class _LiveTestableService(Service): @@ -65,6 +77,7 @@ @log_and_exit @method('com.canonical.SystemImage') def Reset(self): + config.reload() self._api = Mediator() try: self._checking.release() @@ -72,7 +85,6 @@ # Lock is already released. pass self._update = None - self._downloading = False self._rebootable = False self._failure_count = 0 del config.build_number @@ -189,9 +201,9 @@ @method('com.canonical.SystemImage') def ApplyUpdate(self): # Always succeeds. - def _rebooting(): - self.Rebooting(True) - GLib.timeout_add(50, _rebooting) + def _applied(): + self.Applied(True) + GLib.timeout_add(50, _applied) class _UpdateManualSuccess(_UpdateAutoSuccess): @@ -259,9 +271,9 @@ @method('com.canonical.SystemImage') def ApplyUpdate(self): # The update cannot be applied. - def _rebooting(): - self.Rebooting(False) - GLib.timeout_add(50, _rebooting) + def _applied(): + self.Applied(False) + GLib.timeout_add(50, _applied) class _FailResume(Service): === modified file 'systemimage/testing/demo.py' --- systemimage/testing/demo.py 2014-02-20 23:03:24 +0000 +++ systemimage/testing/demo.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -22,12 +22,12 @@ ] +from systemimage.apply import BaseApply from systemimage.device import BaseDevice -from systemimage.reboot import BaseReboot - - -class DemoReboot(BaseReboot): - def reboot(self): + + +class DemoReboot(BaseApply): + def apply(self): print("If I was a phone, I'd be rebooting right about now.") === modified file 'systemimage/testing/helpers.py' --- systemimage/testing/helpers.py 2014-09-17 13:41:31 +0000 +++ systemimage/testing/helpers.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -23,6 +23,7 @@ 'data_path', 'debug', 'debuggable', + 'descriptions', 'find_dbus_process', 'get_channels', 'get_index', @@ -32,14 +33,18 @@ 'setup_keyring_txz', 'setup_keyrings', 'sign', + 'terminate_service', 'touch_build', + 'wait_for_service', 'write_bytes', ] import os import ssl +import dbus import json +import time import gnupg import psutil import shutil @@ -47,8 +52,8 @@ import tarfile import unittest -from contextlib import ExitStack, contextmanager -from functools import partial, wraps +from contextlib import ExitStack, contextmanager, suppress +from functools import partial, partialmethod, wraps from http.server import HTTPServer, SimpleHTTPRequestHandler from pathlib import Path from pkg_resources import resource_filename, resource_string as resource_bytes @@ -57,7 +62,6 @@ from systemimage.config import Configuration, config from systemimage.helpers import MiB, atomic, makedirs, temporary_directory from systemimage.index import Index -from systemimage.state import State from threading import Thread from unittest.mock import patch @@ -78,7 +82,7 @@ def data_path(filename): return os.path.abspath( - resource_filename('systemimage.tests.data', filename)) + resource_filename('systemimage.tests.data', filename)) def make_http_server(directory, port, certpem=None, keypem=None): @@ -117,6 +121,17 @@ except ConnectionResetError: super().handle_one_request() + def do_HEAD(self): + # Just tell the client we have the magic file. + if self.path == '/user-agent.txt': + self.send_response(200) + self.end_headers() + else: + # Canceling a download can cause our internal server to + # see various ignorable errors. No worries. + with suppress(BrokenPipeError, ConnectionResetError): + super().do_HEAD() + def do_GET(self): # If we requested the magic 'user-agent.txt' file, send back the # value of the User-Agent header. Otherwise, vend as normal. @@ -127,12 +142,10 @@ self.end_headers() self.wfile.write(user_agent.encode('utf-8')) else: - try: + # Canceling a download can cause our internal server to + # see various ignorable errors. No worries. + with suppress(BrokenPipeError, ConnectionResetError): super().do_GET() - except (BrokenPipeError, ConnectionResetError): - # Canceling a download can cause our internal server to - # see various ignorable errors. No worries. - pass # Create the server in the main thread, but start it in the sub-thread. # This lets the main thread call .shutdown() to stop everything. Return # just the shutdown method to the caller. @@ -198,47 +211,98 @@ return resources.pop_all() -def configuration(function): - """Decorator that produces a temporary configuration for testing. - - The config_00.ini template is copied to a temporary file and the the - various file system locations are filled in with the location for a, - er, temporary temporary directory. This temporary configuration - file is loaded up and the global configuration object is patched so - that all other code will see it instead of the default global - configuration object. - - Everything is properly cleaned up after the test method exits. - """ - @wraps(function) - def wrapper(*args, **kws): - with ExitStack() as resources: - etc_dir = resources.enter_context(temporary_directory()) - ini_file = os.path.join(etc_dir, 'client.ini') - temp_tmpdir = resources.enter_context(temporary_directory()) - temp_vardir = resources.enter_context(temporary_directory()) +# This defines the @configuration decorator used in various test suites to +# create a temporary config.d/ directory for a test. This is all fairly +# complicated, but here's what's going on. +# +# The _wrapper() function is the inner part of the decorator, and it does the +# heart of the operation, which is to create a temporary directory for +# config.d, along with temporary var and tmp directories. These latter two +# will be interpolated into any configuration file copied into config.d. +# +# The outer decorator function differs depending on whether @configuration was +# given without arguments, or called with arguments at the time of the +# function definition. +# +# In the former case, e.g. +# +# @configuration +# def test_something(self): +# +# The default 00.ini file is interpolated and copied into config.d. Simple. +# +# In the latter case, e.g. +# +# @configuration('some-config.ini') +# def test_something(self): +# +# There's actually another level of interior function, because the outer +# decorator itself is getting called. Here, any named configuration file is +# additionally copied to the config.d directory, renaming it sequentionally to +# something like 01_override.ini, with the numeric part incrementing +# monotonically. +# +# The implementation is tricky because we want the call sites to be simple. +def _wrapper(self, function, ini_files, *args, **kws): + start = 0 + with ExitStack() as resources: + # Create the config.d directory and copy all the source ini files to + # this directory in sequential order, interpolating in the temporary + # tmp and var directories. + config_d = resources.enter_context(temporary_directory()) + temp_tmpdir = resources.enter_context(temporary_directory()) + temp_vardir = resources.enter_context(temporary_directory()) + for ini_file in ini_files: + dst = os.path.join(config_d, '{:02d}_override.ini'.format(start)) + start += 1 template = resource_bytes( - 'systemimage.tests.data', 'config_00.ini').decode('utf-8') - with atomic(ini_file) as fp: + 'systemimage.tests.data', ini_file).decode('utf-8') + with atomic(dst) as fp: print(template.format(tmpdir=temp_tmpdir, vardir=temp_vardir), file=fp) - config = Configuration(ini_file) - resources.enter_context( - patch('systemimage.config._config', config)) - resources.enter_context( - patch('systemimage.device.check_output', - return_value='nexus7')) - # Make sure the cache_partition and data_partition exist. - makedirs(config.updater.cache_partition) - makedirs(config.updater.data_partition) - # The method under test is allowed to specify some additional - # keyword arguments, in order to pass some variables in from the - # wrapper. - signature = inspect.signature(function) - if 'ini_file' in signature.parameters: - kws['ini_file'] = ini_file - return function(*args, **kws) - return wrapper + # Patch the global configuration object so that it can be used + # directly, which is good enough in most cases. Also patch the bit of + # code that detects the device name. + config = Configuration(config_d) + resources.enter_context( + patch('systemimage.config._config', config)) + resources.enter_context( + patch('systemimage.device.check_output', + return_value='nexus7')) + # Make sure the cache_partition and data_partition exist. + makedirs(config.updater.cache_partition) + makedirs(config.updater.data_partition) + # The method under test is allowed to specify some additional + # keyword arguments, in order to pass some variables in from the + # wrapper. + signature = inspect.signature(function) + if 'config_d' in signature.parameters: + kws['config_d'] = config_d + if 'config' in signature.parameters: + kws['config'] = config + # Call the function with the given arguments and return the result. + return function(self, *args, **kws) + + +def configuration(*args): + """Outer decorator which can be called or not at function definition time. + + If called, the arguments are positional only, and name the test data .ini + files which are to be copied to config.d directory. If none are given, + then 00.ini is used. + """ + if len(args) == 1 and callable(args[0]): + # We assume this was the bare @configuration decorator flavor. + function = args[0] + inner = partialmethod(_wrapper, function, ('00.ini',)) + return wraps(function)(inner) + else: + # We assume this was the called @configuration(...) decorator flavor, + # so create the actual decorator that wraps the _wrapper function. + def decorator(function): + inner = partialmethod(_wrapper, function, args) + return wraps(function)(inner) + return decorator def sign(filename, pubkey_ring): @@ -249,6 +313,8 @@ with. This keyring must contain only one key, and its key id must exist in the master secret keyring. """ + # filename could be a Path object. For now, just str-ify it. + filename = str(filename) with ExitStack() as resources: home = resources.enter_context(temporary_directory()) secring = data_path('master-secring.gpg') @@ -268,7 +334,7 @@ def copy(filename, todir, dst=None): src = data_path(filename) - dst = os.path.join(todir, filename if dst is None else dst) + dst = os.path.join(str(todir), filename if dst is None else dst) makedirs(os.path.dirname(dst)) shutil.copy(src, dst) @@ -395,21 +461,24 @@ os.chmod(path, old_mode) -def touch_build(version, timestamp=None): +def touch_build(version, timestamp=None, use_config=None): # LP: #1220238 - assert that no old-style version numbers are being used. assert 0 <= version < (1 << 16), ( - 'old style version number: {}'.format(version)) - with open(config.system.build_file, 'w', encoding='utf-8') as fp: - print(version, file=fp) + 'Old style version number: {}'.format(version)) + if use_config is None: + use_config = config + override = Path(use_config.config_d) / '99_build.ini' + with override.open('wt', encoding='utf-8') as fp: + print("""\ +[service] +build_number: {} +""".format(version), file=fp) + # We have to touch the mtimes for all the files in the config directory. if timestamp is not None: timestamp = int(timestamp) - os.utime(config.system.build_file, (timestamp, timestamp)) - channel_ini = os.path.join( - os.path.dirname(config.config_file), 'channel.ini') - try: - os.utime(channel_ini, (timestamp, timestamp)) - except FileNotFoundError: - pass + for path in Path(use_config.config_d).iterdir(): + os.utime(str(path), (timestamp, timestamp)) + use_config.reload() def write_bytes(path, size_in_mib): @@ -431,13 +500,13 @@ @contextmanager -def debug(*, check_flag=False): +def debug(*, check_flag=False, end='\n'): if not check_flag or os.path.exists('/tmp/debug.enabled'): path = Path('/tmp/debug.log') else: path = Path(os.devnull) with path.open('a', encoding='utf-8') as fp: - function = partial(print, file=fp) + function = partial(print, file=fp, end=end) function.fp = fp yield function fp.flush() @@ -487,6 +556,8 @@ SystemImagePlugin.controller.set_mode(cert_pem='cert.pem') def setUp(self): + # Avoid circular imports. + from systemimage.state import State self._resources = ExitStack() self._state = State() try: @@ -544,3 +615,44 @@ dict(type='device-signing'), os.path.join(self._serverdir, self.CHANNEL, self.DEVICE, 'device-signing.tar.xz')) + + +def descriptions(path): + descriptions = [] + for image in path: + # There's only one description per image so order doesn't + # matter. + descriptions.extend(image.descriptions.values()) + return descriptions + + +def wait_for_service(*, restart=False, reload=True): + bus = dbus.SystemBus() + if restart: + service = bus.get_object('com.canonical.SystemImage', '/Service') + iface = dbus.Interface(service, 'com.canonical.SystemImage') + iface.Exit() + service = dbus.SystemBus().get_object('org.freedesktop.DBus', '/') + iface = dbus.Interface(service, 'org.freedesktop.DBus') + if reload: + iface.ReloadConfig() + # Wait until the system-image-dbus process is actually running. + # http://people.freedesktop.org/~david/eggdbus-20091014/eggdbus-interface-org.freedesktop.DBus.html#eggdbus-method-org.freedesktop.DBus.StartServiceByName + reply = 0 + # 2015-03-09 BAW: This could potentially spin forever, but we'll assume + # D-Bus eventually is successful in starting the service. + while reply != 2: + reply = iface.StartServiceByName('com.canonical.SystemImage', 0) + time.sleep(0.1) + + +def terminate_service(): + # Avoid circular imports. + from systemimage.testing.nose import SystemImagePlugin + proc = find_dbus_process(SystemImagePlugin.controller.ini_path) + if proc is not None: + bus = dbus.SystemBus() + service = bus.get_object('com.canonical.SystemImage', '/Service') + iface = dbus.Interface(service, 'com.canonical.SystemImage') + iface.Exit() + proc.wait() === modified file 'systemimage/testing/nose.py' --- systemimage/testing/nose.py 2014-09-26 14:36:34 +0000 +++ systemimage/testing/nose.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -90,7 +90,7 @@ 'Set the log file for the test run', nargs=1) def set_dbus_loglevel(level): - self.log_level = 'info:{}'.format(level[0]) + self.log_level = level[0] self.addOption(set_dbus_loglevel, 'M', 'loglevel', 'Set the systemimage.dbus log level', nargs=1) @@ -149,3 +149,8 @@ ## from systemimage.testing.helpers import debug ## with debug() as dlog: ## dlog('^^^^^', event.test) + + def describeTest(self, event): + # This is fucked up. + if 'partial' in event.description: + event.description = event.description[:-73] === added file 'systemimage/testing/service.py' --- systemimage/testing/service.py 1970-01-01 00:00:00 +0000 +++ systemimage/testing/service.py 2015-05-20 14:55:53 +0000 @@ -0,0 +1,65 @@ +# Copyright (C) 2014-2015 Canonical Ltd. +# Author: Barry Warsaw + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 3 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +"""DBus service testing pre-load module. + +This is arranged so that the test suite can enable code coverage data +collection as early as possible in the private bus D-Bus activated processes. +""" + +# Uncomment this if the controller won't start. There's no other good way to +# get debugging information about the D-Bus activated process, since their +# stderr just seems to get lost. +## import sys +## sys.stderr = open('/tmp/debug.log', 'a', encoding='utf-8') + + +import os + +# Set this environment variable if the controller won't start. There's no +# other good way to get debugging information about the D-Bus activated +# process, since their stderr just seems to get lost. +if os.environ.get('SYSTEMIMAGE_DEBUG_DBUS_ACTIVATION'): + import sys + sys.stderr = open('/tmp/debug.log', 'a', encoding='utf-8') + + +# It's okay if this module isn't available. +try: + from coverage.control import coverage as _Coverage +except ImportError: + _Coverage = None + + +def main(): + # Enable code coverage. + ini_file = os.environ.get('COVERAGE_PROCESS_START') + if _Coverage is not None and ini_file is not None: + coverage =_Coverage(config_file=ini_file, auto_data=True) + # Stolen from coverage.process_startup() + coverage.erase() + coverage.start() + coverage._warn_no_data = False + coverage._warn_unimported_source = False + # All systemimage imports happen here so that we have the best possible + # chance of instrumenting all relevant code. + from systemimage.service import main as real_main + # Now run the actual D-Bus service. + return real_main() + + +if __name__ == '__main__': + import sys + sys.exit(main()) === removed file 'systemimage/testing/service.py' --- systemimage/testing/service.py 2014-09-17 02:58:58 +0000 +++ systemimage/testing/service.py 1970-01-01 00:00:00 +0000 @@ -1,50 +0,0 @@ -# Copyright (C) 2014 Canonical Ltd. -# Author: Barry Warsaw - -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; version 3 of the License. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -"""DBus service testing pre-load module. - -This is arranged so that the test suite can enable code coverage data -collection as early as possible in the private bus D-Bus activated processes. -""" - -import os - -# It's okay if this module isn't available. -try: - from coverage.control import coverage as _Coverage -except ImportError: - _Coverage = None - - -def main(): - # Enable code coverage. - ini_file = os.environ.get('COVERAGE_PROCESS_START') - if _Coverage is not None and ini_file is not None: - coverage =_Coverage(config_file=ini_file, auto_data=True) - # Stolen from coverage.process_startup() - coverage.erase() - coverage.start() - coverage._warn_no_data = False - coverage._warn_unimported_source = False - # All systemimage imports happen here so that we have the best possible - # chance of instrumenting all relevant code. - from systemimage.service import main as real_main - # Now run the actual D-Bus service. - return real_main() - - -if __name__ == '__main__': - import sys - sys.exit(main()) === renamed file 'systemimage/tests/data/config_00.ini' => 'systemimage/tests/data/00.ini' --- systemimage/tests/data/config_00.ini 2014-01-30 15:41:03 +0000 +++ systemimage/tests/data/00.ini 2015-05-20 14:55:53 +0000 @@ -12,7 +12,6 @@ [system] timeout: 1s -build_file: {tmpdir}/ubuntu-build tempdir: {tmpdir}/tmp logfile: {tmpdir}/client.log loglevel: info @@ -31,7 +30,7 @@ [hooks] device: systemimage.device.SystemProperty scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot +apply: systemimage.apply.Reboot [dbus] lifetime: 2m === renamed file 'systemimage/tests/data/config_03.ini' => 'systemimage/tests/data/01.ini' --- systemimage/tests/data/config_03.ini 2014-09-17 13:41:31 +0000 +++ systemimage/tests/data/01.ini 2015-05-20 14:55:53 +0000 @@ -12,14 +12,13 @@ [system] timeout: 1s -build_file: {tmpdir}/ubuntu-build tempdir: {tmpdir}/tmp logfile: {logfile} loglevel: {loglevel} settings_db: {vardir}/settings.db [gpg] -archive_master: {vardir}/etc/archive-master.tar.xz +archive_master: {vardir}/usr/share/system-image/archive-master.tar.xz image_master: {vardir}/keyrings/image-master.tar.xz image_signing: {vardir}/keyrings/image-signing.tar.xz device_signing: {vardir}/keyrings/device-signing.tar.xz @@ -31,7 +30,7 @@ [hooks] device: systemimage.testing.demo.TestingDevice scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot +apply: systemimage.apply.Reboot [dbus] lifetime: 5m === added file 'systemimage/tests/data/api.channels_01.json' --- systemimage/tests/data/api.channels_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/api.channels_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,13 @@ +{ + "stable": { + "devices": { + "nexus7": { + "index": "/stable/nexus7/index.json", + "keyring": { + "path": "/stable/nexus7/device-signing.tar.xz", + "signature": "/stable/nexus7/device-signing.tar.xz.asc" + } + } + } + } +} === added file 'systemimage/tests/data/api.index_01.json' --- systemimage/tests/data/api.index_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/api.index_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,36 @@ +{ + "global": { + "generated_at": "Thu Aug 01 08:01:00 UTC 2013" + }, + "images": [ + { + "description": "Full", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 3, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 2, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1600, + "bootme": true + } + ] +} === added file 'systemimage/tests/data/api.index_02.json' --- systemimage/tests/data/api.index_02.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/api.index_02.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,251 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "abc", + "order": 1, + "path": "/a/b/c.txt", + "signature": "/a/b/c.txt.asc", + "size": 104857600 + + }, + { + "checksum": "bcd", + "order": 1, + "path": "/b/c/d.txt", + "signature": "/b/c/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "cde", + "order": 1, + "path": "/c/d/e.txt", + "signature": "/c/d/e.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1300 + }, + { + "base": 1300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "def", + "order": 1, + "path": "/d/e/f.txt", + "signature": "/d/e/f.txt.asc", + "size": 104857600 + }, + { + "checksum": "ef0", + "order": 1, + "path": "/e/f/0.txt", + "signature": "/e/f/0.txt.asc", + "size": 104857600 + }, + { + "checksum": "f01", + "order": 1, + "path": "/f/e/1.txt", + "signature": "/f/e/1.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1301 + }, + { + "base": 1301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "012", + "order": 1, + "path": "/0/1/2.txt", + "signature": "/0/1/2.txt.asc", + "size": 104857600 + }, + { + "checksum": "123", + "order": 1, + "path": "/1/2/3.txt", + "signature": "/1/2/3.txt.asc", + "size": 104857600 + }, + { + "checksum": "234", + "order": 1, + "path": "/2/3/4.txt", + "signature": "/2/3/4.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1304 + }, + + { + "description": "Full B", + "description-en": "The full B", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 1, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 10000 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 10001 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 1, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 10002 + } + ], + "type": "full", + "version": 1200 + }, + { + "base": 1200, + "description": "Delta B.1", + "description-en_US": "This is the delta B.1", + "description-xx": "XX This is the delta B.1", + "description-yy": "YY This is the delta B.1", + "description-yy_ZZ": "YY-ZZ This is the delta B.1", + "files": [ + { + "checksum": "cebe3d9d614ba5c19f633566104315854a11353a333bf96f16b5afa0e90abdc4", + "order": 1, + "path": "/6/7/8.txt", + "signature": "/6/7/8.txt.asc", + "size": 20000 + }, + { + "checksum": "35a9e381b1a27567549b5f8a6f783c167ebf809f1c4d6a9e367240484d8ce281", + "order": 1, + "path": "/7/8/9.txt", + "signature": "/7/8/9.txt.asc", + "size": 20001 + }, + { + "checksum": "6bd6c3f7808391e8b74f5c2d58810809eda5c134aaa7f1b27ddf4b445c421ac5", + "order": 1, + "path": "/8/9/a.txt", + "signature": "/8/9/a.txt.asc", + "size": 20002 + } + ], + "type": "delta", + "version": 1201 + }, + { + "base": 1201, + "description": "Delta B.2", + "description-xx": "Oh delta, my delta", + "description-xx_CC": "This hyar is the delta B.2", + "files": [ + { + "checksum": "8c43d75d5b9f1aa9fc3fabb6b60b6c06553324352399a33febce95a1b588d1d6", + "order": 1, + "path": "/9/a/b.txt", + "signature": "/9/a/b.txt.asc", + "size": 30000 + }, + { + "checksum": "20e796c128096d229ba89bf412a53c3151d170a409c2c8c1dd8e414087b7ffae", + "order": 1, + "path": "/f/e/d.txt", + "signature": "/f/e/d.txt.asc", + "size": 30001 + }, + { + "checksum": "278238e8bafa4709c77aa723e168101acd6ee1fb9fcc1b6eca4762e5c7dad768", + "order": 1, + "path": "/e/d/c.txt", + "signature": "/e/d/c.txt.asc", + "size": 30002 + + } + ], + "type": "delta", + "version": 1304 + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "dcb", + "order": 1, + "path": "/d/c/b.txt", + "signature": "/d/c/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "cba", + "order": 1, + "path": "/c/b/a.txt", + "signature": "/c/b/a.txt.asc", + "size": 104857600 + }, + { + "checksum": "ba9", + "order": 1, + "path": "/b/a/9.txt", + "signature": "/b/a/9.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1100 + }, + { + "base": 1100, + "description": "Delta C.1", + "files": [ + { + "checksum": "a98", + "order": 1, + "path": "/a/9/8.txt", + "signature": "/a/9/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "987", + "order": 1, + "path": "/9/8/7.txt", + "signature": "/9/8/7.txt.asc", + "size": 104857600 + }, + { + "checksum": "876", + "order": 1, + "path": "/8/7/6.txt", + "signature": "/8/7/6.txt.asc", + "size": 838860800 + + } + ], + "type": "delta", + "version": 1303 + } + ] +} === added file 'systemimage/tests/data/api.index_03.json' --- systemimage/tests/data/api.index_03.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/api.index_03.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,37 @@ +{ + "global": { + "generated_at": "Thu Aug 01 08:01:00 UTC 2013" + }, + "images": [ + { + "description": "Full", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 3, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 2, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1600, + "version_detail": "ubuntu=101,raw-device=201,version=301", + "bootme": true + } + ] +} === added file 'systemimage/tests/data/candidates.index_01.json' --- systemimage/tests/data/candidates.index_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/candidates.index_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,6 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [] +} === added file 'systemimage/tests/data/candidates.index_02.json' --- systemimage/tests/data/candidates.index_02.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/candidates.index_02.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,23 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "New full build 1", + "files": [], + "minversion": 600, + "type": "full", + "version": 1300 + }, + { + "bootme": true, + "description": "New full build 2", + "files": [], + "minversion": 1100, + "type": "full", + "version": 1400 + } + ] +} === renamed file 'systemimage/tests/data/index_05.json' => 'systemimage/tests/data/candidates.index_03.json' === renamed file 'systemimage/tests/data/index_03.json' => 'systemimage/tests/data/candidates.index_04.json' === renamed file 'systemimage/tests/data/index_04.json' => 'systemimage/tests/data/candidates.index_05.json' === renamed file 'systemimage/tests/data/index_06.json' => 'systemimage/tests/data/candidates.index_06.json' === renamed file 'systemimage/tests/data/index_07.json' => 'systemimage/tests/data/candidates.index_07.json' === added file 'systemimage/tests/data/candidates.index_08.json' --- systemimage/tests/data/candidates.index_08.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/candidates.index_08.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,244 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "abc", + "order": 1, + "path": "/a/b/c.txt", + "signature": "/a/b/c.txt.asc", + "size": 104857600 + + }, + { + "checksum": "bcd", + "order": 1, + "path": "/b/c/d.txt", + "signature": "/b/c/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "cde", + "order": 1, + "path": "/c/d/e.txt", + "signature": "/c/d/e.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1300 + }, + { + "base": 1300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "def", + "order": 1, + "path": "/d/e/f.txt", + "signature": "/d/e/f.txt.asc", + "size": 104857600 + }, + { + "checksum": "ef0", + "order": 1, + "path": "/e/f/0.txt", + "signature": "/e/f/0.txt.asc", + "size": 104857600 + }, + { + "checksum": "f01", + "order": 1, + "path": "/f/e/1.txt", + "signature": "/f/e/1.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1301 + }, + { + "base": 1301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "012", + "order": 1, + "path": "/0/1/2.txt", + "signature": "/0/1/2.txt.asc", + "size": 104857600 + }, + { + "checksum": "123", + "order": 1, + "path": "/1/2/3.txt", + "signature": "/1/2/3.txt.asc", + "size": 104857600 + }, + { + "checksum": "234", + "order": 1, + "path": "/2/3/4.txt", + "signature": "/2/3/4.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1304 + }, + + { + "description": "Full B", + "files": [ + { + "checksum": "345", + "order": 1, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "456", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "567", + "order": 1, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1200 + }, + { + "base": 1200, + "description": "Delta B.1", + "files": [ + { + "checksum": "678", + "order": 1, + "path": "/6/7/8.txt", + "signature": "/6/7/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "789", + "order": 1, + "path": "/7/8/9.txt", + "signature": "/7/8/9.txt.asc", + "size": 104857600 + }, + { + "checksum": "89a", + "order": 1, + "path": "/8/9/a.txt", + "signature": "/8/9/a.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1201 + }, + { + "base": 1201, + "description": "Delta B.2", + "files": [ + { + "checksum": "9ab", + "order": 1, + "path": "/9/a/b.txt", + "signature": "/9/a/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "fed", + "order": 1, + "path": "/f/e/d.txt", + "signature": "/f/e/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "edc", + "order": 1, + "path": "/e/d/c.txt", + "signature": "/e/d/c.txt.asc", + "size": 209715200 + + } + ], + "type": "delta", + "version": 1304 + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "dcb", + "order": 1, + "path": "/d/c/b.txt", + "signature": "/d/c/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "cba", + "order": 1, + "path": "/c/b/a.txt", + "signature": "/c/b/a.txt.asc", + "size": 104857600 + }, + { + "checksum": "ba9", + "order": 1, + "path": "/b/a/9.txt", + "signature": "/b/a/9.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1100 + }, + { + "base": 1100, + "description": "Delta C.1", + "files": [ + { + "checksum": "a98", + "order": 1, + "path": "/a/9/8.txt", + "signature": "/a/9/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "987", + "order": 1, + "path": "/9/8/7.txt", + "signature": "/9/8/7.txt.asc", + "size": 104857600 + }, + { + "checksum": "876", + "order": 1, + "path": "/8/7/6.txt", + "signature": "/8/7/6.txt.asc", + "size": 838860800 + + } + ], + "type": "delta", + "version": 1303 + } + ] +} === renamed file 'systemimage/tests/data/index_11.json' => 'systemimage/tests/data/candidates.index_09.json' === added file 'systemimage/tests/data/candidates.index_10.json' --- systemimage/tests/data/candidates.index_10.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/candidates.index_10.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,36 @@ +{ + "global": { + "generated_at": "Thu Aug 01 08:01:00 UTC 2013" + }, + "images": [ + { + "description": "Full", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 3, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 2, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1600, + "bootme": true + } + ] +} === added file 'systemimage/tests/data/candidates.index_11.json' --- systemimage/tests/data/candidates.index_11.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/candidates.index_11.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,37 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "base": 100, + "description": "Delta", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 3, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 2, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1600, + "bootme": true + } + ] +} === renamed file 'systemimage/tests/data/index_19.json' => 'systemimage/tests/data/candidates.index_12.json' === added file 'systemimage/tests/data/candidates.index_13.json' --- systemimage/tests/data/candidates.index_13.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/candidates.index_13.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,244 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "abc", + "order": 1, + "path": "/a/b/c.txt", + "signature": "/a/b/c.txt.asc", + "size": 104857600 + + }, + { + "checksum": "bcd", + "order": 1, + "path": "/b/c/d.txt", + "signature": "/b/c/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "cde", + "order": 1, + "path": "/c/d/e.txt", + "signature": "/c/d/e.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 300 + }, + { + "base": 300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "def", + "order": 1, + "path": "/d/e/f.txt", + "signature": "/d/e/f.txt.asc", + "size": 104857600 + }, + { + "checksum": "ef0", + "order": 1, + "path": "/e/f/0.txt", + "signature": "/e/f/0.txt.asc", + "size": 104857600 + }, + { + "checksum": "f01", + "order": 1, + "path": "/f/e/1.txt", + "signature": "/f/e/1.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 301 + }, + { + "base": 301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "012", + "order": 1, + "path": "/0/1/2.txt", + "signature": "/0/1/2.txt.asc", + "size": 104857600 + }, + { + "checksum": "123", + "order": 1, + "path": "/1/2/3.txt", + "signature": "/1/2/3.txt.asc", + "size": 104857600 + }, + { + "checksum": "234", + "order": 1, + "path": "/2/3/4.txt", + "signature": "/2/3/4.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 304 + }, + + { + "description": "Full B", + "files": [ + { + "checksum": "345", + "order": 1, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "456", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "567", + "order": 1, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 200 + }, + { + "base": 200, + "description": "Delta B.1", + "files": [ + { + "checksum": "678", + "order": 1, + "path": "/6/7/8.txt", + "signature": "/6/7/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "789", + "order": 1, + "path": "/7/8/9.txt", + "signature": "/7/8/9.txt.asc", + "size": 104857600 + }, + { + "checksum": "89a", + "order": 1, + "path": "/8/9/a.txt", + "signature": "/8/9/a.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 201 + }, + { + "base": 201, + "description": "Delta B.2", + "files": [ + { + "checksum": "9ab", + "order": 1, + "path": "/9/a/b.txt", + "signature": "/9/a/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "fed", + "order": 1, + "path": "/f/e/d.txt", + "signature": "/f/e/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "edc", + "order": 1, + "path": "/e/d/c.txt", + "signature": "/e/d/c.txt.asc", + "size": 209715200 + + } + ], + "type": "delta", + "version": 304 + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "dcb", + "order": 1, + "path": "/d/c/b.txt", + "signature": "/d/c/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "cba", + "order": 1, + "path": "/c/b/a.txt", + "signature": "/c/b/a.txt.asc", + "size": 104857600 + }, + { + "checksum": "ba9", + "order": 1, + "path": "/b/a/9.txt", + "signature": "/b/a/9.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 100 + }, + { + "base": 100, + "description": "Delta C.1", + "files": [ + { + "checksum": "a98", + "order": 1, + "path": "/a/9/8.txt", + "signature": "/a/9/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "987", + "order": 1, + "path": "/9/8/7.txt", + "signature": "/9/8/7.txt.asc", + "size": 104857600 + }, + { + "checksum": "876", + "order": 1, + "path": "/8/7/6.txt", + "signature": "/8/7/6.txt.asc", + "size": 838860800 + + } + ], + "type": "delta", + "version": 303 + } + ] +} === added file 'systemimage/tests/data/channel.channels_01.json' --- systemimage/tests/data/channel.channels_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/channel.channels_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,23 @@ +{ + "daily": { + "devices": { + "nexus7": { + "index": "/daily/nexus7/index.json", + "keyring": { + "path": "/daily/nexus7/device-keyring.tar.xz", + "signature": "/daily/nexus7/device-keyring.tar.xz.asc" + } + }, + "nexus4":{ + "index": "/daily/nexus4/index.json" + } + } + }, + "stable": { + "devices": { + "nexus7":{ + "index": "/stable/nexus7/index.json" + } + } + } +} === added file 'systemimage/tests/data/channel.channels_02.json' --- systemimage/tests/data/channel.channels_02.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/channel.channels_02.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,38 @@ +{ + "daily": { + "devices": { + "grouper": { + "index": "/daily/grouper/index.json" + }, + "maguro": { + "index": "/daily/maguro/index.json" + }, + "mako": { + "index": "/daily/mako/index.json" + }, + "manta": { + "index": "/daily/manta/index.json" + } + } + }, + "daily-proposed": { + "devices": { + "grouper": { + "index": "/daily-proposed/grouper/index.json", + "keyring": { + "path": "/daily-proposed/grouper/device-signing.tar.xz", + "signature": "/daily-proposed/grouper/device-signing.tar.xz.asc" + } + }, + "maguro": { + "index": "/daily-proposed/maguro/index.json" + }, + "mako": { + "index": "/daily-proposed/mako/index.json" + }, + "manta": { + "index": "/daily-proposed/manta/index.json" + } + } + } +} === added file 'systemimage/tests/data/channel.channels_03.json' --- systemimage/tests/data/channel.channels_03.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/channel.channels_03.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,70 @@ +{ + "13.10": { + "devices": { + "grouper": { + "index": "/13.10/grouper/index.json" + }, + "maguro": { + "index": "/13.10/maguro/index.json" + }, + "mako": { + "index": "/13.10/mako/index.json" + }, + "manta": { + "index": "/13.10/manta/index.json" + } + } + }, + "13.10-proposed": { + "devices": { + "grouper": { + "index": "/13.10-proposed/grouper/index.json" + }, + "maguro": { + "index": "/13.10-proposed/maguro/index.json" + }, + "mako": { + "index": "/13.10-proposed/mako/index.json" + }, + "manta": { + "index": "/13.10-proposed/manta/index.json" + } + } + }, + "14.04": { + "devices": { + "grouper": { + "index": "/14.04/grouper/index.json" + }, + "maguro": { + "index": "/14.04/maguro/index.json" + }, + "mako": { + "index": "/14.04/mako/index.json" + }, + "manta": { + "index": "/14.04/manta/index.json" + } + } + }, + "14.04-proposed": { + "devices": { + "grouper": { + "index": "/14.04-proposed/grouper/index.json", + "keyring": { + "path": "/14.04-proposed/grouper/device-signing.tar.xz", + "signature": "/14.04-proposed/grouper/device-signing.tar.xz.asc" + } + }, + "maguro": { + "index": "/14.04-proposed/maguro/index.json" + }, + "mako": { + "index": "/14.04-proposed/mako/index.json" + }, + "manta": { + "index": "/14.04-proposed/manta/index.json" + } + } + } +} === added file 'systemimage/tests/data/channel.channels_04.json' --- systemimage/tests/data/channel.channels_04.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/channel.channels_04.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,56 @@ +{ + "daily": { + "alias": "saucy", + "devices": { + "grouper": { + "index": "/daily/grouper/index.json" + }, + "maguro": { + "index": "/daily/maguro/index.json" + }, + "mako": { + "index": "/daily/mako/index.json" + }, + "manta": { + "index": "/daily/manta/index.json" + } + } + }, + "saucy": { + "devices": { + "grouper": { + "index": "/saucy/grouper/index.json" + }, + "maguro": { + "index": "/saucy/maguro/index.json" + }, + "mako": { + "index": "/saucy/mako/index.json" + }, + "manta": { + "index": "/saucy/manta/index.json", + "keyring": { + "path": "/saucy/manta/device-signing.tar.xz", + "signature": "/saucy/manta/device-signing.tar.xz.asc" + } + } + } + }, + "saucy-proposed": { + "hidden": true, + "devices": { + "grouper": { + "index": "/saucy-proposed/grouper/index.json" + }, + "maguro": { + "index": "/saucy-proposed/maguro/index.json" + }, + "mako": { + "index": "/saucy-proposed/mako/index.json" + }, + "manta": { + "index": "/saucy-proposed/manta/index.json" + } + } + } +} === added file 'systemimage/tests/data/channel.channels_05.json' --- systemimage/tests/data/channel.channels_05.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/channel.channels_05.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,23 @@ +{ + "daily": { + "devices": { + "nexus7": { + "index": "/daily/nexus7/index.json", + "keyring": { + "path": "/daily/nexus7/device-keyring.tar.xz", + "signature": "/daily/nexus7/device-keyring.tar.xz.asc" + } + }, + "nexus4":{ + "index": "/daily/nexus4/index.json" + } + } + }, + "stable": { + "devices": { + "nexus7":{ + "index": "/stable/nexus7/index.json" + } + } + } +} === removed file 'systemimage/tests/data/channel_06.ini' --- systemimage/tests/data/channel_06.ini 2014-09-17 02:58:58 +0000 +++ systemimage/tests/data/channel_06.ini 1970-01-01 00:00:00 +0000 @@ -1,8 +0,0 @@ -[service] -base: localhost -http_port: 8980 -https_port: 8943 -channel: daily -build_number: 300 -channel_target: saucy -device: shoephone === removed file 'systemimage/tests/data/channel_07.ini' --- systemimage/tests/data/channel_07.ini 2014-09-17 02:58:58 +0000 +++ systemimage/tests/data/channel_07.ini 1970-01-01 00:00:00 +0000 @@ -1,8 +0,0 @@ -[service] -base: localhost -http_port: 8980 -https_port: 8943 -channel: daily -build_number: 300 -channel_target: saucy -device: === added file 'systemimage/tests/data/config.config_01.ini' --- systemimage/tests/data/config.config_01.ini 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/config.config_01.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,34 @@ +# Configuration file for specifying relatively static information about the +# upgrade resolution process. + +[service] +base: phablet.example.com +http_port: 80 +https_port: 443 +channel: stable +build_number: 0 + +[system] +timeout: 10s +tempdir: /tmp +logfile: /var/log/system-image/client.log +loglevel: error +settings_db: /var/lib/phablet/settings.db + +[gpg] +archive_master: /usr/share/phablet/archive-master.tar.xz +image_master: /etc/phablet/image-master.tar.xz +image_signing: /var/lib/phablet/image-signing.tar.xz +device_signing: /var/lib/phablet/device-signing.tar.xz + +[updater] +cache_partition: {tmpdir}/android/cache +data_partition: {vardir}/lib/phablet/updater + +[hooks] +device: systemimage.device.SystemProperty +scorer: systemimage.scores.WeightedScorer +apply: systemimage.apply.Reboot + +[dbus] +lifetime: 2m === added file 'systemimage/tests/data/config.config_02.ini' --- systemimage/tests/data/config.config_02.ini 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/config.config_02.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,34 @@ +# Configuration file for specifying relatively static information about the +# upgrade resolution process. + +[service] +base: phablet.example.com +http_port: 80 +https_port: 443 +channel: stable +build_number: 0 + +[system] +timeout: 10s +tempdir: /tmp +logfile: /var/log/system-image/client.log +loglevel: critical:debug +settings_db: /var/lib/phablet/settings.db + +[gpg] +archive_master: /etc/phablet/archive-master.tar.xz +image_master: /etc/phablet/image-master.tar.xz +image_signing: /var/lib/phablet/image-signing.tar.xz +device_signing: /var/lib/phablet/device-signing.tar.xz + +[updater] +cache_partition: {tmpdir}/android/cache +data_partition: {vardir}/lib/phablet/updater + +[hooks] +device: systemimage.device.SystemProperty +scorer: systemimage.scores.WeightedScorer +apply: systemimage.apply.Reboot + +[dbus] +lifetime: 2m === renamed file 'systemimage/tests/data/config_02.ini' => 'systemimage/tests/data/config.config_03.ini' --- systemimage/tests/data/config_02.ini 2014-01-30 15:41:03 +0000 +++ systemimage/tests/data/config.config_03.ini 2015-05-20 14:55:53 +0000 @@ -11,7 +11,6 @@ [system] timeout: 10s -build_file: /etc/ubuntu-build tempdir: /tmp logfile: /var/log/system-image/client.log loglevel: error @@ -24,13 +23,13 @@ device_signing: /var/lib/phablet/device-signing.tar.xz [updater] -cache_partition: /android/cache -data_partition: /var/lib/phablet/updater +cache_partition: {tmpdir}/android/cache +data_partition: {vardir}/lib/phablet/updater [hooks] device: systemimage.device.SystemProperty scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot +apply: systemimage.apply.Reboot [dbus] lifetime: 3s === added file 'systemimage/tests/data/config.config_04.ini' --- systemimage/tests/data/config.config_04.ini 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/config.config_04.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,36 @@ +# TEMPLATE configuration file for specifying relatively static information +# about the upgrade resolution process. +# +# This is used by the DBus tests. + +[service] +base: localhost +http_port: 8980 +https_port: 8943 +channel: stable +build_number: 0 + +[system] +timeout: 1s +tempdir: {tmpdir}/tmp +logfile: {logfile} +loglevel: {loglevel} +settings_db: {vardir}/settings.db + +[gpg] +archive_master: {vardir}/etc/archive-master.tar.xz +image_master: {vardir}/keyrings/image-master.tar.xz +image_signing: {vardir}/keyrings/image-signing.tar.xz +device_signing: {vardir}/keyrings/device-signing.tar.xz + +[updater] +cache_partition: {vardir}/android/cache +data_partition: {vardir}/ubuntu/cache + +[hooks] +device: systemimage.testing.demo.TestingDevice +scorer: systemimage.scores.WeightedScorer +apply: systemimage.apply.Reboot + +[dbus] +lifetime: 5m === renamed file 'systemimage/tests/data/config_05.ini' => 'systemimage/tests/data/config.config_05.ini' --- systemimage/tests/data/config_05.ini 2014-02-25 21:46:55 +0000 +++ systemimage/tests/data/config.config_05.ini 2015-05-20 14:55:53 +0000 @@ -11,7 +11,6 @@ [system] timeout: 10s -build_file: /etc/ubuntu-build tempdir: /tmp logfile: /var/log/system-image/client.log loglevel: error @@ -24,13 +23,13 @@ device_signing: /var/lib/phablet/device-signing.tar.xz [updater] -cache_partition: /android/cache -data_partition: /var/lib/phablet/updater +cache_partition: {tmpdir}/android/cache +data_partition: {vardir}/lib/phablet/updater [hooks] device: systemimage.device.SystemProperty scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot +apply: systemimage.apply.Reboot [dbus] lifetime: 3s === renamed file 'systemimage/tests/data/config_06.ini' => 'systemimage/tests/data/config.config_06.ini' --- systemimage/tests/data/config_06.ini 2014-02-25 21:46:55 +0000 +++ systemimage/tests/data/config.config_06.ini 2015-05-20 14:55:53 +0000 @@ -11,7 +11,6 @@ [system] timeout: 10s -build_file: /etc/ubuntu-build tempdir: /tmp logfile: /var/log/system-image/client.log loglevel: error @@ -24,13 +23,13 @@ device_signing: /var/lib/phablet/device-signing.tar.xz [updater] -cache_partition: /android/cache -data_partition: /var/lib/phablet/updater +cache_partition: {tmpdir}/android/cache +data_partition: {vardir}/lib/phablet/updater [hooks] device: systemimage.device.SystemProperty scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot +apply: systemimage.apply.Reboot [dbus] lifetime: 3s === renamed file 'systemimage/tests/data/config_07.ini' => 'systemimage/tests/data/config.config_07.ini' --- systemimage/tests/data/config_07.ini 2014-02-25 21:46:55 +0000 +++ systemimage/tests/data/config.config_07.ini 2015-05-20 14:55:53 +0000 @@ -11,7 +11,6 @@ [system] timeout: 10s -build_file: /etc/ubuntu-build tempdir: /tmp logfile: /var/log/system-image/client.log loglevel: error @@ -30,7 +29,7 @@ [hooks] device: systemimage.device.SystemProperty scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot +apply: systemimage.apply.Reboot [dbus] lifetime: 3s === renamed file 'systemimage/tests/data/config_08.ini' => 'systemimage/tests/data/config.config_08.ini' --- systemimage/tests/data/config_08.ini 2014-02-26 16:11:09 +0000 +++ systemimage/tests/data/config.config_08.ini 2015-05-20 14:55:53 +0000 @@ -11,7 +11,6 @@ [system] timeout: 10s -build_file: /etc/ubuntu-build tempdir: /tmp logfile: /var/log/system-image/client.log loglevel: error @@ -30,7 +29,7 @@ [hooks] device: systemimage.device.SystemProperty scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot +apply: systemimage.apply.Reboot [dbus] lifetime: 3s === renamed file 'systemimage/tests/data/channel_02.ini' => 'systemimage/tests/data/config.config_09.ini' --- systemimage/tests/data/channel_02.ini 2014-03-05 21:29:23 +0000 +++ systemimage/tests/data/config.config_09.ini 2015-05-20 14:55:53 +0000 @@ -5,8 +5,5 @@ channel: proposed build_number: 833 -[system] -build_file: /etc/path/to/alternative/build-file - [dbus] lifetime: 1h === added file 'systemimage/tests/data/config.config_10.ini' --- systemimage/tests/data/config.config_10.ini 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/config.config_10.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,27 @@ +# Bogus configuration file missing the [system] stanza. + +[service] +base: phablet.example.com +# Negative ports are not allowed. +http_port: 80 +https_port: disabled +channel: stable +build_number: 0 + +[gpg] +archive_master: /etc/phablet/archive-master.tar.xz +image_master: /etc/phablet/image-master.tar.xz +image_signing: /var/lib/phablet/image-signing.tar.xz +device_signing: /var/lib/phablet/device-signing.tar.xz + +[updater] +cache_partition: {tmpdir}/android/cache +data_partition: {vardir}/lib/phablet/updater + +[hooks] +device: systemimage.device.SystemProperty +scorer: systemimage.scores.WeightedScorer +apply: systemimage.apply.Reboot + +[dbus] +lifetime: 3s === added file 'systemimage/tests/data/config.config_11.ini' --- systemimage/tests/data/config.config_11.ini 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/config.config_11.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,2 @@ +[service] +device: nexus8 === removed file 'systemimage/tests/data/config_04.ini' --- systemimage/tests/data/config_04.ini 2014-01-30 15:41:03 +0000 +++ systemimage/tests/data/config_04.ini 1970-01-01 00:00:00 +0000 @@ -1,36 +0,0 @@ -# Configuration file for specifying relatively static information about the -# upgrade resolution process. - -[service] -base: phablet.example.com -# Non-standard ports -http_port: 8080 -https_port: 80443 -channel: stable -build_number: 0 - -[system] -timeout: 30s -build_file: {tmpdir}/ubuntu-build -tempdir: {tmpdir}/tmp -logfile: {vardir}/client.log -loglevel: info -settings_db: {vardir}/settings.db - -[gpg] -archive_master: {vardir}/etc/archive-master.tar.xz -image_master: {vardir}/keyrings/image-master.tar.xz -image_signing: {vardir}/keyrings/image-signing.tar.xz -device_signing: {vardir}/keyrings/device-signing.tar.xz - -[updater] -cache_partition: {vardir}/android/cache -data_partition: {vardir}/ubuntu/cache - -[hooks] -device: systemimage.device.SystemProperty -scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot - -[dbus] -lifetime: 3s === removed file 'systemimage/tests/data/config_09.ini' --- systemimage/tests/data/config_09.ini 2014-09-17 02:58:58 +0000 +++ systemimage/tests/data/config_09.ini 1970-01-01 00:00:00 +0000 @@ -1,27 +0,0 @@ -# Bogus configuration file missing the [system] stanza. - -[service] -base: phablet.example.com -# Negative ports are not allowed. -http_port: 80 -https_port: disabled -channel: stable -build_number: 0 - -[gpg] -archive_master: /etc/phablet/archive-master.tar.xz -image_master: /etc/phablet/image-master.tar.xz -image_signing: /var/lib/phablet/image-signing.tar.xz -device_signing: /var/lib/phablet/device-signing.tar.xz - -[updater] -cache_partition: /android/cache -data_partition: /var/lib/phablet/updater - -[hooks] -device: systemimage.device.SystemProperty -scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot - -[dbus] -lifetime: 3s === removed file 'systemimage/tests/data/config_10.ini' --- systemimage/tests/data/config_10.ini 2014-09-17 02:58:58 +0000 +++ systemimage/tests/data/config_10.ini 1970-01-01 00:00:00 +0000 @@ -1,35 +0,0 @@ -# Configuration file for specifying relatively static information about the -# upgrade resolution process. - -[service] -base: phablet.example.com -http_port: 80 -https_port: 443 -channel: stable -build_number: 0 - -[system] -timeout: 10s -build_file: /etc/ubuntu-build -tempdir: /tmp -logfile: /var/log/system-image/client.log -loglevel: critical:debug -settings_db: /var/lib/phablet/settings.db - -[gpg] -archive_master: /etc/phablet/archive-master.tar.xz -image_master: /etc/phablet/image-master.tar.xz -image_signing: /var/lib/phablet/image-signing.tar.xz -device_signing: /var/lib/phablet/device-signing.tar.xz - -[updater] -cache_partition: /android/cache -data_partition: /var/lib/phablet/updater - -[hooks] -device: systemimage.device.SystemProperty -scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot - -[dbus] -lifetime: 2m === renamed file 'systemimage/tests/data/channels_06.json' => 'systemimage/tests/data/dbus.channels_01.json' === renamed file 'systemimage/tests/data/index_13.json' => 'systemimage/tests/data/dbus.index_01.json' === renamed file 'systemimage/tests/data/index_18.json' => 'systemimage/tests/data/dbus.index_02.json' === added file 'systemimage/tests/data/dbus.index_03.json' --- systemimage/tests/data/dbus.index_03.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/dbus.index_03.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,36 @@ +{ + "global": { + "generated_at": "Thu Aug 01 08:01:00 UTC 2013" + }, + "images": [ + { + "description": "Full", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 3, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 2, + "path": "/5/6/5.txt", + "signature": "/5/6/5.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1600, + "bootme": true + } + ] +} === renamed file 'systemimage/tests/data/index_24.json' => 'systemimage/tests/data/dbus.index_04.json' === renamed file 'systemimage/tests/data/index_25.json' => 'systemimage/tests/data/dbus.index_05.json' === added file 'systemimage/tests/data/dbus.index_06.json' --- systemimage/tests/data/dbus.index_06.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/dbus.index_06.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,37 @@ +{ + "global": { + "generated_at": "Thu Aug 01 08:01:00 UTC 2013" + }, + "images": [ + { + "description": "Full", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 3, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 2, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1600, + "version_detail": "ubuntu=402,mako=502,custom=602", + "bootme": true + } + ] +} === added file 'systemimage/tests/data/download.index_01.json' --- systemimage/tests/data/download.index_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/download.index_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,6 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [] +} === added file 'systemimage/tests/data/gpg.channels_01.json' --- systemimage/tests/data/gpg.channels_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/gpg.channels_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,23 @@ +{ + "daily": { + "devices": { + "nexus7": { + "index": "/daily/nexus7/index.json", + "keyring": { + "path": "/daily/nexus7/device-keyring.tar.xz", + "signature": "/daily/nexus7/device-keyring.tar.xz.asc" + } + }, + "nexus4":{ + "index": "/daily/nexus4/index.json" + } + } + }, + "stable": { + "devices": { + "nexus7":{ + "index": "/stable/nexus7/index.json" + } + } + } +} === added file 'systemimage/tests/data/helpers.config_01.ini' --- systemimage/tests/data/helpers.config_01.ini 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/helpers.config_01.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,7 @@ +[service] +base: systum-imaje.ubuntu.com +http_port: 88 +https_port: 89 +channel: proposed +build_number: 1833 +version_detail: ubuntu=123,mako=456,custom=789 === added file 'systemimage/tests/data/helpers.config_02.ini' --- systemimage/tests/data/helpers.config_02.ini 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/helpers.config_02.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,6 @@ +[service] +base: systum-imaje.ubuntu.com +http_port: 88 +https_port: 89 +channel: proposed +build_number: 1833 === added file 'systemimage/tests/data/index.channels_01.json' --- systemimage/tests/data/index.channels_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/index.channels_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,9 @@ +{ + "stable": { + "devices": { + "nexus7":{ + "index": "/stable/nexus7/index.json" + } + } + } +} === added file 'systemimage/tests/data/index.channels_02.json' --- systemimage/tests/data/index.channels_02.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/index.channels_02.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,13 @@ +{ + "stable": { + "devices": { + "nexus7":{ + "index": "/stable/nexus7/index.json", + "keyring": { + "path": "stable/nexus7/device.tar.xz", + "signature": "stable/nexus7/device.tar.xz.asc" + } + } + } + } +} === renamed file 'systemimage/tests/data/channels_04.json' => 'systemimage/tests/data/index.channels_03.json' === renamed file 'systemimage/tests/data/channels_05.json' => 'systemimage/tests/data/index.channels_04.json' === added file 'systemimage/tests/data/index.channels_05.json' --- systemimage/tests/data/index.channels_05.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/index.channels_05.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,9 @@ +{ + "stable": { + "devices": { + "nexus7":{ + "index": "/stable/nexus7/index.json" + } + } + } +} === added file 'systemimage/tests/data/index.index_01.json' --- systemimage/tests/data/index.index_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/index.index_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,251 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "abc", + "order": 1, + "path": "/a/b/c.txt", + "signature": "/a/b/c.txt.asc", + "size": 104857600 + + }, + { + "checksum": "bcd", + "order": 1, + "path": "/b/c/d.txt", + "signature": "/b/c/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "cde", + "order": 1, + "path": "/c/d/e.txt", + "signature": "/c/d/e.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1300 + }, + { + "base": 1300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "def", + "order": 1, + "path": "/d/e/f.txt", + "signature": "/d/e/f.txt.asc", + "size": 104857600 + }, + { + "checksum": "ef0", + "order": 1, + "path": "/e/f/0.txt", + "signature": "/e/f/0.txt.asc", + "size": 104857600 + }, + { + "checksum": "f01", + "order": 1, + "path": "/f/e/1.txt", + "signature": "/f/e/1.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1301 + }, + { + "base": 1301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "012", + "order": 1, + "path": "/0/1/2.txt", + "signature": "/0/1/2.txt.asc", + "size": 104857600 + }, + { + "checksum": "123", + "order": 1, + "path": "/1/2/3.txt", + "signature": "/1/2/3.txt.asc", + "size": 104857600 + }, + { + "checksum": "234", + "order": 1, + "path": "/2/3/4.txt", + "signature": "/2/3/4.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1304 + }, + + { + "description": "Full B", + "description-en": "The full B", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 1, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 10000 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 10001 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 1, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 10002 + } + ], + "type": "full", + "version": 1200 + }, + { + "base": 1200, + "description": "Delta B.1", + "description-en_US": "This is the delta B.1", + "description-xx": "XX This is the delta B.1", + "description-yy": "YY This is the delta B.1", + "description-yy_ZZ": "YY-ZZ This is the delta B.1", + "files": [ + { + "checksum": "cebe3d9d614ba5c19f633566104315854a11353a333bf96f16b5afa0e90abdc4", + "order": 1, + "path": "/6/7/8.txt", + "signature": "/6/7/8.txt.asc", + "size": 20000 + }, + { + "checksum": "35a9e381b1a27567549b5f8a6f783c167ebf809f1c4d6a9e367240484d8ce281", + "order": 1, + "path": "/7/8/9.txt", + "signature": "/7/8/9.txt.asc", + "size": 20001 + }, + { + "checksum": "6bd6c3f7808391e8b74f5c2d58810809eda5c134aaa7f1b27ddf4b445c421ac5", + "order": 1, + "path": "/8/9/a.txt", + "signature": "/8/9/a.txt.asc", + "size": 20002 + } + ], + "type": "delta", + "version": 1201 + }, + { + "base": 1201, + "description": "Delta B.2", + "description-xx": "Oh delta, my delta", + "description-xx_CC": "This hyar is the delta B.2", + "files": [ + { + "checksum": "8c43d75d5b9f1aa9fc3fabb6b60b6c06553324352399a33febce95a1b588d1d6", + "order": 1, + "path": "/9/a/b.txt", + "signature": "/9/a/b.txt.asc", + "size": 30000 + }, + { + "checksum": "20e796c128096d229ba89bf412a53c3151d170a409c2c8c1dd8e414087b7ffae", + "order": 1, + "path": "/f/e/d.txt", + "signature": "/f/e/d.txt.asc", + "size": 30001 + }, + { + "checksum": "278238e8bafa4709c77aa723e168101acd6ee1fb9fcc1b6eca4762e5c7dad768", + "order": 1, + "path": "/e/d/c.txt", + "signature": "/e/d/c.txt.asc", + "size": 30002 + + } + ], + "type": "delta", + "version": 1304 + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "dcb", + "order": 1, + "path": "/d/c/b.txt", + "signature": "/d/c/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "cba", + "order": 1, + "path": "/c/b/a.txt", + "signature": "/c/b/a.txt.asc", + "size": 104857600 + }, + { + "checksum": "ba9", + "order": 1, + "path": "/b/a/9.txt", + "signature": "/b/a/9.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1100 + }, + { + "base": 1100, + "description": "Delta C.1", + "files": [ + { + "checksum": "a98", + "order": 1, + "path": "/a/9/8.txt", + "signature": "/a/9/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "987", + "order": 1, + "path": "/9/8/7.txt", + "signature": "/9/8/7.txt.asc", + "size": 104857600 + }, + { + "checksum": "876", + "order": 1, + "path": "/8/7/6.txt", + "signature": "/8/7/6.txt.asc", + "size": 838860800 + + } + ], + "type": "delta", + "version": 1303 + } + ] +} === renamed file 'systemimage/tests/data/index_01.json' => 'systemimage/tests/data/index.index_02.json' === renamed file 'systemimage/tests/data/index_02.json' => 'systemimage/tests/data/index.index_03.json' === added file 'systemimage/tests/data/index.index_04.json' --- systemimage/tests/data/index.index_04.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/index.index_04.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,244 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "abc", + "order": 1, + "path": "/a/b/c.txt", + "signature": "/a/b/c.txt.asc", + "size": 104857600 + + }, + { + "checksum": "bcd", + "order": 1, + "path": "/b/c/d.txt", + "signature": "/b/c/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "cde", + "order": 1, + "path": "/c/d/e.txt", + "signature": "/c/d/e.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1300 + }, + { + "base": 1300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "def", + "order": 1, + "path": "/d/e/f.txt", + "signature": "/d/e/f.txt.asc", + "size": 104857600 + }, + { + "checksum": "ef0", + "order": 1, + "path": "/e/f/0.txt", + "signature": "/e/f/0.txt.asc", + "size": 104857600 + }, + { + "checksum": "f01", + "order": 1, + "path": "/f/e/1.txt", + "signature": "/f/e/1.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1301 + }, + { + "base": 1301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "012", + "order": 1, + "path": "/0/1/2.txt", + "signature": "/0/1/2.txt.asc", + "size": 104857600 + }, + { + "checksum": "123", + "order": 1, + "path": "/1/2/3.txt", + "signature": "/1/2/3.txt.asc", + "size": 104857600 + }, + { + "checksum": "234", + "order": 1, + "path": "/2/3/4.txt", + "signature": "/2/3/4.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1304 + }, + + { + "description": "Full B", + "files": [ + { + "checksum": "345", + "order": 1, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "456", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "567", + "order": 1, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1200 + }, + { + "base": 1200, + "description": "Delta B.1", + "files": [ + { + "checksum": "678", + "order": 1, + "path": "/6/7/8.txt", + "signature": "/6/7/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "789", + "order": 1, + "path": "/7/8/9.txt", + "signature": "/7/8/9.txt.asc", + "size": 104857600 + }, + { + "checksum": "89a", + "order": 1, + "path": "/8/9/a.txt", + "signature": "/8/9/a.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1201 + }, + { + "base": 1201, + "description": "Delta B.2", + "files": [ + { + "checksum": "9ab", + "order": 1, + "path": "/9/a/b.txt", + "signature": "/9/a/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "fed", + "order": 1, + "path": "/f/e/d.txt", + "signature": "/f/e/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "edc", + "order": 1, + "path": "/e/d/c.txt", + "signature": "/e/d/c.txt.asc", + "size": 209715200 + + } + ], + "type": "delta", + "version": 1304 + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "dcb", + "order": 1, + "path": "/d/c/b.txt", + "signature": "/d/c/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "cba", + "order": 1, + "path": "/c/b/a.txt", + "signature": "/c/b/a.txt.asc", + "size": 104857600 + }, + { + "checksum": "ba9", + "order": 1, + "path": "/b/a/9.txt", + "signature": "/b/a/9.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1100 + }, + { + "base": 1100, + "description": "Delta C.1", + "files": [ + { + "checksum": "a98", + "order": 1, + "path": "/a/9/8.txt", + "signature": "/a/9/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "987", + "order": 1, + "path": "/9/8/7.txt", + "signature": "/9/8/7.txt.asc", + "size": 104857600 + }, + { + "checksum": "876", + "order": 1, + "path": "/8/7/6.txt", + "signature": "/8/7/6.txt.asc", + "size": 838860800 + + } + ], + "type": "delta", + "version": 1303 + } + ] +} === renamed file 'systemimage/tests/data/sprint_nexus7_index_01.json' => 'systemimage/tests/data/index.index_05.json' === added file 'systemimage/tests/data/main.channels_01.json' --- systemimage/tests/data/main.channels_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/main.channels_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,13 @@ +{ + "stable": { + "devices": { + "nexus7": { + "index": "/stable/nexus7/index.json", + "keyring": { + "path": "/stable/nexus7/device-signing.tar.xz", + "signature": "/stable/nexus7/device-signing.tar.xz.asc" + } + } + } + } +} === renamed file 'systemimage/tests/data/channels_10.json' => 'systemimage/tests/data/main.channels_02.json' === added file 'systemimage/tests/data/main.channels_03.json' --- systemimage/tests/data/main.channels_03.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/main.channels_03.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,13 @@ +{ + "stable": { + "devices": { + "nexus7": { + "index": "/stable/nexus7/index.json", + "keyring": { + "path": "/stable/nexus7/device-signing.tar.xz", + "signature": "/stable/nexus7/device-signing.tar.xz.asc" + } + } + } + } +} === renamed file 'systemimage/tests/data/config_01.ini' => 'systemimage/tests/data/main.config_01.ini' --- systemimage/tests/data/config_01.ini 2014-01-30 15:41:03 +0000 +++ systemimage/tests/data/main.config_01.ini 2015-05-20 14:55:53 +0000 @@ -5,31 +5,30 @@ base: phablet.example.com http_port: 80 https_port: 443 -channel: stable +channel: special build_number: 0 [system] timeout: 10s -build_file: /etc/ubuntu-build tempdir: /tmp logfile: /var/log/system-image/client.log loglevel: error settings_db: /var/lib/phablet/settings.db [gpg] -archive_master: /etc/phablet/archive-master.tar.xz +archive_master: /usr/share/phablet/archive-master.tar.xz image_master: /etc/phablet/image-master.tar.xz image_signing: /var/lib/phablet/image-signing.tar.xz device_signing: /var/lib/phablet/device-signing.tar.xz [updater] -cache_partition: /android/cache -data_partition: /var/lib/phablet/updater +cache_partition: {tmpdir}/android/cache +data_partition: {vardir}/lib/phablet/updater [hooks] device: systemimage.device.SystemProperty scorer: systemimage.scores.WeightedScorer -reboot: systemimage.reboot.Reboot +apply: systemimage.apply.Reboot [dbus] lifetime: 2m === renamed file 'systemimage/tests/data/channel_01.ini' => 'systemimage/tests/data/main.config_02.ini' === renamed file 'systemimage/tests/data/channel_05.ini' => 'systemimage/tests/data/main.config_03.ini' === renamed file 'systemimage/tests/data/channel_03.ini' => 'systemimage/tests/data/main.config_04.ini' === added file 'systemimage/tests/data/main.config_05.ini' --- systemimage/tests/data/main.config_05.ini 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/main.config_05.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,7 @@ +[service] +base: localhost +http_port: 8980 +https_port: 8943 +channel: daily +build_number: 300 +channel_target: saucy === renamed file 'systemimage/tests/data/channel_04.ini' => 'systemimage/tests/data/main.config_07.ini' --- systemimage/tests/data/channel_04.ini 2014-01-30 15:41:03 +0000 +++ systemimage/tests/data/main.config_07.ini 2015-05-20 14:55:53 +0000 @@ -3,4 +3,4 @@ http_port: 8980 https_port: 8943 channel: saucy -build_number: 1 +build_number: 33 === renamed file 'systemimage/tests/data/index_14.json' => 'systemimage/tests/data/main.index_01.json' === renamed file 'systemimage/tests/data/index_20.json' => 'systemimage/tests/data/main.index_02.json' === renamed file 'systemimage/tests/data/index_15.json' => 'systemimage/tests/data/main.index_03.json' === added file 'systemimage/tests/data/main.index_04.json' --- systemimage/tests/data/main.index_04.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/main.index_04.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,36 @@ +{ + "global": { + "generated_at": "Thu Aug 01 08:01:00 UTC 2013" + }, + "images": [ + { + "description": "Full", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 3, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 2, + "path": "/5/6/5.txt", + "signature": "/5/6/5.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1600, + "bootme": true + } + ] +} === added file 'systemimage/tests/data/main.index_05.json' --- systemimage/tests/data/main.index_05.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/main.index_05.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,36 @@ +{ + "global": { + "generated_at": "Thu Aug 01 08:01:00 UTC 2013" + }, + "images": [ + { + "description": "Full", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 3, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 2, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1600, + "bootme": true + } + ] +} === added file 'systemimage/tests/data/scores.index_01.json' --- systemimage/tests/data/scores.index_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/scores.index_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,245 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "abc", + "order": 1, + "path": "/a/b/c.txt", + "signature": "/a/b/c.txt.asc", + "size": 104857600 + + }, + { + "checksum": "bcd", + "order": 1, + "path": "/b/c/d.txt", + "signature": "/b/c/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "cde", + "order": 1, + "path": "/c/d/e.txt", + "signature": "/c/d/e.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 300 + }, + { + "base": 300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "def", + "order": 1, + "path": "/d/e/f.txt", + "signature": "/d/e/f.txt.asc", + "size": 104857600 + }, + { + "checksum": "ef0", + "order": 1, + "path": "/e/f/0.txt", + "signature": "/e/f/0.txt.asc", + "size": 104857600 + }, + { + "checksum": "f01", + "order": 1, + "path": "/f/e/1.txt", + "signature": "/f/e/1.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 301 + }, + { + "base": 301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "012", + "order": 1, + "path": "/0/1/2.txt", + "signature": "/0/1/2.txt.asc", + "size": 104857600 + }, + { + "checksum": "123", + "order": 1, + "path": "/1/2/3.txt", + "signature": "/1/2/3.txt.asc", + "size": 104857600 + }, + { + "checksum": "234", + "order": 1, + "path": "/2/3/4.txt", + "signature": "/2/3/4.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 304 + }, + + { + "description": "Full B", + "files": [ + { + "checksum": "345", + "order": 1, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "456", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "567", + "order": 1, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 200 + }, + { + "base": 200, + "description": "Delta B.1", + "files": [ + { + "checksum": "678", + "order": 1, + "path": "/6/7/8.txt", + "signature": "/6/7/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "789", + "order": 1, + "path": "/7/8/9.txt", + "signature": "/7/8/9.txt.asc", + "size": 104857600 + }, + { + "checksum": "89a", + "order": 1, + "path": "/8/9/a.txt", + "signature": "/8/9/a.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 201 + }, + { + "base": 201, + "description": "Delta B.2", + "files": [ + { + "checksum": "9ab", + "order": 1, + "path": "/9/a/b.txt", + "signature": "/9/a/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "fed", + "order": 1, + "path": "/f/e/d.txt", + "signature": "/f/e/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "edc", + "order": 1, + "path": "/e/d/c.txt", + "signature": "/e/d/c.txt.asc", + "size": 209715200 + + } + ], + "type": "delta", + "version": 304, + "phased-percentage": 0 + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "dcb", + "order": 1, + "path": "/d/c/b.txt", + "signature": "/d/c/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "cba", + "order": 1, + "path": "/c/b/a.txt", + "signature": "/c/b/a.txt.asc", + "size": 104857600 + }, + { + "checksum": "ba9", + "order": 1, + "path": "/b/a/9.txt", + "signature": "/b/a/9.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 100 + }, + { + "base": 100, + "description": "Delta C.1", + "files": [ + { + "checksum": "a98", + "order": 1, + "path": "/a/9/8.txt", + "signature": "/a/9/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "987", + "order": 1, + "path": "/9/8/7.txt", + "signature": "/9/8/7.txt.asc", + "size": 104857600 + }, + { + "checksum": "876", + "order": 1, + "path": "/8/7/6.txt", + "signature": "/8/7/6.txt.asc", + "size": 838860800 + + } + ], + "type": "delta", + "version": 303 + } + ] +} === renamed file 'systemimage/tests/data/index_08.json' => 'systemimage/tests/data/scores.index_02.json' === renamed file 'systemimage/tests/data/index_09.json' => 'systemimage/tests/data/scores.index_03.json' === renamed file 'systemimage/tests/data/index_17.json' => 'systemimage/tests/data/scores.index_04.json' === added file 'systemimage/tests/data/scores.index_05.json' --- systemimage/tests/data/scores.index_05.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/scores.index_05.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,245 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "abc", + "order": 1, + "path": "/a/b/c.txt", + "signature": "/a/b/c.txt.asc", + "size": 104857600 + + }, + { + "checksum": "bcd", + "order": 1, + "path": "/b/c/d.txt", + "signature": "/b/c/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "cde", + "order": 1, + "path": "/c/d/e.txt", + "signature": "/c/d/e.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 300 + }, + { + "base": 300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "def", + "order": 1, + "path": "/d/e/f.txt", + "signature": "/d/e/f.txt.asc", + "size": 104857600 + }, + { + "checksum": "ef0", + "order": 1, + "path": "/e/f/0.txt", + "signature": "/e/f/0.txt.asc", + "size": 104857600 + }, + { + "checksum": "f01", + "order": 1, + "path": "/f/e/1.txt", + "signature": "/f/e/1.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 301 + }, + { + "base": 301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "012", + "order": 1, + "path": "/0/1/2.txt", + "signature": "/0/1/2.txt.asc", + "size": 104857600 + }, + { + "checksum": "123", + "order": 1, + "path": "/1/2/3.txt", + "signature": "/1/2/3.txt.asc", + "size": 104857600 + }, + { + "checksum": "234", + "order": 1, + "path": "/2/3/4.txt", + "signature": "/2/3/4.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 304 + }, + + { + "description": "Full B", + "files": [ + { + "checksum": "345", + "order": 1, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "456", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "567", + "order": 1, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 200 + }, + { + "base": 200, + "description": "Delta B.1", + "files": [ + { + "checksum": "678", + "order": 1, + "path": "/6/7/8.txt", + "signature": "/6/7/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "789", + "order": 1, + "path": "/7/8/9.txt", + "signature": "/7/8/9.txt.asc", + "size": 104857600 + }, + { + "checksum": "89a", + "order": 1, + "path": "/8/9/a.txt", + "signature": "/8/9/a.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 201 + }, + { + "base": 201, + "description": "Delta B.2", + "files": [ + { + "checksum": "9ab", + "order": 1, + "path": "/9/a/b.txt", + "signature": "/9/a/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "fed", + "order": 1, + "path": "/f/e/d.txt", + "signature": "/f/e/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "edc", + "order": 1, + "path": "/e/d/c.txt", + "signature": "/e/d/c.txt.asc", + "size": 209715200 + + } + ], + "type": "delta", + "version": 304, + "phased-percentage": 50 + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "dcb", + "order": 1, + "path": "/d/c/b.txt", + "signature": "/d/c/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "cba", + "order": 1, + "path": "/c/b/a.txt", + "signature": "/c/b/a.txt.asc", + "size": 104857600 + }, + { + "checksum": "ba9", + "order": 1, + "path": "/b/a/9.txt", + "signature": "/b/a/9.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 100 + }, + { + "base": 100, + "description": "Delta C.1", + "files": [ + { + "checksum": "a98", + "order": 1, + "path": "/a/9/8.txt", + "signature": "/a/9/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "987", + "order": 1, + "path": "/9/8/7.txt", + "signature": "/9/8/7.txt.asc", + "size": 104857600 + }, + { + "checksum": "876", + "order": 1, + "path": "/8/7/6.txt", + "signature": "/8/7/6.txt.asc", + "size": 838860800 + + } + ], + "type": "delta", + "version": 303 + } + ] +} === added file 'systemimage/tests/data/scores.index_06.json' --- systemimage/tests/data/scores.index_06.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/scores.index_06.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,253 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "full", + "version": 1300, + "version_detail": "ubuntu=100,raw-device=200,version=300" + }, + { + "base": 1300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "delta", + "version": 1301, + "version_detail": "ubuntu=101,raw-device=201,version=301" + }, + { + "base": 1301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "delta", + "version": 1304, + "version_detail": "ubuntu=102,raw-device=202,version=302" + }, + + { + "description": "Full B", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "full", + "version": 1200, + "version_detail": "ubuntu=103,raw-device=203,version=303" + }, + { + "base": 1200, + "bootme": true, + "description": "Delta B.1", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "delta", + "version": 1201, + "version_detail": "ubuntu=104,raw-device=204,version=304" + }, + { + "base": 1201, + "description": "Delta B.2", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 209715200 + + } + ], + "type": "delta", + "version": 1304, + "version_detail": "ubuntu=105,raw-device=205,version=305" + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "full", + "version": 1100, + "version_detail": "ubuntu=106,raw-device=206,version=306" + }, + { + "base": 1100, + "description": "Delta C.1", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 838860800 + + } + ], + "type": "delta", + "version": 1303, + "version_detail": "ubuntu=107,raw-device=207,version=307" + } + ] +} === added file 'systemimage/tests/data/scores.index_07.json' --- systemimage/tests/data/scores.index_07.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/scores.index_07.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,252 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "full", + "version": 1300, + "version_detail": "ubuntu=100,raw-device=200,version=300" + }, + { + "base": 1300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "delta", + "version": 1301, + "version_detail": "ubuntu=101,raw-device=201,version=301" + }, + { + "base": 1301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "delta", + "version": 1304, + "version_detail": "ubuntu=102,raw-device=202,version=302" + }, + + { + "description": "Full B", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "full", + "version": 1200, + "version_detail": "ubuntu=103,raw-device=203,version=303" + }, + { + "base": 1200, + "bootme": true, + "description": "Delta B.1", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "delta", + "version": 1201, + "version_detail": "ubuntu=104,raw-device=204,version=304" + }, + { + "base": 1201, + "description": "Delta B.2", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 209715200 + + } + ], + "type": "delta", + "version": 1304 + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + } + ], + "type": "full", + "version": 1100, + "version_detail": "ubuntu=106,raw-device=206,version=306" + }, + { + "base": 1100, + "description": "Delta C.1", + "files": [ + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 104857600 + }, + { + "checksum": "", + "order": 1, + "path": "", + "signature": "", + "size": 838860800 + + } + ], + "type": "delta", + "version": 1303, + "version_detail": "ubuntu=107,raw-device=207,version=307" + } + ] +} === added file 'systemimage/tests/data/state.channels_01.json' --- systemimage/tests/data/state.channels_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/state.channels_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,64 @@ +{ + "daily": { + "alias": "tubular", + "devices": { + "grouper": { + "index": "/daily/grouper/index.json" + }, + "maguro": { + "index": "/daily/maguro/index.json" + }, + "mako": { + "index": "/daily/mako/index.json" + }, + "manta": { + "index": "/daily/manta/index.json", + "keyring": { + "path": "/daily/manta/device-signing.tar.xz", + "signature": "/daily/manta/device-signing.tar.xz.asc" + } + } + } + }, + "saucy": { + "devices": { + "grouper": { + "index": "/saucy/grouper/index.json" + }, + "maguro": { + "index": "/saucy/maguro/index.json" + }, + "mako": { + "index": "/saucy/mako/index.json" + }, + "manta": { + "index": "/saucy/manta/index.json", + "keyring": { + "path": "/saucy/manta/device-signing.tar.xz", + "signature": "/saucy/manta/device-signing.tar.xz.asc" + } + } + } + }, + "tubular": { + "hidden": true, + "devices": { + "grouper": { + "index": "/tubular/grouper/index.json" + }, + "maguro": { + "index": "/tubular/maguro/index.json" + }, + "mako": { + "index": "/tubular/mako/index.json" + }, + "manta": { + "index": "/tubular/manta/index.json", + "keyring": { + "path": "/tubular/manta/device-signing.tar.xz", + "signature": "/tubular/manta/device-signing.tar.xz.asc" + } + } + } + } +} === added file 'systemimage/tests/data/state.channels_02.json' --- systemimage/tests/data/state.channels_02.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/state.channels_02.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,13 @@ +{ + "stable": { + "devices": { + "nexus7": { + "index": "/stable/nexus7/index.json", + "keyring": { + "path": "/stable/nexus7/device-signing.tar.xz", + "signature": "/stable/nexus7/device-signing.tar.xz.asc" + } + } + } + } +} === renamed file 'systemimage/tests/data/channels_11.json' => 'systemimage/tests/data/state.channels_03.json' === renamed file 'systemimage/tests/data/channels_07.json' => 'systemimage/tests/data/state.channels_04.json' === renamed file 'systemimage/tests/data/channels_08.json' => 'systemimage/tests/data/state.channels_05.json' === renamed file 'systemimage/tests/data/channels_09.json' => 'systemimage/tests/data/state.channels_06.json' === renamed file 'systemimage/tests/data/channels_01.json' => 'systemimage/tests/data/state.channels_07.json' === added file 'systemimage/tests/data/state.config_01.ini' --- systemimage/tests/data/state.config_01.ini 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/state.config_01.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,2 @@ +[hooks] +apply: systemimage.apply.Noop === added file 'systemimage/tests/data/state.config_02.ini' --- systemimage/tests/data/state.config_02.ini 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/state.config_02.ini 2015-05-20 14:55:53 +0000 @@ -0,0 +1,6 @@ +[service] +base: localhost +http_port: 8980 +https_port: 8943 +channel: saucy +build_number: 1 === added file 'systemimage/tests/data/state.index_01.json' --- systemimage/tests/data/state.index_01.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/state.index_01.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,244 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "abc", + "order": 1, + "path": "/a/b/c.txt", + "signature": "/a/b/c.txt.asc", + "size": 104857600 + + }, + { + "checksum": "bcd", + "order": 1, + "path": "/b/c/d.txt", + "signature": "/b/c/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "cde", + "order": 1, + "path": "/c/d/e.txt", + "signature": "/c/d/e.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 300 + }, + { + "base": 300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "def", + "order": 1, + "path": "/d/e/f.txt", + "signature": "/d/e/f.txt.asc", + "size": 104857600 + }, + { + "checksum": "ef0", + "order": 1, + "path": "/e/f/0.txt", + "signature": "/e/f/0.txt.asc", + "size": 104857600 + }, + { + "checksum": "f01", + "order": 1, + "path": "/f/e/1.txt", + "signature": "/f/e/1.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 301 + }, + { + "base": 301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "012", + "order": 1, + "path": "/0/1/2.txt", + "signature": "/0/1/2.txt.asc", + "size": 104857600 + }, + { + "checksum": "123", + "order": 1, + "path": "/1/2/3.txt", + "signature": "/1/2/3.txt.asc", + "size": 104857600 + }, + { + "checksum": "234", + "order": 1, + "path": "/2/3/4.txt", + "signature": "/2/3/4.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 304 + }, + + { + "description": "Full B", + "files": [ + { + "checksum": "345", + "order": 1, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "456", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "567", + "order": 1, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 200 + }, + { + "base": 200, + "description": "Delta B.1", + "files": [ + { + "checksum": "678", + "order": 1, + "path": "/6/7/8.txt", + "signature": "/6/7/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "789", + "order": 1, + "path": "/7/8/9.txt", + "signature": "/7/8/9.txt.asc", + "size": 104857600 + }, + { + "checksum": "89a", + "order": 1, + "path": "/8/9/a.txt", + "signature": "/8/9/a.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 201 + }, + { + "base": 201, + "description": "Delta B.2", + "files": [ + { + "checksum": "9ab", + "order": 1, + "path": "/9/a/b.txt", + "signature": "/9/a/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "fed", + "order": 1, + "path": "/f/e/d.txt", + "signature": "/f/e/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "edc", + "order": 1, + "path": "/e/d/c.txt", + "signature": "/e/d/c.txt.asc", + "size": 209715200 + + } + ], + "type": "delta", + "version": 304 + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "dcb", + "order": 1, + "path": "/d/c/b.txt", + "signature": "/d/c/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "cba", + "order": 1, + "path": "/c/b/a.txt", + "signature": "/c/b/a.txt.asc", + "size": 104857600 + }, + { + "checksum": "ba9", + "order": 1, + "path": "/b/a/9.txt", + "signature": "/b/a/9.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 100 + }, + { + "base": 100, + "description": "Delta C.1", + "files": [ + { + "checksum": "a98", + "order": 1, + "path": "/a/9/8.txt", + "signature": "/a/9/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "987", + "order": 1, + "path": "/9/8/7.txt", + "signature": "/9/8/7.txt.asc", + "size": 104857600 + }, + { + "checksum": "876", + "order": 1, + "path": "/8/7/6.txt", + "signature": "/8/7/6.txt.asc", + "size": 838860800 + + } + ], + "type": "delta", + "version": 303 + } + ] +} === added file 'systemimage/tests/data/state.index_02.json' --- systemimage/tests/data/state.index_02.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/state.index_02.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,245 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "bootme": true, + "description": "Full A", + "files": [ + { + "checksum": "abc", + "order": 1, + "path": "/a/b/c.txt", + "signature": "/a/b/c.txt.asc", + "size": 104857600 + + }, + { + "checksum": "bcd", + "order": 1, + "path": "/b/c/d.txt", + "signature": "/b/c/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "cde", + "order": 1, + "path": "/c/d/e.txt", + "signature": "/c/d/e.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 300 + }, + { + "base": 300, + "bootme": true, + "description": "Delta A.1", + "files": [ + { + "checksum": "def", + "order": 1, + "path": "/d/e/f.txt", + "signature": "/d/e/f.txt.asc", + "size": 104857600 + }, + { + "checksum": "ef0", + "order": 1, + "path": "/e/f/0.txt", + "signature": "/e/f/0.txt.asc", + "size": 104857600 + }, + { + "checksum": "f01", + "order": 1, + "path": "/f/e/1.txt", + "signature": "/f/e/1.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 301 + }, + { + "base": 301, + "bootme": true, + "description": "Delta A.2", + "files": [ + { + "checksum": "012", + "order": 1, + "path": "/0/1/2.txt", + "signature": "/0/1/2.txt.asc", + "size": 104857600 + }, + { + "checksum": "123", + "order": 1, + "path": "/1/2/3.txt", + "signature": "/1/2/3.txt.asc", + "size": 104857600 + }, + { + "checksum": "234", + "order": 1, + "path": "/2/3/4.txt", + "signature": "/2/3/4.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 304 + }, + + { + "description": "Full B", + "files": [ + { + "checksum": "345", + "order": 1, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "456", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "567", + "order": 1, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 200 + }, + { + "base": 200, + "description": "Delta B.1", + "files": [ + { + "checksum": "678", + "order": 1, + "path": "/6/7/8.txt", + "signature": "/6/7/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "789", + "order": 1, + "path": "/7/8/9.txt", + "signature": "/7/8/9.txt.asc", + "size": 104857600 + }, + { + "checksum": "89a", + "order": 1, + "path": "/8/9/a.txt", + "signature": "/8/9/a.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 201 + }, + { + "base": 201, + "description": "Delta B.2", + "files": [ + { + "checksum": "9ab", + "order": 1, + "path": "/9/a/b.txt", + "signature": "/9/a/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "fed", + "order": 1, + "path": "/f/e/d.txt", + "signature": "/f/e/d.txt.asc", + "size": 104857600 + }, + { + "checksum": "edc", + "order": 1, + "path": "/e/d/c.txt", + "signature": "/e/d/c.txt.asc", + "size": 209715200 + + } + ], + "type": "delta", + "version": 304, + "phased-percentage": 0 + }, + + { + "description": "Full C", + "files": [ + { + "checksum": "dcb", + "order": 1, + "path": "/d/c/b.txt", + "signature": "/d/c/b.txt.asc", + "size": 104857600 + }, + { + "checksum": "cba", + "order": 1, + "path": "/c/b/a.txt", + "signature": "/c/b/a.txt.asc", + "size": 104857600 + }, + { + "checksum": "ba9", + "order": 1, + "path": "/b/a/9.txt", + "signature": "/b/a/9.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 100 + }, + { + "base": 100, + "description": "Delta C.1", + "files": [ + { + "checksum": "a98", + "order": 1, + "path": "/a/9/8.txt", + "signature": "/a/9/8.txt.asc", + "size": 104857600 + }, + { + "checksum": "987", + "order": 1, + "path": "/9/8/7.txt", + "signature": "/9/8/7.txt.asc", + "size": 104857600 + }, + { + "checksum": "876", + "order": 1, + "path": "/8/7/6.txt", + "signature": "/8/7/6.txt.asc", + "size": 838860800 + + } + ], + "type": "delta", + "version": 303 + } + ] +} === added file 'systemimage/tests/data/state.index_03.json' --- systemimage/tests/data/state.index_03.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/state.index_03.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,36 @@ +{ + "global": { + "generated_at": "Thu Aug 01 08:01:00 UTC 2013" + }, + "images": [ + { + "description": "Full", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 3, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 2, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "full", + "version": 1600, + "bootme": true + } + ] +} === added file 'systemimage/tests/data/state.index_04.json' --- systemimage/tests/data/state.index_04.json 1970-01-01 00:00:00 +0000 +++ systemimage/tests/data/state.index_04.json 2015-05-20 14:55:53 +0000 @@ -0,0 +1,37 @@ +{ + "global": { + "generated_at": "Mon Apr 29 18:45:27 UTC 2013" + }, + "images": [ + { + "base": 100, + "description": "Delta", + "files": [ + { + "checksum": "da70dfa4d9f95ac979f921e8e623358236313f334afcd06cddf8a5621cf6a1e9", + "order": 3, + "path": "/3/4/5.txt", + "signature": "/3/4/5.txt.asc", + "size": 104857600 + }, + { + "checksum": "b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0", + "order": 1, + "path": "/4/5/6.txt", + "signature": "/4/5/6.txt.asc", + "size": 104857600 + }, + { + "checksum": "97a6d21df7c51e8289ac1a8c026aaac143e15aa1957f54f42e30d8f8a85c3a55", + "order": 2, + "path": "/5/6/7.txt", + "signature": "/5/6/7.txt.asc", + "size": 104857600 + } + ], + "type": "delta", + "version": 1600, + "bootme": true + } + ] +} === renamed file 'systemimage/tests/data/index_16.json' => 'systemimage/tests/data/state.index_05.json' === renamed file 'systemimage/tests/data/index_21.json' => 'systemimage/tests/data/state.index_06.json' === renamed file 'systemimage/tests/data/index_22.json' => 'systemimage/tests/data/state.index_07.json' --- systemimage/tests/data/index_22.json 2014-01-30 15:41:03 +0000 +++ systemimage/tests/data/state.index_07.json 2015-05-20 14:55:53 +0000 @@ -120,7 +120,6 @@ } ], "type": "full", - "phased-percentage": 50, "version": 200 }, { @@ -180,7 +179,8 @@ } ], "type": "delta", - "version": 304 + "version": 304, + "phased-percentage": 50 }, { @@ -209,7 +209,6 @@ } ], "type": "full", - "phased-percentage": 75, "version": 100 }, { === renamed file 'systemimage/tests/data/index_23.json' => 'systemimage/tests/data/state.index_08.json' === renamed file 'systemimage/tests/data/channels_02.json' => 'systemimage/tests/data/winner.channels_01.json' === renamed file 'systemimage/tests/data/channels_03.json' => 'systemimage/tests/data/winner.channels_02.json' === renamed file 'systemimage/tests/data/index_10.json' => 'systemimage/tests/data/winner.index_01.json' === renamed file 'systemimage/tests/data/index_12.json' => 'systemimage/tests/data/winner.index_02.json' === modified file 'systemimage/tests/test_api.py' --- systemimage/tests/test_api.py 2014-09-17 13:41:31 +0000 +++ systemimage/tests/test_api.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -18,21 +18,17 @@ __all__ = [ 'TestAPI', + 'TestAPIVersionDetail', ] import os -import json +import unittest -from contextlib import ExitStack -from datetime import datetime, timedelta -from gi.repository import GLib -from hashlib import sha256 +from pathlib import Path from systemimage.api import Mediator -from systemimage.config import Configuration, config +from systemimage.config import config from systemimage.download import Canceled -from systemimage.gpg import SignatureError -from systemimage.helpers import MiB from systemimage.testing.helpers import ( ServerTestBase, chmod, configuration, copy, setup_index, sign, touch_build) @@ -41,8 +37,8 @@ class TestAPI(ServerTestBase): - INDEX_FILE = 'index_13.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'api.index_01.json' + CHANNEL_FILE = 'api.channels_01.json' CHANNEL = 'stable' DEVICE = 'nexus7' @@ -106,11 +102,11 @@ self._setup_server_keyrings() # Index 14 has a more interesting upgrade path, and will yield a # richer description set. - index_dir = os.path.join(self._serverdir, self.CHANNEL, self.DEVICE) - index_path = os.path.join(index_dir, 'index.json') - copy('index_14.json', index_dir, 'index.json') + index_dir = Path(self._serverdir) / self.CHANNEL / self.DEVICE + index_path = index_dir / 'index.json' + copy('api.index_02.json', index_dir, 'index.json') sign(index_path, 'device-signing.gpg') - setup_index('index_14.json', self._serverdir, 'device-signing.gpg') + setup_index('api.index_02.json', self._serverdir, 'device-signing.gpg') # Get the descriptions. update = Mediator().check_for_update() self.assertTrue(update.is_available) @@ -144,13 +140,13 @@ mediator = Mediator() self.assertTrue(mediator.check_for_update()) # Make sure a reboot did not get issued. - with patch('systemimage.reboot.Reboot.reboot') as reboot: + with patch('systemimage.apply.Reboot.apply') as mock: mediator.download() - # No reboot got issued. - self.assertFalse(reboot.called) + # The update was not applied. + self.assertFalse(mock.called) # But the command file did get written, and all the files are present. - path = os.path.join(config.updater.cache_partition, 'ubuntu_command') - with open(path, 'r', encoding='utf-8') as fp: + path = Path(config.updater.cache_partition) / 'ubuntu_command' + with path.open('r', encoding='utf-8') as fp: command = fp.read() self.assertMultiLineEqual(command, """\ load_keyring image-master.tar.xz image-master.tar.xz.asc @@ -185,29 +181,43 @@ ])) @configuration - def test_reboot(self): - # Run the intermediate steps, and finish with a reboot. + def test_apply(self): + # Run the intermediate steps, applying the update at the end. self._setup_server_keyrings() mediator = Mediator() # Mock to check the state of reboot. - with patch('systemimage.reboot.Reboot.reboot') as reboot: + with patch('systemimage.apply.Reboot.apply') as mock: mediator.check_for_update() mediator.download() - self.assertFalse(reboot.called) - mediator.reboot() - self.assertTrue(reboot.called) + self.assertFalse(mock.called) + mediator.apply() + self.assertTrue(mock.called) @configuration def test_factory_reset(self): mediator = Mediator() - with patch('systemimage.reboot.Reboot.reboot') as reboot: + with patch('systemimage.apply.Reboot.apply') as mock: mediator.factory_reset() - self.assertTrue(reboot.called) - path = os.path.join(config.updater.cache_partition, 'ubuntu_command') - with open(path, 'r', encoding='utf-8') as fp: - command = fp.read() - self.assertMultiLineEqual(command, dedent("""\ - format data + self.assertTrue(mock.called) + path = Path(config.updater.cache_partition) / 'ubuntu_command' + with path.open('r', encoding='utf-8') as fp: + command = fp.read() + self.assertMultiLineEqual(command, dedent("""\ + format data + """)) + + @configuration + def test_production_reset(self): + mediator = Mediator() + with patch('systemimage.apply.Reboot.apply') as mock: + mediator.production_reset() + self.assertTrue(mock.called) + path = Path(config.updater.cache_partition) / 'ubuntu_command' + with path.open('r', encoding='utf-8') as fp: + command = fp.read() + self.assertMultiLineEqual(command, dedent("""\ + format data + enable factory_wipe """)) @configuration @@ -240,73 +250,41 @@ self.assertNotEqual(received_bytes, 0) self.assertNotEqual(total_bytes, 0) - @configuration - def test_pause_resume(self): - # Pause and resume the download. - self._setup_server_keyrings() - for path in ('3/4/5.txt', '4/5/6.txt', '5/6/7.txt'): - full_path = os.path.join(self._serverdir, path) - with open(full_path, 'wb') as fp: - fp.write(b'x' * 100 * MiB) - # We must update the file checksums in the index.json file, then we - # have to resign it. - index_path = os.path.join( - self._serverdir, 'stable', 'nexus7', 'index.json') - with open(index_path, 'r', encoding='utf-8') as fp: - index = json.load(fp) - checksum = sha256(b'x' * 100 * MiB).hexdigest() - for i in range(3): - index['images'][0]['files'][i]['checksum'] = checksum - with open(index_path, 'w', encoding='utf-8') as fp: - json.dump(index, fp) - sign(index_path, 'device-signing.gpg') - # Now the test is all set up. - mediator = Mediator() - pauses = [] - def do_paused(self, signal, path, paused): - if paused: - pauses.append(datetime.now()) - resumes = [] - def do_resumed(self, signal, path, resumed): - if resumed: - resumes.append(datetime.now()) - def pause_on_start(self, signal, path, started): - if started and self._pausable: - mediator.pause() - GLib.timeout_add_seconds(3, mediator.resume) - with ExitStack() as resources: - resources.enter_context( - patch('systemimage.download.DownloadReactor._do_paused', - do_paused)) - resources.enter_context( - patch('systemimage.download.DownloadReactor._do_resumed', - do_resumed)) - resources.enter_context( - patch('systemimage.download.DownloadReactor._do_started', - pause_on_start)) - mediator.check_for_update() - # We'll get a signature error because we messed with the file - # contents. Since this check happens after all files are - # downloaded, this exception is inconsequential to the thing we're - # testing. - try: - mediator.download() - except SignatureError: - pass - # There should be at one pause and one resume event, separated by - # about 3 seconds. - self.assertEqual(len(pauses), 1) - self.assertEqual(len(resumes), 1) - self.assertGreaterEqual(resumes[0] - pauses[0], timedelta(seconds=2.5)) + from systemimage.testing.controller import USING_PYCURL + @unittest.skipIf(os.getuid() == 0, 'Test cannot succeed when run as root') + @unittest.skipUnless(USING_PYCURL, 'LP: #1411866') @configuration - def test_state_machine_exceptions(self, ini_file): + def test_state_machine_exceptions(self, config): # An exception in the state machine captures the exception and returns # an error string in the Update instance. self._setup_server_keyrings() - config = Configuration(ini_file) with chmod(config.updater.cache_partition, 0): update = Mediator().check_for_update() # There's no winning path, but there is an error. self.assertFalse(update.is_available) self.assertIn('Permission denied', update.error) + + +class TestAPIVersionDetail(ServerTestBase): + INDEX_FILE = 'api.index_03.json' + CHANNEL_FILE = 'api.channels_01.json' + CHANNEL = 'stable' + DEVICE = 'nexus7' + + @configuration + def test_update_available_version(self): + # An update is available. What's the target version number? + self._setup_server_keyrings() + update = Mediator().check_for_update() + self.assertEqual(update.version_detail, + 'ubuntu=101,raw-device=201,version=301') + + @configuration + def test_no_update_available_version(self): + # No update is available, so the target version number is zero. + self._setup_server_keyrings() + touch_build(1600) + update = Mediator().check_for_update() + self.assertFalse(update.is_available) + self.assertEqual(update.version_detail, '') === modified file 'systemimage/tests/test_bag.py' --- systemimage/tests/test_bag.py 2014-07-23 22:51:19 +0000 +++ systemimage/tests/test_bag.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify === modified file 'systemimage/tests/test_candidates.py' --- systemimage/tests/test_candidates.py 2014-02-20 23:03:24 +0000 +++ systemimage/tests/test_candidates.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -29,36 +29,28 @@ from systemimage.candidates import ( delta_filter, full_filter, get_candidates, iter_path) from systemimage.scores import WeightedScorer -from systemimage.testing.helpers import configuration, get_index - - -def _descriptions(path): - descriptions = [] - for image in path: - # There's only one description per image so order doesn't - # matter. - descriptions.extend(image.descriptions.values()) - return descriptions +from systemimage.testing.helpers import ( + configuration, descriptions, get_index) class TestCandidates(unittest.TestCase): def test_no_images(self): # If there are no images defined, there are no candidates. - index = get_index('index_01.json') + index = get_index('candidates.index_01.json') candidates = get_candidates(index, 1400) self.assertEqual(candidates, []) def test_only_higher_fulls(self): # All the full images have a minversion greater than our version, so # we cannot upgrade to any of them. - index = get_index('index_02.json') + index = get_index('candidates.index_02.json') candidates = get_candidates(index, 100) self.assertEqual(candidates, []) def test_one_higher_full(self): # Our device is between the minversions of the two available fulls, so # the older one can be upgraded too. - index = get_index('index_02.json') + index = get_index('candidates.index_02.json') candidates = get_candidates(index, 800) # There is exactly one upgrade path. self.assertEqual(len(candidates), 1) @@ -72,7 +64,7 @@ def test_fulls_with_no_minversion(self): # Like the previous test, there are two full upgrades, but because # neither of them have minversions, both are candidates. - index = get_index('index_05.json') + index = get_index('candidates.index_03.json') candidates = get_candidates(index, 400) self.assertEqual(len(candidates), 2) # Both candidate paths have exactly one image in them. We can't sort @@ -90,13 +82,13 @@ def test_no_deltas_based_on_us(self): # There are deltas in the test data, but no fulls. None of the deltas # have a base equal to our build number. - index = get_index('index_03.json') + index = get_index('candidates.index_04.json') candidates = get_candidates(index, 100) self.assertEqual(candidates, []) def test_one_delta_based_on_us(self): # There is one delta in the test data that is based on us. - index = get_index('index_03.json') + index = get_index('candidates.index_04.json') candidates = get_candidates(index, 500) self.assertEqual(len(candidates), 1) path = candidates[0] @@ -108,7 +100,7 @@ def test_two_deltas_based_on_us(self): # There are two deltas that are based on us, so both are candidates. # They get us to different final versions. - index = get_index('index_04.json') + index = get_index('candidates.index_05.json') candidates = get_candidates(index, 1100) self.assertEqual(len(candidates), 2) # Both candidate paths have exactly one image in them. We can't sort @@ -118,37 +110,37 @@ self.assertEqual(len(path1), 1) # One path gets us to version 1300 and the other 1400. images = sorted([path0[0], path1[0]], key=attrgetter('version')) - self.assertEqual(_descriptions(images), ['Delta 2', 'Delta 1']) + self.assertEqual(descriptions(images), ['Delta 2', 'Delta 1']) def test_one_path_with_full_and_deltas(self): # There's one path to upgrade from our version to the final version. # This one starts at a full and includes several deltas. - index = get_index('index_06.json') + index = get_index('candidates.index_06.json') candidates = get_candidates(index, 1000) self.assertEqual(len(candidates), 1) path = candidates[0] self.assertEqual(len(path), 3) self.assertEqual([image.version for image in path], [1300, 1301, 1302]) - self.assertEqual(_descriptions(path), ['Full 1', 'Delta 1', 'Delta 2']) + self.assertEqual(descriptions(path), ['Full 1', 'Delta 1', 'Delta 2']) def test_one_path_with_deltas(self): # Similar to above, except that because we're upgrading from the # version of the full, the path is only two images long, i.e. the # deltas. - index = get_index('index_06.json') + index = get_index('candidates.index_06.json') candidates = get_candidates(index, 1300) self.assertEqual(len(candidates), 1) path = candidates[0] self.assertEqual(len(path), 2) self.assertEqual([image.version for image in path], [1301, 1302]) - self.assertEqual(_descriptions(path), ['Delta 1', 'Delta 2']) + self.assertEqual(descriptions(path), ['Delta 1', 'Delta 2']) def test_forked_paths(self): # We have a fork in the road. There is a full update, but two deltas # with different versions point to the same base. This will give us # two upgrade paths, both of which include the full. - index = get_index('index_07.json') + index = get_index('candidates.index_07.json') candidates = get_candidates(index, 1200) self.assertEqual(len(candidates), 2) # We can sort the paths by length. @@ -179,9 +171,9 @@ def test_get_downloads(self): # Path B will win; it has one full and two deltas, none of which have # a bootme flag. Download all their files. - index = get_index('index_10.json') + index = get_index('candidates.index_08.json') candidates = get_candidates(index, 600) - winner = WeightedScorer().choose(candidates) + winner = WeightedScorer().choose(candidates, 'devel') descriptions = [] for image in winner: # There's only one description per image so order doesn't matter. @@ -217,9 +209,9 @@ def test_get_downloads_with_bootme(self): # Path B will win; it has one full and two deltas. The first delta # has a bootme flag so the second delta's files are not downloaded. - index = get_index('index_11.json') + index = get_index('candidates.index_09.json') candidates = get_candidates(index, 600) - winner = WeightedScorer().choose(candidates) + winner = WeightedScorer().choose(candidates, 'devel') descriptions = [] for image in winner: # There's only one description per image so order doesn't matter. @@ -242,7 +234,7 @@ # Run a filter over the candidates, such that the only ones left are # those that contain only full upgrades. This can truncate any paths # that start with some fulls and then contain some deltas. - index = get_index('index_10.json') + index = get_index('candidates.index_08.json') candidates = get_candidates(index, 600) filtered = full_filter(candidates) # Since all images start with a full update, we're still left with @@ -251,13 +243,13 @@ self.assertEqual([image.type for image in filtered[0]], ['full']) self.assertEqual([image.type for image in filtered[1]], ['full']) self.assertEqual([image.type for image in filtered[2]], ['full']) - self.assertEqual(_descriptions(filtered[0]), ['Full A']) - self.assertEqual(_descriptions(filtered[1]), ['Full B']) - self.assertEqual(_descriptions(filtered[2]), ['Full C']) + self.assertEqual(descriptions(filtered[0]), ['Full A']) + self.assertEqual(descriptions(filtered[1]), ['Full B']) + self.assertEqual(descriptions(filtered[2]), ['Full C']) def test_filter_for_fulls_one_candidate(self): # Filter for full updates, where the only candidate has one full image. - index = get_index('index_13.json') + index = get_index('candidates.index_10.json') candidates = get_candidates(index, 600) filtered = full_filter(candidates) self.assertEqual(filtered, candidates) @@ -265,7 +257,7 @@ def test_filter_for_fulls_with_just_delta_candidates(self): # A candidate path that contains only deltas will have no filtered # paths if all the images are delta updates. - index = get_index('index_15.json') + index = get_index('candidates.index_11.json') candidates = get_candidates(index, 100) self.assertEqual(len(candidates), 1) filtered = full_filter(candidates) @@ -273,7 +265,7 @@ def test_filter_for_deltas(self): # Filter the candidates, where the only available path is a delta path. - index = get_index('index_15.json') + index = get_index('candidates.index_11.json') candidates = get_candidates(index, 100) self.assertEqual(len(candidates), 1) filtered = delta_filter(candidates) @@ -284,27 +276,27 @@ # Run a filter over the candidates, such that the only ones left are # those that start with and contain only deltas. Since none of the # paths do so, tere are no candidates left. - index = get_index('index_10.json') + index = get_index('candidates.index_08.json') candidates = get_candidates(index, 600) filtered = delta_filter(candidates) self.assertEqual(len(filtered), 0) def test_filter_for_deltas_one_candidate(self): # Filter for delta updates, but the only candidate is a full. - index = get_index('index_13.json') + index = get_index('candidates.index_10.json') candidates = get_candidates(index, 600) filtered = delta_filter(candidates) self.assertEqual(len(filtered), 0) def test_filter_for_multiple_deltas(self): # The candidate path has multiple deltas. All are preserved. - index = get_index('index_19.json') + index = get_index('candidates.index_12.json') candidates = get_candidates(index, 100) filtered = delta_filter(candidates) self.assertEqual(len(filtered), 1) path = filtered[0] self.assertEqual(len(path), 3) - self.assertEqual(_descriptions(path), + self.assertEqual(descriptions(path), ['Delta A', 'Delta B', 'Delta C']) @@ -313,25 +305,25 @@ def test_candidates(self): # Path B will win; it has one full and two deltas. - index = get_index('index_20.json') + index = get_index('candidates.index_13.json') candidates = get_candidates(index, 0) self.assertEqual(len(candidates), 3) path0 = candidates[0] - self.assertEqual(_descriptions(path0), + self.assertEqual(descriptions(path0), ['Full A', 'Delta A.1', 'Delta A.2']) path1 = candidates[1] - self.assertEqual(_descriptions(path1), + self.assertEqual(descriptions(path1), ['Full B', 'Delta B.1', 'Delta B.2']) path2 = candidates[2] - self.assertEqual(_descriptions(path2), ['Full C', 'Delta C.1']) + self.assertEqual(descriptions(path2), ['Full C', 'Delta C.1']) # The version numbers use the new regime. self.assertEqual(path0[0].version, 300) self.assertEqual(path0[1].base, 300) self.assertEqual(path0[1].version, 301) self.assertEqual(path0[2].base, 301) self.assertEqual(path0[2].version, 304) - winner = WeightedScorer().choose(candidates) - self.assertEqual(_descriptions(winner), + winner = WeightedScorer().choose(candidates, 'devel') + self.assertEqual(descriptions(winner), ['Full B', 'Delta B.1', 'Delta B.2']) self.assertEqual(winner[0].version, 200) self.assertEqual(winner[1].base, 200) === modified file 'systemimage/tests/test_channel.py' --- systemimage/tests/test_channel.py 2014-07-23 22:51:19 +0000 +++ systemimage/tests/test_channel.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -45,7 +45,7 @@ # Test that parsing a simple top level channels.json file produces the # expected set of channels. The Nexus 7 daily images have a device # specific keyring. - channels = get_channels('channels_01.json') + channels = get_channels('channel.channels_01.json') self.assertEqual(channels.daily.devices.nexus7.index, '/daily/nexus7/index.json') self.assertEqual(channels.daily.devices.nexus7.keyring.path, @@ -61,30 +61,30 @@ def test_getattr_failure(self): # Test the getattr syntax on an unknown channel or device combination. - channels = get_channels('channels_01.json') + channels = get_channels('channel.channels_01.json') self.assertRaises(AttributeError, getattr, channels, 'bleeding') self.assertRaises(AttributeError, getattr, channels.stable, 'nexus3') def test_daily_proposed(self): # The channel name has a dash in it. - channels = get_channels('channels_07.json') + channels = get_channels('channel.channels_02.json') self.assertEqual(channels['daily-proposed'].devices.grouper.index, '/daily-proposed/grouper/index.json') def test_bad_getitem(self): # Trying to get a channel via getitem which doesn't exist. - channels = get_channels('channels_07.json') + channels = get_channels('channel.channels_02.json') self.assertRaises(KeyError, getitem, channels, 'daily-testing') def test_channel_version(self): # The channel name has a dot in it. - channels = get_channels('channels_08.json') + channels = get_channels('channel.channels_03.json') self.assertEqual(channels['13.10'].devices.grouper.index, '/13.10/grouper/index.json') def test_channel_version_proposed(self): # The channel name has both a dot and a dash in it. - channels = get_channels('channels_08.json') + channels = get_channels('channel.channels_03.json') self.assertEqual(channels['14.04-proposed'].devices.grouper.index, '/14.04-proposed/grouper/index.json') @@ -103,7 +103,7 @@ self._serverdir = self._stack.enter_context(temporary_directory()) self._stack.push(make_http_server( self._serverdir, 8943, 'cert.pem', 'key.pem')) - copy('channels_01.json', self._serverdir, 'channels.json') + copy('channel.channels_01.json', self._serverdir, 'channels.json') self._channels_path = os.path.join( self._serverdir, 'channels.json') except: @@ -124,7 +124,7 @@ '/daily/nexus7/device-keyring.tar.xz.asc') @configuration - def test_load_channel_bad_signature(self, ini_file): + def test_load_channel_bad_signature(self): # We get an error if the signature on the channels.json file is bad. sign(self._channels_path, 'spare.gpg') setup_keyrings() @@ -148,7 +148,7 @@ self.assertRaises(SignatureError, next, self._state) @configuration - def test_load_channel_bad_signature_gets_fixed(self, ini_file): + def test_load_channel_bad_signature_gets_fixed(self, config_d): # Like above, but the second download of the image signing key results # in a properly signed channels.json file. sign(self._channels_path, 'spare.gpg') @@ -165,7 +165,7 @@ os.path.join(self._serverdir, 'gpg', 'image-signing.tar.xz')) # This will succeed by grabbing a new image-signing key. - config = Configuration(ini_file) + config = Configuration(config_d) with open(config.gpg.image_signing, 'rb') as fp: checksum = hashlib.md5(fp.read()).digest() next(self._state) @@ -180,7 +180,7 @@ '/daily/nexus7/device-keyring.tar.xz.asc') @configuration - def test_load_channel_blacklisted_signature(self, ini_file): + def test_load_channel_blacklisted_signature(self, config_d): # We get an error if the signature on the channels.json file is good # but the key is blacklisted. sign(self._channels_path, 'image-signing.gpg') @@ -193,7 +193,7 @@ # cause the state machine to try to download a new image signing key, # so let's put the cached one up on the server. This will still be # backlisted though. - config = Configuration(ini_file) + config = Configuration(config_d) key_path = os.path.join(self._serverdir, 'gpg', 'image-signing.tar.xz') shutil.copy(config.gpg.image_signing, key_path) shutil.copy(config.gpg.image_signing + '.asc', key_path + '.asc') @@ -215,7 +215,7 @@ self._stack = ExitStack() try: self._serverdir = self._stack.enter_context(temporary_directory()) - copy('channels_01.json', self._serverdir, 'channels.json') + copy('channel.channels_01.json', self._serverdir, 'channels.json') sign(os.path.join(self._serverdir, 'channels.json'), 'image-signing.gpg') except: @@ -242,7 +242,7 @@ """LP: #1221841 introduces a new format to channels.json.""" def test_channels(self): # We can parse new-style channels.json files. - channels = get_channels('channels_09.json') + channels = get_channels('channel.channels_04.json') self.assertEqual(channels.daily.alias, 'saucy') self.assertEqual(channels.daily.devices.grouper.index, '/daily/grouper/index.json') @@ -262,23 +262,23 @@ def test_hidden_defaults_to_false(self): # If a channel does not have a hidden field, it defaults to false. - channels = get_channels('channels_09.json') + channels = get_channels('channel.channels_04.json') self.assertFalse(channels.daily.hidden) def test_getattr_failure(self): # Test the getattr syntax on an unknown channel or device combination. - channels = get_channels('channels_09.json') + channels = get_channels('channel.channels_04.json') self.assertRaises(AttributeError, getattr, channels, 'bleeding') self.assertRaises( AttributeError, getattr, channels.daily.devices, 'nexus3') def test_daily_proposed(self): # The channel name has a dash in it. - channels = get_channels('channels_09.json') + channels = get_channels('channel.channels_04.json') self.assertEqual(channels['saucy-proposed'].devices.grouper.index, '/saucy-proposed/grouper/index.json') def test_bad_getitem(self): # Trying to get a channel via getitem which doesn't exist. - channels = get_channels('channels_09.json') + channels = get_channels('channel.channels_04.json') self.assertRaises(KeyError, getitem, channels, 'daily-testing') === modified file 'systemimage/tests/test_config.py' --- systemimage/tests/test_config.py 2014-09-17 13:41:31 +0000 +++ systemimage/tests/test_config.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -23,23 +23,23 @@ import os import sys import stat +import shutil import logging import unittest from contextlib import ExitStack, contextmanager from datetime import timedelta -from pkg_resources import resource_filename from subprocess import CalledProcessError, check_output +from systemimage.apply import Reboot from systemimage.config import Configuration from systemimage.device import SystemProperty -from systemimage.reboot import Reboot from systemimage.scores import WeightedScorer from systemimage.testing.helpers import configuration, data_path, touch_build from unittest.mock import patch @contextmanager -def _patch_device_hook(): +def _patch_device_hook(side_effect=FileNotFoundError): # The device hook has two things that generally need patching. The first # is the logging output, which is just noise for testing purposes, so # silence it. The second is that the `getprop` command may actually exist @@ -48,26 +48,21 @@ with ExitStack() as resources: resources.enter_context(patch('systemimage.device.logging.getLogger')) resources.enter_context( - patch('systemimage.device.check_output', - side_effect=FileNotFoundError)) + patch('systemimage.device.check_output', side_effect=side_effect)) yield class TestConfiguration(unittest.TestCase): def test_defaults(self): - default_ini = resource_filename('systemimage.data', 'client.ini') - config = Configuration(default_ini) + config = Configuration() # [service] self.assertEqual(config.service.base, 'system-image.ubuntu.com') - self.assertEqual(config.service.http_base, - 'http://system-image.ubuntu.com') - self.assertEqual(config.service.https_base, - 'https://system-image.ubuntu.com') + self.assertEqual(config.http_base, 'http://system-image.ubuntu.com') + self.assertEqual(config.https_base, 'https://system-image.ubuntu.com') self.assertEqual(config.service.channel, 'daily') self.assertEqual(config.service.build_number, 0) # [system] self.assertEqual(config.system.tempdir, '/tmp') - self.assertEqual(config.system.build_file, '/etc/ubuntu-build') self.assertEqual(config.system.logfile, '/var/log/system-image/client.log') self.assertEqual(config.system.loglevel, @@ -77,10 +72,10 @@ # [hooks] self.assertEqual(config.hooks.device, SystemProperty) self.assertEqual(config.hooks.scorer, WeightedScorer) - self.assertEqual(config.hooks.reboot, Reboot) + self.assertEqual(config.hooks.apply, Reboot) # [gpg] self.assertEqual(config.gpg.archive_master, - '/etc/system-image/archive-master.tar.xz') + '/usr/share/system-image/archive-master.tar.xz') self.assertEqual( config.gpg.image_master, '/var/lib/system-image/keyrings/image-master.tar.xz') @@ -98,22 +93,19 @@ # [dbus] self.assertEqual(config.dbus.lifetime.total_seconds(), 600) - def test_basic_ini_file(self): - # Read a basic .ini file and check that the various attributes and - # values are correct. - ini_file = data_path('config_01.ini') - config = Configuration(ini_file) + @configuration('config.config_01.ini') + def test_basic_config_d(self, config): + # Read a basic config.d directory and check that the various attributes + # and values are correct. + # # [service] self.assertEqual(config.service.base, 'phablet.example.com') - self.assertEqual(config.service.http_base, - 'http://phablet.example.com') - self.assertEqual(config.service.https_base, - 'https://phablet.example.com') + self.assertEqual(config.http_base, 'http://phablet.example.com') + self.assertEqual(config.https_base, 'https://phablet.example.com') self.assertEqual(config.service.channel, 'stable') self.assertEqual(config.service.build_number, 0) # [system] self.assertEqual(config.system.tempdir, '/tmp') - self.assertEqual(config.system.build_file, '/etc/ubuntu-build') self.assertEqual(config.system.logfile, '/var/log/system-image/client.log') self.assertEqual(config.system.loglevel, @@ -124,10 +116,10 @@ # [hooks] self.assertEqual(config.hooks.device, SystemProperty) self.assertEqual(config.hooks.scorer, WeightedScorer) - self.assertEqual(config.hooks.reboot, Reboot) + self.assertEqual(config.hooks.apply, Reboot) # [gpg] self.assertEqual(config.gpg.archive_master, - '/etc/phablet/archive-master.tar.xz') + '/usr/share/phablet/archive-master.tar.xz') self.assertEqual(config.gpg.image_master, '/etc/phablet/image-master.tar.xz') self.assertEqual(config.gpg.image_signing, @@ -135,114 +127,146 @@ self.assertEqual(config.gpg.device_signing, '/var/lib/phablet/device-signing.tar.xz') # [updater] - self.assertEqual(config.updater.cache_partition, '/android/cache') - self.assertEqual(config.updater.data_partition, - '/var/lib/phablet/updater') + self.assertEqual(config.updater.cache_partition[-14:], + '/android/cache') + self.assertEqual(config.updater.data_partition[-20:], + '/lib/phablet/updater') # [dbus] self.assertEqual(config.dbus.lifetime.total_seconds(), 120) - def test_special_dbus_logging_level(self): + @configuration + def test_should_have_reloaded(self, config_d): + # If a configuration is already loaded, it cannot be loaded again. + # Use .reload() instead. + config = Configuration(config_d) + self.assertRaises(RuntimeError, config.load, config_d) + + @configuration + def test_ignore_some_files(self, config_d): + # Any file that doesn't follow the NN_whatever.ini format isn't loaded. + path_1 = os.path.join(config_d, 'dummy_file') + with open(path_1, 'w', encoding='utf-8') as fp: + print('ignore me', file=fp) + path_2 = os.path.join(config_d, 'nounderscore.ini') + with open(path_2, 'w', encoding='utf-8') as fp: + print('ignore me', file=fp) + path_3 = os.path.join(config_d, 'XX_almost.ini') + with open(path_3, 'w', encoding='utf-8') as fp: + print('ignore me', file=fp) + config = Configuration(config_d) + self.assertNotIn('dummy_file', config.ini_files) + self.assertNotIn('nounderscore.ini', config.ini_files) + self.assertNotIn('XX_almost.ini', config.ini_files) + + @configuration('config.config_02.ini') + def test_special_dbus_logging_level(self, config): # Read a config.ini that has a loglevel value with an explicit dbus # logging level. - ini_file = data_path('config_10.ini') - config = Configuration(ini_file) self.assertEqual(config.system.loglevel, (logging.CRITICAL, logging.DEBUG)) - def test_nonstandard_ports(self): - # config_02.ini has non-standard http and https ports. - ini_file = data_path('config_02.ini') - config = Configuration(ini_file) - self.assertEqual(config.service.base, 'phablet.example.com') - self.assertEqual(config.service.http_base, - 'http://phablet.example.com:8080') - self.assertEqual(config.service.https_base, - 'https://phablet.example.com:80443') - - def test_disabled_http_port(self): - # config_05.ini has http port disabled and non-standard https port. - ini_file = data_path('config_05.ini') - config = Configuration(ini_file) - self.assertEqual(config.service.base, 'phablet.example.com') - self.assertEqual(config.service.http_base, - 'https://phablet.example.com:80443') - self.assertEqual(config.service.https_base, - 'https://phablet.example.com:80443') - - def test_disabled_https_port(self): - # config_06.ini has https port disabled and standard http port. - ini_file = data_path('config_06.ini') - config = Configuration(ini_file) - self.assertEqual(config.service.base, 'phablet.example.com') - self.assertEqual(config.service.http_base, - 'http://phablet.example.com') - self.assertEqual(config.service.https_base, - 'http://phablet.example.com') - - def test_both_ports_disabled(self): - # config_07.ini has both http and https ports disabled. - ini_file = data_path('config_07.ini') + @configuration('config.config_03.ini') + def test_nonstandard_ports(self, config): + # This ini file has non-standard http and https ports. + self.assertEqual(config.service.base, 'phablet.example.com') + self.assertEqual(config.http_base, 'http://phablet.example.com:8080') + self.assertEqual(config.https_base, + 'https://phablet.example.com:80443') + + @configuration('config.config_05.ini') + def test_disabled_http_port(self, config): + # This ini file has http port disabled and non-standard https port. + self.assertEqual(config.service.base, 'phablet.example.com') + self.assertEqual(config.http_base, 'https://phablet.example.com:80443') + self.assertEqual(config.https_base, + 'https://phablet.example.com:80443') + + @configuration('config.config_06.ini') + def test_disabled_https_port(self, config): + # This in i file has https port disabled and standard http port. + self.assertEqual(config.service.base, 'phablet.example.com') + self.assertEqual(config.http_base, 'http://phablet.example.com') + self.assertEqual(config.https_base, 'http://phablet.example.com') + + @configuration + def test_both_ports_disabled(self, config_d): + # This ini file has both http and https ports disabled. + shutil.copy(data_path('config.config_07.ini'), + os.path.join(config_d, '01_override.ini')) config = Configuration() with self.assertRaises(ValueError) as cm: - config.load(ini_file) + config.load(config_d) self.assertEqual(cm.exception.args[0], 'Cannot disable both http and https ports') - def test_negative_port_number(self): - # config_08.ini has a negative port number. - ini_file = data_path('config_08.ini') - config = Configuration() + @configuration + def test_negative_port_number(self, config_d): + # This ini file has a negative port number. + shutil.copy(data_path('config.config_08.ini'), + os.path.join(config_d, '01_override.ini')) with self.assertRaises(ValueError) as cm: - config.load(ini_file) + Configuration(config_d) self.assertEqual(cm.exception.args[0], '-1') @configuration - def test_get_build_number(self, ini_file): + def test_get_build_number(self, config): # The current build number is stored in a file specified in the # configuration file. - config = Configuration(ini_file) touch_build(1500) + config.reload() self.assertEqual(config.build_number, 1500) @configuration - def test_get_build_number_missing(self, ini_file): + def test_get_build_number_after_reload(self, config): + # After a reload, the build number gets updated. + self.assertEqual(config.build_number, 0) + touch_build(801) + config.reload() + self.assertEqual(config.build_number, 801) + + @configuration + def test_get_build_number_missing(self, config): # The build file is missing, so the build number defaults to 0. - config = Configuration(ini_file) self.assertEqual(config.build_number, 0) @configuration - def test_get_device_name(self, ini_file): - config = Configuration(ini_file) + def test_get_device_name(self, config): # The device name as we'd expect it to work on a real image. with patch('systemimage.device.check_output', return_value='nexus7'): self.assertEqual(config.device, 'nexus7') # Get it again to test out the cache. self.assertEqual(config.device, 'nexus7') - def test_get_device_name_fallback(self): + @configuration + def test_get_device_name_fallback(self, config): # Fallback for testing on non-images. - config = Configuration() - # Silence the log exceptions this will provoke. - with patch('systemimage.device.logging.getLogger'): - # It's possible getprop actually does exist on the system. - with patch('systemimage.device.check_output', - side_effect=CalledProcessError(1, 'ignore')): - self.assertEqual(config.device, '?') - - def test_device_no_getprop_fallback(self): + with _patch_device_hook(side_effect=CalledProcessError(1, 'ignore')): + self.assertEqual(config.device, '?') + + @configuration + def test_service_device(self, config_d): + # A configuration file could have a [service]device variable, which + # takes precedence. + shutil.copy(data_path('config.config_11.ini'), + os.path.join(config_d, '01_override.ini')) + with patch('systemimage.device.check_output', return_value='nexus9'): + config = Configuration(config_d) + # This gets the [service]device value from the configuration file, + # not the output of the hook. + self.assertEqual(config.device, 'nexus8') + + @configuration + def test_device_no_getprop_fallback(self, config): # Like above, but a FileNotFoundError occurs instead. - config = Configuration() with _patch_device_hook(): self.assertEqual(config.device, '?') @configuration - def test_get_channel(self, ini_file): - config = Configuration(ini_file) + def test_get_channel(self, config): self.assertEqual(config.channel, 'stable') @configuration - def test_overrides(self, ini_file): - config = Configuration(ini_file) + def test_overrides(self, config): self.assertEqual(config.build_number, 0) self.assertEqual(config.device, 'nexus7') self.assertEqual(config.channel, 'stable') @@ -253,15 +277,30 @@ self.assertEqual(config.device, 'phablet') self.assertEqual(config.channel, 'daily-proposed') - def test_bad_override(self): - config = Configuration() + @configuration + def test_build_number_cli_override(self, config): + # When setting the build number, e.g. --build on the cli, we have an + # additional value we can check. Normally we only care what the build + # number is, but in a one specific case we care whether it was + # overridden on the command line. When a channel alias switch is + # happening we normally set the build number to 0 to force a full + # update. However the user can override this on the cli by setting + # --build, which takes precedence. + self.assertEqual(config.build_number, 0) + self.assertFalse(config.build_number_override) + config.build_number = 108 + self.assertEqual(config.build_number, 108) + self.assertTrue(config.build_number_override) + + @configuration + def test_bad_override(self, config): with self.assertRaises(ValueError) as cm: # Looks like an int, but isn't. config.build_number = '20150801' self.assertEqual(str(cm.exception), 'integer is required, got: str') - def test_reset_build_number(self): - config = Configuration() + @configuration + def test_reset_build_number(self, config): old_build = config.build_number self.assertEqual(old_build, 0) config.build_number = 20990000 @@ -271,57 +310,18 @@ config.build_number = 21000000 self.assertEqual(config.build_number, 21000000) - def test_channel_ini_overrides(self): - # If a /etc/system-image/channels.ini file exists, it overrides any - # previously set options. - default_ini = resource_filename('systemimage.data', 'client.ini') - config = Configuration(default_ini) - # [service] - self.assertEqual(config.service.base, 'system-image.ubuntu.com') - self.assertEqual(config.service.http_base, - 'http://system-image.ubuntu.com') - self.assertEqual(config.service.https_base, - 'https://system-image.ubuntu.com') - self.assertEqual(config.service.channel, 'daily') - self.assertEqual(config.service.build_number, 0) - # Load the overrides. - channel_ini = resource_filename( - 'systemimage.tests.data', 'channel_01.ini') - config.load(channel_ini, override=True) + @configuration('00.ini', 'config.config_09.ini') + def test_later_files_override(self, config): + # This value comes from the 00.ini file. + self.assertEqual(config.system.timeout, timedelta(seconds=1)) + # These get overridden in second ini file. self.assertEqual(config.service.base, 'systum-imaje.ubuntu.com') - self.assertEqual(config.service.http_base, - 'http://systum-imaje.ubuntu.com:88') - self.assertEqual(config.service.https_base, - 'https://systum-imaje.ubuntu.com:89') - self.assertEqual(config.service.channel, 'proposed') - self.assertEqual(config.service.build_number, 1833) - - def test_channel_ini_may_override_system_section(self): - # Overrides may include the [system] section. - default_ini = resource_filename('systemimage.data', 'client.ini') - config = Configuration(default_ini) - channel_ini = data_path('channel_02.ini') - config.load(channel_ini, override=True) - self.assertEqual(config.system.build_file, - '/etc/path/to/alternative/build-file') - # But other [system] settings which come from client.ini are unchanged. - self.assertEqual(config.system.settings_db, - '/var/lib/system-image/settings.db') - - def test_channel_ini_ignored_sections(self): - # Only the [service] and [system] section in channel.ini is used. - default_ini = resource_filename('systemimage.data', 'client.ini') - config = Configuration(default_ini) - channel_ini = data_path('channel_02.ini') - config.load(channel_ini, override=True) - # channel_02.ini sets this to 1h, but it's not overridden. - self.assertEqual(config.dbus.lifetime, timedelta(minutes=10)) - - def test_tempdir(self): + self.assertEqual(config.dbus.lifetime, timedelta(hours=1)) + + @configuration + def test_tempdir(self, config): # config.tempdir is randomly created. - default_ini = resource_filename('systemimage.data', 'client.ini') - config = Configuration(default_ini) - self.assertEqual(config.tempdir[:18], '/tmp/system-image-') + self.assertEqual(config.tempdir[-26:-8], '/tmp/system-image-') self.assertEqual(stat.filemode(os.stat(config.tempdir).st_mode), 'drwx--S---') @@ -345,36 +345,60 @@ self.assertEqual(stdout[:29], 'drwx--S--- /tmp/system-image-') self.assertFalse(os.path.exists(stdout.split()[1])) - def test_constructor(self): - # LP: #1342183: Configuration constructor takes an ini_file argument. - config = Configuration(data_path('config_01.ini')) + @configuration('config.config_01.ini') + def test_constructor(self, config_d): + # Configuration constructor takes an optional directory argument. + config = Configuration(config_d) self.assertEqual(config.service.base, 'phablet.example.com') - - def test_main_ini_file_must_contain_system_stanza(self): - # It's okay if an override is missing the [system] stanza, but the - # main ini file (i.e. non-override) must contain it. - ini_file = data_path('config_09.ini') - config = Configuration() - self.assertRaises(KeyError, config.load, ini_file) - - def test_channel_ini_device_override(self): - # A channel.ini file can override the device name. - config = Configuration(data_path('config_01.ini')) - config.load(data_path('channel_06.ini'), override=True) - self.assertEqual(config.device, 'shoephone') - - def test_channel_ini_missing_device_override(self): - # The channel.ini can omit the [service]device setting, in which case - # the hook is still used. - config = Configuration(data_path('config_01.ini')) - config.load(data_path('channel_05.ini'), override=True) - with _patch_device_hook(): - self.assertEqual(config.device, '?') - - def test_channel_ini_empty_device_override(self): - # The channel.ini can have an empty [service]device setting, in which - # case the hook is still used. - config = Configuration(data_path('config_01.ini')) - config.load(data_path('channel_07.ini'), override=True) - with _patch_device_hook(): - self.assertEqual(config.device, '?') + # Passing in a non-directory is not allowed. + self.assertRaises(TypeError, + Configuration, data_path('config.config_01.ini')) + + @configuration + def test_phased_percentage(self, config): + # By default, the phased percentage override is None. + self.assertIsNone(config.phase_override) + + @configuration + def test_phased_percentage_override(self, config): + # The phased percentage for the device can be overridden. + self.assertIsNone(config.phase_override) + config.phase_override = 33 + self.assertEqual(config.phase_override, 33) + # It can also be reset. + del config.phase_override + self.assertIsNone(config.phase_override) + + @configuration + def test_phased_percentage_override_int(self, config): + # When overriding the phased percentage, the new value must be an int. + self.assertRaises(ValueError, setattr, config, 'phase_override', '!') + + @configuration + def test_crazy_phase(self, config): + config.phase_override = -100 + self.assertEqual(config.phase_override, 0) + config.phase_override = 108 + self.assertEqual(config.phase_override, 100) + config.phase_override = 0 + self.assertEqual(config.phase_override, 0) + config.phase_override = 100 + self.assertEqual(config.phase_override, 100) + + @configuration('config.config_10.ini') + def test_missing_stanza_okay(self, config): + # config_09.ini does not contain a [system] section, so that gets set + # to the built-in default values. + self.assertEqual(config.system.logfile, + '/var/log/system-image/client.log') + + @configuration + def test_user_agent(self, config): + # The User-Agent string contains the device, channel, and build. + config.device = 'geddyboard' + config.channel = 'devel-trio' + config.build_number = 2112 + self.assertEqual( + config.user_agent, + 'Ubuntu System Image Upgrade Client: ' + 'device=geddyboard;channel=devel-trio;build=2112') === modified file 'systemimage/tests/test_dbus.py' --- systemimage/tests/test_dbus.py 2014-09-26 14:36:34 +0000 +++ systemimage/tests/test_dbus.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -21,7 +21,9 @@ 'TestDBusCheckForUpdateToUnwritablePartition', 'TestDBusCheckForUpdateWithBrokenIndex', 'TestDBusDownload', + 'TestDBusDownloadBigFiles', 'TestDBusFactoryReset', + 'TestDBusProductionReset', 'TestDBusGetSet', 'TestDBusInfo', 'TestDBusMiscellaneous', @@ -38,7 +40,6 @@ 'TestDBusRegressions', 'TestDBusUseCache', 'TestLiveDBusInfo', - 'TestLiveDBusInfoWithChannelIni', ] @@ -55,22 +56,41 @@ from dbus.exceptions import DBusException from functools import partial from pathlib import Path +from textwrap import dedent from systemimage.config import Configuration -from systemimage.helpers import MiB, atomic, safe_remove +from systemimage.helpers import MiB, safe_remove from systemimage.reactor import Reactor from systemimage.settings import Settings from systemimage.testing.helpers import ( - copy, find_dbus_process, make_http_server, setup_index, setup_keyring_txz, - setup_keyrings, sign, write_bytes) + copy, data_path, find_dbus_process, make_http_server, setup_index, + setup_keyring_txz, setup_keyrings, sign, terminate_service, touch_build, + wait_for_service, write_bytes) from systemimage.testing.nose import SystemImagePlugin +# Precomputed SHA256 hash for 750MiB of b'x'. +HASH750 = '5fdddb486eeb1aa4dbdada48424418fce5f753844544b6970e4a25879d6d6f52' + + # Use a namedtuple for more convenient argument unpacking. UASRecord = namedtuple('UASRecord', 'is_available downloading available_version update_size ' 'last_update_date error_reason') +def tweak_checksums(checksum): + index_path = os.path.join( + SystemImagePlugin.controller.serverdir, + 'stable', 'nexus7', 'index.json') + with open(index_path, 'r', encoding='utf-8') as fp: + index = json.load(fp) + for i in range(3): + index['images'][0]['files'][i]['checksum'] = checksum + with open(index_path, 'w', encoding='utf-8') as fp: + json.dump(index, fp) + sign(index_path, 'device-signing.gpg') + + class SignalCapturingReactor(Reactor): def __init__(self, *signals): super().__init__(dbus.SystemBus()) @@ -179,15 +199,34 @@ self.quit() +class DoubleFiringReactor(Reactor): + def __init__(self, iface, wait_count=2): + super().__init__(dbus.SystemBus()) + self.iface = iface + self.wait_count = wait_count + self.uas_signals = [] + self.react_to('UpdateAvailableStatus') + + def _do_UpdateAvailableStatus(self, signal, path, *args): + self.uas_signals.append(UASRecord(*args)) + if len(self.uas_signals) >= self.wait_count: + self.quit() + + def run(self): + self.schedule(self.iface.CheckForUpdate, milliseconds=50) + self.schedule(self.iface.CheckForUpdate, milliseconds=55) + super().run() + + class ManualUpdateReactor(Reactor): def __init__(self, iface): super().__init__(dbus.SystemBus()) self.iface = iface - self.rebooting = False + self.applied = False self.react_to('UpdateAvailableStatus') self.react_to('UpdateProgress') self.react_to('UpdateDownloaded') - self.react_to('Rebooting') + self.react_to('Applied') self.react_to('UpdateFailed') self.iface.CheckForUpdate() @@ -207,13 +246,37 @@ def _do_UpdateFailed(self, signal, path, *args, **kws): # Before LP: #1287919 was fixed, this signal would have been sent. - self.rebooting = False - self.quit() - - def _do_Rebooting(self, signal, path, *args, **kws): - # The system is now rebooting . - self.rebooting = True - self.quit() + self.applied = False + self.quit() + + def _do_Applied(self, signal, path, *args, **kws): + # The update was applied. + self.applied = True + self.quit() + + +class AppliedNoRebootingReactor(Reactor): + def __init__(self, iface): + super().__init__(dbus.SystemBus()) + self.iface = iface + # Values here are (received, flag) + self.applied = (False, False) + self.rebooting = (False, False) + self.react_to('Applied') + self.react_to('Rebooting') + self.react_to('UpdateDownloaded') + self.schedule(self.iface.CheckForUpdate) + + def _do_UpdateDownloaded(self, signal, path, *args, **kws): + # The update successfully downloaded, so apply the update now. + self.iface.ApplyUpdate() + + def _do_Applied(self, signal, path, *args): + self.applied = (True, args[0]) + self.quit() + + def _do_Rebooting(self, signal, path, *args): + self.rebooting = (True, args[0]) class _TestBase(unittest.TestCase): @@ -283,7 +346,7 @@ make_http_server(serverdir, 8943, 'cert.pem', 'key.pem')) cls._resources.push(make_http_server(serverdir, 8980)) # Set up the server files. - copy('channels_06.json', serverdir, 'channels.json') + copy('dbus.channels_01.json', serverdir, 'channels.json') sign(os.path.join(serverdir, 'channels.json'), 'image-signing.gpg') # Only the archive-master key is pre-loaded. All the other keys # are downloaded and there will be both a blacklist and device @@ -319,7 +382,7 @@ def setUp(self): super().setUp() - self._prepare_index('index_13.json') + self._prepare_index('dbus.index_01.json') # We need a configuration file that agrees with the dbus client. self.config = Configuration(SystemImagePlugin.controller.ini_path) # For testing reboot preparation. @@ -333,7 +396,7 @@ # Consume the UpdateFailed that results from the cancellation. reactor = SignalCapturingReactor('TornDown') reactor.run(self.iface.TearDown, timeout=15) - safe_remove(self.config.system.build_file) + # Clear out any previously downloaded data files. for updater_dir in (self.config.updater.cache_partition, self.config.updater.data_partition): try: @@ -343,6 +406,11 @@ pass for filename in all_files: safe_remove(os.path.join(updater_dir, filename)) + # Since the controller re-uses the same config_d directory, clear out + # any touched config files that aren't the default. + for ini_file in os.listdir(self.config.config_d): + if ini_file != '00_defaults.ini': + safe_remove(os.path.join(self.config.config_d, ini_file)) safe_remove(self.reboot_log) super().tearDown() @@ -355,18 +423,6 @@ setup_index(index_file, serverdir, 'device-signing.gpg', write_callback) - def _touch_build(self, version, timestamp=None): - # Unlike the touch_build() helper, this one uses our own config object - # rather than the global one. It's not worth messing with - # touch_build() to generalize it. - assert 0 <= version < (1 << 16), ( - 'old style version number: {}'.format(version)) - with open(self.config.system.build_file, 'w', encoding='utf-8') as fp: - print(version, file=fp) - if timestamp is not None: - timestamp = int(timestamp.timestamp()) - os.utime(self.config.system.build_file, (timestamp, timestamp)) - class TestDBusCheckForUpdate(_LiveTesting): """Test the SystemImage dbus service.""" @@ -383,13 +439,12 @@ self.assertFalse(signal.downloading) self.assertEqual(signal.available_version, '1600') self.assertEqual(signal.update_size, 314572800) - # This is the first update applied. - self.assertEqual(signal.last_update_date, 'Unknown') - self.assertEqual(signal.error_reason, '') def test_update_available_auto_download(self): # Automatically download the available update. self.download_always() + timestamp = int(datetime(2022, 8, 1, 10, 11, 12).timestamp()) + touch_build(1701, timestamp, self.config) reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) self.assertEqual(len(reactor.signals), 1) @@ -400,44 +455,38 @@ self.assertEqual(signal.available_version, '1600') self.assertEqual(signal.update_size, 314572800) # This is the first update applied. - self.assertEqual(signal.last_update_date, 'Unknown') + self.assertEqual(signal.last_update_date, '2022-08-01 10:11:12') self.assertEqual(signal.error_reason, '') def test_no_update_available(self): # Our device is newer than the version that's available. - self._touch_build(1701, datetime(2013, 8, 1, 10, 11, 12)) + timestamp = int(datetime(2022, 8, 1, 10, 11, 12).timestamp()) + touch_build(1701, timestamp, self.config) + self.iface.Reset() reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) self.assertEqual(len(reactor.signals), 1) signal = reactor.signals[0] self.assertFalse(signal.is_available) # No update has been previously applied. - self.assertEqual(signal.last_update_date, '2013-08-01 10:11:12') + self.assertEqual(signal.last_update_date, '2022-08-01 10:11:12') # All other values are undefined. def test_last_update_date(self): # Pretend the device got a previous update. Now, there's no update # available, but the date of the last update is provided in the # signal. - timestamp = datetime(2013, 1, 20, 12, 1, 45) - self._touch_build(1701, timestamp) - timestamp = int(timestamp.timestamp()) + timestamp = int(datetime(2022, 1, 20, 12, 1, 45).timestamp()) + touch_build(1701, timestamp, self.config) + self.iface.Reset() # Fake that there was a previous update. - channel_ini = os.path.join( - os.path.dirname(SystemImagePlugin.controller.ini_path), - 'channel.ini') - with ExitStack() as resources: - resources.callback(safe_remove, channel_ini) - with open(channel_ini, 'w', encoding='utf-8'): - pass - os.utime(channel_ini, (timestamp, timestamp)) - reactor = SignalCapturingReactor('UpdateAvailableStatus') - reactor.run(self.iface.CheckForUpdate) + reactor = SignalCapturingReactor('UpdateAvailableStatus') + reactor.run(self.iface.CheckForUpdate) self.assertEqual(len(reactor.signals), 1) signal = reactor.signals[0] self.assertFalse(signal.is_available) # No update has been previously applied. - self.assertEqual(signal.last_update_date, '2013-01-20 12:01:45') + self.assertEqual(signal.last_update_date, '2022-01-20 12:01:45') # All other values are undefined. def test_check_for_update_twice(self): @@ -492,7 +541,8 @@ def test_nothing_to_auto_download(self): # We're auto-downloading, but there's no update available. self.download_always() - self._touch_build(1701) + touch_build(1701, use_config=self.config) + self.iface.Reset() self.assertFalse(os.path.exists(self.command_file)) reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) @@ -546,7 +596,8 @@ def test_nothing_to_manually_download(self): # We're manually downloading, but there's no update available. self.download_manually() - self._touch_build(1701) + touch_build(1701, use_config=self.config) + self.iface.Reset() self.assertFalse(os.path.exists(self.command_file)) reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) @@ -587,7 +638,7 @@ # same local destination file. This is a bug on the server and the # client cannot perform an update. self.download_manually() - self._prepare_index('index_23.json') + self._prepare_index('dbus.index_03.json') reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) self.assertEqual(len(reactor.signals), 1) @@ -603,6 +654,38 @@ self.assertEqual(last_reason[:25], 'DuplicateDestinationError') +class TestDBusDownloadBigFiles(_LiveTesting): + # If the update contains several very large files, ensure that they can be + # successfully downloaded. With the PyCURL downloader, this will ensure + # that the minimum transfer rate error isn't triggered. + def test_download_big_files(self): + # Start by creating some big files which will take a while to + # download. + def write_callback(dst): + # Write a 500 MiB sized file. + write_bytes(dst, 750) + self._prepare_index('dbus.index_04.json', write_callback) + tweak_checksums(HASH750) + # Do the download. + self.download_always() + reactor = SignalCapturingReactor('UpdateDownloaded') + reactor.run(self.iface.CheckForUpdate, timeout=1200) + self.assertEqual(len(reactor.signals), 1) + with open(self.command_file, 'r', encoding='utf-8') as fp: + command = fp.read() + self.assertMultiLineEqual(command, """\ +load_keyring image-master.tar.xz image-master.tar.xz.asc +load_keyring image-signing.tar.xz image-signing.tar.xz.asc +load_keyring device-signing.tar.xz device-signing.tar.xz.asc +format system +mount system +update 6.txt 6.txt.asc +update 7.txt 7.txt.asc +update 5.txt 5.txt.asc +unmount system +""") + + class TestDBusApply(_LiveTesting): def setUp(self): super().setUp() @@ -621,17 +704,42 @@ reboot = fp.read() self.assertEqual(reboot, '/sbin/reboot -f recovery') - def test_reboot_no_update(self): + def test_applied(self): + # Apply the update, and the Applied signal we'll get. + self.assertFalse(os.path.exists(self.reboot_log)) + reactor = SignalCapturingReactor('UpdateDownloaded') + reactor.run(self.iface.CheckForUpdate) + reactor = SignalCapturingReactor('Applied') + reactor.run(self.iface.ApplyUpdate) + self.assertEqual(len(reactor.signals), 1) + self.assertTrue(reactor.signals[0]) + + def test_applied_no_reboot(self): + # Apply the update, but do not reboot. + ini_path = os.path.join( + SystemImagePlugin.controller.ini_path, + '12_noreboot.ini') + shutil.copy(data_path('state.config_01.ini'), ini_path) + self.iface.Reset() + reactor = AppliedNoRebootingReactor(self.iface) + reactor.run() + # We should have gotten only one signal, the Applied. + received, flag = reactor.applied + self.assertTrue(received) + self.assertTrue(flag) + received, flag = reactor.rebooting + self.assertFalse(received) + + def test_applied_no_update(self): # There's no update to reboot to. - self.assertFalse(os.path.exists(self.reboot_log)) - self._touch_build(1701) + touch_build(1701, use_config=self.config) + self.iface.Reset() reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) - reactor = SignalCapturingReactor('Rebooting') + reactor = SignalCapturingReactor('Applied') reactor.run(self.iface.ApplyUpdate) self.assertEqual(len(reactor.signals), 1) self.assertFalse(reactor.signals[0][0]) - self.assertFalse(os.path.exists(self.reboot_log)) def test_reboot_after_update_failed(self): # Cause the update to fail by deleting a file from the server. @@ -646,8 +754,27 @@ failure_count, reason = reactor.signals[0] self.assertEqual(failure_count, 1) self.assertNotEqual(reason, '') - # The reboot fails, so we get an error message. - reactor = SignalCapturingReactor('Rebooting') + # The reboot fails. + reactor = SignalCapturingReactor('Applied') + reactor.run(self.iface.ApplyUpdate) + self.assertEqual(len(reactor.signals), 1) + self.assertFalse(reactor.signals[0][0]) + + def test_applied_after_update_failed(self): + # Cause the update to fail by deleting a file from the server. + self.download_manually() + reactor = SignalCapturingReactor('UpdateAvailableStatus') + reactor.run(self.iface.CheckForUpdate) + os.remove(os.path.join(SystemImagePlugin.controller.serverdir, + '4/5/6.txt.asc')) + reactor = SignalCapturingReactor('UpdateFailed') + reactor.run(self.iface.DownloadUpdate) + self.assertEqual(len(reactor.signals), 1) + failure_count, reason = reactor.signals[0] + self.assertEqual(failure_count, 1) + self.assertNotEqual(reason, '') + # Applying the update fails. + reactor = SignalCapturingReactor('Applied') reactor.run(self.iface.ApplyUpdate) self.assertEqual(len(reactor.signals), 1) self.assertFalse(reactor.signals[0][0]) @@ -663,15 +790,16 @@ def test_exit(self): # There is a D-Bus method to exit the server immediately. + proc = find_dbus_process(SystemImagePlugin.controller.ini_path) self.iface.Exit() - self.assertRaises(DBusException, self.iface.Info) + proc.wait() + self.assertRaises(DBusException, self.iface.Information) # Re-establish a new connection. bus = dbus.SystemBus() service = bus.get_object('com.canonical.SystemImage', '/Service') self.iface = dbus.Interface(service, 'com.canonical.SystemImage') - # There's no update to apply, so we'll get an error string instead of - # the empty string for this call. But it will restart the server. - reactor = SignalCapturingReactor('Rebooting') + # There's no update to apply. + reactor = SignalCapturingReactor('Applied') reactor.run(self.iface.ApplyUpdate) self.assertEqual(len(reactor.signals), 1) self.assertFalse(reactor.signals[0][0]) @@ -801,7 +929,7 @@ self.assertEqual(percentage, i) self.assertEqual(eta, 50 - (i * 0.5)) self.assertTrue(reactor.downloaded) - reactor = SignalCapturingReactor('Rebooting') + reactor = SignalCapturingReactor('Applied') reactor.run(self.iface.ApplyUpdate) self.assertEqual(len(reactor.signals), 1) self.assertTrue(reactor.signals[0]) @@ -922,7 +1050,7 @@ self.assertEqual(percentage, i) self.assertEqual(eta, 50 - (i * 0.5)) self.assertTrue(reactor.downloaded) - reactor = SignalCapturingReactor('Rebooting') + reactor = SignalCapturingReactor('Applied') reactor.run(self.iface.ApplyUpdate) self.assertEqual(len(reactor.signals), 1) self.assertTrue(reactor.signals[0]) @@ -1014,7 +1142,7 @@ '1983-09-13T12:13:14') self.assertEqual(reactor.status.error_reason, '') self.assertTrue(reactor.downloaded) - reactor = SignalCapturingReactor('Rebooting') + reactor = SignalCapturingReactor('Applied') reactor.run(self.iface.ApplyUpdate) self.assertEqual(len(reactor.signals), 1) self.assertFalse(bool(reactor.signals[0][0])) @@ -1116,9 +1244,9 @@ file_path = os.path.join(serverdir, '5', '6', '7.txt') # This index file has a 5/6/7.txt checksum equal to the one we're # going to create below. - setup_index('index_18.json', serverdir, 'device-signing.gpg') + setup_index('dbus.index_02.json', serverdir, 'device-signing.gpg') head, tail = os.path.split(index_path) - copy('index_18.json', head, tail) + copy('dbus.index_02.json', head, tail) sign(index_path, 'device-signing.gpg') write_bytes(file_path, 50) sign(file_path, 'device-signing.gpg') @@ -1202,7 +1330,7 @@ # Set a key, restart the dbus server, and the key's value persists. self.iface.SetSetting('permanent', 'waves') self.assertEqual(self.iface.GetSetting('permanent'), 'waves') - self.iface.Exit() + terminate_service() self.assertRaises(DBusException, self.iface.GetSetting, 'permanent') # Re-establish a new connection. bus = dbus.SystemBus() @@ -1300,7 +1428,7 @@ mode = 'more-info' def test_info(self): - # .Info() with a channel.ini containing version details. + # .Info() with some version details. buildno, device, channel, last_update, details = self.iface.Info() self.assertEqual(buildno, 45) self.assertEqual(device, 'nexus11') @@ -1309,7 +1437,7 @@ self.assertEqual(details, dict(ubuntu='123', mako='456', custom='789')) def test_information(self): - # .Information() with a channel.ini containing version details. + # .Information() with some version details. response = self.iface.Information() self.assertEqual( sorted(str(key) for key in response), [ @@ -1333,8 +1461,10 @@ class TestLiveDBusInfo(_LiveTesting): def test_info_no_version_detail(self): - # .Info() where there is no channel.ini with version details. - self._touch_build(45, datetime(2022, 8, 1, 4, 45, 45)) + # .Info() where there are no version details. + timestamp = int(datetime(2022, 8, 1, 4, 45, 45).timestamp()) + touch_build(45, timestamp, self.config) + self.iface.Reset() buildno, device, channel, last_update, details = self.iface.Info() self.assertEqual(buildno, 45) self.assertEqual(device, 'nexus7') @@ -1343,9 +1473,11 @@ self.assertEqual(details, {}) def test_information_before_check_no_details(self): - # .Information() where there is no channel.ini with version details, - # and no previous CheckForUpdate() call was made. - self._touch_build(45, datetime(2022, 8, 1, 4, 45, 45)) + # .Information() where there are no version details, and no previous + # CheckForUpdate() call was made. + timestamp = int(datetime(2022, 8, 1, 4, 45, 45).timestamp()) + touch_build(45, timestamp, self.config) + self.iface.Reset() response = self.iface.Information() self.assertEqual(response['current_build_number'], '45') self.assertEqual(response['device_name'], 'nexus7') @@ -1356,9 +1488,11 @@ self.assertEqual(response['target_build_number'], '-1') def test_information_no_details(self): - # .Information() where there is no channel.ini with version details, - # but a previous CheckForUpdate() call was made. - self._touch_build(45, datetime(2022, 8, 1, 4, 45, 45)) + # .Information() where there are no version details, but a previous + # CheckForUpdate() call was made. + timestamp = int(datetime(2022, 8, 1, 4, 45, 45).timestamp()) + touch_build(45, timestamp, self.config) + self.iface.Reset() reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) # Before we get the information, let's poke a known value into the @@ -1383,19 +1517,19 @@ self.assertEqual(response['target_build_number'], '1600') def test_information(self): - # .Information() where there is a channel.ini with version details, - # and a previous CheckForUpdate() call was made. - self._touch_build(45) + # .Information() where there there are version details, and a previous + # CheckForUpdate() call was made. + timestamp = int(datetime(2022, 8, 1, 4, 45, 45).timestamp()) + touch_build(45, timestamp, use_config=self.config) ini_path = Path(SystemImagePlugin.controller.ini_path) - channel_path = ini_path.with_name('channel.ini') - with channel_path.open('w', encoding='utf-8') as fp: + override_ini = ini_path / '03_override.ini' + with override_ini.open('w', encoding='utf-8') as fp: print("""\ [service] version_detail: ubuntu=222,mako=333,custom=444 """, file=fp) + self.iface.Reset() # Set last_update_date. - timestamp = int(datetime(2022, 8, 1, 4, 45, 45).timestamp()) - os.utime(str(channel_path), (timestamp, timestamp)) reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) # Before we get the information, let's poke a known value into the @@ -1415,7 +1549,8 @@ self.assertEqual(response['device_name'], 'nexus7') self.assertEqual(response['channel_name'], 'stable') self.assertEqual(response['last_update_date'], '2022-08-01 04:45:45') - self.assertEqual(response['version_detail'], '') + self.assertEqual(response['version_detail'], + 'ubuntu=222,mako=333,custom=444') # We can't really check the returned last check date against anything # in a robust way. E.g. what if we just happen to be at 12:59:59 on # December 31st? Let's at least make sure it has a sane format. @@ -1425,7 +1560,8 @@ def test_information_no_update_available(self): # .Information() where we know that no update is available, gives us a # target build number equal to the current build number. - self._touch_build(1701) + touch_build(1701, use_config=self.config) + self.iface.Reset() reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) signal = reactor.signals[0] @@ -1436,7 +1572,7 @@ def test_information_workflow(self): # At first, .Information() won't know whether there is an update # available or not. Then we check, and it tells us there is one. - self._touch_build(45) + touch_build(45, use_config=self.config) response = self.iface.Information() self.assertEqual(response['target_build_number'], '-1') reactor = SignalCapturingReactor('UpdateAvailableStatus') @@ -1446,41 +1582,60 @@ response = self.iface.Information() self.assertEqual(response['target_build_number'], '1600') - -class TestLiveDBusInfoWithChannelIni(_LiveTesting): - @classmethod - def setUpClass(cls): - # This must be done before the D-Bus service is started. - ini_path = Path(SystemImagePlugin.controller.ini_path) - channel_path = ini_path.with_name('channel.ini') - with channel_path.open('w', encoding='utf-8') as fp: - print("""\ -[service] -version_detail: ubuntu=222,mako=333,custom=444 -build_number: 42 -""", file=fp) - # Set last_update_date. - timestamp = int(datetime(2022, 8, 1, 4, 45, 45).timestamp()) - os.utime(str(channel_path), (timestamp, timestamp)) - # Set the build number. - config = Configuration(SystemImagePlugin.controller.ini_path) - # We can't use _touch_build(). - with open(config.system.build_file, 'w', encoding='utf-8') as fp: - print(45, file=fp) - super().setUpClass() - - def test_information_before_check(self): - self._touch_build(45) - # .Information() where there is a channel.ini with version details, - # and no previous CheckForUpdate() call was made. - response = self.iface.Information() - self.assertEqual(response['current_build_number'], '42') - self.assertEqual(response['device_name'], 'nexus7') - self.assertEqual(response['channel_name'], 'stable') - self.assertEqual(response['last_update_date'], '2022-08-01 04:45:45') - self.assertEqual(response['version_detail'], - 'ubuntu=222,mako=333,custom=444') - self.assertEqual(response['last_check_date'], '') + def test_target_version_detail_before_check(self): + # Before we do a CheckForUpdate, there is no target version detail. + timestamp = int(datetime(2022, 8, 1, 4, 45, 45).timestamp()) + touch_build(45, timestamp, self.config) + self.iface.Reset() + response = self.iface.Information() + self.assertEqual(response['version_detail'], '') + self.assertEqual(response['target_version_detail'], '') + + def test_target_version_detail_after_check_no_update_available(self): + # After a CheckForUpdate, if there is no update available, the target + # version detail is the same as the version detail. + ini_path = Path(SystemImagePlugin.controller.ini_path) + override_ini = ini_path / '03_override.ini' + with override_ini.open('w', encoding='utf-8') as fp: + print("""\ +[service] +version_detail: ubuntu=401,mako=501,custom=601 +""", file=fp) + timestamp = int(datetime(2022, 8, 1, 4, 45, 45).timestamp()) + touch_build(1700, timestamp, use_config=self.config) + self.iface.Reset() + reactor = SignalCapturingReactor('UpdateAvailableStatus') + reactor.run(self.iface.CheckForUpdate) + response = self.iface.Information() + self.assertEqual(response['version_detail'], + 'ubuntu=401,mako=501,custom=601') + self.assertEqual(response['target_version_detail'], + 'ubuntu=401,mako=501,custom=601') + + def test_target_version_detail_after_check_update_available(self): + # After a CheckForUpdate, if there is an update available, the target + # version detail is the new update. + ini_path = Path(SystemImagePlugin.controller.ini_path) + override_ini = ini_path / '03_override.ini' + with override_ini.open('w', encoding='utf-8') as fp: + print("""\ +[service] +version_detail: ubuntu=401,mako=501,custom=601 +""", file=fp) + timestamp = int(datetime(2022, 8, 1, 4, 45, 45).timestamp()) + touch_build(45, timestamp, use_config=self.config) + # This index.json file is exactly like the tests's default + # dbus.index_01.json file except that it has version_detail keys in + # the image sections. + self._prepare_index('dbus.index_06.json') + self.iface.Reset() + reactor = SignalCapturingReactor('UpdateAvailableStatus') + reactor.run(self.iface.CheckForUpdate) + response = self.iface.Information() + self.assertEqual(response['version_detail'], + 'ubuntu=401,mako=501,custom=601') + self.assertEqual(response['target_version_detail'], + 'ubuntu=402,mako=502,custom=602') class TestDBusFactoryReset(_LiveTesting): @@ -1502,10 +1657,32 @@ self.assertEqual(command, 'format data\n') +class TestDBusProductionReset(_LiveTesting): + def test_production_reset(self): + # A production factory reset is applied. + command_file = os.path.join( + self.config.updater.cache_partition, 'ubuntu_command') + self.assertFalse(os.path.exists(self.reboot_log)) + self.assertFalse(os.path.exists(command_file)) + reactor = SignalCapturingReactor('Rebooting') + reactor.run(self.iface.ProductionReset) + self.assertEqual(len(reactor.signals), 1) + self.assertTrue(reactor.signals[0]) + with open(self.reboot_log, encoding='utf-8') as fp: + reboot = fp.read() + self.assertEqual(reboot, '/sbin/reboot -f recovery') + with open(command_file, encoding='utf-8') as fp: + command = fp.read() + self.assertMultiLineEqual(command, dedent("""\ + format data + enable factory_wipe + """)) + + class TestDBusProgress(_LiveTesting): def test_progress(self): self.download_manually() - self._touch_build(0) + touch_build(0, use_config=self.config) reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) self.assertEqual(len(reactor.signals), 1) @@ -1538,25 +1715,16 @@ full_path = os.path.join( SystemImagePlugin.controller.serverdir, path) write_bytes(full_path, 750) - # Disable the checksums - they just get in the way of these tests. - index_path = os.path.join(SystemImagePlugin.controller.serverdir, - 'stable', 'nexus7', 'index.json') - with open(index_path, 'r', encoding='utf-8') as fp: - index = json.load(fp) - for i in range(3): - index['images'][0]['files'][i]['checksum'] = '' - with open(index_path, 'w', encoding='utf-8') as fp: - json.dump(index, fp) - sign(index_path, 'device-signing.gpg') + tweak_checksums('') def test_pause(self): self.download_manually() - self._touch_build(0) + touch_build(0, use_config=self.config) reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) self.assertEqual(len(reactor.signals), 1) # There must be an update available. - self.assertTrue(reactor.signals[0][0]) + self.assertTrue(reactor.signals[0].is_available) # We're ready to start downloading. We schedule a pause to happen in # a little bit and then ensure that we get the proper signal. reactor = PausingReactor(self.iface) @@ -1612,7 +1780,7 @@ # download the ancillary files (i.e. channels.json, index.json, # keyrings), but not the data files. self.download_always() - self._touch_build(0) + touch_build(0, use_config=self.config) reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) self.assertEqual(len(reactor.signals), 1) @@ -1655,19 +1823,16 @@ path = os.path.join(config.updater.cache_partition, filename) if filename.endswith('.txt') or filename.endswith('.txt.asc'): mtimes[filename] = os.stat(path).st_mtime_ns - # Don't issue the reboot. Instead, kill the services, which throws - # away all state, but does not delete the cached files. - # Re-establish a new connection. - process = find_dbus_process(SystemImagePlugin.controller.ini_path) - if process is not None: - self.iface.Exit() - process.wait(60) + # Don't issue the reboot. Instead, kill the service, which throws away + # all state, but does not delete the cached files. Re-establish a new + # connection. + terminate_service() bus = dbus.SystemBus() service = bus.get_object('com.canonical.SystemImage', '/Service') self.iface = dbus.Interface(service, 'com.canonical.SystemImage') # Now, if we just apply the update, it will succeed, since it knows # that the cached files are valid. - reactor = SignalCapturingReactor('Rebooting') + reactor = SignalCapturingReactor('Applied') reactor.run(self.iface.ApplyUpdate) self.assertEqual(len(reactor.signals), 1) self.assertTrue(reactor.signals[0]) @@ -1727,7 +1892,9 @@ def write_callback(dst): # Write a 100 MiB sized file. write_bytes(dst, 100) - self._prepare_index('index_24.json', write_callback) + self._prepare_index('dbus.index_04.json', write_callback) + timestamp = int(datetime(2022, 8, 1, 10, 11, 12).timestamp()) + touch_build(0, timestamp, self.config) # Create a reactor that will exit when the UpdateDownloaded signal is # received. We're going to issue a CheckForUpdate with automatic # updates enabled. As soon as we receive the UpdateAvailableStatus @@ -1748,7 +1915,7 @@ self.assertTrue(signal.downloading) self.assertEqual(signal.available_version, '1600') self.assertEqual(signal.update_size, 314572800) - self.assertEqual(signal.last_update_date, 'Unknown') + self.assertEqual(signal.last_update_date, '2022-08-01 10:11:12') self.assertEqual(signal.error_reason, '') def test_multiple_check_for_updates_with_manual_downloading(self): @@ -1767,8 +1934,8 @@ def write_callback(dst): # Write a 100 MiB sized file. write_bytes(dst, 100) - self._prepare_index('index_24.json', write_callback) - self._touch_build(0) + self._prepare_index('dbus.index_04.json', write_callback) + touch_build(0, use_config=self.config) # Create a reactor that implements the following test plan: # * Set the device to download manually. # * Flash to an older revision @@ -1782,28 +1949,59 @@ # the update, and reboot reactor = ManualUpdateReactor(self.iface) reactor.run() - self.assertTrue(reactor.rebooting) - - + self.assertTrue(reactor.applied) + + def test_schedule_lots_of_checks(self): + # There is a checking lock in the D-Bus layer. If that lock cannot be + # acquired *and* the results of a previous check have already been + # cached, then the cached results are returned. + self.download_manually() + reactor = SignalCapturingReactor('UpdateAvailableStatus') + reactor.run(self.iface.CheckForUpdate) + # At this point, we now have a cached update status. Although this is + # timing dependent, schedule two more CheckForUpdates right after each + # other. The second one should get caught by the checking lock. + reactor = DoubleFiringReactor(self.iface) + reactor.run() + self.assertEqual(reactor.uas_signals[0], reactor.uas_signals[1]) + + +from systemimage.testing.controller import USING_PYCURL + +@unittest.skipIf(os.getuid() == 0, 'Test cannot succeed when run as root') +@unittest.skipUnless(USING_PYCURL, 'LP: #1411866') class TestDBusCheckForUpdateToUnwritablePartition(_LiveTesting): @classmethod def setUpClass(cls): - ini_path = SystemImagePlugin.controller.ini_path - with open(ini_path, 'r', encoding='utf-8') as fp: - lines = fp.readlines() - with atomic(ini_path) as fp: - for line in lines: - key, sep, value = line.partition(': ') - if key == 'cache_partition': - cls.bad_path = os.path.join(value.strip(), 'unwritable') - print('{}: {}'.format(key, cls.bad_path), file=fp) - os.makedirs(cls.bad_path, mode=0) - else: - fp.write(line) super().setUpClass() + # Put cache_partition in an unwritable directory. + config = Configuration(SystemImagePlugin.controller.ini_path) + cache_partition = config.updater.cache_partition + cls.bad_path = Path(cache_partition) / 'unwritable' + cls.bad_path.mkdir(mode=0, parents=True) + # Write a .ini file to override the cache partition. + cls.override = os.path.join(config.config_d, '10_override.ini') + with open(cls.override, 'w', encoding='utf-8') as fp: + print("""\ +[updater] +cache_partition: {} +""".format(cls.bad_path), file=fp) + + @classmethod + def tearDownClass(cls): + safe_remove(cls.override) + shutil.rmtree(str(cls.bad_path)) + super().tearDownClass() + + def setUp(self): + # wait_for_service() must be called befor the upcall to setUp(), + # otherwise self will have an iface attribute pointing to a defunct + # proxy. + wait_for_service(restart=True) + super().setUp() def tearDown(self): - os.chmod(self.bad_path, 0o777) + self.bad_path.chmod(0o777) super().tearDown() def test_check_for_update_error(self): @@ -1821,7 +2019,7 @@ # LP: #1222910. A broken index.json file contained an image with type # == 'delta' but no base field. This breaks the hash calculation of # that image and causes the check-for-update to fail. - self._prepare_index('index_25.json') + self._prepare_index('dbus.index_05.json') reactor = SignalCapturingReactor('UpdateAvailableStatus') reactor.run(self.iface.CheckForUpdate) self.assertEqual(len(reactor.signals), 1) === modified file 'systemimage/tests/test_download.py' --- systemimage/tests/test_download.py 2014-09-17 13:41:31 +0000 +++ systemimage/tests/test_download.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -16,8 +16,10 @@ """Test asynchronous downloads.""" __all__ = [ + 'TestCURL', + 'TestDownload', 'TestDownloadBigFiles', - 'TestDownloads', + 'TestDownloadManagerFactory', 'TestDuplicateDownloads', 'TestGSMDownloads', 'TestHTTPSDownloads', @@ -29,41 +31,47 @@ import os -import sys import random import unittest from contextlib import ExitStack -from datetime import datetime, timedelta -from gi.repository import GLib +from dbus.exceptions import DBusException from hashlib import sha256 from systemimage.config import Configuration, config +from systemimage.curl import CurlDownloadManager from systemimage.download import ( - Canceled, DBusDownloadManager, DuplicateDestinationError, Record) + Canceled, DuplicateDestinationError, Record, get_download_manager) from systemimage.helpers import temporary_directory from systemimage.settings import Settings +from systemimage.testing.controller import USING_PYCURL from systemimage.testing.helpers import ( - configuration, data_path, make_http_server, write_bytes) + configuration, data_path, make_http_server, reset_envar, write_bytes) from systemimage.testing.nose import SystemImagePlugin +from systemimage.udm import DOWNLOADER_INTERFACE, UDMDownloadManager from unittest.mock import patch from urllib.parse import urljoin +if USING_PYCURL: + import pycurl + def _http_pathify(downloads): return [ - (urljoin(config.service.http_base, url), + (urljoin(config.http_base, url), os.path.join(config.tempdir, filename) ) for url, filename in downloads] def _https_pathify(downloads): return [ - (urljoin(config.service.https_base, url), + (urljoin(config.https_base, url), os.path.join(config.tempdir, filename) ) for url, filename in downloads] -class TestDownloads(unittest.TestCase): +class TestDownload(unittest.TestCase): + """Base class for testing the PyCURL and udm downloaders.""" + def setUp(self): super().setUp() self._resources = ExitStack() @@ -80,12 +88,15 @@ self._resources.close() super().tearDown() + def _downloader(self, *args): + return get_download_manager(*args) + @configuration def test_good_path(self): # Download a bunch of files that exist. No callback. - DBusDownloadManager().get_files(_http_pathify([ - ('channels_01.json', 'channels.json'), - ('index_01.json', 'index.json'), + self._downloader().get_files(_http_pathify([ + ('channel.channels_05.json', 'channels.json'), + ('download.index_01.json', 'index.json'), ])) self.assertEqual( set(os.listdir(config.tempdir)), @@ -94,18 +105,19 @@ @configuration def test_empty_download(self): # Empty download set completes successfully. LP: #1245597. - DBusDownloadManager().get_files([]) + self._downloader().get_files([]) # No TimeoutError is raised. @configuration def test_user_agent(self): # The User-Agent request header contains the build number. version = random.randint(0, 99) - with open(config.system.build_file, 'w', encoding='utf-8') as fp: - print(version, file=fp) + config.build_number = version + config.device = 'geddyboard' + config.channel = 'devel-trio' # Download a magic path which the server will interpret to return us # the User-Agent header value. - DBusDownloadManager().get_files(_http_pathify([ + self._downloader().get_files(_http_pathify([ ('user-agent.txt', 'user-agent.txt'), ])) path = os.path.join(config.tempdir, 'user-agent.txt') @@ -113,7 +125,9 @@ user_agent = fp.read() self.assertEqual( user_agent, - 'Ubuntu System Image Upgrade Client; Build {}'.format(version)) + 'Ubuntu System Image Upgrade Client: ' + 'device=geddyboard;channel=devel-trio;build={}'.format( + version)) @configuration def test_download_with_callback(self): @@ -124,10 +138,10 @@ nonlocal received_bytes, total_bytes received_bytes = received total_bytes = total - downloader = DBusDownloadManager(callback) + downloader = self._downloader(callback) downloader.get_files(_http_pathify([ - ('channels_01.json', 'channels.json'), - ('index_01.json', 'index.json'), + ('channel.channels_05.json', 'channels.json'), + ('download.index_01.json', 'index.json'), ])) self.assertEqual( set(os.listdir(config.tempdir)), @@ -144,30 +158,18 @@ def capture(message): nonlocal exception exception = message - downloader = DBusDownloadManager(callback) + downloader = self._downloader(callback) with patch('systemimage.download.log.exception', capture): downloader.get_files(_http_pathify([ - ('channels_01.json', 'channels.json'), + ('channel.channels_05.json', 'channels.json'), ])) # The exception got logged. self.assertEqual(exception, 'Exception in progress callback') # The file still got downloaded. self.assertEqual(os.listdir(config.tempdir), ['channels.json']) - @configuration - def test_no_dev_package(self): - # system-image-dev contains the systemimage.testing subpackage, but - # this is not normally installed on the device. When it's missing, - # the DownloadReactor's _print() debugging method should no-op. - # - # To test this, we patch systemimage.testing in sys.modules so that an - # ImportError is raised when it tries to import it. - with patch.dict(sys.modules, {'systemimage.testing.helpers': None}): - DBusDownloadManager().get_files(_http_pathify([ - ('channels_01.json', 'channels.json'), - ])) - self.assertEqual(os.listdir(config.tempdir), ['channels.json']) - + # This test helps bump the udm-based downloader test coverage to 100%. + @unittest.skipIf(USING_PYCURL, 'Test is not relevant for PyCURL') @configuration def test_timeout(self): # If the reactor times out, we get an exception. We fake the timeout @@ -176,12 +178,12 @@ def finish_with_timeout(self, *args, **kws): self.timed_out = True self.quit() - with patch('systemimage.download.DownloadReactor._do_finished', + with patch('systemimage.udm.DownloadReactor._do_finished', finish_with_timeout): self.assertRaises( TimeoutError, - DBusDownloadManager().get_files, - _http_pathify([('channels_01.json', 'channels.json')]) + self._downloader().get_files, + _http_pathify([('channel.channels_05.json', 'channels.json')]) ) @@ -200,8 +202,8 @@ with ExitStack() as stack: stack.push(make_http_server( self._directory, 8943, 'cert.pem', 'key.pem')) - DBusDownloadManager().get_files(_https_pathify([ - ('channels_01.json', 'channels.json'), + get_download_manager().get_files(_https_pathify([ + ('channel.channels_05.json', 'channels.json'), ])) self.assertEqual( set(os.listdir(config.tempdir)), @@ -223,9 +225,9 @@ with make_http_server(self._directory, 8943, 'cert.pem', 'key.pem'): self.assertRaises( FileNotFoundError, - DBusDownloadManager().get_files, + get_download_manager().get_files, _https_pathify([ - ('channels_01.json', 'channels.json'), + ('channel.channels_05.json', 'channels.json'), ])) @configuration @@ -238,9 +240,9 @@ stack.push(make_http_server(self._directory, 8943)) self.assertRaises( FileNotFoundError, - DBusDownloadManager().get_files, + get_download_manager().get_files, _https_pathify([ - ('channels_01.json', 'channels.json'), + ('channel.channels_05.json', 'channels.json'), ])) @@ -261,9 +263,9 @@ self._directory, 8943, 'expired_cert.pem', 'expired_key.pem')) self.assertRaises( FileNotFoundError, - DBusDownloadManager().get_files, + get_download_manager().get_files, _https_pathify([ - ('channels_01.json', 'channels.json'), + ('channel.channels_05.json', 'channels.json'), ])) @@ -284,12 +286,16 @@ self._directory, 8943, 'nasty_cert.pem', 'nasty_key.pem')) self.assertRaises( FileNotFoundError, - DBusDownloadManager().get_files, + get_download_manager().get_files, _https_pathify([ - ('channels_01.json', 'channels.json'), + ('channel.channels_05.json', 'channels.json'), ])) +# These tests don't strictly improve coverage for the udm-based downloader, +# but they are still useful to keep because they test a implicit code path. +# These can be removed once GSM-testing is pulled into s-i via LP: #1388886. +@unittest.skipIf(USING_PYCURL, 'Test is not relevant for PyCURL') class TestGSMDownloads(unittest.TestCase): def setUp(self): super().setUp() @@ -311,9 +317,9 @@ directory = os.path.dirname(data_path('__init__.py')) self._resources.push(make_http_server(directory, 8980)) # Patch the GSM setting method to capture what actually happens. - self._original = getattr(DBusDownloadManager, '_set_gsm') + self._original = getattr(UDMDownloadManager, '_set_gsm') self._resources.enter_context(patch( - 'systemimage.download.DBusDownloadManager._set_gsm', set_gsm)) + 'systemimage.udm.UDMDownloadManager._set_gsm', set_gsm)) self._resources.callback(setattr, self, '_original', None) except: self._resources.close() @@ -324,42 +330,44 @@ super().tearDown() @configuration - def test_manual_downloads_gsm_allowed(self, ini_file): + def test_manual_downloads_gsm_allowed(self, config_d): # When auto_download is 0, manual downloads are enabled so assuming # the user knows what they're doing, GSM downloads are allowed. - config = Configuration(ini_file) + config = Configuration(config_d) Settings(config).set('auto_download', '0') - DBusDownloadManager().get_files(_http_pathify([ - ('channels_01.json', 'channels.json') + get_download_manager().get_files(_http_pathify([ + ('channel.channels_05.json', 'channels.json') ])) self.assertTrue(self._gsm_set_flag) self.assertTrue(self._gsm_get_flag) @configuration - def test_wifi_downloads_gsm_disallowed(self, ini_file): + def test_wifi_downloads_gsm_disallowed(self, config_d): # Obviously GSM downloads are not allowed when downloading # automatically on wifi-only. - config = Configuration(ini_file) + config = Configuration(config_d) Settings(config).set('auto_download', '1') - DBusDownloadManager().get_files(_http_pathify([ - ('channels_01.json', 'channels.json') + get_download_manager().get_files(_http_pathify([ + ('channel.channels_05.json', 'channels.json') ])) self.assertFalse(self._gsm_set_flag) self.assertFalse(self._gsm_get_flag) @configuration - def test_always_downloads_gsm_allowed(self, ini_file): + def test_always_downloads_gsm_allowed(self, config_d): # GSM downloads are allowed when always downloading. - config = Configuration(ini_file) + config = Configuration(config_d) Settings(config).set('auto_download', '2') - DBusDownloadManager().get_files(_http_pathify([ - ('channels_01.json', 'channels.json') + get_download_manager().get_files(_http_pathify([ + ('channel.channels_05.json', 'channels.json') ])) self.assertTrue(self._gsm_set_flag) self.assertTrue(self._gsm_get_flag) class TestDownloadBigFiles(unittest.TestCase): + # This test helps bump the udm-based downloader test coverage to 100%. + @unittest.skipIf(USING_PYCURL, 'Test is not relevant for PyCURL') @configuration def test_cancel(self): # Try to cancel the download of a big file. @@ -373,12 +381,12 @@ # The download service doesn't provide reliable cancel # granularity, so instead, we mock the 'started' signal to # immediately cancel the download. - downloader = DBusDownloadManager() + downloader = get_download_manager() def cancel_on_start(self, signal, path, started): if started: downloader.cancel() stack.enter_context(patch( - 'systemimage.download.DownloadReactor._do_started', + 'systemimage.udm.DownloadReactor._do_started', cancel_on_start)) self.assertRaises( Canceled, downloader.get_files, _http_pathify([ @@ -407,53 +415,11 @@ ('missing.txt', 'missing.txt'), ]) self.assertRaises(FileNotFoundError, - DBusDownloadManager().get_files, + get_download_manager().get_files, downloads) # The temporary directory is empty. self.assertEqual(os.listdir(config.tempdir), []) - @configuration - def test_download_pause_resume(self): - with ExitStack() as stack: - serverdir = stack.enter_context(temporary_directory()) - stack.push(make_http_server(serverdir, 8980)) - # Create a couple of big files to download. - write_bytes(os.path.join(serverdir, 'bigfile_1.dat'), 10) - write_bytes(os.path.join(serverdir, 'bigfile_2.dat'), 10) - write_bytes(os.path.join(serverdir, 'bigfile_3.dat'), 10) - downloads = _http_pathify([ - ('bigfile_1.dat', 'bigfile_1.dat'), - ('bigfile_2.dat', 'bigfile_2.dat'), - ('bigfile_3.dat', 'bigfile_3.dat'), - ]) - downloader = DBusDownloadManager() - pauses = [] - def do_paused(self, signal, path, paused): - if paused: - pauses.append(datetime.now()) - resumes = [] - def do_resumed(self, signal, path, resumed): - if resumed: - resumes.append(datetime.now()) - def pause_on_start(self, signal, path, started): - if started: - downloader.pause() - GLib.timeout_add_seconds(3, downloader.resume) - stack.enter_context( - patch('systemimage.download.DownloadReactor._do_paused', - do_paused)) - stack.enter_context( - patch('systemimage.download.DownloadReactor._do_resumed', - do_resumed)) - stack.enter_context( - patch('systemimage.download.DownloadReactor._do_started', - pause_on_start)) - downloader.get_files(downloads, pausable=True) - self.assertEqual(len(pauses), 1) - self.assertEqual(len(resumes), 1) - self.assertGreaterEqual(resumes[0] - pauses[0], - timedelta(seconds=2.5)) - class TestRecord(unittest.TestCase): def test_record(self): @@ -506,7 +472,7 @@ checksum = sha256(content).hexdigest() with open(os.path.join(self._serverdir, 'source.dat'), 'wb') as fp: fp.write(content) - downloader = DBusDownloadManager() + downloader = get_download_manager() downloads = [] for url, dst in _http_pathify([('source.dat', 'local.dat'), ('source.dat', 'local.dat'), @@ -525,7 +491,7 @@ fp.write(content) with open(os.path.join(self._serverdir, 'source2.dat'), 'wb') as fp: fp.write(content) - downloader = DBusDownloadManager() + downloader = get_download_manager() downloads = [] for url, dst in _http_pathify([('source1.dat', 'local.dat'), ('source2.dat', 'local.dat'), @@ -549,8 +515,8 @@ checksum = sha256(content).hexdigest() with open(os.path.join(self._serverdir, 'source.dat'), 'wb') as fp: fp.write(content) - downloader = DBusDownloadManager() - url = urljoin(config.service.http_base, 'source.dat') + downloader = get_download_manager() + url = urljoin(config.http_base, 'source.dat') downloads = [ Record(url, 'local.dat', checksum), # Mutate the checksum so they won't match. @@ -581,8 +547,8 @@ checksum = sha256(content).hexdigest() with open(os.path.join(self._serverdir, 'source.dat'), 'wb') as fp: fp.write(content) - downloader = DBusDownloadManager() - url = urljoin(config.service.http_base, 'source.dat') + downloader = get_download_manager() + url = urljoin(config.http_base, 'source.dat') downloads = [ Record(url, 'local.dat', checksum), # Mutate the checksum so they won't match. @@ -598,3 +564,151 @@ ( 'http://localhost:8980/source.dat', 'local.dat', '809ecb6ebc8bcefc733f6f2ec44f791abeed6a99edf0cc31519637898aebd52d')])]""") + + +# This class only bumps coverage to 100% for the cURL-based downloader, so it +# can be skipped when the test suite runs under u-d-m. Checking the +# environment variable wouldn't be enough for production (see download.py +# get_download_manager() for other cases where the downloader is chosen), but +# it's sufficient for the test suite. See tox.ini. +@unittest.skipUnless(USING_PYCURL, 'Test is not relevant for UDM') +class TestCURL(unittest.TestCase): + def setUp(self): + super().setUp() + self._resources = ExitStack() + try: + # Start the HTTP server running, vending files out of our test + # data directory. + directory = os.path.dirname(data_path('__init__.py')) + self._resources.push(make_http_server(directory, 8980)) + except: + self._resources.close() + raise + + def tearDown(self): + self._resources.close() + super().tearDown() + + @configuration + def test_multi_perform(self): + # PyCURL's multi.perform() can return the E_CALL_MULTI_PEFORM status + # which tells us to just try again. This doesn't happen in practice, + # but the code path needs coverage. However, .perform() itself can't + # be mocked because pycurl.CurlMulti is a built-in. Fun. + class FakeMulti: + def perform(self): + return pycurl.E_CALL_MULTI_PERFORM, 2 + done_once = False + class Testable(CurlDownloadManager): + def _do_once(self, multi, handles): + nonlocal done_once + if done_once: + return super()._do_once(multi, handles) + else: + done_once = True + return super()._do_once(FakeMulti(), handles) + Testable().get_files(_http_pathify([ + ('channel.channels_05.json', 'channels.json'), + ('download.index_01.json', 'index.json'), + ])) + self.assertTrue(done_once) + # The files still get downloaded. + self.assertEqual( + set(os.listdir(config.tempdir)), + set(['channels.json', 'index.json'])) + + @configuration + def test_multi_fail(self): + # PyCURL's multi.perform() can return a failure code (i.e. not E_OK) + # which triggers a FileNotFoundError. It doesn't really matter which + # failure code it returns. + class FakeMulti: + def perform(self): + return pycurl.E_READ_ERROR, 2 + class Testable(CurlDownloadManager): + def _do_once(self, multi, handles): + return super()._do_once(FakeMulti(), handles) + with self.assertRaises(FileNotFoundError) as cm: + Testable().get_files(_http_pathify([ + ('channel.channels_05.json', 'channels.json'), + ('download.index_01.json', 'index.json'), + ])) + # One of the two files will be contained in the error message, but + # which one is undefined, although in practice it will be the first + # one. + self.assertRegex( + cm.exception.args[0], + 'http://localhost:8980/(channel.channels_05|index_01).json') + + +class TestDownloadManagerFactory(unittest.TestCase): + """We have a factory for creating the download manager to use.""" + + def test_get_downloader_forced_curl(self): + # Setting SYSTEMIMAGE_PYCURL envar to 1, yes, or true forces the + # PyCURL downloader. + with reset_envar('SYSTEMIMAGE_PYCURL'): + os.environ['SYSTEMIMAGE_PYCURL'] = '1' + self.assertIsInstance(get_download_manager(), CurlDownloadManager) + with reset_envar('SYSTEMIMAGE_PYCURL'): + os.environ['SYSTEMIMAGE_PYCURL'] = 'tRuE' + self.assertIsInstance(get_download_manager(), CurlDownloadManager) + with reset_envar('SYSTEMIMAGE_PYCURL'): + os.environ['SYSTEMIMAGE_PYCURL'] = 'YES' + self.assertIsInstance(get_download_manager(), CurlDownloadManager) + + def test_get_downloader_forced_udm(self): + # Setting SYSTEMIMAGE_PYCURL envar to anything else forces the udm + # downloader. + with reset_envar('SYSTEMIMAGE_PYCURL'): + os.environ['SYSTEMIMAGE_PYCURL'] = '0' + self.assertIsInstance(get_download_manager(), UDMDownloadManager) + with reset_envar('SYSTEMIMAGE_PYCURL'): + os.environ['SYSTEMIMAGE_PYCURL'] = 'false' + self.assertIsInstance(get_download_manager(), UDMDownloadManager) + with reset_envar('SYSTEMIMAGE_PYCURL'): + os.environ['SYSTEMIMAGE_PYCURL'] = 'nope' + self.assertIsInstance(get_download_manager(), UDMDownloadManager) + + def test_auto_detect_udm(self): + # If the environment variable is not set, we do auto-detection. For + # backward compatibility, if udm is available on the system bus, we + # use it. + with reset_envar('SYSTEMIMAGE_PYCURL'): + if 'SYSTEMIMAGE_PYCURL' in os.environ: + del os.environ['SYSTEMIMAGE_PYCURL'] + with patch('dbus.SystemBus.get_object') as mock: + self.assertIsInstance( + get_download_manager(), UDMDownloadManager) + mock.assert_called_once_with(DOWNLOADER_INTERFACE, '/') + + def test_auto_detect_curl(self): + # If the environment variable is not set, we do auto-detection. If udm + # is not available on the system bus, we use the cURL downloader. + import systemimage.download + with ExitStack() as resources: + resources.enter_context(reset_envar('SYSTEMIMAGE_PYCURL')) + if 'SYSTEMIMAGE_PYCURL' in os.environ: + del os.environ['SYSTEMIMAGE_PYCURL'] + mock = resources.enter_context( + patch('dbus.SystemBus.get_object', side_effect=DBusException)) + resources.enter_context( + patch.object(systemimage.download, 'pycurl', object())) + self.assertIsInstance( + get_download_manager(), CurlDownloadManager) + mock.assert_called_once_with(DOWNLOADER_INTERFACE, '/') + + def test_auto_detect_none_available(self): + # Again, we're auto-detecting, but in this case, we have neither udm + # nor pycurl available. + import systemimage.download + with ExitStack() as resources: + resources.enter_context(reset_envar('SYSTEMIMAGE_PYCURL')) + if 'SYSTEMIMAGE_PYCURL' in os.environ: + del os.environ['SYSTEMIMAGE_PYCURL'] + mock = resources.enter_context( + patch('dbus.SystemBus.get_object', side_effect=DBusException)) + resources.enter_context( + patch.object(systemimage.download, 'pycurl', None)) + self.assertRaises(ImportError, get_download_manager) + mock.assert_called_once_with(DOWNLOADER_INTERFACE, '/') === modified file 'systemimage/tests/test_gpg.py' --- systemimage/tests/test_gpg.py 2014-02-20 23:03:24 +0000 +++ systemimage/tests/test_gpg.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -19,6 +19,7 @@ 'TestKeyrings', 'TestSignature', 'TestSignatureError', + 'TestSignatureWithOverrides', ] @@ -201,18 +202,16 @@ class TestSignature(unittest.TestCase): def setUp(self): self._stack = ExitStack() + self.addCleanup(self._stack.close) self._tmpdir = self._stack.enter_context(temporary_directory()) - def tearDown(self): - self._stack.close() - @configuration def test_good_signature(self): # We have a channels.json file signed with the imaging signing key, as # would be the case in production. The signature will match a context # loaded with the public key. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) sign(channels_json, 'image-signing.gpg') with temporary_directory() as tmpdir: keyring = os.path.join(tmpdir, 'image-signing.tar.xz') @@ -227,7 +226,7 @@ # In this case, the file is signed with the device key, so it will not # verify against the image signing key. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) sign(channels_json, 'device-signing.gpg') # Verify the signature with the pubkey. with temporary_directory() as tmpdir: @@ -243,7 +242,7 @@ # Like above, the file is signed with the device key, but this time we # include both the image signing and device signing pubkeys. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) sign(channels_json, 'device-signing.gpg') with temporary_directory() as tmpdir: keyring_1 = os.path.join(tmpdir, 'image-signing.tar.xz') @@ -261,7 +260,7 @@ # The file is signed with the image master key, but it won't verify # against the image signing and device signing pubkeys. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) sign(channels_json, 'image-master.gpg') # Verify the signature with the pubkey. with temporary_directory() as tmpdir: @@ -279,8 +278,8 @@ def test_bad_not_even_a_signature(self): # The signature file isn't even a signature file. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst=channels_json) - copy('channels_01.json', self._tmpdir, dst=channels_json + '.asc') + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json + '.asc') with temporary_directory() as tmpdir: dst = os.path.join(tmpdir, 'device-signing.tar.xz') setup_keyring_txz('device-signing.gpg', 'image-signing.gpg', @@ -297,7 +296,7 @@ # though, we also have a blacklist keyring, but none of the keyids in # the blacklist match the keyid that the file was signed with. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst='channels.json') + copy('gpg.channels_01.json', self._tmpdir, dst='channels.json') sign(channels_json, 'device-signing.gpg') # Verify the signature with the pubkey. with temporary_directory() as tmpdir: @@ -318,7 +317,7 @@ def test_bad_signature_in_blacklist(self): # Like above, but we put the device signing key id in the blacklist. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) sign(channels_json, 'device-signing.gpg') # Verify the signature with the pubkey. with temporary_directory() as tmpdir: @@ -340,7 +339,7 @@ def test_good_validation(self): # The .validate() method does nothing if the signature is good. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) sign(channels_json, 'image-signing.gpg') with temporary_directory() as tmpdir: keyring = os.path.join(tmpdir, 'image-signing.tar.xz') @@ -354,11 +353,9 @@ class TestSignatureError(unittest.TestCase): def setUp(self): self._stack = ExitStack() + self.addCleanup(self._stack.close) self._tmpdir = self._stack.enter_context(temporary_directory()) - def tearDown(self): - self._stack.close() - def test_extra_data(self): # A SignatureError includes extra information about the path to the # signature file, and the path to the data file. You also get the md5 @@ -382,7 +379,7 @@ # The .validate() method raises a SignatureError exception with extra # information when the signature is invalid. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) sign(channels_json, 'device-signing.gpg') # Verify the signature with the pubkey. with temporary_directory() as tmpdir: @@ -416,7 +413,7 @@ def test_signature_invalid_due_to_blacklist(self): # Like above, but we put the device signing key id in the blacklist. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) sign(channels_json, 'device-signing.gpg') # Verify the signature with the pubkey. with temporary_directory() as tmpdir: @@ -464,7 +461,7 @@ # The repr/str of the SignatureError should contain lots of useful # information that will make debugging easier. channels_json = os.path.join(self._tmpdir, 'channels.json') - copy('channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) sign(channels_json, 'device-signing.gpg') # Verify the signature with the pubkey. tmpdir = self._stack.enter_context(temporary_directory()) @@ -503,3 +500,116 @@ self.assertTrue(line.endswith("/image-signing.tar.xz']")) elif i == 8: self.assertEqual(line, ' blacklist: no blacklist ') + + +class TestSignatureWithOverrides(unittest.TestCase): + """system-image-cli supports a --skip-gpg-verification flag.""" + + def setUp(self): + self._stack = ExitStack() + self.addCleanup(self._stack.close) + self._tmpdir = self._stack.enter_context(temporary_directory()) + + @configuration + def test_bad_signature(self, config): + # In this case, the file is signed with the device key, so it will not + # verify against the image signing key, unless the + # --skip-gpg-verification flag is set. + channels_json = os.path.join(self._tmpdir, 'channels.json') + channels_asc = channels_json + '.asc' + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) + sign(channels_json, 'device-signing.gpg') + # Verify the signature with the pubkey. + with temporary_directory() as tmpdir: + dst = os.path.join(tmpdir, 'image-signing.tar.xz') + setup_keyring_txz('image-signing.gpg', 'image-master.gpg', + dict(type='image-signing'), dst) + with Context(dst) as ctx: + self.assertFalse(ctx.verify(channels_asc, channels_json)) + # But with the --skip-gpg-verification flag set, the verify + # call returns success. + config.skip_gpg_verification = True + self.assertTrue(ctx.verify(channels_asc, channels_json)) + + @configuration + def test_bad_signature_with_multiple_keyrings(self, config): + # The file is signed with the image master key, but it won't verify + # against the image signing and device signing pubkeys, unless the + # --skip-gpg-verification flag is set. + channels_json = os.path.join(self._tmpdir, 'channels.json') + channels_asc = channels_json + '.asc' + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) + sign(channels_json, 'image-master.gpg') + # Verify the signature with the pubkey. + with temporary_directory() as tmpdir: + keyring_1 = os.path.join(tmpdir, 'image-signing.tar.xz') + keyring_2 = os.path.join(tmpdir, 'device-signing.tar.xz') + setup_keyring_txz('image-signing.gpg', 'image-master.gpg', + dict(type='image-signing'), keyring_1) + setup_keyring_txz('device-signing.gpg', 'image-signing.gpg', + dict(type='device-signing'), keyring_2) + with Context(keyring_1, keyring_2) as ctx: + self.assertFalse(ctx.verify(channels_asc, channels_json)) + config.skip_gpg_verification = True + self.assertTrue(ctx.verify(channels_asc, channels_json)) + + @configuration + def test_bad_not_even_a_signature(self, config): + # The signature file isn't even a signature file. Verification will + # fail unless the --skip-gpg-verification flag is set. + channels_json = os.path.join(self._tmpdir, 'channels.json') + channels_asc = channels_json + '.asc' + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json + '.asc') + with temporary_directory() as tmpdir: + dst = os.path.join(tmpdir, 'device-signing.tar.xz') + setup_keyring_txz('device-signing.gpg', 'image-signing.gpg', + dict(type='device-signing'), + dst) + with Context(dst) as ctx: + self.assertFalse(ctx.verify(channels_asc, channels_json)) + config.skip_gpg_verification = True + self.assertTrue(ctx.verify(channels_asc, channels_json)) + + @configuration + def test_bad_signature_in_blacklist(self): + # Like above, but we put the device signing key id in the blacklist. + channels_json = os.path.join(self._tmpdir, 'channels.json') + channels_asc = channels_json + '.asc' + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) + sign(channels_json, 'device-signing.gpg') + # Verify the signature with the pubkey. + with temporary_directory() as tmpdir: + keyring_1 = os.path.join(tmpdir, 'image-signing.tar.xz') + keyring_2 = os.path.join(tmpdir, 'device-signing.tar.xz') + blacklist = os.path.join(tmpdir, 'blacklist.tar.xz') + setup_keyring_txz('image-signing.gpg', 'image-master.gpg', + dict(type='image-signing'), keyring_1) + setup_keyring_txz('device-signing.gpg', 'image-signing.gpg', + dict(type='device-signing'), keyring_2) + # We're letting the device signing pubkey stand in for a blacklist. + setup_keyring_txz('device-signing.gpg', 'image-master.gpg', + dict(type='blacklist'), blacklist) + with Context(keyring_1, keyring_2, blacklist=blacklist) as ctx: + self.assertFalse(ctx.verify(channels_asc, channels_json)) + config.skip_gpg_verification = True + self.assertTrue(ctx.verify(channels_asc, channels_json)) + + @configuration + def test_bad_signature_with_validate(self, config): + # This is similar to the above, except that the .validate() API is + # used instead. + channels_json = os.path.join(self._tmpdir, 'channels.json') + channels_asc = channels_json + '.asc' + copy('gpg.channels_01.json', self._tmpdir, dst=channels_json) + sign(channels_json, 'device-signing.gpg') + # Verify the signature with the pubkey. + with temporary_directory() as tmpdir: + dst = os.path.join(tmpdir, 'image-signing.tar.xz') + setup_keyring_txz('image-signing.gpg', 'image-master.gpg', + dict(type='image-signing'), dst) + with Context(dst) as ctx: + self.assertRaises(SignatureError, ctx.validate, + channels_asc, channels_json) + config.skip_gpg_verification = True + ctx.validate(channels_asc, channels_json) === modified file 'systemimage/tests/test_helpers.py' --- systemimage/tests/test_helpers.py 2014-09-17 13:41:31 +0000 +++ systemimage/tests/test_helpers.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -34,11 +34,13 @@ from contextlib import ExitStack from datetime import datetime, timedelta +from pathlib import Path from systemimage.bag import Bag -from systemimage.config import Configuration, config +from systemimage.config import Configuration from systemimage.helpers import ( - MiB, as_loglevel, as_object, as_timedelta, calculate_signature, - last_update_date, phased_percentage, temporary_directory, version_detail) + MiB, NO_PORT, as_loglevel, as_object, as_port, as_stripped, as_timedelta, + calculate_signature, last_update_date, phased_percentage, + temporary_directory, version_detail) from systemimage.testing.helpers import configuration, data_path, touch_build from unittest.mock import patch @@ -109,6 +111,21 @@ def test_as_bad_dbus_loglevel(self): self.assertRaises(ValueError, as_loglevel, 'error:basicConfig') + def test_as_port(self): + self.assertEqual(as_port('801'), 801) + + def test_as_non_int_port(self): + self.assertRaises(ValueError, as_port, 'not-a-port') + + def test_as_port_disabled(self): + self.assertIs(as_port('disabled'), NO_PORT) + self.assertIs(as_port('disable'), NO_PORT) + self.assertIs(as_port('DISABLED'), NO_PORT) + self.assertIs(as_port('DISABLE'), NO_PORT) + + def test_stripped(self): + self.assertEqual(as_stripped(' field '), 'field') + class TestLastUpdateDate(unittest.TestCase): @configuration @@ -116,93 +133,95 @@ # The last upgrade data can come from /userdata/.last_update. with ExitStack() as stack: tmpdir = stack.enter_context(temporary_directory()) - userdata_path = os.path.join(tmpdir, '.last_update') + userdata_path = Path(tmpdir) / '.last_update' + stack.enter_context(patch('systemimage.helpers.LAST_UPDATE_FILE', + str(userdata_path))) + timestamp = int(datetime(2012, 11, 10, 9, 8, 7).timestamp()) + userdata_path.touch() + os.utime(str(userdata_path), (timestamp, timestamp)) + self.assertEqual(last_update_date(), '2012-11-10 09:08:07') + + @configuration + def test_date_from_config_d(self, config): + # The latest mtime from all the config.d files is taken as the last + # update date. Add a bunch of ini files where the higher numbered + # ones have higher numbered year mtimes. + for year in range(18, 22): + ini_file = Path(config.config_d) / '{:02d}_config.ini'.format(year) + ini_file.touch() + timestamp = int(datetime(2000 + year, 1, 2, 3, 4, 5).timestamp()) + os.utime(str(ini_file), (timestamp, timestamp)) + config.reload() + self.assertEqual(last_update_date(), '2021-01-02 03:04:05') + + @configuration + def test_date_from_config_d_reversed(self, config): + # As above, but the higher numbered ini files have earlier mtimes. + for year in range(22, 18, -1): + ini_file = Path(config.config_d) / '{:02d}_config.ini'.format(year) + ini_file.touch() + timestamp = int(datetime(2040-year, 1, 2, 3, 4, 5).timestamp()) + os.utime(str(ini_file), (timestamp, timestamp)) + config.reload() + self.assertEqual(last_update_date(), '2021-01-02 03:04:05') + + @configuration + def test_date_from_userdata_takes_precedence(self, config_d): + # The last upgrade data will come from /userdata/.last_update, even if + # there are .ini files with later mtimes in them. + for year in range(18, 22): + ini_file = Path(config_d) / '{:02d}_config.ini'.format(year) + ini_file.touch() + timestamp = int(datetime(2000 + year, 1, 2, 3, 4, 5).timestamp()) + os.utime(str(ini_file), (timestamp, timestamp)) + with ExitStack() as stack: + tmpdir = stack.enter_context(temporary_directory()) + userdata_path = Path(tmpdir) / '.last_update' + stack.enter_context(patch('systemimage.helpers.LAST_UPDATE_FILE', + str(userdata_path))) + timestamp = int(datetime(2012, 11, 10, 9, 8, 7).timestamp()) + userdata_path.touch() + os.utime(str(userdata_path), (timestamp, timestamp)) + self.assertEqual(last_update_date(), '2012-11-10 09:08:07') + + def test_date_unknown(self): + # If there is no /userdata/.last_update file and no ini files, then + # the last update date is unknown. + with ExitStack() as stack: + config_d = stack.enter_context(temporary_directory()) + tempdir = stack.enter_context(temporary_directory()) + userdata_path = os.path.join(tempdir, '.last_update') stack.enter_context(patch('systemimage.helpers.LAST_UPDATE_FILE', userdata_path)) - timestamp = int(datetime(2012, 11, 10, 9, 8, 7).timestamp()) - with open(userdata_path, 'w'): - # i.e. touch(1) - pass - os.utime(userdata_path, (timestamp, timestamp)) - self.assertEqual(last_update_date(), '2012-11-10 09:08:07') - - @configuration - def test_date_from_channel_ini(self, ini_file): - # The last update date can come from the mtime of the channel.ini - # file, which lives next to the configuration file. - channel_ini = os.path.join( - os.path.dirname(ini_file), 'channel.ini') - with open(channel_ini, 'w'): - pass - timestamp = int(datetime(2022, 1, 2, 3, 4, 5).timestamp()) - os.utime(channel_ini, (timestamp, timestamp)) - self.assertEqual(last_update_date(), '2022-01-02 03:04:05') - - @configuration - def test_date_from_channel_ini_instead_of_ubuntu_build(self, ini_file): - # The last update date can come from the mtime of the channel.ini - # file, which lives next to the configuration file, even when there is - # an /etc/ubuntu-build file. - channel_ini = os.path.join( - os.path.dirname(ini_file), 'channel.ini') - with open(channel_ini, 'w', encoding='utf-8'): - pass - # This creates the ubuntu-build file, but not the channel.ini file. - timestamp_1 = int(datetime(2022, 1, 2, 3, 4, 5).timestamp()) - touch_build(2, timestamp_1) - timestamp_2 = int(datetime(2022, 3, 4, 5, 6, 7).timestamp()) - os.utime(channel_ini, (timestamp_2, timestamp_2)) - self.assertEqual(last_update_date(), '2022-03-04 05:06:07') - - @configuration - def test_date_fallback(self, ini_file): - # If the channel.ini file doesn't exist, use the ubuntu-build file. - channel_ini = os.path.join( - os.path.dirname(ini_file), 'channel.ini') - with open(channel_ini, 'w', encoding='utf-8'): - pass - # This creates the ubuntu-build file, but not the channel.ini file. - timestamp_1 = int(datetime(2022, 1, 2, 3, 4, 5).timestamp()) - touch_build(2, timestamp_1) - timestamp_2 = int(datetime(2022, 3, 4, 5, 6, 7).timestamp()) - os.utime(channel_ini, (timestamp_2, timestamp_2)) - # Like the above test, but with this file removed. - os.remove(channel_ini) - self.assertEqual(last_update_date(), '2022-01-02 03:04:05') - - @configuration - def test_date_unknown(self, ini_file): - # No fallbacks. - config = Configuration(ini_file) - channel_ini = os.path.join(os.path.dirname(ini_file), 'channel.ini') - self.assertFalse(os.path.exists(channel_ini)) - self.assertFalse(os.path.exists(config.system.build_file)) - self.assertEqual(last_update_date(), 'Unknown') - - @configuration - def test_date_no_microseconds(self, ini_file): + config = Configuration(config_d) + stack.enter_context(patch('systemimage.config._config', config)) + self.assertEqual(last_update_date(), 'Unknown') + + @configuration + def test_date_no_microseconds(self, config): # Resolution is seconds. - channel_ini = os.path.join( - os.path.dirname(ini_file), 'channel.ini') - with open(channel_ini, 'w', encoding='utf-8'): - pass - timestamp = datetime(2013, 12, 11, 10, 9, 8, 7).timestamp() + ini_file = Path(config.config_d) / '01_config.ini' + ini_file.touch() + timestamp = datetime(2022, 12, 11, 10, 9, 8, 7).timestamp() # We need nanoseconds. timestamp *= 1000000000 - os.utime(channel_ini, ns=(timestamp, timestamp)) - self.assertEqual(last_update_date(), '2013-12-11 10:09:08') + os.utime(str(ini_file), ns=(timestamp, timestamp)) + config.reload() + self.assertEqual(last_update_date(), '2022-12-11 10:09:08') @configuration - def test_version_detail(self, ini_file): - channel_ini = data_path('channel_03.ini') - config.load(channel_ini, override=True) + def test_version_detail(self, config): + shutil.copy(data_path('helpers.config_01.ini'), + os.path.join(config.config_d, '00_config.ini')) + config.reload() self.assertEqual(version_detail(), dict(ubuntu='123', mako='456', custom='789')) @configuration - def test_no_version_detail(self, ini_file): - channel_ini = data_path('channel_01.ini') - config.load(channel_ini, override=True) + def test_no_version_detail(self, config): + shutil.copy(data_path('helpers.config_02.ini'), + os.path.join(config.config_d, '00_config.ini')) + config.reload() self.assertEqual(version_detail(), {}) def test_version_detail_from_argument(self): @@ -212,115 +231,108 @@ def test_no_version_in_version_detail(self): self.assertEqual(version_detail('ubuntu,mako,custom'), {}) - @configuration - def test_date_from_userdata_ignoring_fallbacks(self, ini_file): - # Even when /etc/system-image/channel.ini and /etc/ubuntu-build exist, - # if there's a /userdata/.last_update file, that takes precedence. - with ExitStack() as stack: - # /userdata/.last_update - tmpdir = stack.enter_context(temporary_directory()) - userdata_path = os.path.join(tmpdir, '.last_update') - stack.enter_context(patch('systemimage.helpers.LAST_UPDATE_FILE', - userdata_path)) - with open(userdata_path, 'w'): - # i.e. touch(1) - pass - timestamp = int(datetime(2010, 9, 8, 7, 6, 5).timestamp()) - os.utime(userdata_path, (timestamp, timestamp)) - # /etc/channel.ini - channel_ini = os.path.join( - os.path.dirname(ini_file), 'channel.ini') - with open(channel_ini, 'w'): - pass - timestamp = int(datetime(2011, 10, 9, 8, 7, 6).timestamp()) - os.utime(channel_ini, (timestamp, timestamp)) - # /etc/ubuntu-build. - timestamp = int(datetime(2012, 11, 10, 9, 8, 7).timestamp()) - touch_build(2, timestamp) - # Run the test. - self.assertEqual(last_update_date(), '2010-09-08 07:06:05') - - @configuration - def test_last_date_no_permission(self, ini_file): + @unittest.skipIf(os.getuid() == 0, 'Test cannot succeed when run as root') + @configuration + def test_last_date_no_permission(self, config): # LP: #1365761 reports a problem where stat'ing /userdata/.last_update - # results in a PermissionError. In that case it should just use a - # fall back, in this case the channel.ini file. - channel_ini = os.path.join( - os.path.dirname(ini_file), 'channel.ini') - with open(channel_ini, 'w', encoding='utf-8'): - pass - # This creates the ubuntu-build file, but not the channel.ini file. + # results in a PermissionError. In that case it should fall back to + # using the mtimes of the config.d ini files. timestamp_1 = int(datetime(2022, 1, 2, 3, 4, 5).timestamp()) touch_build(2, timestamp_1) - # Now, the channel.ini file. - timestamp_2 = int(datetime(2022, 3, 4, 5, 6, 7).timestamp()) - os.utime(channel_ini, (timestamp_2, timestamp_2)) - # Now create an stat'able /userdata/.last_update file. + # Now create an unstat'able /userdata/.last_update file. with ExitStack() as stack: tmpdir = stack.enter_context(temporary_directory()) - userdata_path = os.path.join(tmpdir, '.last_update') + userdata_path = Path(tmpdir) / '.last_update' stack.enter_context(patch('systemimage.helpers.LAST_UPDATE_FILE', - userdata_path)) + str(userdata_path))) timestamp = int(datetime(2012, 11, 10, 9, 8, 7).timestamp()) - with open(userdata_path, 'w'): - # i.e. touch(1) - pass - os.utime(userdata_path, (timestamp, timestamp)) # Make the file unreadable. + userdata_path.touch() + os.utime(str(userdata_path), (timestamp, timestamp)) stack.callback(os.chmod, tmpdir, 0o777) os.chmod(tmpdir, 0o000) - # The last update date will be the date of the channel.ini file. - self.assertEqual(last_update_date(), '2022-03-04 05:06:07') + config.reload() + # The last update date will be the date of the 99_build.ini file. + self.assertEqual(last_update_date(), '2022-01-02 03:04:05') class TestPhasedPercentage(unittest.TestCase): def setUp(self): - phased_percentage(reset=True) + self._resources = ExitStack() + tmpdir = self._resources.enter_context(temporary_directory()) + self._mid_path = os.path.join(tmpdir, 'machine-id') + self._resources.enter_context(patch( + 'systemimage.helpers.UNIQUE_MACHINE_ID_FILES', [self._mid_path])) def tearDown(self): - phased_percentage(reset=True) + self._resources.close() + + def _set_machine_id(self, machine_id): + with open(self._mid_path, 'w', encoding='utf-8') as fp: + fp.write(machine_id) def test_phased_percentage(self): - # This function returns a percentage between 0 and 100. If this value - # is greater than a similar value in the index.json's 'image' section, - # that image is completely ignored. - with ExitStack() as stack: - tmpdir = stack.enter_context(temporary_directory()) - path = os.path.join(tmpdir, 'machine-id') - stack.enter_context(patch( - 'systemimage.helpers.UNIQUE_MACHINE_ID_FILE', - path)) - stack.enter_context(patch( - 'systemimage.helpers.time.time', - return_value=1380659512.983512)) - with open(path, 'wb') as fp: - fp.write(b'0123456789abcdef\n') - self.assertEqual(phased_percentage(), 81) - # The value is cached, so it's always the same for the life of the - # process, at least until we reset it. - self.assertEqual(phased_percentage(), 81) - - def test_phased_percentage_reset(self): - # Test the reset API. - with ExitStack() as stack: - tmpdir = stack.enter_context(temporary_directory()) - path = os.path.join(tmpdir, 'machine-id') - stack.enter_context(patch( - 'systemimage.helpers.UNIQUE_MACHINE_ID_FILE', - path)) - stack.enter_context(patch( - 'systemimage.helpers.time.time', - return_value=1380659512.983512)) - with open(path, 'wb') as fp: - fp.write(b'0123456789abcdef\n') - self.assertEqual(phased_percentage(), 81) - # The value is cached, so it's always the same for the life of the - # process, at least until we reset it. - with open(path, 'wb') as fp: - fp.write(b'x0123456789abcde\n') - self.assertEqual(phased_percentage(reset=True), 81) - # The next one will have a different value. - self.assertEqual(phased_percentage(), 17) + # The phased percentage is used to determine whether a calculated + # winning path is to be applied or not. It returns a number between 0 + # and 100 based on the machine's unique machine id (as kept in a + # file), the update channel, and the target build number. + self._set_machine_id('0123456789abcdef') + self.assertEqual(phased_percentage(channel='ubuntu', target=11), 51) + # The phased percentage is always the same, given the same + # machine-id, channel, and target. + self.assertEqual(phased_percentage(channel='ubuntu', target=11), 51) + + def test_phased_percentage_different_machine_id(self): + # All else being equal, a different machine_id gives different %. + self._set_machine_id('0123456789abcdef') + self.assertEqual(phased_percentage(channel='ubuntu', target=11), 51) + self._set_machine_id('fedcba9876543210') + self.assertEqual(phased_percentage(channel='ubuntu', target=11), 25) + + def test_phased_percentage_different_channel(self): + # All else being equal, a different channel gives different %. + self._set_machine_id('0123456789abcdef') + self.assertEqual(phased_percentage(channel='ubuntu', target=11), 51) + self._set_machine_id('0123456789abcdef') + self.assertEqual(phased_percentage(channel='devel', target=11), 96) + + def test_phased_percentage_different_target(self): + # All else being equal, a different target gives different %. + self._set_machine_id('0123456789abcdef') + self.assertEqual(phased_percentage(channel='ubuntu', target=11), 51) + self._set_machine_id('0123456789abcdef') + self.assertEqual(phased_percentage(channel='ubuntu', target=12), 1) + + @configuration + def test_phased_percentage_override(self, config): + # The phased percentage can be overridden. + self._set_machine_id('0123456789abcdef') + self.assertEqual(phased_percentage(channel='ubuntu', target=11), 51) + config.phase_override = 33 + self.assertEqual(phased_percentage(channel='ubuntu', target=11), 33) + # And reset. + del config.phase_override + self.assertEqual(phased_percentage(channel='ubuntu', target=11), 51) + + def test_phased_percentage_machine_id_file_fallback(self): + # Ensure that the fallbacks for the machine-id file. + with ExitStack() as resources: + resources.enter_context(patch( + 'systemimage.helpers.UNIQUE_MACHINE_ID_FILES', + ['/does/not/exist', self._mid_path])) + self._set_machine_id('0123456789abcdef') + self.assertEqual( + phased_percentage(channel='ubuntu', target=11), 51) + + def test_phased_percentage_machine_id_file_fallbacks_exhausted(self): + # Not much we can do if there are no machine-id files. + with ExitStack() as resources: + resources.enter_context(patch( + 'systemimage.helpers.UNIQUE_MACHINE_ID_FILES', + ['/does/not/exist', '/is/not/present'])) + self._set_machine_id('0123456789abcdef') + self.assertRaises(RuntimeError, phased_percentage, + channel='ubuntu', target=11) class TestSignature(unittest.TestCase): === modified file 'systemimage/tests/test_image.py' --- systemimage/tests/test_image.py 2014-02-20 23:03:24 +0000 +++ systemimage/tests/test_image.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify === modified file 'systemimage/tests/test_index.py' --- systemimage/tests/test_index.py 2014-02-20 23:03:24 +0000 +++ systemimage/tests/test_index.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -33,9 +33,6 @@ configuration, copy, get_index, make_http_server, makedirs, setup_keyring_txz, setup_keyrings, sign) from systemimage.testing.nose import SystemImagePlugin -# FIXME -from systemimage.tests.test_candidates import _descriptions -from unittest.mock import patch class TestIndex(unittest.TestCase): @@ -44,19 +41,19 @@ SystemImagePlugin.controller.set_mode(cert_pem='cert.pem') def test_index_global(self): - index = get_index('index_01.json') + index = get_index('index.index_02.json') self.assertEqual( index.global_.generated_at, datetime(2013, 4, 29, 18, 45, 27, tzinfo=timezone.utc)) def test_index_image_count(self): - index = get_index('index_01.json') + index = get_index('index.index_02.json') self.assertEqual(len(index.images), 0) - index = get_index('index_02.json') + index = get_index('index.index_03.json') self.assertEqual(len(index.images), 2) def test_image_20130300_full(self): - index = get_index('sprint_nexus7_index_01.json') + index = get_index('index.index_05.json') image = index.images[0] self.assertEqual( image.descriptions, @@ -83,7 +80,7 @@ def test_image_20130500_minversion(self): # Some full images have a minimum version older than which they refuse # to upgrade from. - index = get_index('sprint_nexus7_index_01.json') + index = get_index('index.index_05.json') image = index.images[5] self.assertEqual(image.type, 'full') self.assertEqual(image.version, 20130500) @@ -92,7 +89,7 @@ def test_image_descriptions(self): # Image descriptions can come in a variety of locales. - index = get_index('index_14.json') + index = get_index('index.index_01.json') self.assertEqual(index.images[0].descriptions, { 'description': 'Full A'}) self.assertEqual(index.images[3].descriptions, { @@ -113,45 +110,6 @@ 'description-xx_CC': 'This hyar is the delta B.2', }) - def test_image_phased_percentage(self): - # This index has two full updates with a phased-percentage value and - # one without (which defaults to 100). We'll set the system's - # percentage right in the middle of the two so that the one with 50% - # will not show up in the list of images. - with patch('systemimage.index.phased_percentage', return_value=66): - index = get_index('index_22.json') - descriptions = set(_descriptions(index.images)) - # This one does not have a phased-percentage, so using the default of - # 100, it gets in. - self.assertIn('Full A', descriptions) - # This one has a phased-percentage of 50 so it gets ignored. - self.assertNotIn('Full B', descriptions) - # This one has a phased-percentage of 75 so it gets added. - self.assertIn('Full C', descriptions) - - def test_image_phased_percentage_100(self): - # Like above, but with a system percentage of 100, so nothing but the - # default gets in. - with patch('systemimage.index.phased_percentage', return_value=100): - index = get_index('index_22.json') - descriptions = set(_descriptions(index.images)) - # This one does not have a phased-percentage, so using the default of - # 100, it gets in. - self.assertIn('Full A', descriptions) - # This one has a phased-percentage of 50 so it gets ignored. - self.assertNotIn('Full B', descriptions) - # This one has a phased-percentage of 75 so it gets added. - self.assertNotIn('Full C', descriptions) - - def test_image_phased_percentage_0(self): - # Like above, but with a system percentage of 0, everything gets in. - with patch('systemimage.index.phased_percentage', return_value=0): - index = get_index('index_22.json') - descriptions = set(_descriptions(index.images)) - self.assertIn('Full A', descriptions) - self.assertIn('Full B', descriptions) - self.assertIn('Full C', descriptions) - class TestDownloadIndex(unittest.TestCase): maxDiff = None @@ -186,10 +144,11 @@ # Load the index.json pointed to by the channels.json. All signatures # validate correctly and there is no device keyring or blacklist. self._copysign( - 'channels_02.json', 'channels.json', 'image-signing.gpg') - # index_10.json path B will win, with no bootme flags. + 'index.channels_05.json', 'channels.json', 'image-signing.gpg') + # index.index_04.json path B will win, with no bootme flags. self._copysign( - 'index_10.json', 'stable/nexus7/index.json', 'image-signing.gpg') + 'index.index_04.json', 'stable/nexus7/index.json', + 'image-signing.gpg') setup_keyrings() state = State() state.run_thru('get_index') @@ -203,10 +162,11 @@ def test_load_index_with_device_keyring(self): # Here, the index.json file is signed with a device keyring. self._copysign( - 'channels_03.json', 'channels.json', 'image-signing.gpg') - # index_10.json path B will win, with no bootme flags. + 'index.channels_02.json', 'channels.json', 'image-signing.gpg') + # index.index_04.json.json path B will win, with no bootme flags. self._copysign( - 'index_10.json', 'stable/nexus7/index.json', 'device-signing.gpg') + 'index.index_04.json', 'stable/nexus7/index.json', + 'device-signing.gpg') setup_keyrings() setup_keyring_txz( 'device-signing.gpg', 'image-signing.gpg', @@ -225,10 +185,11 @@ # Here, the index.json file is signed with the image signing keyring, # even though there is a device key. That's fine. self._copysign( - 'channels_03.json', 'channels.json', 'image-signing.gpg') - # index_10.json path B will win, with no bootme flags. + 'index.channels_02.json', 'channels.json', 'image-signing.gpg') + # index.index_04.json.json path B will win, with no bootme flags. self._copysign( - 'index_10.json', 'stable/nexus7/index.json', 'image-signing.gpg') + 'index.index_04.json', 'stable/nexus7/index.json', + 'image-signing.gpg') setup_keyrings() setup_keyring_txz( 'device-signing.gpg', 'image-signing.gpg', @@ -246,10 +207,10 @@ def test_load_index_with_bad_keyring(self): # Here, the index.json file is signed with a defective device keyring. self._copysign( - 'channels_03.json', 'channels.json', 'image-signing.gpg') + 'index.channels_02.json', 'channels.json', 'image-signing.gpg') # This will be signed by a keyring that is not the device keyring. self._copysign( - 'index_10.json', 'stable/nexus7/index.json', 'spare.gpg') + 'index.index_04.json', 'stable/nexus7/index.json', 'spare.gpg') setup_keyrings() setup_keyring_txz( 'device-signing.gpg', 'image-signing.gpg', @@ -263,10 +224,11 @@ def test_load_index_with_blacklist(self): # Here, we've blacklisted the device key. self._copysign( - 'channels_03.json', 'channels.json', 'image-signing.gpg') + 'index.channels_02.json', 'channels.json', 'image-signing.gpg') # This will be signed by a keyring that is not the device keyring. self._copysign( - 'index_10.json', 'stable/nexus7/index.json', 'device-signing.gpg') + 'index.index_04.json', 'stable/nexus7/index.json', + 'device-signing.gpg') setup_keyrings() setup_keyring_txz( 'device-signing.gpg', 'image-signing.gpg', @@ -283,10 +245,11 @@ def test_missing_channel(self): # The system's channel does not exist. self._copysign( - 'channels_04.json', 'channels.json', 'image-signing.gpg') - # index_10.json path B will win, with no bootme flags. + 'index.channels_03.json', 'channels.json', 'image-signing.gpg') + # index.index_04.json path B will win, with no bootme flags. self._copysign( - 'index_10.json', 'stable/nexus7/index.json', 'image-signing.gpg') + 'index.index_04.json', 'stable/nexus7/index.json', + 'image-signing.gpg') setup_keyrings() # Our channel (stable) isn't in the channels.json file, so there's # nothing to do. Running the state machine to its conclusion leaves @@ -300,10 +263,11 @@ def test_missing_device(self): # The system's device does not exist. self._copysign( - 'channels_05.json', 'channels.json', 'image-signing.gpg') - # index_10.json path B will win, with no bootme flags. + 'index.channels_04.json', 'channels.json', 'image-signing.gpg') + # index.index_04.json path B will win, with no bootme flags. self._copysign( - 'index_10.json', 'stable/nexus7/index.json', 'image-signing.gpg') + 'index.index_04.json', 'stable/nexus7/index.json', + 'image-signing.gpg') setup_keyrings() # Our device (nexus7) isn't in the channels.json file, so there's # nothing to do. Running the state machine to its conclusion leaves === modified file 'systemimage/tests/test_keyring.py' --- systemimage/tests/test_keyring.py 2014-02-20 23:03:24 +0000 +++ systemimage/tests/test_keyring.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify === modified file 'systemimage/tests/test_main.py' --- systemimage/tests/test_main.py 2014-09-17 13:41:31 +0000 +++ systemimage/tests/test_main.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -17,21 +17,26 @@ __all__ = [ 'TestCLIDuplicateDestinations', + 'TestCLIFactoryReset', 'TestCLIFilters', 'TestCLIListChannels', 'TestCLIMain', 'TestCLIMainDryRun', 'TestCLIMainDryRunAliases', 'TestCLINoReboot', + 'TestCLIProductionReset', + 'TestCLIProgress', 'TestCLISettings', - 'TestCLIFactoryReset', + 'TestCLISignatures', 'TestDBusMain', + 'TestDBusMainNoConfigD', ] import os import sys import dbus +import json import stat import time import shutil @@ -43,14 +48,14 @@ from functools import partial from io import StringIO from pathlib import Path -from pkg_resources import resource_filename, resource_string as resource_bytes from systemimage.config import Configuration, config from systemimage.helpers import safe_remove from systemimage.main import main as cli_main from systemimage.settings import Settings from systemimage.testing.helpers import ( ServerTestBase, chmod, configuration, copy, data_path, find_dbus_process, - temporary_directory, touch_build) + sign, temporary_directory, terminate_service, touch_build, + wait_for_service) from systemimage.testing.nose import SystemImagePlugin from textwrap import dedent from unittest.mock import MagicMock, patch @@ -71,6 +76,33 @@ os.umask(old_mask) +def machine_id(mid): + with ExitStack() as resources: + tempdir = resources.enter_context(temporary_directory()) + path = os.path.join(tempdir, 'machine-id') + with open(path, 'w', encoding='utf-8') as fp: + print(mid, file=fp) + resources.enter_context( + patch('systemimage.helpers.UNIQUE_MACHINE_ID_FILES', [path])) + return resources.pop_all() + + +def capture_print(fp): + return patch('builtins.print', partial(print, file=fp)) + + +def argv(*args): + args = list(args) + args.insert(0, 'argv0') + with ExitStack() as resources: + resources.enter_context(patch('systemimage.main.sys.argv', args)) + # We need a fresh global Configuration object to mimic what the + # command line script would see. + resources.enter_context( + patch('systemimage.config._config', Configuration())) + return resources.pop_all() + + class TestCLIMain(unittest.TestCase): def setUp(self): super().setUp() @@ -81,11 +113,12 @@ # We patch builtin print() rather than sys.stdout because the # latter can mess with pdb output should we need to trace through # the code. - self._resources.enter_context( - patch('builtins.print', partial(print, file=self._stdout))) + self._resources.enter_context(capture_print(self._stdout)) # Patch argparse's stderr to capture its error messages. self._resources.enter_context( patch('argparse._sys.stderr', self._stderr)) + self._resources.push( + machine_id('feedfacebeefbacafeedfacebeefbaca')) except: self._resources.close() raise @@ -94,50 +127,43 @@ self._resources.close() super().tearDown() - def test_config_file_good_path(self): - # The default configuration file exists. - self._resources.enter_context( - patch('systemimage.main.sys.argv', ['argv0', '--info'])) - # Patch default configuration file. + def test_config_directory_good_path(self): + # The default configuration directory exists. + self._resources.enter_context(argv('--info')) + # Patch default configuration directory. tempdir = self._resources.enter_context(temporary_directory()) - ini_path = os.path.join(tempdir, 'client.ini') - shutil.copy( - resource_filename('systemimage.data', 'client.ini'), tempdir) + copy('main.config_01.ini', tempdir, '00_config.ini') self._resources.enter_context( - patch('systemimage.main.DEFAULT_CONFIG_FILE', ini_path)) + patch('systemimage.main.DEFAULT_CONFIG_D', tempdir)) # Mock out the initialize() call so that the main() doesn't try to # create a log file in a non-existent system directory. self._resources.enter_context(patch('systemimage.main.initialize')) cli_main() - self.assertEqual(config.config_file, ini_path) - self.assertEqual(config.system.build_file, '/etc/ubuntu-build') - - def test_missing_default_config_file(self): - # The default configuration file is missing. - self._resources.enter_context( - patch('systemimage.main.sys.argv', ['argv0'])) - # Patch default configuration file. - self._resources.enter_context( - patch('systemimage.main.DEFAULT_CONFIG_FILE', - '/does/not/exist/client.ini')) - with self.assertRaises(SystemExit) as cm: - cli_main() - self.assertEqual(cm.exception.code, 2) - self.assertEqual( - self._stderr.getvalue().splitlines()[-1], - 'Configuration file not found: /does/not/exist/client.ini') - - def test_missing_explicit_config_file(self): - # An explicit configuration file given with -C is missing. - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', '/does/not/exist.ini'])) - with self.assertRaises(SystemExit) as cm: - cli_main() - self.assertEqual(cm.exception.code, 2) - self.assertEqual( - self._stderr.getvalue().splitlines()[-1], - 'Configuration file not found: /does/not/exist.ini') + self.assertEqual(config.config_d, tempdir) + self.assertEqual(config.channel, 'special') + + def test_missing_default_config_directory(self): + # The default configuration directory is missing. + self._resources.enter_context(argv()) + # Patch default configuration directory. + self._resources.enter_context( + patch('systemimage.main.DEFAULT_CONFIG_D', '/does/not/exist')) + with self.assertRaises(SystemExit) as cm: + cli_main() + self.assertEqual(cm.exception.code, 2) + self.assertEqual( + self._stderr.getvalue().splitlines()[-1], + 'Configuration directory not found: /does/not/exist') + + def test_missing_explicit_config_directory(self): + # An explicit configuration directory given with -C is missing. + self._resources.enter_context(argv('-C', '/does/not/exist')) + with self.assertRaises(SystemExit) as cm: + cli_main() + self.assertEqual(cm.exception.code, 2) + self.assertEqual( + self._stderr.getvalue().splitlines()[-1], + 'Configuration directory not found: /does/not/exist') def test_ensure_directories_exist(self): # The temporary and var directories are created if they don't exist. @@ -145,8 +171,8 @@ dir_2 = self._resources.enter_context(temporary_directory()) # Create a configuration file with directories that point to # non-existent locations. - config_ini = os.path.join(dir_1, 'client.ini') - with open(data_path('config_00.ini'), encoding='utf-8') as fp: + config_ini = os.path.join(dir_1, '00_config.ini') + with open(data_path('00.ini'), encoding='utf-8') as fp: template = fp.read() # These paths look something like they would on the real system. tmpdir = os.path.join(dir_2, 'tmp', 'system-image') @@ -155,9 +181,7 @@ with open(config_ini, 'wt', encoding='utf-8') as fp: fp.write(configuration) # Invoking main() creates the directories. - self._resources.enter_context(patch( - 'systemimage.main.sys.argv', - ['argv0', '-C', config_ini, '--info'])) + self._resources.enter_context(argv('-C', dir_1, '--info')) self.assertFalse(os.path.exists(tmpdir)) cli_main() self.assertTrue(os.path.exists(tmpdir)) @@ -169,8 +193,8 @@ dir_2 = self._resources.enter_context(temporary_directory()) # Create a configuration file with directories that point to # non-existent locations. - config_ini = os.path.join(dir_1, 'client.ini') - with open(data_path('config_04.ini'), encoding='utf-8') as fp: + config_ini = os.path.join(dir_1, '00_config.ini') + with open(data_path('00.ini'), encoding='utf-8') as fp: template = fp.read() # These paths look something like they would on the real system. tmpdir = os.path.join(dir_2, 'tmp', 'system-image') @@ -179,12 +203,10 @@ with open(config_ini, 'w', encoding='utf-8') as fp: fp.write(configuration) # Invoking main() creates the directories. - config = Configuration(config_ini) + config = Configuration(dir_1) self.assertFalse(os.path.exists(config.system.tempdir)) self.assertFalse(os.path.exists(config.system.logfile)) - self._resources.enter_context(patch( - 'systemimage.main.sys.argv', - ['argv0', '-C', config_ini, '--info'])) + self._resources.enter_context(argv('-C', dir_1, '--info')) cli_main() mode = os.stat(config.system.tempdir).st_mode self.assertEqual(stat.filemode(mode), 'drwx--S---') @@ -194,14 +216,11 @@ self.assertEqual(stat.filemode(mode), '-rw-------') @configuration - def test_info(self, ini_file): + def test_info(self, config_d): # -i/--info gives information about the device, including the current # build number, channel, and device name. - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--info'])) - # Set up the build number. touch_build(1701, TIMESTAMP) + self._resources.enter_context(argv('-C', config_d, '--info')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ current build number: 1701 @@ -211,62 +230,40 @@ """)) @configuration - def test_info_last_update_channel_ini(self, ini_file): - # --info's last update date uses the mtime of channel.ini even when - # /etc/ubuntu-build exists. - channel_ini = os.path.join(os.path.dirname(ini_file), 'channel.ini') - head, tail = os.path.split(channel_ini) - copy('channel_01.ini', head, tail) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--info'])) - # Set up the build number. - config = Configuration(ini_file) - touch_build(1701) + def test_info_last_update_timestamps(self, config_d): + # --info's last update date uses the latest mtime of the files in the + # config.d directory. + copy('main.config_02.ini', config_d, '00_config.ini') + copy('main.config_02.ini', config_d, '01_config.ini') + copy('main.config_02.ini', config_d, '02_config.ini') + # Give the default ini file an even earlier timestamp. + timestamp_0 = int(datetime(2010, 11, 8, 2, 3, 4).timestamp()) + touch_build(1701, timestamp_0) + # Make the 01 ini file the latest. timestamp_1 = int(datetime(2011, 1, 8, 2, 3, 4).timestamp()) - os.utime(config.system.build_file, (timestamp_1, timestamp_1)) + os.utime(os.path.join(config_d, '00_config.ini'), + (timestamp_1, timestamp_1)) + os.utime(os.path.join(config_d, '02_config.ini'), + (timestamp_1, timestamp_1)) timestamp_2 = int(datetime(2011, 8, 1, 5, 6, 7).timestamp()) - os.utime(channel_ini, (timestamp_2, timestamp_2)) + os.utime(os.path.join(config_d, '01_config.ini'), + (timestamp_2, timestamp_2)) + self._resources.enter_context(argv('-C', config_d, '--info')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ - current build number: 1833 + current build number: 1701 device name: nexus7 channel: proposed last update: 2011-08-01 05:06:07 """)) @configuration - def test_info_last_update_date_fallback(self, ini_file): - # --info's last update date falls back to the mtime of - # /etc/ubuntu-build when no channel.ini file exists. - channel_ini = os.path.join(os.path.dirname(ini_file), 'channel.ini') - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--info'])) - # Set up the build number. - config = Configuration(ini_file) - touch_build(1701) - timestamp_1 = int(datetime(2011, 1, 8, 2, 3, 4).timestamp()) - os.utime(config.system.build_file, (timestamp_1, timestamp_1)) - self.assertFalse(os.path.exists(channel_ini)) - cli_main() - self.assertEqual(self._stdout.getvalue(), dedent("""\ - current build number: 1701 - device name: nexus7 - channel: stable - last update: 2011-01-08 02:03:04 - """)) - - @configuration - def test_build_number(self, ini_file): + def test_build_number(self, config_d): # -b/--build overrides the build number. touch_build(1701, TIMESTAMP) # Use --build to override the default build number. self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, - '--build', '20250801', - '--info'])) + argv('-C', config_d, '--build', '20250801', '--info')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ current build number: 20250801 @@ -276,14 +273,11 @@ """)) @configuration - def test_device_name(self, ini_file): + def test_device_name(self, config_d): # -d/--device overrides the device type. touch_build(1701, TIMESTAMP) self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, - '--device', 'phablet', - '--info'])) + argv('-C', config_d, '--device', 'phablet', '--info')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ current build number: 1701 @@ -293,14 +287,11 @@ """)) @configuration - def test_channel_name(self, ini_file): + def test_channel_name(self, config_d): # -c/--channel overrides the channel. touch_build(1701, TIMESTAMP) self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, - '--channel', 'daily-proposed', - '--info'])) + argv('-C', config_d, '--channel', 'daily-proposed', '--info')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ current build number: 1701 @@ -310,16 +301,12 @@ """)) @configuration - def test_channel_name_with_alias(self, ini_file): + def test_channel_name_with_alias(self, config_d): # When the current channel has an alias, this is reflected in the # output for --info - channel_ini = os.path.join(os.path.dirname(ini_file), 'channel.ini') - head, tail = os.path.split(channel_ini) - copy('channel_05.ini', head, tail) + copy('main.config_03.ini', config_d, '01_config.ini') touch_build(300, TIMESTAMP) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--info'])) + self._resources.enter_context(argv('-C', config_d, '--info')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ current build number: 300 @@ -330,17 +317,16 @@ """)) @configuration - def test_all_overrides(self, ini_file): + def test_all_overrides(self, config_d): # Use -b -d and -c together. touch_build(1701, TIMESTAMP) # Use --build to override the default build number. self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, - '-b', '20250801', - '-c', 'daily-proposed', - '-d', 'phablet', - '--info'])) + argv('-C', config_d, + '-b', '20250801', + '-c', 'daily-proposed', + '-d', 'phablet', + '--info')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ current build number: 20250801 @@ -350,11 +336,9 @@ """)) @configuration - def test_bad_build_number_override(self, ini_file): + def test_bad_build_number_override(self, config_d): # -b/--build requires an integer. - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--build', 'bogus'])) + self._resources.enter_context(argv('-C', config_d, '--build', 'bogus')) with self.assertRaises(SystemExit) as cm: cli_main() self.assertEqual(cm.exception.code, 2) @@ -363,49 +347,12 @@ 'system-image-cli: error: -b/--build requires an integer: bogus') @configuration - def test_channel_ini_override_build_number(self, ini_file): - # The channel.ini file can override the build number. - copy('channel_01.ini', os.path.dirname(ini_file), 'channel.ini') - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '-i'])) - # Set up the build number. - touch_build(1701, TIMESTAMP) - cli_main() - self.assertEqual(self._stdout.getvalue(), dedent("""\ - current build number: 1833 - device name: nexus7 - channel: proposed - last update: 2013-08-01 12:11:10 - """)) - - @configuration - def test_channel_ini_override_channel(self, ini_file): - # The channel.ini file can override the channel. - channel_ini = os.path.join(os.path.dirname(ini_file), 'channel.ini') - head, tail = os.path.split(channel_ini) - copy('channel_01.ini', head, tail) - os.utime(channel_ini, (TIMESTAMP, TIMESTAMP)) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '-i'])) - cli_main() - self.assertEqual(self._stdout.getvalue(), dedent("""\ - current build number: 1833 - device name: nexus7 - channel: proposed - last update: 2013-08-01 12:11:10 - """)) - - @configuration - def test_switch_channel(self, ini_file): + def test_switch_channel(self, config_d): # `system-image-cli --switch ` is a convenience equivalent to # `system-image-cli -b 0 --channel `. touch_build(801, TIMESTAMP) self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--switch', 'utopic-proposed', - '--info'])) + argv('-C', config_d, '--switch', 'utopic-proposed', '--info')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ current build number: 0 @@ -415,14 +362,13 @@ """)) @configuration - def test_switch_channel_with_overrides(self, ini_file): + def test_switch_channel_with_overrides(self, config_d): # The use of --switch is a convenience only, and if -b and/or -c is # given explicitly, they override the convenience. touch_build(801, TIMESTAMP) self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--switch', 'utopic-proposed', - '-b', '1', '-c', 'utopic', '--info'])) + argv('-C', config_d, '--switch', 'utopic-proposed', + '-b', '1', '-c', 'utopic', '--info')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ current build number: 1 @@ -432,9 +378,8 @@ """)) @configuration - def test_log_file(self, ini_file): + def test_log_file(self, config): # Test that the system log file gets created and written. - config = Configuration(ini_file) self.assertFalse(os.path.exists(config.system.logfile)) class FakeState: def __init__(self, candidate_filter): @@ -443,9 +388,7 @@ return self def __next__(self): raise StopIteration - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file])) + self._resources.enter_context(argv('-C', config.config_d)) self._resources.enter_context( patch('systemimage.main.State', FakeState)) cli_main() @@ -462,11 +405,11 @@ r'\[systemimage\] [^(]+ \(\d+\) ' r'state machine finished\n') + @unittest.skipIf(os.getuid() == 0, 'Test cannot succeed when run as root') @configuration - def test_log_file_permission_denied(self, ini_file): + def test_log_file_permission_denied(self, config): # LP: #1301995 - some tests are run as non-root, meaning they don't # have access to the system log file. Use a fallback in that case. - config = Configuration(ini_file) # Set the log file to read-only. system_log = Path(config.system.logfile) system_log.touch(0o444, exist_ok=False) @@ -474,9 +417,7 @@ tmpdir = self._resources.enter_context(temporary_directory()) self._resources.enter_context( patch('systemimage.logging.xdg_cache_home', tmpdir)) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--dry-run'])) + self._resources.enter_context(argv('-C', config.config_d, '--dry-run')) cli_main() # There should now be nothing in the system log file, and something in # the fallback log file. @@ -487,11 +428,10 @@ self.assertEqual(stat.filemode(fallback.stat().st_mode), '-rw-------') @configuration - def test_bad_filter_type(self, ini_file): + def test_bad_filter_type(self, config_d): # --filter option where value is not `full` or `delta` is an error. self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--filter', 'bogus'])) + argv('-C', config_d, '--filter', 'bogus')) with self.assertRaises(SystemExit) as cm: cli_main() self.assertEqual(cm.exception.code, 2) @@ -500,18 +440,14 @@ 'system-image-cli: error: Bad filter type: bogus') @configuration - def test_version_detail(self, ini_file): - # --info where channel.ini has [service]version_detail - channel_ini = os.path.join(os.path.dirname(ini_file), 'channel.ini') - head, tail = os.path.split(channel_ini) - copy('channel_03.ini', head, tail) - os.utime(channel_ini, (TIMESTAMP, TIMESTAMP)) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '-i'])) + def test_version_detail(self, config_d): + # --info where a config file has [service]version_detail. + copy('main.config_04.ini', config_d, '01_config.ini') + touch_build(1933, TIMESTAMP) + self._resources.enter_context(argv('-C', config_d, '-i')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ - current build number: 1833 + current build number: 1933 device name: nexus7 channel: proposed last update: 2013-08-01 12:11:10 @@ -521,30 +457,24 @@ """)) @configuration - def test_no_version_detail(self, ini_file): - # --info where channel.ini does not hav [service]version_detail - channel_ini = os.path.join(os.path.dirname(ini_file), 'channel.ini') - head, tail = os.path.split(channel_ini) - copy('channel_01.ini', head, tail) - os.utime(channel_ini, (TIMESTAMP, TIMESTAMP)) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '-i'])) + def test_no_version_detail(self, config_d): + # --info where there is no [service]version_detail setting. + copy('main.config_02.ini', config_d, '01_config.ini') + touch_build(1933, TIMESTAMP) + self._resources.enter_context(argv('-C', config_d, '-i')) cli_main() self.assertEqual(self._stdout.getvalue(), dedent("""\ - current build number: 1833 + current build number: 1933 device name: nexus7 channel: proposed last update: 2013-08-01 12:11:10 """)) @configuration - def test_state_machine_exceptions(self, ini_file): + def test_state_machine_exceptions(self, config): # If an exception happens during the state machine run, the error is # logged and main exits with code 1. - config = Configuration(ini_file) - self._resources.enter_context( - patch('systemimage.main.sys.argv', ['argv0', '-C', ini_file])) + self._resources.enter_context(argv('-C', config.config_d)) # Making the cache directory unwritable is a good way to trigger a # crash. Be sure to set it back though! with chmod(config.updater.cache_partition, 0): @@ -552,27 +482,22 @@ self.assertEqual(exit_code, 1) @configuration - def test_state_machine_exceptions_dry_run(self, ini_file): + def test_state_machine_exceptions_dry_run(self, config): # Like above, but doing only a --dry-run. - config = Configuration(ini_file) - # Making the cache directory unwritable is a good way to trigger a - # crash. Be sure to set it back though! - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--dry-run'])) + self._resources.enter_context(argv('-C', config.config_d, '--dry-run')) with chmod(config.updater.cache_partition, 0): exit_code = cli_main() self.assertEqual(exit_code, 1) class TestCLIMainDryRun(ServerTestBase): - INDEX_FILE = 'index_14.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'main.index_01.json' + CHANNEL_FILE = 'main.channels_01.json' CHANNEL = 'stable' DEVICE = 'nexus7' @configuration - def test_dry_run(self, ini_file): + def test_dry_run(self, config_d): # `system-image-cli --dry-run` prints the winning upgrade path. self._setup_server_keyrings() # We patch builtin print() rather than sys.stdout because the @@ -580,17 +505,18 @@ # the code. capture = StringIO() with ExitStack() as resources: - resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--dry-run'])) + resources.enter_context(capture_print(capture)) + resources.push(machine_id('0000000000000000aaaaaaaaaaaaaaaa')) + resources.enter_context(argv('-C', config_d, '--dry-run')) cli_main() - self.assertEqual(capture.getvalue(), - 'Upgrade path is 1200:1201:1304\n') + self.assertEqual( + capture.getvalue(), """\ +Upgrade path is 1200:1201:1304 +Target phase: 12% +""") @configuration - def test_dry_run_no_update(self, ini_file): + def test_dry_run_no_update(self, config_d): # `system-image-cli --dry-run` when there are no updates available. self._setup_server_keyrings() # We patch builtin print() rather than sys.stdout because the @@ -600,16 +526,13 @@ # Set up the build number. touch_build(1701) with ExitStack() as resources: - resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--dry-run'])) + resources.enter_context(capture_print(capture)) + resources.enter_context(argv('-C', config_d, '--dry-run')) cli_main() self.assertEqual(capture.getvalue(), 'Already up-to-date\n') @configuration - def test_dry_run_bad_channel(self, ini_file): + def test_dry_run_bad_channel(self, config_d): # `system-image-cli --dry-run --channel ` should say it's # already up-to-date. self._setup_server_keyrings() @@ -618,69 +541,148 @@ # the code. capture = StringIO() with ExitStack() as resources: - resources.enter_context( - patch('builtins.print', partial(print, file=capture))) + resources.enter_context(capture_print(capture)) # Use --build to override the default build number. resources.enter_context( - patch('systemimage.main.sys.argv', [ - 'argv0', '-C', ini_file, - '--channel', 'daily-proposed', - '--dry-run'])) + argv('-C', config_d, + '--channel', 'daily-proposed', + '--dry-run')) cli_main() self.assertEqual(capture.getvalue(), 'Already up-to-date\n') + @configuration + def test_percentage(self, config_d): + # --percentage overrides the device's target percentage. + self._setup_server_keyrings() + capture = StringIO() + with ExitStack() as resources: + resources.enter_context(capture_print(capture)) + resources.push(machine_id('0000000000000000aaaaaaaaaaaaaaaa')) + resources.enter_context(argv('-C', config_d, '--dry-run')) + cli_main() + self.assertEqual( + capture.getvalue(), """\ +Upgrade path is 1200:1201:1304 +Target phase: 12% +""") + capture = StringIO() + with ExitStack() as resources: + resources.enter_context(capture_print(capture)) + resources.push(machine_id('0000000000000000aaaaaaaaaaaaaaaa')) + resources.enter_context( + argv('-C', config_d, '--dry-run', '--percentage', '81')) + cli_main() + self.assertEqual( + capture.getvalue(), """\ +Upgrade path is 1200:1201:1304 +Target phase: 81% +""") + + @configuration + def test_p(self, config_d): + # -p overrides the device's target percentage. + self._setup_server_keyrings() + capture = StringIO() + with ExitStack() as resources: + resources.enter_context(capture_print(capture)) + resources.push(machine_id('0000000000000000aaaaaaaaaaaaaaaa')) + resources.enter_context(argv('-C', config_d, '--dry-run')) + cli_main() + self.assertEqual( + capture.getvalue(), """\ +Upgrade path is 1200:1201:1304 +Target phase: 12% +""") + capture = StringIO() + with ExitStack() as resources: + resources.enter_context(capture_print(capture)) + resources.push(machine_id('0000000000000000aaaaaaaaaaaaaaaa')) + resources.enter_context( + argv('-C', config_d, '--dry-run', '-p', '81')) + cli_main() + self.assertEqual( + capture.getvalue(), """\ +Upgrade path is 1200:1201:1304 +Target phase: 81% +""") + + @configuration + def test_crazy_p(self, config_d): + # --percentage/-p value is floored at 0% and ceilinged at 100%. + self._setup_server_keyrings() + capture = StringIO() + with ExitStack() as resources: + resources.enter_context(capture_print(capture)) + resources.push(machine_id('0000000000000000aaaaaaaaaaaaaaaa')) + resources.enter_context( + argv('-C', config_d, '--dry-run', '-p', '10000')) + cli_main() + self.assertEqual( + capture.getvalue(), """\ +Upgrade path is 1200:1201:1304 +Target phase: 100% +""") + capture = StringIO() + with ExitStack() as resources: + resources.enter_context(capture_print(capture)) + resources.push(machine_id('0000000000000000aaaaaaaaaaaaaaaa')) + resources.enter_context( + argv('-C', config_d, '--dry-run', '-p', '-10')) + cli_main() + self.assertEqual( + capture.getvalue(), """\ +Upgrade path is 1200:1201:1304 +Target phase: 0% +""") + class TestCLIMainDryRunAliases(ServerTestBase): - INDEX_FILE = 'index_20.json' - CHANNEL_FILE = 'channels_10.json' + INDEX_FILE = 'main.index_02.json' + CHANNEL_FILE = 'main.channels_02.json' CHANNEL = 'daily' DEVICE = 'manta' @configuration - def test_dry_run_with_channel_alias_switch(self, ini_file): + def test_dry_run_with_channel_alias_switch(self, config_d): # `system-image-cli --dry-run` where the channel alias the device was # on got switched should include this information. self._setup_server_keyrings() - channel_ini = os.path.join(os.path.dirname(ini_file), 'channel.ini') - head, tail = os.path.split(channel_ini) - copy('channel_05.ini', head, tail) + copy('main.config_05.ini', config_d, '01_config.ini') capture = StringIO() - self._resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--dry-run'])) # Do not use self._resources to manage the check_output mock. Because # of the nesting order of the @configuration decorator and the base # class's tearDown(), using self._resources causes the mocks to be # unwound in the wrong order, affecting future tests. - with patch('systemimage.device.check_output', return_value='manta'): + with ExitStack() as resources: + resources.enter_context(capture_print(capture)) + resources.enter_context(argv('-C', config_d, '--dry-run')) + # Patch the machine id. + resources.push(machine_id('0000000000000000aaaaaaaaaaaaaaaa')) + resources.enter_context( + patch('systemimage.device.check_output', return_value='manta')) cli_main() self.assertEqual( - capture.getvalue(), - 'Upgrade path is 200:201:304 (saucy -> tubular)\n') + capture.getvalue(), """\ +Upgrade path is 200:201:304 (saucy -> tubular) +Target phase: 25% +""") class TestCLIListChannels(ServerTestBase): - INDEX_FILE = 'index_20.json' - CHANNEL_FILE = 'channels_10.json' + INDEX_FILE = 'main.index_02.json' + CHANNEL_FILE = 'main.channels_02.json' CHANNEL = 'daily' DEVICE = 'manta' @configuration - def test_list_channels(self, ini_file): + def test_list_channels(self, config_d): # `system-image-cli --list-channels` shows all available channels, # including aliases. self._setup_server_keyrings() - channel_ini = os.path.join(os.path.dirname(ini_file), 'channel.ini') - head, tail = os.path.split(channel_ini) - copy('channel_05.ini', head, tail) + copy('main.config_05.ini', config_d, '01_config.ini') capture = StringIO() - self._resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--list-channels'])) + self._resources.enter_context(capture_print(capture)) + self._resources.enter_context(argv('-C', config_d, '--list-channels')) # Do not use self._resources to manage the check_output mock. Because # of the nesting order of the @configuration decorator and the base # class's tearDown(), using self._resources causes the mocks to be @@ -695,19 +697,14 @@ """)) @configuration - def test_list_channels_exception(self, ini_file): + def test_list_channels_exception(self, config_d): # If an exception occurs while getting the list of channels, we get a # non-zero exit status. self._setup_server_keyrings() - channel_ini = os.path.join(os.path.dirname(ini_file), 'channel.ini') - head, tail = os.path.split(channel_ini) - copy('channel_05.ini', head, tail) + copy('main.config_05.ini', config_d, '01_config.ini') capture = StringIO() - self._resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--list-channels'])) + self._resources.enter_context(capture_print(capture)) + self._resources.enter_context(argv('-C', config_d, '--list-channels')) # Do not use self._resources to manage the check_output mock. Because # of the nesting order of the @configuration decorator and the base # class's tearDown(), using self._resources causes the mocks to be @@ -723,15 +720,15 @@ class TestCLIFilters(ServerTestBase): - INDEX_FILE = 'index_15.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'main.index_03.json' + CHANNEL_FILE = 'main.channels_03.json' CHANNEL = 'stable' DEVICE = 'nexus7' maxDiff = None @configuration - def test_filter_full(self, ini_file): + def test_filter_full(self, config_d): # With --filter=full, only full updates will be considered. self._setup_server_keyrings() # We patch builtin print() rather than sys.stdout because the @@ -741,17 +738,14 @@ # Set up the build number. touch_build(100) with ExitStack() as resources: - resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - resources.enter_context( - patch('systemimage.main.sys.argv', [ - 'argv0', '-C', ini_file, '--dry-run', - '--filter', 'full'])) + resources.enter_context(capture_print(capture)) + resources.enter_context( + argv('-C', config_d, '--dry-run', '--filter', 'full')) cli_main() self.assertMultiLineEqual(capture.getvalue(), 'Already up-to-date\n') @configuration - def test_filter_delta(self, ini_file): + def test_filter_delta(self, config_d): # With --filter=delta, only delta updates will be considered. self._setup_server_keyrings() # We patch builtin print() rather than sys.stdout because the @@ -761,33 +755,33 @@ # Set up the build number. touch_build(100) with ExitStack() as resources: - resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - resources.enter_context( - patch('systemimage.main.sys.argv', [ - 'argv0', '-C', ini_file, '--dry-run', - '--filter', 'delta'])) + resources.enter_context(capture_print(capture)) + resources.enter_context( + argv('-C', config_d, '--dry-run', '--filter', 'delta')) + resources.push(machine_id('0000000000000000aaaaaaaaaaaaaaaa')) cli_main() - self.assertMultiLineEqual(capture.getvalue(), 'Upgrade path is 1600\n') + self.assertMultiLineEqual(capture.getvalue(), """\ +Upgrade path is 1600 +Target phase: 80% +""") class TestCLIDuplicateDestinations(ServerTestBase): - INDEX_FILE = 'index_23.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'main.index_04.json' + CHANNEL_FILE = 'main.channels_03.json' CHANNEL = 'stable' DEVICE = 'nexus7' @configuration - def test_duplicate_destinations(self, ini_file): - # index_23.json has the bug we saw in the wild in LP: #1250181. + def test_duplicate_destinations(self, config_d): + # main.index_04.json has the bug we saw in the wild in LP: #1250181. # There, the server erroneously included a data file twice in two # different images. This can't happen and indicates a server # problem. The client must refuse to upgrade in this case, by raising # an exception. self._setup_server_keyrings() with ExitStack() as resources: - resources.enter_context( - patch('systemimage.main.sys.argv', ['argv0', '-C', ini_file])) + resources.enter_context(argv('-C', config_d)) exit_code = cli_main() self.assertEqual(exit_code, 1) # 2013-11-12 BAW: IWBNI we could assert something about the log @@ -801,25 +795,76 @@ class TestCLINoReboot(ServerTestBase): - INDEX_FILE = 'index_13.json' - CHANNEL_FILE = 'channels_10.json' + INDEX_FILE = 'main.index_05.json' + CHANNEL_FILE = 'main.channels_02.json' CHANNEL = 'daily' DEVICE = 'manta' @configuration - def test_no_reboot(self, ini_file): + def test_no_apply(self, config_d): + # `system-image-cli --no-apply` downloads everything but does not + # apply the update. + self._setup_server_keyrings() + capture = StringIO() + self._resources.enter_context(capture_print(capture)) + self._resources.enter_context( + argv('-C', config_d, '--no-apply', '-b', 0, '-c', 'daily')) + mock = self._resources.enter_context( + patch('systemimage.apply.Reboot.apply')) + # Do not use self._resources to manage the check_output mock. Because + # of the nesting order of the @configuration decorator and the base + # class's tearDown(), using self._resources causes the mocks to be + # unwound in the wrong order, affecting future tests. + with patch('systemimage.device.check_output', return_value='manta'): + cli_main() + # The reboot method was never called. + self.assertFalse(mock.called) + # All the expected files should be downloaded. + self.assertEqual(set(os.listdir(config.updater.data_partition)), set([ + 'blacklist.tar.xz', + 'blacklist.tar.xz.asc', + ])) + self.assertEqual(set(os.listdir(config.updater.cache_partition)), set([ + '5.txt', + '5.txt.asc', + '6.txt', + '6.txt.asc', + '7.txt', + '7.txt.asc', + 'device-signing.tar.xz', + 'device-signing.tar.xz.asc', + 'image-master.tar.xz', + 'image-master.tar.xz.asc', + 'image-signing.tar.xz', + 'image-signing.tar.xz.asc', + 'ubuntu_command', + ])) + path = os.path.join(config.updater.cache_partition, 'ubuntu_command') + with open(path, 'r', encoding='utf-8') as fp: + command = fp.read() + self.assertMultiLineEqual(command, """\ +load_keyring image-master.tar.xz image-master.tar.xz.asc +load_keyring image-signing.tar.xz image-signing.tar.xz.asc +load_keyring device-signing.tar.xz device-signing.tar.xz.asc +format system +mount system +update 6.txt 6.txt.asc +update 7.txt 7.txt.asc +update 5.txt 5.txt.asc +unmount system +""") + + @configuration + def test_no_reboot(self, config_d): # `system-image-cli --no-reboot` downloads everything but does not - # reboot into recovery. + # apply the update. THIS IS DEPRECATED IN SI 3.0. self._setup_server_keyrings() capture = StringIO() - self._resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--no-reboot', - '-b', 0, '-c', 'daily'])) + self._resources.enter_context(capture_print(capture)) + self._resources.enter_context( + argv('-C', config_d, '--no-reboot', '-b', 0, '-c', 'daily')) mock = self._resources.enter_context( - patch('systemimage.reboot.Reboot.reboot')) + patch('systemimage.apply.Reboot.apply')) # Do not use self._resources to manage the check_output mock. Because # of the nesting order of the @configuration decorator and the base # class's tearDown(), using self._resources causes the mocks to be @@ -864,18 +909,16 @@ """) @configuration - def test_g(self, ini_file): + def test_g(self, config_d): # `system-image-cli -g` downloads everything but does not reboot into # recovery. self._setup_server_keyrings() capture = StringIO() - self._resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '-g', '-b', 0, '-c', 'daily'])) + self._resources.enter_context(capture_print(capture)) + self._resources.enter_context( + argv('-C', config_d, '-g', '-b', 0, '-c', 'daily')) mock = self._resources.enter_context( - patch('systemimage.reboot.Reboot.reboot')) + patch('systemimage.apply.Reboot.apply')) # Do not use self._resources to manage the check_output mock. Because # of the nesting order of the @configuration decorator and the base # class's tearDown(), using self._resources causes the mocks to be @@ -920,18 +963,16 @@ """) @configuration - def test_rerun_after_no_reboot_reboots(self, ini_file): + def test_rerun_after_no_reboot_reboots(self, config_d): # Running system-image-cli again after a `system-image-cli -g` does # not download anything the second time, but does issue a reboot. self._setup_server_keyrings() capture = StringIO() - self._resources.enter_context( - patch('builtins.print', partial(print, file=capture))) + self._resources.enter_context(capture_print(capture)) mock = self._resources.enter_context( - patch('systemimage.reboot.Reboot.reboot')) + patch('systemimage.apply.Reboot.apply')) self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '-g', '-b', 0, '-c', 'daily'])) + argv('-C', config_d, '-g', '-b', 0, '-c', 'daily')) # Do not use self._resources to manage the check_output mock. Because # of the nesting order of the @configuration decorator and the base # class's tearDown(), using self._resources causes the mocks to be @@ -945,8 +986,11 @@ shutil.rmtree(os.path.join(self._serverdir, '3')) shutil.rmtree(os.path.join(self._serverdir, '4')) shutil.rmtree(os.path.join(self._serverdir, '5')) - with patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '-b', 0, '-c', 'daily']): + # Run main again without the -g flag this time we reboot. + with ExitStack() as stack: + stack.enter_context(argv('-C', config_d, '-b', 0, '-c', 'daily')) + stack.enter_context( + patch('systemimage.device.check_output', return_value='manta')) cli_main() # The reboot method was never called. self.assertTrue(mock.called) @@ -956,25 +1000,46 @@ """Test the --factory-reset option for factory resets.""" @configuration - def test_factory_reset(self, ini_file): + def test_factory_reset(self, config_d): # system-image-cli --factory-reset capture = StringIO() with ExitStack() as resources: - resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - mock = resources.enter_context( - patch('systemimage.reboot.Reboot.reboot')) - resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--factory-reset'])) - cli_main() - # A reboot was issued. - self.assertTrue(mock.called) - path = os.path.join(config.updater.cache_partition, 'ubuntu_command') - with open(path, 'r', encoding='utf-8') as fp: - command = fp.read() - self.assertMultiLineEqual(command, dedent("""\ - format data + resources.enter_context(capture_print(capture)) + mock = resources.enter_context( + patch('systemimage.apply.Reboot.apply')) + resources.enter_context(argv('-C', config_d, '--factory-reset')) + cli_main() + # A reboot was issued. + self.assertTrue(mock.called) + path = os.path.join(config.updater.cache_partition, 'ubuntu_command') + with open(path, 'r', encoding='utf-8') as fp: + command = fp.read() + self.assertMultiLineEqual(command, dedent("""\ + format data + """)) + + +class TestCLIProductionReset(unittest.TestCase): + """Test the --production-reset option for production factory resets.""" + + @configuration + def test_production_reset(self, config_d): + # system-image-cli --production-reset + capture = StringIO() + with ExitStack() as resources: + resources.enter_context(capture_print(capture)) + mock = resources.enter_context( + patch('systemimage.apply.Reboot.apply')) + resources.enter_context(argv('-C', config_d, '--production-reset')) + cli_main() + # A reboot was issued. + self.assertTrue(mock.called) + path = os.path.join(config.updater.cache_partition, 'ubuntu_command') + with open(path, 'r', encoding='utf-8') as fp: + command = fp.read() + self.assertMultiLineEqual(command, dedent("""\ + format data + enable factory_wipe """)) @@ -990,8 +1055,7 @@ # We patch builtin print() rather than sys.stdout because the # latter can mess with pdb output should we need to trace through # the code. - self._resources.enter_context( - patch('builtins.print', partial(print, file=self._stdout))) + self._resources.enter_context(capture_print(self._stdout)) # Patch argparse's stderr to capture its error messages. self._resources.enter_context( patch('argparse._sys.stderr', self._stderr)) @@ -1004,16 +1068,14 @@ super().tearDown() @configuration - def test_show_settings(self, ini_file): + def test_show_settings(self, config_d): # `system-image-cli --show-settings` shows all the keys and values in # sorted order by alphanumeric key name. settings = Settings() settings.set('peart', 'neil') settings.set('lee', 'geddy') settings.set('lifeson', 'alex') - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--show-settings'])) + self._resources.enter_context(argv('-C', config_d, '--show-settings')) cli_main() self.assertMultiLineEqual(self._stdout.getvalue(), dedent("""\ lee=geddy @@ -1022,29 +1084,25 @@ """)) @configuration - def test_get_key(self, ini_file): + def test_get_key(self, config_d): # `system-image-cli --get key` prints the key's value. settings = Settings() settings.set('ant', 'aunt') - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--get', 'ant'])) + self._resources.enter_context(argv('-C', config_d, '--get', 'ant')) cli_main() self.assertMultiLineEqual(self._stdout.getvalue(), dedent("""\ aunt """)) @configuration - def test_get_keys(self, ini_file): + def test_get_keys(self, config_d): # `--get key` can be used multiple times. settings = Settings() settings.set('s', 'saucy') settings.set('t', 'trusty') settings.set('u', 'utopic') self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, - '--get', 's', '--get', 'u', '--get', 't'])) + argv('-C', config_d, '--get', 's', '--get', 'u', '--get', 't')) cli_main() self.assertMultiLineEqual(self._stdout.getvalue(), dedent("""\ saucy @@ -1053,13 +1111,11 @@ """)) @configuration - def test_get_missing_key(self, ini_file): + def test_get_missing_key(self, config_d): # Since by definition a missing key has a default value, you can get # missing keys. Note that `auto_download` is the one weirdo. self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, - '--get', 'missing', '--get', 'auto_download'])) + argv('-C', config_d, '--get', 'missing', '--get', 'auto_download')) cli_main() # This produces a blank line, since `missing` returns the empty # string. For better readability, don't indent the results. @@ -1069,38 +1125,33 @@ """) @configuration - def test_set_key(self, ini_file): + def test_set_key(self, config_d): # `system-image-cli --set key=value` sets a key/value pair. - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--set', 'bass=4'])) + self._resources.enter_context(argv('-C', config_d, '--set', 'bass=4')) cli_main() self.assertEqual(Settings().get('bass'), '4') @configuration - def test_change_key(self, ini_file): + def test_change_key(self, config_d): # `--set key=value` changes an existing key's value. settings = Settings() settings.set('a', 'ant') settings.set('b', 'bee') settings.set('c', 'cat') - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--set', 'b=bat'])) + self._resources.enter_context(argv('-C', config_d, '--set', 'b=bat')) cli_main() self.assertEqual(settings.get('a'), 'ant') self.assertEqual(settings.get('b'), 'bat') self.assertEqual(settings.get('c'), 'cat') @configuration - def test_set_keys(self, ini_file): + def test_set_keys(self, config_d): # `--set key=value` can be used multiple times. self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, - '--set', 'a=ant', - '--set', 'b=bee', - '--set', 'c=cat'])) + argv('-C', config_d, + '--set', 'a=ant', + '--set', 'b=bee', + '--set', 'c=cat')) cli_main() settings = Settings() self.assertEqual(settings.get('a'), 'ant') @@ -1108,15 +1159,13 @@ self.assertEqual(settings.get('c'), 'cat') @configuration - def test_del_key(self, ini_file): + def test_del_key(self, config_d): # `system-image-cli --del key` removes a key from the database. settings = Settings() settings.set('ant', 'insect') settings.set('bee', 'insect') settings.set('cat', 'mammal') - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--del', 'bee'])) + self._resources.enter_context(argv('-C', config_d, '--del', 'bee')) cli_main() settings = Settings() self.assertEqual(settings.get('ant'), 'insect') @@ -1125,15 +1174,14 @@ self.assertEqual(settings.get('bee'), '') @configuration - def test_del_keys(self, ini_file): + def test_del_keys(self, config_d): # `--del key` can be used multiple times. settings = Settings() settings.set('ant', 'insect') settings.set('bee', 'insect') settings.set('cat', 'mammal') self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--del', 'bee', '--del', 'cat'])) + argv('-C', config_d, '--del', 'bee', '--del', 'cat')) cli_main() settings = Settings() self.assertEqual(settings.get('ant'), 'insect') @@ -1142,27 +1190,23 @@ self.assertEqual(settings.get('bee'), '') @configuration - def test_del_missing_key(self, ini_file): + def test_del_missing_key(self, config_d): # When asked to delete a key that's not in the database, nothing # much happens. - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, '--del', 'missing'])) + self._resources.enter_context(argv('-C', config_d, '--del', 'missing')) cli_main() self.assertEqual(Settings().get('missing'), '') @configuration - def test_mix_and_match(self, ini_file): + def test_mix_and_match(self, config_d): # Because argument order is not preserved, and any semantics for # mixing and matching database arguments would be arbitrary, it is not # allowed to mix them. capture = StringIO() - self._resources.enter_context( - patch('builtins.print', partial(print, file=capture))) - self._resources.enter_context( - patch('systemimage.main.sys.argv', - ['argv0', '-C', ini_file, - '--set', 'c=cat', '--del', 'bee', '--get', 'dog'])) + self._resources.enter_context(capture_print(capture)) + self._resources.enter_context( + argv('-C', config_d, + '--set', 'c=cat', '--del', 'bee', '--get', 'dog')) with self.assertRaises(SystemExit) as cm: cli_main() self.assertEqual(cm.exception.code, 2) @@ -1173,45 +1217,44 @@ class TestDBusMain(unittest.TestCase): def setUp(self): + super().setUp() self._stack = ExitStack() try: - old_ini = SystemImagePlugin.controller.ini_path - self._stack.callback( - setattr, SystemImagePlugin.controller, 'ini_path', old_ini) - self.tmpdir = self._stack.enter_context(temporary_directory()) - template = resource_bytes( - 'systemimage.tests.data', 'config_04.ini').decode('utf-8') - self.ini_path = os.path.join(self.tmpdir, 'client.ini') - with open(self.ini_path, 'w', encoding='utf-8') as fp: - print(template.format(tmpdir=self.tmpdir, vardir=self.tmpdir), - file=fp) - SystemImagePlugin.controller.ini_path = self.ini_path SystemImagePlugin.controller.set_mode() + config_d = SystemImagePlugin.controller.ini_path + override = os.path.join(config_d, '06_override.ini') + self._stack.callback(safe_remove, override) + with open(override, 'w', encoding='utf-8') as fp: + print('[dbus]\nlifetime: 3s\n', file=fp) + # The testing framework will have caused system-image-dbus to be + # started by now. The tests below assume it is not yet running, so + # let's be sure to stop it. + terminate_service() except: self._stack.close() raise def tearDown(self): - bus = dbus.SystemBus() - service = bus.get_object('com.canonical.SystemImage', '/Service') - iface = dbus.Interface(service, 'com.canonical.SystemImage') - iface.Exit() - self._stack.close() + try: + terminate_service() + finally: + self._stack.close() def _activate(self): - # Start the D-Bus service. + # Re-start and reload the D-Bus service. + wait_for_service() bus = dbus.SystemBus() service = bus.get_object('com.canonical.SystemImage', '/Service') - iface = dbus.Interface(service, 'com.canonical.SystemImage') - return iface.Info() + self._iface = dbus.Interface(service, 'com.canonical.SystemImage') + return self._iface.Information() def test_service_exits(self): # The dbus service automatically exits after a set amount of time. - # + config_d = SystemImagePlugin.controller.ini_path # Nothing has been spawned yet. - self.assertIsNone(find_dbus_process(self.ini_path)) + self.assertIsNone(find_dbus_process(config_d)) self._activate() - process = find_dbus_process(self.ini_path) + process = find_dbus_process(config_d) self.assertTrue(process.is_running()) # Now wait for the process to self-terminate. If this times out # before the process exits, a TimeoutExpired exception will be @@ -1221,37 +1264,32 @@ def test_service_keepalive(self): # Proactively calling methods on the service keeps it alive. - self.assertIsNone(find_dbus_process(self.ini_path)) + config_d = SystemImagePlugin.controller.ini_path + self.assertIsNone(find_dbus_process(config_d)) self._activate() - process = find_dbus_process(self.ini_path) + process = find_dbus_process(config_d) self.assertTrue(process.is_running()) # Normally the process would exit after 3 seconds, but we'll keep it # alive for a bit. for i in range(3): - self._activate() + self._iface.Information() time.sleep(2) self.assertTrue(process.is_running()) - def test_channel_ini_override(self): - # An optional channel.ini can override the build number and channel. - # - # The config.ini file names the `stable` channel. Let's create an - # ubuntu-build file with a fake version number. - config = Configuration(self.ini_path) - with open(config.system.build_file, 'w', encoding='utf-8') as fp: - print(33, file=fp) - # Now, write a channel.ini file to override both of these. - dirname = os.path.dirname(self.ini_path) - copy('channel_04.ini', dirname, 'channel.ini') + def test_config_override(self): + # Other ini files can override the build number and channel. + config_d = SystemImagePlugin.controller.ini_path + copy('main.config_07.ini', config_d, '07_override.ini') info = self._activate() # The build number. - self.assertEqual(info[0], 1) + self.assertEqual(info['current_build_number'], '33') # The channel - self.assertEqual(info[2], 'saucy') + self.assertEqual(info['channel_name'], 'saucy') def test_temp_directory(self): # The temporary directory gets created if it doesn't exist. - config = Configuration(self.ini_path) + config_d = SystemImagePlugin.controller.ini_path + config = Configuration(config_d) # The temporary directory may have already been created via the # .set_mode() call in the setUp(). That invokes a 'stopper' for the # -dbus process, which has the perverse effect of first D-Bus @@ -1271,7 +1309,7 @@ def test_permissions(self): # LP: #1235975 - The created tempdir had unsafe permissions. - config = Configuration(self.ini_path) + config = Configuration(SystemImagePlugin.controller.ini_path) # See above. try: shutil.rmtree(config.system.tempdir) @@ -1289,9 +1327,10 @@ def test_single_instance(self): # Only one instance of the system-image-dbus service is allowed to # remain active on a single system bus. - self.assertIsNone(find_dbus_process(self.ini_path)) + config_d = SystemImagePlugin.controller.ini_path + self.assertIsNone(find_dbus_process(config_d)) self._activate() - proc = find_dbus_process(self.ini_path) + proc = find_dbus_process(config_d) # Attempt to start a second process on the same system bus. env = dict( DBUS_SYSTEM_BUS_ADDRESS=os.environ['DBUS_SYSTEM_BUS_ADDRESS']) @@ -1299,7 +1338,7 @@ if coverage_env is not None: env['COVERAGE_PROCESS_START'] = coverage_env args = (sys.executable, '-m', 'systemimage.testing.service', - '-C', self.ini_path) + '-C', config_d) second = subprocess.Popen(args, universal_newlines=True, env=env) # Allow a TimeoutExpired exception to fail the test. try: @@ -1310,3 +1349,257 @@ raise self.assertNotEqual(second.pid, proc.pid) self.assertEqual(code, 2) + + +class TestDBusMainNoConfigD(unittest.TestCase): + def test_start_with_missing_config_d(self): + # Trying to start the D-Bus service with a configuration directory + # that doesn't exist yields an error. + terminate_service() + wait_for_service(reload=False) + # Try to start a new process with a bogus configuration directory. + env = dict( + DBUS_SYSTEM_BUS_ADDRESS=os.environ['DBUS_SYSTEM_BUS_ADDRESS']) + coverage_env = os.environ.get('COVERAGE_PROCESS_START') + if coverage_env is not None: + env['COVERAGE_PROCESS_START'] = coverage_env + args = (sys.executable, '-m', 'systemimage.testing.service', + '-C', '/does/not/exist') + with temporary_directory() as tempdir: + stdout_path = os.path.join(tempdir, 'stdout') + stderr_path = os.path.join(tempdir, 'stderr') + with ExitStack() as files: + tempdir = files.enter_context(temporary_directory()) + stdout = files.enter_context( + open(stdout_path, 'w', encoding='utf-8')) + stderr = files.enter_context( + open(stderr_path, 'w', encoding='utf-8')) + try: + subprocess.check_call(args, + universal_newlines=True, env=env, + stdout=stdout, stderr=stderr) + except subprocess.CalledProcessError as error: + self.assertNotEqual(error.returncode, 0) + with open(stdout_path, 'r', encoding='utf-8') as fp: + stdout = fp.read() + with open(stderr_path, 'r', encoding='utf-8') as fp: + stderr = fp.readlines() + self.assertEqual(stdout, '') + self.assertEqual( + stderr[-1], + 'Configuration directory not found: .load() requires a ' + 'directory: /does/not/exist\n') + + +class TestCLISignatures(ServerTestBase): + INDEX_FILE = 'main.index_01.json' + CHANNEL_FILE = 'main.channels_01.json' + CHANNEL = 'stable' + DEVICE = 'nexus7' + + @configuration + def test_update_attempt_with_bad_signatures(self, config_d): + # Let's say the index.json file has a bad signature. The update + # should refuse to apply. + self._setup_server_keyrings() + # Sign the index.json file with the wrong (i.e. bad) key. + index_path = os.path.join( + self._serverdir, self.CHANNEL, self.DEVICE, 'index.json') + sign(index_path, 'spare.gpg') + stdout = StringIO() + with ExitStack() as resources: + resources.enter_context(capture_print(stdout)) + # Patch argparse's stderr to capture its error messages. + resources.push(machine_id('feedfacebeefbacafeedfacebeefbaca')) + resources.enter_context(argv('-C', config_d, '--dry-run')) + # Now that the index.json on the server is signed with the wrong + # keyring, try to upgrade. + code = cli_main() + # The upgrade failed because of the signature. + self.assertEqual(code, 1) + with open(config.system.logfile, encoding='utf-8') as fp: + logged = fp.readlines() + # Slog through the log output and look for evidence that the upgrade + # failed because of the faulty signature on the index.json file. + # Then assert on those clues, but get rid of the trailing newlines. + exception_found = False + data_path = sig_path = None + i = 0 + while i < len(logged): + line = logged[i][:-1] + i += 1 + if line.startswith('systemimage.gpg.SignatureError'): + # There should only be one of these lines. + self.assertFalse(exception_found) + exception_found = True + elif line.strip().startswith('sig path'): + sig_path = logged[i][:-1] + i += 1 + elif line.strip().startswith('data path'): + data_path = logged[i][:-1] + i += 1 + # Check the clues. + self.assertTrue(exception_found) + self.assertTrue(sig_path.endswith('index.json.asc'), repr(sig_path)) + self.assertTrue(data_path.endswith('index.json'), repr(data_path)) + + @configuration + def test_update_attempt_with_bad_signatures_overridden(self, config_d): + # Let's say the index.json file has a bad signature. Normally, the + # update should refuse to apply, but we override the GPG checks so it + # will succeed. + self._setup_server_keyrings() + # Sign the index.json file with the wrong (i.e. bad) key. + index_path = os.path.join( + self._serverdir, self.CHANNEL, self.DEVICE, 'index.json') + sign(index_path, 'spare.gpg') + stdout = StringIO() + stderr = StringIO() + with ExitStack() as resources: + resources.enter_context(capture_print(stdout)) + resources.enter_context( + patch('systemimage.main.sys.stderr', stderr)) + # Patch argparse's stderr to capture its error messages. + resources.push(machine_id('feedfacebeefbacafeedfacebeefbaca')) + resources.enter_context( + argv('-C', config_d, '--dry-run', '--skip-gpg-verification')) + # Now that the index.json on the server is signed with the wrong + # keyring, try to upgrade. + code = cli_main() + # The upgrade failed because of the signature. + self.assertEqual(code, 0) + self.assertEqual(stdout.getvalue(), """\ +Upgrade path is 1200:1201:1304 +Target phase: 64% +""") + # And we get the scary warning on the console. + self.assertMultiLineEqual(stderr.getvalue(), """\ +WARNING: All GPG signature verifications have been disabled. +Your upgrades are INSECURE. +""") + + +class TestCLIProgress(ServerTestBase): + INDEX_FILE = 'main.index_01.json' + CHANNEL_FILE = 'main.channels_01.json' + CHANNEL = 'stable' + DEVICE = 'nexus7' + + @configuration + def test_dots_progress(self, config_d): + # --progress=dots prints a bunch of dots to stderr. + self._setup_server_keyrings() + stderr = StringIO() + with ExitStack() as resources: + resources.enter_context( + patch('systemimage.main.LINE_LENGTH', 10)) + resources.enter_context( + patch('systemimage.main.sys.stderr', stderr)) + resources.enter_context( + argv('-C', config_d, '-b', '0', '--no-reboot', + '--progress', 'dots')) + cli_main() + # There should be some dots in the stderr. + self.assertGreater(stderr.getvalue().count('.'), 2) + + @configuration + def test_json_progress(self, config_d): + # --progress=json prints some JSON to stdout. + self._setup_server_keyrings() + stdout = StringIO() + with ExitStack() as resources: + resources.enter_context( + patch('systemimage.main.sys.stdout', stdout)) + resources.enter_context( + argv('-C', config_d, '-b', '0', '--no-reboot', + '--progress', 'json')) + cli_main() + # stdout is now filled with JSON goodness. We can't assert too much + # about the contents though. + line_count = 0 + for line in stdout.getvalue().splitlines(): + line_count += 1 + record = json.loads(line) + self.assertEqual(record['type'], 'progress') + self.assertIn('now', record) + self.assertIn('total', record) + self.assertGreater(line_count, 4) + + @configuration + def test_logfile_progress(self, config_d): + # --progress=logfile dumps some messages to the log file. + self._setup_server_keyrings() + log_mock = MagicMock() + from systemimage.main import _LogfileProgress + class Testable(_LogfileProgress): + def __init__(self, log): + super().__init__(log) + self._log = log_mock + with ExitStack() as resources: + resources.enter_context( + patch('systemimage.main._LogfileProgress', Testable)) + resources.enter_context( + argv('-C', config_d, '-b', '0', '--no-reboot', + '--progress', 'logfile')) + cli_main() + self.assertGreater(log_mock.debug.call_count, 4) + positional, keyword = log_mock.debug.call_args + self.assertTrue(positional[0].startswith('received: ')) + + @configuration + def test_all_progress(self, config_d): + # We can have more than one --progress flag. + self._setup_server_keyrings() + stdout = StringIO() + stderr = StringIO() + log_mock = MagicMock() + from systemimage.main import _LogfileProgress + class Testable(_LogfileProgress): + def __init__(self, log): + super().__init__(log) + self._log = log_mock + with ExitStack() as resources: + resources.enter_context( + patch('systemimage.main.LINE_LENGTH', 10)) + resources.enter_context( + patch('systemimage.main.sys.stderr', stderr)) + resources.enter_context( + patch('systemimage.main.sys.stdout', stdout)) + resources.enter_context( + patch('systemimage.main._LogfileProgress', Testable)) + resources.enter_context( + argv('-C', config_d, '-b', '0', '--no-reboot', + '--progress', 'dots', + '--progress', 'json', + '--progress', 'logfile')) + cli_main() + self.assertGreater(stderr.getvalue().count('.'), 2) + line_count = 0 + for line in stdout.getvalue().splitlines(): + line_count += 1 + record = json.loads(line) + self.assertEqual(record['type'], 'progress') + self.assertIn('now', record) + self.assertIn('total', record) + self.assertGreater(line_count, 4) + self.assertGreater(log_mock.debug.call_count, 4) + positional, keyword = log_mock.debug.call_args + self.assertTrue(positional[0].startswith('received: ')) + + @configuration + def test_bad_progress(self, config_d): + # An unknown progress type results in an error. + stderr = StringIO() + with ExitStack() as resources: + resources.enter_context( + patch('systemimage.main.sys.stderr', stderr)) + resources.enter_context( + argv('-C', config_d, '-b', '0', '--no-reboot', + '--progress', 'not-a-meter')) + with self.assertRaises(SystemExit) as cm: + cli_main() + exit_code = cm.exception.code + self.assertEqual(exit_code, 2) + self.assertEqual( + stderr.getvalue().splitlines()[-1], + 'system-image-cli: error: Unknown progress meter: not-a-meter') === modified file 'systemimage/tests/test_scores.py' --- systemimage/tests/test_scores.py 2014-02-20 23:03:24 +0000 +++ systemimage/tests/test_scores.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -14,6 +14,8 @@ # along with this program. If not, see . __all__ = [ + 'TestPhasedUpdates', + 'TestVersionDetail', 'TestWeightedScorer', ] @@ -22,7 +24,8 @@ from systemimage.candidates import get_candidates from systemimage.scores import WeightedScorer -from systemimage.testing.helpers import get_index +from systemimage.testing.helpers import descriptions, get_index +from unittest.mock import patch class TestWeightedScorer(unittest.TestCase): @@ -31,20 +34,20 @@ def test_choose_no_candidates(self): # If there are no candidates, then there is no path to upgrade. - self.assertEqual(self.scorer.choose([]), []) + self.assertEqual(self.scorer.choose([], 'devel'), []) def test_score_no_candidates(self): self.assertEqual(self.scorer.score([]), []) def test_one_path(self): - index = get_index('index_08.json') + index = get_index('scores.index_02.json') candidates = get_candidates(index, 600) # There's only one path. scores = self.scorer.score(candidates) # The score is 200 for the two extra bootme flags. self.assertEqual(scores, [200]) # And we upgrade to the only path available. - winner = self.scorer.choose(candidates) + winner = self.scorer.choose(candidates, 'devel') # There are two images in the winning path. self.assertEqual(len(winner), 2) self.assertEqual([image.version for image in winner], [1300, 1301]) @@ -62,27 +65,136 @@ # a huge score making it impossible to win. # # Path B wins. - index = get_index('index_09.json') + index = get_index('scores.index_03.json') candidates = get_candidates(index, 600) # There are three paths. The scores are as above. scores = self.scorer.score(candidates) self.assertEqual(scores, [300, 200, 9401]) - winner = self.scorer.choose(candidates) + winner = self.scorer.choose(candidates, 'devel') self.assertEqual(len(winner), 3) self.assertEqual([image.version for image in winner], [1200, 1201, 1304]) - descriptions = [] - for image in winner: - # There's only one description per image so order doesn't matter. - descriptions.extend(image.descriptions.values()) - self.assertEqual(descriptions, ['Full B', 'Delta B.1', 'Delta B.2']) + self.assertEqual(descriptions(winner), + ['Full B', 'Delta B.1', 'Delta B.2']) def test_tied_candidates(self): # LP: #1206866 - TypeError when two candidate paths scored equal. # - # index_17.json was captured from real data causing the traceback. - index = get_index('index_17.json') + # index_04.json was captured from real data causing the traceback. + index = get_index('scores.index_04.json') candidates = get_candidates(index, 1) - path = self.scorer.choose(candidates) + path = self.scorer.choose(candidates, 'devel') self.assertEqual(len(path), 1) self.assertEqual(path[0].version, 1800) + + +class TestPhasedUpdates(unittest.TestCase): + def setUp(self): + self.scorer = WeightedScorer() + + def test_inside_phase_gets_update(self): + # When the final image on an update path has a phase percentage higher + # than the device percentage, the candidate path is okay. In this + # case, the `Full B` has phase of 50%. + index = get_index('scores.index_05.json') + candidates = get_candidates(index, 100) + with patch('systemimage.scores.phased_percentage', return_value=22): + winner = self.scorer.choose(candidates, 'devel') + descriptions = [] + for image in winner: + descriptions.extend(image.descriptions.values()) + self.assertEqual(descriptions, ['Full B', 'Delta B.1', 'Delta B.2']) + + def test_outside_phase_gets_update(self): + # When the final image on an update path has a phase percentage lower + # than the device percentage, the scorer falls back to the next + # candidate path. + index = get_index('scores.index_05.json') + candidates = get_candidates(index, 100) + with patch('systemimage.scores.phased_percentage', return_value=66): + winner = self.scorer.choose(candidates, 'devel') + self.assertEqual(descriptions(winner), + ['Full A', 'Delta A.1', 'Delta A.2']) + + def test_equal_phase_gets_update(self): + # When the final image on an update path has a phase percentage exactly + # equal to the device percentage, the candidate path is okay. In this + # case, the `Full B` has phase of 50%. + index = get_index('scores.index_05.json') + candidates = get_candidates(index, 100) + with patch('systemimage.scores.phased_percentage', return_value=50): + winner = self.scorer.choose(candidates, 'devel') + self.assertEqual(descriptions(winner), + ['Full B', 'Delta B.1', 'Delta B.2']) + + def test_pulled_update(self): + # When the final image on an update path has a phase percentage of + # zero, then regardless of the device's percentage, the candidate path + # is not okay. In this case, the `Full B` has phase of 0%. + index = get_index('scores.index_01.json') + candidates = get_candidates(index, 100) + with patch('systemimage.scores.phased_percentage', return_value=0): + winner = self.scorer.choose(candidates, 'devel') + self.assertEqual(descriptions(winner), + ['Full A', 'Delta A.1', 'Delta A.2']) + + def test_pulled_update_insanely_negative_randint(self): + # When the final image on an update path has a phase percentage of + # zero, then regardless of the device's percentage (even if randint + # returned some insane value), the candidate path is not okay. In this + # case, the `Full B` has phase of 0%. + index = get_index('scores.index_01.json') + candidates = get_candidates(index, 100) + with patch('systemimage.scores.phased_percentage', return_value=-100): + winner = self.scorer.choose(candidates, 'devel') + self.assertEqual(descriptions(winner), + ['Full A', 'Delta A.1', 'Delta A.2']) + + def test_pulled_update_insanely_positive_randint(self): + # When the final image on an update path has a phase percentage of + # zero, then regardless of the device's percentage (even if randint + # returned some insane value), the candidate path is not okay. In this + # case, the `Full B` has phase of 0%. + index = get_index('scores.index_01.json') + candidates = get_candidates(index, 100) + with patch('systemimage.scores.phased_percentage', return_value=1000): + winner = self.scorer.choose(candidates, 'devel') + self.assertEqual(len(winner), 0) + + +class TestVersionDetail(unittest.TestCase): + def setUp(self): + self.scorer = WeightedScorer() + + def test_version_detail(self): + # The index.json file has three paths for updates, but only one is + # selected. The winning path lands on an image with a version_detail + # key. + index = get_index('scores.index_06.json') + candidates = get_candidates(index, 600) + scores = self.scorer.score(candidates) + self.assertEqual(scores, [300, 200, 9401]) + winner = self.scorer.choose(candidates, 'devel') + self.assertEqual(len(winner), 3) + self.assertEqual([image.version for image in winner], + [1200, 1201, 1304]) + self.assertEqual(descriptions(winner), + ['Full B', 'Delta B.1', 'Delta B.2']) + self.assertEqual(winner[-1].version_detail, + "ubuntu=105,raw-device=205,version=305") + + def test_no_version_detail(self): + # The index.json file has three paths for updates, but only one is + # selected. The winning path lands on an image without a + # version_detail key. + index = get_index('scores.index_07.json') + candidates = get_candidates(index, 600) + scores = self.scorer.score(candidates) + self.assertEqual(scores, [300, 200, 9401]) + winner = self.scorer.choose(candidates, 'devel') + self.assertEqual(len(winner), 3) + self.assertEqual([image.version for image in winner], + [1200, 1201, 1304]) + self.assertEqual(descriptions(winner), + ['Full B', 'Delta B.1', 'Delta B.2']) + self.assertEqual(winner[-1].version_detail, '') === modified file 'systemimage/tests/test_settings.py' --- systemimage/tests/test_settings.py 2014-07-31 23:20:10 +0000 +++ systemimage/tests/test_settings.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -25,7 +25,6 @@ from contextlib import ExitStack from pathlib import Path -from systemimage.config import Configuration from systemimage.helpers import temporary_directory from systemimage.settings import Settings from systemimage.testing.helpers import chmod, configuration @@ -34,8 +33,7 @@ class TestSettings(unittest.TestCase): @configuration - def test_creation(self, ini_file): - config = Configuration(ini_file) + def test_creation(self, config): self.assertFalse(os.path.exists(config.system.settings_db)) settings = Settings() self.assertTrue(os.path.exists(config.system.settings_db)) @@ -104,13 +102,13 @@ keyval.sort() self.assertEqual(keyval, [('a', 'ant'), ('b', 'bee'), ('c', 'cat')]) + @unittest.skipIf(os.getuid() == 0, 'Test cannot succeed when run as root') @configuration - def test_settings_db_permission_denied(self, ini_file): + def test_settings_db_permission_denied(self, config): # LP: #1349478 - some tests are run as non-root, meaning they don't # have write permission to /var/lib/system-image. This is where # settings.db gets created, but if the process can't create files # there, we get a sqlite3 exception. - config = Configuration(ini_file) db_file = Path(config.system.settings_db) self.assertFalse(db_file.exists()) with ExitStack() as resources: === modified file 'systemimage/tests/test_state.py' --- systemimage/tests/test_state.py 2014-09-17 13:41:31 +0000 +++ systemimage/tests/test_state.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -25,10 +25,10 @@ 'TestKeyringDoubleChecks', 'TestMiscellaneous', 'TestPhasedUpdates', - 'TestRebooting', 'TestState', 'TestStateDuplicateDestinations', 'TestStateNewChannelsFormat', + 'TestUpdateApplied', ] @@ -48,12 +48,10 @@ from systemimage.state import ChecksumError, State from systemimage.testing.demo import DemoDevice from systemimage.testing.helpers import ( - ServerTestBase, configuration, copy, data_path, get_index, + ServerTestBase, configuration, copy, data_path, descriptions, get_index, make_http_server, setup_keyring_txz, setup_keyrings, sign, temporary_directory, touch_build) from systemimage.testing.nose import SystemImagePlugin -# FIXME -from systemimage.tests.test_candidates import _descriptions from unittest.mock import call, patch BAD_SIGNATURE = 'f' * 64 @@ -73,7 +71,7 @@ self._serverdir = self._stack.enter_context(temporary_directory()) self._stack.push(make_http_server( self._serverdir, 8943, 'cert.pem', 'key.pem')) - copy('channels_01.json', self._serverdir, 'channels.json') + copy('state.channels_07.json', self._serverdir, 'channels.json') self._channels_path = os.path.join( self._serverdir, 'channels.json') except: @@ -385,11 +383,11 @@ self.assertRaises(SignatureError, next, state) -class TestRebooting(ServerTestBase): - """Test various state transitions leading to a reboot.""" +class TestUpdateApplied(ServerTestBase): + """Test various state transitions leading to the applying of the update.""" - INDEX_FILE = 'index_13.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'state.index_03.json' + CHANNEL_FILE = 'state.channels_02.json' CHANNEL = 'stable' DEVICE = 'nexus7' @@ -418,7 +416,7 @@ self.assertFalse(os.path.exists(signing_path + '.asc')) self.assertFalse(os.path.exists(device_path + '.asc')) # None of the data files are found yet. - for image in get_index('index_13.json').images: + for image in get_index('state.index_03.json').images: for filerec in image.files: path = os.path.join(cache_dir, os.path.basename(filerec.path)) asc = os.path.join( @@ -443,7 +441,7 @@ self.assertTrue(os.path.exists(signing_path + '.asc')) self.assertTrue(os.path.exists(device_path + '.asc')) # All of the data files are found. - for image in get_index('index_13.json').images: + for image in get_index('state.index_03.json').images: for filerec in image.files: path = os.path.join(cache_dir, os.path.basename(filerec.path)) asc = os.path.join( @@ -452,26 +450,27 @@ self.assertTrue(os.path.exists(asc)) @configuration - def test_reboot_issued(self): - # The reboot gets issued. + def test_update_applied(self, config): + # The update gets applied self._setup_server_keyrings() - with ExitStack() as resources: - mock = resources.enter_context( - patch('systemimage.reboot.check_call')) + ini_path = os.path.join(config.config_d, '10_state.ini') + shutil.copy(data_path('state.config_01.ini'), ini_path) + config.reload() + with patch('systemimage.apply.Noop.apply') as mock: list(State()) - self.assertEqual(mock.call_args[0][0], - ['/sbin/reboot', '-f', 'recovery']) + self.assertEqual(mock.call_count, 1) @configuration - def test_no_update_available_no_reboot(self): + def test_no_update_available_no_apply(self, config): # LP: #1202915. If there's no update available, running the state - # machine to completion should not result in a reboot. + # machine to completion should not make the call to apply it. self._setup_server_keyrings() + ini_path = os.path.join(config.config_d, '10_state.ini') + shutil.copy(data_path('state.config_01.ini'), ini_path) + config.reload() # Hack the current build number so that no update is available. touch_build(5000) - with ExitStack() as resources: - mock = resources.enter_context( - patch('systemimage.reboot.Reboot.reboot')) + with patch('systemimage.apply.Noop.apply') as mock: list(State()) self.assertEqual(mock.call_count, 0) @@ -483,10 +482,13 @@ self.assertRaises(CalledProcessError, list, State()) @configuration - def test_run_until(self): + def test_run_until(self, config): # It is possible to run the state machine either until some specific # state is completed, or it runs to the end. self._setup_server_keyrings() + ini_path = os.path.join(config.config_d, '10_state.ini') + shutil.copy(data_path('state.config_01.ini'), ini_path) + config.reload() state = State() self.assertIsNone(state.channels) state.run_thru('get_channel') @@ -496,26 +498,19 @@ # Run it some more. state.run_thru('get_index') self.assertIsNotNone(state.index) - # Run until just before the reboot. - # - # Mock the reboot to make sure a reboot did not get issued. - got_reboot = False - def reboot_mock(self): - nonlocal got_reboot - got_reboot = True - with patch('systemimage.reboot.Reboot.reboot', reboot_mock): - state.run_until('reboot') - # No reboot got issued. - self.assertFalse(got_reboot) - # Finish it off. - with patch('systemimage.reboot.Reboot.reboot', reboot_mock): + # Run until just before the apply step. + with patch('systemimage.apply.Noop.apply') as mock: + state.run_until('apply') + self.assertEqual(mock.call_count, 0) + # Run to the end of the state machine. + with patch('systemimage.apply.Noop.apply', mock): list(state) - self.assertTrue(got_reboot) + self.assertEqual(mock.call_count, 1) class TestRebootingNoDeviceSigning(ServerTestBase): - INDEX_FILE = 'index_13.json' - CHANNEL_FILE = 'channels_11.json' + INDEX_FILE = 'state.index_03.json' + CHANNEL_FILE = 'state.channels_03.json' CHANNEL = 'stable' DEVICE = 'nexus7' SIGNING_KEY = 'image-signing.gpg' @@ -546,7 +541,7 @@ self.assertFalse(os.path.exists(signing_path + '.asc')) self.assertFalse(os.path.exists(device_path + '.asc')) # None of the data files are found yet. - for image in get_index('index_13.json').images: + for image in get_index('state.index_03.json').images: for filerec in image.files: path = os.path.join(cache_dir, os.path.basename(filerec.path)) asc = os.path.join( @@ -572,7 +567,7 @@ self.assertTrue(os.path.exists(signing_path + '.asc')) self.assertFalse(os.path.exists(device_path + '.asc')) # All of the data files are found. - for image in get_index('index_13.json').images: + for image in get_index('state.index_03.json').images: for filerec in image.files: path = os.path.join(cache_dir, os.path.basename(filerec.path)) asc = os.path.join( @@ -582,8 +577,8 @@ class TestCommandFileFull(ServerTestBase): - INDEX_FILE = 'index_13.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'state.index_03.json' + CHANNEL_FILE = 'state.channels_02.json' CHANNEL = 'stable' DEVICE = 'nexus7' @@ -591,7 +586,7 @@ def test_full_command_file(self): # A full update's command file gets properly filled. self._setup_server_keyrings() - State().run_until('reboot') + State().run_until('apply') path = os.path.join(config.updater.cache_partition, 'ubuntu_command') with open(path, 'r', encoding='utf-8') as fp: command = fp.read() @@ -630,8 +625,8 @@ class TestCommandFileDelta(ServerTestBase): - INDEX_FILE = 'index_15.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'state.index_04.json' + CHANNEL_FILE = 'state.channels_02.json' CHANNEL = 'stable' DEVICE = 'nexus7' @@ -641,7 +636,7 @@ self._setup_server_keyrings() # Set the current build number so a delta update will work. touch_build(100) - State().run_until('reboot') + State().run_until('apply') path = os.path.join(config.updater.cache_partition, 'ubuntu_command') with open(path, 'r', encoding='utf-8') as fp: command = fp.read() @@ -658,8 +653,8 @@ class TestFileOrder(ServerTestBase): - INDEX_FILE = 'index_16.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'state.index_05.json' + CHANNEL_FILE = 'state.channels_02.json' CHANNEL = 'stable' DEVICE = 'nexus7' @@ -670,7 +665,7 @@ self._setup_server_keyrings() # Set the current build number so a delta update will work. touch_build(100) - State().run_until('reboot') + State().run_until('apply') path = os.path.join(config.updater.cache_partition, 'ubuntu_command') with open(path, 'r', encoding='utf-8') as fp: command = fp.read() @@ -696,8 +691,8 @@ class TestDailyProposed(ServerTestBase): """Test that the daily-proposed channel works as expected.""" - INDEX_FILE = 'index_13.json' - CHANNEL_FILE = 'channels_07.json' + INDEX_FILE = 'state.index_03.json' + CHANNEL_FILE = 'state.channels_04.json' CHANNEL = 'daily-proposed' DEVICE = 'grouper' @@ -731,8 +726,8 @@ class TestVersionedProposed(ServerTestBase): - INDEX_FILE = 'index_13.json' - CHANNEL_FILE = 'channels_08.json' + INDEX_FILE = 'state.index_03.json' + CHANNEL_FILE = 'state.channels_05.json' CHANNEL = '14.04-proposed' DEVICE = 'grouper' @@ -753,8 +748,8 @@ class TestFilters(ServerTestBase): - INDEX_FILE = 'index_15.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'state.index_04.json' + CHANNEL_FILE = 'state.channels_02.json' CHANNEL = 'stable' DEVICE = 'nexus7' @@ -781,16 +776,20 @@ class TestStateNewChannelsFormat(ServerTestBase): - CHANNEL_FILE = 'channels_09.json' + CHANNEL_FILE = 'state.channels_06.json' CHANNEL = 'saucy' DEVICE = 'manta' - INDEX_FILE = 'index_21.json' + INDEX_FILE = 'state.index_06.json' @configuration - def test_full_reboot(self): + def test_full_reboot(self, config_d): # Test that state transitions through reboot work for the new channel # format. Also check that the right files get moved into place. - config.load(data_path('channel_04.ini'), override=True) + shutil.copy(data_path('state.config_01.ini'), + os.path.join(config_d, '11_state.ini')) + shutil.copy(data_path('state.config_02.ini'), + os.path.join(config_d, '12_state.ini')) + config.reload() self._setup_server_keyrings() state = State() # Do not use self._resources to manage the check_output mock. Because @@ -798,7 +797,7 @@ # class's tearDown(), using self._resources causes the mocks to be # unwound in the wrong order, affecting future tests. with patch('systemimage.device.check_output', return_value='manta'): - state.run_until('reboot') + state.run_until('apply') path = os.path.join(config.updater.cache_partition, 'ubuntu_command') with open(path, 'r', encoding='utf-8') as fp: command = fp.read() @@ -812,29 +811,25 @@ update 5.txt 5.txt.asc unmount system """) - got_reboot = False - def reboot_mock(self): - nonlocal got_reboot - got_reboot = True - with patch('systemimage.reboot.Reboot.reboot', reboot_mock): + with patch('systemimage.apply.Noop.apply') as mock: list(state) - self.assertTrue(got_reboot) + self.assertEqual(mock.call_count, 1) class TestChannelAlias(ServerTestBase): - CHANNEL_FILE = 'channels_10.json' + CHANNEL_FILE = 'state.channels_01.json' CHANNEL = 'daily' DEVICE = 'manta' - INDEX_FILE = 'index_20.json' + INDEX_FILE = 'state.index_01.json' @configuration - def test_channel_alias_switch(self): + def test_channel_alias_switch(self, config_d): # Channels in the channel.json files can have an optional "alias" key, # which if set, describes the other channel this channel is based on # (only in a server-side generated way; the client sees all channels # as fully "stocked"). # - # The channel.ini file can have a channel_target key which names the + # The [service] section can have a `channel_target` key which names the # channel alias this device has been tracking. If the channel_target # does not match the channel alias, then the client considers its # internal version number to be 0 and does a full update. @@ -850,9 +845,8 @@ # upgrade from build number 0 to get on the right track. # # To test this condition, we calculate the upgrade path first in the - # absence of a channels.ini file. The device is tracking the daily - # channel, and there isno channel_target attribute, so we get the - # latest build on that channel. + # absence of a [service]channel_target key. The device is tracking the + # daily channel, so we get the latest build on that channel. self._setup_server_keyrings() touch_build(300) config.channel = 'daily' @@ -876,12 +870,16 @@ # Set the build number back to 300 for the next test. del config.build_number touch_build(300) - # Now we pretend there was a channel.ini file, and load it. This also - # tells us the current build number is 300, but through the - # channel_target field it tells us that the previous daily channel - # alias was saucy. Now (via the channels.json file) it's tubular, and - # the upgrade path starting at build 0 is different. - config.load(data_path('channel_05.ini'), override=True) + # Now we drop in a configuration file which sets the + # [service]channel_target key. This also tells us the current build + # number is 300, but through the channel_target field it tells us that + # the previous daily channel alias was saucy. Now (via the + # channels.json file) it's tubular, and the upgrade path starting at + # build 0 is different. + override_path = os.path.join(config_d, '02_override.ini') + with open(override_path, 'w', encoding='utf-8') as fp: + print('[service]\nchannel_target: saucy\n', file=fp) + config.reload() # All things being equal to the first test above, except that now # we're in the middle of an alias switch. The upgrade path is exactly # the same as if we were upgrading from build 0. @@ -892,7 +890,7 @@ [200, 201, 304]) @configuration - def test_channel_alias_switch_with_cli_option(self): + def test_channel_alias_switch_with_cli_option(self, config_d): # Like the above test, but in similating the use of `system-image-cli # --build 300`, we set the build number explicitly. This prevent the # channel alias squashing of the build number to 0. @@ -909,11 +907,19 @@ state.run_thru('calculate_winner') self.assertEqual([image.version for image in state.winner], [301, 304]) - # Now we pretend there was a channel.ini file, and load it. This also - # tells us the current build number is 300, but through the - # channel_target field it tells us that the previous daily channel - # alias was saucy. Now (via the channels.json file) it's tubular. - config.load(data_path('channel_05.ini'), override=True) + # Now we have an override file. This also tells us the current build + # number is 300, but through the channel_target field it tells us that + # the previous daily channel alias was saucy. Now (via the + # channels.json file) it's tubular. + override_path = os.path.join(config_d, '02_override.ini') + with open(override_path, 'w', encoding='utf-8') as fp: + print("""\ +[service] +channel_target: saucy +channeL: daily +build_number: 300 +""", file=fp) + config.reload() # All things being equal to the first test above, except that now # we're in the middle of an alias switch. The upgrade path is exactly # the same as if we were upgrading from build 0. @@ -923,7 +929,7 @@ state.run_thru('calculate_winner') self.assertEqual([image.version for image in state.winner], [200, 201, 304]) - # Finally, this mimic the effect of --build 300, thus giving us back + # Finally, this mimics the effect of --build 300, thus giving us back # the original upgrade path. config.build_number = 300 state = State() @@ -933,31 +939,65 @@ class TestPhasedUpdates(ServerTestBase): - CHANNEL_FILE = 'channels_10.json' + CHANNEL_FILE = 'state.channels_01.json' CHANNEL = 'daily' DEVICE = 'manta' - INDEX_FILE = 'index_22.json' - - @configuration - def test_phased_updates(self): - # With our threshold at 66, the "Full B" image is suppressed, thus the - # upgrade path is different than it normally would be. In this case, - # the 'A' path is taken (in fact, the B path isn't even considered). - self._setup_server_keyrings() - config.channel = 'daily' - state = State() - self._resources.enter_context( - patch('systemimage.index.phased_percentage', return_value=66)) - # Do not use self._resources to manage the check_output mock. Because - # of the nesting order of the @configuration decorator and the base - # class's tearDown(), using self._resources causes the mocks to be - # unwound in the wrong order, affecting future tests. - with patch('systemimage.device.check_output', return_value='manta'): - state.run_thru('calculate_winner') - self.assertEqual(_descriptions(state.winner), + INDEX_FILE = 'state.index_07.json' + + @configuration + def test_inside_phased_updates_0(self): + # With our threshold at 22, the normal upgrade to "Full B" image is ok. + self._setup_server_keyrings() + config.channel = 'daily' + state = State() + self._resources.enter_context( + patch('systemimage.scores.phased_percentage', return_value=22)) + # Do not use self._resources to manage the check_output mock. Because + # of the nesting order of the @configuration decorator and the base + # class's tearDown(), using self._resources causes the mocks to be + # unwound in the wrong order, affecting future tests. + with patch('systemimage.device.check_output', return_value='manta'): + state.run_thru('calculate_winner') + self.assertEqual(descriptions(state.winner), + ['Full B', 'Delta B.1', 'Delta B.2']) + + @configuration + def test_outside_phased_updates(self): + # With our threshold at 66, the normal upgrade to "Full B" image is + # discarded, and the previous Full A update is chosen instead. + self._setup_server_keyrings() + config.channel = 'daily' + state = State() + self._resources.enter_context( + patch('systemimage.scores.phased_percentage', return_value=66)) + # Do not use self._resources to manage the check_output mock. Because + # of the nesting order of the @configuration decorator and the base + # class's tearDown(), using self._resources causes the mocks to be + # unwound in the wrong order, affecting future tests. + with patch('systemimage.device.check_output', return_value='manta'): + state.run_thru('calculate_winner') + self.assertEqual(descriptions(state.winner), ['Full A', 'Delta A.1', 'Delta A.2']) @configuration + def test_equal_phased_updates_0(self): + # With our threshold at 50, i.e. exactly equal to the image's + # percentage, the normal upgrade to "Full B" image is ok. + self._setup_server_keyrings() + config.channel = 'daily' + state = State() + self._resources.enter_context( + patch('systemimage.scores.phased_percentage', return_value=50)) + # Do not use self._resources to manage the check_output mock. Because + # of the nesting order of the @configuration decorator and the base + # class's tearDown(), using self._resources causes the mocks to be + # unwound in the wrong order, affecting future tests. + with patch('systemimage.device.check_output', return_value='manta'): + state.run_thru('calculate_winner') + self.assertEqual(descriptions(state.winner), + ['Full B', 'Delta B.1', 'Delta B.2']) + + @configuration def test_phased_updates_0(self): # With our threshold at 0, all images are good, so it's a "normal" # update path. @@ -965,40 +1005,102 @@ config.channel = 'daily' state = State() self._resources.enter_context( - patch('systemimage.index.phased_percentage', return_value=0)) + patch('systemimage.scores.phased_percentage', return_value=0)) # Do not use self._resources to manage the check_output mock. Because # of the nesting order of the @configuration decorator and the base # class's tearDown(), using self._resources causes the mocks to be # unwound in the wrong order, affecting future tests. with patch('systemimage.device.check_output', return_value='manta'): state.run_thru('calculate_winner') - self.assertEqual(_descriptions(state.winner), + self.assertEqual(descriptions(state.winner), ['Full B', 'Delta B.1', 'Delta B.2']) @configuration def test_phased_updates_100(self): - # With our threshold at 100, only the image without a specific - # phased-percentage key is allowed. That's the 'A' path again. - self._setup_server_keyrings() - config.channel = 'daily' - state = State() - self._resources.enter_context( - patch('systemimage.index.phased_percentage', return_value=77)) - # Do not use self._resources to manage the check_output mock. Because - # of the nesting order of the @configuration decorator and the base - # class's tearDown(), using self._resources causes the mocks to be - # unwound in the wrong order, affecting future tests. - with patch('systemimage.device.check_output', return_value='manta'): - state.run_thru('calculate_winner') - self.assertEqual(_descriptions(state.winner), - ['Full A', 'Delta A.1', 'Delta A.2']) + # With our threshold at 100, the "Full B" image is discarded and the + # backup "Full A" image is chosen. + self._setup_server_keyrings() + config.channel = 'daily' + state = State() + self._resources.enter_context( + patch('systemimage.scores.phased_percentage', return_value=77)) + # Do not use self._resources to manage the check_output mock. Because + # of the nesting order of the @configuration decorator and the base + # class's tearDown(), using self._resources causes the mocks to be + # unwound in the wrong order, affecting future tests. + with patch('systemimage.device.check_output', return_value='manta'): + state.run_thru('calculate_winner') + self.assertEqual(descriptions(state.winner), + ['Full A', 'Delta A.1', 'Delta A.2']) + + +class TestPhasedUpdatesPulled(ServerTestBase): + CHANNEL_FILE = 'state.channels_01.json' + CHANNEL = 'daily' + DEVICE = 'manta' + INDEX_FILE = 'state.index_02.json' + + @configuration + def test_pulled_update(self): + # Regardless of the device's phase percentage, when the image has a + # percentage of 0, it will never be considered. In this case Full B + # has a phased percentage of 0, so the fallback Full A is chosen. + self._setup_server_keyrings() + config.channel = 'daily' + state = State() + self._resources.enter_context( + patch('systemimage.scores.phased_percentage', return_value=0)) + # Do not use self._resources to manage the check_output mock. Because + # of the nesting order of the @configuration decorator and the base + # class's tearDown(), using self._resources causes the mocks to be + # unwound in the wrong order, affecting future tests. + with patch('systemimage.device.check_output', return_value='manta'): + state.run_thru('calculate_winner') + self.assertEqual(descriptions(state.winner), + ['Full A', 'Delta A.1', 'Delta A.2']) + + @configuration + def test_pulled_update_insanely_negative_randint(self): + # Regardless of the device's phase percentage, when the image has a + # percentage of 0, it will never be considered. In this case Full B + # has a phased percentage of 0, so the fallback Full A is chosen. + self._setup_server_keyrings() + config.channel = 'daily' + state = State() + self._resources.enter_context( + patch('systemimage.scores.phased_percentage', return_value=-100)) + # Do not use self._resources to manage the check_output mock. Because + # of the nesting order of the @configuration decorator and the base + # class's tearDown(), using self._resources causes the mocks to be + # unwound in the wrong order, affecting future tests. + with patch('systemimage.device.check_output', return_value='manta'): + state.run_thru('calculate_winner') + self.assertEqual(descriptions(state.winner), + ['Full A', 'Delta A.1', 'Delta A.2']) + + @configuration + def test_pulled_update_insanely_positive_randint(self): + # Regardless of the device's phase percentage, when the image has a + # percentage of 0, it will never be considered. + self._setup_server_keyrings() + config.channel = 'daily' + state = State() + self._resources.enter_context( + patch('systemimage.scores.phased_percentage', return_value=1000)) + # Do not use self._resources to manage the check_output mock. Because + # of the nesting order of the @configuration decorator and the base + # class's tearDown(), using self._resources causes the mocks to be + # unwound in the wrong order, affecting future tests. + with patch('systemimage.device.check_output', return_value='manta'): + state.run_thru('calculate_winner') + self.assertEqual(len(state.winner), 0) class TestCachedFiles(ServerTestBase): - CHANNEL_FILE = 'channels_11.json' + CHANNEL_FILE = 'state.channels_03.json' CHANNEL = 'stable' DEVICE = 'nexus7' - INDEX_FILE = 'index_13.json' + INDEX_FILE = 'state.index_03.json' SIGNING_KEY = 'image-signing.gpg' @configuration @@ -1278,7 +1380,7 @@ # Otherwise we clean those files out and start from scratch. self._setup_server_keyrings() state = State() - state.run_until('reboot') + state.run_until('apply') self.assertTrue(os.path.exists( os.path.join(config.updater.cache_partition, 'ubuntu_command'))) # Now, to prove that the data files are not re-downloaded with a new @@ -1294,11 +1396,11 @@ path = os.path.join(config.updater.cache_partition, filename) mtimes[filename] = os.stat(path).st_mtime_ns self.assertGreater(len(mtimes), 0) - # Now create a new state machine, and run until reboot again. Even - # though there are no data files on the server, this still completes - # successfully. + # Now create a new state machine, and run until the update gets applied + # again. Even though there are no data files on the server, this still + # completes successfully. state = State() - state.run_until('reboot') + state.run_until('apply') # Check all the mtimes. for filename in os.listdir(config.updater.cache_partition): if filename.endswith('.txt') or filename.endswith('.txt.asc'): @@ -1348,17 +1450,17 @@ with open(asc_path, 'rb') as fp: checksum = hashlib.md5(fp.read()).digest() mtime = os.stat(txt_path).st_mtime_ns - state.run_until('reboot') + state.run_until('apply') with open(asc_path, 'rb') as fp: self.assertNotEqual(checksum, hashlib.md5(fp.read()).digest) self.assertNotEqual(mtime, os.stat(txt_path).st_mtime_ns) class TestKeyringDoubleChecks(ServerTestBase): - CHANNEL_FILE = 'channels_11.json' + CHANNEL_FILE = 'state.channels_03.json' CHANNEL = 'stable' DEVICE = 'nexus7' - INDEX_FILE = 'index_13.json' + INDEX_FILE = 'state.index_03.json' SIGNING_KEY = 'image-signing.gpg' @configuration @@ -1523,14 +1625,14 @@ class TestStateDuplicateDestinations(ServerTestBase): """An index.json with duplicate destination files is broken.""" - INDEX_FILE = 'index_23.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'state.index_08.json' + CHANNEL_FILE = 'state.channels_02.json' CHANNEL = 'stable' DEVICE = 'nexus7' @configuration def test_duplicate_destinations(self): - # index_23.json has the bug we saw in the wild in LP: #1250181. + # state.index_08.json has the bug we saw in the wild in LP: #1250181. # There, the server erroneously included a data file twice in two # different images. This can't happen and indicates a server # problem. The client must refuse to upgrade in this case, by raising @@ -1558,8 +1660,8 @@ class TestMiscellaneous(ServerTestBase): """Test a few additional things for full code coverage.""" - INDEX_FILE = 'index_13.json' - CHANNEL_FILE = 'channels_06.json' + INDEX_FILE = 'state.index_03.json' + CHANNEL_FILE = 'state.channels_02.json' CHANNEL = 'stable' DEVICE = 'nexus7' @@ -1614,7 +1716,7 @@ call('No blacklist found on second attempt')) # Even though there's no blacklist file, everything still gets # downloaded correctly. - state.run_until('reboot') + state.run_until('apply') path = os.path.join(config.updater.cache_partition, 'ubuntu_command') with open(path, 'r', encoding='utf-8') as fp: command = fp.read() === modified file 'systemimage/tests/test_winner.py' --- systemimage/tests/test_winner.py 2014-07-23 22:51:19 +0000 +++ systemimage/tests/test_winner.py 2015-05-20 14:55:53 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2014 Canonical Ltd. +# Copyright (C) 2013-2015 Canonical Ltd. # Author: Barry Warsaw # This program is free software: you can redistribute it and/or modify @@ -28,7 +28,7 @@ from systemimage.config import config from systemimage.gpg import SignatureError from systemimage.helpers import temporary_directory -from systemimage.state import ChecksumError, State +from systemimage.state import State from systemimage.testing.helpers import ( configuration, copy, make_http_server, setup_index, setup_keyring_txz, setup_keyrings, sign, touch_build) @@ -49,17 +49,18 @@ self._stack = ExitStack() try: self._serverdir = self._stack.enter_context(temporary_directory()) - copy('channels_02.json', self._serverdir, 'channels.json') + copy('winner.channels_01.json', self._serverdir, 'channels.json') sign(os.path.join(self._serverdir, 'channels.json'), 'image-signing.gpg') - # index_10.json path B will win, with no bootme flags. + # Path B will win, with no bootme flags. self._indexpath = os.path.join('stable', 'nexus7', 'index.json') - copy('index_12.json', self._serverdir, self._indexpath) + copy('winner.index_02.json', self._serverdir, self._indexpath) sign(os.path.join(self._serverdir, self._indexpath), 'image-signing.gpg') # Create every file in path B. The file contents will be the # checksum value. We need to create the signatures on the fly. - setup_index('index_12.json', self._serverdir, 'image-signing.gpg') + setup_index('winner.index_02.json', self._serverdir, + 'image-signing.gpg') self._stack.push( make_http_server(self._serverdir, 8943, 'cert.pem', 'key.pem')) self._stack.push(make_http_server(self._serverdir, 8980)) @@ -175,7 +176,7 @@ setup_keyrings() # To set up the device signing key, we need to load channels_03.json # and copy the device keyring to the server. - copy('channels_03.json', self._serverdir, 'channels.json') + copy('winner.channels_02.json', self._serverdir, 'channels.json') sign(os.path.join(self._serverdir, 'channels.json'), 'image-signing.gpg') setup_keyring_txz( @@ -186,7 +187,8 @@ # signed with the device key. sign(os.path.join(self._serverdir, self._indexpath), 'device-signing.gpg') - setup_index('index_12.json', self._serverdir, 'device-signing.gpg') + setup_index('winner.index_02.json', self._serverdir, + 'device-signing.gpg') touch_build(100) # Run the state machine until we download the files. state = State() @@ -214,9 +216,9 @@ # they are signed by the device key instead of the image signing # master. setup_keyrings() - # To set up the device signing key, we need to load channels_03.json - # and copy the device keyring to the server. - copy('channels_03.json', self._serverdir, 'channels.json') + # To set up the device signing key, we need to load this channels.json + # file and copy the device keyring to the server. + copy('winner.channels_02.json', self._serverdir, 'channels.json') sign(os.path.join(self._serverdir, 'channels.json'), 'image-signing.gpg') setup_keyring_txz( @@ -226,7 +228,8 @@ sign(os.path.join(self._serverdir, self._indexpath), 'device-signing.gpg') # All the downloadable files are now signed with the image signing key. - setup_index('index_12.json', self._serverdir, 'image-signing.gpg') + setup_index('winner.index_02.json', self._serverdir, + 'image-signing.gpg') touch_build(100) # Run the state machine until we download the files. state = State() @@ -251,11 +254,12 @@ @configuration def test_download_winners_bad_checksums(self): # Similar to the various good paths, except because the checksums are - # wrong in index_10.json, we'll get a error when downloading. - copy('index_10.json', self._serverdir, self._indexpath) + # wrong in this index.json file, we'll get a error when downloading. + copy('winner.index_01.json', self._serverdir, self._indexpath) sign(os.path.join(self._serverdir, self._indexpath), 'image-signing.gpg') - setup_index('index_10.json', self._serverdir, 'image-signing.gpg') + setup_index('winner.index_01.json', self._serverdir, + 'image-signing.gpg') setup_keyrings() state = State() touch_build(100) @@ -272,9 +276,9 @@ # signing key, which according to the spec means the files are not # signed correctly. setup_keyrings() - # To set up the device signing key, we need to load channels_03.json - # and copy the device keyring to the server. - copy('channels_03.json', self._serverdir, 'channels.json') + # To set up the device signing key, we need to load this channels.json + # file and copy the device keyring to the server. + copy('winner.channels_02.json', self._serverdir, 'channels.json') sign(os.path.join(self._serverdir, 'channels.json'), 'image-signing.gpg') setup_keyring_txz( @@ -284,7 +288,7 @@ sign(os.path.join(self._serverdir, self._indexpath), 'device-signing.gpg') # All the downloadable files are now signed with a bogus key. - setup_index('index_12.json', self._serverdir, 'spare.gpg') + setup_index('winner.index_02.json', self._serverdir, 'spare.gpg') touch_build(100) # Run the state machine until just before we download the files. state = State() === added file 'systemimage/udm.py' --- systemimage/udm.py 1970-01-01 00:00:00 +0000 +++ systemimage/udm.py 2015-05-20 14:55:53 +0000 @@ -0,0 +1,212 @@ +# Copyright (C) 2014-2015 Canonical Ltd. +# Author: Barry Warsaw + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 3 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +"""Download files via ubuntu-download-manager.""" + +__all__ = [ + 'UDMDownloadManager', + ] + + +import os +import dbus +import logging + +from systemimage.config import config +from systemimage.download import Canceled, DownloadManagerBase +from systemimage.reactor import Reactor +from systemimage.settings import Settings + +log = logging.getLogger('systemimage') + +# Parameterized for testing purposes. +DOWNLOADER_INTERFACE = 'com.canonical.applications.Downloader' +MANAGER_INTERFACE = 'com.canonical.applications.DownloadManager' +OBJECT_NAME = 'com.canonical.applications.Downloader' +OBJECT_INTERFACE = 'com.canonical.applications.GroupDownload' + + +def _headers(): + return {'User-Agent': config.user_agent} + + +def _print(*args, **kws): + # We must import this here to avoid circular imports. + ## from systemimage.testing.helpers import debug + ## with debug() as ddlog: + ## ddlog(*args, **kws) + pass + + +class DownloadReactor(Reactor): + def __init__(self, bus, object_path, callback=None, pausable=False): + super().__init__(bus) + self._callback = callback + self._pausable = pausable + # For _do_pause() percentage calculation. + self._received = 0 + self._total = 0 + self.error = None + self.canceled = False + self.local_paths = None + self.react_to('canceled', object_path) + self.react_to('error', object_path) + self.react_to('finished', object_path) + self.react_to('paused', object_path) + self.react_to('progress', object_path) + self.react_to('resumed', object_path) + self.react_to('started', object_path) + + def _do_started(self, signal, path, started): + _print('STARTED:', started) + + def _do_finished(self, signal, path, local_paths): + _print('FINISHED:', local_paths) + self.local_paths = local_paths + self.quit() + + def _do_error(self, signal, path, error_message): + _print('ERROR:', error_message) + log.error(error_message) + self.error = error_message + self.quit() + + def _do_progress(self, signal, path, received, total): + _print('PROGRESS:', received, total) + # For _do_pause() percentage calculation. + self._received = received + self._total = total + self._callback(received, total) + + def _do_canceled(self, signal, path, canceled): + # Why would we get this signal if it *wasn't* canceled? Anyway, + # this'll be a D-Bus data type so converted it to a vanilla Python + # boolean. + _print('CANCELED:', canceled) + self.canceled = bool(canceled) + self.quit() + + def _do_paused(self, signal, path, paused): + _print('PAUSE:', paused, self._pausable) + send_paused = self._pausable and config.dbus_service is not None + if send_paused: # pragma: no branch + # We could plumb through the `service` object from service.py (the + # main entry point for system-image-dbus, but that's actually a + # bit of a pain, so do the expedient thing and grab the interface + # here. + percentage = (int(self._received / self._total * 100.0) + if self._total > 0 else 0) + config.dbus_service.UpdatePaused(percentage) + + def _do_resumed(self, signal, path, resumed): + _print('RESUME:', resumed) + # There currently is no UpdateResumed() signal. + + def _default(self, *args, **kws): + _print('SIGNAL:', args, kws) # pragma: no cover + + +class UDMDownloadManager(DownloadManagerBase): + """Download via ubuntu-download-manager (UDM).""" + + def __init__(self, callback=None): + super().__init__() + if callback is not None: + self.callbacks.append(callback) + self._iface = None + + def _get_files(self, records, pausable): + assert self._iface is None + bus = dbus.SystemBus() + service = bus.get_object(DOWNLOADER_INTERFACE, '/') + iface = dbus.Interface(service, MANAGER_INTERFACE) + object_path = iface.createDownloadGroup( + records, + 'sha256', + False, # Don't allow GSM yet. + # https://bugs.freedesktop.org/show_bug.cgi?id=55594 + dbus.Dictionary(signature='sv'), + _headers()) + download = bus.get_object(OBJECT_NAME, object_path) + self._iface = dbus.Interface(download, OBJECT_INTERFACE) + # Are GSM downloads allowed? Yes, except if auto_download is set to 1 + # (i.e. wifi-only). + allow_gsm = Settings().get('auto_download') != '1' + UDMDownloadManager._set_gsm(self._iface, allow_gsm=allow_gsm) + # Start the download. + reactor = DownloadReactor( + bus, object_path, self._reactor_callback, pausable) + reactor.schedule(self._iface.start) + log.info('[{}] Running group download reactor', object_path) + reactor.run() + # This download is complete so the object path is no longer + # applicable. Setting this to None will cause subsequent cancels to + # be queued. + self._iface = None + log.info('[{}] Group download reactor done', object_path) + if reactor.error is not None: + log.error('Reactor error: {}'.format(reactor.error)) + if reactor.canceled: + log.info('Reactor canceled') + # Report any other problems. + if reactor.error is not None: + raise FileNotFoundError(reactor.error) + if reactor.canceled: + raise Canceled + if reactor.timed_out: + raise TimeoutError + # Sanity check the downloaded results. + # First, every requested destination file must exist, otherwise + # udm would not have given us a `finished` signal. + missing = [record.destination for record in records + if not os.path.exists(record.destination)] + if len(missing) > 0: # pragma: no cover + local_paths = sorted(reactor.local_paths) + raise AssertionError( + 'Missing destination files: {}\nlocal_paths: {}'.format( + missing, local_paths)) + + def _reactor_callback(self, received, total): + self.received = received + self.total = total + self._do_callback() + + @staticmethod + def _set_gsm(iface, *, allow_gsm): + # This is a separate method for easier testing via mocks. + iface.allowGSMDownload(allow_gsm) + + def cancel(self): + """Cancel any current downloads.""" + if self._iface is None: + # Since there's no download in progress right now, there's nothing + # to cancel. Setting this flag queues the cancel signal once the + # reactor starts running again. Yes, this is a bit weird, but if + # we don't do it this way, the caller will immediately get a + # Canceled exception, which isn't helpful because it's expecting + # one when the next download begins. + super().cancel() + else: + self._iface.cancel() + + def pause(self): + """Pause the download, but only if one is in progress.""" + if self._iface is not None: # pragma: no branch + self._iface.pause() + + def resume(self): + """Resume the download, but only if one is in progress.""" + if self._iface is not None: # pragma: no branch + self._iface.resume() === modified file 'systemimage/version.txt' --- systemimage/version.txt 2014-09-26 14:36:34 +0000 +++ systemimage/version.txt 2015-05-20 14:55:53 +0000 @@ -1,1 +1,1 @@ -2.5 +3.0 === modified file 'tools/demo.ini' --- tools/demo.ini 2014-01-30 15:41:03 +0000 +++ tools/demo.ini 2015-05-20 14:55:53 +0000 @@ -8,7 +8,6 @@ [system] channel: daily -build_file: /tmp/system-image/etc/ubuntu-build tempdir: /tmp logfile: /tmp/system-image/logs/client.log loglevel: info === added file 'tools/runme.sh' --- tools/runme.sh 1970-01-01 00:00:00 +0000 +++ tools/runme.sh 2015-05-20 14:55:53 +0000 @@ -0,0 +1,10 @@ +where=udm/build +root=$HOME/projects/phone/${where}/src/downloads/daemon +logfile=$HOME/.cache/ubuntu-download-manager/ubuntu-download-manager.INFO +# export GLOG_logtostderr=1 +# export GLOG_v=100 +echo -n `date --rfc-3339=ns` >> ${logfile} +echo -n " " >> ${logfile} +echo $* >> ${logfile} +#exec env -u DBUS_SESSION_BUS_ADDRESS ${root}/ubuntu-download-manager $* +exec ${root}/ubuntu-download-manager $* === modified file 'tox.ini' --- tox.ini 2014-09-17 13:41:31 +0000 +++ tox.ini 2015-05-20 14:55:53 +0000 @@ -1,22 +1,27 @@ [tox] -envlist = py34 -recreate=True +envlist = {py34,coverage}-{udm,curl} +recreate = True + +[coverage] +rcfile = {toxinidir}/{envname}.ini +rc = --rcfile={[coverage]rcfile} +dir = --directory={envname} [testenv] -commands = python -m nose2 -v +commands = + py34: python -m nose2 -v + coverage: python /usr/bin/python3-coverage run {[coverage]rc} -m nose2 -v + coverage: python3-coverage combine {[coverage]rc} + coverage: python3-coverage html {[coverage]rc} {[coverage]dir} sitepackages = True +indexserver = + default = http://missing.example.com +usedevelop = True +whitelist_externals = python3-coverage setenv = SYSTEMIMAGE_REACTOR_TIMEOUT=60 - -[testenv:coverage] -basepython = python3 -commands = - python /usr/bin/python3-coverage run --rcfile={toxinidir}/coverage.ini -m nose2 -v - python3-coverage combine --rcfile={toxinidir}/coverage.ini - python3-coverage html --rcfile={toxinidir}/coverage.ini -sitepackages = True -setenv = - SYSTEMIMAGE_REACTOR_TIMEOUT=120 - COVERAGE_PROCESS_START={toxinidir}/coverage.ini - COVERAGE_OPTIONS="-p" - COVERAGE_FILE={toxinidir}/.coverage + coverage: COVERAGE_PROCESS_START={[coverage]rcfile} + coverage: COVERAGE_OPTIONS="-p" + coverage: COVERAGE_FILE={toxinidir}/.coverage + udm: SYSTEMIMAGE_PYCURL=0 + curl: SYSTEMIMAGE_PYCURL=1