Over the Air Updates (OTAU)

Performing Over the Air Updates to gateways is the core function of Gateway Engine and, more specifically, the gwe process.

OTAU State Machine

Once gwe is started and initialized, it’s default behavior is to execute a continuous loop whereby it checks for commands issued to it via the engine_fetch resource. If an OTAU command is detected, it operates on the command according to a specific state machine. Below is a list of states gwe can occupy during an OTAU.

States

GWE OTAU States.
State Description  
init Begin processing new OTAU.  
test Determine if OTAU can be processed.  
test_error Not enough space on filesystem.  
test_success Proceeding with download and installation.  
download Begin download of tarball.  
download_success Download of tarball complete.  
download_error Unable to download tarball.  
install Begin installation of OTAU.  
install_success Installation complete.  
install_error Installer exited with non-zero code.  

State Object Format

GWE OTAU State Object Format.
State Object Format
init {<tarball>: {info: <message>,state: init,version: <ver>, name: my_custom_app}}
test {<tarball>: {info: <message>,name: my_custom_app,state: test,version: <ver>, content_available: true,size: <bytes>}}
test_error {<tarball>: {info: <message>, name: my_custom_app, state: test_error, version: <ver>, content_available: true, size: <bytes>}}
test_success {<tarball>: {info: <message>, name: my_custom_app, state: test_success, version: <ver>, content_available: true, size: <bytes>}}
download {<tarball>: {info: <message>, name: my_custom_app, state: download, version: <ver>, content_available: true, size: <bytes>}}
download_success {<tarball>: { info: <message>, name: my_custom_app, install_path: /tmp/<tarball>, state: download_success, version: <ver>, content_available: true, size: <bytes>}}
download_error {<tarball>: {info: <message>, name: my_custom_app, install_path: /tmp/<tarball>, state: download_error, version: <ver>, content_available: true, size: <bytes>}}
install {<tarball>: { info: <message>, name: my_custom_app, install_path: /tmp/<tarball>, state: install, version: <ver>, content_available: true, size: <bytes>}}
install_success {<tarball>: { info: <message>, name: ota_bundle_test, install_path: /tmp/<tarball>, state: install_success, version: <ver>, content_available: false, size: <bytes>}}
install_error {<tarball>: {info: <message>, name: ota_bundle_test, install_path: /tmp/<tarball>, state: install_error, version: <ver>, content_available: false, size: <bytes>}}

State Diagram

To help you understand how Gateway Engine’s OTAU feature works, the diagram below was put together to show the sequence that results in a Gateway Engine Over the Air Update.

!include otau_state_machine.uml

In words, this diagram states that Gateway Engine will check for the engine_fetch alias for specially formatted JSON objects that contain instructions for new Custom Gateway Applications or updates to existing ones to be downloaded and installed. If there’s nothing to download and install it will sleep for the amount of seconds configured in the update_interval alias.

Note

The default value update_interval that GatewayEngine uses when it starts for the 1st time is 43200 seconds. This comes out to 12 hours (twice/day). If you want to shorten this, write the amount of seconds you want GatewayEngine to sleep before it checks in again in the update_interval alias. Once it reads this new value, it will “clear” the alias (i.e. write an empty string).

The update_interval can be changed before gwe ever starts with the following command:

gwe --set-update-interval <interval>

Adding commands like this to Gateway Engine release tarballs in the gateway-engine/init/post_install_commands.sh file is a handy way to configure Gateway Engine for your specific environment when prepping for a manufacturing order of bulk gateways.

If gwe is already running, the commands are:

gwe --set-update-interval <interval>
supervisorctl restart gwe

When changing the update_interval remotely, gwe will fetch the contents of its update_interval resource the next time it wakes up from sleep and not require a restart:

murano device write <id> update_interval=<interval>

During the development of you Custom Gateway Application, if you’re not on a cellular network or otherwise aren’t concerned with network bandwidth usage, use a really fast update_interval. You can set the update_interval to as low as one second. Doing so will ensure that as soon as you have updates to deploy to the gateway, they will be installed immediately.

OTAU Deployments

The goal of this section is to enable system maintainers to execute successful OTAU deployments. Deployments to remote devices are risky since failing to do so successfully can result in unexpected data loss and device failure.

These deployments can be difficult to manage when the number of gateways deployed in an IoT environment begins to grow. A common problem that maintainers encounter is that some problems only occur in specific environments. For instance, if a Custom Gateway Application was developed using Wifi and is later implemented on a Cellular network it is the case that the bandwidth consumption of the application was underestimated. Other cases involve unexpected behavior that only occur at a single remote location or a small subset.

For reasons like these, the environment present at the initial release of the Production system is monolithic and then evolves to heterogeneous. While it is both useful and expected that systems are well documented it is often unlikely and unscalable to expect the documentation to be current to a continuously changing enviroment.

This section contains methods and best-practices that fleet maintainers/admins can employ to for successful OTAU rollouts.

  1. Determine Components Being Upgraded
  2. Identify Affected Gateways In Fleet
  3. Requisition Suitable Test Environtment
  4. Deploy OTAU In Test Environment
  5. Determine Success
  6. Deploy OTAU To Affected Gateways
  7. Keeping Track Of Versions

Prod Environment

The production environment is the Murano Product(s) that contain the actual business hardware. Keep this separate from test devices.

Dev Environment

It is highly recommended that maintainers/admins of gateway fleets retain a testing or staging environment with which to test their OTAUs before deploying them to units in their production/customer-facing solutions. This is especially true when performing OTAUs of Gateway Engine itself.

During the lifetime of a typical IoT solution things like hardware and software can change. For this reason it is important to have on-hand all of the versions of hardware deployed, or that are about to be deployed in the production system.

Test Plan

For a given hardware version of a gateway, make a plan with steps to do the following before a deploying OTAUs to production gateways.

  1. Recreate Production Environment
  • Obtain copy of the gateway image (e.g. filesystem and bootloader) and flash the gateway.
  • Install and configure the version of Gateway Engine running on the target device in your production environment (if this isn’t already done on the image build).
  • Connect all hardware and/or start simulation software needed by your Custom Gateway Application in order to function normally.
  1. Install the Custom Gateway Application

    Upload the OTAU.
    murano content upload /path/to/the_application.vX.tar.gz
    
    Apply the OTAU command to a specific gateway.
    murano content upload /path/to/the_application.vX.tar.gz
    murano device write 02:42:AC:11:00:04 engine_fetch='{"install": [{"name": "the_application.vX.tar.gz"}]}'
    
    OPTIONAL: List all gateway identities in the Murano Product.
    murano device list --json | jq -r '.devices[] | .identity'
    
    OPTIONAL: Use a bash for-loop to iterate over a set of gateways.
    for uuid in 00:00:08:4F:AB:0C 00:00:08:4F:AB:0D 00:00:08:4F:AB:0E ... ; do
        murano device write $uuid engine_fetch='{"install": [{"name": "the_application.vX.tar.gz"}]}'
    done
    
    If the app is already copied to the gateway, use gwe CLI to install the update.
    gwe --install-apps /root/the_application.vX.tar.gz
    
  2. Monitor the progress of the OTAU

    Read the entire OTAU state object.
    murano device read 02:42:AC:11:00:02 fetch_status
    
    Pretty-print the OTAU state object.
    murano device read 02:42:AC:11:00:02 fetch_status --json | jq -r .fetch_status.reported | jq .
    
    Drilling down further into the OTAU state object.
    murano device read 02:42:AC:11:00:02 fetch_status --json | jq -r .fetch_status.reported | jq -r '."the_application.vX.1.tar.gz".state'
    
  3. Verify CGA Behavior

Once the OTAU completes successfully, verify that the gateway processes are behaving normally by inspecting the gwe resources as well as any custom data resources the CGA reports to (e.g. raw_data, gateway_data, temperature, etc.).

Read the entire engine_report object.
murano device read 02:42:AC:11:00:02 engine_report
Pretty-print the engine_report object.
murano device read 02:42:AC:11:00:02 engine_report --json | jq -r   .engine_report.reported | jq .

Bundled OTAUs

In many cases, you want to update Gateway Engine as well as you custom application. Since Gateway Engine version 1.5.24, the bundling option is available. This option gives gateway fleet maintainers the option of bundling custom applications and libraries along with a Gateway Engine update package. I order to do this, make sure you Get the GWE Development Tools and then use the gwe CLI to compose the desired bundle.

  1. Determine the Release Area from which to get the Gateway Engine update package. It is recommended that you use the gmq-master/otau release area. If your gateway do not use gmq, then it is recommended that you use baseline-master/otau.

  2. Determine the path(s) to your custom application update tarballs (i.e. /home/me/sandbox/example/my_app.v23.tar.gz).

  3. Run the bundling command:

    Bundling command example.
    gwe --bundle gmq-master/otau --app_to_bundle /home/me/sandbox/example/my_app.v23.tar.gz
    
  4. Test the bundle against your development fleet (Test Plan).

Below is an example of bunding a test application.

An example bundling command
$ gwe --bundle gmq-master/otau --app_to_bundle /home/will/test/example.v23.tar.gz
/home/will/test
2018-06-04 12:23:04,881-INFO-GWE.tball:bundle:403 ::> Creating temporary staging dir for gwe bundle: /tmp/gwe-bundle-tmp
2018-06-04 12:23:04,910-WARNING-GWE.tball:bundle:408 ::> Removing gwe.v1.6.6.tar.gz.
2018-06-04 12:23:04,911-INFO-GWE.tball:bundle:410 ::> Contents of pre-bundled OTAU tarball:
2018-06-04 12:23:04,911-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/install.sh
2018-06-04 12:23:04,911-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/setup.py
2018-06-04 12:23:04,911-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/otau.py
2018-06-04 12:23:04,911-INFO-GWE.tball:bundle_walk:394 ::> /tmp/gwe-bundle-tmp/deps
2018-06-04 12:23:04,911-INFO-GWE.tball:bundle_walk:394 ::> /tmp/gwe-bundle-tmp/apps_to_install
2018-06-04 12:23:04,912-INFO-GWE.tball:bundle_walk:394 ::> /tmp/gwe-bundle-tmp/test
2018-06-04 12:23:04,912-INFO-GWE.tball:bundle_walk:394 ::> /tmp/gwe-bundle-tmp/init
2018-06-04 12:23:04,912-INFO-GWE.tball:bundle_walk:394 ::> /tmp/gwe-bundle-tmp/GatewayEngine
2018-06-04 12:23:04,912-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/deps/acp.v4.tar.gz
2018-06-04 12:23:04,912-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/deps/device-client.v1.5.9.tar.gz
2018-06-04 12:23:04,912-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/apps_to_install/gmq.v1.3.8.tar.gz
2018-06-04 12:23:04,912-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/test/self_test.py
2018-06-04 12:23:04,912-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/test/__init__.py
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/init/gwe.conf
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/init/supervisord.conf
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/installer.py
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/__version__.py
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/tarball.py
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/constants.py
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/Gateway.cfg
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/Engine.config
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/cli.py
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/utils.py
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/README.md
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/__init__.py
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/GatewayEngine.py
2018-06-04 12:23:04,913-INFO-GWE.tball:bundle:414 ::> Copying /home/will/test/example.v23.tar.gz to /tmp/gwe-bundle-tmp/apps_to_install
2018-06-04 12:23:04,914-INFO-GWE.tball:bundle:416 ::> Bundling /tmp/gwe-bundle-tmp
2018-06-04 12:23:04,914-INFO-GWE.tball:bundle:417 ::> Contents of bundle:
2018-06-04 12:23:04,914-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/install.sh
2018-06-04 12:23:04,915-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/setup.py
2018-06-04 12:23:04,916-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/otau.py
2018-06-04 12:23:04,916-INFO-GWE.tball:bundle_walk:394 ::> /tmp/gwe-bundle-tmp/deps
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:394 ::> /tmp/gwe-bundle-tmp/apps_to_install
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:394 ::> /tmp/gwe-bundle-tmp/test
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:394 ::> /tmp/gwe-bundle-tmp/init
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:394 ::> /tmp/gwe-bundle-tmp/GatewayEngine
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/deps/acp.v4.tar.gz
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/deps/device-client.v1.5.9.tar.gz
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/apps_to_install/example.v23.tar.gz
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/apps_to_install/gmq.v1.3.8.tar.gz
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/test/self_test.py
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/test/__init__.py
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/init/gwe.conf
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/init/supervisord.conf
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/installer.py
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/__version__.py
2018-06-04 12:23:04,917-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/tarball.py
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/constants.py
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/Gateway.cfg
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/Engine.config
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/cli.py
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/utils.py
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/README.md
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/__init__.py
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle_walk:390 ::> /tmp/gwe-bundle-tmp/GatewayEngine/GatewayEngine.py
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle:422 ::> Bundle difference: set(['/tmp/gwe-bundle-tmp/apps_to_install/example.v23.tar.gz'])
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle:426 ::> Creating bundle with files: ['/tmp/gwe-bundle-tmp/deps', '/tmp/gwe-bundle-tmp/install.sh', '/tmp/gwe-bundle-tmp/apps_to_install', '/tmp/gwe-bundle-tmp/setup.py', '/tmp/gwe-bundle-tmp/otau.py', '/tmp/gwe-bundle-tmp/test', '/tmp/gwe-bundle-tmp/init', '/tmp/gwe-bundle-tmp/GatewayEngine']
2018-06-04 12:23:04,918-INFO-GWE.tball:bundle:430 ::> Adding /tmp/gwe-bundle-tmp/deps as deps
2018-06-04 12:23:04,949-INFO-GWE.tball:bundle:430 ::> Adding /tmp/gwe-bundle-tmp/install.sh as install.sh
2018-06-04 12:23:04,949-INFO-GWE.tball:bundle:430 ::> Adding /tmp/gwe-bundle-tmp/apps_to_install as apps_to_install
2018-06-04 12:23:04,951-INFO-GWE.tball:bundle:430 ::> Adding /tmp/gwe-bundle-tmp/setup.py as setup.py
2018-06-04 12:23:04,951-INFO-GWE.tball:bundle:430 ::> Adding /tmp/gwe-bundle-tmp/otau.py as otau.py
2018-06-04 12:23:04,951-INFO-GWE.tball:bundle:430 ::> Adding /tmp/gwe-bundle-tmp/test as test
2018-06-04 12:23:04,952-INFO-GWE.tball:bundle:430 ::> Adding /tmp/gwe-bundle-tmp/init as init
2018-06-04 12:23:04,953-INFO-GWE.tball:bundle:430 ::> Adding /tmp/gwe-bundle-tmp/GatewayEngine as GatewayEngine
2018-06-04 12:23:04,986-INFO-GWE.tball:bundle:435 ::> Bundle success: True