GatewayEngine package

Submodules

GatewayEngine.GatewayEngine module

class GatewayEngine.GatewayEngine.DisableLogger[source]

Class used for disabling logging for specific context.

class GatewayEngine.GatewayEngine.Engine(engine_cfg_file, log_level, gateway_cfg_file='/var/lib/jenkins/etc/exosite/gwe/Gateway.cfg')[source]

Bases: object

Engine defines the main program loop for Gateway Engine.

apps()[source]

Protected member getter. List of dicts. Like this: [{‘counter’: 1}, {‘ntp_updater’: 1}]

apps_report()[source]

Returns a list of dictionaries of apps that contains runtime info from supervisord as well as version info from gwe.

dump_to_cfg(section, option, value)[source]

Utility to dump configuration data to the instance cfg file.

get_supervisor_apps_shortlist(supervisor_apps)[source]

Parses supervisor.getAllProcessInfo() and returns a dictionary.

installation_summary(status, test, info)[source]

A JSON object containing the summary of the installation status of a given OTAU.

loop(once)[source]
set_apps(apps)[source]

Protected member setter. Sets instance member and cfg file.

set_update_interval(interval)[source]

Protected member setter. Sets instance member and cfg file.

set_user_agent(user_agent)[source]

Protected member setter. Sets instance member and cfg file.

start(once=False)[source]

Main function for GatewayEngine.py

supervisor_add_process_group(group)[source]

Method to add a process group (new application) to supervisord control.

supervisor_get_all_process_info()[source]

Connect to supervisord and return the getAllProcessInfo() data.

If supervisor is not installed and/or not running, a message will be logged and an empty list returned.

supervisor_get_app_state(app)[source]

Method that returns the ‘statename’ of the app in question.

supervisor_reload_config()[source]

Method for making supervisor reload its config files.

supervisor_restart()[source]

Method for restarting supervisord. Use with extreme caution as this method will result in gwe restarting as well.

supervisor_start_app(app)[source]

Method to start a supervisor-controlled application. Uses the XML RPC connection to supervisor to call startProcess ‘max_supervisor_rpc_attempts’ or until process is in one of the following states:

  • ‘STARTING’
  • ‘RUNNING’

For more information on the supervisor states your process can occupy, see:

supervisor_stop_app(app)[source]

Method to stop a supervisor-controlled application. Uses the XML RPC connection to supervisor to call stopProcess ‘max_supervisor_rpc_attempts’ or until process is in one of the following states:

  • ‘STOPPED’
  • ‘EXITED’
  • ‘BACKOFF’
  • ‘FATAL’

The ‘STOPPING’ state is not included in this list because a given process can specify their ‘stopwaitsecs’ in thier supervisor.conf files and should not block gwe from its other tasks. Additionally, if a process is in ‘STOPPING’ state, further TERM signals would be redundant.

For more information on the supervisor states your process can occupy, see:

update_interval()[source]

Protected member getter.

update_ota_state(state)[source]
update_self_from_cfg()[source]

Updates Device instance with values from cfg.

user_agent()[source]

Protected member getter.

class GatewayEngine.GatewayEngine.Gateway(cfg_file, log_level)[source]

Bases: exo.device.Device

Object containing Gateway-specific variables and methods.

Gateway doesn’t require any configuration file settings other than the ones needed by Device().

activate()[source]

This function runs until a successful activate call is made.

device_info()[source]

Sends device information to device_info dataport.

Contents include:
  • uname -a output for kernel and os info
  • ip addresses for gdc and gwe configured interfaces
  • df output for current disk space usage
  • free output for memory usage
exo_fetch_new_apps_list()[source]

Returns a list of tarball names to look for in content area. After successfully reading the engine_fetch dataport, we clear it.

fetch dataport format and examples:
  • {“install”: [{“name”: “new_app.v1.tar.gz”},

    {“name”: “newer_app.v3.tar.gz”}]}

  • {“install”: [{“name”: “test_app2.v3.tar.gz”}]}

exo_get_update_interval(device)[source]

Fetches new update interval from update_interval dataport, then clears the dataport. If we’re unable to retrieve any new values or if it isn’t a valid value, just return the old one.

exo_report(apps_report)[source]

Exosite communication function that updates dataports with report data.

getSerialNumber()[source]

Get Serial Number of Gateway.

If the Gateway.cfg file has the ‘uuid’ option set, the value provided will be used as the serial number for the gateway.

If the ‘iface’ option is set, the serial number of the gateway will be the MAC address of the hardware interface provided (i.e. eth0, en0, ppp0, etc.).

If neither ‘uuid’ or ‘iface’ options are set in the Gateway.cfg file, GWE will not be able to run because it will have no serial number with which to provision itself with Exosite.

get_and_parse_new_apps_to_name_version_dict()[source]

Calls self.exo_fetch_new_apps_list() and parses the tarball names into a python dict containing

fetch dataport format:
{‘tarball_name’: {‘name’: <name>, ‘version’: <version>}}
static get_mac_address(ifname)[source]

Get MAC address from OS.

gwe_debug_upload(debug_info)[source]

Sends debug information to ‘gwe_debug’ dataport.

iface()[source]

Get the hardware internet interface name from the Gateway.cfg file.

ip_addr(iface)[source]

Returns a string that represents the ip address in octet form of the iface parameter given.

Example:

$ print(ip_addr(‘ppp0’)) ‘142.54.82.147’

$ print(ip_addr(‘eth0’)) ‘192.168.0.50’

static network_connected(host='8.8.8.8', port=53, timeout=3)[source]

Host: 8.8.8.8 (google-public-dns-a.google.com) OpenPort: 53/tcp Service: domain (DNS/TCP)

run_cmd(cmd)[source]

Run a shell command and return its output.

If raised, catches OSError and returns the error message.

Examples:
free_space = self.run_cmd(‘free’) ifconfig = self.run_cmd(‘ifconfig ‘+self.iface())
usage_report()[source]

Gets the usage report from

Device.interfaces.interface_report()

and JSONifies it.

class GatewayEngine.GatewayEngine.TimeoutServerProxy(uri, timeout=60, transport=None, encoding=None, verbose=0, allow_none=0, use_datetime=0)[source]

Bases: xmlrpclib.ServerProxy

An override class for xmlrpclib.Server that allows for the same type of connection to the Supervisord XML RPC Server but you can specify important override parameters like timeout.

class GatewayEngine.GatewayEngine.TimeoutTransport(timeout, use_datetime=0)[source]

Bases: xmlrpclib.Transport

An override class that provides a supervisord a transport you can specify the timeout for in the xml rpc connection.

make_connection(host)[source]

GatewayEngine.cli module

GatewayEngine.cli.main()[source]

Entry point for Gateway Engine process.

GatewayEngine.cli.parse_args(args)[source]

GatewayEngine.constants module

Separate some logging attributes for global imports.

class GatewayEngine.constants.EngineCfgOpts[source]

Enum for cfg file options

UpdateInterval = 'update_interval'
UserAgent = 'user_agent'
class GatewayEngine.constants.EngineCfgSects[source]

Enum for cfg file sections.

Apps = 'apps'
Device = 'device'
class GatewayEngine.constants.GatewayCfgOpts[source]

Enum for cfg file options

CaptureInstallerStdout = 'capture_installer_stdout'
DeviceInfo = 'device_info'
EngineFetch = 'engine_fetch'
EngineReport = 'engine_report'
ExtractPath = 'extract_path'
FetchStatus = 'fetch_status'
GWEDebug = 'gwe_debug'
InstallerTimeoutMinutes = 'installer_timeout_minutes'
MaxSupervisorRpcAttempts = 'max_supervisor_rpc_attempts'
SupervisorRpcConnectionTimeout = 'supervisor_rpc_connection_timeout'
UpdateInterval = 'update_interval'
UsageReport = 'usage_report'
class GatewayEngine.constants.GatewayCfgSects[source]

Enum for cfg file sections.

Dataports = 'dataports'
Gwe = 'gwe'

GatewayEngine.installer module

class GatewayEngine.installer.Installer(path_to_app, engine_cfg, **kwargs)[source]

Bases: object

Class for installing a single application either from a GatewayEngine base install, or from new content retrieved from vendor content area.

For installing multiple applications, put this in a for-loop.

create_supervisord_conf_file(app_path, app_name)[source]

With a the path to the newly extracted app and its name as inputs, create the supervisor.d conf file from the sections within.

dir_name_and_version()[source]

If we’re installing from a directory, get the name and version from the directory name.

exec_install_dot_sh()[source]

This function executes the new application’s installer.

Supported installers are ‘install.sh’ and ‘setup.py’ files.

Any non-zero numeric return code indicates a problem with installing the application.

Returns:
  • retval: a tuple of (errorcode, description)
NUMERIC_CODES:

8 - Please check to make sure the ‘shebang’ is present in install.sh. 7 - Unused. 6 - Unused. 5 - Unused. 4 - Unused. 3 - Unused.

2 - Starting installation, setting up environment for installation. Receiving
this error code indicates that something went wrong during the setup steps during the installation.
1 - Executing installer. Receiving this error code indicates that something
went wrong before the installation took place.

0 - Installer returns this value to indicate successful script execution. -1 - Supported installer not found. Either ‘install.sh’ or ‘setup.py’ not found. -2 - Can’t execute install.sh script. Execute permissions not set. -3 - Caught general exception during installer execution. See gwe log for details. -4 - Supported installer not found. Either ‘install.sh’ or ‘setup.py’ not found. -5 - Unused. -6 - Unused. -7 - Unused. -8 - Unused. -9 - Unused. -10 - Unused. -11 - Unused. -12 - Unused. -13 - Unused. -14 - Unused.

-15 - Installer was aborted due to exceeding installer_timeout_minutes GWE config
setting.
install()[source]

Run installation steps for new app. New app can be a directory (usually a base install), or a tarball (usually installing from OneP.

update_engine_config()[source]

Add new application name and version to Engine.config.

class GatewayEngine.installer.SupervisorDefault[source]

Enum for supervisor config file OPTIONS defaults.

Opts = {'command': 'command', 'redirect_stderr': 'true', 'stdout_logfile': '/var/log', 'stdout_logfile_backups': '1', 'stdout_logfile_maxbytes': '200KB'}
GatewayEngine.installer.main()[source]

GatewayEngine.tarball module

Module for handling tarfile operations.

class GatewayEngine.tarball.ExistingTarball(path, **kwargs)[source]

Bases: GatewayEngine.tarball.Tarball

Class for dealing with existing tarfiles.

class GatewayEngine.tarball.Namespace(**kw)[source]

Bases: object

This is a helper class that allows ‘.’ access to dictionary members instead of dict[‘member’] notation.

class GatewayEngine.tarball.NewTarball(tarball_fname, response, **kwargs)[source]

Bases: GatewayEngine.tarball.Tarball

Class for creating new tarfiles.

create_tarball_from_build_file()[source]
Inputs:
  • self.tarball_fname: is not used directy. It is overridden

    by the desired tarball name defined by JSON input file in self.response.

  • self.response: is the path to a json file that defines

    a valid gwe app tarball we can create.

Output:
  • ‘/pwd/<app.vN.tar.gz>’
create_tarball_from_download()[source]
Inputs:
None
Output:
  • ‘self.extract_path<app.vN.tar.gz>’
class GatewayEngine.tarball.OTAUBundleTarball(path, **kwargs)[source]

Bases: GatewayEngine.tarball.ExistingTarball

Methods and members for Gateway Engine OTAU tarballs (e.g. gwe.v1.5.23.tar.gz).

bundle(apps_path_list)[source]

Creates tmp directory in self.extract_path, unpacks the self.path to the tmp directory, adds apps in apps_path_list to apps_to_install/ and then re-tar’s everything to create the bundle.

extractmember(member)[source]

Extracts stuff from tarfiles.

unbundle()[source]

Extracts all apps in ‘.*apps_to_install/.*’, removes them from the bundle, then recreates the bundle.

exception GatewayEngine.tarball.SectionInvalid[source]

Bases: exceptions.Exception

exception GatewayEngine.tarball.SectionMissing[source]

Bases: exceptions.Exception

class GatewayEngine.tarball.Tarball(path, **kwargs)[source]

Bases: object

Class containing methods and variables for dealing with tarfiles.

extract()[source]

Extract a tarball and return the path to its extract directory.

get_name_and_version()[source]

Populates self.name and self.version based on self.path.

getfile(file)[source]
getmember(member)[source]

If member exists in tarball, return a file-like object representing the member.

grepmemberpath(member_regex, one=True)[source]

Returns the path in the tarfile to the member (regex supported) if it exists.

The order of the tarfile matters to which result is returned.

Optional parameter one (default: True), when set to False will return a list of paths to every member that matches the provided regex.

Example:
$ tar zcvf requests_check.v1.tar.gz install.sh requests_check.v1 install.sh requests_check.v1/ requests_check.v1/install.sh requests_check.v1/tmp/ requests_check.v1/tmp/install.sh $ python -c “from GatewayEngine.tarball import Tarball ; print(Tarball(‘requests_check.v1.tar.gz’).grepmemberpath(‘.*install.sh’))” install.sh $ tar zcvf requests_check.v1.tar.gz requests_check.v1 install.sh requests_check.v1/ requests_check.v1/install.sh requests_check.v1/tmp/ requests_check.v1/tmp/install.sh install.sh $ python -c “from GatewayEngine.tarball import Tarball ; print(Tarball(‘requests_check.v1.tar.gz’).grepmemberpath(‘.*install.sh’))” requests_check.v1/install.sh
has_install_dot_sh()[source]

Returns a boolean for determining if the tarball contains an ‘install.sh’ script.

install_dot_sh_executable()[source]

Returns a boolean for determining if the ‘install.sh’ script has the ‘x’ flag set.

is_tarball()[source]

Determines if a given file is a properly formatted tarball.

open(tarball)[source]

Wrapper function for tarfile.open().

If tarball is not readable, return None.

static parse_filename(basename)[source]

Verifies if a given filename satisfy <app_name>.v<#>.tar.gz format

remove()[source]

Get rid of the tarball.

remove_install_package()[source]

Method that removes the install package (i.e. the contents of the extracted tarball) from its temporary location.

GatewayEngine.utils module

Utility functions for GatewayEngine.

GatewayEngine.utils.check_build_file(path_to_json_file, verbose=False)[source]
GatewayEngine.utils.check_tarball(full_path_of_tarball, cleanup=False, verbose=False, show_results=True, cleanup_tmp=False)[source]

cleanup_tmp only removes the dir used to extract the file for testing

class GatewayEngine.utils.colors[source]

Enum class for console colors.

BOLD = '\x1b[1m'
ENDC = '\x1b[0m'
FAIL = '\x1b[91m'
HEADER = '\x1b[95m'
OKBLUE = '\x1b[94m'
OKGREEN = '\x1b[92m'
UNDERLINE = '\x1b[4m'
WARNING = '\x1b[93m'
GatewayEngine.utils.conf_dir()[source]

Returns the GWE configuration directory.

GatewayEngine.utils.create_build_file(**kwargs)[source]

Call this function with kwargs to avoid interactive mode. This function helps you create a gwe.build file for a Custom Gateway Application.

GatewayEngine.utils.display_test_results(test_results)[source]

Helper function to check_tarball function.

Takes the test_results object from check_tarball and pretty-prints the results to STDOUT.

GatewayEngine.utils.engine_conf()[source]

Returns the GWE apps database file path.

GatewayEngine.utils.gwe_cik()[source]

Returns the GWE cik.

GatewayEngine.utils.gwe_conf()[source]

Returns the GWE configuration file path.

GatewayEngine.utils.isFileNameValid(fileName)[source]

checks that a file meets the required format of myapp.v1.tar.gz regex “.+.vd+.tar.gz”

Returns False if there is any whitespace in the tarball name.

Module contents