gmq package


gmq.cli module

Gateway Message Queue (GMQ)

GMQ is a localhost HTTP server for use as a general purpose request queue for data intended for Exosite Murano Products and Solutions. The primary use cases for GMQ is for lossy cellular and intermittent WiFi networks, but can be used as a store-and-forward queue for outgoing data.

gmq gmq [-v|–version] gmq [-g|–cfg-path] gmq [-l|–log-cfg-path] gmq (cfg|log) set <section> <option> <value> gmq (cfg|log) [ get <section> <option> ] gmq reread
cfg Option for adjusting real-time behavior of the gmq server. log Option for adjusting the logging behavior of the gmq server.
-h –help Show this screen. -v –version Show version. -g –cfg-path Print the path to the gmq configuration file. -l –log-cfg-path Print the path to the gmq logging configuration file. cfg Print the contents of the gmq configuration. cfg get Print a list of the gmq configuration sections. cfg get <section> Print a list of the configuration options for <section>. cfg get <section> <option> Print the configuration option. cfg set <section> <option> <value> Set the configuration option. log Print the contents of the gmq logging configuration. log get Print a list of the logging configuration sections. log get <section> Print a list of the logging configuration options for <section>. log get <section> <option> Print the logging configuration option. log set <section> <option> <value> Set the logging configuration option.

Changes in gmq config file comes into effect after execution of this.


Entry point for Gateway Engine process.

gmq.db module

class gmq.db.DATA(vms, dataport, data, method, timestamp=None)[source]
class gmq.db.DB[source]

Bases: object

Base class for all databases

classmethod query_callback(*args)[source]

Callback for every query made to the database.

class gmq.db.GMQ_DB[source]

Bases: gmq.db.DB, object

The GMQ database is the identity map for all Vendor, Model, Serial entries and their CIKs.


Inserts a single VMS() entry into the database.


Removes a single VMS() entry from the database.


Fetch and return all rows from the database.


Select and return the VMS.CIK entry in the database.


Updates the VMS.CIK member in the database.

class gmq.db.VMS_DB(vms, create=True)[source]

Bases: gmq.db.DB, object

The VMS database is where the ‘queued’ data ends up on a failed write attempt or a deliberate record request.

All data to be recorded into a 1P client is put in the ‘record’ table of this database.

The VMS database checks the available free space left on the host disk to determine whether or not old ‘record’ entries should be expired.


AssertionError if None is any of VENDOR, MODEL or SERIAL.

The process of ‘complete’ing records is to delete thier entries from the db.


The process of marking an entry as ‘conflicted’ is (at the moment) simply to log it and remove it from the record database.

TODO: Is this the right approach for conflicting records?


Method for deleting the database file.


The process of expiring records is to simply delete the oldest ‘qty’ records.


Clear CIK flag by setting to 0.

Due to the multithreaded nature of GMQ, there has to be a safe way to signal the main thread the handles activations and manages ciks that the cik in use is bad and should be removed from the GMQ_DB().

Update the ‘cik’ table with an integer-boolean that signifies a 401 response from 1P as a result from a ‘record’ request.

  • CIK Good: 1
  • CIK Bad: 0

Return list of dataports from the record table.

get_records(dataport, num_rows)[source]

Return num_rows Rows from the record table.


Query the cik table to see if the flag is set to 0.

False if flag is 0. True if flag is 1.
record(dataport, data)[source]

Cache data in db to write into 1P. Check available free disk space to determine if old records need to be expired to make room for latest data.


Return the current row count from the record table.


The ‘id’ column is set to AUTOINCREMENT. Though it is unlikely to overflow, this method is provided so the AUTOINCREMENT number can be reset given all records have been deleted from the ‘record’ table.


Set CIK flag by setting to 1.

Due to the multithreaded nature of GMQ, there has to be a safe way to signal the main thread the handles activations and manages ciks that the cik in use is bad and should be removed from the GMQ_DB().

Update the ‘cik’ table with an integer-boolean that signifies a 401 response from 1P as a result from a ‘record’ request.

  • CIK Good: 1
  • CIK Bad: 0

gmq.logger module

gmq.logger.setup_logging(default_path='/var/lib/jenkins/shiningpanda/jobs/fa78b320/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/gmq-1.2-py2.7.egg/gmq/logging.cfg', default_level=20, env_key='GMQ_LOG_CFG')[source]

Setup logging configuration

gmq.q module

class gmq.q.Q(q)[source]

Bases: threading.Thread, object

class gmq.q.VMS_Activator(gmq_db, vms)[source]

Bases: exo.api.ExositeAPI, object

Class to handle all cik management in the GMQ database.

gmq.server module

Gateway Message Queue, or GMQ, is a standalone HTTP server on localhost:8090 that attempts to behave like Exosite One Platform with a Gateway-centric design philosophy that targets the IoT Gateway as the main application of this program.

Designer’s Note on twisted implementation:
It is more common in the twisted framework to unpack POST data with the request.args member when using Content-Type of application/x-www-form-urlencoded. However, this doesn’t seem allow ‘;’, ‘+’ and other characters.
class gmq.server.Activate[source]

Bases: twisted.web.resource.Resource, object

Endpoint for getting and requesting an activation for a VMS client.

curl -XPOST http://localhost:8090/activate -H ‘Content-Type: application/x-www-form-urlencoded; charset=utf-8’ -d ‘vendor=gateway-engine-demo&model=GatewayEngine_V1&sn=00:00:00:00:00:02’

class gmq.server.RecordAPI[source]

Bases: twisted.web.resource.Resource, object

Class for implementing the /onep:v1/stack/record endpoint.
POST /onep:v1/stack/record HTTP/1.1 Host: X-Exosite-CIK: <CIK> X-Exosite-PID: <PROJECT_ID> Content-Type: application/x-www-form-urlencoded; charset=utf-8 Content-Length: <length> <blank line> alias=<alias 1>&<timestamp 1>=<value 1>&<timestamp 2>=<value 2>&alias=<alias 2>&<timestamp 3>=<value 3>&<timestamp 4>=<value 4>
class gmq.server.RereadConfigs[source]

Bases: twisted.web.resource.Resource, object

Resource that forces gmq to re-read gmq.cfg and logging.cfg, applying these changes during run-time. This makes it so you can tune gmq and logging while it’s running so restarts aren’t necessary for config changes (in most cases).

curl -XPOST http://localhost:8090/reread

class gmq.server.WriteAPI[source]

Bases: twisted.web.resource.Resource

Main resource for local GMQ resource. The render_POST method IS the HTTP Write API.

curl -XPOST http://localhost:8090/onep:v1/stack/alias -H ‘X-Exosite-VMS: gateway-engine-demo GatewayEngine_V1 00:00:00:00:00:02’ -H ‘Content-Type: application/x-www-form-urlencoded; charset=utf-8’ -d “device_info=I wrote this data.”

render_POST(*args, **kw)

Populates VMS object based on request headers.

Returns a tuple of (VMS(), ErrMsg).

gmq.server.re_read_cfg(*args, **kwargs)[source]

gmq.unq module

class gmq.unq.Unq(gmq_db)[source]

Bases: threading.Thread, object

Parent thread that spawns VMS threads that record entries in VMS databases.

class gmq.unq.VMS_Thread(vms)[source]

Bases: threading.Thread, object


gmq.utils module

class gmq.utils.VMS(vendor, model, serial, cik='', certfile=None, keyfile=None, certfile_reprovision=None, keyfile_reprovision=None)[source]

Base identity class for all databases.


Module contents