QGIS Dev Meeting Essen 2014

Goals

Provide hooks to modify existing services by manipulating input and output
Provide hooks to extend QGIS server by adding new services
Lower the barrier to develop new services by using Python
Maintain deployment and configuration extensions as easy as possibile (GUI-based one-click install & deployment)
Prototyping of new services

Not goals

Replace the existing professional solutions available for heavy duty webservices

Possible applications

web clients configuration
authentication/authorization
new services (WPS etc.)
new output formats

Implementations

Two possibilities:

404 handler - Simplest, hack-ish but unobtrusive and FAST! and KISS and DRY and [place your favourite pattern here!]
Observer - Full-featured but with a minimal overhead and will require an heavy refactoring of the existing request handling code

Standard flow

System Message: ERROR/3

index.rstline 46

Unknown directive type "graph".

.. graph:: images/qgis-server-stdflow.png
    :scale: 100%

    digraph g {

        #rankdir="LR"
        node [color=greenyellow, shape=box, style=filled, fontname="sans-serif"]
        "Core SERVICE?" [shape=hexagon]
        "Output to FCGI stdin"[color=greenyellow, shape=box, style="filled,rounded"]
        "Incoming request" [color=greenyellow, shape=box, style="filled,rounded"]

        "Incoming request" -> "Get the SERVICE"
        "Get the SERVICE" -> "Core SERVICE?"
        "Core SERVICE?" -> "Execute" [ label="yes" ]
        "Execute" -> "Output to FCGI stdin"
        "Core SERVICE?" -> "Exception" [ label="no" ]
        "Exception" -> "Output to FCGI stdin"

    }

404 handler

System Message: ERROR/3

index.rstline 71

Unknown directive type "graph".

.. graph:: images/qgis-server-404flow.png
    :scale: 100%

    digraph g {

        #rankdir="LR"
        node [color=greenyellow, shape=box, style=filled, fontname="sans-serif"]
        "Core SERVICE?" [shape=hexagon]
        "Plugin SERVICE?" [shape=hexagon color=yellow style=filled]
        "Output to FCGI stdin" [color=greenyellow, shape=box, style="filled,rounded"]
        "Incoming request" [color=greenyellow, shape=box, style="filled,rounded"]

        "Incoming request" -> "Get the SERVICE"
        "Get the SERVICE" -> "Core SERVICE?"
        "Core SERVICE?" -> "Execute core" [ label="yes" ]
        "Execute core" -> "Output to FCGI stdin"
        "Core SERVICE?" -> "Plugin SERVICE?" [ label="no" ]
        "Plugin SERVICE?" -> "Execute plugin" [ label="yes" ]
        "Plugin SERVICE?" -> "Exception" [ label="no" ]
        "Exception" -> "Output to FCGI stdin"
        "Execute plugin" -> "Output to FCGI stdin"

    }

404 pros & cons

Pros

unobtrusive: just three lines of codes added to main loop
CGI-style plugins
Minimal overhead when FCGI starts
0 (zero) overhead on core SERVICES: FAST!
re-use current Python Plugins loading mechanism
proof-of-concept simplest implementation done

Cons

no way to manipulate request and response generated from core services

Observer overview

Observer pattern: plugins will register themselves and listen to signals. Plugins will receive the request/response objects and they will be able to manipulate them:

at FCGI startup
after the request object is created
before the response is sent to the client
if there is no match with core services (404 handler)

Observer flowchart

System Message: ERROR/3

index.rstline 131

Unknown directive type "graph".

.. graph:: images/qgis-server-signals-slots.png
    :scale: 70%

    digraph g {

        rankdir="LR"

        node  [color=greenyellow, shape=box, style="filled,rounded", fontname="sans-serif"]
            "FCGI Start"
            "Incoming request"
            "Output to FCGI stdin"


        node [color=greenyellow, shape=box, style=filled, fontname="sans-serif"]
            "Execute"


        # Signals
        node [shape=box color=yellow style=filled, fontname="sans-serif"]
            "Emit STARTED"
            "Emit REQUEST"
            "Emit RESPONSE"
            "Emit 404"

        node  [color=greenyellow, shape=box, style="filled", fontname="sans-serif"]
            "Core SERVICE?" [shape=hexagon, style=filled]
            "Response ready?" [shape=hexagon, style=filled]
            "Get the SERVICE"

        {rank=same "FCGI Start" -> "Emit STARTED"}
        "Emit STARTED" -> "Incoming request"

        subgraph cluster_0 {
            label = "FCGI Loop";
            style=filled;
            color=lightgrey;

            {rank=same  "Incoming request" "Emit REQUEST" "Get the SERVICE"}
            {rank=same  "Core SERVICE?" "Emit 404" "Response ready?"}
            {rank=same  "Execute" "Exception"  "Emit RESPONSE"}

            "Incoming request" -> "Emit REQUEST"
            "Emit REQUEST" -> "Get the SERVICE"
            "Get the SERVICE" -> "Core SERVICE?"
            "Core SERVICE?" -> "Execute" [ label="yes" ]
            "Execute" -> "Emit RESPONSE"
            "Emit RESPONSE" -> "Output to FCGI stdin"
            "Core SERVICE?" -> "Emit 404" [ label="no" ]
            "Emit 404" -> "Response ready?"
            "Response ready?"  -> "Output to FCGI stdin" [ label="yes" ]
            "Response ready?"  -> "Exception" [ label="no" ]
            "Exception" -> "Output to FCGI stdin"

            "Output to FCGI stdin" -> "Incoming request"

        }

        #"Core SERVICE?" -> "Response ready?" [style=invis]
    }

Observer pros & cons

Pros

Provides all the possibilities to manipulate the request and response and extend QGIS Server

Cons

requires heavy refactoring of request handler --> request/repsponse handler
could introduce some (probably tiny) overhead even if there aren't any plugins installed/enabled

404 PoC Implementation Details

FCGI application starts and maintain an hash of services exposed by the plugins
When FCGI loop ends without a match for the core services (WMS, WFS, WCS) the 404 handler matches SERVICE with the service metatag exposed by the plugin.
If there is a match we do more checks:
- the REQUEST parameter must match the CSV list of plugin's exposed methods, as declared in the methods metadata
- the plugin loads without errors
- the plugin has a method named as the REQUEST

Example query string:

http://localhost/cgi-bin/qgis_mapserv.fcgi?SERVICE=HELLO&REQUEST=SayHello

404 PoC metadata

Python methods (static!) are invoked and their output is captured and passed on to FCGI

Plugins exposed SERVICE and REQUEST methods are listed in Plugin's metadata:

Example metadata:

service=HELLO
methods=GetCapabilities,GetOutput,RemoteConsole,SayHello

404 PoC methods

When all checks are done the plugin runs by calling the method in the REQUEST, the method receives:

the project path and
the query string as parameters.

The plugin (static) method has access to global python environment and can run arbitrary python commands, it can optionally return a content type string (the server defaults to text/plain).

The plugin CGI-style output is captured diverting stdout and stderr to a custom buffer which becomes the server response.

404 PoC example method

@staticmethod
def SayHello(project_path, parameters):
    """Just say something"""
    print "HelloServer"
    return 'text/plain'

Example response:

200 OK
Connection: close
Date: Thu, 04 Sep 2014 09:56:36 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: Accept-Encoding
Content-Length: 12
Content-Type: text/plain
Client-Date: Thu, 04 Sep 2014 09:56:36 GMT
Client-Peer: 127.0.0.1:80
Client-Response-Num: 1

HelloServer

Observer refactoring

System Message: ERROR/3

index.rstline 287

Unknown directive type "graph".

.. graph:: images/qgis-server-refactoring.png
    :scale: 70%


    digraph g {

        #rankdir="LR"
        node [color=greenyellow, shape=box, style=filled, fontname="sans-serif"]

        request[label="Incoming request", style="filled,rounded"]
        is_wms [label="Is WMS?", shape=hexagon]
        is_wfs [label="Is WFS?", shape=hexagon]
        output [label="Output", shape=box, style="filled,rounded"]
        exception [label="Exception"]

        request -> is_wms
        is_wms -> output  [ label="yes" ]
        is_wms -> is_wfs [ label="no" ]
        is_wfs -> output [ label="yes" ]
        is_wfs -> exception [ label="no" ]
        exception -> output
        output -> request

    }

Observer current loop

The current main loop:

Get the request
- create GET/POST/SOAP request handler
- if SERVICE is WMS/WFS/WCS
  create WMS/WFS/WCS server passing in request handler take()
  call server's executeRequest()
  call request handler output method
- else Exception

Observer refactoring

The new main loop:

Get the request
- create GET/POST/SOAP request handler
- pass request to iFace
- call plugins (emit request signal)
- if there is not a response
  if SERVICE is WMS/WFS/WCS
  create WMS/WFS/WCS server
  call server's executeRequest()
  store the byte stream output and content type in the request handler
  call plugins (emit 404 signal)
- call plugins (emit response signal)
- if there is not a response
  store Exception in the request handler
- request handler output the response

Server iFace

Not sure about what should be exposed:

request/response handler
capabilities cache
renderer
logger ?

To GUI or not to GUI?

Should we use the same system for desktop and server plugins?

What is the (mostly) unique selling point of QGIS Server ?

GUI-based configuration of the map and the services!

A Server plugin can have (not must) a desktop interface aimed to configure itself

One-click install and deployment

Example(s)

An example plugin is available:

https://github.com/elpaso/qgis-helloserver

A running implementation of the sever plugins basic functionalities is available here:

https://github.com/elpaso/QGIS/tree/serverplugins

Local test:

http://qwc/cgi-bin/qgis_mapserv.cgi?SERVICE=HELLO&request=GetCapabilities http://qwc/cgi-bin/qgis_mapserv.cgi?SERVICE=HELLO&request=SayHello http://qwc/cgi-bin/qgis_mapserv.cgi?SERVICE=HELLO&request=RemoteConsole http://qwc/cgi-bin/qgis_mapserv.cgi?SERVICE=WPS&request=GetCapabilities

Restrictions

If the plugin has a Desktop interface it cannot usually access to user's QSettings, this means that plugins options have to be stored somewhere else in order to be accessible by the server side.