QGIS Server With Python Superpowers

Workshop

Alessandro Pasotti

QCooperative / ItOpen

Workshop Program

QGIS Server

The WYSIWYG GIS Server

From the desktop to the web!

Typical Workflow

Supported Standards

Compliance Tests

OGC CITE Compliance Testing

CI tests:

http://test.qgis.org/ogc_cite/

System Overview

Project Configuration

Official documentation: https://docs.qgis.org/testing/en/docs/user_manual/working_with_ogc/server/index.html

Data Storage

Project File Storage

Use rewrite!

Specifiers:

Deployment Strategies

1. Docker Containers

  • - you have to know Docker
  • + you can easily replicate/move/scale deployments

2. Bare Metal or VM

  • + maybe easier to setup/customize

Docker Images

Demo VM Stack

Server

Port

Mapped to host

Nginx FastCGI

80

8080

Apache (Fast)CGI

81

8081

Nginx Python

82

8082

Nginx MapProxy

83

8083

Development server

8000

8000

Plain CGI is only useful for testing!

The Development Server

Not suitable for production!

Usage: qgis_mapserver [options] [address:port]
QGIS Development Server

Options:
-l <logLevel>     Sets log level (default: 0)
                    0: INFO
                    1: WARNING
                    2: CRITICAL
-p <projectPath>  Path to a QGIS project file (*.qgs or *.qgz),
                    if specified it will override the query string MAP argument
                    and the QGIS_PROJECT_FILE environment variable

Arguments:
addressAndPort    Listen to address and port (default: "localhost:8000")
                    address and port can also be specified with the environment
                    variables QGIS_SERVER_ADDRESS and QGIS_SERVER_PORT

FCGI Requirements Summary

xvfb is required for features like printing and HTML labels.

Advanced QGIS Server Configuration

12 factors app https://12factor.net/

Configuration through environment variables

Authenticated Layers in QGIS Server

QGIS authentication DB qgis-auth.db path can be specified with the environment variable QGIS_AUTH_DB_DIR_PATH

QGIS_AUTH_PASSWORD_FILE environment variable can contain the master password required to decrypt the authentication DB.

Make sure that the permissions on the file are set to be only readable by the Server’s process user and check that the file is not accessible via any URL.

TODO for QGIS4: QGIS_AUTH_PASSWORD needs to be added.

Parallel Rendering

QGIS_SERVER_PARALLEL_RENDERING

Activates parallel rendering for WMS GetMap requests. It’s disabled (false) by default. Available values are:

0 or false (case insensitive) 1 or true (case insensitive)

QGIS_SERVER_MAX_THREADS

Number of threads to use when parallel rendering is activated. Default value is -1 to use the number of processor cores.

Logging

QGIS_SERVER_LOG_FILE (deprecated)

Specify path and filename. Make sure that server has proper permissions for writing to file. File should be created automatically, just send some requests to server. If it’s not there, check permissions.

QGIS_SERVER_LOG_STDERR (best option)

QGIS_SERVER_LOG_LEVEL

Specify desired log level. Available values are:

0 or INFO (log all requests) 1 or WARNING 2 or CRITICAL (log just critical errors, suitable for production purposes)

Caching

A QGIS Server instance caches:

Caches are not shared among instances, layers are not cached.

Caching is generally delegated to different tier, caching solutions are expecially recommended for serving tiles:

Look for metatiles and/or activate TILE buffer support if your layers contain labels.

WFS3/OAPIF and the new OGC APIs

Resources overrides (HTML templates, JS/CSS etc.):

Base directory for all WFS3 static resources (HTML templates, CSS, JS etc.) QGIS_SERVER_API_RESOURCES_DIRECTORY

https://docs.qgis.org/testing/en/docs/user_manual/working_with_ogc/server/services.html#the-html-template-language

VM Stack Summary

Alternative:

Optional:

Bare Metal - OS Setup

We are using Ubuntu Bionic 64bit

https://github.com/elpaso/qgis3-server-vagrant

in Vagrant it is provided by the box:

https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64-vagrant.box

Setup Steps

Provided VMs

  1. Unprovisioned (software installed, no configuration)

    You need to make the configuration manually or run the provisioning scripts from:

    /vagrant/provisioning
  2. Fully provisioned (ready to run)

SSH into the Machine

Plain VM (username: qgis, password: qgis):

ssh -p 2222 qgis@localhost # password: qgis
sudo su - # become superuser

Vagrant:

vagrant up
vagrant ssh
sudo su - # become superuser

Checkpoint: you need to be able to log into the machine and become root

Add Resources from Workshop Repository

Only for unprovisioned machines!

wget https://github.com/elpaso/qgis3-server-vagrant/archive/master.zip
unzip master.zip
rm -rf /vagrant/ # if exists
mv qgis3-server-vagrant-master/ /vagrant
rm master.zip
cd /vagrant/provisioning

The Provisioning Scripts

Steps:

Add Required Repositories

# Add QGIS repositories
apt-key adv --keyserver keyserver.ubuntu.com --recv-key 51F523511C7028C3
echo 'deb http://qgis.org/ubuntu-nightly bionic main' > /etc/apt/sources.list.d/ubuntu-qgis.list
apt-get update && apt-get -y upgrade

Which repository? https://qgis.org/en/site/forusers/alldownloads.html#debian-ubuntu

Check for New Packages

Checkpoint: the available version of qgis-server must be >= 3 from qgis.org

apt-cache policy qgis-server
# output follows:
qgis-server:
Installed: 1:3.11.0+git20200214+51ba7e8a89+28bionic
Candidate: 1:3.11.0+git20200214+51ba7e8a89+28bionic
Version table:
*** 1:3.11.0+git20200214+51ba7e8a89+28bionic 500
       500 http://qgis.org/ubuntu-nightly bionic/main amd64 Packages
       100 /var/lib/dpkg/status
    2.18.17+dfsg-1 500
       500 http://it.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

Install System Software

Install the software, see:

/vagrant/provisioning/config.sh
/vagrant/provisioning/common.sh
# Common configuration
export QGIS_SERVER_DIR=/qgis-server
export DEBIAN_FRONTEND=noninteractive
# Install QGIS server and deps (overwrite is a temporary solution)
apt-get -y install -o Dpkg::Options::="--force-overwrite" qgis-server python3-qgis xvfb
# Install utilities (optional)
apt-get -y install vim unzip ipython3

Install System Software I

Checkpoint: qgis installed with no errors, you can check it with

/usr/lib/cgi-bin/qgis_mapserv.fcgi 2> /dev/null
Content-Length: 54
Content-Type: text/xml; charset=utf-8
Server:  Qgis FCGI server - QGis version 3.0.0-Girona
Status:  500

<ServerException>Project file error</ServerException>

Install System Software II

Copy resources

. /vagrant/provisioning/config.sh

# Install sample projects and plugins
mkdir -p $QGIS_SERVER_DIR/logs
cp -r /vagrant/resources/web/htdocs $QGIS_SERVER_DIR
cp -r /vagrant/resources/web/plugins $QGIS_SERVER_DIR
cp -r /vagrant/resources/web/projects $QGIS_SERVER_DIR
chown -R www-data.www-data $QGIS_SERVER_DIR

Install System Software III

Setup xvfb and plain CGI

# Setup xvfb
cp /vagrant/resources/xvfb/xvfb.service \
    /etc/systemd/system/xvfb.service
systemctl enable /etc/systemd/system/xvfb.service
service xvfb start

# Symlink to cgi for apache CGI mode
ln -s /usr/lib/cgi-bin/qgis_mapserv.fcgi \
    /usr/lib/cgi-bin/qgis_mapserv.cgi

Apache2

Installation (with FCGI module)

The Apache HTTP Server Project is an effort to develop and maintain an open-source HTTP server for modern operating systems including UNIX and Windows.

apt-get -y install apache2 libapache2-mod-fcgid

Apache2 architecture

Apache2 Configuration I

Configure the web server

cp /vagrant/resources/apache2/001-qgis-server.conf \
    /etc/apache2/sites-available
# sed: replace QGIS_SERVER_DIR with actual path
sed -i -e "s@QGIS_SERVER_DIR@${QGIS_SERVER_DIR}@g" \
    /etc/apache2/sites-available/001-qgis-server.conf
# sed: replace port from 80 to 81
sed -i -e 's/VirtualHost \*:80/VirtualHost \*:81/' \
    /etc/apache2/sites-available/001-qgis-server.conf
sed -i -e "s@QGIS_SERVER_DIR@${QGIS_SERVER_DIR}@g" \
    $QGIS_SERVER_DIR/htdocs/index.html

Apache2 Configuration II

VirtualHost configuration for both FastCGI and CGI

<VirtualHost *:81>
    # [ ... ] Standard config goes here
    FcgidInitialEnv DISPLAY ":99"
    FcgidInitialEnv LC_ALL "en_US.UTF-8"
    # FcgidInitialEnv QGIS_DEBUG 1
    # FcgidInitialEnv QGIS_PLUGINPATH "QGIS_SERVER_DIR/plugins"
    # FcgidInitialEnv QGIS_AUTH_DB_DIR_PATH "QGIS_SERVER_DIR"
    # Path to the QGIS3.ini settings file
    # FcgidInitialEnv QGIS_OPTIONS_PATH "QGIS_SERVER_DIR"
    # Path to the user profile directory
    # FcgidInitialEnv QGIS_CUSTOM_CONFIG_PATH "QGIS_SERVER_DIR"

Apache2 Configuration III

Logging

# FcgidInitialEnv QGIS_DEBUG 1
# Deprecated log to file (bad practice!)
# FcgidInitialEnv QGIS_SERVER_LOG_FILE "QGIS_SERVER_DIR/logs/qgis-apache-001.log"
# Log to stderr instead:
FcgidInitialEnv QGIS_SERVER_LOG_STDERR 1
# FcgidInitialEnv QGIS_SERVER_LOG_LEVEL 0

Apache2 Configuration IV

    # Required by QGIS plugin HTTP BASIC auth
    <IfModule mod_fcgid.c>
        RewriteEngine on
        RewriteCond %{HTTP:Authorization} .
        RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
    </IfModule>
    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride All
        Options +ExecCGI -MultiViews +FollowSymLinks
        Allow from all
        AddHandler cgi-script .cgi
        AddHandler fcgid-script .fcgi
        Require all granted
    </Directory>
</VirtualHost>

Apache2 Configuration V

Enable sites and restart

a2enmod rewrite # Only required by some plugins
# a2enmod cgid # Required by plain old CGI
a2dissite 000-default
a2ensite 001-qgis-server
# Listen on port 81 instead of 80 (nginx)
sed -i -e 's/Listen 80/Listen 81/' /etc/apache2/ports.conf
service apache2 restart # Restart the server

Checkpoint: check whether Apache is listening on localhost port 8081 http://localhost:8081

Nginx Installation

nginx [engine x] is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server

# Install the software
export DEBIAN_FRONTEND=noninteractive
apt-get -y install nginx

Nginx architecture

Nginx configuration I

# Enable site
rm /etc/nginx/sites-enabled/default
cp /vagrant/resources/nginx/qgis-server-fcgi \
    /etc/nginx/sites-enabled/qgis-server
# sed: replace QGIS_SERVER_DIR with actual path
sed -i -e "s@QGIS_SERVER_DIR@${QGIS_SERVER_DIR}@" \
    /etc/nginx/sites-enabled/qgis-server

Nginx Configuration II

# Extract server name and port from HTTP_HOST, this
# is required because we are behind a VMs mapped port

map $http_host $parsed_server_name {
    default  $host;
    "~(?P<h>[^:]+):(?P<p>.*+)" $h;
}

map $http_host $parsed_server_port {
    default  $server_port;
    "~(?P<h>[^:]+):(?P<p>.*+)" $p;
}

Nginx Configuration III

Load balancing (round robin default, or least_conn;)

upstream qgis_mapserv_backend {
    ip_hash;
    server unix:/run/qgis_mapserv4.sock;
    server unix:/run/qgis_mapserv3.sock;
    server unix:/run/qgis_mapserv2.sock;
    server unix:/run/qgis_mapserv1.sock;
}

Nginx Configuration IV

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    # This is vital
    underscores_in_headers on;

    root /qgis-server/htdocs;

    location / {
            # First attempt to serve request as file, then
            # as directory, then fall back to displaying a 404.
            try_files $uri $uri/ =404;
    }

Nginx Configuration V

Rewrite!

# project file set by env var
# example: http://localhost:8080/project/project_base_name/
location ~ ^/project/([^/]+)/?(.*)$
{
  set $qgis_project /qgis-server/projects/$1.qgs;
  rewrite ^/project/(.*)$ /cgi-bin/qgis_mapserv.fcgi last;
}

Nginx Configuration VI

location /cgi-bin/ {
    # Disable gzip (it makes scripts feel slower since they
    # have to complete before getting gzipped)
    gzip off;

    # Fastcgi socket
    fastcgi_pass  qgis_mapserv_backend;

    # $http_host contains the original server name and port, such as: "localhost:8080"
    fastcgi_param SERVER_NAME       $parsed_server_name;
    fastcgi_param SERVER_PORT       $parsed_server_port;

    # [ continue ... ]

Nginx Configuration VII

        # [ ... continued ]

        # Set project file from env var
        fastcgi_param QGIS_PROJECT_FILE $qgis_project;

        # Fastcgi parameters, include the standard ones
        # (note: this needs to be last or it will overwrite fastcgi_param set above)
        include /etc/nginx/fastcgi_params;

    }
}

Systemd Socket Config for FastCGI

Socket

# Path: /etc/systemd/system/qgis-server-fcgi@.socket
# systemctl enable qgis-server-fcgi@{1..4}.socket && systemctl start qgis-server-fcgi@{1..4}.socket

[Unit]
Description = QGIS Server FastCGI Socket (instance %i)
[Socket]
SocketUser = www-data
SocketGroup = www-data
SocketMode = 0660
ListenStream = /run/qgis_mapserv%i.sock
[Install]
WantedBy = sockets.target

Systemd Service Config for FastCGI

# Path: /etc/systemd/system/qgis-server-fcgi@.service
# systemctl start qgis-server-fcgi@{1..4}.service

[Unit]
Description = QGIS Server Tracker FastCGI backend (instance %i)
[Service]
User = www-data
Group = www-data
ExecStart = /usr/lib/cgi-bin/qgis_mapserv.fcgi
StandardInput = socket
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=qgis-server-fcgi
WorkingDirectory=/tmp
Restart = always

Systemd Service Config for FastCGI

Service

# Environment
Environment="QGIS_AUTH_DB_DIR_PATH=QGIS_SERVER_DIR/projects"
Environment="QGIS_SERVER_LOG_FILE=QGIS_SERVER_DIR/logs/qgis-server-fcgi.log"
Environment="QGIS_SERVER_LOG_LEVEL=0"
Environment="QGIS_DEBUG=1"
Environment="DISPLAY=:99"
Environment="QGIS_PLUGINPATH=QGIS_SERVER_DIR/plugins"
Environment="QGIS_OPTIONS_PATH=QGIS_SERVER_DIR"
Environment="QGIS_CUSTOM_CONFIG_PATH=QGIS_SERVER_DIR"

[Install]
WantedBy = multi-user.target

Checkpoint: Nginx

Check WMS on localhost 8080 in the browser

http://localhost:8080

Follow the links!

Checkpoint: QGIS as a Client

Check WMS and WFS using QGIS as a client.

Check that WFS requires HTTP Basic auth (username and password = "qgis")

Check that WWS GetFeatureInfo returns a (blueish) formatted HTML

Note: a test project with pre-configured endpoints is available in the resources/qgis/ directory.

Checkpoint: WMS Search FILTER

Searching features with WMS

http://localhost:8080/cgi-bin/qgis_mapserv.fcgi?
MAP=/qgis-server/projects/helloworld.qgs&SERVICE=WMS
&REQUEST=GetFeatureInfo&CRS=EPSG%3A4326&WIDTH=1794&HEIGHT=1194
&LAYERS=world&QUERY_LAYERS=world&
FILTER=world%3A%22NAME%22%20%3D%20%27SPAIN%27

The filter is a QGIS Expression:

FILTER=world:"NAME" = 'SPAIN'

WMS Vendor Parameters

Full list: https://docs.qgis.org/testing/en/docs/user_manual/working_with_ogc/server/services.html

http://localhost:8081/cgi-bin/qgis_mapserv.fcgi?
INFO_FORMAT=text/plain&MAP=/qgis-server/projects/helloworld.qgs
&SERVICE=WMS&REQUEST=GetFeatureInfo&CRS=EPSG%3A4326&WIDTH=1794&HEIGHT=1194&LAYERS=world&
WITH_GEOMETRY=TRUE&QUERY_LAYERS=world&FILTER=world%3A%22NAME%22%20%3D%20%27SPAIN%27

Checkpoint: Highlighting

The SELECTION parameter can highlight features from one or more layers: Vector features can be selected by passing comma separated lists with feature ids in GetMap and GetPrint. Example: SELECTION=mylayer1:3,6,9;mylayer2:1,5,6

http://localhost:8080/cgi-bin/qgis_mapserv.fcgi?
MAP=/qgis-server/projects/helloworld.qgs&SERVICE=WMS&VERSION=1.3.0&
SELECTION=world%3A44&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&
LAYERS=world&CRS=EPSG%3A4326&STYLES=&DPI=180&WIDTH=1794&HEIGHT=1194&
BBOX=31.7944%2C-18.2153%2C58.0297%2C21.20361

Checkpoint: Printing

From composer templates (with substitutions!)

<Layouts>
  <Layout units="mm" printResolution="300" name="Printable World"
  worldFileMap="{db75b0bf-f2f1-42e6-9727-1b6b21d8862e}">
  ...

FORMAT can be any of PDF, PNG, JPG

See also: DXF Export

Checkpoint: Printing URL

http://localhost:8080/cgi-bin/qgis_mapserv.fcgi?
MAP=/qgis-server/projects/helloworld.qgs&SERVICE=WMS&VERSION=1.1.1&
REQUEST=GetPrint&TEMPLATE=Printable%20World&CRS=EPSG%3A4326&
map0:EXTENT=4,52,14,58&FORMAT=png&LAYERS=bluemarble,world

Checkpoint: Printing Substitutions

http://localhost:8080/cgi-bin/qgis_mapserv.fcgi?
MAP=/qgis-server/projects/helloworld.qgs&SERVICE=WMS&
VERSION=1.1.1&REQUEST=GetPrint&TEMPLATE=Printable%20World
&CRS=EPSG%3A4326&map0:EXTENT=4,52,14,58&FORMAT=png
&LAYERS=bluemarble,world&print_title=Custom%20print%20title!

Python Development

QGIS Server and Python

What can we do?

QGIS Server Modules

Server API Documentation

C++

https://qgis.org/api/group__server.html

Python

https://qgis.org/pyqgis/master/server/index.html

I/O Filters

Applications:

Legacy Architecture

SERVICE modules

Customization

New OGC API Architecture

OGC API modules

Customization

Python Development Topics

QGIS Server API

For standalone or embedding:

Python API Basics

from qgis.core import QgsApplication
from qgis.server import *
app = QgsApplication([], False)
server = QgsServer()
request = QgsBufferServerRequest(
    'http://localhost:8081/?MAP=/qgis-server/projects/helloworld.qgs' +
    '&SERVICE=WMS&REQUEST=GetCapabilities')
response = QgsBufferServerResponse()
server.handleRequest(request, response)
print(response.headers())
print(response.body().data().decode('utf8'))
app.exitQgis()

Full script: https://github.com/qgis/QGIS/blob/master/tests/src/python/qgis_wrapped_server.py

Plugins Anatomy

Plugins are loaded from QGIS_PLUGINPATH directory.

The Server Interface

A QgsServerInterface instance is made available to plugins and it provides methods to register filters, services and APIs and methods to manage the capabilities cache for legacy services.

Plugins Workflow

Filter Plugins Registration

Type

Base Class

QgsServerInterface registration

I/O

QgsServerFilter

registerFilter()

Access Control

QgsAccessControlFilter

registerAccessControl()

Cache

QgsServerCacheFilter

registerServerCache()

Note: custom SERVICE and API handlers are registered in the serverInterface.serviceRegistry()

I/O Filters Hooks

Server plugins register one or more QgsServerFilters that "listen to signals". Plugin filters receive the request/response objects and they can manipulate them with the following methods:

I/O Filters Flowchart

I/O Filters Examples

Access Control Filter Plugins

Fine-grained control over layers, features and attributes!

Example: https://github.com/elpaso/qgis3-server-vagrant/blob/master/resources/web/plugins/accesscontrol/accesscontrol.py

Docs: https://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/server.html#access-control-plugin

Cache Filter Plugins

from qgis.server import QgsServerCacheFilter
import hashlib

class StupidCache(QgsServerCacheFilter):
    """A simple in-memory and not-shared cache for demonstration purposes"""
    _cache = {}
    def _get_hash(self, request):
        # create a unique hash from the request
        paramMap = request.parameters()
        urlParam = "&".join(["%s=%s" % (k, paramMap[k]) for k in paramMap.keys()])
        m = hashlib.md5()
        m.update(urlParam.encode('utf8'))
        return m.hexdigest()

Cache Plugins II

    def getCachedDocument(self, project, request, key):
        hash = self._get_hash(request)
        try:
            result = self._cache[self._get_hash(request)]
            return result
        except KeyError:
            return QByteArray()

    def setCachedDocument(self, doc, project, request, key):
        hash = self._get_hash(request)
        self._cache[hash] = doc
        return True

serverIface.registerServerCache(StupidCache(serverIface), 100 )

Legacy Custom Services

New server plugin-based service architecture!

You can now create custom services in pure Python.

Example: https://github.com/elpaso/qgis3-server-vagrant/blob/master/resources/web/plugins/xyz/xyz.py

OGC API Custom Services

Since QGIS 3.10

New server plugin-based API architecture!

You can now create custom APIs in pure Python.

Example: https://github.com/elpaso/qgis3-server-vagrant/blob/master/resources/web/plugins/customapi/customapi.py

Other examples

The Python QGIS tests contain a comprehensive set of scripts to test services implementations in QGIS Server:

https://github.com/qgis/QGIS/tree/master/tests/src/python

Release cycle

LTR: 12 months support

https://www.qgis.org/it/site/getinvolved/development/roadmap.html#release-schedule

Presentation links

https://github.com/elpaso/qgis3-server-vagrant/ (docs folder)

SpaceForward
Right, Down, Page DownNext slide
Left, Up, Page UpPrevious slide
GGo to slide number
POpen presenter console
HToggle this help