Alessandro Pasotti
QCooperative / ItOpen
The WYSIWYG GIS Server
From the desktop to the web!
Official documentation: https://docs.qgis.org/testing/en/docs/user_manual/working_with_ogc/server/index.html
Use rewrite!
Specifiers:
- - you have to know Docker
- + you can easily replicate/move/scale deployments
- + maybe easier to setup/customize
Server | Port | Mapped to host |
Nginx FastCGI | 80 | 8080 |
Apache (Fast)CGI | 81 | 8081 |
Nginx Python | 82 | 8082 |
Nginx MapProxy | 83 | 8083 |
Development server | 8000 | 8000 |
Plain CGI is only useful for testing!
Not suitable for production!
Usage: qgis_mapserver [options] [address:port] QGIS Development Server Options: -l <logLevel> Sets log level (default: 0) 0: INFO 1: WARNING 2: CRITICAL -p <projectPath> Path to a QGIS project file (*.qgs or *.qgz), if specified it will override the query string MAP argument and the QGIS_PROJECT_FILE environment variable Arguments: addressAndPort Listen to address and port (default: "localhost:8000") address and port can also be specified with the environment variables QGIS_SERVER_ADDRESS and QGIS_SERVER_PORT
xvfb is required for features like printing and HTML labels.
12 factors app https://12factor.net/
Configuration through environment variables
QGIS authentication DB qgis-auth.db path can be specified with the environment variable QGIS_AUTH_DB_DIR_PATH
QGIS_AUTH_PASSWORD_FILE environment variable can contain the master password required to decrypt the authentication DB.
Make sure that the permissions on the file are set to be only readable by the Server’s process user and check that the file is not accessible via any URL.
TODO for QGIS4: QGIS_AUTH_PASSWORD needs to be added.
QGIS_SERVER_PARALLEL_RENDERING
Activates parallel rendering for WMS GetMap requests. It’s disabled (false) by default. Available values are:
0 or false (case insensitive) 1 or true (case insensitive)
QGIS_SERVER_MAX_THREADS
Number of threads to use when parallel rendering is activated. Default value is -1 to use the number of processor cores.
QGIS_SERVER_LOG_FILE (deprecated)
Specify path and filename. Make sure that server has proper permissions for writing to file. File should be created automatically, just send some requests to server. If it’s not there, check permissions.
QGIS_SERVER_LOG_STDERR (best option)
QGIS_SERVER_LOG_LEVEL
Specify desired log level. Available values are:
0 or INFO (log all requests) 1 or WARNING 2 or CRITICAL (log just critical errors, suitable for production purposes)
A QGIS Server instance caches:
Caches are not shared among instances, layers are not cached.
Caching is generally delegated to different tier, caching solutions are expecially recommended for serving tiles:
Look for metatiles and/or activate TILE buffer support if your layers contain labels.
Resources overrides (HTML templates, JS/CSS etc.):
Base directory for all WFS3 static resources (HTML templates, CSS, JS etc.) QGIS_SERVER_API_RESOURCES_DIRECTORY
Alternative:
Optional:
We are using Ubuntu Bionic 64bit
https://github.com/elpaso/qgis3-server-vagrant
in Vagrant it is provided by the box:
https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64-vagrant.box
You need to make the configuration manually or run the provisioning scripts from:
/vagrant/provisioning
Plain VM (username: qgis, password: qgis):
ssh -p 2222 qgis@localhost # password: qgis sudo su - # become superuser
Vagrant:
vagrant up
vagrant ssh
sudo su - # become superuser
Checkpoint: you need to be able to log into the machine and become root
Only for unprovisioned machines!
wget https://github.com/elpaso/qgis3-server-vagrant/archive/master.zip unzip master.zip rm -rf /vagrant/ # if exists mv qgis3-server-vagrant-master/ /vagrant rm master.zip cd /vagrant/provisioning
Steps:
# Add QGIS repositories apt-key adv --keyserver keyserver.ubuntu.com --recv-key 51F523511C7028C3 echo 'deb http://qgis.org/ubuntu-nightly bionic main' > /etc/apt/sources.list.d/ubuntu-qgis.list apt-get update && apt-get -y upgrade
Which repository? https://qgis.org/en/site/forusers/alldownloads.html#debian-ubuntu
Checkpoint: the available version of qgis-server must be >= 3 from qgis.org
apt-cache policy qgis-server # output follows: qgis-server: Installed: 1:3.11.0+git20200214+51ba7e8a89+28bionic Candidate: 1:3.11.0+git20200214+51ba7e8a89+28bionic Version table: *** 1:3.11.0+git20200214+51ba7e8a89+28bionic 500 500 http://qgis.org/ubuntu-nightly bionic/main amd64 Packages 100 /var/lib/dpkg/status 2.18.17+dfsg-1 500 500 http://it.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
Install the software, see:
/vagrant/provisioning/config.sh /vagrant/provisioning/common.sh
# Common configuration export QGIS_SERVER_DIR=/qgis-server export DEBIAN_FRONTEND=noninteractive # Install QGIS server and deps (overwrite is a temporary solution) apt-get -y install -o Dpkg::Options::="--force-overwrite" qgis-server python3-qgis xvfb # Install utilities (optional) apt-get -y install vim unzip ipython3
Checkpoint: qgis installed with no errors, you can check it with
/usr/lib/cgi-bin/qgis_mapserv.fcgi 2> /dev/null Content-Length: 54 Content-Type: text/xml; charset=utf-8 Server: Qgis FCGI server - QGis version 3.0.0-Girona Status: 500 <ServerException>Project file error</ServerException>
Copy resources
. /vagrant/provisioning/config.sh # Install sample projects and plugins mkdir -p $QGIS_SERVER_DIR/logs cp -r /vagrant/resources/web/htdocs $QGIS_SERVER_DIR cp -r /vagrant/resources/web/plugins $QGIS_SERVER_DIR cp -r /vagrant/resources/web/projects $QGIS_SERVER_DIR chown -R www-data.www-data $QGIS_SERVER_DIR
Setup xvfb and plain CGI
# Setup xvfb cp /vagrant/resources/xvfb/xvfb.service \ /etc/systemd/system/xvfb.service systemctl enable /etc/systemd/system/xvfb.service service xvfb start # Symlink to cgi for apache CGI mode ln -s /usr/lib/cgi-bin/qgis_mapserv.fcgi \ /usr/lib/cgi-bin/qgis_mapserv.cgi
Installation (with FCGI module)
The Apache HTTP Server Project is an effort to develop and maintain an open-source HTTP server for modern operating systems including UNIX and Windows.
apt-get -y install apache2 libapache2-mod-fcgid
Configure the web server
cp /vagrant/resources/apache2/001-qgis-server.conf \ /etc/apache2/sites-available # sed: replace QGIS_SERVER_DIR with actual path sed -i -e "s@QGIS_SERVER_DIR@${QGIS_SERVER_DIR}@g" \ /etc/apache2/sites-available/001-qgis-server.conf # sed: replace port from 80 to 81 sed -i -e 's/VirtualHost \*:80/VirtualHost \*:81/' \ /etc/apache2/sites-available/001-qgis-server.conf sed -i -e "s@QGIS_SERVER_DIR@${QGIS_SERVER_DIR}@g" \ $QGIS_SERVER_DIR/htdocs/index.html
VirtualHost configuration for both FastCGI and CGI
<VirtualHost *:81> # [ ... ] Standard config goes here FcgidInitialEnv DISPLAY ":99" FcgidInitialEnv LC_ALL "en_US.UTF-8" # FcgidInitialEnv QGIS_DEBUG 1 # FcgidInitialEnv QGIS_PLUGINPATH "QGIS_SERVER_DIR/plugins" # FcgidInitialEnv QGIS_AUTH_DB_DIR_PATH "QGIS_SERVER_DIR" # Path to the QGIS3.ini settings file # FcgidInitialEnv QGIS_OPTIONS_PATH "QGIS_SERVER_DIR" # Path to the user profile directory # FcgidInitialEnv QGIS_CUSTOM_CONFIG_PATH "QGIS_SERVER_DIR"
Logging
# FcgidInitialEnv QGIS_DEBUG 1 # Deprecated log to file (bad practice!) # FcgidInitialEnv QGIS_SERVER_LOG_FILE "QGIS_SERVER_DIR/logs/qgis-apache-001.log" # Log to stderr instead: FcgidInitialEnv QGIS_SERVER_LOG_STDERR 1 # FcgidInitialEnv QGIS_SERVER_LOG_LEVEL 0
# Required by QGIS plugin HTTP BASIC auth <IfModule mod_fcgid.c> RewriteEngine on RewriteCond %{HTTP:Authorization} . RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}] </IfModule> ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ <Directory "/usr/lib/cgi-bin"> AllowOverride All Options +ExecCGI -MultiViews +FollowSymLinks Allow from all AddHandler cgi-script .cgi AddHandler fcgid-script .fcgi Require all granted </Directory> </VirtualHost>
Enable sites and restart
a2enmod rewrite # Only required by some plugins # a2enmod cgid # Required by plain old CGI a2dissite 000-default a2ensite 001-qgis-server # Listen on port 81 instead of 80 (nginx) sed -i -e 's/Listen 80/Listen 81/' /etc/apache2/ports.conf service apache2 restart # Restart the server
Checkpoint: check whether Apache is listening on localhost port 8081 http://localhost:8081
nginx [engine x] is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server
# Install the software export DEBIAN_FRONTEND=noninteractive apt-get -y install nginx
# Enable site rm /etc/nginx/sites-enabled/default cp /vagrant/resources/nginx/qgis-server-fcgi \ /etc/nginx/sites-enabled/qgis-server # sed: replace QGIS_SERVER_DIR with actual path sed -i -e "s@QGIS_SERVER_DIR@${QGIS_SERVER_DIR}@" \ /etc/nginx/sites-enabled/qgis-server
# Extract server name and port from HTTP_HOST, this # is required because we are behind a VMs mapped port map $http_host $parsed_server_name { default $host; "~(?P<h>[^:]+):(?P<p>.*+)" $h; } map $http_host $parsed_server_port { default $server_port; "~(?P<h>[^:]+):(?P<p>.*+)" $p; }
Load balancing (round robin default, or least_conn;)
upstream qgis_mapserv_backend { ip_hash; server unix:/run/qgis_mapserv4.sock; server unix:/run/qgis_mapserv3.sock; server unix:/run/qgis_mapserv2.sock; server unix:/run/qgis_mapserv1.sock; }
server { listen 80 default_server; listen [::]:80 default_server; # This is vital underscores_in_headers on; root /qgis-server/htdocs; location / { # First attempt to serve request as file, then # as directory, then fall back to displaying a 404. try_files $uri $uri/ =404; }
Rewrite!
# project file set by env var # example: http://localhost:8080/project/project_base_name/ location ~ ^/project/([^/]+)/?(.*)$ { set $qgis_project /qgis-server/projects/$1.qgs; rewrite ^/project/(.*)$ /cgi-bin/qgis_mapserv.fcgi last; }
location /cgi-bin/ { # Disable gzip (it makes scripts feel slower since they # have to complete before getting gzipped) gzip off; # Fastcgi socket fastcgi_pass qgis_mapserv_backend; # $http_host contains the original server name and port, such as: "localhost:8080" fastcgi_param SERVER_NAME $parsed_server_name; fastcgi_param SERVER_PORT $parsed_server_port; # [ continue ... ]
# [ ... continued ] # Set project file from env var fastcgi_param QGIS_PROJECT_FILE $qgis_project; # Fastcgi parameters, include the standard ones # (note: this needs to be last or it will overwrite fastcgi_param set above) include /etc/nginx/fastcgi_params; } }
Socket
# Path: /etc/systemd/system/qgis-server-fcgi@.socket # systemctl enable qgis-server-fcgi@{1..4}.socket && systemctl start qgis-server-fcgi@{1..4}.socket [Unit] Description = QGIS Server FastCGI Socket (instance %i) [Socket] SocketUser = www-data SocketGroup = www-data SocketMode = 0660 ListenStream = /run/qgis_mapserv%i.sock [Install] WantedBy = sockets.target
# Path: /etc/systemd/system/qgis-server-fcgi@.service # systemctl start qgis-server-fcgi@{1..4}.service [Unit] Description = QGIS Server Tracker FastCGI backend (instance %i) [Service] User = www-data Group = www-data ExecStart = /usr/lib/cgi-bin/qgis_mapserv.fcgi StandardInput = socket StandardOutput=syslog StandardError=syslog SyslogIdentifier=qgis-server-fcgi WorkingDirectory=/tmp Restart = always
Service
# Environment Environment="QGIS_AUTH_DB_DIR_PATH=QGIS_SERVER_DIR/projects" Environment="QGIS_SERVER_LOG_FILE=QGIS_SERVER_DIR/logs/qgis-server-fcgi.log" Environment="QGIS_SERVER_LOG_LEVEL=0" Environment="QGIS_DEBUG=1" Environment="DISPLAY=:99" Environment="QGIS_PLUGINPATH=QGIS_SERVER_DIR/plugins" Environment="QGIS_OPTIONS_PATH=QGIS_SERVER_DIR" Environment="QGIS_CUSTOM_CONFIG_PATH=QGIS_SERVER_DIR" [Install] WantedBy = multi-user.target
Check WMS and WFS using QGIS as a client.
Check that WFS requires HTTP Basic auth (username and password = "qgis")
Check that WWS GetFeatureInfo returns a (blueish) formatted HTML
Note: a test project with pre-configured endpoints is available in the resources/qgis/ directory.
Searching features with WMS
http://localhost:8080/cgi-bin/qgis_mapserv.fcgi? MAP=/qgis-server/projects/helloworld.qgs&SERVICE=WMS &REQUEST=GetFeatureInfo&CRS=EPSG%3A4326&WIDTH=1794&HEIGHT=1194 &LAYERS=world&QUERY_LAYERS=world& FILTER=world%3A%22NAME%22%20%3D%20%27SPAIN%27
The filter is a QGIS Expression:
FILTER=world:"NAME" = 'SPAIN'
Full list: https://docs.qgis.org/testing/en/docs/user_manual/working_with_ogc/server/services.html
http://localhost:8081/cgi-bin/qgis_mapserv.fcgi? INFO_FORMAT=text/plain&MAP=/qgis-server/projects/helloworld.qgs &SERVICE=WMS&REQUEST=GetFeatureInfo&CRS=EPSG%3A4326&WIDTH=1794&HEIGHT=1194&LAYERS=world& WITH_GEOMETRY=TRUE&QUERY_LAYERS=world&FILTER=world%3A%22NAME%22%20%3D%20%27SPAIN%27
The SELECTION parameter can highlight features from one or more layers: Vector features can be selected by passing comma separated lists with feature ids in GetMap and GetPrint. Example: SELECTION=mylayer1:3,6,9;mylayer2:1,5,6
http://localhost:8080/cgi-bin/qgis_mapserv.fcgi? MAP=/qgis-server/projects/helloworld.qgs&SERVICE=WMS&VERSION=1.3.0& SELECTION=world%3A44&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true& LAYERS=world&CRS=EPSG%3A4326&STYLES=&DPI=180&WIDTH=1794&HEIGHT=1194& BBOX=31.7944%2C-18.2153%2C58.0297%2C21.20361
From composer templates (with substitutions!)
<Layouts> <Layout units="mm" printResolution="300" name="Printable World" worldFileMap="{db75b0bf-f2f1-42e6-9727-1b6b21d8862e}"> ...
FORMAT can be any of PDF, PNG, JPG
See also: DXF Export
http://localhost:8080/cgi-bin/qgis_mapserv.fcgi? MAP=/qgis-server/projects/helloworld.qgs&SERVICE=WMS&VERSION=1.1.1& REQUEST=GetPrint&TEMPLATE=Printable%20World&CRS=EPSG%3A4326& map0:EXTENT=4,52,14,58&FORMAT=png&LAYERS=bluemarble,world
http://localhost:8080/cgi-bin/qgis_mapserv.fcgi? MAP=/qgis-server/projects/helloworld.qgs&SERVICE=WMS& VERSION=1.1.1&REQUEST=GetPrint&TEMPLATE=Printable%20World &CRS=EPSG%3A4326&map0:EXTENT=4,52,14,58&FORMAT=png &LAYERS=bluemarble,world&print_title=Custom%20print%20title!
What can we do?
Applications:
For standalone or embedding:
from qgis.core import QgsApplication from qgis.server import * app = QgsApplication([], False) server = QgsServer() request = QgsBufferServerRequest( 'http://localhost:8081/?MAP=/qgis-server/projects/helloworld.qgs' + '&SERVICE=WMS&REQUEST=GetCapabilities') response = QgsBufferServerResponse() server.handleRequest(request, response) print(response.headers()) print(response.body().data().decode('utf8')) app.exitQgis()
Full script: https://github.com/qgis/QGIS/blob/master/tests/src/python/qgis_wrapped_server.py
Plugins are loaded from QGIS_PLUGINPATH directory.
A QgsServerInterface instance is made available to plugins and it provides methods to register filters, services and APIs and methods to manage the capabilities cache for legacy services.
Type | Base Class | QgsServerInterface registration |
I/O | QgsServerFilter | registerFilter() |
Access Control | QgsAccessControlFilter | registerAccessControl() |
Cache | QgsServerCacheFilter | registerServerCache() |
Note: custom SERVICE and API handlers are registered in the serverInterface.serviceRegistry()
Server plugins register one or more QgsServerFilters that "listen to signals". Plugin filters receive the request/response objects and they can manipulate them with the following methods:
Fine-grained control over layers, features and attributes!
Docs: https://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/server.html#access-control-plugin
from qgis.server import QgsServerCacheFilter import hashlib class StupidCache(QgsServerCacheFilter): """A simple in-memory and not-shared cache for demonstration purposes""" _cache = {} def _get_hash(self, request): # create a unique hash from the request paramMap = request.parameters() urlParam = "&".join(["%s=%s" % (k, paramMap[k]) for k in paramMap.keys()]) m = hashlib.md5() m.update(urlParam.encode('utf8')) return m.hexdigest()
def getCachedDocument(self, project, request, key): hash = self._get_hash(request) try: result = self._cache[self._get_hash(request)] return result except KeyError: return QByteArray() def setCachedDocument(self, doc, project, request, key): hash = self._get_hash(request) self._cache[hash] = doc return True serverIface.registerServerCache(StupidCache(serverIface), 100 )
New server plugin-based service architecture!
You can now create custom services in pure Python.
Example: https://github.com/elpaso/qgis3-server-vagrant/blob/master/resources/web/plugins/xyz/xyz.py
Since QGIS 3.10
New server plugin-based API architecture!
You can now create custom APIs in pure Python.
The Python QGIS tests contain a comprehensive set of scripts to test services implementations in QGIS Server:
LTR: 12 months support
https://www.qgis.org/it/site/getinvolved/development/roadmap.html#release-schedule
https://github.com/elpaso/qgis3-server-vagrant/ (docs folder)
Space | Forward |
---|---|
Right, Down, Page Down | Next slide |
Left, Up, Page Up | Previous slide |
G | Go to slide number |
P | Open presenter console |
H | Toggle this help |