Thanks to the QGIS Grant 2019 program, we (myself and Nyall Dawson) have developed a new QGIS core API, fully exposed to Python, that makes it possible to manage stored connections to various data provider source in a unified and consistent way.
The goal of the new API is twofold:
- provide a unified way to store and retrieve data provider connections in the QGIS settings
- provide an abstract set of methods to perform most common operation on DB data sources (e.g. browse tables, drop/create a table/schema, run arbitrary SQL commands etc.)
The new API is documented in https://qgis.org/api/classQgsAbstractProviderConnection.html and it provides a few specializations for DB connections (https://qgis.org/api/classQgsAbstractDatabaseProviderConnection.html) and an initial PR implementation for web service-based connections (https://github.com/qgis/QGIS/pull/33045).
While the whole of the desired refactoring work was too big for a single grant request, the first work package has been completed and the following data providers have been partially or totally refactored to make use of the new connections API:
- postgres
- geopackage (OGR)
- spatialite
The new API was also used to implement the automatic loading of layer dependencies (not part of the grant program).
For developers interested in working with the new API, a set Python tests are available to show how to use the methods: https://github.com/qgis/QGIS/blob/master/tests/src/python/test_qgsproviderconnection_ogr_gpkg.py (see also the postgres and spatialite companion tests).
There is still a large amount of work to be done in order to complete all the desired refactoring and to remove all the Python and C++ code that will be ultimately be made redundant. In particular, future work should be undertaken to:
- port all remaining data providers to the new API
- refactor and eliminate the remaining DB-manager connectors to make use of the abstract API
- eliminate duplicate and untested code inside the Processing framework for working with Postgres databases and port the code to the new, stable, well-tested API
- refactor and eliminate the remaining QGIS browser data items to make use of the abstract API
For further information, the following paragraphs (taken from the original grant proposal) will provide full details about the background of this work.
Background/motivation
- DB-Manager is an important part of the QGIS interface, which allows browsing/previews of different DB-based data sources, complex queries, management of layers, import-export etc., DB creation and deletion etc.
- After the QGIS 3.0 release, improvements within the core browser widgets implemented in C++ have resulted in a (constantly growing) degree of overlapping functionality between the browser and db manager.
- After QGIS 3 API improvements concerning layer import and export functionality, there are many duplicated implementations between browser and db manager – some functionality is better in browser, some functionality is better in db manager. Users are forced to choose between two competing semi-complete alternatives, instead of having one, complete, well integrated solution.
- There are no unit tests for DB-Manager and this leads to frequent regressions, which (aside from being frustrating for users) consume a substantial part of our development time and budget during the bugfixing programs. Furthermore the nature of large Python codebases like db manager makes it very easy to accidentally break functionality with no warning or errors during development.
Proposed solution
We propose to start refactoring the DB-manager plugin functionality into core C++ implementation, reusing existing core API and replacing redundant duplicate functionality.
The clear advantages are:
- no duplicate functionality, so it’s easier for users to understand and use
- more usage of well-tested and well-maintained core C++ API
- testability and immediate feedback on API breaks (an advantage of C++ is that the application won’t even build if an API is changed or accidentally misused)
- better performance
- the ability to expose database management functionality via stable PyQGIS API, allowing other plugins and scripts to utilise this functionality. In future, Processing algorithms may also be developed which would take advantage of these functions (e.g. “create schema”, “drop table”, “vacuum table” algorithms)
- DB management functionality would be available within the main QGIS window (from the Browser panel), instead of as a separate dialog.
Grant proposal package
The above mentioned work is too large to be completed within a single grant, so what we propose here is to start the refactoring needed in order to have a core stable C++ API that can be used by the application and the plugins and that will be available to fully move DB manager to C++ API in the future avoiding duplication of code and functionality.
- create an interface for databases that expose the required functions to a coherent API
- add missing tests and documentation for the a.m. API
- porting some basic functions from db manager to the new api:
- create table (with native field types support)
- create schema
- delete table
- Rename table
The API will be exposed through the browser and it will be used by the DB manager instead of the Python implementation that is currently used.