Welcome to Planet PostGIS

September 19, 2017

PostGIS Development

PostGIS 2.1.9 Released

The PostGIS development team has uploaded the final release of the PostGIS 2.1 branch. The 2.1 branch is now end-of-life. As befits a patch release, the focus is on bugs and breakages.

Continue Reading by clicking title hyperlink ..

by Paul Ramsey at September 19, 2017 12:00 AM

September 14, 2017

Paul Ramsey

PostGIS Operators in 2.4

TL;DR: If you are using ORDER BY or GROUP BY on geometry columns in your application and you have expectations regarding the order or groups you obtain, beware that PostGIS 2.4 changes the behaviour or ORDER BY and GROUP BY. Review your applications accordingly.


The first operators we learn about in elementary school are =, > and <, but they are the operators that are the hardest to define in the spatial realm.

PostGIS Operators in 2.4

When is = equal?

For example, take “simple” equality. Are geometry A and B equal? Should be easy, right?

But are we talking about:

  1. A and B have exactly the same vertices in the same order and with the same starting points?
  2. A and B have exactly the same vertices in any order? (see ST_OrderingEquals)
  3. A and B have the same vertices in any order but different starting points?
  4. A has some extra vertices that B does not, but they cover exactly the same area in space? (see ST_Equals)
  5. A and B have the same bounds?

Confusingly, for the first 16 years of its existence, PostGIS used definition 5, “A and B have the same bounds” when evaluating the = operator for two geometries.

However, for PostGIS 2.4, PostGIS will use definition 1: “A and B have exactly the same vertices in the same order and with the same starting points”.

Why does this matter? Because the behavour of the SQL GROUP BY operation is bound to the “=” operator: when you group by a column, an output row is generated for all groups where every item is “=” to every other item. With the new definition in 2.4, the semantics of GROUP BY should be more “intuitive” when used against geometries.

What is > and <?

Greater and less than are also tricky in the spatial domain:

  • Is POINT(0 0) less than POINT(1 1)? Sort of looks like it, but…
  • Is POINT(0 0) less than POINT(-1 1) or POINT(1 -1)? Hm, that makes the first one look less obvious…

Greater and less than are concepts that make sense for 1-dimensional values, but not for higher dimensions. The “>” and “<” operators have accordingly been an ugly hack for a long time: they compared the minima of the bounding boxes of the two geometries.

  • If they were sortable using the X coordinate of the minima, that was the sorting returned.
  • If they were equal in X, then the Y coordinate of the minima was used.
  • Etc.

This process returned a sorted order, but not a very satisfying one: a “good” sorting would tend to place objects that are near to each other in space, near to each other in the sorted set.

Here’s what the old sorting looked like, applied to world populated places:

Geometry sorting in PostGIS 2.3

The new sorting system for PostGIS 2.4 calculates a very simple “morton key” using the center of the bounds of a feature, keeping things simple for performance reasons. The result is a sorted order that tends to keep spatially nearby features closer together in the sorted set.

Geometry sorting in PostGIS 2.4

Just as the “=” operator is tied to the SQL GROUP BY operation, the “>” and “<” operators are tied to the SQL ORDER BY operation. The pictures above were created by generating a line string from the populated places points as follows:

CREATE TABLE places_line AS 
SELECT ST_MakeLine(geom ORDER BY geom) AS geom 
FROM places;

September 14, 2017 04:00 PM

September 13, 2017

PostGIS Development

PostGIS 2.4.0rc1 Released

The PostGIS development team is pleased to announce the release of PostGIS 2.4.0rc1. Best served with PostgreSQL 10beta4 or rc1 which is coming soon and pgRouting 2.5.0rc 2.5.0 release is eminent. See the full list of changes in the news file.

We encourage everyone to test and in particular package maintainers to insure no surprises at final release time.

IMPORTANT NOTES

If you are upgrading from an existing PostGIS install, make sure after installing PostGIS 2.4.0rc1 binaries to do.

ALTER EXTENSION postgis UPDATE;

— if you have additional postgishy extensions below upgrade them too

ALTER EXTENSION postgis_sfcgal UPDATE;
ALTER EXTENSION postgis_topology UPDATE;
ALTER EXTENSION postgis_tiger_geocoder UPDATE;
--pgRouting 2.5.0 is imminent
ALTER EXTENSION pgrouting UPDATE;

In order to have Map Box Vector Tiles support enabled, you’ll need to compile with protobuf support and pkg-config to verify the correct minimum version of protobuf-c see protobuf for details. ST_FrechetDistance function will not be enabled if PostGIS is compiled with lower than GEOS 3.7.0. GEOS 3.7.0 will probably not be released before PostGIS 2.4.0 is out.

Continue Reading by clicking title hyperlink ..

by Regina Obe at September 13, 2017 12:00 AM

September 11, 2017

Anita Graser (Underdark)

Drive-time Isochrones from a single Shapefile using QGIS, PostGIS, and Pgrouting

This is a guest post by Chris Kohler .

Introduction:

This guide provides step-by-step instructions to produce drive-time isochrones using a single vector shapefile. The method described here involves building a routing network using a single vector shapefile of your roads data within a Virtual Box. Furthermore, the network is built by creating start and end nodes (source and target nodes) on each road segment. We will use Postgresql, with PostGIS and Pgrouting extensions, as our database. Please consider this type of routing to be fair, regarding accuracy, as the routing algorithms are based off the nodes locations and not specific addresses. I am currently working on an improved workflow to have site address points serve as nodes to optimize results. One of the many benefits of this workflow is no financial cost to produce (outside collecting your roads data). I will provide instructions for creating, and using your virtual machine within this guide.

Steps:–Getting Virtual Box(begin)–

Intro 1. Download/Install Oracle VM(https://www.virtualbox.org/wiki/Downloads)

Intro 2. Start the download/install OSGeo-Live 11(https://live.osgeo.org/en/overview/overview.html).

Pictures used in this workflow will show 10.5, though version 11 can be applied similarly. Make sure you download the version: osgeo-live-11-amd64.iso. If you have trouble finding it, here is the direct link to the download (https://sourceforge.net/projects/osgeo-live/files/10.5/osgeo-live-10.5-amd64.iso/download)
Intro 3. Ready for virtual machine creation: We will utilize the downloaded OSGeo-Live 11 suite with a virtual machine we create to begin our workflow. The steps to create your virtual machine are listed below. Also, here are steps from an earlier workshop with additional details with setting up your virtual machine with osgeo live(http://workshop.pgrouting.org/2.2.10/en/chapters/installation.html).

1.  Create Virutal Machine: In this step we begin creating the virtual machine housing our database.

Open Oracle VM VirtualBox Manager and select “New” located at the top left of the window.

VBstep1

Then fill out name, operating system, memory, etc. to create your first VM.

vbstep1.2

2. Add IDE Controller:  The purpose of this step is to create a placeholder for the osgeo 11 suite to be implemented. In the virtual box main window, right-click your newly-created vm and open the settings.

vbstep2

In the settings window, on the left side select the storage tab.

Find “adds new storage controller button located at the bottom of the tab. Be careful of other buttons labeled “adds new storage attachment”! Select “adds new storage controller button and a drop-down menu will appear. From the top of the drop-down select “Add IDE Controller”.

vbstep2.2

vbstep2.3

You will see a new item appear in the center of the window under the “Storage Tree”.

3.  Add Optical Drive: The osgeo 11 suite will be implemented into the virtual machine via an optical drive. Highlight the new controller IDE you created and select “add optical drive”.

vbstep3

A new window will pop-up and select “Choose Disk”.

vbstep3.2

Locate your downloaded file “osgeo-live 11 amd64.iso” and click open. A new object should appear in the middle window under your new controller displaying “osgeo-live-11.0-amd64.iso”.

vbstep3.3

Finally your virtual machine is ready for use.
Start your new Virtual Box, then wait and follow the onscreen prompts to begin using your virtual machine.

vbstep3.4

–Getting Virtual Box(end)—

4. Creating the routing database, and both extensions (postgis, pgrouting): The database we create and both extensions we add will provide the functions capable of producing isochrones.

To begin, start by opening the command line tool (hold control+left-alt+T) then log in to postgresql by typing “psql -U user;” into the command line and then press Enter. For the purpose of clear instruction I will refer to database name in this guide as “routing”, feel free to choose your own database name. Please input the command, seen in the figure below, to create the database:

CREATE DATABASE routing;

You can use “\c routing” to connect to the database after creation.

step4

The next step after creating and connecting to your new database is to create both extensions. I find it easier to take two-birds-with-one-stone typing “psql -U user routing;” this will simultaneously log you into postgresql and your routing database.

When your logged into your database, apply the commands below to add both extensions

CREATE EXTENSION postgis;
CREATE EXTENSION pgrouting;

step4.2

step4.3

5. Load shapefile to database: In this next step, the shapefile of your roads data must be placed into your virtual machine and further into your database.

My method is using email to send myself the roads shapefile then download and copy it from within my virtual machines web browser. From the desktop of your Virtual Machine, open the folder named “Databases” and select the application “shape2pgsql”.

step5

Follow the UI of shp2pgsql to connect to your routing database you created in Step 4.

step5.2

Next, select “Add File” and find your roads shapefile (in this guide we will call our shapefile “roads_table”) you want to use for your isochrones and click Open.

step5.3

Finally, click “Import” to place your shapefile into your routing database.

6. Add source & target columns: The purpose of this step is to create columns which will serve as placeholders for our nodes data we create later.

There are multiple ways to add these columns into the roads_table. The most important part of this step is which table you choose to edit, the names of the columns you create, and the format of the columns. Take time to ensure the source & target columns are integer format. Below are the commands used in your command line for these functions.

ALTER TABLE roads_table ADD COLUMN "source" integer;
ALTER TABLE roads_table ADD COLUMN "target" integer;

step6

step6.2

7. Create topology: Next, we will use a function to attach a node to each end of every road segment in the roads_table. The function in this step will create these nodes. These newly-created nodes will be stored in the source and target columns we created earlier in step 6.

As well as creating nodes, this function will also create a new table which will contain all these nodes. The suffix “_vertices_pgr” is added to the name of your shapefile to create this new table. For example, using our guide’s shapefile name , “roads_table”, the nodes table will be named accordingly: roads_table_vertices_pgr. However, we will not use the new table created from this function (roads_table_vertices_pgr). Below is the function, and a second simplified version, to be used in the command line for populating our source and target columns, in other words creating our network topology. Note the input format, the “geom” column in my case was called “the_geom” within my shapefile:

pgr_createTopology('roads_table', 0.001, 'geom', 'id',
 'source', 'target', rows_where := 'true', clean := f)

step7

Here is a direct link for more information on this function: http://docs.pgrouting.org/2.3/en/src/topology/doc/pgr_createTopology.html#pgr-create-topology

Below is an example(simplified) function for my roads shapefile:

SELECT pgr_createTopology('roads_table', 0.001, 'the_geom', 'id')

8. Create a second nodes table: A second nodes table will be created for later use. This second node table will contain the node data generated from pgr_createtopology function and be named “node”. Below is the command function for this process. Fill in your appropriate source and target fields following the manner seen in the command below, as well as your shapefile name.

To begin, find the folder on the Virtual Machines desktop named “Databases” and open the program “pgAdmin lll” located within.

step8

Connect to your routing database in pgAdmin window. Then highlight your routing database, and find “SQL” tool at the top of the pgAdmin window. The tool resembles a small magnifying glass.

step8.2

We input the below function into the SQL window of pgAdmin. Feel free to refer to this link for further information: (https://anitagraser.com/2011/02/07/a-beginners-guide-to-pgrouting/)

CREATE TABLE node AS
   SELECT row_number() OVER (ORDER BY foo.p)::integer AS id,
          foo.p AS the_geom
   FROM (     
      SELECT DISTINCT roads_table.source AS p FROM roads_table
      UNION
      SELECT DISTINCT roads_table.target AS p FROM roads_table
   ) foo
   GROUP BY foo.p;

step8.3

  1.  Create a routable network: After creating the second node table from step 8,  we will combine this node table(node) with our shapefile(roads_table) into one, new, table(network) that will be used as the routing network. This table will be called “network” and will be capable of processing routing queries.  Please input this command and execute in SQL pgAdmin tool as we did in step 8. Here is a reference for more information:(https://anitagraser.com/2011/02/07/a-beginners-guide-to-pgrouting/)   

step8.2

 

CREATE TABLE network AS
   SELECT a.*, b.id as start_id, c.id as end_id
   FROM roads_table AS a
      JOIN node AS b ON a.source = b.the_geom
      JOIN node AS c ON a.target = c.the_geom;

step9.2

10. Create a “noded” view of the network:  This new view will later be used to calculate the visual isochrones in later steps. Input this command and execute in SQL pgAdmin tool.

CREATE OR REPLACE VIEW network_nodes AS 
SELECT foo.id,
 st_centroid(st_collect(foo.pt)) AS geom 
FROM ( 
  SELECT network.source AS id,
         st_geometryn (st_multi(network.geom),1) AS pt 
  FROM network
  UNION 
  SELECT network.target AS id, 
         st_boundary(st_multi(network.geom)) AS pt 
  FROM network) foo 
GROUP BY foo.id;

step10

11.​ Add column for speed:​ This step may, or may not, apply if your original shapefile contained a field of values for road speeds.

In reality a network of roads will typically contain multiple speed limits. The shapefile you choose may have a speed field, otherwise the discrimination for the following steps will not allow varying speeds to be applied to your routing network respectfully.

If values of speed exists in your shapefile we will implement these values into a new field, “traveltime“, that will show rate of travel for every road segment in our network based off their geometry. Firstly, we will need to create a column to store individual traveling speeds. The name of our column will be “traveltime” using the format: ​double precision.​ Input this command and execute in the command line tool as seen below.

ALTER TABLE network ADD COLUMN traveltime double precision;

step11

Next, we will populate the new column “traveltime” by calculating traveling speeds using an equation. This equation will take each road segments geometry(shape_leng) and divide by the rate of travel(either mph or kph). The sample command I’m using below utilizes mph as the rate while our geometry(shape_leng) units for my roads_table is in feet​. If you are using either mph or kph, input this command and execute in SQL pgAdmin tool. Below further details explain the variable “X”.

UPDATE network SET traveltime = shape_leng / X*60

step11.2

How to find X​, ​here is an example​: Using example 30 mph as rate. To find X, we convert 30 miles to feet, we know 5280 ft = 1 mile, so we multiply 30 by 5280 and this gives us 158400 ft. Our rate has been converted from 30 miles per hour to 158400 feet per hour. For a rate of 30 mph, our equation for the field “traveltime”  equates to “shape_leng / 158400*60″. To discriminate this calculations output, we will insert additional details such as “where speed = 30;”. What this additional detail does is apply our calculated output to features with a “30” value in our “speed” field. Note: your “speed” field may be named differently.

UPDATE network SET traveltime = shape_leng / 158400*60 where speed = 30;

Repeat this step for each speed value in your shapefile examples:

UPDATE network SET traveltime = shape_leng / X*60 where speed = 45;
UPDATE network SET traveltime = shape_leng / X*60 where speed = 55;

The back end is done. Great Job!

Our next step will be visualizing our data in QGIS. Open and connect QGIS to your routing database by right-clicking “PostGIS” in the Browser Panel within QGIS main window. Confirm the checkbox “Also list tables with no geometry” is checked to allow you to see the interior of your database more clearly. Fill out the name or your routing database and click “OK”.

If done correctly, from QGIS you will have access to tables and views created in your routing database. Feel free to visualize your network by drag-and-drop the network table into your QGIS Layers Panel. From here you can use the identify tool to select each road segment, and see the source and target nodes contained within that road segment. The node you choose will be used in the next step to create the views of drive-time.

12.Create views​: In this step, we create views from a function designed to determine the travel time cost. Transforming these views with tools will visualize the travel time costs as isochrones.

The command below will be how you start querying your database to create drive-time isochrones. Begin in QGIS by draging your network table into the contents. The visual will show your network as vector(lines). Simply select the road segment closest to your point of interest you would like to build your isochrone around. Then identify the road segment using the identify tool and locate the source and target fields.

step12

step12.2

Place the source or target field value in the below command where you see ​VALUE​, in all caps​.

This will serve you now as an isochrone catchment function for this workflow. Please feel free to use this command repeatedly for creating new isochrones by substituting the source value. Please input this command and execute in SQL pgAdmin tool.

*AT THE BOTTOM OF THIS WORKFLOW I PROVIDED AN EXAMPLE USING SOURCE VALUE “2022”

CREATE OR REPLACE VIEW "​view_name" AS 
SELECT di.seq, 
       di.id1, 
       di.id2, 
       di.cost, 
       pt.id, 
       pt.geom 
FROM pgr_drivingdistance('SELECT
     gid::integer AS id, 
     Source::integer AS source, 
     Target::integer AS target,                                    
     Traveltime::double precision AS cost 
       FROM network'::text, ​VALUE::bigint, 
    100000::double precision, false, false)
    di(seq, id1, id2, cost)
JOIN network_nodes pt ON di.id1 = pt.id;

step12.3

13.Visualize Isochrone: Applying tools to the view will allow us to adjust the visual aspect to a more suitable isochrone overlay.

​After creating your view, a new item in your routing database is created, using the “view_name” you chose. Drag-and-drop this item into your QGIS LayersPanel. You will see lots of small dots which represent the nodes.

In the figure below, I named my view “take1“.

step13

Each node you see contains a drive-time value, “cost”, which represents the time used to travel from the node you input in step 12’s function.

step13.2

Start by installing the QGIS plug-in Interpolation” by opening the Plugin Manager in QGIS interface.

step13.3

Next, at the top of QGIS window select “Raster” and a drop-down will appear, select “Interpolation”.

step13.4

 

A new window pops up and asks you for input.

step13.5

Select your “​view”​ as the​ vector layer​, select ​”cost​” as your ​interpolation attribute​, and then click “Add”.

step13.6

A new vector layer will show up in the bottom of the window, take care the type is Points. For output, on the other half of the window, keep the interpolation method as “TIN”, edit the ​output file​ location and name. Check the box “​Add result to project​”.

Note: decreasing the cellsize of X and Y will increase the resolution but at the cost of performance.

Click “OK” on the bottom right of the window.

step13.7

A black and white raster will appear in QGIS, also in the Layers Panel a new item was created.

step13.8

Take some time to visualize the raster by coloring and adjusting values in symbology until you are comfortable with the look.

step13.9

step13.10

14. ​Create contours of our isochrone:​ Contours can be calculated from the isochrone as well.

Find near the top of QGIS window, open the “Raster” menu drop-down and select Extraction → Contour.

step14

Fill out the appropriate interval between contour lines but leave the check box “Attribute name” unchecked. Click “OK”.

step14.2

step14.3

15.​ Zip and Share:​ Find where you saved your TIN and contours, compress them in a zip folder by highlighting them both and right-click to select “compress”. Email the compressed folder to yourself to export out of your virtual machine.

Example Isochrone catchment for this workflow:

CREATE OR REPLACE VIEW "2022" AS 
SELECT di.seq, Di.id1, Di.id2, Di.cost,                           
       Pt.id, Pt.geom 
FROM pgr_drivingdistance('SELECT gid::integer AS id,                                       
     Source::integer AS source, Target::integer AS target, 
     Traveltime::double precision AS cost FROM network'::text, 
     2022::bigint, 100000::double precision, false, false) 
   di(seq, id1, id2, cost) 
JOIN netowrk_nodes pt 
ON di.id1 = pt.id;

References: Virtual Box ORACLE VM, OSGeo-Live 11  amd64 iso, Workshop FOSS4G Bonn(​http://workshop.pgrouting.org/2.2.10/en/index.html​),


by ckohler4692 at September 11, 2017 08:01 PM

September 02, 2017

PostGIS Development

PostGIS 2.4.0beta1 Released

The PostGIS development team is pleased to announce the release of PostGIS 2.4.0beta1. PostGIS 2.4.0 will be the first version to support PostgreSQL 10. Best served with PostgreSQL 10beta4 and pgRouting 2.5.0beta See the full list of changes in the news file.

We encourage everyone to test and in particular package maintainers to insure no surprises at final release time.

IMPORTANT NOTES

From this point forward until release of PostGIS 2.4.0, no new functions will be added.
Only bug fixes will be addressed in future 2.4.0 betas and rcs.

In order to have Map Box Vector Tiles support enabled, you’ll need to compile with protobuf support and pkg-config to verify the correct minimum version of protobuf-c see protobuf for details. ST_FrechetDistance function will not be enabled if PostGIS is compiled with lower than GEOS 3.7.0. GEOS 3.7.0 will be released around the same time as PostGIS 2.4.0 and will have a beta release in a week.

Continue Reading by clicking title hyperlink ..

by Regina Obe at September 02, 2017 12:00 AM

August 20, 2017

Boston GIS (Regina Obe, Leo Hsu)

geography type is not limited to earth

It's a little known fact that the PostGIS geography type since PostGIS 2.2 (well was introduced in 2.1 I think but had a bug in it until 2.1.4 or so), supports any spheroidal spatial reference system that measures in degrees. When an srid is not specified, it defaults to using 4326, Earth WGS 84 long lat.


Continue reading "geography type is not limited to earth"

by Regina Obe (nospam@example.com) at August 20, 2017 02:50 PM

August 05, 2017

PostGIS Development

PostGIS 2.4.0alpha Released

The PostGIS development team is pleased to announce the release of PostGIS 2.4.0alpha. This is the first version to support PostgreSQL 10. Best served with PostgreSQL 10beta2. See the full list of changes in the news file.

Continue Reading by clicking title hyperlink ..

by Regina Obe at August 05, 2017 12:00 AM

August 01, 2017

Boston GIS (Regina Obe, Leo Hsu)

Code Sprint in Boston after FOSS4G 2017

Reminder: Right after the Free and Open Source GIS conference in Boston is the OSGeo / LocationTech code sprint on Saturday August 19th 9AM-5PM at District Hall where project members from various Open Source Geospatial projects will be fleshing out ideas, documenting, coding, and introducing new folks to open source development. All are welcome including those who are unable to make the conference.

We are getting a final head-count this week to plan for food arrangements. If you are planning to attend, add your name to the list https://wiki.osgeo.org/wiki/FOSS4G_2017_Code_Sprint#Registered_Attendees. If you are unable to add your name to the list, feel free to send Regina an email at lr@pcorp.us with your name and projects you are interested in so I can add you to the list. Looking forward to hanging out with folks interested in PostgreSQL and PostGIS development.

District Hall is a gorgeous community space. Check out the District Hall View http://bit.ly/2f61J8c

by Regina Obe (nospam@example.com) at August 01, 2017 09:22 PM

July 31, 2017

BigSQL Holly

BigSQL + PostGIS: Reprojecting Your Spatial Data with ST_Transform

With our latest release of PostGIS (release 2.3.3), projection datum shift files have been included across all the platforms. You will find them located in: bigsql/pg96/share/postgresql/contrib/postgis-2.3/proj Why do you care? Well, this means you can easily reproject your PostGIS data using the ST_Transform command. > ST_Transform — Return a new geometry with its coordinates transformed to a different spatial reference.

Exercise Prerequisites

Previous knowledge of projections and spatial references are assumed for this tutorial. To learn more about projections, I recommend these resources:

Previous blog posts to help you get setup with PostGIS:

What we will do in this exercise

Suppose we would like to find all the subway entrances that are withing 220 meters (about 2 football fields as the crow flies) to the Empire State Building (located at 40.748855, -73.985773).

A good projection for measuring distances in NYC is based on the Transverse Mercator projection: UTM Zone 19N (srid 32619).

Unfortunately, the data available by the MTA, Subway Entrances, has a spatial reference of WSGS:84 (srid 4326). This is a good choice if you want a map which shows the whole world, but not for measuring distances within New York City.

So.. we need to:

  • import the shapefile into our PostGIS database
  • create a copy of the table and reproject it
  • select all subway entrances located within 220 meters of the Empire State building converted from lat, long (40.748855, -73.985773) to it’s location in UTM Zone 19N.

Import Shapefile to PostGIS

I’ve made it easier for you by providing the data for you to download here: subway_entrances.zip

Unzip the shapefile, navigate to the location of your subway_entrances.shp file, and run the following command:

shp2pgsql -I -s 4326 subway_entrances.shp subway_entrances | psql -U postgres -d postgis_sample

Connect to your database to make sure it was successfully imported. If using Windows, you may need to include -U postgres with the psql command:

psql 
postgres=# \connect postgis_sample;

See that subway_entrances table is in the database:

postgis_sample=# \dt

Now look at the attributes in subway_entrances table:

\d+ subway_entrances

Column Type Modifiers Storage Stats target Description
gid integer not null default nextval(‘subwayentrancesgid_seq’::regclass) plain
objectid numeric main
url character varying(254) extended
name character varying(254) extended
line character varying(254) extended
geom geometry(Point,4326) main

Reproject subway_entrances from srid 4326 to 32619

Create a table subway_entrances_utm that is a duplicate of subway_entrances:

CREATE TABLE subway_entrances_utm AS SELECT * FROM subway_entrances;

Reproject the new table to UTM Zone 19N (srid 32619):

ALTER TABLE subway_entrances_utm
    ALTER COLUMN geom TYPE geometry(POINT,32619) 
    USING ST_Transform(geom,32619);

Select stations 220 meters from Empire State Building

Run the following query composed of these PostGIS commands:

  • ST_DWithin
  • ST_Transform
  • ST_SetSRID
  • ST_MakePoint

    SELECT name, line
    FROM subway_entrances_utm
    WHERE ST_DWithin(geom,ST_Transform(ST_SetSRID(ST_MakePoint(-73.985773,40.748855),4326),32619),220);
    

Your result should show 7 rows, including one having no value for the name field:

name               |     line
---------------------------------+---------------
 6th Ave & 34th St at SE corner  | B-D-F-M-N-Q-R
 6th Ave & 34th St at NE corner  | B-D-F-M-N-Q-R
 Broadway & 32nd St at NE corner | B-D-F-M-N-Q-R
 Broadway & 32nd St at NW corner | B-D-F-M-N-Q-R
 6th Ave & 35th St at SE corner  | B-D-F-M-N-Q-R
 6th Ave & 35th St at NE corner  | B-D-F-M-N-Q-R
                                 | B-D-F-M-N-Q-R

If you want, you can create a new table, subway_entrances_empire, that only contains these subway stations:

CREATE TABLE subway_entrances_empire AS SELECT *
        FROM subway_entrances_utm
        WHERE ST_DWithin(geom,ST_Transform(ST_SetSRID(ST_MakePoint(-73.985773,40.748855),4326),32619),220);

by Holly Orr at July 31, 2017 10:18 PM

July 29, 2017

Boston GIS (Regina Obe, Leo Hsu)

Using OSGeoLive with VirtualBox

FOSS4G 2017 is just a few weeks away. Many of the workshops will utilize OSGeoLive as a means for workshop participants to get up and running quickly with OSGeo Free and Open Source GIS tools and Boston data. OSGeoLive11 is a LUbuntu 16.04 distribution. OSGeoLive11 is going thru the final stages of prep. You can download the OSGeoLiveRC1 ISO for it from http://aiolos.survey.ntua.gr/gisvm/11.0/osgeo-live-11.0rc1-amd64.iso.

Once OSGeoLive11 is fully prepped, it will be linked on the osgeolive site http://live.osgeo.org. If you want to run OSGeoLive11 from bootable media, you can burn the ISO to DVD or thumb drive and boot. In final prep, you can expect to have a Virtual Image ready to go you can host on Virtual Box or VMWare and make further customizations. OSGeoLive11 thumb drives will be handed out at the conference.

If you are doing any PostgreSQL/ PostGIS / pgRouting / or other GIS training, OSGeoLive is pretty handy. OSGeoLive11 contains PostgreSQL 9.5 (I know slightly dated) , PostGIS 2.3.2 and cli tools, pgRouting 2.4.1, and osm2pgrouting 2.2.0. In addition it contains popular GIS Desktop friends QGIS, OpenJump, gvSig, uDig as well as power tools like GRASS, R, GDAL CLI toolkit, and Jupyter notebooks. Mapping Servers MapServer and GeoServer. We'll be using pgRouting, osm2pgRouting, PostGIS, PostgreSQL, QGIS, and OpenJump in our workshop Problem Solving with pgRouting. A good chunk of FOSS GIS relies on PostgreSQL via PostGIS so you'll find a lot of already setup PostgreSQL databases on this disk.

For this set of exercises, we're going to go thru using the ISO media linked above on a Windows 7 VirtualBox setup. If you are using any other OS (e.g. Mac OSX, Linux, Unix), instructions should be much the same.


Continue reading "Using OSGeoLive with VirtualBox"

by Regina Obe (nospam@example.com) at July 29, 2017 07:55 AM

July 24, 2017

BigSQL Holly

PostGIS 2.3.3 is Now Available with BigSQL!

The good folks on the PostGIS development team recently the released of PostGIS 2.3.3:

As befits a patch release, the focus is on bugs and breakages. Best served with PostgreSQL 9.6.3+ and pgRouting 2.4.1.

See the full list of changes here.

And now… the new release is available in the BigSQL distribution!

If you haven’t installed the BigSQL distribution and setup your PostGIS database start here:
PostGIS – How to create a Spatial Database with PGC Command Line

If you have been using our distribution (good for you!) you can easily upgrade to the latest release with the following command lines:

./pgc update
./pgc upgrade postgis23-pg96

And then connect to database you want to upgrade and run the following command:

ALTER EXTENSION postgis UPDATE;

Now check to make sure your distribution has been updates:

./pgc list

You should see the following row listed for the postgis23-pg96 extension:

Category Component Version Stage ReleaseDt Status Cur? Updates
Extensions postgis23-pg96 2.3.3-1 2017-07-13 Installed 1

The version number consists of:

  • 2.3.3 is the release number from the PostGIS development community
  • -1 is the BigSQL release where we have made improvements our fixes to our distribution

Keep a lookout for our dash releases as we make more improvements to our internal builds.

And… in fact, stay-tuned for an upcoming -2 release the improves our ST_Transform and adds JSON-C (ST_GeomFromGeoJson) functionality to the mix.

by Holly Orr at July 24, 2017 11:37 PM

July 12, 2017

BigSQL Holly

Setting Up LDAP with Active Directory in PostgreSQL

In the example below, we are setting up LDAP with Active Directory (AD), one of the more popular server implementations, using the BigSQL PostgreSQL 96 distribution. Available for Windows and Linux.

Download BigSQL Distribution

Linux:

python -c "$(curl -fsSL http://s3.amazonaws.com/pgcentral/install.py)" 

Windows:

@powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('http://s3.amazonaws.com/pgcentral/install.ps1'))"

Find the bigsql directory and install PostgreSQL. Windows users don’t prefix the pgc command with ./ as shown in the following commands:

cd bigsql
./pgc install pg96
./pgc start pg96

Set your environment variables:

cd <directory of installation>/pg96

Linux:

source pg96.env

Windows:

pg96-env.bat

Configure pg_hba.conf

From the PostgreSQL documentaton for pg_hba.conf:

Client authentication is controlled by the pg_hba.conf (HBA stands for host-based authentication.)

The general format of the pg_hba.conf file is a set of records, one per line. A record is made up of a number of fields which are separated by spaces and/or tabs. The fields in the each record specifies a connection type, a client IP address range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters.

Modify the LDAP connection record

There are many authentication methods in postgres (md5, trust, peer, etc). In this exercise, you will modify pg_hba.conf to use LDAP as the authentication method.

Navigate to the pg_hba.conf

cd bigsql/data/pg96/

Use your favorite editor to modify the line:

TYPE DATABASE USER CIDR-ADDRESS METHOD
host all all 127.0.0.1/32 trust

When we finish our exercise, the line will look something like this:

TYPE DATABASE USER CIDR-ADDRESS METHOD
host all all 127.0.0.1/32 ldap ldapserver=192.0.2.0 ldapprefix=”” ldapsuffix=”@oscg.com”

Modifying the METHOD field

The METHOD field provides connection credentials for Active Directory. There are 4 parameters you will need to create:

  1. ldap (the authentication method)
  2. ldapserver
  3. ldapprefix
  4. ldapsuffix

ldapserver

You can enter either the Active Directory server name or the ip address. For example:

ldapserver=192.0.2.0 

or

ldapserver=oscg.com

‘ldapprefix’ and ‘ldapsuffix’

Active Directory uses the following structure for login:

  • “DOMAIN_NAME\USERNAME”
  • “USERNAME@DOMAIN_NAME”

The ‘ldapprefix’ and ‘ldapsuffix’ serve as the variables for storing the DOMAIN_NAME value.

So, if the domain name for our active directory instance is OSCG, then we can provide the domain and user login credentials in one of 2 ways:

  • ldapprefix=”OSCG\” (the domain name at login – Note: the backslash is required)
  • ldapsuffix=”@oscg.com” (the domain name of the Active Directory server – Note: the @ is required)

Then, when the user (e.g. Vice Admiral Kathryn Janeway) enters her username (e.g. “kjaneway”), it is prepended or appended to the domain name depending on which method you chose to store the DOMAIN_NAME value:

  • OSCG\kjaneway
  • kjaneway@oscg.com

Note: Use either ldapprefix or ldapsuffix, not both!

In this example, we have modified the ldappreffix parameter:

METHOD
ldap ldapserver=192.0.2.0 ldapprefix=”OSCG\” ldapsuffix=””

and in this example, we have modified the ldapsuffix parameter:

METHOD
ldap ldapserver=192.0.2.0 ldapprefix=”” ldapsuffix=”@oscg.com”

Note: The empty quotes are required for whichever method you are not using – ldapprefix or ldapsuffix

Now, replace the parameters we used in the examples with your own Active Directory credentials:

METHOD
ldap ldapserver=”IP ADDRESS OR DOMAIN NAME OF YOUR ACTIVE DIRECTORY SERVER” ldapprefix=”YOUR DOMAIN_NAME” ldapsuffix=””

or

METHOD
ldap ldapserver=”IP ADDRESS OR DOMAIN NAME OF YOUR ACTIVE DIRECTORY SERVER” ldapprefix=”” ldapsuffix=”@” + “YOUR ACTIVE DIRECTORY SERVER NAME”

There are many more modes and configuration options to choose from when setting up your LDAP authentication. To learn more go here.

Create role for the AD domain user

Login to postgres database as postgres user:

psql -U postgres     

Create role on postgres that exists in active directory:

CREATE ROLE "kjaneway" WITH LOGIN;

Exit psql

\q

Reload the postgres to load new pg_hba.conf

cd bigsql
./pgc reload pg96

Login to postgres with Active Directory/Domain User

psql -h 127.0.0.1 -d postgres -U "kjaneway" 

Now, you should be prompted to enter a password. Enter the password you use for the active directory domain:

Password for user kjaneway: 

And, voila! You should now see the following prompt giving you access to the postgres database using LDAP:

psql (9.6.3)
Type "help" for help.

postgres=>

Final test – disable and enable Active Directory

Login to your Active Directory server and disable the domain account you have been testing against:

Connecting to the postgres database with the domain user and password should now fail:

psql -h 127.0.0.1 -d postgres -U "kjaneway" 
Password for user ldaptest: 
psql: FATAL:  LDAP authentication failed for user "kjaneway"

Now, enable the ‘holly.orr’ account in Active Directory:

Connecting to the postgres database with the domain user and password should now suceed:

psql -h 127.0.0.1 -d postgres -U "kjaneway" 
Password for user kjaneway: 
psql (9.6.3)
Type "help" for help.

openscg=# 

The What and Why of LDAP: A Little Context to Help You Understand

Remember phone books? Those big heavy directories, ordered alphabetically, that you would heave onto your kitchen table and thumb through to find a friend’s phone number and address?

With the advent of the internet and email, a new virtual phonebook was in order. One that allowed you look up an address for someone who’s never sent you email or enabled your organization to keep one centralized up-to-date “phone book” that permissable users had access to.

The solution came in the 1980’s. After 70 years of honing the art of managing phone books, the Telecommunications companies answered the call (get it) by applying the concept of directory services to information technology and computer networking. This resulted in the comprehensive X.500 specification.

In the 1990’s, geeks from the University of Michigan (of course) created a alternative protocal, LDAP, Lightweight Directory Access Protocol. As the name suggests, LDAP was lightweight compared to it’s predeccesor x.500 and quickly became widely adopted

Today, LDAP is a staple of enterprise IT infrastructures for authenticating and storing data about users. It is not limited to contact information and is used to look up encryption certificates, services on a network, and provide “single sign-on” where one password for a user is shared between many services.

Popular commercial LDAP server implementations include:

  • Microsoft Active Directory
  • UnboundID Directory Server
  • Oracle Internet Directory
  • Oracle Unified Directory
  • IBM Security Directory Server
  • NetIQ eDirectory
  • CA Directory

Popular open source LDAP server implementations include:

  • OpenLDAP
  • ForgeRock OpenDJ
  • Apache DS
  • 389 Directory Server

And lately, there have been Cloud LDAP solutions like JumpCloud or OneLogin. But this may come at a risk.

What are you using? Have you tried out Cloud solutions? Hit reply and tell us how you use LDAP with PostgreSQL.

by Holly Orr at July 12, 2017 12:17 PM

July 03, 2017

Postgres OnLine Journal (Leo Hsu, Regina Obe)

Installing pgTap in windows with msys2 and mingw64

This weekend we spent sometime moving PostGIS/pgRouting windows buildbot Winnie to new hardware. Leo did the hardware and I handled installing and reconfiguring stuff. While I was at it, I upgraded to new Jenkins. Vicky Vergara has been bugging me to setup pgTap so she can run her pgRouting pgTap tests to make sure they work on windows. She's got 22488 tests. She just loves pgTap. Last time I tried installing pgTap I gave up, but I was in mood for experimentation so gave it another chance.


Continue reading "Installing pgTap in windows with msys2 and mingw64"

by Leo Hsu and Regina Obe (nospam@example.com) at July 03, 2017 04:42 PM

July 01, 2017

PostGIS Development

PostGIS 2.3.3 Released

The PostGIS development team is pleased to announce the release of PostGIS 2.3.3 As befits a patch release, the focus is on bugs and breakages. Best served with PostgreSQL 9.6.3+ and pgRouting 2.4.1.

Continue Reading by clicking title hyperlink ..

by Regina Obe at July 01, 2017 12:00 AM

June 22, 2017

BigSQL Holly

pgAdmin3 LTS Now Supports PostgreSQL 8.4 – 10

We are pleased to announce pgAdmin3 LTS is now available on OSX and Windows for PostgreSQL 10!

As v1.0 of pgAdmin4 was released in September of 2016, the pgAdmin Development Team decided to no longer support pgAdmin III with any more fixes. We believe that pgAdmin3 usage should fade out over time and have forked the code to include basic support for PostgreSQL 9.6+.

Try it out!

Install pgAdmin3 LTS by BigSQL

If you haven’t already installed PostgreSQL, start here.

Via Command Line

Windows users don’t prefix the pgc command with ./ as shown in the following commands:

cd <directory of installation>
./pgc install pgadmin3
./pgc start pgadmin3

Via GUI

If you haven’t already, install pgDevOps.

In pgDevOps, navigate to Package Manager, select the pgAdmin3 component, and install.

Already have pgAdmin3 installed?

cd <directory of installation>
./pgc update
./pgc upgrade pgadmin3

Wanna delete pgAdmin3?

cd <directory of installation>
./pgc remove pgadmin3

As always, give us your feedback so that we can continue to support and respond to the needs of the PostgreSQL community.

by Holly Orr at June 22, 2017 06:15 PM

June 19, 2017

BigSQL Holly

GDAL New Release 2.2 – BigSQL Makes it Easy to Install

The GDAL community just announced the release of GDAL 2.2… and BigSQL just made it easier to implement!

Install it Now

We support Diegos and Yodas with 2 ways to install:

Option 1: Install with the pgDevOps GUI

For GIS analysts (scientists, statisticians, analysts, etc) who find themselves as the de facto DBA (among other things).

“I don’t have time for this! I’m saving this sea turtle. Just give me a GUI!”

1. Go to www.bigsql.org/postgresql/installers and download and run the installer for your operating system.

2. Make sure to check the box for including the pgDevOps component with your install.

3. Open pgDevOps in your web browser:

    <ip of machine>:8051

If installing on local machine:

    localhost:8051  
4. In the pgDevOps dashboard click the Package Manager icon.

5. Click the PostgreSQL installation you want to install on.

6. Click on the PostGIS icon and follow the instructions to install.

That’s it!

Option 2: Install via Command Line and PGC

For DBAs who find themselves supporting these strange spatially enabled databases.

“Real men only use command line! And what the hell is GIS?”

  1. Install pgc via command line to a sandbox.
    *Note: this command will install the BigSQL distribution into the directory you are currently located.

    MAC / Linux:

    python -c "$(curl -fsSL http://s3.amazonaws.com/pgcentral/install.py)" 
    

    Windows:

    @powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('http://s3.amazonaws.com/pgcentral/install.ps1'))"
    
  2. Get into the product home directory by navigating via command line to the bigsql directory. Windows users don’t prefix the pgc command with ./ as shown in the below examples:

    cd bigsql
    
  3. Using the pgc command line tool, run the update command to get the lastest distribution:

    ./pgc update
    
  4. Then run the install, init, and start commands on pg96:

    ./pgc install pg96 
    ./pgc start pg96
    
  5. Next, install postgis:

    ./pgc install postgis23-pg96
    

That’s it!

The GDAL_DATA Directory Path

In order for all your PostGIS extensions to properly work, you will need to set the GDAL_DATA environment variable to the location of the directory gdal.

Lucky for you, with the BigQL distribution this is as easy as running the following commands via a command line tool.

OSX:

 cd <directory of installation>/pg96
 source pg96.env

Windows:

 cd <directory of installation>\pg96
 pg96-env

Both of these commands set environment variables that will live during your current session. Note that if you close your terminal or command prompt, these variables are removed from memory.

It’s also possible to set GDAL_DATA as a persistant environment variable. But that is the beyond the scope of this tutorial.

To check to see what version of GDAL you have installed:

gdalinfo --version

WTH is GDAL?

In 1998, Frank Warmerdam began developing GDAL, which stands for Geospatial Data Abstraction Library. The official definition, found on gdal.org, is as follows:

GDAL is a translator library for raster and vector geospatial data formats that is released under an X/MIT style Open Source license by the Open Source Geospatial Foundation. As a library, it presents a single raster abstract data model and single vector abstract data model to the calling application for all supported formats. It also comes with a variety of useful command line utilities for data translation and processing.

Clear as mud, right? Let me refer to a better definition written by Planet Labs’ Robert Simmon in the post, A Gentle Introduction to GDAL, Part 1. I higly recommend reading the entire article:

Even the description is complicated. Put another way, GDAL (pronounced ‘gee-dal’ not ‘goodle’ (too close to Google!)) is a collection of software that helps with the translation of data from different file formats, data types, and map projections. It’s used in free software like QGIS and GRASS, and even some commercial applications like ESRI ArcGIS and Avenza Geographic Imager/MAPublisher.

WTH is OGR?

To make things even a bit more confusing, OGR (which I discussed in an earlier post, is part of GDAL:

In theory it is separate from GDAL, but currently they reside in the same source tree and are somewhat entangled.

So what’s the difference between GDAL and OGR?

The GDAL library is historically for raster data extract, transform, load (ETL) processes, while OGR supports vector data. Sometimes people will interchange the names GDAL, OGR, and GDAL/OGR to refer to the same thing. I know… geeks and their esoteric acronyms, right?

Regardless of the complexities, I can’t overemphasize how essential it is to add GDAL/OGR libraries to your spatial data management toolkit!

by Holly Orr at June 19, 2017 04:44 PM

June 17, 2017

Postgres OnLine Journal (Leo Hsu, Regina Obe)

Dollar-quoting for escaping single quotes

PostgreSQL has a feature called dollar-quoting, which allows you to include a body of text without escaping the single quotes. This feature has existed for quite some time. You've probably seen this in action when defining functions for example:

CREATE OR REPLACE FUNCTION hello_world(param_your_name text)
RETURNS text AS
$$
SELECT 'Hello world. My name is ' || param_your_name || '.';
$$
language sql STRICT;

Which is easier to read, than the equivalent escape quoted function:


CREATE OR REPLACE FUNCTION hello_world(param_your_name text)
RETURNS text AS
'
SELECT ''Hello world. My name is '' || param_your_name || ''.'';
'
language sql STRICT;

Continue reading "Dollar-quoting for escaping single quotes"

by Leo Hsu and Regina Obe (nospam@example.com) at June 17, 2017 11:54 PM

June 09, 2017

Bill Dollins

Refreshing a PostGIS Materialized View in FME

I am following up my previous post with an extremely simple example using FME to kick off the refresh of a materialized view (matview) after a data import. I had never used FME prior to coming to Spatial Networks, but now I’m hooked. I’m having a really hard time finding things it can’t do.

As I mentioned in my last post, it’s really easy to refresh a matview in PostgreSQL using the REFRESH MATERIALIZED VIEW statement. This leaves open the possibility of automating the refresh as appropriate in an application or other process.

I decided to illustrate this using a basic FME example. Using the cellular tower data set from my past post, I extracted a table containing only the records for the state of Maryland. The towers data set contains the two letter abbreviation for the state, but not the full state name. So, I built a matview to join the state name to a subset of columns from the towers data set. The SQL for that matview is here:

View the code on Gist.

I will use FME to append the records for the state of Virginia from a GeoJSON file to the PostGIS table containing the records for Maryland.

Disclaimer: This example wastes the power of FME. If this were all I needed to do, I’d probably just use OGR. I kept it simple for this post.

Here is a screenshot of my import process from FME workbench:

As can be seen, 656 records from the GeoJSON file will be appended to the PostGIS table.

To make FME run the REFRESH statement after the import, I just put the SQL into the advanced parameters for the PostGIS writer.

For demonstration purposes, I’ll run the import without the REFRESH statement. This will result in the source table having a different record count than the matview, as shown here.

Now, I’ll clear out the Virginia records from the base table, add the REFRESH statement back into FME, and rerun the process. This time, the table and the matview are in sync, as seen here.

This basic example illustrates one way to automate the refresh of matviews as part of a data update process. Whether using FME, GeoKettle, or your own application code, it’s very easy to keep matviews in sync.

It’s important to remember however, that every refresh is writing data to a new table. Care should be taken to tune how often matviews are refreshed to ensure that performance doesn’t suffer. They aren’t appropriate for high-velocity data sets, but can work nicely for data that is updated less frequently.

by Bill Dollins at June 09, 2017 06:04 PM

June 06, 2017

Bill Dollins

Working with Materialized Views in PostGIS

It’s been a few months since I’ve posted, owing mainly to getting my feet under me at Spatial Networks. About a month after I started, the company re-merged with Fulcrum, which had previously been spun off as a separate company. As a result, I’ve gotten to know the Fulcrum engineering team and have gotten to peer under the hood of the product.

Of course, Spatial Networks is also a data company. What had originally attracted me was the opportunity to help streamline the delivery of their data products, and this remains a pressing issue. This has kept me elbow-deep in PostGIS, and has led me to delve into using materialized views more than I have before.

What is a materialized view? If you are familiar with relational databases, then you are familiar with views, which are saved queries that are stored in the database. Similar to tables, you can select data from a view; but, rather than directly selecting physically stored data, you are executing the SQL that defines the view, which will stitch together data at the time of execution.

A materialized view is a useful hybrid of a table and a view. It is technically a table, because it is physically stored on disk, but it is generated from a SQL statement like a view. This is can be useful for increasing performance because costly joins and functions (ahem, spatial) are not executed every time the data is accessed. The source SQL is executed and the result written to disk. As a result, the data can be indexed and queries can executed quickly. It could be reasonably thought of as a more streamlined SELECT INTO.

So why use materialized views? If properly designed, a relational database (spatial or otherwise) is built using normalization to optimize storage, reduce data redundancy, provide constraints and enhance data quality. This is excellent for storage, but not always ideal for analytics, which is why views exist.

A materialized view provides the ability to prepare a persisted version of data that is better suited for analysis and/or human readability. In a spatial database such as PostGIS, it also provides the ability to pre-process spatial analysis to enhance database and application performance. That is what got me interested in them.

From here, it’s probably best to discuss materialized views in terms of a scenario. I’ll discuss a fairly straightforward example, data binning, which touches upon the advantages of materialized views for me.

I’ll work with four data sets:

  1. An extract of cellular towers from the FCC downloaded from ArcGIS Online
  2. US counties extracted from GADM
  3. 0.5-degree rectangular grid created in QGIS
  4. 0.5-degree hexagonal grid created in QGIS

I imported all the above into PostGIS.

My goal is to calculate the number of cellular tower in each grid square, hexagon, and county. I’ll use a fairly simple bit of SQL to do that.

View the code on Gist.

The SQL above generates a count of the towers that fall within each hex, as governed by the st_contains function and the GROUP BY clause. In a traditional view, this spatial query would executed every time I issued a SELECT statement against the view.

You will notice that I don’t return the hexagon geometry. I will create a second view that joins the geometry to the results of this query. I did that strictly to keep the SQL clean for this post, but it is perfectly fine to get all of the data in one step. The SQL that joins the hexagons is:

View the code on Gist.

Notice the left join so that all of the hexagons are returned regardless of whether or not they contain towers. I included the full CREATE statement this time to highlight the contrast with the next example.

It’s now time to create the materialized view, which calls upon the view created in the previous step. This is the SQL to create the materialized view:

View the code on Gist.

Note the creation of the spatial index. This is important because a materialized view creates a new data set, so the data is divorced from any indices in the source data.

The SQL above physically writes the data to the database, similar to a SELECT INTO statement. Unlike that approach however, we can easily refresh the data by issuing the following command:

REFRESH MATERIALIZED VIEW mvw_cellular_count_geom_hex;

This enables a user or application to automatically updated the stored data whenever the underlying source data changes.

Notice in the SQL above, I am calculating a UUID column. This is being done to aid visualization in QGIS. To add a view of any kind to a QGIS map, you need to tell it what the unique identifier is for the data set. I tried numerous different approaches, but found using a UUID to work most consistently. I will keep exploring this. The binned hex grid layer (with a bit of styling) looks like this in QGIS:

I mentioned earlier that materialized views can help performance. This really becomes apparent with complex geometries. There is minimal difference between the standard view and the materialized view when working with squares or hexes. But, to produce the county map shown at the top of this post, the standard view took 4.2 seconds to run on a Linux machine with quad-core, SSD, and 64GB of RAM. The materialized view returned in 292 milliseconds. This is where not having to re-run spatial queries using the details GADM polygons really pays off.

Materialized views are not a panacea. If you have rapidly updating data, the refresh process with probably introduce too much latency. You are also storing data, such as geometries, twice. But, depending on your needs, the benefits may outstrip the drawbacks of using materialized view in spatial applications.

by Bill Dollins at June 06, 2017 09:38 PM

Oslandia

OSGeo CS 2016 report : PostGIS

The annual OSGeo Codesprint took place from 23d to 26th of February in Paris, at Mozilla’s Foundation offices. Here is a summary of some achievements done during this event by the PostGIS team.

PostGIS

A big team of PostGIS developers worked together during the sprint on numerous issues.

New functions and dev guide

Remi Cura from IGN had prepared a full list of improvements to existing PostGIS functions and new features to develop. Together with Marc Ducobu ( Champs Libres ) , they managed to go all the way from simple PostGIS users to PostGIS developers. They dived into the code and managed to write clean patches for functions and documentation.

They documented their work and created a step-by-step guide for new PostGIS contributors.

New clustering features


New features have been introduced for PostGIS 2.3, among them clustering methods : ST_ClusterKMeans() and ST_ClusterDBSCAN() by Paul Ramsey and Dan Baston. These are really nice Window functions for PostGIS !
On a side note, the documentation for PostGIS Window functions has been improved too ( R. Obe )

OGR ODBC FDW

Regina Obe managed to travel to Paris, and she put efforts analyzing why the ODBC performances of the ogr_fdw PostGIS foreign data wrapper were not as good as expected. This should lead to performance improvement later on. She also worked on problems encountered when restoring PostGIS data ( especially rasters), related to how the search_path is handled. There is still some work to experiment with EVENT TRIGGERS, but the issue is on a good path to be solved.

3D in PostGIS : SFCGAL

 

Our team at Oslandia has been working on SFCGAL, the 3D library behind PostGIS 3D features. Vincent Mora and Hugo Mercier teamed with Mickael Borne ( IGN ) and Sébastien Loriot ( GeometryFactory ) to break things and rebuild them, better. They focused mainly on performances:

  • different computation and precision kernels in CGAL, to use them efficiently and be much faster
  • set a flag for valid geometries, so as not to re-test for validity in every operation
  • lower serialization overhead when passing data between PostGIS, CGAL and SFCGAL

This effort lead to significant improvement in speed. Preliminary tests on 3D intersection showed improvement from 15 seconds down to 100 milliseconds, which is impressive result.
A lot of refactoring of the code has to be done, and this work also started, to simplify and ease the use of SFCGAL.

New indexes for big data

Another significant contribution is the work on BRIN indexes for PostGIS. At PGConf Europe, we already spoke with the team at Dalibo about the new index type in PostgreSQL 9.5 : BRIN use cases and implementation in a GIS context. Some time before OSGeo code sprint, we realized that Giuseppe Broccolo from 2ndQuadrant had started a prototype implementation, so we invited him to join. Giuseppe teamed with Ronan Dunklau and Julien Rouhaud from Dalibo, and together they managed to have a working implementation of this new type of indexes for PostGIS.

Having the opportunity for PostgreSQL developers to meet in this geospatial environment was the ideal way to get things done efficiently.

PostGIS BRIN will be detailled after some code consolidation and benchmarks, but here is the idea. BRIN indexes arepartial indexes: they deal with data blocks ( a given amount of pages ) and not rows. You can set the number of pages per block. This makes indexes much faster to build, and a lot smaller than classic indexes ( GiST for PostGIS ). Of course, the tradeoff is that they are slower to query. And since they group data in blocks, they loose their efficiency if the data is not clustered on disk.

Some of our use cases, with tables full of Billions of points ( using PgPointCloud ), are a good match for BRIN indexes : the data is naturally clustered spatially ( patches of points are loaded sequencially ), and sometimes the data is too big for a table’s index to fit entirely in memory. E.g. we have a 3TB table, resulting in a 20GB GiST index.

A BRIN index can be a really good compromise for this : preliminary tests on a medium-size dataset show a performance degradation of 5x in querying, while the index is 1000x smaller. And this tradeoff is adjustable, according to your dataset size and hardware configuration.

And more

Other topics which have been worked on (mainly by Paul, see his blog post) :

  • Expanded object header, to avoid serialization / deserialization steps
  • A nasty upgrade bug ( 2.2 blocker )

You will find more details in the following reports :

  • Paul Ramsey – http://blog.cleverelephant.ca/2016/03/paris-code-sprint-postgis-recap.html
  • Giuseppe Broccolo – http://blog.2ndquadrant.com/brin-postgis-codesprint2016-paris/
  • Regina Obe (PostgresOnLine) – http://www.postgresonline.com/journal/archives/363-Paris-OSGeo-Code-Sprint-2016-Highlights.html
  • Regina Obe (BostonGIS) – http://www.bostongis.com/blog/index.php?/archives/252-Paris-OSGeo-Code-Sprint-2016-PostGIS-Highlights.html

Thanks to the PostGIS team for such hard work and for their reports !

Figures credits : Openstreetmap’s Soup, OpenScienceMap, Wikipedia

 

by oslandia at June 06, 2017 09:54 AM

June 05, 2017

PostGIS Development

Causes for 'postgis.backend' is already set

The error ‘postgis.backend’ is already set comes up every so often in PostGIS mailing list. The issue arises often during or after an upgrade. I’ll go over causes for this I am aware of and how to fix.

The question goes something like this

After upgrading to Postgis 2.3 from 2.1, my server log is filled with these messages :

“WARNING ‘postgis.backend’ is already set and cannot be changed until you reconnect”

Continue Reading by clicking title hyperlink ..

by Regina Obe at June 05, 2017 12:00 AM

May 25, 2017

BigSQL Holly

Connecting to Remote Spatial Data Sources with ogr_fdw

Shout out goes to Regina Obe and Paul Ramsey for their work and documentation on ogr_fdw. The ogr_fdw project can be found here: https://github.com/pramsey/pgsql-ogr-fdw.

ogr+fdw = ogr_fdw

In an earlier post, I showed you how to use the ogr2ogr (or as my geogeek friends like to call it “ogre to ogre”) command line tool to import features in different data formats to Postgres (e.g. shapefiles or geojson).

Foreign Data Wrappers (FDWs) allow you to connect to remote data sources from within Postgres. From there you can query them with SQL, join across disparate data sets, or join across different systems. There are FDW implementations to connect to MySQL, Oracle, SQLite, as well as flat files.

ogr_fdw allows you to connect your PostGIS database directly to an existing GIS file or database and read from it without importing the data.

It’s like getting a hug from an ogre!

Exercise: Intro to ogr_fdw

Exercise Prerequisites

ogr_fdw is packaged with BigSQL’s PostgreSQL installers. If you have not installed the BigSQL PostgreSQL distribution and setup your PostGIS database, start here.

The GDAL_DATA Directory Path

In order for ogr_fdw to properly work, you will need to set the GDAL_DATA environment variable to the location of the directory gdal (if you haven’t already).

Lucky for you, with the BigQL distribution this is as easy as running the following commands via a command line tool.

Linux / OSX:

 cd <directory of installation>/pg96
 source pg96.env

Windows:

 cd <directory of installation>\pg96
 pg96-env

Both of these commands set environment variables that will live during your current session. Note that if you close your terminal or command prompt, these variables are removed from memory.

It’s also possible to set GDAL_DATA as a persistant environment variable. But that is the beyond the scope of this tutorial.

Download the data

For this exercise, I downloaded the NYC Subway Lines shapefile available at the New York City Open Data site.

To download this data click here: nyc_subway_lines shapefile

ogr_fdw_info Commands

Begin by running the ogr_fdw_info command to show a list of supported formats:

ogr_fdw_info -f

Supported Formats:
  -> "PCIDSK" (read/write)
  -> "netCDF" (read/write)
  ...
  -> "HTTP" (readonly)

Navigate to the nyc_subway_lines directory you downloaded and run the following command to see the layers in the directory:

ogr_fdw_info -s <nyc_subway_lines directory>
    example OSX/Linux: ogr_fdw_info -s /Users/hollyorr/gis_data/nyc_subway_lines
    example Windows: ogr_fdw_info -s C:\gis_data\nyc_subway_lines 

Layers:
  nyc_subway

Now use the following command to read an OGR data source and output a server and table definition for this particular layer:

ogr_fdw_info -s <nyc_subway_lines directory> -l nyc_subway
    example OSX: ogr_fdw_info -s /Users/hollyorr/gis_data/nyc_subway_lines -l nyc_subway
    example Windows: ogr_fdw_info -s C:\gis_data\nyc_subway_lines -l nyc_subway

Output:

CREATE SERVER myserver
  FOREIGN DATA WRAPPER ogr_fdw
  OPTIONS (
    datasource 'nyc_subway_lines',
    format 'ESRI Shapefile' );

CREATE FOREIGN TABLE nyc_subway (
  fid bigint,
  geom Geometry(LineString,4326),
  objectid real,
  shape_len real,
  url varchar,
  name varchar,
  rt_symbol varchar,
  id real
) SERVER myserver
OPTIONS (layer 'nyc_subway');

You can run the commands generated as output in pgAdmin or command line. In this exercise we will run the sql statement in command line with psql.

Connect to psql:

Linux / OSX:

 cd <directory of installation>/pg96
 source pg96.env    

Windows:

 cd <directory of installation>\pg96\bin

Open psql and connect to postgis database:

$ psql
postgres=# \connect postgis_sample
    You are now connected to database "postgis_sample" as user "postgres".

Create the remote server and foreign table:

CREATE SERVER myserver FOREIGN DATA WRAPPER ogr_fdw OPTIONS (datasource '<location of nyc_subway_lines directory>', format 'ESRI Shapefile');

CREATE FOREIGN TABLE nyc_subway (fid bigint, geom Geometry(LineString,4326), objectid real, shape_len real, url varchar, name varchar, rt_symbol varchar, id real) SERVER myserver OPTIONS (layer 'nyc_subway');

Now open pgDevOps by BigSQL and switch to the pgAdmin4 Web view.

Check to see that your remote server was successfully created:

In the pgAdmin4 browser, navigate to:

Servers >> pg96 >> Databases >> postgis_sample >> Foreign Data Wrappers >> ogr_fdw >> Foreign Servers >> myserver

Check to see that your foreign table was successfully created:

In the pgAdmin4 browser, navigate to:

Servers >> pg96 >> Databases >> postgis_sample >> Schemas >> public >> Foreign Tables >> nyc_subway

Finally, query the nyc_subway table to see that your remote data using the ogr_fdw:

SELECT * FROM public.nyc_subway LIMIT 100;

by Holly Orr at May 25, 2017 07:39 PM

May 21, 2017

Boston GIS (Regina Obe, Leo Hsu)

PostGIS 2.4.0, Code Sprints and other extensions to try with PostgreSQL 10 beta1

So PostgreSQL 10beta1 came out recently as Holly mentioned. When first mention of beta hits, things start getting serious for me. I have to make sure that PostGIS compiles against said distribution to make sure eager testers aren't held back.

As with other releases, PostGIS didn't compile against the new PostgreSQL version without some nurturing. We've still got one regress failure, but at least PostGIS 2.4 now compiles cleanly against PostgreSQL 10beta1. I'm hoping that we can release PostGIS 2.4.0 just in time for PostgreSQL 10 planned release in September so I don't have to backport PostgreSQL 10 patches I made to lower PostGIS versions.

For PostGIS 2.4 the main focus will be cleaning up the parallel work so that all aggregate functions can enjoy use of parallel optimization. This is even more important with PostgreSQL 10 now that more kinds of queries can benefit from parallelization work. I'm also hoping to focus more energy on the raster side of things.


Continue reading "PostGIS 2.4.0, Code Sprints and other extensions to try with PostgreSQL 10 beta1"

by Regina Obe (nospam@example.com) at May 21, 2017 07:37 AM

April 24, 2017

BigSQL Holly

ETL (Extract, Transform, Load) PostGIS Data with ogr2ogr

In an earlier post, I showed you how to use the shp2pgsql tool to import shapefiles into PostGIS. But what if you need to import other geodata formats (e.g. GeoJSON, MapInfo, KML, CSV, etc.)?

ogr2ogr – ETL for PostGIS

Shrek

Lucky for you, the BigSQL PostGIS distribution comes bundled with GDAL and the ogr2ogr (or as my geogeek friends like to call it “ogre to ogre”) command line tool.

From gdal.org/ogr2gr:

This program can be used to convert simple features data between file formats performing various operations during the process such as spatial or attribute selections, reducing the set of attributes, setting the output coordinate system or even reprojecting the features during translation.

In other words, ogr2ogr is your go to ETL tool for importing and exporting PostGIS data.

Exercise: Intro to ogr2ogr

In this exercise, we will import and export GeoJSON files to and from our postgis_sample database using ogr2ogr commands.

Exercise Prerequisites

The ogr2ogr command line tool is packaged with BigSQL’s PostgreSQL installers. If you have not installed the BigSQL PostgreSQL distribution and setup your PostGIS database, start here.

GeoJSON

GeoJSON is an extension of JSON open standard format and designed for representing simple geographical features, along with their non-spatial attributes.

Example:

    {
        "type": "Feature",
        "geometry": {
            "type": "Point",
            "coordinates": [125.6, 10.1]
        },
        "properties": {
            "name": "Dinagat Islands"
        }
    }

Getting to know ogr2ogr Command Line

Open your command line utility and enter the this help command to see ogr2ogr parameters and usage:

ogr2ogr --long-usage

You should see the following parameter and under it a list of all the data formats your installation of ogr2ogr supports:

-f format_name

Make sure that the these two formats are included in the list:

-f "GeoJSON"
-f "PostgreSQL"

Import GeoJSON to PostGIS

For this exercise, I downloaded the NYC Wi-Fi Hotspot Locations GeoJSON file available at the New York City Open Data site.

To download this data click here: nyc_hotspots.geojson

To import a GeoJSON file into your PostGIS database, use the following command:

ogr2ogr -f "PostgreSQL" PG:"dbname=my_database user=postgres" "source_data.json" -nln new_table_name

If you want to append the new data to already existing records, use the -append flag:

ogr2ogr -f "PostgreSQL" PG:"dbname=my_database user=postgres" "source_data.geojson" -nln destination_table -append

So, for our exercise run the following commands:

cd <location of nyc_hotspots.geojson>
ogr2ogr -f "PostgreSQL" PG:"dbname=postgis_sample user=postgres" "nyc_hotspots.geojson" -nln nyc_hotspots

Now, run the following sql query on your postgis_sample database to see the table’s attributes:

select * from public.nyc_hotspots;

Export from PostGIS to GeoJSON

To generate a geojson file from one of your PostGIS tables, run the following command.

ogr2ogr -f "GeoJson" out.geojson PG:"host=localhost dbname=postgis_sample user=postgres password=<your_password>" \  -sql "SELECT * from nyc_hotspots"

*Note: Although you can use ogr2ogr to import and export shapefile format to and from PostGIS, it is recommended you use the shp2pgsql tool I discussed in an earlier post.

by Holly Orr at April 24, 2017 12:37 PM

April 05, 2017

Boston GIS (Regina Obe, Leo Hsu)

Slides from PGConfUS 2017 PostGIS talk

At PGConfUS 2017 last week, we presented a talk: Top 10 Problems Solved by PostGIS. The slides for the talk in HTML format and PDF format. The pgRouting examples at the end of the talk seemed to be the most popular.

We'll be giving a pgRouting workshop at FOSS4G Boston 2017 Aug 14-19th where we'll go into much more depth about topics like drive time analysis and Vehicle Route schedule planning (VRP). We also hope to give a talk at FOSS4G 2017 on PostGIS spatial tricks.

by Regina Obe (nospam@example.com) at April 05, 2017 07:23 AM

Boston GIS (Regina Obe, Leo Hsu)

Google Cloud SQL for PostgreSQL supports PostGIS

Google recently put in beta Google Cloud SQL for PostgreSQL which targets PostgreSQL 9.6 and seems like they plan to stay in line with PostgreSQL offerings similar to the Amazon PostgreSQL RDS and Aurora offerings. I was curious to see what kind of extensions they support. In Google cloud blog article Cloud SQL for PostgreSQL for your mobile and Geospatial applications in Google Cloud PostGIS is explicitly mentioned as supported. Presumably because Descartes is one of their case studies which I can only guess probably uses PostGIS based on the quote from Tim Kelton, Descartes co-founder:

"Our individual teams, who are building micro services, can quickly provision a database on Cloud SQL. They don't need to bother compiling Geos, Proj4, GDAL, and Lib2xml to leverage PostGIS. And when PostGIS isn’t needed, our teams use PostgreSQL without extensions or MySQL, also supported by Cloud SQL."

Excerpted from their article:
Geospatial support: Easily enable the popular PostGIS extension for geospatial objects in Postgres.

It's unclear from the article what other extensions they support though. Maybe I'll have to try it.

by Regina Obe (nospam@example.com) at April 05, 2017 06:18 AM

March 26, 2017

Boston GIS (Regina Obe, Leo Hsu)

PGConfUS 2017 Getting Stuff done in PostGIS

A reminder, PGConfUS 2017 conference is just days away, and we'll be giving a training March 28th 2017, Jersey City, NJ at 1 PM. If you are coming, keep an eye on this page PGConf 2017 US Getting Stuff done with PostGIS materials.

If you haven't signed up already, there are still spots, make sure to buy your tickets at http://pgconf.us/conferences/2017#registration.


Continue reading "PGConfUS 2017 Getting Stuff done in PostGIS"

by Regina Obe (nospam@example.com) at March 26, 2017 04:42 AM

March 23, 2017

BigSQL Holly

Importing Shapefiles into PostGIS with shp2pgsql

Geojson may be catching up, but by and far the most popular data format for sharing geospatial data is (and has been since the early 1990’s) the shapefile. Although a shapefile is a simple way to share data that can be easily consumed by almost any GIS system, it comes with many limitations and is not suitable for your enterprise data needs.

My personal least favorite limitation is that shapefiles truncate column names greater than 10 characters – ouch!

Luckily, the PostGIS community has bundled with PostGIS distributions a great tool that simplifies the task of importing shapefiles into your PostgreSQL database: shp2pgsql.

Shapefile Definitions:

Vector Data: Points, lines, or polygons that are used to represent real-world features on a map. Used for data that has discrete boundaries like administrative boundaries (e.g. country borders), physical features (e.g. lakes or streams) or infrastructure (e.g. streets). Attributes in a table are associated with each vector feature. vector_small

Shapefile: The most popular geospatial vector data format. First developed by ESRI. Because of it’s open specification it is now predominantly used to share GIS data between open source and proprietary software. A shapefile actually consists of 3 mandatory files:

  • .shp — shape format; the feature geometry itself
  • .shx — shape index format; a positional index of the feature geometry to allow seeking forwards and backwards quickly
  • .dbf — attribute format; columnar attributes for each shape, in dBase IV format

Exercise: Shapefiles to PostGIS with shp2pgsql

Prerequisite: Before you begin, create a spatial database by following the instructions laid out in our previous post, PostGIS – How to create a Spatial Database with PGC Command Line.

Bundled with your BigSQL PostGIS distribution is a tool called shp2pgsql. In this exercise, we will download a shapefile, find the correct SRID code, and use the command line tool to upload it to our postgis_sample database.

Download the data

Over the past 10 years, most U.S. municipalities have enacted open data policies. This means you can download lots of geospatial data from government web sites. Most geospatial data is provided as a shapefile or geojson file.

For this exercise, I downloaded NYC borough boundaries shapefile available at the New York City Open Data site.

To download this data click here: nybb_17a

After downloading, unzip the files and put it on your drive somewhere you will remember.

Navigate to shp2pgsql

Before you begin, make sure shp2pgsql is installed in your bin directory:

cd <directory of installation>/pg96/bin 

shp2pgsql syntax

Syntax for running command line:

shp2pgsql -I -s <SRID> <PATH/TO/SHAPEFILE> <SCHEMA>.<DBTABLE> | psql -U postgres -d <DBNAME>

Parameters

< SRID > – The Spatial Reference System Identifier (SRID or ESPG) is a unique value that represents the data’s coordinate system.

Findng the SRID/ESPG code:

All geospatial data has a coordinate system it is projected in. The topic of projections is beyond the scope of this posting, but you can learn more here.

Navigate to the shapefile’s .prj file which contains information about the shapefile’s coordinate system.

  cd <location of unzipped nybb_17a directory> 

Option 1:
Upload the .prj file to prj2epsg.org

Option 2:
Open the .prj file in your favorite editor and put the values in the first parameter, PROJCS, into the following sql statement that you will run against the postgis_example database:

nybb.prj parameter:

  PROJCS["NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet"

SQL query:

    SELECT srid, srtext,proj4text FROM spatial_ref_sys 
    WHERE srtext ILIKE '%New_York_Long_Island%' 
    AND proj4text ILIKE '%nad83%' AND proj4text ILIKE '%ft%';  

srid

The SRID for the nybb data is 2263.

<PATH/TO/SHAPEFILE> – The full path to the shapefile:

<location of unzipped nybb_17a directory>  

< SCHEMA > — Target schema where the new table will be created:

This is optional. 

< DBTABLE > — New database table to be created:

nybb

< DATABASE > —Target database where the table will be created:

postgis_sample

For a complete listing of the shp2pgslq command line parameters go here.

Putting it all together

So, putting this all together for this exercise you would run the following command:

Note: before running the command make sure psql has been started.

cd <location of unzipped nybb_17a directory>
shp2pgsql -I -s 2263 nybb.shp nybb | psql -U postgres -d postgis_sample

Now, run the following sql query on your postgis_sample database:

select * from public.nybb;

And you will see the nybb table’s attributes:

nybb_table

by Holly Orr at March 23, 2017 01:51 PM

February 21, 2017

Stephen Mather

Taking Slices from LiDAR data: Part IX

Part 9 of N… , see e.g. my previous post on the topic.

We’ve been working to reduce the effect of overlapping samples on statistics we run on LiDAR data, and to do so, we’ve been using PDAL’s filters.sample approach. One catch: this handles the horizontal sampling problem well, but we might want to intentionally retain samples from high locations — after all, I want to see the trees for the forest and vice versa. So, it might behoove us to sample within each of our desired height classes to retain as much vertical information as possible.


by smathermather at February 21, 2017 03:26 AM

February 19, 2017

Stephen Mather

Taking Slices from LiDAR data: Part VIII

Part 8 of N… , see e.g. my previous post on the topic.

I didn’t think my explanation of sampling problems with LiDAR data in my previous post was adequate. Here are a couple more figures for clarification.

We can take this dataset over trees, water, fences, and buildings that is heavily sampled in some areas and sparsely sampled in others and use PDAL’s filters.sample (Poisson dart-throwing) to create an evenly sampled version of the dataset.

Figure showing overlap of LiDAR scanlines Figure showing data resampled for eveness
Figure showing overlap of LiDAR scanlines

Figure showing overlap of LiDAR scanlines

Figure showing data resampled for eveness

Figure showing data resampled for evenness

An extra special thanks to the PDAL team for not only building such cool software, but being so responsive to questions!


by smathermather at February 19, 2017 01:04 AM

February 16, 2017

Stephen Mather

Taking Slices from LiDAR data: Part VII

Part 7 of N… , see e.g. my previous post on the topic.

More work on taking LiDAR slices. This time, the blog post is all about data preparation. LiDAR data, in its raw form, often has scan line effects when we look at density of points.

lidar_flightlines

This can affect statistics we are running, as our sampling effort is not even. To ameliorate this affect a bit, we can decimate our point cloud before doing further work with it. In PDAL, we have three choices for decimation: filters.decimation, which samples every Nth point from the point cloud; filters.voxelgrid, which does volumetric pixel based resampling; and filters.sample or “Poisson sampling via ‘Dart Throwing'”.

filters.decimation won’t help us with the above problem. Voxelgrid sampling could help, but it’s very regular, so I reject this on beauty grounds alone. This leaves filters.sample.

The nice thing about both the voxelgrid and the poisson sampling is that they retain much of the shape of the point cloud while down sampling the data:

subsample-ex1

subsample-ex2

We will execute the poisson sampling in PDAL. As many things in PDAL are best done with a (json) pipeline file, we construct a pipeline file describing the filtering we want to do, and then call that from the command line:

We can slice our data up similar to previous posts, and then look at the point density per slice. R-code for doing this forthcoming (thanks to Chris Tracey at Western Pennsylvania Conservancy and the LidR project), but below is a graphic as a teaser. For the record, we will probably pursue a fully PDAL solution in the end, but really interesting results in the interim:

image001

More to come. Stay tuned.


by smathermather at February 16, 2017 04:46 AM

February 10, 2017

BigSQL Holly

PostGIS – How to create a Spatial Database with PGC Command Line

postgis_plus_bigsqlWe are excited to announce that BigSQL distributions of PostGIS now come bundled with several popular community spatial extensions:

This is the only community distribution to offer PostGIS with these extensions across ALL platforms (Mac, Linux, and Windows)! So, for all you GeoGeeks using Linux / Mac – spatially enabling your PostgreSQL database with PostGIS functionality just got a lot easier.

In addition, we’ve included a sample script to help guide you through creating your first PostGIS database.

In the upcoming weeks, we will offer guidance for both GUI and command line work flows (see Are you a Yoda or a Diego?) and blogging about what you can do with spatially enabled data.

So, let’s get started…

What is PostGIS?

PostGIS is powerful spatial database extender for PostgreSQL. It adds support for geographic objects allowing location queries to be run in SQL. For a full list of supported features look here: http://postgis.net/features

How to install the BigSQL distribution

If you have installed the BigSQL distribution, skip to the next section, “Getting started with BigSQL PGCLI.”

Getting started with BigSQL PGCLI

PGCLI (Pretty Good Command Line Interface) is a package and update manager, similar to yum or apt-get, specifically for PostgreSQL and related community projects. PGC enables you to easily find and install packages, configure multiple versions of PostgreSQL, update existing applications, and control services by starting and stopping from the command line.

You can run pgc commands with the following call:

 cd <directory of installation>
 ./pgc (Linux and Mac)
 pgc (Windows)

For the purpose of this tutorial, we will be using the Linux/Mac syntax: ./pgc

To learn more about PGC, go to the BigSQL PGCLI tutorial.

Update components to latest versions and install PostGIS

  1. Open a terminal and navigate to the installation directory.

  2. On windows it could be C:\Postgres, on Linux and Mac it could be any directory based on the installation method. You will be able to locate “pgc” bash script OR “pgc.bat” file in this directory.

     cd <directory of installation>
    
  3. Run the pgc status command to show your current Postgres server status.

    In this example Postgres 9.6 is running on the default port:

     ./pgc status
        pg96 running on port 5432
    

    If your Postgres instance is not running, start it with the following pgc command:

     ./pgc start
    
  4. Run the pgc update command to get the latest versions:

     ./pgc update
    
  5. Install PostGIS:

     ./pgc install postgis23-pg96
    

Create a PostGIS sample database

We have provided you with a sql script that will create a Postgis enabled database and includes the following extensions:

Before you begin, navigate to directory that contains the sql script you will run to create a PostGIS database and check to make sure the file create_postgis_sample_db.sql downloaded with your install. You can also take a look at what is in the script using your favorite editor tool:

 cd <directory of installation>/pg96/share/doc/postgresql/extension 
 ls

 Optional:
 vi create_postgis_sample_db.sql

The GDAL_DATA Directory Path

In order for all your PostGIS extensions to properly work, you will need to set the GDAL_DATA environment variable to the location of the directory gdal.

Lucky for you, with the BigQL distribution this is as easy as running the following commands via a command line tool.

Linux / OSX:

 cd <directory of installation>/pg96
 source pg96.env

Windows:

 cd <directory of installation>\pg96
 pg96-env

Both of these commands set environment variables that will live during your current session. Note that if you close your terminal or command prompt, these variables are removed from memory.

It’s also possible to set GDAL_DATA as a persistent environment variable. But that is the beyond the scope of this tutorial.

Run the script using the psql terminal.

Linux / OSX:

 psql -f share/doc/postgresql/extension/create_postgis_sample_db.sql

Windows:

 psql -f ..\share\doc\postgresql\extension\create_postgis_sample_db.sql

Check to see that the database and extensions were successfully created:

 psql (if using Windows, you may need to include -U postgres with the psql command)
 postgres=# \connect postgis_sample;
 You are now connected to database "postgis_sample" as user "postgres".

 \dx
                 List of installed extensions
                 Name             | Version |   Schema               
    ------------------------------+---------+------------
     address_standardizer_data_us | 2.3.2   | public     
     fuzzystrmatch                | 1.1     | public     
     ogr_fdw                      | 1.0     | public     
     plpgsql                      | 1.0     | pg_catalog 
     postgis                      | 2.3.2   | public     
     postgis_tiger_geocoder       | 2.3.2   | tiger      
     postgis_topology             | 2.3.2   | topology   
    (8 rows)

Exit psql:

 \q

Install pgadmin3 if it is not already installed:

 cd <directory of installation>  
 ./pgc install pgadmin3

Launch pgadmin3 (the GUI should popup when you run this command):

 ./pgc start pgadmin3    

Output should look something like this:

pgadmin_postgis_sample

Get me outta here! (CLI edition):

You can always uninstall your BigSQL Postgres distribution at any time with the following commands:

 cd <directory of installation>
 ./pgc stop
 cd uninstall
 open uninstall.app

Finally, cd to location of your PostgreSQL directory and delete the PostgreSQL directory and it’s contents:

 rm -r <directory of installation> (Linux or Mac)
 rmdir <directory of installation> /s (Windows)

by Holly Orr at February 10, 2017 02:18 PM

February 08, 2017

BigSQL Holly

Are you a Yoda or a Diego?

In the geospatial world there are usually 2 types of people supporting spatially enabled databases:

diego 1. GIS analysts  (scientist, statisticians, analysts, etc) who find themselves as the de facto DBA (among other things). “I don’t have time for this! I’m saving this sea turtle. Just give me a GUI!”

yoda 2. DBAs who find themselves supporting these strange spatially enabled databases. “Real men only use command line! And what the hell is GIS?”

arcinfo Of course, some of you old school GISers may really love to geek out on the CLI (ArcI/Info anyone?).

But GIS has come a long way since 1982. For one thing, the learning curve for running GIS analysis is becoming less intimidating due to new SaaS geospatial companies like our friends at Carto. For another, open source (FREE!) server and desktop software has caught up with (and in some cases surpassed) its proprietary counterparts (see out post “Why are you still paying for your GIS“).

At BigSQL, we want you to know we love you for who you are. And in that spirit, just in time for the release of PostgreSQL 9.6.2 and PostGIS 2.3.2, we are introducing tools to help you build and monitor your spatially enabled databases that will satisfy whichever tribe you belong to: the GUIs or the CLIs.

In our next post we will help Tribe Yoda (aka CLIs) leverage the PGCLI: Pretty Good Command Line Interface to make installing, updating, configuring, and monitoring PostgreSQL/PostGIS simple and FREE!

by Holly Orr at February 08, 2017 08:25 PM

January 31, 2017

PostGIS Development

PostGIS 2.3.2 Released

The PostGIS development team is pleased to announce the release of PostGIS 2.3.2 As befits a patch release, the focus is on bugs and breakages. Best served with PostgreSQL 9.6.1+ and pgRouting 2.3.2.

Continue Reading by clicking title hyperlink ..

by Regina Obe at January 31, 2017 12:00 AM

January 30, 2017

PostGIS Development

PostGIS 2.2.5 Released

The PostGIS development team is pleased to announce the release of PostGIS 2.2.5 As befits a patch release, the focus is on bugs and breakages.

Continue Reading by clicking title hyperlink ..

by Regina Obe at January 30, 2017 12:00 AM

January 24, 2017

Oslandia

QGIS versioning plugin

We developped a tool to manage data history, branches, and to work offline with your PostGIS-stored data and QGIS. Read more to get the insight of QGIS Versioning plugin.

The QGIS plugin is available in QGIS plugin repository, and you can `fork it on GitHub too https://github.com/Oslandia/qgis-versioning !

Introduction

Even if the necessity of data versioning often arises, no standard solution exist for databases.

The GeoGit project proposes a solution to store versioned geospatial data. There is also an existing plugin for QGIS, pgversion, which uses views and triggers to version a PostGIS database. Unfortunately those solutions were not adapted to the specific constrains of this project, namely: using a PostGIS database as the main repository (excludes GeoGit) and the ability to working off-line (excludes pgversion).The project we developed QGIS/PostGIS versioning looks like the following.

 

Design

The database is stored in a PostGIS schema, the complete schema is versioned (i.e. not individual tables). Revisions are identified by a revision number. A revision table in the versioned schema, called ‘revisions’, keeps track of the date, author, commit message and branch of all revisions.

Once a table structure is defined, three operations can be performed on rows: INSERT, DELETE and UPDATE. To be able to track history, every row is kept in the tables. Deleted rows are marked as such and updated rows are a combined insertion-deletion where the deleted and added rows are linked to one another as parent and child.|

A total of five columns are needed for versioning the first branch:

PRIMARY KEY
a unique identifier across the table

branch_rev_begin
revision when this record was inserted

branch_rev_end
last revision for which this record exist (i.e. revision when it was deleted minus one)

branch_parent
in case the row has been inserted as the result of an update, this fields stores the hid of the row that has been updated

branch_child
in case the row has been marked as deleted as the result of an update, this field stores the hid of the row that has been inserted in its place.

For each additional branch, four additional columns are needed (the ones with the prefix branch_).

Note:
If the branch_rev_begin is null, it means that a row belongs to another branch.

SQL views are used to see the database for a given revision number. If we note ‘rev’ the revision we want to see. For each table, the condition for a row to be present is the view is::

(branch_rev_end IS NULL OR branch_rev_end >= rev) AND branch_rev_begin <= rev

In the special case of the current revision, or head revision, the condition reads::

branch_rev_end IS NULL AND branch_rev_begin IS NOT NULL

Note:
Since elements are not deleted (but merely marked as such) from an historized table, care must be taken with the definition of constrains, in particular the conceptual unicity of a field values.

Withing the PostGIS database, the views on revisions must be read-only  and historized tables should not be edited directly. This is a basic principle for version control: editions must be made to working copies an then committed to the database. Please note that by default PostGIS 9.3 creates updatable views.

Workflow schema

This setup allows for multiple users to use and edit data offline from a central repository, and commit their modifications concurrently.

Working copies

Two kinds of working copies are available:

SpatiaLite working copies
They are meant to be used off-line. They consist of the versioned tables of a given versioned database (i.e. PostGIS schema) or any subset. For each table, only the elements that have not been marked as deleted in the head revision need to be present. Furthermore only a subset of the elements the user needs to edit can be selected (e.g. a spatial extend).  To create a working copy (i.e. to checkout), tables from the versioned schema (or the aforementioned subsets) are converted to a SpatiaLite database using ogr2ogr.

PostGIS working copies
They are meant to be used when the connection to the original database will remain available. They are quite similar to pgversion working copies since they only store differences from a given revision (the one checked out).

The following description is aimed at understanding the inner workings of the qgis versioning plugin. The user does not need to perform the described operations manually.

For each versioned table in the working copy, a view is created with the suffix _view (e.g. mytable_view). Those views typically filters out the historization columns and shows the head revision. A set of triggers is defined to allow updating on those views (DELETE, UPDATE and INSERT).

The DELETE trigger simply marks the end revision of a given record.

The INSERT trigger create a new record and fills the branch_rev_begin field.

The UPDATE trigger create a new record and fills the branch_rev_begin and branch_parent fields. It then marks the parent record as deleted, and fills the branch_rev_end and branch_child fields.

Updating the working copy

Changes can be made to the database while editing the working copy. In order to reconcile those edition, the user needs to update the working copy.

When updating, a set of records can be in conflicts: the records for which the end revision has been set since the initial checkout or last update if any.

Multiple editions can be made to the same record. Therefore the child relation must be followed to the last child in order to present tu user with the latest state of a given conflicting feature.

Conflicts are stored in a table and identified with a conflict id and the tag ‘theirs’ or ‘mine’. A DELETE trigger on this table is used for conflict resolution. On deletion of ‘mine’, the working copy edition is discarded, on deletion of ‘theirs’ the working copy edition is appended to the feature history (i.e. the working copy feature becomes a child of the last state of the feature in the historized database).

Committing the editions to the versionned database

If a working copy is up to date, the editions can be integrated in the versioned database. This operation consists simply in the insertion of a record in the revisions table, and, for each versioned table, the update of rows that are different and inserting rows that are not present.

Branching

A branch can be created from any revision by adding the four history columns and setting the branch_rev_begin field of features that are present in their revision.

Plugin interface tutorial

Groups are used for all versioning operations in QGIS since the versions are for a complete PostGIS schema or SpatiaLite database.

The versioning toolbar will change depending on the group selected in the QGIS legend.

Note:
The group elements must share the same connection information (i.e. share the same database and schema for PostGIS working copies and revision views or share same SpatiaLite database for SpatiaLite working copies).

Versioning a PostGIS schema

Starting with an unversioned database, import a number of layers from a schema that needs to be versioned into a QGIS project.

Once the layers are imported, they must be grouped together.

Selecting the newly created group will cause the versioning toolbar to display the historize button (green V). On click a confirmation is requested to version the database schema.

The versioned layers are imported in a new group and the original layers are removed from the project.

Note:
The symobology is not kept in the process.

Working with a versioned PostGIS schema

Versioned layers can be imported in QGIS. The layers must be from a head revision or a view on any revision.

 

Once the layers are in QGIS, they must be grouped.

 

For PostGIS groups at head revision, the versioning plugin allows the user to create a SpatiaLite or a PostGIS working copy, create a view on a given revision or create a branch. A corresponding group will be imported in QGIS.

If the user chooses to create a SpatiaLite working copy, he will be asked to select a file to store the working copy.

 

Commiting changes

Once the working copy is imported in QGIS, the user can start edition of its layers. With SpatiaLite working copies, this edition can be done off-line.

When the user is done with edition, he can commit the changes to the database and if commit is feasible (i.e. the working copy is up to date with the versioned database), he will be prompted for a commit message and subsequently be informed of the revision number he committed.

 

If the commit is not feasible, the user will be informed that he must update his working copy prior to commit.

Resolving conflicts

Conflicts are detected during update, the user is informed, and conflicts layers are imported into QGIS.

To resolve conflicts, the user can open the conflict layer’s attribute table. Selected entries are also selected on the map canvas and the user can decide which version, either his or the database’s, he wants to keep. User version is tagged with ‘mine’ and database version with ‘theirs’. The conflict is resolved by deleting the unwanted entry in the conflict layer.

Note:
On deletion of one conflict entry, both entries are removed (by a trigger) but the attribute table (and canvas) are not refreshed. As a workaround, the user can close and re-open the attribute table to see the actual state of the conflict table.

Once the conflict table is empty, the commit can be done.

Restrictions

Due to design choices and tools used for conversion to SpatiaLite, a number of restrictions apply to the versioned database:

  • |schemas, tables and branch names should not have space, caps or quotes
  • tables must have primary keys
  • columns are lowercase (because of conversion to SpatiaLite) but can have spaces (not that it’s recommended
  • geometry column is geom in PostGIS, GEOMETRY in SpatiaLite

Note
Do not edit OGC_FID or ROWID

Note
The constrains on the tables are be lost in the PostGIS to SpatiaLite conversion.

Known bug

The conflict layer won’t be loaded automatically is it has no geometry. The user will have to load it manually.

by oslandia at January 24, 2017 02:40 PM

Oslandia

QGIS plugin for water management

Oslandia releases today a new plugin for the QGIS processing framework, allowing for water distribution network simulation. It integrates the opensource EPANET simulation software. EPANET models water distribution networks. It’s a widely used public-domain simulation software developed by the US Environmental Protection Agency.

Hydraulic simulation is used to understand water distribution in distribution network, to forecast the impact of network alterations, to dimension network elements or study extreme case scenarios (e.g. important demand for firefighting, pipes breakages, interruption in supply).

QGIS provides a graphical user interface that can be used to import/edit/export hydraulic model elements and simulation parameters from various sources, launch simulation and visualize results directly inside QGIS.

Hydraulic model

A hydraulic model consists of junctions (POINT) and pipes (LINESTRING) along with various other elements like tanks, pumps and valves. Those elements can be stored as features in a spatially enabled database. Features attributes can be simple (e.g. pipe diameter) or complex (e.g. pumps characteristic curves or water consumption). Complex attributes are stored via a foreign key in other alphanumeric tables.

This is the kind of data QGIS is designed to handle. It can import/export them from/to a variety of sources and also display and edit them.

Simulation parameters

Simulation parameters and options (e.g. simulation time step or accuracy) are key-value pairs. The values can be stored in a table which columns are keys. Each set of simulation parameters is then a record in this table. This kind of table can be loaded in QGIS as a vector layer without geometry.

Integration in the processing framework

Once the hydraulic model and simulation parameters are loaded in QGIS, the simulation can be launched through the Processing toolbox. The plugin uses the standalone command line interface of EPANET (CLI) which path needs to be specified in processing Options and configuration.

The plugin assembles an EPANET input file, runs EPANET and parses its output to generate result layers.

One interesting aspect with processing modules is that they can be used for chained processing: the user can use other modules to do additional transformations of simulation results, as feeding them into another simulation model.

Result visualization

Simulation results are water pressure and velocity at all points in the network along with state of network elements (e.g. volume in tanks, power of pumps) for all simulation time steps . This represent a huge amount of data that are usually displayed either as time-plots or as map-plots of time aggregated data (e.g. max and min during simulation).

Results of particular interest are:

  • time-plots of:
    • volume in reservoirs
    • flow at pumps
    • pressure in pipes and at junctions
  • map-plots of:
    • low speed (stagnation)
    • high and low pressure (risk of breakage, unhappy consumer)
    • lack of level variation in reservoirs (stagnation)
    • empty reservoir
    • reservoir overflow
    • abnormal pressure (typical of error in the altitude of a node in the model)
    • flow direction

QGIS is naturally suited for map-plots. Time-aggregated simulation results are automatically joined to map layers when the result table is added to the map. Rule-based symbology is used to highlight zones of concern (e.g. low water velocity or empty reservoirs).

The matplotlib library provides 2D plotting facilities in python and QGIS provides an extensive set of selection tools (on the map or in tables). The plugin’s button plots the appropriate value depending on the selected feature type (e.g. water level for tanks, pressure for junctions).

Screencast

For a full demo of this plugin, see the following video :

 

Where and who

The plugin is available on GitHub and should be available soon on QGIS plugin repository : https://github.com/Oslandia/qgis-epanet

This work has been funded by European Funds. Many thanks to the GIS Office of Apavil, Valcea County (Romania). Oslandia has developped this plugin, and provides support and development around QGIS, PostGIS and this plugin. Get in touch if you need more : infos@oslandia.com

We are looking for a free dataset with full informations (pumps, tanks, valves, pipes and their characteristics…) to distribute with this plugin as a test case and demonstration. If you can provide this, mail us !

We also are implementing a Processing plugin for SWMM, the public domain Waste-water simulation tool. If you are interested to participate to the development, please contact us.

by oslandia at January 24, 2017 01:53 PM

Oslandia

PostGIS 3D – Foss4g video and workshop

The latest PostGIS and QGIS 3D enhancements presented at FOSS4G by Oslandia are available online.We suggest you to have a look on our PostGIS 3D / QGIS 3D video demonstration using SFCGAL library and the QGIS Horao plugin.

A step by step workshop, (really close to the video workflow) is also available online  https://github.com/Oslandia/Workshops/tree/master/FOSS4G_2013_PostGIS_3D

We can provide you the full virtual machine on demand, with proper software environment (6GB Virtual Box Image).

We would be really interested in having your advice on these new 3D features, and the use cases you could be interested in. Do not hesitate to get in touch.

Contact us at infos+foss4g@oslandia.com for any information.

by oslandia at January 24, 2017 01:12 PM

Oslandia

QGIS Community meeting in Brighton

Developers and contributors from the QGIS project are used to gather physically twice a year across different countries. Such an event allows people to synchronize their effort, and discuss new possible developments.cThe latest QGIS community meeting took place in Brighton from the 12th to the 16th of September, just before the FOSS4G event. It was the biggest community meeting organized so far, with close to 50 people attending ! Everything went smooth thanks to the perfect organization by Lutra Consulting.

This session was of particular interest in the project’s history, since it was dedicated to the release of the eagerly-awaited new 2.0 version of QGIS.

Oslandia is used to take part in the event and even organized the march 2012 session in Lyon.


Presentations

Despite being originally oriented toward code and translations, some presentations took place during the event. Some of them have been video recorded, some did not. Hereafter is a subset of them.

A new website

In parallel to the release of the 2.0 version, the QGIS website has been updated. Its look and feel, but also the way it is now build. Richard Duivenvoorde presented the efforts that have been put on the support of multiple languages, adaptation to mobile devices, and the reuse of tools used for building the documentation of the project. The new website is now online.

Richard presenting the new website

 

Presentation of the new website : http://www.ustream.tv/recorded/38687971

Constraints on attributes

Some more developer-oriented presentations and discussions also took place. Matthias Kuhn and Nathan Woodrow presented an idea about extending the way attributes are handled by QGIS. In particular, the concept of constrained attributes emerged. The idea is to be able to express, manipulate and edit contrains on attributes (possible range of values for instance) as it is found in databases. This could then be used to constrain user editing of layers, presenting to the user an appropriate widget (combo box for an enumeration for instance), especially for layers that do not have native support for these constraints.

QGIS for Android tablets

RealworldSystems presented their work on what they called the “QGIS Mobility framework”, based on previous works by Marco Bernasocchi on QGIS for Android. It is dedicated to the design of custom QGIS applications for deployment on Android tablets (for on-the-field editing campains for instance). It looks promising and has already been used in a real-world application for gaz pipeline inspection. The framework can be found on github.

QGIS webserver

Andreas Neumann presented evolutions of QGIS webserver and webclient. More can be found in the corresponding video.

Andreas presenting the work on QGIS webserver and webclient

Video 1 http://www.ustream.tv/recorded/38741015

Evolution of the Globe plugin

Matthias Kuhn presented evolutions he made to the Globe plugin that allows to display a 3D earth with different kinds of data on it. Lots of osgearth features are now integrated into the Globe plugin (in particular the support for 2D vector layers).

Matthias presenting its work on the Globe plugin

Video 2 http://www.ustream.tv/recorded/38737991

Visualisation of 3D data

Oslandia presented also its ongoing work on the integration of Postgis 3D. After a thourought evaluation of osgearth, which is the base of the Globe plugin, we decided to develop our own 3D visualisation stack directly on top of OpenSceneGraph.

A QGIS plugin has also been developed in order to be able to view QGIS layers in 3D.

With this new 3D visualisation stack we are able to display and manipulate data of a whole city between 20 and 60 frames per second on a laptop (here the demo has been designed on data from the city of Lyon) , when we were hardly able to display a small city quarter with Globe.

Oslandia presenting its work on its 3D visualisation stack

Video 3 http://www.ustream.tv/recorded/38738897

Slides https://github.com/Oslandia/presentations/tree/master/qgis_hf_2013

QGIS 2.0

All the work done during this community meeting allowed to polish the 2.0 version of QGIS which has been publicly announced during the FOSS4G in Nottingham by Tim Sutton.
Waiting now for the 2.1 release 🙂

by oslandia at January 24, 2017 11:48 AM

Oslandia

Oslandia and Mappy towards OpenSource

For more than two years, Oslandia has been working with Mappy for its transition towards OpenSource. The first step is represented by the database migration project of the cartographic backoffice, from Oracle to PostGIS. This has just been finalized and is currently running in production.

Mappy offers routing and cartography services. Supporting several billions of requests per month, the LBS platform (Location Based Services) is at the heart of Mappy’s activity and serves as a base for new products such as Web To Store. Oslandia works together with Mappy to migrate this cartographic platform from a solution developped in-house, to a new solution based entirely on free and opensource softwares, in order to prepare forthcoming technical and operational challenges.

The migration

Data preparation and cartographic map creation backoffices have been migrated successfully.
This software stack was initially based on Oracle Spatial, SQL Server, and other non-standard internal tools to create the map. It only runs opensource components by now. Oracle and SQL Server have completely been replaced by PostgreSQL and PostGIS, which is the key component for spatial data storage and preprocessing. Mapnik, Python, Tornado, Varnish, MemCached, Debian are other Opensource components used in this architecture.

The migration to OpenSource softwares allowed Mappy to rationalize and optimize the architecture of their “map” component :

  • Reduced data processing time
  • API and data format Standardization
  • Strong reduction of the technical debt
  • Reduced amount of lines of code
  • Cost reduction for the platform, and cost-effective scaling
  • Technical skills improvements for the team and stronger motivation
  • Full technical mastership of their tools

Oslandia helped Mappy to integrate the Opensource culture and methods into their teams, and provided them with the technical skills required to implement this high-load architecture. Oslandia’s competences in geographical information systems, and its high expertise on PostGIS let the project be achieved successfully.

Mappy benefited from the latest OpenSource technologies, at the state of the art, and was also able to contribute back to some free software, like Mapnik.

Migration steps

The migration has been achieved with several steps, so as to keep the project under control, validating the migrated components through early tests.

The initial platform was the following one :

The target platform, with all components migrated to OpenSource, is now the following one :

Thanks to the availability of OpenSource components, a prototype has been developped within two weeks at the beginning of the project, so as to validate architectural decisions. Then the real migration work has begun.

The choice concerning OpenSource components has been fast, as already said. The target architecture could therefore be setup quickly, and the development and migration processes have been setup once this first step validated.

This migration gave the opportunity for Mappy to contribute to Mapnik. A 25% performance gain was achieved when reading data from PostGIS. Mappy also opensourced Pycnik, a Python tool to generate styles for Mapnik.

PostGIS

PostGIS, the database supporting the whole infrastructure, allowed to reach very high performances and a rich functional level. The database – 75GB – benefits from all latest enhancements of PostgreSQL and PostGIS, such as streaming replication, PostGIS new geographical functions, recursive CTEs, JSON native support and much more.

Audrey Malherbe ( @AudreyMalherbe ), project manager at Mappy, stresses that choosing PostgreSQL and PostGIS was obvious:

“We wanted to dive into OpenSource, leaning on reknown and performant technologies like PostGIS. It was important for us to contribute to OpenSource, and Oslandia’s expertise and its implication in the community has allowed us to rush into this adventure with full confidence.”

Some facts

Most geeks like to see this kind of numbers :

  • for 85% of all Mappy traffic
  • 14 servers (4 PostGIS, 8 Tornik, 2 Varnish)
  • 240 req/s peak without cache
  • 2500 req/s peak with cache
  • 94% Cache Hit Ratio
  • 2M objects in cache warming
  • 75 GB for PostGIS database
  • 300 GB for the relief database

Next steps

More technical informations on this migration are available in the `Audrey’s presentation at FOSS4G 2013.

Mappy intends to go on with this OpenSource orientation and extend this migration method to the other services of the cartographic platform.

 

About Mappy

Routing and cartographic services specialist, Mappy is reknown as the leader for local search through maps, on internet, tablets, mobiles and GPS.

Mappy offers three kinds of search to its users : map search, which allows to visualize a neighborhood, to experience city immersion thanks to 360° views in 320 french cities, but also to open the door of several thousands of shops ; the routing service available for cars, public transportation, bicycle and pedestrian mode ; and product search, allowing to locate a specific product in a given geographical zone, to know its price and availability.

As a major actor of urban transportation, Mappy offers to advertisers a geolocated solution on the whole country, facilitating web-to-store applications and trafic generation to their retail centers.
Mappy counts as of now more than 10 millions users on Internet, tablets and mobile (Mappy and MappyGPS Free).

Mappy is a subsidiary of the Solocal Group.

by oslandia at January 24, 2017 10:51 AM

January 23, 2017

Oslandia

Full Spatial database power in 2 lines

This post explains how to setup a powerful spatial data store with a wide range of features on Ubuntu 14.04 (Trusty) in 2 command lines. And how it works.

TL;DR

Run this on Ubuntu 14.04 Trusty Tahr, and enjoy :

sudo apt install docker.io
sudo docker.io run --rm -P --name pggis_test oslandia/pggis /sbin/my_init

Database credentials : pggis/pggis/pggis for database/user/password

(DO NOT USE THIS IMAGE FOR PRODUCTION).

More info

You want to store and manipulate spatial data. Lots of spatial data. Various spatial data. This Docker image lets you install the latest components based on the most powerful opensource database (PostgreSQL, in case you wonder) and benefit from a wide range of features.
Docker is a Linux container based technology. See containers like something between the good old chroot and a Virtual Machine. Fast, efficient and painless. See Docker as a mix between a meta-package manager, git and a whale. Docker is available natively on Ubuntu 14.04 (and named docker.io), and can also run on other Linux flavors, or in a minimalist Virtual Machine with Boot2docker for other environments (Windows, MacOSX).

This image focus on installing and running the following components :

  • PostgreSQL 9.3
  • PostGIS 2.1.3 with SFCGAL support
  • PgRouting
  • PostgreSQL PointCloud
  • PDAL

PostgreSQL is the advanced database providing a lot of features for data store and management. Version 9.3 is the latest stable version (9.4 is just around the corner), providing support for streaming replication, JSON, CTE, window queries and much, much more.

PostGIS is a PostgreSQL extension, enabling spatial support. It features new data types, such as 2D (and 2.5D) vector data, be it projected or in lat/lon. It also offers raster data support, to store gridded image data, and topological vector data storage.

SFCGAL support for PostGIS adds 3D features to the database. You can store 3D meshes, TIN, and operate on those objects.

PgRouting is a PostgreSQL/PostGIS extension allowing you to perform routing in the database on topological spatial data. It can be used to dynamically compute shortest paths, driving distances and more.

PostgreSQL PointCloud is a PostgreSQL/PostGIS extension which lets you deal with huge amounts of points, e.g. LIDAR data. PDAL is a library and set of tools allowing you to load and extract data from and to PointCloud (among other things).

All of this makes the best platform for spatial data crushing. Loads of spatial data.

Using it

Below is a longer full explanation to install this image and use it.

Install Docker

You first need to install docker tools on your Ubuntu system. You can install Docker on a lot of recent Linux flavors or in a Linux Virtual Machine ( with Boot2docker for example).

sudo apt install docker.io

Get the image and run the container

The following commands use docker to download the image `from the docker registry <https://index.docker.io/u/oslandia/pggis/>`_ (~1GB) and then run it in the foreground. If you omit the **pull** command and try to run it directly, Docker will look for the image on the Docker registry if not already present locally.

sudo docker.io pull oslandia/pggis
sudo docker.io run --rm -P --name pggis_test oslandia/pggis /sbin/my_init

Your container is initiated from the oslandia/pggis image. It is named pggis_test, runs in the foreground (default), will redirect all exposed ports to a host port ( -P ), and will delete all created filesystem and container image when the container exits ( -rm ). It will run the /sbin/my_init startup script of baseimage to launch all necessary processes inside the container.

Now PostgreSQL runs in your container with all extensions activated, and a new database created just for you.

Connect to the database

Assuming you have the postgresql-client installed on your host, you can use the host-mapped port to connect to the database. You need to use **docker.io ps** to find out what local host port the container is mapped to first:

$ sudo docker.io ps
CONTAINER ID        IMAGE                   COMMAND                CREATED             STATUS              PORTS                     NAMES
75fec271dc5e        oslandia/pggis:latest   /sbin/my_init       51 seconds ago      Up 50 seconds       0.0.0.0:49154->5432/tcp   pggis_test

You can see that te container’s port 5432 has been mapped to the host’s port 49154 in this case (yours may differ). We can now connect to the database server through this host port. A pggis user has been created (with pggis as password) and a corresponding database with extensions activated.

$ psql -h localhost -p 49154 -d pggis -U pggis --password

Enjoy GIS features

You are now ready to use the database. Inside the psql console, you can check that everything is in order with queries like below.

pggis=# SELECT postgis_version();
postgis_version
---------------------------------------
2.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
(1 row)

pggis=# SELECT st_astext(st_makepoint(random() * 100, random() * 100)) from generate_series(1, 10);
st_astext
-------------------------------------------
POINT(90.1295741088688 97.7623191196471)
POINT(79.2798819020391 0.146342813968658)
POINT(56.9643571972847 50.8249804843217)
POINT(14.1903728246689 26.436754828319)
POINT(46.5733857825398 39.7641783114523)
POINT(89.1805129591376 36.6665146779269)
POINT(17.3397121019661 20.6000158563256)
POINT(11.5902675781399 93.6640297528356)
POINT(96.0820932406932 53.891480108723)
POINT(88.9889035373926 15.435611223802)
(10 rows)

Describing all features of installed components would require a lot more posts, coming later. If you are interested in learning more, Oslandia can provide training, assistance and support.

Warning

Do not use this in production !

The PostgreSQL configuration in this image opens the database connection without any IP restriction for the pggis super-user. The default pggis password is very weak too. This should NOT BE USED IN PRODUCTION, and you should adapt the configuration to your specific setup and security level. See below to rebuild the image with different configuration. In a production environment, you would also want to run the container in the background ( -d option) and keep the container filesystem and images after running ( no –rm option ).

How it works

This image is a Docker image, which uses Linux Containers (LXC). The base image is `Phusion baseimage <http://phusion.github.io/baseimage-docker>`_ , which it itself a Ubuntu 14.04 (Trusty). It is is built using a Dockerfile (see the source on Github). This Dockerfile is used by Docker to build the pggis image, by processing all installation, download, compilation and setup of the various components. The result is a Docker image, which is then published on docker.io.

You can either directly download and use the image from docker.io as explained above, or rebuild it from the Dockerfile in the git repository like below.

sudo apt-get install docker.io

Then clone the GitHub repository to get the Dockerfile :

git clone https://github.com/vpicavet/docker-pggis.git

cd docker-pggis

Then you can build the image, using the Dockerfile in the current repository.

sudo docker.io build -t oslandia/pggis .

Note that building the image requires quite a lot of download from internet, as well as enough RAM and CPU. Compilations are achieved along the way, with make -j3, thus enabling parallel compilation. You should adapt this value according to the number of CPUs you have.

Once you have built the image, you can run a container based on it just like above.

sudo docker.io run --rm -P --name pggis_test oslandia/pggis /sbin/my_init

If you want to change how the container works, or the default setup, you can edit the Dockerfile and rebuild the image. Docker caches every step of the build process (after each RUN command), so rebuilding can be very fast.

If you have a docker.io account, you can upload the image to the image registry like this

    sudo docker.io login
    sudo docker.io push oslandia/pggis

Conclusion

This is a first dive into Docker, with a spatial GIS database container. There is a lot more you can do with Docker, and plenty of use cases for this spatial database setup. This Docker image, the setup and the way it is built can be improved a lot as well, and will certainly change along with package availability, ppa repositories, next versions of components, better Docker practices and versions, but it is already a good and efficient way to get the stack up quickly. Do not hesitate to fork the GitHub repository  and send Pull Requests.

You can also report any issue on GitHub https://github.com/vpicavet/docker-pggis/issues

If you want to go further, or if you have any remark concerning the tools covered in this blog post, do not hesitate to contact us : infos+pggis@oslandia.com

by oslandia at January 23, 2017 03:41 PM

December 11, 2016

Paul Ramsey

PostgreSQL "Compatible" Aurora

PostgreSQL "Compatible" Aurora

While I know full well that Amazon’s marketing department doesn’t need my help, I cannot resist flagging this new development from the elves in Santa’s AWS workshop:

Today we are launching a preview of Amazon Aurora PostgreSQL-Compatible Edition. It offers … high durability, high availability, and the ability to quickly create and deploy read replicas. Here are some of the things you will love about it:

Performance – Aurora delivers up to 2x the performance of PostgreSQL running in traditional environments.

Compatibility – Aurora is fully compatible with the open source version of PostgreSQL (version 9.6.1). On the stored procedure side, we are planning to support Perl, pgSQL, Tcl, and JavaScript (via the V8 JavaScript engine). We are also planning to support all of the PostgreSQL features and extensions that are supported in Amazon RDS for PostgreSQL.

Cloud Native – Aurora takes full advantage of the fact that it is running within AWS.

The language Amazon uses around Aurora is really wierd – they talk about “MySQL compatibility” and “PostgreSQL compatibility”. At an extreme, one might interpret that to mean that Aurora is a net-new database providing wire- and function-level compatibility to the target databases. However, in the PostgreSQL case, the fact that they are additionally supporting PostGIS, the server-side languages, really the whole database environment, hints strongly that most of the code is actually PostgreSQL code.

There is not a lot of reference material about what’s going on behind the scenes, but this talk from re:Invent shows that most of the action is in the storage layer. For MySQL, since storage back-ends are pluggable, it’s possible that AWS has added their own back-end. Alternately, they may be running a hacked up version of the InnoDB engine.

For PostgreSQL, with only one storage back-end, it’s pretty much a foregone conclusion that AWS have taken a fork and added some secret sauce to it. However, the fact that they are tracking the community version almost exactly (they currently offer 9.6.1) indicates that maybe their fork isn’t particularly invasive.

I’d want to wait a while before trusting a production system of record to Aurora PgSQL, but the idea of the cloud native PostgreSQL, with full PostGIS support, excites me to no end. RDS is already very very convenient, so RDS-with-better-performance and integration is just icing on the cake for me.

I, for one, welcome our new cloud database overlords.

December 11, 2016 04:09 PM

December 09, 2016

Boundless Geo

LocationTech Tour 2016

If you caught our update on FOSS4G 2016 you know we enjoy taking part in community events and open source advocacy. The LocationTech Tour is the second big open source GIS mapping conference & community event we take part in; …

The post LocationTech Tour 2016 appeared first on Boundless Geo.

by Jody Garnett at December 09, 2016 05:13 PM

Boundless Geo

LocationTech Tour 2016

If you caught our update on FOSS4G 2016 you know we enjoy taking part in community events and open source advocacy. The LocationTech Tour is the second big open source GIS mapping conference & community event we take part in; …

The post LocationTech Tour 2016 appeared first on Boundless Geo.

by Jody Garnett at December 09, 2016 05:13 PM

November 28, 2016

PostGIS Development

PostGIS 2.3.1 Released

The PostGIS development team is pleased to announce the release of PostGIS 2.3.1. Best served with pgRouting 2.3.1 and PostgreSQL 9.6.1.

As befits a patch release, the focus is on bugs and breakages.

Continue Reading by clicking title hyperlink ..

by Regina Obe at November 28, 2016 12:00 AM

November 26, 2016

PostGIS Development

PostGIS 2.2.4 Released

The PostGIS development team is pleased to announce the release of PostGIS 2.2.4 As befits a patch release, the focus is on bugs and breakages.

Continue Reading by clicking title hyperlink ..

by Regina Obe at November 26, 2016 12:00 AM

November 23, 2016

Postgres OnLine Journal (Leo Hsu, Regina Obe)

ODBC FDW now supports 9.5 and 9.6

A while ago when Foreign Data Wrappers in PostgreSQL was a fairly new thing, we talked about the ODBC_FDW foreign data wrapper. Since then, people have been asking us how to get the ODBC FDW to work on newer PostgreSQL. Sadly the ODBC_FDW was stuck in time not having updated to newer FDW API standards. Our recommendation was just to use OGR_FDW, which many distributions both Linux and Windows have compiled OGR_FDW with ODBC support. True that OGR_FDW is coined as a spatial data wrapper, but the reality is spatial data rarely lives apart from regular attribute data so a good spatial vector driver supports both vector data and bread and butter data types. OGR_FDW is still our go to for working with spreadsheets and folders of CSV files.

Recently the fine folks at Carto patched the ODBC FDW to work with PostgreSQL 9.5. I do hope they accept my modest patch to make it work with PostgreSQL 9.6 as well. So now 2 FDWs to choose from for connecting to ODBC datasources. Which one is better? The answer as most always is IT DEPENDS.


Continue reading "ODBC FDW now supports 9.5 and 9.6"

by Leo Hsu and Regina Obe (nospam@example.com) at November 23, 2016 05:39 AM

October 17, 2016

BigSQL Holly

Two great things that taste great together: PostGIS and PostgreSQL

In my past life as a geospatial DBA, I had to navigate users and managers who belonged to different database teams (PostgreSQL, MySQL, SQL Server, Oracle, etc). When I was lucky enough to work in a shop that supported open source solutions, I was often asked by members of team MySQL, Why should we use PostgreSQL as our geospatial database when MySQL has spatial data types?” 

The answer: PostGIS.

Sure, MySQL has the ability to run spatial analysis (with some prodding). But PostgreSQL + PostGIS wins in:

  • functionality
  • performance
  • adoption by 3rd party solutions (QGIS, ArcGIS Server, GeoServer…)
  • So if you like to do things the free and easy way, go with PostgreSQL and PostGIS.

    by Holly Orr at October 17, 2016 02:59 PM

    October 10, 2016

    UpStats blog (Stefan Petrea)

    Off the streets, Land subdivision in PostGIS - Part 2

    Intro

    In a previous post, we've seen how PostGIS and Openstreetmap can be used to leverage geographical data.

    A common task in GIS(in particular land use planning) is land subdivision. This involves taking different shapes, polygons usually, and cutting them into smaller polygons.

    This post will focus on describing an algorithm for partitioning a land polygon into parcels of a given area. Parcels will be cut off from the initial polygon until no more parcels can be formed.

    This is part2 of a series:

    Algorithm description

    We have a polygon P and the nearest road R. We get the bounding box B for P and all our searches for cutting points/lines will be confined to B. We compute the extreme points on the bounding box and we label them with the cardinal directions they represent.

    Our goal is to cut a corner C (also called subdivision) from P such that it contains the nearest boundary point to a road. The cut will be done using two lines, one horizontal and one vertical. We want C to be of a given area A.

    Now in order to find the corner to cut, we look at the extreme points and check which one of them is closest to a road.

    We'll use sweeping-lines to find the parcel we need. Any sweeping-lines mentioned will be moving away from the corner (in other words, away from the closest road point).

    In what follows, we assume the north-west corner needs to be cut.

    We place an inset (a horizontal line) that will be located sqrt(A) to the south (relative to the north edge). The inset is positioned there because we anticipate the target area to be in the form of a square.

    If the area above the inset (the one we aim for) is larger than our target, we split the polygon, take the upper half and use another sweeping line that goes from west to east, to find another cutting line that allows us to find a parcel with the required area.

    If the area above the inset is insufficient (below the target area), we search for a better position for it, using binary search, along the north-south direction.

    Additional details: The way the cut search works, using the inset, is such that we avoid getting thin horizontal strips when our initial polygon is a square/rectangle (and it is expected to be a square in the vast majority of cases).

    Details about corner cases (other than NW which was covered above):

    • NE corner: horizontal goes north->south and vertical goes east->west
    • SE corner: horizontal goes south->north and vertical goes east->west
    • SW corner: horizontal goes south->north and vertical goes west->east

    So the sweep lines always move away from the corner.

    After the parcel with target area was found, it will be cut off from the polygon, the original polygon in the GIS database will be updated and the new parcel will be inserted (separate from the original polygon that we split).

    Demo

    OSM was used to extract test data. Specifically, in the run below, data for the Herastrau Park in Bucharest was used. In OSM, the park is represented as a series of polygons, and one of the polygons was partitioned in parcels, each measuring 8000 square meters.

    On the left side you can see the actual partition. On the right side, the same partition is displayed, except we also have some of the objects used throughout development, including bounding boxes, and extreme boundary points.

    And to visualize how the algorithm works, here's an animation of how it partitions the polygon, parcel by parcel:

    Challenges

    • While working on this algorithm implementation, at one point, it was required to find the extreme boundary points and assign them cardinal orientation labels. A first approach to determining the cardinal direction of the boundary points was to order them by ST_Azimuth relative to the centroid of the polygon (as the circle center passed to ST_Azimuth). This is a pitfall because the clock-wise order doesn't guarantee in any way membership to the N,E,S,W edges of the bounding box. It's perfectly possible that two such extreme points belong to the north edge, and one belongs to the west edge, and one to the south edge.
    • As it turns out, splitting the polygon multiple times, and using ST_Union to glue the remaining parts back together will create MULTIPOLYGON structures and which ST_ExteriorRing is not compatible with. So a convex hull was used in order to get one single polygon and compute the bounding box for it. This problem will surface after repeatedly cutting the input polygon.
    • Due to lack of knowledge about QGIS, some logic for an SVG-based 1 visualization was written in order to check the position of the cuts and the shape of the polygon after the cuts were made. Although QGIS has virtual layer feature, that was quite hard to use.
    • Early on, there was a need to visualize some objects from OSM in order to assess if they are fit for test data. It was easier to create a VIEW of those objects in PostgreSQL and then visualize that in QGIS (it was a bit hard to deal with all the menus in QGIS)
    • It would've been nice to have some automatically generated test polygons for each position of the road relative to the polygon. And some polygons that would be thiner near the top and wider near the bottom. However, the polygons extracted from OSM were used instead.
    • To get the extreme points of the polygon, the first approach used was to intersect exterior ring of the polygon with the exterior ring of the polygon's envelope (envelope meaning bounding box). This proved to be complicated and would definitely add complexity if one of the polygon's edges were axis-parallel as the result of the intersection would've been a line, and not a point. So the second approach was much simpler, just decomposing the polygon into its vertices (using ST_DumpPoints) and then picking the northmost, eastmost, westmost and southmost vertices. While this works, it only works for POLYGON and MULTIPOLYGON but it's not expected to work for CURVEPOLYGON.

    After-thoughts

    While implementing the logic for each corner-cut, it became more obvious that the logic for one corner would be enough, and the entire drawing could be rotated, the cut made, and then the resulting pieces rotated back. This could have decreased the size of the implementation.

    No strict constraints are made on the shape of the parcels. Most of them are square, but some diverge from that shape quite a lot.

    Some polygons may have U-shapes and if the closest-point on the boundary is one of the two tips of the U shape, the parcel that gets cut could turn out to be one made up of disconnected pieces.

    Another gap that was not covered by the initial spec and is not currently handled is the row-excess which can form at the end of a row (in the example it will be at the end of the row). This row-excess is another factor that can generate thin parcels because once formed, the next split will use this excess and add a very thin portion beneath it to form the new parcel. This is not very helpful, but it's a known issue.

    Someone pointed out that there should also be access paths/roads but that wasn't part of the problem statement. Another aspect that was not taken into consideration was surface inclination, it's assumed that subdivided area is flat.

    Conclusion

    In this post we've seen how PostGIS can be leveraged to build a land subdivision algorithm. The implementation for this algorithm is available on Github under MIT license.

    Footnotes:

    1

    Satoshi Koda has a great blog post about this, and that was very useful reading while writing the SVG visualization for this project. His new blog is hosted here.

    October 10, 2016 09:00 PM

    October 06, 2016

    PostGIS Development

    PostGIS 2.2.3 Released

    The PostGIS development team is pleased to announce the release of PostGIS 2.2.3 As befits a patch release, the focus is on bugs and breakages.

    Continue Reading by clicking title hyperlink ..

    by Regina Obe at October 06, 2016 12:00 AM