Blogging Offline Using Markdown and Brackets

This post is not GIS related, but I thought I'd write about it. I'm using Squarespace for this website, and it can be a pain to write blog posts in the online editor. I want to be able to write when I'm offline and post when I'm done. So, I've been thinking of a workflow to do this. I'm experimenting with Markdown and Brackets to do this. I love Brackets for coding - it's an elegant piece of software with lots of extensions/plugins to do everything I need it to. For more info on Markdown, see John Gruber's site for the original/standard Markdown specs and usage. Squarespace has a Markdown cheatsheet as well. You can also find other flavors of Markdown as well. Note that you don't have to use Brackets to write your blog post in Markdown - any text editor (e.g. Wordpad, notepad, Word, etc) as long as you use the Markdown syntax.

Here is my workflow. If anyone happens to know of a more efficient way to blog offline with Squarespace, please let know.

Install Brackets with Markdown Plugins

  1. Download and install Brackets. Then use the Extension Manager in Brackets to download and install the Markdown Preview and Markdown Toolbar plugins.

  2. Write your blog post in Brackets using Markdown syntax as I am doing here. See Squarespace's Markdown cheatsheet for more info.

Below is a snapshot of my Brackets workspace.
https://www.opengislab.com/s/Fig0001.png

Set up Squarespace "Media" Folder

  1. In my Squarespace Pages directory, I've created a folder in the "Not Linked" section to upload and organize my images. I save my images in the order that I will insert them into my blog page (e.g. Fig0001.png, Fig00002.png). Each image is added as a "Link" page in the folder.

To access the URL of the images, click the linkname and see the properties. You'll see that it begin with "/s/filename.xxx". Just append the URL of your site in front of that - e.g. https://www.yoursite.com/s/filename.xxx

https://www.opengislab.com/s/Fig0002.png

2.. When you're done writing, just copy and paste the Markdown text into your blog page using Squarespace's Markdown Block

https://www.opengislab.com/s/Fig0003.png

https://www.opengislab.com/s/Fig0004.png

That's it. And for anyone who is interested, I used a software called FastStone Capture to capture my screenshots - I've been using this for years and like it a lot. You can't beat $20 for a lifetime license. It's not as fancy as something like Snagit but has a low learning curve and does everything I need.

Hope you enjoy this post. Until next time...

Using USGS TNM Vector Data to Create Seamless Topographic Maps in ArcGIS

One of the most popular download on this site is how to convert USGS geopdf to geotiffs using ArcGIS.  So I started to thinking about why not use the USGS Topo Vector Data to create seamless maps that are very similar to the USGS geopdfs? Why do I want to use the USGS Topo Vector Data to create topographic maps? Because it’s a hassle and a pain in neck to get USGS GeoPDFs into a GIS, and sometimes the web mapping services just don’t cut it (e.g. resolution not good enough, limited control over styling, etc.).

This post is a follow up post to the previous, USGS TNM Style Template ArcGIS. Data for Hawaii is now available to use with the ArcGIS map document template. I like the USGS TNM Topographic maps as basemaps – it provides an informative but clean background for whatever data layers I overlay on top of it and plus most people are familiar with it. If you are looking to recreate a USGS Topographic Quadrangle maps (7.5-Minute topo quads) in ArcGIS – I suggest you follow the instructions given on the USGS site. What I want is to assemble a statewide dataset for use with the TNM style template to create a seamless USGS topo maps. You should read the USGS FAQ on using the USGS TNM style template.

Step 1: Download Datasets and TNM Style Template

So, the first step is to download the datasets needed for Hawaii and the style template.

1. A good place to look for data is here: https://viewer.nationalmap.gov/basic/ or go directly to https://viewer.nationalmap.gov/tools/topotemplate/

First you should download the TNM Style template and Topo Map Vector Data GDBs for all the 7.5-Minute quads of interest. Here I have the Topo_TNM_Style Template which contains the ArcMap map document that will be used for the styling and map making later.

Fig0002.png

Also I have downloaded all the vector data by island. Make sure you download data as file geodatabases and not shapefiles. If you know what you're doing then you can go directly to the Cloud Browse FTP site and download what you need. Unfortunately the listing of staged vector data doesn't include Hawaii so I have use the data download service instead.

Topo Vector Data for 7.5-Minute cells for Oahu

Topo Vector Data for 7.5-Minute cells for Oahu

If you look at the topo vector data GDB file you'll see that it contains the following data - shown below is a example for Pearl Harbor quad, but all the GDBs should contains the same datasets. Using this as a reference I'm going to download the statewide (Hawaii) equivalents to created my Statewide Topo Vector Data GDB.

Fig0004.png

You will need to download the following datasets:

Fig0005.png
  • GOVTUNIT_Hawaii_State_GDB.gdb - file GDB containing boundary layers for Hawaii
  • MAPINDICES_Hawaii_State_GDB.gdb - file GDB containing map indices for Hawaii. You'll want to use the 7.5-Minute map index
  • NHD_H_Hawaii_GDB.gdb - file GDB containing the National Hydro Dataset for Hawaii
  • STRUCT_Hawaii_State.GDB.gdb - file GDB containing point structures for Hawaii
  • TRAN_15_Hawaii_GU_STATEORTERRITORY.gdb - file GDB containing transportation layers for Hawaii
  • WOODLAND_15_Hawaii_GU_STATEORTERRITORY.gdb - file GDB containing woodland land use for Hawaii

For the Elevation Data - you can download the contour data directly from the Cloud Browse FTP site or use the Download GIS Data web service. Here I am using the data download service.

Fig0006.png
Fig0007.png

Step 2: Create an empty geodatabase schema

So after downloading the data I need, I going to create an empty geodatabase schema based on one of the Vector Data Quad GDB (e.g. Vector_34545_Pearl_Harbor_7_5_Min.gdb). I'm going to then load the statewide datasets into this new GDB and this will be my new Topo Vector Data GDB for Hawaii.

Export existing Vector_xxx_7_5_Min.gdb schema

1. In ArcCatalog, right click on the Vector_xxx_7_5_Min.gdb >> Export >> XML Workspace Document

2. In the Export XML Workspace window: Select only the schema to export and give the output xml a name. Click Next and make sure you Select all features to included in the export

Fig0008.png
Fig0009.png

Create a new file geodatabase with existing schema

1. In ArcCatalog, create a new file geodatabase then import the existing XML schema your exported previously (Right click on file GDB >> Import >> XML Workspace Document)

2. In the Import XML Workspacec Document, select to import only the schema

Fig0010.png

Step 3: Load Data into the New Statewide GDB

1. You should now have an empty file GDB with the proper schema to load data into.

Fig0011.png

2. In ArcCatalog start loading the corresponding statewide data into the correct feature classes. For example, here I'm going to load the features from my MAPINDICES_Hawaii_State_GDB.gdb\Cells\CellGrid_7_5Minute layer into the corresponding TopoVectorDataHawaii_2017.gdb\CELLS\CellGrid_7_5Minute layer. Right click on the feature class for which you want to load features into >> Load >> Load Data. In the Simple Data Loader window: input your parameters.

Fig0012.png
Fig0013.png
Fig0014.png
Fig0015.png

3. Use the Simple Loader method to load features into all the corresponding feature classes - exceptions below. You can load features from multiple layers -- see example below for elev_contours.

Note: For Transportation Road features the Target Fields and Matching Source Fields will not all match (some Source Fields will be blank or None) - this should be ok since the fields for display (TNMFRC) and label (FULL_STREET_NAME) match. Also the TNMDerivedNames feature class does not have a statewide equivalent (explained further below) - follow Step 3A below for a work around solution.

Fig0016.png

Step 3A: Batch Selecting and Merging TNM Derived Names Points

The USGS TNM Style guide says that "TNM Derived Names is a filtered and enriched dataset intended specifically to be used for labeling named features. TNM Derived Names data are provided only in conjunction with Topo Map Vector Data products." So there are no national or statewide data for TNMDerivedNames layer - you'll have to merged all the points from each of the Vector Quad GDB. Note that you can't just merged the points together because a quad can contain points for other adjacent quads as well so you'll end up having duplicates if you don't filter them first. What you'll need to do is select only the points in the quad of interest then do the merge so the resulting layer won't contain duplicates (see example image below:

Fig0017.png

Batch Selecting TNM Derived Names Points of Interest

The example shown here is batch selecting features for all 6 quads for Lanai.

1. Create a new file GDB for the data staging (e.g. LanaiStaging.gdb)

2. In ArcToolbox, go to Analysis Tools >> Extract >> Right click on Select >> Batch

3. In the Batch Select grid: (there are 6 quad cells so there should be 6 output feature classes)

  • Input Features: TNMDerivedNames layer from each quad
  • Setup your Environment settings to use your staging GDB as the Scratch Workspace (e.g. LanaiStaging.gdb)
  • Output Feature Class: Use the default or input a name
Fig0018.png

Here is an example for the LanaiStaging.gdb

Fig0019.png

Merging or Loading the TNMDerivedNames Output Feature Classes

After you are done with the batch feature selections, use the Simple Data Loader (right click on the target layer >> Load >> Load Data) to load in the features from your staging GDB into the TNMDerivedNames feature class. Alternatively you could also use the Merge Tool in ArcCatalog.

Fig0020.png

Once you are done with the merging or feature loading, you can then link the data sources in the TNM Style Template MXD.

Step 4: Linking Data Sources to TNM Style Template

1. Open the TNM Style Template map document (e.g. TopoTNMStyleTemplate_v10_0.mxd)

2. Modify the map document to fit your needs:

  • Delete data frames that you don't need - I only kept the Map Layers and Images Data Frames
  • Change the coordinate system of the Map Layers Data Frame
  • Set the Data Source links for layers with missing data links. Note PLSS does not exist for Hawaii.
  • Save a copy of the map document
Fig0021.png

3. "Layers" under the Images Data Frame are web services. Note that the WetlandsTopoService needs fixing (exising URL to service is doesn't work).

  • In a web browser, go to https://viewer.nationalmap.gov/services/ and scroll down to Themed Overlays  >> FWS Wetlands - Topo Symbols >> Click on REST - this should open a new page to the REST service - copy this URL up to "...services/"
Fig0022.png
  • In ArcCatalog, add this web service. Go to GIS Servers >> Add ArcGIS Server. In the Add GIS Server window: select use GIS server >> Next. Then Paste the Server URL you copied and click Finish
Fig0023.png

You should now have the FWS Wetlands Topo Services added to your GIS server in ArcCaalog - see below.

Fig0024.png

4. Back in the ArcMap document, what you have to do is remove the existing WetlandsTopoService then add it back in using the new FWS GIS Server. Note: Setting the data source/link to the existing WetlandsTopoService didn't seem to work for me - not sure why (I'm using ArcGIS Desktop 10.5.1).

Fig0027.png

5. Modify your map document and map layout to fit your needs. That's it.

Example showing a "seamless" TNM Style map -- very similar to the USGS topo quad maps

Example showing a "seamless" TNM Style map -- very similar to the USGS topo quad maps

Another example showing a more close up view with no modifications of labels or symbology - note since this is all vector data (with exception of web services) everything looks sharp and can be modified as needed.

Another example showing a more close up view with no modifications of labels or symbology - note since this is all vector data (with exception of web services) everything looks sharp and can be modified as needed.

Installing Apache, QGIS Server, and Lizmap Web Client on Windows OS

This post is my notes on the process/procedures I went through to install QGIS Server, Lizmap Web Client (Lizmap WC), and Apache on a Windows 10 x64 laptop and Windows Server 2012 R2 machine. For the most part I followed the instructions on Lizmap site, Installing Lizmap Web Client on Windows (last accessed 6/13/2017), however the instructions there seems like it hasn't been updated in a while and some things didn’t quite work as expected when I tried them. So, this blog post is my version of those instructions including notes on what I did to make all the components work. A look at the demo is shown below with notes/instructions below if you want to try it yourself. Sorry there are no screenshots.

Lizmap Web App Demo Slides

Requirements/Software used:

  • Lizmap WC and QGIS Server must be on same machine
  • QGIS Server LTR 2.14 (installed via OSGeo4W x64)
  • QGIS Desktop (installed via OSGeo4W x64)
  • Lizmap Web Plugin (installed via QGIS desktop plugin manager)
  • Lizmap Web Client 3.1.1 – download link
  • Apache 2.4
    • Mod fcgid
    • C++ Redistributable VS 2015
    • PHP 7.1

For my test on my Windows 10 laptop, I have everything on the same machine. But on the Windows Server 2012 machine, I have QGIS Desktop with Lizmap Web Plugin on a different machine. For my installation I put Apache and Lizmap side-by-side inside a C:/webserver directory. You can however, put your installations inside of Apache root directory (e.g. C:/Apache24/Lizmap) or inside Apache htdocs directory (e.g. C:/Apache24/htdocs/lizmap). If installation of lizmap and Apache are side by side or lizmap is inside Apache root directory then you need to properly configure the virtual hosts.

Part 1: Apache 2.4 Server Installation and Configuration

For more info on installing Apache on Windows, see: https://httpd.apache.org/docs/2.4/platform/windows.html#winsvc

1.       Create a webserver directory on C: Drive (e.g. C:\webserver)

2.       Go to apachelounge.com/download page

3.       Download Apache 2.4  httpd-2.4.25-win64-VC14.zip

You should read the read me file for more info on installing Apache.

4.       Download  mod_fcgid-2.3.9-win64-VC14.zip – make sure this matches you Apache download above (e.g. xxx-win64-VC14)

5.      Download C++ Redistributable VS 2015 – the link to vc_redist_x64/86.exe is broken when I tried it on 6/12/2017. You can try this link  here: https://www.microsoft.com/en-us/download/details.aspx?id=52685

Note: The Visual C Redistributable Package is needed to run Apache HTTPD as a Windows Service but the version required will depend on which binary you are installing: If you are installing Apache 2.4 VC14 then you need Microsoft Visual Studio C 2015 Redistributable

6.       Install C++ VS 2015 (vc_redist.x64.exe) that you downloaded.

7.       Unzip or extract from httpd-2.4.25-win64-VC14.zip\Apache24 directory to C: \webserver folder (e.g. C:\webserver\Apache24)

8.       Unzip or extract from mod_fcgid-2.3.9-win64-VC14.zip\mod_fcgid-2.3.9\mod_fcgid.so to C:\webserver\Apcahce24\modules folder

9.       Open the Apache configuration file: C:\webserver\Apache24\conf\httpd.conf in a text editor (I use Notepad++)

  • Replace all occurrences of c:/Apache24 to c:/webserver/Apache24
  • Find “ServerName” – this should be ServerName localhost:80 (if using default port 80) or if not using default port then it should be ServerName localhost:port (e.g. ServerName localhost:90)
  • Change the default port if needed. Find “Listen 80” to Listen port (e.g. Listen 90). You need to use a port that is not already in used by another service.
  • Find “DirectoryIndex index.html” and add index.php (e.g. DirectoryIndex index.html index.php)
  • Save and close the file when done

10.   Check to see is Apache will start:

  • Open the Command Prompt window: type cmd and run it as administrator
  • Command window: Type in C:\webserver\Apache24\bin\httpd.exe
  • Notice of Windows Firewall popup: Allow Access for All

Note from Lizmap Installation docs: If a popup didn’t display, it probably means you have an anti-virus software managing your firewall in which case you will need to allow the Apache service. Also you may need to add port 80 or whatever port you use as inbound/outbound port in Advanced Windows Firewall Properties (Control Panel >> Administrative Tools >> Windows Firewall with Advanced Security)

Note: On Windows 10 laptop the popup showed and on Windows Server 2012 R2 I didn’t get a popup. However, when I opened http:localhost in web browser it works.

  • Open a web browser and type in http://localhost if using default port 80 or localhost:port if using a different port (e.g. http://localhost:90)

You should see a webpage with “It Works” – this means Apache is installed correctly

  • Back in the Command Prompt window: Crtl + C to stop Apache. This may take a few seconds.

11.   Append Apache to your Windows System Path so you can directly access Apache by typing httpd.exe instead of the full path to the exe.

12.   Start Apache as windows service: https://httpd.apache.org/docs/2.4/platform/windows.html#winsvc

  • In Command Prompt window: cmd (as administrator)
  • To install Apache as a service: httpd.exe -k install
  • Check windows services to see that Apache in installed and started
  • Stop Apache for now if it’s started – you’ll configure it and then restart later

13.   Open the Apache configuration file: C:\webserver\Apache24\conf\httpd.conf in a text editor. Enable the following modules by uncommenting or removing the # character.

Note: if you don’t see the module already in the file then you need to type it in

  • LoadModule actions_module modules/mod_actions.so
  • LoadModule deflate_module modules/mod_deflate.so
  • LoadModule expires_module modules/mod_expires.so
  • LoadModule ext_filter_module modules/mod_ext_filter.so
  • LoadModule fcgid_module modules/mod_fcgid.so
  • LoadModule headers_module modules/mod_headers.so
  • LoadModule ident_module modules/mod_ident.so
  • LoadModule rewrite_module modules/mod_rewrite.so
  • LoadModule ssl_module modules/mod_ssl.so

Note: If you are using a different port than the default 80 then you need to also enable these modules as well.

  • LoadModule cache_module modules/mod_cache.so
  • LoadModule cache_disk_module modules/mod_cache_disk.so
  • LoadModule proxy_module modules/mod_proxy.so
  • LoadModule proxy_connect_module modules/mod_proxy_connect.so
  • LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
  • LoadModule proxy_http_module modules/mod_proxy_http.so

Add the following code to the end of the httpd configuration file:

<IfModule mod_deflate.c>
SetOutputFilter DEFLATE
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\\.0[678] no-gzip
BrowserMatch \\bMSIE !no-gzip !gzip-only-text/html
SetEnvIfNoCase Reuqest_URI \\.(?:gif|jpe?g|png|rar|zip)$ no-gzip dont-vary
Header append Vary User-Agent env=!dont-vary
</IfModule>

14.   Save the changes to httpd.conf file then restart Apache

Part 2: PHP 7.1 Configuration

Note: Lizmap Web Client require at least PHP version 5.4. Also make sure you use the non-thread safe file

1.       Go to PHP downloads page and download the appropriate file for your OS and Apache version installed earlier – I install VC14 x64 non-thread safe file, php-7.1.6-nts-Win32-VC14-x64.zip

2.       Unzip or extract the zipped file contents to your php directory (e.g. C:\webserver\php-7.1). Create the php folder if you need to inside the main webserver directory)

3.       Make a backup copy of C:\webserver\php-7.1\php.ini-production just in case. Then rename it to php.ini

4.       Open C:\webserver\php-7.1\php.ini in a text editor

  • Uncomment out the extension_dir = “ext” to say the following (this is the path to your php ext folder) extension_dir = "C:\webserver\php-7.1\ext"
  • Enable the following php modules:
    •  extension=php_curl.dll
    • extension=php_fileinfo.dll
    • extension=php_gd2.dll
    • extension=php_mbstring.dll
    • extension=php_pdo_pgsql.dll
    • extension=php_pdo_sqlite.dll
    • extension=php_pgsql.dll
    • extension=php_sqlite3.dll
  • Find where it says upload_max_filesize and change to 15M
  • Find where it says post_max_size and change to 15M

5.       Save the php.ini file

6.       Add PHP to your Windows System Path

7.       Open Command Prompt window: check PHP is correctly declared in System Path. Type php-m

8.       In Command Prompt window: check Apache configuration. Type httpd -S

9.       Create a php configuration in Apache for using PHP.

  • Open a new file in a text editor then put in the following code:
FcgidInitialEnv PHPRC "C:\\webserver\\php-7.1"
 <FilesMatch \\.php$>
AddHandler fcgid-script .php
FcgidWrapper "C:/webserver/php-7.1/php-cgi.exe" .php
</FilesMatch>
  • Save the file as php-7.1.conf in C:\webserver\Apache24\conf\extra

10.       Open C:\webserver\Apache24\conf\httpd.conf in text editor

  •  Uncomment out the line to include virtual host configuration file and save the file

 Include conf/extra/httpd-vhosts.conf

11.       Open C:\webserver\Apache24\conf\extra\httpd-vhosts.conf in text editor

At the end of the file copy and paste following code:

Note: change the port if using something other than 80

<VirtualHost *:80>
  Include conf/extra/php-7.1.conf
  ServerName localhost
  DocumentRoot "C:/webserver/lizmap/"

  <Directory "C:/webserver/lizmap">
  Options -Indexes +FollowSymLinks +ExecCGI
  AllowOverride All
  Require all granted
 </Directory>

 CustomLog "logs/lizmap-access.log" common
 ErrorLog "logs/lizmap-error.log"
</VirtualHost>

12.       Create Lizmap folder inside the webserver directory (e.g. C:\webserver\lizmap)

13.       Create a text file with the following code and save as index.php inside lizmap directory (e.g. C:\webserver\lizmap\index.php)

php <?php phpinfo(); ?>

14.       Restart Apache service

15.       Open a web browser and type in http://localhost/index.php (don’t forget to include the port if you are using something other than default 80 – e.g. http://localhost:90/index.php)

You should get a webpage with the PHP information.

          Note: If you are getting a 403 Forbidden Access Error. Open the C:\webserver\Apache24\conf\httpd.conf

In a text editor and change the access permission to your server’s filesystem to Require all granted. Save the file and restart Apache service then try to access index.php again.

Fig01.png

Part 3: QGIS Server Installation and Configuration

I installed QGIS Server using OSGeo4W installer x64 version – download on QGIS site or on OSGeo4W site. Lizmap Web Client doesn’t work properly with the newer versions of QGIS Server, so you must use the LTR 2.14.

Note: When I installed QGIS desktop and server LTR 2.14 via the Advanced Setup – I had issues with QGIS starting up properly (something about missing components), even though the installation itself didn’t give me any errors. So I uninstalled and started fresh with Express Setup first then Advanced Setup and that resolved my issues.

Express Setup

  • Run the installer: Select Express Setup
  • Install from Internet
  • Root Directory: use default (C:\OSGeo4W64) and install for all users
  • Keep default Local Package Directory and Start Menu Name
  • Select the Packages: GDAL, Grass, QGIS Desktop
  • Install software dependencies required and let installation complete.

Advanced Setup

  • Run the installer again: Select Advance Setup
  • Install from Internet
  • Keep Root Directory default and install for all users
  • Keep default Local Package Directory and Start Menu Name
  • Select these Packages:
    • Commandline_Utilities: GDAL
    • Desktop: QGIS LTR 2.14 full
    • Libs: fcgi
    • Optional Lib: filegdb (only if you want to be able to use ESRI's file gdb)
    • Web: QGIS Server LTR 2.14
  • Resolve Unmet Dependencies: Install the required dependencies, accept license agreements, and allow installation to complete
  • After installation completes, check in C:\OSGeo4W64\apps\qgis-ltr\bin folder to make sure these files are in there.
    • qgis_mapserv.fcgi.exe
    • qgis_server.dll
    • wms_metadata.xml

Part 4: QGIS Server Configuration

1.       Open C:\webserver\Apache24\conf\extra\httpd-vhosts.conf in text editor and modified to contain this content:

Note: change host port if needed if other then 80

<VirtualHost *:80>
 Include conf/extra/php-7.1.conf
ServerName localhost

# Lizmap Production
DocumentRoot "C:/webserver/lizmap/prod/"
<Directory "C:/webserver/lizmap/prod">
 Options -Indexes +FollowSymLinks +ExecCGI
AllowOverride All
Require all granted
</Directory>

# LizMap Pre-production
Alias /preprod/ "C:/webserver/lizmap/preprod/"
<Directory "C:/webserver/lizmap/preprod">
Options -Indexes +FollowSymLinks +ExecCGI
AllowOverride All
Require all granted
</Directory>

Alias /qgis/ "C:/OSGeo4W64/apps/qgis-ltr/bin/"
<Directory "C:/OSGeo4W64/apps/qgis-ltr/bin/">
SetHandler fcgid-script
Options +ExecCGI
AllowOverride All
Require all granted
</Directory>

Alias /document/ "C:/webserver/data/document/"
<Directory "C:/webserver/Data/document">
Options -Indexes +FollowSymLinks
AllowOverride All
Require all granted
</Directory>

CustomLog "logs/lizmap-access.log" common
ErrorLog "logs/lizmap-error.log"
</VirtualHost>

2.       Save changes in file

3.       Open C:/webserver/Apache24/conf/extra/php-7.1.conf and modify to contain this content:

FcgidInitialEnv PHPRC "C:\\webserver\php-7.1"

FcgidInitialEnv PATH "C:\OSGeo4W64\bin;C:\OSGeo4W64\apps\qgis-ltr\bin;C:\OSGeo4W64\apps\grass\grass-7.2.1\lib;C:\OSGeo4W64\apps\grass\grass-7.2.1\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\WBem"
FcgidInitialEnv QT_PLUGIN_PATH "C:\OSGeo4W64\apps\qgis-ltr\qtplugins;C:\OSGeo4W64\apps\Qt4\plugins"
FcgidInitialEnv PYTHONHOME "C:\OSGeo4W64\apps\Python27"
FcgidInitialEnv PYTHONPATH "C:\OSGeo4W64\apps\qgis-ltr\.\python;C:\OSGeo4W64\apps\qgis-ltr\.\python\plugins;C:\OSGeo4W64\apps\Python27\DLLs;C:\OSGeo4W64\apps\Python27\lib;C:\OSGeo4W64\bin;C:\OSGeo4W64\apps\Python27;C:\OSGeo4W64\apps\Python27\lib\site-packages"

FcgidInitialEnv QGIS_SERVER_LOG_LEVEL 0
FcgidInitialEnv QGIS_SERVER_LOG_FILE "C:\\webserver\\Apache24\\logs\\qgis_server.log"

FcgidIOTimeout 120
  FcgidInitialEnv LC_ALL "en_US.UTF-8"
  FcgidInitialEnv PYTHONIOENCODING UTF-8
  FcgidInitialEnv LANG "en_US.UTF-8"
  FcgidInitialEnv QGIS_DEBUG 1
  FcgidInitialEnv QGIS_SERVER_LOG_FILE "C:\\webserver\Apache24\logs\\qgis_server.log"
  FcgidInitialEnv QGIS_SERVER_LOG_LEVEL 0
  FcgidInitialEnv QGIS_PLUGINPATH "C:\OSGeo4W64\apps\qgis-ltr\python\plugins"

SetEnvIf Request_URI ^/qgis QGIS_PREFIX_PATH "C:\OSGeo4W64\apps\qgis-ltr"
SetEnvIf Request_URI ^/qgis TEMP "C:\Windows\Temp"

SetEnvIf Request_URI ^/qgis GDAL_DATA "C:\OSGeo4W64\share\gdal"
SetEnvIf Request_URI ^/qgis GDAL_DRIVER_PATH "C:\OSGeo4W64\bin"
SetEnvIf Request_URI ^/qgis PDAL_DRIVER_PATH "C:\OSGeo4W64\bin"
SetEnvIf Request_URI ^/qgis GDAL_SKIP "JP2ECW"
SetEnvIf Request_URI ^/qgis PROJ_LIB "C:\OSGeo4W64\share\proj"

<FilesMatch \.php$>
AddHandler fcgid-script .php
FcgidWrapper "C:/webserver/php-7.1/php-cgi.exe" .php
</FilesMatch>

4.       Save the file

5.       Restart Apache service

6.       Open a web browser and type in http://localhost/qgis/qgis_mapserv.fcgi.exe - include the port after localhost if you’re using something different than the default port (e.g. http://localhost:90/qgis/qgis_mapserv.fcgi.exe )

You will most likely get a page where it says Service unknown or unsupported.

Try typing in the following in a browser:

http://localhost:90/qgis/qgis_mapserv.fcgi.exe?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities

http://localhost:90/qgis/qgis_mapserv.fcgi.exe?SERVICE=WMS&REQUEST=GetCapabilities

If you get something back that isn’t an error then you are ok – means QGIS Server installed properly and you can call WMS Get Capabilites

Part 5: Lizmap Web Client Installation and Configuration

Using Lizmap Web Client 3.1.1 Other versions like 3.0.x or Master did not work for me. When I did install version 3.0.x the Lizmap demo site wouldn’t load in the web browser. Also for this portion I have several directories and subfolders for production and preproduction work on my server.  

1.       Download Lizmap Web Client – I used lizmap-web-client-3.1.1.zip.

2.       Create in C:\webserver\lizmap directoy, 2 new folders for your production and preproduction work

C:\webserver\lizmap\prod

C:\webserver\lizmap\preprod

3.       Create in C:\webserver directory a new cache directory to hold 2 new subfolders: prod and preprod

C:\webserver\cache\prod

C:\webserver\cache\preprod

4.       Unzip or extract lizmap-web-client-3.1.1.zip to C:\webserver\lizmap\prod directory. So what you end up having is C:\webserver\lizmap\prod\lizmap-web-client-3.1.1

5.       Open C:/webserver/Apache24/conf/extra/php-7.1.conf in text editor and modified the section to the virtual host to point to C:/webserver/lizmap/prod/lizmap-web-client-3.1.1/lizmap/www folder

Note: change host port if needed if other then 80

<VirtualHost *:80>
Include conf/extra/php-7.1.conf
 ServerName localhost

# Lizmap Production
DocumentRoot " C:/webserver/lizmap/prod/lizmap-web-client-3.1.1/lizmap/www/"
<Directory " C:/webserver/lizmap/prod/lizmap-web-client-3.1.1/lizmap/www ">
Options -Indexes +FollowSymLinks +ExecCGI
AllowOverride All
Require all granted
</Directory>

# LizMap Pre-production
Alias /preprod/ "C:/webserver/lizmap/preprod/lizmap-web-client-3.1.1/lizmap/www/"
<Directory "C:/webserver/lizmap/preprod/lizmap-web-client-3.1.1/lizmap/www">
Options -Indexes +FollowSymLinks +ExecCGI
AllowOverride All
Require all granted
</Directory>

6.       Save your changes

7.       Restart Apache service

8.       Open the Command Prompt window (run as Administrator) and type in the following lines:

cd C:\\webserver\\lizmap\\prod\\lizmap-web-client-3.1.1\\
cd lizmap/var/config
copy lizmapConfig.ini.php.dist lizmapConfig.ini.php
copy localconfig.ini.php.dist localconfig.ini.php
copy profiles.ini.php.dist profiles.ini.php
cd ../../..

9.       Keep the Command Prompt window open.

10.   Open C:\webserver\lizmap\prod\lizmap311\lizmap\var\config\ localconfig.ini.php in a text editor and include the following code (if you want to install the lizmap demo websites) – make sure you have admin rights to modify and save this file (e.g. run notepad++ as Administrator then modify and save changes).

[modules]
lizmap.installparam=demo

11.   Save your changes and close file

12.   Back in the Command Prompt window:  launch lizmap installation. After installation completes, close the command prompt window.

cd C:\webserver\lizmap\prod\lizmap-web-client-3.1.1\
php lizmap/install/installer.php

14.   In the Lizmap home page, click the Connect button on the top right - or go to http://localhost/admin.php

  • Login with default username = admin, password = admin
  • Change password
  • Lizmap Configuration menu: Modify the input as follows and then Save new configuration
  • WMS Server URL: http://127.0.0.1/qgis/qgis_mapserv.fcgi.exe
  • Cache root directory: C:/webserver/cache/prod/

15.   From the Administrator page, you can click on the Project and view the Montpellier – Transport demo. Click Load the Map button. Or you can just type in http://localhost/index.php/view/map/?repository=montpellier&project=montpellier

You should see the demo Montpellier web application.

16.   If I go to my website: http://xx.xx.xx/index.php  and you should also be able to see the demo application.

Lizmap Folder Organization (optional)

1.       Create Lizmap directories for organizing production and preproduction work. Create the directories via a *.bat file (e.g. write the following lines into a text editor and save as with .bat extension, such as CreateLZDir.bat). The examples below are directories for a main webapp directory with subfolders for map repositories in a prod and preprod environments.

mkdir C:\webserver\webapp\common\
mkdir C:\webserver\webapp\document\
mkdir C:\webserver\webapp\document\svg\
mkdir C:\webserver\webapp\prod\
mkdir C:\webserver\webapp\prod\maprep\
mkdir C:\webserver\webapp\prod\maprep\data
mkdir C:\webserver\webapp\prod\maprep\data\vector
mkdir C:\webserver\webapp\prod\maprep\data\raster
mkdir C:\webserver\webapp\prod\maprep\media\
mkdir C:\webserver\webapp\prod\maprep\media\js\
mkdir C:\webserver\webapp\prod\maprep\media\image\
mkdir C:\webserver\webapp\preprod
mkdir C:\webserver\webapp\preprod\maprep\
mkdir C:\webserver\webapp\preprod\maprep\data
mkdir C:\webserver\webapp\preprod\maprep\data\vector
mkdir C:\webserver\webapp\preprod\maprep\data\raster
mkdir C:\webserver\webapp\preprod\maprep\media\
mkdir C:\webserver\webapp\preprod\maprep\media\js\
mkdir C:\webserver\webapp\preprod\maprep\media\image\

2.       Save your *.bat file (e.g. createDir.bat) and then run it in the Command Prompt window.

  • In the Command Prompt window: cd into your directory that contains the *.bat file
  • Run the *.bat file

SVG and QGIS Server Bug:

I’m not sure if this still holds true or not, but not a bad idea to do it. 

In Apache you need to Add a vhost to publish SVG and images files via HTTP this will avoid the bug in QGIS Server under Windows which cannot display SVG icon when you have a relative path. Create a folder to contain the svg files then modify C:/webserver/Apache24/conf/extra/httpd-vhosts.conf . Add the codes before the CustomLog portion.

Alias /document/ "C:/webserver/webapp/document/"
<Directory "C:/webserver/webapp/document">
Options -Indexes +FollowSymLinks
AllowOverride All
Require all granted
</Directory>

Save and restart Apache service. Check if you can add svg file in the folder C:/webserver/webapp/document/svg then access it via http://localhost/document/svg/my_icon.svg and use it as the SVG path in the style properties of a vector layer.

After you have the installation set up, see the LizWeb Map Client Documentation on how to create your maps in QGIS and web application.

GIS4DAR ArcGIS Training Workshop - September 7-8, 2017

Sorry, it has been a while since my last posting. I have been thinking of re-arranging the site to also include ArcGIS guides/tutorials as well as other software applications that I'm learning and experimenting with. For example, I'm currently learning about Docker and how to use it as a way to easily build web apps in a container environment. Also I've been learning Sphinx for documentation writing. I'll write something up once I'm done with experimenting.

So with that said, this post is a summary of an ArcGIS training workshop that I did earlier in September of this year that I'd like to share. Special thanks go out to Paulo Maurin, Gretchen Chiques, Annette DeRoches, and Risa Orem at the NOAA Inouye Regional Center, Ford Island for organizing the GIS4DAR workshop and inviting me to do the training.

The purpose of this training workshop was to teach the Hawaii Department of Land and Natural Resources, Division of Aquatic Resources (DAR) staff how to apply GIS to their research. 

Below is a synopsis taken from the GIS manual and a preview of the Table of Contents so you can see what the training was all about.  You can also download the GIS4DAR Training Manual - Please be patient. The pdf file is about 100 pages and is ~6 MB. The data for the training is really large (~1 GB) so I'm still trying to figure what is the best way to post it for downloading. But in the meantime if you would like it  - please email me directly.

About the Training/Tutorial

A GIS (Geographic Information System) is a computer-based system for the input, storage, analyses, and display of spatial data. This training workshop provides an introductory to intermediate training in GIS using ArcGIS Desktop 10.4.1. No GIS experience is required. The purpose of this training is to get you up and going using and applying GIS to your research without spending a lot of time learning it. The training is hands-on and project-based, meaning you will learn how to apply GIS to a project from start to finish. More specific to the software, ArcGIS 10.4.1 that will be taught, you will learn how to:

  • Use the ArcCatalog application to store, organize and manage spatial data such as creating geodatabase and metadata.
  • Use the ArcMap application to view spatial data and make maps/visuals; and learn the basics of cartographic design.
  • Use the ArcToolbox application to process/analyze spatial data, such as deriving slope from a bathymetry model (or DEM) and doing data conversion. 
  • Use the Model Builder application to create models and create a project work-flow process.
  • Use the Spatial Analyst Toolbar to do raster analyses, such as calculating the shortest or least cost path from one location to another.

The training is broken up into 2 sections, an ArcCatalog section with 3 exercises and an ArcMap section with 10 exercises. The ArcGIS Desktop 10.4.1 with a Standard or Advanced license plus separate Spatial Analyst and an optional 3D Analyst Extensions are required.  

Case Example: Interisland Ocean Cable Project
The case example being used in this tutorial is based on a project done in 2009 conducted by UH Manoa, School of Ocean and Earth Science and Technology (SOEST). More information for the survey project can be at: http://www.hawaiicleanenergyinitiative.org/storage/interisland_cable.pdf (last accessed 8/24/2017).

Note that the methods used in this tutorial are only for the purpose of teaching GIS/ArcGIS, and are not necessarily reflective of the actual methods used in the real project. 

Case scenario: As part of a renewable energy initiative, the Hawaiian Electric Company (HECO), with assistance from the State of Hawaii (DBEDT) is planning to integrate wind energy into the Oahu electrical system. The wind energy is expected to come from one or more wind farms on Lanai and/or Molokai and transmitted to Oahu via an interisland, undersea cable system.  The proposed plan is to have either 1) two, 200 MW wind farms: one on Lanai and another on Molokai to integrate with Oahu’s electric grid, or 2) a single 400 MW facility either on Lanai or Molokai to integrate with Oahu’s electrical grid system. A later phased cable extension to Maui is also being looked at. 

Your Task: To determine if it’s feasible to have such an undersea power transmission system and if so suggest possible cable transmission routes.

Preview of the Table of Contents 

Note: Appendix B is not included in the tutorial for download here as it only pertains to Hawaii State Employees

TOC.jpg
TOC2.jpg

Open Source GIS Implementation - Part 4 Installing GeoServer

Sorry for the delay in posting part 4. I've been a little busy lately but here it is. Part 4 goes over how to install GeoServer and how to publish a PostGIS database to it. Here I am using GeoServer 2.10.1 Installation was done on both a Windows 10 laptop and Windows Server 2012R machine for testing.

There is more detailed instructions on the GeoServer site which you should read. The instructions were easy to follow, so there's no need to rewrite it so here. I'm just giving a summary of what I did.

JRE 8

  • JRE 8 is required.
  • Download JRE 8 from Oracle
  • I installed both x86 (jre-8u121-windows-i586.exe) and x64 (jre-8u12-windows-x64.exe) versions.

Installing GeoServer 2.10.1

I found the installation process to be fairly easy. The only thing I had to do was make sure that the JRE path was correct. I have both the x64 and x86 JRE 8 versions installed - the default option during the installation process uses the path to the the x86 JRE (e.g. C:\Program Files \(x86\)\Java\jre8\). I left the path as the default since I want to run GeoServer as a service, which requires 32bit JRE (i.e. x86 JRE version\).   

Once the installation completes and if you're able access the Geoserver web admin page from the program menu or by opening a web browser and typing http://localhost:8080/geoserver/web then you're good.   

If you can access this page then it means GeoServer was installed properly

If you can access this page then it means GeoServer was installed properly


Installing GeoServer Extensions

There are many GeoServer Extensions -- you should read about them and decide which one you want to install to extend the functionalities of your GeoServer. These are the ones that I installed on my GeoServer. They seemed to have been installed properly but haven't been fully tested out yet. There are tutorials on the GeoServer site you can explore.

  • Vector Tiles - outputs json, geojson, topojson, mbtile 
  • Importer - batch uploading of data via a GUI
  • YSLD -  styling of layers in YSLD format (easier to do/read then SLD)
  • Printing Module - printing using Mapfish printing service. I installed this put haven't figured it out or worked with it yet. I will post my notes when I get it working.

You can download the extension(s), here. Mark sure the extension(s) you download matches your GeoServer version, in this case it is 2.10.1. The Vector Tiles extension for GeoServer 2.10.1 can be found in the community-latest folder. With GeoServer 2.11 Vector Tiles is part of the core extensions. The instructions from the GeoServer site to install the Vector Tiles extension is for GeoServer 2.11 but it is sill applicable to version 2.10. You can read the instructions here

Basically you just download and extract the jar files for the extension and put them in the directory of your GeoServer’s WEB-INF\Lib folder (e.g. C:\Program Files\(x86)\GeoServer 2.10.1\webapps\geoserver\WEB-INF\lib\).

After you copy and paste the jar files into the correct directory, the instruction guide (above links) says to restart GeoServer. That didn't work for me on my Windows 10 laptop. I had to restart my computer then restart GeoServer before I could see the added extension. The same thing happened on a Windows 2012R Server - I had to restart the server, then restart GeoServer and wait for a few minutes (up to 5 minutes) then refresh the GeoServer Web Admin page to see the extensions showed up.

Verifying Extensions Installed properly

Vector Tile Extension

If you log into the Geoserver through the web admin page and click on a layer and then click the Tile Caching Tab you should see the vector tile output options that you just added to GeoServer.

Output options available after installing the Vector Tiles extension

Output options available after installing the Vector Tiles extension

Importer Extension

If the Importer extension installed properly then you should see a new tool, called Import Data under Data Section in the left hand side of the GeoServer web admin page. 

YSLD Extension

If the YSLD extension installed properly then you should be able to see the available format under Data >> Styles >> New Style. With this extension, you can style layers in YSLD and SLD formats. As you can see from the example below, I've also added the CSS extension for styling layers in CSS format. See link here for more information on styling in GeoServer. There are even styling cookbooks available to make it easier.


Importing Data to GeoServer

This section goes over how to use the Import Data tool to batch import a PostGIS database. Using the Import Data tool is a much easier option than having to use cURL command line option to do batching data importing to GeoServer. 

Create a new Worskspace and Data Store

Open the GeoServer web admin page - login if needed. Then under the Data section, click Workspaces >> Add new workspace

In the new workspace window that comes up, type in a name for your new workspace - limited to 10 characters and no spaces (e.g. CCH). For Namespace URI (Uniform Resource Identifier) this can be anything - according to the GeoServer site, a URI is similar to a URL except it does not need to point to an actual location on the web. It just needs to be a unique identifier, such as http://g70.design/geoserver/CCH.

Type in a Name (limit to 10 characters & no spaces) and a URI

Type in a Name (limit to 10 characters & no spaces) and a URI

Batch Import Data

When you are done creating a new workspace, it's time to import some data into it. For this example, I'm batch importing some layers from a PostGIS database that I created earlier in part 3. 

In GeoServer, click Import Data. Then in the right hand panel, 1) Choose PostGIS as the data source to import from; 2) Under Configure the data source: fill in the appropriate information to connect to your PostGIS database; 3) Specify the Target for the Import: fill in the appropriate info for your workspace and data store - you can use an existing data store or create a new. If you create a new one - the store name will default to the name of your PostGIS table (e.g. workspace = CCH, store = CCHGIS (which is the name of my PostGIS database).

Substitute your own information in the fields above. For Store you can create a new one (default name store name will be the name of your database) or use an existing one. Click Next button

Substitute your own information in the fields above. For Store you can create a new one (default name store name will be the name of your database) or use an existing one. Click Next button

In the Import panel that comes up you should be able to see the PostGIS database you just connected to. Select the layer(s) that you want to import into the data Store. I think of the workspace as the directory and the store as a container for the data.  

Note: When batch importing I was only able to batch import layers that were on the same page. If I selected layers that were listed on page 1 then clicked to page 2 to select more layers then GeoServer only imported the layers I selected from the current page that I was on. This meant that I had to select all the layers I wanted to imported on page 1 then click Import and wait for it to successfully import, then go to  page 2 and repeat the process, etc. 

  Select the layers you want to import >> Click Import then want for the layers to be successfully imported before importing more layers on page 2, 3, etc. See Note above.

 

Select the layers you want to import >> Click Import then want for the layers to be successfully imported before importing more layers on page 2, 3, etc. See Note above.

If you go to the Stores panel, you will see the newly created Workspace (e.g CCH) and the Store (CCHGIS, which is also the name of my PostGIS database).

If you click on Layers in the left hand panel, you will see the layers that were imported. Notice under Name, it is the workspace name followed by the layer name (e.g. CCH:park) and the Store is CCHGIS.

You can also preview the layers by Clicking Layer Preview in the left hand panel then click OpenLayers (under Common Formats heading). Notice that the layer is given a default feature color. 

Below is an example layer preview of the park layer with polygon features shown in gray  (default color).


Styling Layers

Styling layers in GeoServer can be done using SLD, YSLD, or CSS. See the GeoServer Styling Workshop for more info. Supposedly you can use QGIS to create SLDs – this would be an easy option, however, when I tried it using QGIS 2.18.3 it didn't work. Out of curiosity I also tried exporting an SLD using an extension in ArcGIS, called Arc2Earth -  this didn't work either.  So for now I guess the best way to create and edit styling in GeoServer is directly edit XML or use the GeoServer style editor.

Click on Styles in the left hand panel then in the right hand panel you can either Add a new style or click on an existing style to edit it. Notice there are already default styles for the layers I batched imported (e.g. CCH_citynopass, CCH_coastline, etc). You can either delete then and create new ones or edit them - whatever you prefer.

Below is a snapshot of the New Style window. Choose a format then either upload an existing style file or use the Style Editor to write your own styles. I highly recommend reading the GeoServer Styling Section. It has lots of useful info, including "cookbook recipes" you can copy and edit for your purpose.

Here is a sample of a style in YSLD format I created for a zoning layer. Notice under Format, you can't change the format from one to another (e.g. can't switch format from YSLD to SLD). Also notice there are 3 tabs: Publishing, Layer Preview, and Layer Attributes -- these are shown below.

With the  Publishing tab you can select which layer the style is associated with and whether it is the default style for that layer. Here I am associating my CCHzoning style to the zoning layer and making it the default symbology.

The Layer Preview tab gives you a preview of your style. Any changes/edits made in the Style Editor will be reflected in the preview. Notice where it says "Previewing on layer: CCH:zoning_luo" - make sure this corresponds correctly to your style. I've had it where maybe I forgot to check of something and I'm not previewing on the correct layer (i.e. Previewing on layer: CCH_TMK but my style is for zoning so nothing shows up in preview). If this happens, you can click on the layer name and then you can select the correct layer. 

  Make style changes/edits using the Style Editor to see a preview of what changes will look like. You may have to click the Apply button to see the changes. Use the Validate button to help you "debug" your style structure. Use the Submit button to finalize/save the changes.

 

Make style changes/edits using the Style Editor to see a preview of what changes will look like. You may have to click the Apply button to see the changes. Use the Validate button to help you "debug" your style structure. Use the Submit button to finalize/save the changes.

The Layer Attributes tab will allow you to see the attribute field names, and type associate with the layer.  

It takes a bit of work to do the styling but it's not that hard to do. It's just a hassle and not as easy as just being able to use SLD created in QGIS. I'm hoping the SLD from the next version of QGIS will work with GeoServer. 


Publishing Layers

Once you've create styles for your layer then the next thing you'd want to do is publish them as WMS, WFS, etc. Click on Layers in the left hand panel then select a layer to edit its properties. There are 4 tabs: Data, Publishing, Dimension, and Tile Caching. I would suggest reading up on all the options within these tabs to properly configure and publish your layers. Don't forget to click the SAVE at the bottom of each tab after you make your changes. 

The Data tab, shown below is basically the metadata for your layer. Fill in any input that is needed, make sure the layer is enabled though.

The Publishing tab allows you to edit the WFS and WMS settings for the layer. Notice the default style for WMS is the zoning style I created earlier.

The Dimension tab allows you to enable a time and elevation dimension if your layer has these properties.

The Tile Caching tab is where you can create caches for your layer. and select the tile image format. Notice you will not have these formats: geojson, topojson, utfgrid, or mapbox-vector if you didn't install their respective extensions. 

Once your layers are published you should test it out to see if your GeoServer services are working by trying to call them in a web mapping application or in a desktop GIS software (e.g. ArcGIS, QGIS). Below are some examples of my GeoServer services as shown in a Leaflet and Open Layers web maps, and  in ArcGIS and QGIS.

Leaflet map showing layers from my GeoServer

Leaflet map showing layers from my GeoServer

Open Layers web map showing the park layer from my GeoServer

Open Layers web map showing the park layer from my GeoServer

My GeoServer added in QGIS with zoning layer displayed

My GeoServer added in QGIS with zoning layer displayed

My GeoServer added in ArcCatlog with a preview of the zoning layer.

My GeoServer added in ArcCatlog with a preview of the zoning layer.

 

 

Open Source GIS Implementation - Part 3 Creating PostGIS Database

Continuing on my open source GIS server experiment, part 3 goes over how to create a PostGIS database and batch import to PostGIS from an ESRI file gdb.

Most of my data are in geodatabases so this is what I will be using for this portion of the experiment. I downloaded some shapefiles from the City and County of Honolulu (CCH) HoLIS website and imported them into a file geodatabase, specifying the Web Mercator projection (EPSG: 3857).

Connecting to PostgreSQL and create a new PostGIS database 

  • Open pgAdmin 3 or 4 (I am using pgAdmin 3 here) - if you haven't previously provided login creditials  (e.g. postgres/postgres) then enter them and save it for next time 
  • Double click on your Server (e.g. PostgreSQL 9.6 localhost: 5432) to connect to it.
  • Right click on Databases >> New Database.  
Properties tab: type in a name for your database (e.g CCHGIS) and the owner (e.g. postgres)  

Properties tab: type in a name for your database (e.g CCHGIS) and the owner (e.g. postgres)  

Definition tab: for template, select a sample template to use if you have one (e.g. postgis_23_sample). Note if you are using a sample database as the template to create a new database, make sure it is not in use or connected otherwise you will get an error.   

Definition tab: for template, select a sample template to use if you have one (e.g. postgis_23_sample). Note if you are using a sample database as the template to create a new database, make sure it is not in use or connected otherwise you will get an error.   

Here is the newly created CCHGIS database based on the postgis_23_sample which already has the extensions added that I want. 


Loading File GDB to PostGIS Database Using Ogr2ogr 

Required:

  • GDAL with ESRI file gdb support must be installed. See Part 1 of this series for installing QGIS with OSgeo4W that you can use to install GDAL with fgdb support - download pdf.
  • Have your own file geodatabase and make sure you have write access to it.. Optionally to make things easier when you build your web mapping app, make sure the layers in your geodatabase are in either Web Mercator projection (EPSG: 3857) or WGS84 (EPSG: 4326). 

Optional

  • Add the path to OSGeo4W/bin (ex: “C:\OSGeo4W\bin”) to the Path Environment Variables (Right click Computer > Properties > Advanced system settings > Environment Variables > System Variables > Path). This keeps you from having to type the full path of the application so you can type ogr2ogr instead of having to type C:\OSGeo4W\bin\ogr2ogr.exe

Using the Ogr2ogr command:

Type the following command in cmd terminal (Start >> run cmd)

ogr2ogr -f "PostgreSQL" “PG:host=[host] port=[port] dbname=[dbName] user=[user]" “[PathtoFileGDB]” -overwrite -skipfailures –progress --config PG_USE_COPY YES

Example:

ogr2ogr –f “PostgreSQL” “PG:host=localhost port=5432 dbname=CCHGIS user=postgres” “C:\Temp\CCH_HoLIS_PostGIS.gdb” –overwrite –skipfailures -progress –-ogr2ogrconfig PG_USE_COPY YES
Example of command as shown in the cmd terminal

Example of command as shown in the cmd terminal

Note: by default, postgresql deny all connection if it's not from "localhost". If you get an error that says PGConnectdb failed; no pg_hba.conf entry for host “x.x.x.xx” , user “postgres”, database “CCHGIS”, SSL off – you need to change the host = localhost instead of host = 1.2.3.45.6 or you need to modify the pg_hba.conf file. 

Try this: Find pg_hba.conf file and open it in a text editor (e.g. C:\Program Files\PostgreSQL\9.6\data\pg_hba.conf)
Add the following line as the first line of pg_hba.conf. This should allow access to all databases for all users with an encrypted password:

# TYPE DATABASE USER CIDR-ADDRESSMETHOD
hostallall 0.0.0.0/0 md5

Here is a snapshot showing all the layers from my CCHGIS geodatabase imported into my PostGIS database (e.g CCHGIS that I created earlier)

File geodatabase layers are imported into PostGIS CCHGIS database under Tables.

File geodatabase layers are imported into PostGIS CCHGIS database under Tables.

The next part of this series, Part 4 will be on Installing GeoServer and Importing PostGIS to GeoServer. I will try to post it soon - maybe the first week of April 3rd, 2017.

Open Source GIS Implementation - Part 2 PostgreSQL/PostGIS Installation

This post is part 2 of my Open Source GIS Implementation Experiment. Part 1 was installing QGIS, an open source desktop GIS - download pdf.

Thoughts on the installation process

I tried both installers from both EnterpriseDB and BigSQL to install PostgreSQL 9.6. Both installers include the option to install PostGIS 2.3.1 bundle. Although the EnterpriseDB included the current pgAdmin 4 and the BigSQL installer package I used included pgAdmin 3 LTR. The installation process was fairly easy. After testing and playing around I decided to use the EnterpriseDB installer to install PostgreSQL 9.6 with PostGIS bundle 2.3.1. I then did a separate installation of pgAdmin 3.

Thoughts on pgAdmin 4

On the laptop pgAdmin did not work the first time I tried it. Just said "loading" for a long time before I manually ended the process. The second time worked but the display resolution was horrible on my laptop's very high resolution (3200x1800) screen. The pgAdmin 4 GUI was tiny; text and graphics were so small I couldn't see anything. My laptop OS is Windows 10 so I'm not sure if it's just and issue particular to Windows 10 or not. I tried changing the program's properties to "Disable display scaling on high DPI settings "as well as "Run this program in compatibility mode for" for Windows 7 and 8; these options didn't seem to make a difference.

PgAdmin 4 on the Windows Server had no display resolution problem. However, it was very slow to start up. Slow start up was an issue on the laptop as well. I started application a few times and everytime it took anywhere from 1-5 minutes or it just hangs. At this point, I'd rather stick with pgAdmin 3 for now.

Installation Notes

  • PostgreSQL must be installed first before PostGIS. Versions for both should match - e.g. if you are using the x64 PostgreSQL make sure you also use the x64 PostGIS installer too
  • A caveat about BigSQL installer - it will install pgAdmin 3 but it does not include an option to install a sample PostGIS database, which I find useful for doing a quick test since it already has most of the extensions enabled. It also doesn't include shp2pgsql-gui to import shapefiles to PostGIS - you have to do this manually - see section on Installing pgAdmin 3 below.
  • The EnterpriseDB installer does include shp2pgsql-gui but it is not added to pgAdmin 4 as a plugin like with pgAdmin 3. It is a separate application under the PostGIS Bundle 2.3 in the startup menu. If you don't see it in your startup menu, it should be where you install PostgreSQL (e.g. C:\Program Files\PostgreSQL\9.6\bin\postgisgui).
  • See PostgreSQL website for more information

Install PostgreSQL/PostGIS using EnterpriseDB Installer

Download the PostgreSQL 9.6.1 from EnterpriseDB site. I am using the x64 installer. After downloading PostgreSQL, start the installation and follow prompts:    

Specify installation folder or use the default  

Specify installation folder or use the default  

Specify data folder or use default  

Specify data folder or use default  

 Enter a password for superuser postgres. You can use the default user/password of postgres/postgres for now and change it later.

 Enter a password for superuser postgres. You can use the default user/password of postgres/postgres for now and change it later.

Use the default port 5432 or enter one that is not being used. If you don't know then leave as default

Use the default port 5432 or enter one that is not being used. If you don't know then leave as default

Advanced Options: Choose default locale

Advanced Options: Choose default locale

After completion it will ask if you want to launch Stack Builder – check the box and click Finish. This will lauch Stack Builder

After completion it will ask if you want to launch Stack Builder – check the box and click Finish. This will lauch Stack Builder


Stack Builder: Download and install PostGIS 2.3 bundle 

Wait for the Stack Builder to Launch. Once it does follow the prompts to install PostGIS.

After the Stack Builder launches, select PostgreSQL 9.6 (x64) on port 5432

After the Stack Builder launches, select PostgreSQL 9.6 (x64) on port 5432

Expand Spatial Extensions and check the appropriate PostGIS bundle to download and install

Expand Spatial Extensions and check the appropriate PostGIS bundle to download and install

After downloading finishes, start the installation process. Note: you may want to check Skip Installation if the profile you are using to do the installation do not have administrator rights, in which case you skip installation and go to your PostGIS download directory (e.g. C:\Users\Stephanie) and run the installation as an administrator.  

Check the box to Skip Installation if you don't have administrative rights to do installations on your computer. Leave uncheck to initial installation process immediately

Check the box to Skip Installation if you don't have administrative rights to do installations on your computer. Leave uncheck to initial installation process immediately

Choose Components: Select PostGIS and Create Spatial database (if you want a sample db to be created for testing or to use as a template)

Choose Components: Select PostGIS and Create Spatial database (if you want a sample db to be created for testing or to use as a template)

Chose a destination folder or use the default

Chose a destination folder or use the default

Database Connection:   Username: postgres   Password: postgres (or whatever you entered above)   Port: 5432 (or whatever you entered above)  

Database Connection:
  Username: postgres
  Password: postgres (or whatever you entered above)
  Port: 5432 (or whatever you entered above)  

If you chose to create a sample database then enter a name: use default or enter new name then click Install

If you chose to create a sample database then enter a name: use default or enter new name then click Install

Register GDAL_DATA environment – say yes. This will automatically add GDAL_DATA environment variable putting this path in for you. If you use GDAL already chances are you already have this environment variable set and you may not want to overwrite it.  

Register GDAL_DATA environment – say yes. This will automatically add GDAL_DATA environment variable putting this path in for you. If you use GDAL already chances are you already have this environment variable set and you may not want to overwrite it.  

Enable Raster Drivers (default option is off) – say yes  

Enable Raster Drivers (default option is off) – say yes  

Enable Out of Database Raster – say yes  

Enable Out of Database Raster – say yes  

Close the window when PostGIS installation complete. Also close the Stack Builder window when Finish.

Successful installation of PostgreSQL and PostGIS bundle. The EnterpriseDB installer installs pgAdmin 4 as part of PostgreSQL 9.6 menu - there is no pgAdmin 3. Notice PostGIS 2.0 Shapefile to Database is a separate application under PostGIS bundle menu. 

2017-01-30_112851.jpg

Here is a screenshot of pgAdmin 4 on Windows Server

Here is a screenshot of pgAdmin 4 on Windows 10 Laptop. I have a high resolution monitor. Everything on the GUI is really tiny making it impossible to read or even use. Changing the screen resolution or the option to use the application in compatibility mode didn't make a difference.


Installing pgAdmin 3

The screenshots above showed the pgAdmin 4 GUI. However, as I noted earlier not only was it slow to start - basically start up was inconsistent. Sometime it would start up right away, sometimes it took a few minutes, and sometimes it would just hang. It was true for me on both the Windows 10 laptop and Windows Server. You may have a different experience with pgAdmin 4 but for now I will install and use pgAdmin 3.

Download pgAdmin 3. I used version 1.22.2. It looks like it is available only as x32 bit. I did not see a version for x64 bit. Unzip the file and start installation.

Custom Setup: use the default options or make changes as needed

Custom Setup: use the default options or make changes as needed

After sucessful installation, check to see if pgAdmin III is in your start up menu

After sucessful installation, check to see if pgAdmin III is in your start up menu

Below is a screenshot of pgAdmin 3 on Windows Server. Note: I ended up with 2 "servers". PostGIS and PostgreSQL 9.6. Connect to the Server. I deleted the one that said PostGIS since it doesn't contain the sample db I wanted. 

After connecting to the Server, I can see the sample database (e.g postgis_23_sample) created during the installation process earlier.


Manually add/enable shp2pgsql-gui plugin

This part is optional. You should only do this portion if you want to be able to access the shapefile loader while in pgAdmin III -  But if you want to access the application as a plug-in when you are using pgAdmin, then this is how you would manually add/enable it for pgAdmin 3. If you have installed PostgreSQL with the PostGIS bundle then you should already have the shp2pgsql-gui application installed. It is a separate application under the PostGIS bundle program menu, called PostGIS Shapefile and DBF Loader -  you can access it in your the startup menu.

  • Open up pgAdmin 3 and check the Plugins menu, noticed there is only the PSQL Console and no PostGIS Shapefile Loader.  

 

  • Check to see if you have the postgisgui folder in your PostgreSQL's bin directory (e.g. C:\Program Files\PostgreSQL\9.6\bin\). If you used EnterpriseDB installer to install PostgreSQL with the PostGIS bundle then you should have a postgisgui folder on your computer.

Note: if you don't have this postgisgui folder then you can download the PostGIS bundle, unzipped it, and copy the postgisgui folder to the bin directory of your PostgreSQL installation. Make sure you download the correct version of the zipped file.

  • Find the plugins.d folder in your pgAdmin 3 installation folder (e.g C:\Program Files \(x86\)\pgAdmin III\1.22\plugins.d\).
  •  In this folder there is a plugins.ini file. Change the permission on it so you can edit it
  • Right click on file >> Properites >> Security tab >> Users >> Edit >> check box for Full Control or Modify and Write option.
Change the permission on the plugins.ini file so you can edit it

Change the permission on the plugins.ini file so you can edit it

  • After changing the file permission, open plugins.ini in a text editor such as Notepad++ or Notepad and enter the following text near the end of file then save it.

Note: The text below comes from the postgis.shp2pgsgql-gui.ini file that is part of the downloaded PostGIS zipped bundle (e.g. C:\Downloads\postgis-bundle-pg96-2.3.1x64\pgAdmin III\plugins.d\postgis.shp2pgsql-gui.ini)

;
; PostGIS shp2pgsql-gui (Windows):
;
Title=PostGIS Shapefile and DBF loader 2.2
Command="$$PGBINDIR\postgisgui\shp2pgsql-gui.exe" -h "$$HOSTNAME" -p $$PORT -U "$$USERNAME" -d "$$DATABASE" -W "$$PASSWORD"
Description=Open a PostGIS ESRI Shapefile or Plain dbf loader console to the current database.
KeyFile=$$PGBINDIR\postgisgui\shp2pgsql-gui.exe
Platform=windows
ServerType=postgresql
Database=Yes
SetPassword=Yes

Here is an example shown in Notepad++. The highlighted portion is the inserted code.  

  • Open pgAdmin 3 and go to File menu >> Options.
  • In the Options window, under Browser, click Binary Paths. Where it says PG Bin Path - browse to or input the path to your PostgreSQL bin directory (e.g. C:\Program Files\PostgreSQL\9.6\bin).
  • Click OK when done.

After you do the above, the PostGIS Shapefile and DBF Loader (i.e. shp2pgsql-gui application) should be in the Plugins menu. If it's not there then try closing pgAdmin 3 and reopening it.  

Here is what the PostGIS Shapefile and DBF Loader application looks like.  

The shp2pgsql loader gui

The shp2pgsql loader gui

That is all for part 2 of the open source implementation process. Part 3 will cover how to install GeoServer and how to import data from PostGIS.

Open Source GIS Implementation - an Experiment

This post documents my attempt at implementing opensource GIS Desktop and Server. It describes the trials and errors of the process. It may or may not work for your purpose, but I hope others will find it useful in terms of knowing what is involved.

This is the open source GIS software stack I used.

The implementaion process is broken up into serveral parts:

Part 1: Installing QGIS. This is just the installation of the opensource desktop GIS, QGIS, which I've already written a document on how to do this.  Although it is for QGIS 2.14 the process will be the same for the current version 2.18.

Part 2: PostgreSQL/PostGIS Installation.

Part 3: Creating a PostGIS database. This part goes over how to import an ESRI file GDB to a PostGIS database using ogr2ogr command line utility.

Part 4: Installing GeoServer.

Part 5: Publishing PostGIS to GeoServer. This part goes over how to publish or push PostGIS database to GeoServer

Part 6:  Using Leaflet and OpenLayers to test that layers GeoServer can be called. 

I tried OpenGeoSuite 4.8 (free version from Boundless but require email sign up) which was really easy to install and used. The version of OpenGeoSuite I tested uses slightly older versions of GeoServer and PostgreSQL/PostGIS. I wanted to use more current versions of the software in the stack as well as wanting to know how to each component worked, so I tried my hand at implemnting each components of the stack myself. Of course it's not as easy as just  using OpenGeoSuite but at least I learned a lot in the process.

Here are the specs on the machines that I tested.

Tested on Windows OS Machines
Installation of Postgres/PostGIS on both a Windows 10 OS x64 laptop and a Windows Server 2012 R2 virtual machine for testing. The Windows Server VM already has a web server (Windows IIS) already set up at my work by IT so I used that. Also for my laptop I just used/enabled the Windows IIS already available. 

Laptop Specs: Dell XPS 13

Windows Server 2012R VM

2017-01-19_091615.jpg


 

HIGICC GIS Expo - March 24th, 2017

For those who missed the HIGICC GIS Expo held at the Hawaii Convention Center on March 24th, 2017 it was a great turn out. There were around 120 participants and 19 presenters from all sectors and various industries. 

Presentations from all the presenters for the one day conference will be posted at HIGICC site - please check their site. Below is a short presentation I gave on open source GIS implementation at G70 if anyone is interested. I will be posting more details on the implementation process soon for those who want to know more about how to install the software stack used. 

Open Source GIS User Group - Intro to QGIS

The 3rd Open Source GIS User Group meeting was a presentation on QGIS. The following is an outline of what was presented.  A corresponding short guide is provided for you to use to explore QGIS: Download IntroQGIS pdf.

Quick Introduction to QGIS 2.14 LTR

Presented by: Stephanie Saephan

1. What is QGIS?

  • Free, cross platform, opensource GIS software
  • One of the most widely used OS GIS – very popular
  • Started in 2002 as more of a data viewer and has since grown into a full blown GIS application
  • On-going development with bug fixes and new features added on a regular basis. 

2. Installing QGIS

  • Stand-alone installer (current 2.16 and LTR 2.14) – recommended for beginners
  • OSGEO4W windows installer (32/64bit, bundles other opensource packages) – recommended for advanced users.
  • Can have multiple versions on same machine, but can only use one version at a time

3. QGIS Desktop – a demonstration of the software to include the following:

  • Components of the user interface
  • Installing QGIS plugins
  • QuickMapService (basemaps)
  • Configuring the user interface
  • Moving toolbars and panels
  • Hide/show toolbars and panels
  • General global settings
  • Coordinate Reference System (CRS) and Map Scale
    • EPSG: European Petroleum Survey Group (open standard codes for coordinate systems, datums, etc. See spatialreference.org, prj2epsg.org)
    • Default global CRS in QGIS (EPSG: 4326 or WGS84)
    • Map Scale: ratio, measurement unit defaults (project CRS)
  • Adding and viewing spatial data
    • Shapefiles (zipped and unzipped)
    • Geodatabases (file gdb – read only for gdb created with AG10.x)
    • WMS, WFS
  • Symbology options
  • Labeling options
  • Print Composer – making a map
    • New composer
    • Guide lines
    • Linking map project with print composer
    • Adding map inset/overview map
    • Scale bar
    • North arrow (add as image)
    • Legend
    • Export map
  • QGIS Browser
  • Questions? Comments?

I am planning the next open source GIS user group meeting, for sometime in September. If you know open source GIS and would like to do a presentation, demonstration, or workshop please let me know. 

Thanks.

 

 

Open Source GIS User Group Meetup - Notes & Summary

The following is a summary of the Open Source GIS User Group Meetup events that took place on June 10th and June 13th, 2016. I thought the meetups went well. There were lots of discussion on some interesting OS GIS topics. As this user group is just starting up, I feel that a monthly meeting with revolving topics will allow for more people in the community to join and participant, especially if they didn't get to attend these first 2 events. So, for now I'm thinking maybe every 3rd Friday of the month, which means the next meetup is to be held on July 15th, 2016. Time and location is to be announced on the site (1-2 weeks prior). Save the date!

A big thank you to those who participated and volunteered to help out. 

Read on for the summary or download pdf version: OSGISUG_Meeting_Summary_610132016.pdf

Open Source GIS User Group Meeting Summary

I.    Introduction/Purpose of Meetups

Two meetups were held: one at Local Joe downtown Honolulu on June 10th, 2016, and another one at UH Manoa campus on June 13th, 2016, each with 10 attendees. The purpose of these 2 first meetups were getting a sense of people’s interests in open source GIS and what those interests are. It was also to find out if there is enough people to even start a user group. General feedbacks and comments seem to suggest that people are interested in learning more about open source GIS, and having such a user group to share and learn from each other.

Stephanie, host of the meetups talked about why she’s organizing this user group and what the purpose of this group is.

Here's my explanation: I just feel that open source GIS (OS GIS) has come a long way and has really matured within the last 5 years that now feels like a good time to start a user group in Hawaii. I first started looking into OS GIS as a way to create a fuller set of tools to make my job more efficient, but it was really frustrating not being able to talk to anyone locally about OS GIS. The online forums, blogs, etc. that I’m looked into weren’t much help. I know there are others who also use OS GIS, but I didn’t know who they were or how they use OS GIS. This user group is a chance to connect and share with others. 


II.    Purpose of OS GIS User Group

The purpose of this user group is to make learning GIS accessible to everyone via an open source platform, and to foster the sharing of knowledge within the local GIS community. 

Following up on the purpose of the user group was a discussion on existing OS GIS activities, some which included the following:

  • UHM SOEST Environmental Science Program has taught courses using Matlab, GMT, QGIS, 
  • Matt Barbee (SOEST) currently teaching Summer introductory GIS course (although primary using ArcGIS).
  • Some attendees talked about their own experience with OS GIS. Experiences ran the gamut with some having used OS GIS such as GRASS, QGIS, GDAL quite a bit, others have tinkered a little bit with OS GIS here and there and some have no GIS experience at all but would like to learn. 

III.    Goals of OS GIS User Group

Group discussed some goals that are related to the purpose of the user group:

  • Create a consortium/network to connect people, labs, organizations/agencies for the purpose of sharing data and knowledge.  
    • Labs, agencies, organizations with Points of Contact (POC)
      • UH Manoa Labs: SOEST (JP, MB) MAGIS/Hamilton Map Library (TK)
      • Agencies: DLNR/CRWM (JK), NOAA (RW, GC), USGS (LM)
      • Private organizations: Group 70 (SS), HHF(JH), HCRI/RMH (MC)
    • Other possible labs, agencies, organizations to reach out to: UHM DURP, UHM Geography Dept, UHM Botany/EECB, NREM, TPSS, PDC, CCH, State GIS Program, Invasive Species Committees (OICS, BISC, MISC, etc), other companies from private sector.
  • Develop curriculum for learning OS GIS
    • Attendees discuss importance of having curriculum that uses case studies/research/projects and data to showcase how GIS is used to help solve issues relevant to Hawaii.
    • Draft outline OS GIS curriculum. Input from attendees include:
      • Intro to GIS/QGIS
      • Spatial Analyses: tutorials using case studies/research and data that pertain to Hawaii. Also mention of 3D analyse
      • Cartography/Data Visualization
      • Open Data and Standards
      • Web Application Development
      • Remote Sensing/Lidar
  • Provide online tutorials for learning on your own  
    • This is related to the curriculum development. Tutorials shouldn’t just be about how to use the software, but also focus on how GIS principals are applied to a real world issues or context that people can understand.
  • Offer instructor led hands-on training/workshops: In lab setting? BYOL (Bring Your Own Laptop) setting? 
    • Potential collaboration with DURP lab, UH MAGIS/Hamilton Library, and SOEST lab for hands-on trainings.
    • BYOL setting more preferable (?) as it allows for participants to do training/tutorials on their own machine, and at their convenience after the trainings/workshops. Can use OSGeo-Live, which provides a bootable ISO image on USB stick or DVD that allows users to try a variety of OS GIS software without directly needing to do installation on the hardrive – this will for easier for training/workshops as everyone will be using the same setup regardless of machine and OS. Possible venue for BYOL setting: Kakaako Agora, ProtoHUB, other space? More preferable to find place that doesn’t charge a fee. 

IV.    GIS Community Interests

 Some community interests that were discussed:

  • Introduction to OS GIS such as QGIS
  • Linking students with organizations/companies who are looking for GIS interns
  • Crowd sourcing of projects of local interests benefitting the community
  • Possible GIS “Firm” Crawl where a lab, organization or agency host an onsite show and tell of their GIS projects. 
  • Open data, standards, and accessibility
  • Mention of existing open courseware at Penn State (https://www.e-education.psu.edu/geog585/). Perhaps the user group could use their curriculum as a start.

Out of curiosity, Steph asked attendees, “What does open source GIS mean to you?” Answers included:

  • Software: QGIS, OGC, Free
  • Not Arc (meaning not ArcGIS)
  • Open Data and Standards
  • To grow base users
  • Share and support partners who don’t have access to ArcGIS
  • Accessibility and Affordability
  • Cross platform

V.    Next Steps

Attendees/Group came up with 3 tasks to help move the user group initiatives forward. All of these tasks need volunteers/contributors.

1. GIS Software (Introduction to GIS software – QGIS, GRASS, GDAL)

  •  Stephanie is working on a quick intro to QGIS tutorial. To be posted soon to opengislab.com
  • Need volunteers who have experience with GRASS, GDAL, PostGIS, etc.

2. Open Data, Standards, and Accessibility

  • Matt Barbee/Jim Potemra (SOEST) volunteered to take the lead on this topic

3. Applied GIS (applicattions/tutorials): Remote sensing, Lidar, others.

  • Lidar/DEM Applications: Ross Winans (NOAA) volunteered to take the lead
  • Remote Sensing/Imagery/Classification: Miguel Castrence (HCGI/RMP) volunteered to take the lead
  • Other areas of GIS Applications needed: any suggestions appreciated

Future Tasks on more advanced topics such as Coding & Web Applications.

VI.    Future Meetings

  • Venue for future meetings
    • Group 70 or other downtown location?
    • UH Manoa location?
    • Other locations? Possible State sponsored space if non-profit use (e.g. DLNR)
    • Suggestion of doing webcasts on HIGICC site.
  •  Frequency and meeting times
    • Don’t want to have too many meetings that people will get tired attending, also don’t want to be infrequent that people lose interests or forget what was said or done.
    • Monthly meeting with revolving focus, such as Open Data/Standards/Accessibility, RS, QGIS, etc. Key contributors/volunteers could meet as needed to work on specific tasks. 
    • Meeting times could be mornings before work, evenings after work, or possible weekend so as not to cut into people’s work schedule. 
  • Next Meeting
    • If holding monthly meetings – every 3rd Friday of the month?
    • Tentative date of Friday, July 15th, 2016. Time and location TBA. See website opengislab.com for update, one week prior to date of meeting.
    • Tentative topic: Quick look/Intro to QGIS; plan hands-on workshop for Fall (Sept?)

VII.    Website

I (Stephanie) started the website solely as a personal site, but depending on how the user group develops, Stephanie is looking at changing the website opengislab.com to opengislab.org or create a subdomain just for the user group. Either way, searching for typing opengislab.com, *.org  or *.net domain names will still go to the same site. More information to come.

USGS TNM Style Map Template - ArcGIS

While I'm still on the topic of USGS topo maps, I wanted to say how excited I am about the USGS Topo TNM Style Template. You can use this template to re-create a GIS ready USGS topographic styled maps. The Topo TNM Style Template is currently only available as an ArcGIS 10.x map document (mxd).

More information on the TNM Style Template can be found at http://viewer.nationalmap.gov/tools/topotemplate. You can download a zipped demo file, which includes a document on how to use the map template and tailor it to your area of interest. 

The template is recommended for used with Topo Map Vector Data products, which are still under production. The last I checked (5/21/2016) for available data for Hawaii, only staged data for the Ewa quad is available to use with the TNM Style Template. However, I was able to download other publicly available data and use them in the template. It's not perfect but it still looks very much like the current USGS topo maps in GIS ready format. I've included some sample images of what the map template I've tailored for Hawaii looks like.

USGS staged TNM data available for Hawaii, last checked on 5/21/2016 - only data for Ewa quad is available.A

USGS staged TNM data available for Hawaii, last checked on 5/21/2016 - only data for Ewa quad is available.A

Figure 1: TNM mxd displayed in ArcGIS layout view. Notice the similarity to a printed USGS topo map.

Figure 1: TNM mxd displayed in ArcGIS layout view. Notice the similarity to a printed USGS topo map.

Figure 2: TNM mxd as displayed in ArcGIS data view. This is similar to a printed topo map.

Figure 2: TNM mxd as displayed in ArcGIS data view. This is similar to a printed topo map.

How to convert geopdf to geotiff using GDAL

This tutorial shows you how to convert USGS geoPDF topo maps to geoTiffs using GDAL. It assumes that you have: 1) downloaded some USGS geopdfs, 2) a pdf reader such as Adobe Acrobat, and 3) GDAL installed on your computer. Below are the steps I used in converting USGS topo maps from geopdfs to geotiffs with GDAL. I am accessing GDAL using OSGeo4W Shell utility.  The OSGeo4W installer bundles a set of open source geospatial software (e.g. QGIS, GDAL/OGR, GRASS, etc.) for the Windows OS environment. For more information, see website at https://trac.osgeo.org/osgeo4w/.

The USGS has a good, detailed document on how to convert GeoPDF to Geottiffs using GDAL and other software. It can be found here at http://nationalmap.gov/ustopo/documents/ustopo2gtif_current.pdf.

The steps below is the workflow I used to convert a USGS geopdf topo map to geotiff. For a more detailed, step-by-step instruction see the pdf tutorial, How to Convert GeoPDF to GeoTIFF Using GDAL. The method shown here will not create a collarless geotiff image. It will convert a geodpf to geotiff without the collar information but will not actually clip the collar off. 

Step 1: Open OSGeo4W Shell and PDF Layer Names

  • Open the OSGeo4W Shell. It should look something like the image below.
Figure 1: OSGeo4W Shell

Figure 1: OSGeo4W Shell

  • Navigate to your drive and directory where your GeoPDFs are stored, by using the cd or change directory command and inputting your own parameters where denoted by < >.

cd <your drive letter>:\<path to directory>

  • To get a list of files in the directory, type in dir 
  • To get information about your pdf, including coordinate system and pdf layer names, type in gdalinfo <GeoPDF filename> -mdd layers

Step 2: Convert a single geoPDF to geoTIFF

To convert a single geopdf to geotiff, use the gdal_translate command and input your own parameters where denoted by < >. You can use gdal_translate with or without optional parameters. 

Option 1: gdal_translate <geopdf filename> <output geotiff filename> -of GTiff

Example: gdal_translate HI_Honolulu_20130410_TM_geo.pdf HonoluluTopo.tif -of GTiff 

Note: This option will convert all pdf layers to a geotiff at 96dpi. It can take a long time to process and the resulting tiff can be really large especially when including the orthoimagery and shaded terrain. I did a test on the Honolulu quad and it took maybe 3-4 hours with the resulting tiff being ~955MB. 

Option 2: Gdal_translate <GeoPDF filename> <Output Geotiff Filename> -of GTiff
--config gdal_pdf_layers_off “<pdf layername 1>,<pdf layername 2>,<pdf layername 3>” –-config gdal_pdf_dpi <output dpi>

Example: gdal_translate HI_Honolulu_20130410_TM_geo.pdf HonoluluTopo.tif -of GTiff --config GDAL_PDF_LAYERS_OFF “Map_Collar, Map_Frame.Projections_and_Grids, Map_Frame.Terrain.Shaded_Relief, Images.Orthoimage” --config GDAL_PDF_DPI 100

Step 3: Convert multiple geopdfs to geotiffs using batch file

Create a batch file to convert multiple geopdfs to geotiffs. You'll need a text editor such as Notepad. Enter the following into Notepad and save as a *.bat file.

@echo=on
SETLOCAL EnableDelayedExpansion
FOR /F %%i IN (‘DIR /B *.pdf’) DO (        
SET infile=%%i
SET outfile=!infile:.pdf=.tif!

Gdal_translate <GeoPDF filename> <Output Geotiff Filename> -of GTiff
--config gdal_pdf_layers_off “<pdf layername 1>,<pdf layername 2>,<pdf layername 3>” –-config gdal_pdf_dpi <output dpi>
)

Example: Notepad file saved as *.bat

batchfile.jpg

Step 4: Run the batch file in oSGeo4W Shell

Open up the OSGeo4W Shell and navigate to your directory where your geopdfs and the batch file are stored. Then use the TAB key to scroll through the files in your directory to select the batch (*.bat) file. Hit ENTER key when you the *.bat file selected. This will loop through all the geopdfs in the folder and convert them to geotiffs. 

After the conversion process you can then clip the geotiffs with its corresponding USGS 7.5 minute grid cells either in ArcGIS, QGIS or another GIS software. Make sure the grid cell layer is in the same coordinate system as your geotiffs before you do the clipping procedure.

The images below show examples of the Honolulu and Koko Head topo quads as they are viewed in QGIS after having the collars clipped off.

Example of Honolulu and Koko Head topo quad geotiffs with the corresponding USGS 7.5 minute grid cells as displayed in QGIS.

Example of Honolulu and Koko Head topo quad geotiffs with the corresponding USGS 7.5 minute grid cells as displayed in QGIS.

Example of Honolulu and Koko Head topo quad geotiffs without the USGS 7.5 minute grid cells as displayed in QGIS. Notice it is almost seamless -- you could get a seamless image by creating a mosaic and blending the edges.

Example of Honolulu and Koko Head topo quad geotiffs without the USGS 7.5 minute grid cells as displayed in QGIS. Notice it is almost seamless -- you could get a seamless image by creating a mosaic and blending the edges.

How to convert geopdf to geotiff using ArcGIS

Finally! After many years, the new release of ArcGIS Dekstop 10.4 now supports conversion of USGS geopdf topo quads to geotiffs.  ArcGIS 10.3.1 had support for geopdf conversion, but it did not work on USGS topos. The steps below are the workflow I went through to create a seamless, collar-less USGS topographic map of Oahu. A more detailed step-by-step instruction is in the tutorial here, How To Convert GeoPDF to GeoTIFF using AcrcGIS. 

Step 1: Set the pdf layer's default visibility property

Open the geopdf and set the default property of each layer to on or off.  In the Layer Property window make sure that the Initial State section: Export is set to "Exports When Visible". Save the geopdf so the default properties of the layers are set. Double check by re-opening the geopdf again.

Step 2: Convert geoPDF to geoTIFF using PDF To TIFF tool

Access ArcToolbox in either ArcCatalog or ArcMap. Go to ArcToolbox >> Conversion Tools >> PDF To TIFF tool. In the PDF To TIFF dialog window, set your input parameters. Take note of these input parameter options:

  • PDF Map (optional): Map_Layers
  • Clip Output to Map (optional): check the box if you want collarless tiff
  • Write GeoTiff Tags (optional): check the box to ensure ArcGIS writes out the correct coordinates. Note however, the converted tiff image will say "Unknown Spatial Reference" You will not be able to change/edit the spatial reference of the tiff in ArcGIS, even though it is showing the image to be in the correct coordinates. More details on this and on how to work around the "Unknown Spatial Reference" issue are in the pdf document.

Step 3: Batch convert geoPDF using Model Builder

If you want to batch convert many geopdfs to geotiffs, you can use a file iterator in Model Builder or the PDF To TIFF tool's Batch Grid mode. I prefer to use the Model Builder method as the model is more portable and can be easily modified and shared. With The PDF To TIFF Batch Grid, you have to input each individual file and set its parameters, which if you have many geopdfs to convert gets to be tedious.

A Model Builder example

A Model Builder example

PDF To TIFF Batch Grid Mode

PDF To TIFF Batch Grid Mode

Step 4: Create raster catalog and/or mosaic of the geotiffs

After converting the geopdfs to geotiffs, you can create a raster catalog or mosaic to create a seamless set of quads. Make sure to specify the correct spatial reference when creating the catalog or mosaic. 

Raster catalog of 17 topo quads for Oahu.  

Raster catalog of 17 topo quads for Oahu.