Differences

This shows you the differences between two versions of the page.

--- arma3:terrain:qgis-real-world-data-tutorial [2019-01-17 08:33]
snakeman
+++ arma3:terrain:qgis-real-world-data-tutorial [2019-05-04 03:37]
snakeman
@@ Line 3: / Line 3: @@
 ArmA 3 QGIS Real World Data Tutorial By Ross
-This process has often been described using Global Mapper, but I wanted to detail the equivalent steps within the open source GIS app QGIS.
+This process has often been described using Global Mapper, but I wanted to detail the equivalent steps within the open source (free) GIS app - QGIS.
-The goal is to achieve an export satellite image and heightmap with the same or better quality than would be possible with Global Mapper.
+The goal was to achieve an export satellite image and heightmap with the same or better quality than would be possible with Global Mapper.
-The is my second revision of this process as I recently discovered that due to an update to GDAL (**2.4.0**) one of the key steps in the original tutorial using '**Clip Raster by Mask Layer**' no longer works perfectly!
+I've used GDAL command lines where possible, which simplifies the process a little, and reduced additional processing on the data.
-So I reworked the process by getting a bit more under the hood, and using GDAL command line where possible, this also simplifies the process a little and reduced even further processing on the data.
+The following software versions were used for this tutorial:
-Using following versions:
 **[[https://qgis.org/en/site/forusers/download.html|QGIS v3.4.3]]**
 **[[https://sourceforge.net/projects/terraincognita2/files/|Terra Incognita v2.45]]**
+I have referenced Terra Incognita because i felt it was the easiest way to obtain satellite data in the oziexplorer map format - see this **[[https://pmc.editing.wiki/doku.php?id=arma3:terrain:satellite-texture-terra-incognita|tutorial]]** for help using Terra Incognita.
 **First an overview of the steps required:**
@@ Line 21: / Line 21: @@
 ==== For Heightmap: ====
-  * Load heightmap into QGIS (drag and drop)
+  * Load heightmap into QGIS (drag and drop)
   * Set QGIS to appropriate CRS for location of your HM
-  * Create shapefile & generate square feature using Advanced Digitizing panel
+  * Create shapefile & generate perfect square using Advanced Digitizing panel
   * Obtain extents from square feature
   * Run 2 GDAL commands
-    * First to set cell size, CRS and then clip to shapefile square extents
+    * First to set cell size, CRS and then clip to shapefile square extents
     * Second to convert to .asc file
@@ Line 34: / Line 34: @@
 **For single tile export**
-  * Load in just the .map files into QGIS
+  * Load in just the '.map' files generated from Terra Incognita into QGIS
-  * Merge Rasters
+  * Merge rasters
   * Obtain extents from square feature
-  * gdalwarp command will set cell size, CRS and then clip to shapefile square extents
+  * gdalwarp command will set cell size, CRS and then clip to shapefile square extents
-**For 4 tile export**
+**For 4 tile export (large projects 40960 and above)**
-  * Load in just the .map files into QGIS
+  * Load in just the '.map' files generated from Terra Incognita into QGIS
   * Merge Rasters
   * Create grid -- 2 x 2 Feature
   * Move each quarter to its own shapefile layer
   * Obtain extents from each quarter feature
-  * Run gdalwarp command for each quarter to set cell size and CRS and clip to shapefile square feature extents
+  * Run gdalwarp command for each quarter to set cell size and CRS and clip to shapefile square extents
 **And now detailed steps for:**
@@ Line 52: / Line 52: @@
 ===== Heightmap: =====
+This tutorial covers single raster heightmap as per the kind you will find on opentopo [[http://opentopo.sdsc.edu/raster?opentopoID=OTSRTM.082015.4326.1|here]], I will detail steps for heightmaps spread across muliptle rasters in another update. as they will require additional steps likely including Merge / Build Virtual Raster commands. This section does not cover any re scaling of heightmap, which I would recommend against anyway. Remember [[https://pmc.editing.wiki/doku.php?id=arma:terrain:grid_cell_size|Grid size x Cell size]] should = the same size (meters) as your source heightmap selection.
 **Drag and drop your heightmap '.asc' into QGIS**
+If you do use opentopo for source of Heightmap, make sure extract both the **.asc** and **.prj** files before proceeding with below. They should both be in the zip file from opentopo.
 **Set QGIS to appropriate CRS for location of your HM**
@@ Line 61: / Line 65: @@
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-11b.png}}
-Select the CRS to match your real world data, in this example its ‘**WGS 84 UTM zone 20N**’ because the terrain is from the Montserrat in the Caribbean. See also [[https://pmc.editing.wiki/doku.php?id=arma3:terrain:utm-projection|UTM projection]].
+Select the CRS to match your real world data, in this example its ‘**WGS 84 UTM zone 20N**’ because the terrain is from the Montserrat in the Caribbean. See also [[https://pmc.editing.wiki/doku.php?id=arma3:terrain:utm-projection|UTM projection]].\\
+This will ensure your QGIS project space works appropriately with any other data you want to add - eg road shapefiles etc. Terrain Builder will only use **UTM 31N** but we'll get to that later.
-This will ensure your QGIS project space works appropriately with any other data you want to add - eg road shapefiles etc. Terrain Builder will only use UTM 31N but we'll get to that later.
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-13b.png}}
@@ Line 69: / Line 72: @@
 **Create shapefile & generate square feature using Advanced Digitizing panel**
-Top menu -- **Layer **> **Create Layer** > **New Shapefile Layer**
+Top menu -- **Layer **> **Create Layer** > **New Shapefile Layer**
 Save to project folder location
@@ Line 83: / Line 86: @@
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-15b.png}}
-From menu -- **View** → **Toolbars** → **Advanced Digitizing Toolbar**
+From menu -- **View** → **Toolbars** → **Advanced Digitizing Toolbar**
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-16.png}}
@@ Line 95: / Line 98: @@
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-18.png}}
-To create a perfect square, **left mouse click** for your **top left** starting point, move your mouse to the right a little, then press ‘**d**’ on keyboard, type in the exact width (distance) required (in this example ‘20480’) then immediately press **Enter** -- be careful not to move the mouse before pressing **Enter** or it will mess up the number. Press **left mouse click** and you will have drawn your first horizontal line which turns red. By default this tool snaps to 90 degree angles, making it easy to draw your lines. Now to draw the vertical line start moving mouse your down, press ‘d’ again and type in same figure as above, press **Enter** and another **left mouse click**, repeat the process for the last 2 lines of the square.
+To create a perfect square, **left mouse click** for your **top left** starting point, move your mouse to the right a little, then press ‘**d**’ on keyboard, type in the exact width (distance) required (in this example ‘20480’) then immediately press **Enter** -- be careful not to move the mouse before pressing **Enter** or it will mess up the number. Press **left mouse click** and you will have drawn your first horizontal line which turns red. By default this tool snaps to 90 degree angles, making it easy to draw your lines. Now to draw the vertical line start moving mouse your down, press ‘d’ again and type in same figure as above, press **Enter** and another **left mouse click**, repeat the process for the last 2 lines of the square.
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-19b.png}}
-See a short video example using the **Advanced Digitizing** tool:
+See a short video example using the **Advanced Digitizing** tool:
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-AdvancedDigitizing.gif}}
-After the last **left mouse click** from the step above, the tool is still waiting to plot more points, so to finalise the square do a **right mouse click**
+After the last **left mouse click** from the step above, the tool is still waiting to plot more points, so to finalise the square do a **right mouse click**
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-20b.png}}
-This will then bring up a dialogue -- just type any number and Enter
+This will then bring up an attributes dialogue -- just type any number and Enter
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-21.png}}
+Save the above edits to the shapefile by clicking on the layer & selecting ‘**Toggle Editing**’
 **Obtain extents from square feature**
@@ Line 119: / Line 124: @@
 Start typing '**vector**' in Processing Toolbox search bar, and you will see '**Vector information**'
-Double click it and make sure your square **input layer** is selected - then just click **Run**
+Double click it and make sure your square **input layer** is selected - then just click **Run**
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-01.png}}
-You will only need to copy the **Extent** figures, which we will use within the GDAL commands in the next step
+You will only need to copy the **Extent** figures, which we will use within the GDAL commands in the next step
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-02.png}}
-**Run the first GDAL command to set CRS, cell size and then clip to shapefile square extents**
+**Run the first GDAL command which will - set CRS, cell size and then clip to shapefile square extents**
-From menu - **Plugins** > **Python Console**
+From menu - **Plugins** > **Python Console**
 Click '**Show Editor**'
@@ Line 135: / Line 140: @@
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-03.png}}
-The blank window on the right is where you will enter GDAL commands
+The blank window on the right is where you will paste GDAL commands
 Now you are ready to edit the command below to match your data - sections in bold are the parts to edit:
 import os\\
-os.system(r%%'''%%gdalwarp **-t_srs EPSG:32620** -wo SOURCE_EXTRA=1000 **-tr 5.0 5.0** -srcnodata "-9999" -r cubic -of GTiff **-te 576787.687480 1841104.815839 597267.687480 1861584.815839 D:/Arma/Heightmaps/Opentopo/output_srtm.asc** **D:/Arma/QGIS/Montserrat/converted.tif**%%'''%%)
+os.system(r%%'''%%gdalwarp -t_srs EPSG:**32620** -wo SOURCE_EXTRA=1000 -tr **5.0 5.0** -srcnodata "-9999" -r cubic -ot Float32 -of GTiff -te **576787.687480 1841104.815839 597267.687480 1861584.815839 D:/Arma/Heightmaps/Opentopo/output_srtm.asc** **D:/Arma/QGIS/Montserrat/converted.tif**%%'''%%)
 Some explanation of the key parameters:
 **-t_srs EPSG:32620 **\\
 CRS to match where the heightmap is from in the world.
 **-tr 5.0 5.0**\\
-The desired resolution of your heightmap, match with Cell size set within your Mapframe properties in Terrain Builder:
+The desired resolution of your heightmap, match with **Cell size** set within your Mapframe properties in Terrain Builder:
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-04.png}}
 **-te 576787.687480 1841104.815839 597267.687480 1861584.815839**\\
-The 4 long figures represent the **extents** of your square, and should be replaced with your own figures obtained in previous step.
+The 4 long figures represent the **extents** of your square, and should be replaced with your own figures obtained in previous step.
 Update the paths to match where you are storing your source heightmap, and where you want the output tif saved to.
 Input:\\
 **D:/Arma/Heightmaps/Opentopo/output_srtm.asc**  use double quotes around your input path if it contains any blanks
 Output:\\
@@ Line 164: / Line 169: @@
 **Run second GDAL command - converting tif from above step to a Terrain Builder friendly .asc**
+Remember to change the paths in below command to match where your files are located
 import os\\
-os.system(r%%'''%%gdal_translate -of AAIGrid D:/Arma/QGIS/Montserrat/converted.tif D:/Arma/QGIS/Montserrat/final.asc%%'''%%)
+os.system(r%%'''%%gdal_translate -of AAIGrid **D:/Arma/QGIS/Montserrat/converted.tif D:/Arma/QGIS/Montserrat/final.asc**%%'''%%)
@@ Line 173: / Line 180: @@
 ==== For single tile export ====
-**Load in just the .map files into QGIS**
+**Load in just the '.map' files generated from Terra incognita into QGIS**
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-1b.png}}
@@ Line 179: / Line 186: @@
 **Merge Rasters**
-Top Menu -** Raster **→ **Miscellaneous **→ **Merge**
+Top Menu -** Raster **→ **Miscellaneous **→ **Merge**
 Select all your .map files as input layers
@@ Line 185: / Line 192: @@
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-3b.png}}
-change **Output data type** to ‘Byte’ as the default is ‘Float32’.
+change **Output data type** to ‘Byte’ as the default is ‘Float32’.
 Select ‘**Save to File**’ under Merged -- as the default is to a temporary file.
@@ Line 193: / Line 200: @@
 **Obtain extents from square feature**
-You should already have this from working on heightmap above   **576787.687480 1841104.815839 597267.687480 1861584.815839**
+You should already have this from working on heightmap above   **576787.687480 1841104.815839 597267.687480 1861584.815839**
 **Run GDAL command**
 import os\\
-os.system(r%%'''%%gdalwarp **-t_srs EPSG:32620** -r cubic -wo SOURCE_EXTRA=1000 **-tr 1.0 1.0** -r cubic -of BMP **-te 576787.687480 1841104.815839 597267.687480 1861584.815839** D:/Arma/QGIS/Montserrat/merged.tif D:/Arma/QGIS/Montserrat/mont.bmp%%'''%%)
+os.system(r%%'''%%gdalwarp **-t_srs EPSG:32620** -r cubic -wo SOURCE_EXTRA=1000 **-tr 1.0 1.0** -r cubic -of BMP **-te 576787.687480 1841104.815839 597267.687480 1861584.815839** D:/Arma/QGIS/Montserrat/merged.tif D:/Arma/QGIS/Montserrat/mont.bmp%%'''%%)
 Some explanation of the key parameters:
@@ Line 224: / Line 231: @@
 **Create grid -- 2 x 2 Feature**
-You will need to have created the **square shapefile** first -- see above steps
+You will need to have created the **square shapefile** first -- see above steps
-Top Menu -- **View** → **Panels** → **Processing Toolbox**
+Top Menu -- **View** → **Panels** → **Processing Toolbox**
-Processing Toolbox Menu -- **Vector Creation** > **Create Grid**
+Processing Toolbox Menu -- **Vector Creation** > **Create Grid**
-Set **Grid type** to ‘**Rectangle (polygon)**’ & the **Horizontal & Vertical** to half the distance of your main square
+Set **Grid type** to ‘**Rectangle (polygon)**’ & the **Horizontal & Vertical** to half the distance of your main square
 Set it to save new grid feature to .shp file
@@ Line 236: / Line 243: @@
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-26b.png}}
-Select the shapefile layer for your square
+After clicking '**Use Layer Extent**' Select the shapefile layer for your square
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-13.png}}
 After clicking **Run**, your original square will then be Covered with a new grid of 4 tiles -- perfect quarters of your original square
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-30b.png}}
@@ Line 248: / Line 255: @@
 Because you will need to export each quarter separately for use in Terrain Builder, we will move each quarter onto its own shapefile layer.
-Click **Select Features** -
+Click **Select Features** -
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-31.png}}
@@ Line 256: / Line 263: @@
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-32b.png}}
-Top Menu -- **Edit** > **Copy Features**
+Top Menu -- **Edit** > **Copy Features**
-Then… **Edit** → **Paste Features As **→ **New Vector Layer**
+Then… **Edit** → **Paste Features As **→ **New Vector Layer**
 Save shapefile layer to something obvious
@@ Line 284: / Line 291: @@
 And list them out for safe keeping -
-TL -  Extent: (576787.687480, 1851344.815839) - (587027.687480, 1861584.815839)\\
+**TL -  Extent**: (576787.687480, 1851344.815839) - (587027.687480, 1861584.815839)\\
-TR -  Extent: (587027.687480, 1851344.815839) - (597267.687480, 1861584.815839)\\
+**TR -  Extent**: (587027.687480, 1851344.815839) - (597267.687480, 1861584.815839)\\
-BL -  Extent: (576787.687480, 1841104.815839) - (587027.687480, 1851344.815839)\\
+**BL -  Extent**: (576787.687480, 1841104.815839) - (587027.687480, 1851344.815839)\\
-BR - Extent: (587027.687480, 1841104.815839) - (597267.687480, 1851344.815839)\\
+**BR - Extent**: (587027.687480, 1841104.815839) - (597267.687480, 1851344.815839)\\
 **Run GDAL command for each quarter to set cell size and CRS and clip to shapefile square extents**
@@ Line 297: / Line 304: @@
 ===== Loading your assets into Terrain Builder =====
+I would recommend saving copies of any files to be used in **Terrain Builder** into a separate terrain project folder. This will prevent your source project files being adversely affected, and potentially no longer working within **QGIS** - particularly relevant to those changes mentioned below.
 **Heightmap (.asc) edits**
-Open your asc and change the following:
+Open your **asc** in notepad and change the following to ensure it loads in **Terrain builder**:
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-14.png}}
-Before loading the asc file into Terrain Builder, delete the .prj file. Otherwise it will not import into TB.
+Save to a new asc, whilst maintaining the original for use in your QGIS project space
+Before loading the asc file into **Terrain Builder**, delete the .prj file. Otherwise it will not import into TB.
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-09.png}}
@@ Line 326: / Line 336: @@
 {{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-11.png}}
+===== Preparing Road Shapefile =====
+**Load into QGIS (drag and drop) road shapefile**
+In this example I got a good dataset from - [[https://data.humdata.org/|https://data.humdata.org/]]
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-37.png}}
+**Clip vector to square shape**
+Top Menu - **Vector** > **Geoprocessing Tools** > **Clip**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-38.png}}
+Save features from clip to new shapefile -  selecting appropriate CRS, in this example **CRS UTM-20**\\
+Top Menu - **Layer** > **Save As**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-39.png}}
+**Set road shapefile to CRS - 31N**
+RMB on road shapefile > **Set CRS** > **Set Layer CRS**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-40.png}}
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-41.png}}
+**Set QGIS project CRS to UTM- 31N (bottom right corner):**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-42.png}}
+**Collect extents from your square shapefile:**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-43.png}}
+Extent: (**576787.687480**, **1841104.815839**) - (597267.687480, 1861584.815839)
+The first two values will be used in the v.transform step below.
+**v.transform on road shapefile using calculated extents from square shapefile**
+Your heightmap asc must adhere to Terrain Builders required values of **easting 200000** and **northing 0**
+Your road shapefile also needs to line up to the same values - we will use **v.transform** to achieve this
+So taking the extent values we grabbed in previous step -\\
+We will do the following calculations to bring these values to (**200000** and **0**):\\
+**Easting:  576787.687480  - 376787.68748 = 200000**\\
+**Northing: 1841104.815839 - 1841104.815839 = 0**\\
+**-376787.68748**\\
+**-1841104.815839**
+Having figured out the correct figures plug them into the **v.transform** process
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-44.png}}
+Drag and Drop into QGIS your heightmap asc - the one you are able to load into Terrain builder - already adjusted to (**200000**, **0**)
+Set CRS on above asc to **UTM - 31N**
+Right click on asc in **Layers** > **Zoom to Layer**
+In **Layers** panel drag the asc below your **transformed road shapefile** - so you can see the roads layered above the asc
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-45.png}}
+Your roads should now be perfectly aligned above your heightmap
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-46.png}}
+==== Setting ID and ORDER fields ====
+RMB on transformed shapefile layer > Toggle **Editing**
+RMB on transformed shapefile layer  > Open **Attribute Table**
+Click **Delete field** - select all fields and press **OK**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-47.png}}
+New Field - Name '**ID**' - length **0**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-48.png}}
+New Field - Name '**ORDER**' - length 0
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-49.png}}
+Toggle - **Multi Edit mode**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-50.png}}
+ID - Enter **0** - click **Update All**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-51.png}}
+Select ORDER from drop down - Enter **1** - click **Update all**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-52.png}}
+Click - **Switch to table view**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-53.png}}
+Click **Save**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-54.png}}
+RMB on transformed road shapefile - **Toggle Editing** to save the changes
+**Switching to Terrain Builder -**
+**Load your road shapefile**
+Top menu - **File** > **Import** > **Shapes**
+Click **OK**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-55.png}}
+Perfectly overlaid within **Terrain builder:**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-56.png}}
+===== 3D Preview using Qgis2threejs =====
+If you want to see an instant 3d preview of your terrain within QGIS you can with the amazing plugin **Qgis2threejs**
+First install it:
+Top menu > **Plugins** > **Manage and Install Plugins**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-57.png}}
+RMB on the Merged layer and select **Zoom to Layer**
+Then select both the **Merged** raster, and the original source heightmap - deselect all other layers
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-58.png}}
+Start the Qgis2threejs plugin - Top Menu > **Web** > **Qgis2threejs** > **Qgis2threejs Exporter**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-59.png}}
+Select source heightmap and nothing else
+RMB on heightmap > **Properties**
+Set the **Resampling** level to **6**
+Set **Resolution** to **400%**
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-60.png}}
+Enjoy browsing around your terrain in 3D:
+{{https://pmc.editing.wiki/images/Ross-QGIS-Tutorial-61.png}}
+You can also export your terrain to a browser interface if you want - **File** > **Export to web**
 **Coming Soon:**
-  * Road shapefile creation
   * Assisted image classification - for creation of mask
-  * 3d preview using Qgis2threejs

PMC Editing Wiki

User Tools

Site Tools

Differences

Page Tools