Differences

This shows you the differences between two versions of the page.

--- arma3:terrain:qgis-real-world-data [2017/01/29 23:11] – moved terra incognita and global mapper headers to their own pages. snakeman
+++ arma3:terrain:qgis-real-world-data [2024/08/02 18:54] (current) – links added. snakeman
@@ Line 1: / Line 1: @@
-====== Disclaimer WIP ======
+====== ArmA 3 Real World Data using QGIS ======
-This page is work in progress, **this is not a complete tutorial**. This is notes written down from our discord arma server #terrain_makers channel discussion. I will try to finish this page into tutorial as soon as possible, so far I have not been successfully with my efforts to get real world data through QGIS into terrain builder and arma3.
+[[https://www.pmctactical.org/forum/viewforum.php?f=68|ArmA 3 Forum]], [[:arma3|ArmA 3]], [[arma3:config|ArmA 3 Config]], [[arma3:missions|ArmA 3 Missions]], [[arma3:modeling|ArmA 3 3D Modeling]], [[arma3:scripting|ArmA 3 Scripting]], [[arma3:terrain|ArmA 3 Terrain]], [[arma3:texturing|ArmA 3 Texturing]], [[arma3:tools|ArmA 3 Tools]]
-Please [[:contact|contact]] us if you can provide any more information about QGIS real world data subject.
+How to get real world data through QGIS into terrain builder and arma3. See our main [[arma3:terrain:real-world-data-tutorial|Real world data tutorial]] page for the main tutorial part and [[qgis-real-world-data-tutorial|QGIS real world data tutorial]] by Ross.
-btw you can scroll to bottom of this page to see last update time in lower right corner.
@@ Line 12: / Line 10: @@
 Goal of this work in progress page is to learn how to get real world data for:
-  * terrain
+  * <del>terrain</del> see [[arma3:terrain:heightmap-real-world-data]]
-  * satellite texture
+  * <del>satellite texture</del> see [[arma3:terrain:satellite-texture-terra-incognita]]
   * satellite mask
-  * roads
+  * <del>roads</del> see [[arma3:terrain:road-shapefiles-real-world-data]]
-  * vegetation
+  * <del>vegetation</del> see [[real-world-data-tutorial]]
 And process these through [[http://www.qgis.org/en/site/|QGIS]] (free open source program) into terrain builder and finally into finished arma3 terrain addon.
-**I do not know if all of the above is possible, terrain is but mask? vegetation? No idea yet, I'm learning one step at the time.**
-====== Real World Terrain Data to ArmA 3 ======
-**HOWTO get real world terrain data into ArmA 3**
-====== Terrain Heightmap ======
-**ALOS World 3D - 30m data from opentopo.sdsc.edu**
-Go to [[http://opentopo.sdsc.edu/lidar?format=sd|opentopo.sdsc.edu]] website, click Global Data tab, click satellite data -> "ALOS World 3D - 30m". Now you should see world map with red boxes marking the areas for available data.
-Use mouse wheel (middle button) to zoom in / out of the globe map, LMB and drag to move the map.
-Once you have zoomed enough in to a smallish area, click "select a region" button in left side of the globe map screen. Mouse cursor changes to a cross now and you can drag a box to select region. Below you can see the data selection coordinates.
-In coordinates box, if you choose to do so, you can tick "manually enter selection coordinates" box and then manually type in lat/lon coordinates. This is the only sure way to grab perfect square power of two data selection.
-Once you are satisfied with your selection, scroll down to data output formats. Choose "Arc ASCII Grid" from the drop down.
-On visualization untick "Generate hillshade images from DEMs" box.
-Finally on job description box "Enter your e-mail address" just type in whatever asdf@aol.com some fake email address, unless you specifically want them to email you of course. At some point if you try to get huge amount of data that there is long waiting period while site is processing it, perhaps its good way to add your own email here so you get notification when the download is ready. But for normal small size data selections, there is no need for real email.
-When fake email is in, click submit.
-Once processing is done you get "Raster Job Report" page which has your download. On "Download compressed raster results:" click the "rasters_alos.tar.gz" downloadable file to... you know, download.
-This download is always the same file name so **please be careful and unpack this file into a directory**, use winrar RMB "extract to rasters_alos/" and then **rename the directory to some good name which you recognize** because the terrain Asc ASCII Grid file inside is also named just "output_alos.asc" every time you download new data. This way you never overwrite your existing terrain data downloads.
-You should be able to import the tar gunzip (.gz) file into QGIS, however you should unpack it first like explained above. You can save the heightmap from QGIS into TIF file format and then read the tif with L3DT. L3DT can then save the heightmap as ASCII grid or XYZ.
@@ Line 77: / Line 40: @@
 ====== OpenStreetMap ======
-Once you have terrain loaded in QGIS, you can get streets / roads to show transparent on top of it.
+Once you have terrain loaded in QGIS, you can get streets / roads to show transparent on top of it. This is just an image, not actual road shapefiles, see [[arma3:terrain:road-shapefiles-real-world-data|Road Shapefiles Real World Data]] how to download roads.
 Use web -> openlayers plugin -> openstreetmap -> openstreetmap (heh yes twice).
@@ Line 86: / Line 49: @@
 You can click the X button in layer panel on each layer to turn them on/off.
+You should download [[http://download.geofabrik.de/|OSM shapefiles from geofabrik.de]] instead.
@@ Line 98: / Line 63: @@
 Hydrra: Yeah, The plugin provides both sat image and street map. just make sure to separate the layer groups so it's easier to keep track of.
+You should download [[arma3:terrain:satellite-texture-terra-incognita|Satellite Texture with Terra Incognita]] instead.
-====== Convert Geodetic .ASC ======
-Terrain Processor should be able to convert geodetic .ASC heightmap into something that terrain builder would read. I have not managed to get it to work, terrain processor always does (Not Responding) freeze, meaning my file and/or settings were wrong.
-Source [[https://community.bistudio.com/wiki/Terrain_Processor:_Tutorial#Heightmap|bis wiki terrain processor tutorial: heightmap]].
-====== QGIS Print Composer ======
-Print Composer will save an image of your current view.
-Open it by Project -> new print composer, or use CTRL-P
-Make it a square pare size, for example 100mm x 100mm.
-Select layout -> add map, drag a box around the white area.
-The view should look like your current QGIS main view, hide/show layers to get the view you want like google satellite.
-Item properties -> extents, no idea what to do with those :)
-FinCenturion: you need to know now exactly how large of a chunk you need. and the coordinates / extents of this data. I have changed up everything to match arma worldspace by using the methods in my [[arma3:terrain:qgis-reproject-transform-tutorial|QGIS Reproject & Transform Tutorial]].
-FinCenturion: but you need lower left corner x,y and upper right corner x,y coords\\
-FinCenturion: if you have more data than you will need you can of course use an area that is in power of two - so you can keep the horizontal scale 1:1 with real life\\
-FinCenturion: say you want to do a 4096 x 4096 m map, and your lower corner is in the (fictive) coords of x 1000 and y 1000 - > exents 1000, 1000 and 5096 and 5096\\
-FinCenturion: if that made any sense :smile:
-To save the image use composer -> export as image
@@ Line 169: / Line 105: @@
-====== Universal Map Downloader ======
+====== QGIS Refactor Fields ======
-See [[arma2:terrain:google-satellite-maps-downloader|arma2 google satellite maps downloader]] page first.
-Or **Offline Map Maker** which is better version of **Universal Map Downloader** I think.
-This tutorial is supposed to be QGIS real world data... umm err yes but, heh :)
-Some comments from bis forum:
-While I [ZeroG] was using UMD myself at times I created my tutorial, I nowadays only use Global Mapper, as it allows you to download layers just the exact size of the map and you can export them just fitting on the square you defined, projected in UTM and automatically stitched together. Furthermore, there are far more layers accessible on the web than UMD allows you. Just look for WMS/TMS layers on google and you will find lots of data for your area which can be easily added to Global Mapper.
-For satellite images, I use "World Imagery" provided by ESRI (without watermark ofc) but in general I think that it is not a too good idea to use high detail satmaps at all, as you can never recreate the detail displayed on the image (expect for large landscapes without buildings maybe). It might look good at height but silly on the ground. Detail level as on Chernarus satmap is enough.
-What I know find quite useful is "World Topographic Map", as the terrain features there are represented in one color and not so detailed like on satellite images, which means you can easily separate them and fill them with (Chernarus/Takistan)-textures as satmask or with colors for texture mask.
-for a small terrain the world imagery is usefull, but with a bigger terrain size u run into trouble realy fast. as u encounter issue's like different tiles from different seasons/years in your downloaded sat image. i must admit it does give u a 100% fit on your terrain wich is realy a + but i ran in this issue myself for atleast a few times with world imagery.
-you download your hight data here:
-http://gdex.cr.usgs.gov/gdex/
-there u get your hight data and coords those coords u put into universal maps downloader to download satallite image. Global mapper only needs the .tiff file, coord info is embedded in the .tiff file.
-If you want to get geo-referenced images of any zoom level, with the GPS stuff in the file, use this program: http://www.zubak.sk/TerraIncognita/ Global mapper can import OziExplorer data. When downloading the image, just select an area bigger than what you think you need, and use global explorer to export both the .xyz and a .png. Best of all, Terra Incognita is free!
-**Snake Man Offline Map Maker notes:**
-https://earthexplorer.usgs.gov/ first coord is LATITUDE, second is LONGITUDE.
-latitude = up/down\\
-longitude = left/right
-zoom level low number == low quality, high number high quality. Offline map maker allows zoom level 19 while universal map downloader only 17 unless I'm mistaken.
--04-17 when downloading, the opening of earthexplorer did NOT avoid my IP getting banned (opening that site doesn't even make any sense).
-Not sure if zoom level 19 is too much, back in the arma2 terrain era I used universal maps downloader zoom level 17. not sure if the scale is better or perfect for arma terrains in general.
-Using Offline Map Maker's image combiner with 74k x 74k image took a really long time with the program being (Not Responding) mode, just wait and it finishes. The image combiner did 10.4gb bmp image as well as jpg which both did not open in photoshop. QGIS opened the .map file but the downloaded project was like same image stacked three times, really weird.
-====== Easy Bing Maps Downloader ======
-[6:12 AM] M1lkm8n: Easy Bing map downloader\\
-[6:14 AM] M1lkm8n: You just input the top left lat/long and bottom right lat/long and set you zoom level and go....it even stitches them together and georeferences   It for you.\\
-[6:15 AM] M1lkm8n: It's awesome in combination with global mapper
-There doesn't seem to be official homepage for it, only tons of "download sites" of various kinds.
-====== Open Questions ======
-Snake Man: ok when you choose area of terrain you want to download, do you just get "bulk" data with large amounts of room to maneuver or do you get square power of two exact size for arma3 use like 10km x 10km?
-====== Misc Notes ======
-**QGIS**
-[4:09 PM] NigeyS: oo reproject and transform... yeah thats gonna give me a headache\\
-[4:19 PM] FinCenturion: for image export purposes you dont need to reproject and transform\\
-[4:19 PM] FinCenturion: but if you wish to use shapefiles straight up in TP or TB you do\\
-[4:20 PM] NigeyS: atm i just need to get it into photoshop for the layer mask\\
-[4:21 PM] FinCenturion: to use print composer make  a new print composer, then under the composition tab in the workspace make sure the page is square, i use 1000x1000mm\\
-[4:22 PM] FinCenturion: then use the add map function from the layout drop down box\\
-[4:22 PM] FinCenturion: scale the newly added map to cover the white sheet\\
-[4:22 PM] FinCenturion: then click on your new map, go to the item proprties tab on the right again\\
-[4:23 PM] FinCenturion: use the Extents to get the correct display\\
-[4:23 PM] FinCenturion: so you basically need to know pretty exact coords of your map\\
-[4:24 PM] FinCenturion: that is why sometimes it's better to have all your shape data reprojected to arma 3 TB worldspace\\
-[4:24 PM] FinCenturion: anyhoo, then you use the export image button and set a file name, then final resolution\\
-[4:25 PM] FinCenturion: it's a bit of a bother at first but QGIS is a massive help once you get used to it\\
-[4:26 PM] NigeyS: that'll help thanks, i have all the co-ords etc
-**OpenStreetMaps Shape Files**
-The .shp and the .osm packages are "nominally" the same. the main visible difference being the shapefile downloads are individual "layers" and the ".osm" is "All-In-One".
-However, you'll sometimes find that the ".osm" version of an area has more detail than the equivalent shapefiles pack, or there's additional layers not included in the matching .shp pack. So it's always a good idea to grab both sets of data for any area you're seriously interested in, and check them both out thoroughly - cherrypick the best of the useable data from whichever seems best.
+OSM place names data sometimes comes with different order for the fields, this will break up PMC OSM Places CSV2HPP tool. You can fix this in QGIS by using "Refactor fields" plugin.
-You'll find that the .osm's seem to consist of all the shapefiles muddled together on a single layer... However, all the individual shapes are carrying "metadata" - they have ID's, names, types, etc...
+In processing toolbox search for "refactor fields", open it. Now you can reorder the fields mapping dialog list, so if you need to swap "name" and "type" fields, just click one and use the arrow keys to move up or down.
-Most GIS programs, GM included, have a "metadata search" function which will allow you to search that whole jumbled layer of shapes and select only those labelled "residential road" for example, or "highway"... you can then select only that "class" of shape and isolate them from the rest by copying to a new layer...
+When reorder is OK then just click run button and it creates refactored layer with the new reordered fields.
-Keep doing that and you can extract a whole bunch of useful layers from lakes & rivers, thru roads, vegetation areas, agricultural areas, etc, etc... with OSM it's all down to luck as far as coverage and quality is concerned, but most times you should be able to acquire some useable stuff, and save a considerable amount of time....