PMC Editing Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== ArmA 3 Road Shapefiles from Real World Data ======

[[https://www.pmctactical.org/forum/viewforum.php?f=68|ArmA 3 Forum]], [[:arma3|ArmA 3]], [[arma3:config|ArmA 3 Config]], [[arma3:missions|ArmA 3 Missions]], [[arma3:modeling|ArmA 3 3D Modeling]], [[arma3:scripting|ArmA 3 Scripting]], [[arma3:terrain|ArmA 3 Terrain]], [[arma3:texturing|ArmA 3 Texturing]], [[arma3:tools|ArmA 3 Tools]]

**How to use real world data road shapefiles from OpenStreetMap (OSM) in ArmA 3**.

In this tutorial we download [[https://www.openstreetmap.org|OSM]] shapefiles from the web, crop them to our terrain size and reproject them to match terrain builder UTM zone.

If not otherwise mentioned, global mapper is assumed.


====== OpenStreetMap Download ======

[[http://download.geofabrik.de/|Geofabrik]] site has OpenStreetMap (OSM) shapefiles available for download.

Do not try to download "Asia" or "Africa" as those files are **huge**, click on [[http://download.geofabrik.de/asia.html|Asia]] or africa instead and from there you can find individual country pages like [[http://download.geofabrik.de/asia/iraq.html|iraq]] and afghanistan so your downloads are mere few megabytes instead of gigabytes (not to mention the unpacked data is insanely large).

OSM includes roads but also many other features, they even have buildings and vegetation areas. Although no idea how to use the building information with terrain builder or other tools.

If you cannot find country from geofabrik, try [[http://extract.bbbike.org/|extract.bbbike.org]] instead.


====== OSM Road Shapefile Tutorial ======

**OSM Roads Using Global Mapper and QGIS**

We need BOTH of these software's to achieve OSM roads in terrain builder because global mapper v18 does not shift road shapefiles properly (v15 does apparently). If you know how to do this alone in either software please [[:contact|contact us]]. :(

//You cannot replace OSM road features with arma3 ID/ORDER before you transform them! If you do, QGIS will fail the v.transform process.//

  - download road shapefiles from geofabrik as SHP (shapefiles) files for your terrain country(s)
  - unpack and move them to directory where you want to store OSM shapefiles, but do not put in your terrain builder project dir
  - open your terrains global mapper project (UTM projected) and open your road shape file(s), hide other layers
  - select digitizer tool (ALT-D) and select / hightlight user created feature grid
  - RMB grid -> crop/combine/split functions -> crop loaded features to selected areas (or simply click crop to selected areas button, haha)
  - setup dialog choose feature layer -> <create new layer for feature>, click ok
  - give name to your layer, good name would be cropped_roads for example, on layer projection dialog click ok
  - hide osm road shapefile layer, now you see road shapes inside cropped_roads layer, cool!
  - file -> export -> export vector/lidar format, choose dropdown shapefile, click ok
  - shapefile export options -> export lines / select file -> give filename click save
  - enable "generate projection (prj) file for each exported shp file, untick rest, click ok

**DOES NOT WORK IN V18!:**
  - file -> unload all (CTRL-U)
  - open data files -> select the shp file you just exported
  - RMB the only layer which is your cropped road shapefile -> shift - shift selected layers a fixed distance
  - choose dropdown shift layers a fixed distance, click ok
  - specify offset dialog insert the X/longitude and Y/latitude values matching to your terrain coordinates (see below), click ok

What does not work in v18? Well the coordinates jump to some crazy large number 3999999999999900 something like that, I have no idea why and what I do wrong in that part. Read below how to transform roads in QGIS instead (or use older global mapper like v15 which should work).

**QGIS Transform:**
  - open your heightmap in UTM 31N, this is just for visual reference. add raster layer -> choose your asc file. this needs to be set to UTM zone 31N.
  - add vector layer -> browse, choose your global mapper cropped and exported road shapefile, click open
  - RMB roads layer -> set layer CRS -> choose WGS 84 / UTM zone 31N, click ok
  - RMB roads layer -> zoom to layer
  - next we use stuff from [[arma3:terrain:qgis-reproject-transform-tutorial|QGIS Reproject & Transform Tutorial]], CTRL-ALT-T to open processing toolbox
  - type in v.transform and double click open the v.transform option
  - input vector layer needs to be your cropped roads, X and Y shift needs to be your terrain coordinates calculation (see below)
  - click run to execute, wait patiently it takes a while depending on your shapefile size
  - select transformed layer, RMB -> zoom to layer, now you see roads aligning perfectly with your heightmap, congrats!
  - transformed layer RMB -> save as -> browse, this is your final road shape file name to be used in terrain builder, click ok

**Global Mapper ID / ORDER**

  - Fix shapefile attributes for ID and ORDER with [[arma3:terrain:road-shapefiles-real-world-data#id_and_order]] tutorial.

All done! :-)

Enjoy your real world data openstreetmap roads in terrain builder :-)

{{ https://pmc.editing.wiki/images/Terrain-Builder-Global-Mapper-QGIS-Shapefile-Roads.png }}


====== Shift Projection to Terrain Builder ======

In global mapper when you have chosen your terrain area by creating user created features grid, you need to export ASCII Grid ASC heightmap file to see the header for coordinates.

Use proper text editor to open this ASC file and check the header for xllcorner and yllcorner values. These values are used to calculate the terrain builder easting 200000 and northing 0 values.

Easting 200000 will be done by calculating 200000 - xllcorner value. And northing is dead simple, you just take yllcorner value as-is.

So now you have these two values, they can me inputted to QGIS to transform your project coordinates into those of arma3 terrain builder.

Here is example **heightmap.asc** header:
<code>
ncols         512
nrows         512
xllcorner     618849.895
yllcorner     3433172.322
cellsize      5
</code>
So you would take xllcorner and calculate that like this "200000 - 618849.895" equals to -418849.895 value. Then as terrain builder northing is just 0, we dont have to do any calculations we use it as is.

Result shift / transform would be:
<code>
X: -418849.895
Y: -3433172.322
</code>


====== Remove "Bad" Roads ======

ArmA 3 crashes if you have more than roughly 9000 roads / features in the roads.shp shapefile. This is not precise number as it also depends on your other terrain properties, highest we have had in-game is 9,198 and lowest that caused crash was 4,941 so its not exact science. This is only issue for those large terrains or well, terrains that have large number of roads in the shapefile.

Note: for example plain deserts (heightmap) of iraq easily allows OSM road features of 9000 but the rugged mountains (heightmap) of afghanistan crashes for less then 5000. So use your own judgement with all these gazillion variables when you track down the amount of roads your terrain can handle.

Mexico juarez 81km terrain had 7469 road features which caused virtual blocks crash. After global mapper 6m combine 4056 it still crashed. Finally after I cleanly reduced the features by deleting FCLASS roads to number I didn't write down (damn) and then 6m combine down to 1139 the crash was avoided.

OSM shapefiles come with all kinds of backyard "path" roads etc, you might want if not have to remove some of these.

OSM Road attribute types are described in [[http://wiki.openstreetmap.org/wiki/Map_Features#Highway|openstreetmap map features: highway]].

Use search -> search by attributes. This search vector data dialog you can do all kinds of search & delete operations.

If you click to sort the FCLASS column, you can most likely see "unknown" and "unclassified" types, its up to you to decide which road types to remove (see the above OSM road attribute types list), as you really don't need all in arma3 terrain.

**How to delete**

Choose attribute/item dropdown for FCLASS, then type your search string to compare value field, then hit new search button.

Now you can click select all and delete selected buttons to delete all these search matches.

Move the search vector data window away from normal global mapper view and orient the road features so you can see them. For those pimps with triple screen setup this is awesome but its doable even with single screen.

You can see the selected roads highlighed in yellow.

While you click to delete roads, monitor the cropped_roads layer for feature count, once it reached 9000 or below you can stop because that is the rough limit arma3 crashes.

Once you have searched roads, you can search * to return all roads, because every search is only searching what is listed currently.

Check the FCLASS or type, sort it and browse to see all listed there. Sometimes there are really odd types like "yes".

Once you're done with deleting "bad" roads, simply export your roads by file -> export -> export vector/lidar format, as usual.

**ArmA 3 Specific Roads Deletion**

Delete these roads as they are mostly too excessive for arma3 terrains:
<code>
bridleway
steps
foot*
cycle*
path
pedestrian
track_grade*
*link*
living_street
service
unknown
unclassified
residential
tertiary
track
secondary
</code>
track* takes too many roads from ukraine luhansk, from 17k to 380 heh.

tertiary has lot of country roads in ukraine.

residential has those nice city roads, depends on your terrain if you need to delete these.

Anything below 9000 OSM features in roads.shp should work in arma3.

List of important features for arma3 gameplay:
<code>
motorway
trunk
primary
</code>


====== Simplify Road Shapefiles ======

**Simplify Road Polylines**

Most likely your OSM downloaded road shapefiles are too complex for terrain builder, so you //might// need to simplify them. Meaning there are way too many vertices in the polylines which can be reduced. **Note 2018-03-02 update:** arma3 loads any OSM poly line shape file just fine, there is no need to simplify for arma3's sake, PMC has done hundred terrains with various size and quality OSM road poly lines and all of them have been working great.

If you import too complex or "large" road shapefiles to terrain builder it will start to stutter and freezes up.

In global mapper select all your road features, digitizer tool RMB -> move / reshape features -> simplify reduce vertices of selected line area features.

From enter simplification threshold dialog, use horizontal threshold 10 and vertical threshold elevation (meters) 0.2 for good results.


====== Combine Road Features ======

USA Los Angels 40km terrain openstreetmap.org road polylines even after "bad roads" reduction were over 17,000 features which is like ten thousand over the limit. They need to be reduced. Please note that this was some very odd issue on these specific OSM shapes because normal roads / streets were split up to hundreds of small features when it should have been one feature.

I have not experienced such splitting up of polylines in other terrains in this scale.

Select all roads -> RMB in the viewport -> crop/combine/split functions -> combine. Then use about 6 meter value, should work fine (we used also 20m but that might be too large). In select combine options menu let it be default (combine any overlapping and everything else ticked).


====== ID and ORDER ======

When you are using OSM road shapefiles there are lot of crap info attributes which are not used by terrain builder, you must delete these and replace with ID and ORDER attributes. If you try to bring OSM shapefiles as-is to arma3, it will crash.

Terrain Builder uses ID and ORDER values for road shapefile database. See further details from [[arma3:terrain:ultimate-terrain-tutorial#roads_-_export|PMC Ultimate Terrain Tutorial: Roads - Export]].

  - select digitizer tool (ALT-D)
  - select all features (there are many ways, for example drag a box around your roads, or RMB layer -> select all features)
  - RMB feature layer to bring up modify selected line features dialog
  - feature attributes area click delete button, select attributes to delete dialog click check all, click ok (attributes are now deleted)
  - back in modify selected line features dialog click new button, name ID, value 1, click ok. now ID attribute is set
  - click new, name ORDER, value 0, click ok. now ORDER attribute is set
  - modify selected line features dialog click ok
  - ready to export your ID and ORDER attributes roads shape file

Make road orders for roads and always connect a smaller road to bigger road so that the bigger road has smaller order than the road joining it. This way the joining road should go automatically under the road that it's joining.

If you don't use proper ID value, then you get some placeholder textureless road.

**In-Game Crash**

If your roads shapefile is faulty, for example if you forgot to tweak the ID, ORDER etc values when exporting from Global Mapper or QGIS, then arma3 will simply crash when you load terrain using the shapefile.


====== OSM vs SHP Files ======

**by Bushlurker**

The .shp and the .osm packages are "nominally" the same. the main visible difference being the shapefile downloads are individual "layers" and the ".osm" is "All-In-One". 

However, you'll sometimes find that the ".osm" version of an area has more detail than the equivalent shapefiles pack, or there's additional layers not included in the matching .shp pack. So it's always a good idea to grab both sets of data for any area you're seriously interested in, and check them both out thoroughly - cherrypick the best of the useable data from whichever seems best.

You'll find that the .osm's seem to consist of all the shapefiles muddled together on a single layer... However, all the individual shapes are carrying "metadata" - they have ID's, names, types, etc...

Most GIS programs, GM included, have a "metadata search" function which will allow you to search that whole jumbled layer of shapes and select only those labelled "residential road" for example, or "highway"... you can then select only that "class" of shape and isolate them from the rest by copying to a new layer...

Keep doing that and you can extract a whole bunch of useful layers from lakes & rivers, thru roads, vegetation areas, agricultural areas, etc, etc... with OSM it's all down to luck as far as coverage and quality is concerned, but most times you should be able to acquire some useable stuff, and save a considerable amount of time....


====== OSM Download Links ======

[[https://www.openstreetmap.org|openstreetmap.org]] homepage.\\
[[http://download.geofabrik.de/|Geofabrik]]\\
[[http://www.mapcruzin.com/free-world-country-arcgis-maps-shapefiles.htm|mapcruzin.com]]\\
[[http://extract.bbbike.org/|extract.bbbike.org]]