Digital Elevation Models & Mining History

Digital elevation models are expressions of the surface of a planet as data. These, and the software to view or process the data, have become freely available over recent years. The software ranges from easily usable online viewers to PC-based tools requiring intermediate levels of IT skill. This all makes for an interesting and useful resource for people interested in mining history, whether as a casual interest or as a more focused amateur historian. This article seeks to provide an introduction to the topic and to illustrate what is possible. A shortened version is being published in the Peak District Mines Historical Society (PDMHS) members’ newsletter. If you find this article interesting, you may be interested in joining and participating in membership activities, but we also have public activities when pandemics allow.

I am avoiding technical detail, and not providing a “how to” guide; there is an abundance of information available on the web which should suffice, although a mining history focused “how to” guide may be created for PDMHS members if there is interest.

A Brief Introduction to Digital Elevation Models (and Lidar)

I am using “digital elevation model” (hereafter: DEM) as a generic term as well as two more specific terms: “digital terrain models” (DTM) and “digital surface models” (DSM). A DSMs give the surface with things on it – trees, buildings, walls, etc – whereas DTMs, in UK parlance, show the ground level (in the USA, a DTM is augmented with information about surface features such as rivers). Someone with an interest in forest canopies would normally be interested in a DSM, whereas we mining history people are much more likely to be interested in what is beneath the vegetation, hence in DTMs.

DEMs are usually created using the data from flying aircraft, with a considerable amount of sophisticated data processing to get from the raw data to usable DEM data. Radar has been used, although Lidar (an acronym for “light detection and ranging”) is the most likely source of DEMs which we will use. Lidar and radar use similar techniques: bouncing light or radio waves off objects and measuring the pulses which arrive at the sensor to “see” objects. Lidar equipment is extremely expensive. A cheaper alternative is to use photogrammetry, which entails taking numerous overlapping photographs from an aircraft, and using software to work out what the surface elevations must be to cause the observed changes between adjacent photographs. Photogrammetry can be achieved from professional grade unmanned aerial vehicles (aka “drones”). See, for example the 3D model of Magpie Mine, with online viewer, created by Peak Drone Imaging. Some specialist providers also fly Lidar drones.

What are DEMs Useful For?

Before listing what we can use DEMs for, it is fair to mention that even the higher-resolution images are no substitute for an experienced archaeologist on the ground. Such a person will see things, feel them under their feet, draw inferences from changes in vegetation, and observe traces of mineral, etc. Unfortunately such people are in short supply. Aerial photographs may also be more revealing than images created from DTMs, especially high resolution black and white images, but also when the character of vegetation is shown, although the best resolution of freely available aerial photographs is generally not good enough to compete with the latest Lidar-based DTMs.

Fieldwork (preparation) from your desk chair: before venturing out, 2m or higher resolution DTMs, can be a useful way of “seeing” what is there. This kind of prospecting can be really useful when the terrain is difficult. Alternatively, there may be places of interest on private property with no access.

Seeing though undergrowth is a major advantage of DTMs, making visible what would otherwise remain fairly well hidden.

Validating grid references and sketch maps; there are abundant imprecise locations from before the days of GPS, when sometimes an approximate 100m grid reference was the best that could be achieved. This also applies to British Geological Survey maps, which are often based on very old surveys and locate veins quite inaccurately.

Visualising the 3D character of the landscape is one nice application, and one which works even with 50m spatial resolution. This can be either quite subtle shading on a flat map, to given an impression of shadow, or a more virtual-reality-like presentation. Contour lines are really in this category too.

What Data is Available?

There are several good sources of freely-downloadable data (but with some licence terms):

Ordnance Survey OS Terrain 50 is a DTM with a spatial resolution of 50m. It is not available as GeoTiff (see below), but in alternative grid-based formats and as contours.

NASA mapped almost the entire land surface of planet Earth in the Shuttle Radar Tomography Mission (SRTM) at 1 arc second spatial resolution, which is approximately 30m. You can browse maps online and download the data using the USGS Earth Explorer, and there is a SRTM downloader plugin for GQIS (see below).

The Environment Agency has surveyed many parts of England over the last 20 years, at a range of spatial resolutions between 2m and 0.25m. This work was particularly driven by a desire to model and predict flooding, so the smallest resolution surveys closely follow draining networks near large urban/suburban areas such as Sheffield. Since 2016 they have been undertaking the National Lidar Programme, which will complete its survey of the whole of England at 1m spatial resolution in 2021/2 (Lidar surveys are taken during the winter months while vegatation is largely dormant). The vertical accuracy is claimed to be below 0.15m and both DTM and DSM datasets are available. This is all available under an Open Government Licence. This is outstanding!

While the OS and NASA datasets are useful for visualising the physical geography, a spatial resolution of 1m or less is essential for picking up mining historical features other than large scale open-casting. Consequently, most of the rest of this article will look at the Environment Agency data, in particular the 1m National Lidar Programme data as it is available in nice square blocks, whereas the 0.5m and 0.25m surveys are quite limited in coverage and tightly follow streams and rivers.

Visualising DEMs

The simplest way of representing a DEM is as a square grid of altitudes, and this is the most likely way in which data is provided. The grid spacing, known as the spatial resolution, typically varies from around 100m down to 0.5m, or sometimes as small as 0.25m. A grid of elevations is also quite easy for computer programmes to process. The most favoured current format is GeoTiff, which is an extension of the Tag Image File Format (Tiff) to include data about the geospatial location which the data relates to. The simplest way of rendering a DEM for viewing is to have the elevation values correspond with different brightness values, from black (lowest elevation) to white (highest elevation), so GeoTiff is a neat solution. A bit of desk research will turn up lots of alternative file formats. Fortunately, most geospatial software can handle several of them.

Unfortunately, just opening one of these GeoTiff files on your PC will usually just give you what looks like a plain white image, but with the right software we can get something looking a little surreal or medical.
(All images are linked to a larger version)
We can do a bit better by creating a pseudo-colour image, where the elevations are mapped to a colour. The colour scheme can be semi-naturalistic like a traditional atlas, a simple grading from one colour to another via white, a rainbow or a garish scheme designed to highlight a particular elevation range. One really effective technique is to generate “hill shading” by  simulating light and shadow from an imaginary sun. If you visit the OS Terrain 50 web page, there is an interactive map where you can adjust the azimuth to alter the hill shading. Hillshade works well when overlaying a base map or aerial photography.

Grey-scale representation of a DTM. Monsal Head is at the top and Lathkill Dale at the bottom

Wye to Lathkill Pseudocolour DTM

Wye to Lathkill Hillshade Overlaying OpenStreetMap

Hill shading turns out to be quite effective at picking out even quite slight surface features with the higher resolution DEM data. When hill shading is used to augment a map, steep slopes can appear very dark, so people often use multi-directional hill shading, with three or more imaginary suns all shining at the same time. This sounds very unrealistic, but the images are pleasantly usable.

Some other ways of representing a DEM are as contours, which are not so easy to work with “as data” but super if you are map-making, and as the surfaces of 3D objects, which would be useful for creating photo-realistic images and “virtual reality” environments. This article will not look at 3D visualisations.

Visualising DEMs on the Web

My top recommendation is to use the Environment Agency Geomatics Team’s Lidar Composite viewer, but there are other services to explore via their Open Data Products page. This is a very interesting resource which includes clear information about Lidar, the accuracy of the data, and the way raw data is turned into a DTM or DSM. At the time of writing, the latest data which is included is the “2020 Composite”, but this will change as the National Lidar Programme progresses (link to catalogue of all completed and planned surveys).

Environment Agency Lidar Composite Viewer, with key controls annotated: Map layer, Base map, Spatial Resolution, and Placename Search.

At this point, I urge you to go and explore! The most important controls are the spatial resolution selection and the map layer. Once the Map layer control is opened, click on the little eye symbols to switch different effects on and off, or on the three dots to change the opacity of the layer. Experiment with hill shading, pseudo-colour, overlaying hill shade onto base maps (you will have to decrease the opacity of the hill shade), and looking at slope and aspect. The area shown in the image, above, is Grin Low, above Poole’s Cavern in Buxton, providing a dramatic illustration of the legacy of lime burning which is only partially evident as one walks around the woodland. The Thatch Marsh and Burbage Colliery area, not far to the West of Grin Low shows the causeways and pit locations quite clearly, as well as the pack horse hollow-way leading from the pits towards the lime kilns where much of the coal was consumed. The area and its history is thoroughly described in “Coal Mining near Buxton: Thatch Marsh, Orchard Common and Goyt’s Moss” by John Barnatt, in Mining History 19-2 (not currently online) which has several maps drawn from proper fieldwork, which make for an instructive comparison with Lidar armchair exploration.

An alternative viewer is the Environment Agency Survey Open Data Index Catalogue. This gives access to more datasets, including DSMs (although only in the 2017 Composite, not the 2019 version), but I don’t find it quite as pleasant to use.

For the More Adventurous – QGIS

QGIS (formerly Quantum GIS) is open source software with professional-level geographical information system (GIS) capabilities. Software with such power comes with challenges, and I would only recommend people who are confident IT users even taking a look. I hope those who are confident will be able to repeat the following examples, and are maybe motivated to learn more. There are a lot of resources explaining how to use QGIS on the web; just be aware that the latest version is QGIS 3, and many resources refer to the previous release. If you do download QGIS, choose the latest “long term release” version. I am going to use some more technical terms in this section, and will not explain them all, expecting readers can do their own web searches.

Before doing anything else, I recommend adding OpenStreetMap as a base map layer, if only so you know where you are! This is achieved by adding an “XYZ Tiles layer” (see this guide). The URL to add is “http://a.tile.openstreetmap.org/{z}/{x}/{y}.png” (without the quote marks, but with those curley brackets). There are lots of XYZ servers available – try a google search – but the Bing Maps satellite imagery service is a good companion to the OpenStreetMap: “http://ecn.t3.tiles.virtualearth.net/tiles/a{q}.jpeg?g=1“.

The starting point for all which follows is QGIS with a map in view and two panels – headed “Browser” and “Layers” – on the left hand side. If these panels are not visible, use menu View > Panels.

Getting Data for QGIS

I am only going to consider using Environment Agency data, for which there are two ways of using the DEMs in QGIS: downloading GeoTiff files, and accessing a WMS map tile server. Once set up, the WMS approach is quicker and easier but you get less control over how the DEM is rendered. Consequently, I suggest using the Defra Survey Data Download service. The workflow for using this site is generally: draw your area of interest on the map, click “Get Available Tiles”, then choose which dataset you want. Ignore the shapefile upload option; the icons for drawing your area are just beneath the upload grey box. This catalogue covers all the published datasets, with downloads being 5km x 5km squares. At 1m spatial resolution, that amounts to 5000 x 5000 = 25 million data points, so even these tiles are quite large files.

Once downloaded, you can just drag-and-drop the file into QGIS. If you do this, you will see that QGIS chooses a grey scale for each based on black = the lowest elevation and white = the highest elevation. This makes things look blocky if you load several 5km square tiles since each has a different min/max elevation. This can be fixed quite easily, but doesn’t matter for some uses (see below). I generally work with the OpenStreetMap and satellite imagery as the bottom layers and put DEM layers above (and the notes below assume this).

Playing with Hill Shading, Pseudo Colouring, and Overlaying Maps

For this section, we will work within a single 5km tile. I chose the SK16NE DTM from the 1m National Lidar Programme (aka DTM_1565) to work through what follows, which has Magpie Mine almost at the centre. The Magpie Mine is easily accessible and has been thoroughly surveyed and described, so it is a good place to see what Lidar data can (and cannot) reveal. Choose a place you know!

All of the following is achieved by double-clicking on the DTM layer in the “Layers” panel then choosing the “Symbology” option. When you open this up, it will not look exactly as below, because the default “Render type” is the rather boring “Singleband gray”. Change this to “Hillshade”. Also change “Zoomed in” to “Cubic”; if you don’t do this, the appearance when zoomed in looks a little bit like linen (this is an easy thing to forget). You must either “Apply” the changes or “OK” to close the properties window for the settings to take effect.

To begin with, “Multidirectional” will be un-ticked. In that state you can play around with the altitude and azimuth (compass direction) of the imaginary sun for which the shading is created. It will quickly become clear that different features are revealed for different angles, but that if you are interested in showing several features, there isn’t a good answer. This is where multidirectional hillshade comes in; it combines the effect of several simulated suns (3 for QGIS). Since the hillshade algorithm uses gradient and aspect to determine the shade of grey, it will work if you have several 5km tiles loaded.

To combine the hillshade with either the OpenStreetMap, change the “Blending mode” from “Normal” to “Multiply”. This merges the DTM and the map images so that the map appears to be draped over a 3D surface. I find this the best way to interpret features in relation to landscape features. A similar effect can be obtained with the Bing aerial photography layer, although if there are buildings and trees in the area of interest, it may be better to use a DSM instead. Combining the DTM and aerial photographs can often be very revealing since the photographs can show variations in vegetation or surface material in topographically indifferent ground, whereas the DTM reveals aspects which are indifferent in ground cover.

An alternative to hillshading is to set the “Render type” to “Singleband pseudocolor” (the TIFF images are “single band” because there is a single value for each pixel, which corresponds with the height, whereas colour photographs usually have three bands with separate red, green, and blue values per pixel). The magic button is “Classify“! There are lots of colour schemes to experiment with – change “Color ramp” – but most are horrible.

The Min/Max values can be useful to limit the colour range to the elevation range in a particular area of interest. A 10m range can nicely bring out spoil heaps and hollows. These values can also be used to force several 5km tiles to have the same colour scale, so to appear seamless.

Most of the other options will not be useful for general experimentation, but the brightness/contrast/saturation slides can be useful for preparing images for print.

Some tricks:

  • Right-click on the DTM layer and choose “Duplicate Layer”. Now set one of the two layers to be hillshade and one to be pseudocolour. This can really bring out relief.
  • Make your own multidirectional hillshade by duplicating the DTM layer 2 or more times and setting the azimuth separately on each layer. This can help to reveal features which are not showing nicely with the QGIS fixed azimuth angles.

Here is a composite image of Magpie Mine using the DTM mentioned above (so the buildings have been magically transported away). The spoil heap near the 1869 engine house and the reservoir are easily seen, as are the main veins outside the heavily re-modelled central area, but can you make out the four gin circles and the crushing circle? Gin/crushing circles are much easier to see on the ground than from a DTM when the terrain is quite flat. How about the covered flue from the square chimney to the long engine house, the slime ponds and dressing area, and the straight tramroad from Dirty Redsoil into the centre of the main site? In this case, adding the pseudocolour actually makes it harder to make out the more subtle features where the changes in slope and aspect are more significant than changes in elevation.

Composite of Magpie Mine comprising OpenStreetMap base map overlain by a multidirectional hillshade and a single band pseudocolor constrained to the altitude range of 315m to 325m. (Click to open larger image)

QGIS can also generate contour lines from DTMs; change the “Render type” again. The contour options are fairly self-explanatory and work well down to as low as 1m interval with the National Lidar Programme data. The “index contour” can be used to make every 5th, 10th, etc contour be styled differently. The “input downscaling” setting default of 4 usually works well; this smooths out the contour lines to make them less jittery as the limits of the spatial resolution (and to a lesser degree the vertical accuracy) come into play. Larger values make for smoother contours.

I find there isn’t a single magic setting which works for all cases; different sites and different exploratory questions indicate different settings, but there is enough variety and power in what QGIS provides that there is usually a satisfactory approach, within the limits of the data.

Creating Terrain Profiles

Since the DEM encodes height on a grid, it is possible to construct profiles. The standard QGIS plugin “Profile tool” is the easiest way of doing this and there are a few more advanced plugins available. The tool is quite self-explanatory, so here is an example without a “how to”. The area around Wass’ Level (also known as Moorhigh New Level) on Maury Rake is in an accessible grassland location just South of the old Midland Railway bed through Millers Dale. A visit to the site allows the spoil heaps and settling ponds to be easily seen along with the presumed collapsed level at around 245m elevation, at the edge of the woodland. The Lidar image reveals some interesting features up-slope which cannot be seen from the open area but the elevation helpfully reveals enough to motivate a bit of struggling through the undergrowth. Could the feature at 255m be evidence for the excavation from the surface of a winze down from Wass’ Level to the now-lost Upper Level (destroyed when the railway was build just above it)? The paper entitled “The Maury and Burfoot Mines, Taddington and Brushfield, Derbyshire” by John Barnatt & Chris Heathcote in Mining History 15-3 describes the site and its history.

Aside: the vertical line near the right is just an artefact of having used two DEM tiles which have been separately hill-shaded; the hillshade algorithm can’t work out what the slope/aspect is at the edge of a tile. This can be avoided by combining the tiles into a single layer in QGIS, which should be done for publication but I’ve left it in here as there is educational value in showing and explaining the issue.

The Area Around Wass’ Level (SK149730) on Maury Rake, Millers Dale.
(Click for larger image)

Some Oddments…

Once you have got to grips with the basics, there are numerous plugins and processing algorithms to experiment with. The Visibility Analysis plugin uses the DTM to determine those parts of a map which show what can and cannot be seen from a vantage point. There are algorithms which will calculate surface slope and aspect, or roughness (Processing Toolbox > Raster Terrain Analysis); these values can then me mapped to colours using the same technique as above. Mapping slope to pseudocolour does a better job of revealing gin circles than hillshade when the terrain is quite level.

Using a WMS (Web Map Service) is generally more convenient than adding 5km square Tiff tiles in QGIS but the available services provide a combined hillshaded and pseudocoloured image and only a DTM is currently available (things might change). The separate services for 2m and 1m DTM (etc) are:

  • https://environment.data.gov.uk/spatialdata/lidar-composite-digital-terrain-model-dtm-1m-2020/wms
  • https://environment.data.gov.uk/spatialdata/lidar-composite-digital-terrain-model-dtm-2m-2020/wms

The following URL may be added as a WMS to discover the extent of coverage without the DEM: https://environment.data.gov.uk/spatialdata/survey-index-files/wms . NB this will not work in a web browser, but the same source can be accessed using an online service via web browser.

The elevation data is available (as opposed to a hillshaded and pseudocoloured image), but only using the ArcGIS Image Server protocol. In QGIS this requires the use of the “ArcGIS ImageServer Connector” plugin, which works but the image redraws rather slowly. The URL to use is of the form: https://environment.data.gov.uk/image/rest/services/SURVEY/LIDAR_Composite_2m_DTM_2020_Elevation/ImageServer.

We can hope that, once the National Lidar Programme has concluded, the WMS provision will improve, along with the rather confusing array of data download, WMS, and online map viewers. Quite a few things changed during the drafting of this article.

Credits

OpenStreetMap base maps have been produced from the OSM raster map tile service. This is © OpenStreetMap contributors and used under the terms of the Open Database Licence: https://opendatacommons.org/licenses/odbl/.

Digital terrain models used for creating profiles and DEM visualisations are from Defra, used under the Open Government Licence v3.0: http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/

Review of CTC Tools – Don’t Buy from CTC Tools!

The prices look good but CTC tools is hopeless.

I ordered some tools and got a tracking code, with a shipping date of Nov 25th. The tracking said the local courier had not yet received the package. After several weeks I queried this and was told that the package had in fact been returned due to an inadequate address. They send a pdf of the package label. The address was correct. The courier (Hermes) must just have been lazy. OK this may not be CTC’s fault but their failure to inform me about the return IS.

They did give me a refund but had taken payment from paypal in USD, in spite of the website being priced in £ sterling, so the refund came out £6.66 short due to the exchange rate changing. They refused to make up the difference, citing a policy (who reads those) which said prices were given in sterling “as a service”.

So, six weeks after shipping, I have no tools and am £6.66 the poorer for it.

No doubt they will protest that none of this is their fault, but it is just cause for poor reputation; I could have spend a few pounds extra and received my tools in good time. I suggest you do the same!

Searching Ordnance Survey Names for Evidence of Mining History

This article describes some informal/experimental work in which various sources of data were processed to find traces of mining history in the names of places.

A few of the results from the place name search, displayed over an Open Street Map

The Geographic Area

My area of interest is broadly-speaking the Peak District and adjacent areas. For practical purposes, I have defined both a detailed boundary and a rectangular bounding box. The latter takes in all of the former, which is defined using Parish (and similar town/urban) boundaries. Since the boundary of Sheffield stretches into outlying areas which are of interest, this way of defining the boundary ends up including the city and Eastern areas. The same is not the case for Greater Manchester. Noteworthy cases where areas outside the Peak District are included are: the Churnet Valley and Alderley Edge.

The bounding box is (E-min, N-min, E-max, N-max): (381355, 340577, 445079, 402069)

Coordinates will either use fully-numerical eastings and northings to 1m, or use 100km letters (SK covers most of the area) and a variable number of digits.

Where source data is chunked according to 100km or 20km tiles, the small excursion of the boundary north of 400000 is neglected.

Data Sources and Pre-processing

All data sources are ultimately from the Ordnance Survey, but with some qualifications:

  • OS Open Map Local* for NGR squares SJ and SK, was used, limited to shapefile data files “NamedPlace” and “Road”.
  • OS Open Names* is provided in 20km tiles; CSV files with the following names were used: SJ 84, 86, and 88; SK 04, 06, 08, 24, 26, 28.
  • OS 1:50,000 Scale Gazetteer is no longer available from OS but I had a copy from 2009 on disk. This contains names which are not present in currently-available OS downloads. It was processed to extract entries occurring in the same area as used for Open Names (the data includes the tile designator).
  • The Visions of Britain “GB 1900 Gazetteer” (the abridged version) was initially limited to the area extent then an attempt was made to remove entries which are descriptive of features (e.g. “Old Lead Mines”) rather than being names. This is a somewhat subjective exercise. Proper names for lead mines are quite common in this gazetteer; these were left in, although the main intent of the activity is to find place names which had arisen from mining activity, rather than finding named mines.

* – obtained from the OS Open Data download area.

A final spatial filter was applied in all cases to limit the input data to the “detailed area”.

Search Terms

The range of possible name-parts of interest is split up, largely according to variation-spellings of the same root, but with some “misc” categories which contain various thematically-related words. The categories/terms are given below, with the category name in bold.

  • coal: words starting with “coal”, “cole”, or “collier”. Historically, charcoal was also referred to this way.
  • pit: names ending, or having a word ending, in “pit” or “pits”.
  • lead: words beginning “lead”, “led”, “lyd”, “lidgat” or “lidyat”.
  • mine: words starting “mine”.
  • cost: words starting “coars”, “cost”, or “coast”. These could be due to Old English names indicating a mining trial (see PDMHS Bulletin 7-6 pp 339-341).
  • mining misc: a word beginning with one of “rake” or “raike”, “delf” or “delph”, “gin” or “engine”, or “ochre”.
  • bole: a word containing either “bole” or “brenn”. The latter catches “brenner”, the operator of a bole.
  • smelting misc: terms other than “bole”, which signify smelting: “sm.lt” (where . is any letter), “cupola”, “pig”, or “slag”.
  • misc: catches names with words containing “jagger”, “belland”, “forge”, or beginning with “copper”, “bloomery”, “furnace”, or “furnes”.

While it is clear that these search terms will include obviously-erroneous names, these are not excluded from the results; given the status of this work as “for interest and as stimulus”, this feels appropriate.

Search results were further processed to try to remove unwanted multiplicity arising from either the same name appearing in several sources, and from linear features such as roads having multiple entries. This was done by checking for identical names within 500m. This sometimes fails, as different sources may differ in capitalisation or make composite words such as “Bolehill”.

Results

Three map exports are available:

Three maps were created due to difficulties combining all the options using the Leaflet technology which underpins the web maps. For maps which do not immediately show the names, simply place the mouse cursor over a point.

Some Observations

Some of the search terms will give less-obvious false “hits”, and so all should be taken with some caution; place name specialists rely on written records from far earlier than those used here. Some example false friends are: “coal” might originate from “cold”, although we might reason that local geology makes the former more likely; “lydgate” is often though to derive from Old English “hlid-geat”, a swing gate. I will consider the case of “lyd” in a later post.

Almost all of the hits for “mine” are named mines, with a few exceptions such as “Miner’s Cottage”, “Miners Hill”, or “Miners’ Standard PH”.

Licences and Acknowledgements

This work includes data from:

The maps were created in QGIS using the QGIS2Web plugin and the Leaflet web mapping open source toolkit (see links at the bottom of the map screens).

The maps use data from Open Street Map and the National Library of Scotland, under an open licence for non-commercial use.

This blog post and the derived data created by me are licensed under Creative Commons BY-SA.

 

Adding British Geological Survey (BGS) WMS Layers to geojson.io

This took a little while to do; there doesn’t seem to be anywhere explaining how to add WMS layers and controls in simple terms… so here it is for anyone else. Pieced together from various places, the leaflet documentation, and some guesswork, here is how to add WMS layers for BGS solid and superficial deposit geology and linear features to a geojson.io view. The layer opacity has been set to 0.5 to allow the base map to be seen.

You will need to access the developers’ console (F12 on Firefox).

It will look like this:

Enter the following commands where the “>>” is:

var solid = L.tileLayer.wms("https://map.bgs.ac.uk/arcgis/services/BGS_Detailed_Geology/MapServer/WMSServer", {layers: 'BGS.50k.Bedrock', format: 'image/png', version: '1.3.0'});
var drift = L.tileLayer.wms("https://map.bgs.ac.uk/arcgis/services/BGS_Detailed_Geology/MapServer/WMSServer", {layers: 'BGS.50k.Superficial.deposits', format: 'image/png', version: '1.3.0'});
var linear = L.tileLayer.wms("https://map.bgs.ac.uk/arcgis/services/BGS_Detailed_Geology/MapServer/WMSServer", {layers: 'BGS.50k.Linear.features', format: 'image/png', version: '1.3.0'});
var layer_control = {"Solid": solid, "Drift": drift, "Linear Features": linear};
solid.setOpacity(0.5);
drift.setOpacity(0.5);
linear.setOpacity(0.5);
window.api.map.addLayer(drift);
window.api.map.addLayer(solid);
window.api.map.addLayer(linear);
L.control.layers(null, layer_control).addTo(window.api.map);

I find I have to leave the console now.

One thing to note is that the BGS WMS will not return an image if you are very “zoomed out”. Zoom in until the scale bar shows 500m or 1km.

 

Other BGS WMS layers to add:

var artificial = L.tileLayer.wms("https://map.bgs.ac.uk/arcgis/services/BGS_Detailed_Geology/MapServer/WMSServer", {layers: 'BGS.50k.Artificial.ground', format: 'image/png', version: '1.3.0'});
var movement = L.tileLayer.wms("https://map.bgs.ac.uk/arcgis/services/BGS_Detailed_Geology/MapServer/WMSServer", {layers: 'BGS.50k.Mass.movement', format: 'image/png', version: '1.3.0'});

Spreadsheet to Assist Melodeon Tuning

Here is a simple spreadsheet to assist with tuning two voiced melodeons: Melodeon Tuning Spreadsheet (Excel).

It has been set up for, and contains working data from, my D/G Hohner Pokerwork retune. I created it for two main purposes: 1) to plot the existing tuning; 2) to calculate the amount of de-tune for a Viennese tuning with an equal beat all the way up the scale (also known as Dedic tuning from Ian Dedic). I also used it to measure the tuning on a fairly new Serenellini melodeon with drier tuning, to get a better idea of what the tuning of a newish mid-range melodeon is like.

Some notes on using the spreadsheet for those who don’t want to find out by fiddling:

  • leave the “Notes Lookup” sheet alone; it contains the look-up from the piano key numbers to the note name and frequency.
  • On the sheets “G Row” and “D Row”:
    • change the entries in the “Piano Key” column if your button layout or keys are different; “Note Name” and “Concert Freq” change automatically.
    • If you want to find the tuning to achieve a constant beat frequency, alter the values in cells F3 and G3. I have chosen 4Hz (beats per second), which is in the tremolo range, and set this as -2Hz/+2Hz for Viennese tuning.
    • Alternatively… it is common for accordions to be tuning with tapering amounts of reed de-tuning going up the scale, with the beat increasing somewhat. Entering values into row K will compute the beat frequency in row L.
  • The sheets “… Measurements” should automatically populate the left-most columns. This should be mostly self-explanatory. I made columns to record measurements with the reed in the box and on the reed block on a tuning table, computing the difference (“delta”). Since tuning on the table is far easier than in the box, I use this to estimate a correction to my target tuning when using the table.
    • The “block hole number” is just a reference to my numbering of the holes on the chord block… its very confusing working out which reed is which!

 

I you are interested in melodeon tunings, or in the practicalities of tuning a melodeon, the melnet forums are essential reading.

Chinley Churn Underground Quarrying (Mines)

From a visit with Beth Knight earlier this year, here are the locations of some underground workings on Chinley Churn. This post is not a recommendation to visit; the workings contain dangerous loose rock and areas liable to collapse.

Point Name WGS84 Longitude WGS84 Latitude OS Grid Ref. 10m Altitude
1 -1.94595 53.34832 SK 0369 8349 407.5 m
2 -1.94561 53.34882 SK 0371 8355 416.6 m
3 -1.94563 53.34927 SK 0371 8360 415.2 m
4 -1.94546 53.34964 SK 0372 8364 423.6 m
5 -1.94625 53.35133 SK 0367 8382 422.1 m

Photographs showing the entrances, in the same order as the points above:

Chinley Churn Entrance 1

 

Chinley Churn Entrance 2

 

Chinley Churn Entrance 3

 

Chinley Churn Entrance 4

 

Chinley Churn Entrance 5

Decorative and Irregular PCBs in Eagle (with Inkscape)

Eagle does not lend itself to unusual, possibly artistic, board outlines copper tracks, silk screen etc. There are a number of Eagle ULPs (user language programs) that can import DXF files, and yet more which import bit maps as a mass of polygons, but I struggled to get any of them to work, or simply balked at the number of fiddly steps required. Basically: it was just too hard or error-prone. Then I stumbled across an article by Shabaz Yousaf complaining about the same problem but pointing to HPGL (Hewlett Packard Graphics Language) as a solution. He provides a C program to do the conversion but: a) I wanted to make a ULP for ease of use and b) I wanted to add some configurability at run-time. This article outlines the approach I took. I am assuming that images are created from the outset in Inkscape but Inkscape can import many formats, so it should be possible to use it on existing images, and any graphics software supporting HPGL output should work (HPGL appears to be very simple, but it may be the case that other software creates HPGL in a way my code does not recognise… so your mileage may vary!).

The User Language Program – hpgl2scr

The latest version is in my GitHub ULP repository (direct link to ULP). Please report bugs via GitHub if possible. Also any friendly guidance; this is my first ULP. This is available under an open source licence.

hpgl2scr

It should be fairly self-explanatory to use after a bit of messing about. Note that the wire (or polygon boundary) width auto-changes depending on the layer that is chosen. Adding more layers is easy by editing the ULP.

Some notes on usage:

  • HPGL files can contain several separate lines (think a pen moving about in an old-school plotter). Hpgl2scr creates one or more paths of wires or polygon boundaries in the specified layer.
  • A .scr file is created in the same folder as the .hpgl input. This file may be used to repeat the import.
  • In normal use choose the Wire output type. When using polygon output, note that setting the pour type = cutout may be used after import for subtractive effects (but only for copper layers).
  • All units are mm.
  • The imported image can be positioned in four ways:
  1. Aligned to the x and y axes with all wires at positive coordinate values, i.e. in the correct position for a board outline.
  2. Centred at a chosen position.
  3. Off-board (on the opposite side of the origin compared to option 1). This is the best place to group-select the shape and manually move it into place.
  4. Absolute positioning; (0,0) of the original graphic maps to (0,0) on the board. Absolute positioning may be useful if several layers are used in an image and each converted to separate HPGL files for separate importing. Beware that the origin for Inkscape is top left, whereas for boards in Eagle it is bottom left.
  • The Scale to Fit Box setting will rescale the image (preserving aspect ratio) so that it fits within a box of the specified size (in mm). This is quite useful in that it allows Inkscape to be used without worrying about absolute size, but the final board can be within the limits for PCB cost etc.

Using Copper Layers

As imported, copper layers do not form part of the circuit as far as Eagle is concerned. You cannot just connect to them and use them as pre-routed tracks (OK, so you can leave partially-routed tracks but dangling airwires just feels so untidy, and I always check for unrouted airwires so this would just be another cause of missed mistakes). The solution to this problem is to  set the name of the imported wire to be the same name as an existing net. Use the “Name” tool to find the name of the airwire you are interested in, and the Name tool again to change the name of the imported wire to match it. Once done, an airwire should appear from your imported wire. The you can now manually route a trace to the imported wire and it connects nicely.

Note: if you now do a “rip-up all”, your imported wire gets ripped up!

Hints and Tips for Inkscape

I recommend setting document properties to give mm as the default units (File > Document Properties).

inkscape_mmNote that the inkscape image elements MUST be saved as paths (vectors) to export to HPGL. Use the Path menu: Path > Object to Path. If your image does not appear in inkscape check this first!

Simply “save as”, selecting the HP Graphics Language file type.

inkscape_hpgl
Pen number and flatness are irrelevant. Other parameters should be as shown. Make sure resolution is 1016 dpi if the absolute size matters.

The thickness of the lines in inkscape is irrelevant; the HPGL just follows the path.

Being Ambitious

If you want to be ambitious and to compose several different elements (board outline, various bits of copper and silk) in a single inkscape image you will end up using layers in inkscape. Use a separate inkscape layer whenever the HPGL import parameters will differ (e.g. Eagle layer, wire vs polygon, wire thickness). Make all inkscape layers but one be invisible and export to HPGL, ensuring that the “Plot invisible layers” checkbox is not checked (see the “HPGL Output” screenshot above).

The import will probably be best done with the “absolute” positioning. It would be possible to import “off-board” and to move the elements into place, but positioning will not be easy to replicate in eagle. The trickiness arises because Inkscape uses the top left as origin (0,0), whereas Eagle uses the bottom left. This is easily fixed by checking the “Mirror Y-axis” option on export. Make sure that the image is right down in the bottom left corner so that ends up in roughly the right place in Eagle.

An Example

Here is a quickly executed (and slightly wonky) LED star for a “magic wand”. This followed the approach mentioned above of using 2 layers in inkscape. The inner and outer stars were positioned and sized using typed-in values in inkscape and imported one layer at a time with absolute positioning. The imported copper wire was re-named to GND (I had already named the net in the schematic).

wand

It look about 20 minutes, including some time forgetting most of the hints I’ve given!

Xmega DFU Bootloader – Changing the Sensing Pin

The DFU bootloaders that Atmel supplies, as Intel HEX files, for its Xmegas have pre-determined pins used to sense whether or not to skip the bootloader and start the application, or to register the device to the host PC via USB. These are slightly hidden on the Atmel website and the source code requires IAR to compile it (and an easy port to AVR-GCC is thought to be impossible on size grounds). A number of threads on AVRFreaks have discussed this matter, including: “Building bootloader for ATXMega16A4U from avr1916.zip” and “Where’s the DFU bootloader?“. I didn’t fancy installing IAR, but was intrigued by comments about dis-assembling the HEX files and tweaking bytes. The process was not explained, but in spite of starting off totally “in the dark”, an evening of fumbling and fiddling got me through to a result that worked. This post explains the process, in case anyone else wants to do the same. It is also my notes-to-self for when I forget!

Background

The Atmel application note AVR1916 has essential background info and the associated zip file contains the Intel HEX files for different xmega chips. The zip file also contains source code but: a) this needs IAR (see intro); b) it is deeply hard to follow!

Atmel AVR1916: USB DFU Boot Loader for Atmel XMEGA: pdf zip

The zip file is not so easy to find by google/bing; you pretty much have to go to the Atmel site, find the xmega section, go to documents, and select “application notes” from the drop-down.

Tools

There are probably lots of options, but I used:

  • avr-objdump, which is part of the avr-gcc toolchain. I used a recent toolchain download from the Atmel site. If you use Atmel Studio, look in {install directory}\Atmel Toolchain\AVR8 GCC\Native\3.4.1061\avr8-gnu-toolchain\bin.
  • Tiny Hexer by Markus Stephany. This is available from numerous places on the web, although not from the author’s original site mirkes.de .

Procedure

In what follows, I am dealing with an ATxmega32A4U. The method should translate to other chips very easily. I am ONLY concerned with changing which pin is sensed on a hardware reset to determine whether to start the application or wait for USB communication of a new application.

From a command window, execute:

avr-objdump -m avr -D file.hex >> disassembly.txt

That looks somewhat daunting to a non-assembler hobbyist person like me but it turns out not to be quite as bad as all that. The key thing to note is that we’re looking for references to a single I/O pin. Which pin it is may be found in AVR1916.

Default pin configuration table from AVR1916.
Default boot-sensing pin configuration table from AVR1916.

So, for my target it is Port C, pin 3. Consulting the xmega manual…

Base IO memory map addresses for the GPIO ports.
Base IO memory map addresses for the GPIO ports.
IO port memory map offsets (apply to all ports, simply add to the base address for the port to get the address of the appropriate register)
IO port memory map offsets (apply to all ports, simply add to the base address for the port to get the address of the appropriate register)

So, setting the direction for port C will address 0x0640 (offset is 0) and PIN3CTRL will be at 0x0653 (offset is 0x13).

To cut a long story short, the important lines in the disassembly are:

    800a:    f0 92 40 06     sts    0x0640, r15
...
    8010:    00 93 53 06     sts    0x0653, r16
...
    801c:    00 91 48 06     lds    r16, 0x0648
    8020:    03 ff           sbrs    r16, 3
...
    80de:    f0 92 53 06     sts    0x0653, r15

As an exercise, work out which registers are being accessed. The STS and LDS commands are direct store and load instructions. SBRS is “skip if bit in register is set”. In this case we can see bit 3 is being used, which is as it should be, and the value in register 16 is the IN value for the port.

Suppose we want to change the sensing pin to be Port D pin4 (physically pin 24 on a 44 pin xmega). This indicates a base address of 0x0660 and changing the instructions to:

STS  0x0660, R15
...
STS  0x0674, R16
...
LDS  R16, 0x0668
SBRS R16, 4
...
STS  0x0674, R15

Open Tiny Hexer and import the Intel Hex file appropriate to your chip from the AVR1916 zip file. Now it is just a case of changing the hex representation of the appropriate bytes. Note that the byte order of 0x0640 is reversed in the disassembly and in the HEX file. Here are some screenshots of Tiny Hexer after changing the required bytes:

tinyhexer1

Now export from Tiny Hexer as Intel HEX, saving with a new name (!).

At this point, it is a good idea to run avr-objdump again, on the new HEX file, to make sure that it still makes sense to an AVR. I also did a file-diff on the new vs original disassembled files. The results are as follows, although there is also a block of FF that got inserted between code blocks. This has no effect, and simply reflects the fact that Tiny Hexer exported a continuous block rather than two blocks with a space in between.

Compare PC3 PD4 a

Compare PC3 PD4 b

Looks like I got it right! Only the instructions I wanted to change got changed, and the flash memory addresses came out correct for a bootloader.

All that remains is to program this onto the chip with your programmer of choice (I use the Olimex AVR-ISP-MK2 clone since my Dragon utterly fails to even get the device ID 95% of the time), making sure the BOOTRST flag is correctly set if it is a new chip. Erase the chip, program and go!

A Cheat

If you can find two HEX DFU bootloader files for the same target chip, but with different sensing pins, a quick thing to do is to disassemble both and do a diff. I bet they will be from the same basic source code, and will quickly reveal which instructions need changing. Confession: that is how I got started on this problem.

Uploading Using DFU

To be honest, I think FLIP is a bit rubbish, and using the command line batchisp.exe is a pain in the arse. I’ve found dfu-programmer to be much nicer to play with (although the 0.7.1 version does contain a bug that fails to account for the xmega bootloader living in separate flash and so reduces the available flash for applications; this is fixed in 0.7.2 but is not released AFAIK at the time of writing).

I use dfu-programmer from Atmel Studio as an “external tool” via a small “bat file” command script.

In Atmel studio, point the external tool command at the .bat file and add the following into the “arguments” slot:

"$(ProjectDir)Debug\$(ItemFileName).hex" atxmega32a4u

To work, you do need to select the project file with the same name as the hex file before invoking the tool.

My .bat file just contains:

@echo off
REM supply the hex file location as 1st argument e.g. "dfu-programmer script argument blink.hex",
REM supply the target device as the 2nd arg e.g. atxmega32a4u

ECHO Erase, flash %1 and application launch.
ECHO Device = %2
ECHO ==========================================================
ECHO.

SET PATH=%PATH%;C:\Users\Adam\Desktop\dfu-programmer
dfu-programmer %2 erase --force
dfu-programmer %2 flash %1 --debug 2
dfu-programmer %2 launch --no-reset
ECHO -----------------------------------------------------------

Further Adventure?

I have not done this, but see this line:

    800e:    08 e1           ldi    r16, 0x18    ; 24

That is setting the value of register 16, which is immediately stored to PIN3CTRL in the original code. A quick look at the datasheet shows that, as expected, this is setting a pull-up on Port C pin 3. Fancy changing that?

… Or maybe change the SBRS to SBRC ?

Decoding microKorg Patch Files (.prg) to PlainText

MicroKorg “prg” files files are downloaded from the microKorg by the Korg Sound Editor software, and are often posted online (e.g. the microKorg sound bank). But sometimes it is nice to be able to see what the patch is, or post it online in a plain text form. Plain text patches are also a handy way of documenting your own patches in a nice durable format: on paper. Also, if you want to share a patch, posting a readable version of the patch is a nice thing to do, and ideally without having to work your way through the whole matrix, transcribing the settings. Transcribing settings is boring, error-prone…

I have written a small program to convert “.prg” files for the microKorg to a nice readable format. So, if you have Sound Editor working, you can just download your custom patches, locate the place where the .prg files are stored, and away you go. Similarly, it is really quick to see what the Korg sound bank patches (or whatever) are doing at a glance, which I find is a good way of working out the tricks…

e.g. If you drop the “atmos” patch from the microKorg sound bank, you get:

atmos
 [Voice]: Synth , Single , Poly , -- , --
 [Pitch]: 0 , 0 , 0 , 2 , 0
 [Osc 1]: DWGS , 0 , 48 , -- , --
 [Osc 2]: Saw , Sync , -12 , 0 , --
 [Mix Levels]: 127 , 91 , 0 , -- , --
 [Filter]: -12dB LP , 101 , 50 , 0 , 0
 [Filter EG]: 0 , 64 , 127 , 0 , yes
 [Amp]: 127 , cnt , on , 0 , --
 [Amp EG]: 0 , 79 , 0 , 53 , yes
 [LFO 1]: Triangle , Off , off , 59 , --
 [LFO 2]: Sine , Off , off , 70 , --
 [Patch 1]: LFO1 , Pitch , 10 , -- , --
 [Patch 2]: Mod Wheel , Cutoff , -63 , -- , --
 [Patch 3]: LFO1 , Cutoff , 0 , -- , --
 [Patch 4]: LFO2 , Cutoff , 0 , -- , --
 [Mod FX]: Ensemble , 20 , 56 , -- , --
 [Delay]: Stero , off , 80 , 74 , --
 [EQ]: 320Hz , 0 , 6kHz , 0 , --
 [Areggio A]: 120 , 1/16 , 100% , Up , 1
 [Arpeggio B]: off , 0 , off , 8 , Both
 [Arp. Pattern]: yes,yes,yes,yes,yes,yes,yes,yes
 [Arp. State]: off

It only works for single-patch files right now, but is can be used for single and layered patches and vocoder patches. On most web browsers (recent versions of Firefox, Chrome, Internet Explorer), you should be able to drag-and-drop a .prg file. Otherwise you have to use the old-school “Browse” button. There is more technical information on the decoder page, and a link to report bugs or make suggestions (although you can do that as comments to this post if you do not wish to create a GitHub account).

Orchard Common and Drystone Edge Geological Field Trip Notes

The following are field notes from a visit to the valley just to the W of Drystone Edge to inspect the Upper Namurian, especially the Ringinglow coal outcrop and associated seatearth and marine shales (contorted beds). The locations were logged using a Garmin GPS60, and may be downloaded in GPX format. The points are named in the format “ORC %N”, and referred to below.

ORC 001 –  SK 02262 68251 – Roadside parking.

Proceed N up the road to Readyleech Green. Just past the house, the road swings NW, following a fault with downthrow on the SW side. To the right is the Upper Namurian Chatsworth Grit (CG), while the higher ground to the W of the road (Knotbury Common) is Westphalian (Coal Measures) Woodhead Hill Rock (WH). The right of way track around the S side of Knotbury Common approximately follows the lower boundary of the WH.

ORC 002 – SK 02198 69014 –  Shaft Hummock. This is in the shales between the Chatsworth Grit (CG) and Rough Rock (RR). The coal sits just above the CG.

Section through Drystone Edge and Orchard Common
Section through Drystone Edge and Orchard Common looking North

At this point you are on Open Access Land. Descend to the stream, which is easy enough if you follow the fence approx Southwards on its stream-side.

ORC 003 – SK 02170 68842 – Good clean exposure just above the stream bed of  CG followed by seatearth then coal. There may be a detectable marine band here but I didn’t look. The seatearth shows interesting patterning. There is also some evidence of fossilised vegetation.

ORC 004 – SK 02330 69112 – Contorted Bed.

ORC 005 – SK 02319 69450 – Quarry in Rough Rock. Look at the texture, grain size, bedding etc to compare with CG on Drystone Edge (ORC 009). The top of the RR is the top of the Namurian; a band of marine shale with distinctive fossils marks the bottom of the Westphalian (Coal Measures).

ORC 006 – SK 02598 69608 – Contorted Beds.

ORC 007 – SK 02680 69725 – Some nice pieces of loose coal (apparently not in situ, but 5-10cm longest dimension).

ORC 008 – SK 02781 69373 – CG boulders showing Karstic features.

ORC 009 – SK 02698 69152 – Scarp exposure of CG on Drystone Edge. Look NW towards the RR quarry visited earlier and consider the section (above), which follows the same line.


A 5km x 5km square of the solid geology with a lower left corner at SK 000 650 may be obtained using the British Geological Survey map server using the link https://map.bgs.ac.uk/arcgis/services/BGS_Detailed_Geology/MapServer/WMSServer?REQUEST=GetMap&VERSION=1.3.0&LAYERS=BGS.50k.Bedrock&STYLES=default&FORMAT=image/gif&CRS=EPSG:27700&BBOX=400000,365000,405000,370000&WIDTH=500&HEIGHT=500

These are personal notes, shared for whoever may find them useful. I am not a qualified geologist; more details may be found in the Geologist Association Guide No. 26, by F. Wolverson Cope. I used this book to visit the site, although the account above contains additional sites, and more precise location for some of Cope’s sites.

Look up: upper carboniferous cyclothems.