WORKING NOTES FOR OUR DRUPAL 7 IMPLEMENTATION - MAY CHANGE IN DRUPAL 8/9
Parish Boundary Mapping
Currently GENUKI is in course of implementing parish boundary mapping on many of its key place pages. In England and Wales this typically involves mapping according to Kain and Oliver ("K&O") parish maps, in Scotland a different map set of indeterminate origin is in use (seemingly dating to around the 1890s, but not really that accurate, although some some effort has been since been made to correct these, e.g. Caithness), whilst Ireland remains a separate case (although currently parish boundaries based on Openstreetmap data from townlands.ie are being incorporated). There is currently no mapping activity ongoing in the Channel Islands (or Isle of Man?).
This document tries to collect together relevant mapping knowledge and experience in the areas of Scotland, England & Wales, and Ireland - the author has not done any work on mapping in Wales, although in Wales the common experience of K&O mapping in England should apply - indeed Wales is already fully covered by draft map outlines.
In the absence of a current agreed GENUKI-wide strategy on mapping, the assumption in this document is that we are creating "parish maps" or "parish outlines" for those (and only those) place pages in GENUKI which relate to a formally recognised ecclesiastical or civil parish (for England this document does not specifically aim to address non-parish place pages such as those for towns, chapels, chapelries, etc., and e.g. in Scotland "quoad sacra" or similar parishes.)
In this context, GENUKI's parish mapping data resides in the relevant gazetteer pages, where it is held in WKT (Well Known Text) format, containing longitude and latitude data either in point and/or polygon formats. In most cases in the UK, the longitude will be a negative number (i.e. west of Greenwich), whilst the latitude will always be a positive number (i.e. north of the equator).
Relevant Mapping Formats
WKT - (Well Known Text)
In GENUKI, we use WKT format, and all locations are held in lat/lon format, although at times they may be initially provided using other formats, such as Ordnance Survey National Grid References (NGR), and subsequently saved as lat/lon. In GENUKI lat/lon in WKT is usually provided with the longitude as the first of a pair of numbers, separated by a space. Coordinate number pairs are separated by a comma followed by a space. Sets of multiple coordinate pairs are enclosed within round brackets " ( )".
KML - Keyhole Markup Language
Keyhole Markup Language is an XML notation for expressing geographic annotation and visualization within Internet-based, two-dimensional maps and three-dimensional Earth browsers. KML was developed for use with Google Earth, which was originally named Keyhole Earth Viewer.
Providing Mapping data for a GENUKI Place Page
As mentioned above, initially we are creating "parish maps" or "parish outlines" for those (and only those) place pages in GENUKI which relate to a formally recognised ecclesiastical or civil parish. Non-parish place pages are not specifically considered here, although other features may be considered in the future, such as for example, Townlands and Baronies in Ireland.
To do this we actually provide data on the associated Gazetteer node (thus each place fo which we wish to provide a map boundary requires a suitable gazetteer entry).
Creating a Parish Polygon
It is important to bear in mind is that OpenStreetMap (OSM) shows current (modern-day) boundaries, whilst in GENUKI we are normally using older ones, without specifying any date. That is one of the problems with our current approach. So for now we should adopt the policy of drawing boundaries around the area covered by a GENUKI place page. As displayed in GENUKI, OpenStreetMap (OSM) usually shows current (modern-day) civil parish boundaries.
Typically it is best try to follow 19th C boundaries where possible. It is found that particularly in rural areas the administrative boundaries have not changed over time, so OSM lines are (usually) still in the same place as they were in the 19th C.
To create a new parish outline, open the relevant Gazetteer node for editing.
As a starting point (for England), select the K&O map set then enter full-screen mode, click the "Add Polygon" icon and follow over the lines on a K&O map (for which we don't seem to have a date), which seem to be mid 19th C drawn on a mid 20th C map. The fine detail isn't there though as they use a 1" map and don't always get the lines in the right place. Close the polygon outline to complete it, then exit full-screen mode and click the Save button on the gazetteer edit screen. (K&O maps are only visible up to Zoom Level ?, no map tiles exists above that level.)
So to get the finer detail switch to OSM and where the modern boundaries were close by, adjust the lines to follow them. However, often the modern boundaries are quite different, especially in built up areas. For OSM higher Zoom Levels can be used , up to Zoom Level ?, at which point they also no longer appear.) However higher zoom levels can still be used to edit the more dense collections of adjacent points , with care.
The old NLS maps (NLS1900) if used as a base layer give a lot more information that can be used in problem areas, but not in all cases. Before using zoom in it is based on 1" maps around 1890 but as zoom is increased at some point it switches to 2.5" OS from around 1910. So where boundaries changed between 1890 & 1910 there may be differences according to zoom level.
In general the 1910 maps have the best detail needed to get exact boundaries, but it is not clear whether water courses have changed. The problems remaining are where boundaries changed between 1850 and 1890, so use K&O there. But if a water course is nearby, typically use that as it is very likely the old boundary followed it. Similarly for field boundaries.
In NLS 1900 it is convenient to follow the indicated boundary (usually a string of dots, although that is really a map-makers convenience to avoid overlapping the adjacent field boundaries for visibility). But this seems to be the most effective approach, to assist later alignment of the boundaries of adjacent parishes. The dots are actually next to the actual boundary (often a field boundary) as otherwise they could not be seen, and with some water courses they ignore some of the bends, which may change every year.)
Ideally what we would wish to have in the future is a range of gazetteer entries showing different boundaries according to time, which we could then use to show how administrative or other boundaries changed over time.
As an alternative, it is possible to obtain current (modern-day) boundaries, go to "Maps" on a place page and select the GeoHack link. Then near the bottom of the first table on the right find MaPit and click 'Containing areas' where it is possible (in theory at least) get all the modern ones in WKT. This is data that may be cut and pasted into a gazetteer entry. But note that GeoHack's "WKT format" download link DOES NOT provide a valid WKT text file, instead it is better to use the KML link as a data source for subsequent work (see below).
Of course these modern boundaries tend not to be the areas that we really want, especially in urban areas, however this can be a useful source for data to get a parish outline started, if only for later editing, or even to gather data for a spreadsheet.
Editing a Parish Polygon
To edit an existing polygon, open the relevant Gazetteer node for editing.
Select the desired map set then enter full-screen mode, click the "Edit Layers" icon and make the desired changes. When completed, click the small "Save" link over the map, then exit full-screen mode and click the Save button on the gazetteer edit screen.
Coordination and Harmonisation of Parish Boundaries
Once a number of contiguous parish polygons have been created, it is then necessary to resolve errors along shared boundaries.
Traditionally this is done using an iterative, visual, and largely trial-and-error process to reduce these errors sufficiently so as to remove obvious occurrences of gaps (lack of coverage) and slivers (overlaps of coverage).
Short-Cut Solutions
To obtain current (modern-day) boundaries, go to "Maps" on a place page and select the GeoHack link. Then near the bottom of the first table on the right find MaPit and click 'Containing areas' where it is possible (in theory at least) get all the modern ones in WKT. This is data that may be cut and pasted into a gazetteer entry. But note that GeoHack's "WKT format" download link DOES NOT provide a valid WKT text file, it is better to use the KML link as the data source (see below).
This could be a useful source for data to get a parish outline started, if only for later editing, or to even gather data for a spreadsheet.
Using KML
Click the KML link, which will download the KML data as a KML file. Then upload it to https://mygeodata.cloud/converter/kml-to-wkt which can be used to produce WKT. However, for free use the converter has a limit of 3 data sets per month, and also an overall data quantity limit.
The KML file is in content a text file which can however be saved as or renamed with a txt file extension and then locally edited using a few simple global edits (although better perhaps to load into an Excel spreadsheet (less error-prone), then export as a csv file). Save this suitably edited text file which can then be cut and pasted into the gazetteer entry in a suitable form, wrapping the Point and Polygon in GEOMETRYCOLLECTION (). e.g.
GEOMETRYCOLLECTION (POINT (1.459269 52.348472), POLYGON ((1.488218 52.347295, 1.486158 52.34719, 1.48118 52.346404, 1.480064 52.346876, 1.471739 52.349497, 1.467791 52.353377, 1.453371 52.356365, 1.446848 52.35951, 1.449766 52.361187, 1.442728 52.36276, 1.433544 52.366324, 1.422472 52.368368, 1.420584 52.367215, 1.415434 52.364699, 1.409254 52.363756, 1.405735 52.36082, 1.413374 52.35647, 1.421356 52.352066, 1.430111 52.347295, 1.426764 52.344516, 1.437492 52.339325, 1.440411 52.338276, 1.445303 52.339639, 1.445732 52.338905, 1.450882 52.337699, 1.450968 52.335759, 1.455259 52.334552, 1.456976 52.336545, 1.459208 52.336021, 1.470022 52.337332, 1.47294 52.33922, 1.477747 52.339639, 1.479893 52.337489, 1.483755 52.335601, 1.485043 52.336755, 1.484871 52.337699, 1.489677 52.339902, 1.48839 52.341632, 1.489077 52.342261, 1.488218 52.347295)))
(Surprisingly, LibreOffice Calc does not currently appear to have a suitable facility for handling XML imports, which was at one time available experimentally, the "XML source" option seems to have been disabled, and there now seem to be no options available to activate it. Web advice on this topic is ambiguous.)
Experiments so far in SFK and RUT have seemed promising. However currently information derived in this way is too detailed (too many points) to be edited in the current configuration of our GENUKI mapping tools (since resolved by increasing the available map zoom level described above).
Neither approach is really fast or simple enough but shows that it is possible, and with some work a faster system could be achieved. Perhaps a batch-based facility on the server, although as this mapping effort is in effect (for now at least) possibly really only a one-time only activity?
England and Wales - Kain and Oliver (K&O) Maps
The K&O maps as provided in the GENUKI editor are fairly self-explanatory. It is helpful however to have also a copy of the associated data spreadsheet which lists all the places addressed.
The shorthand codes which relate to the Excel columns are: the unique identifier for each record entry (REC), which combines a numerical identifier indicating the position each place is listed in its unique county and a unique number for each county (arranged in alphabetical order for England followed by Wales); a simplified version of the same identifier (NUM); the name of the area associated with that unique identifier (PLA); a term to indicate the category of place (CAT), whether borough, chapelry, division, hamlet, parish or township; the mother parish in which the ‘place’ was located (PAR); and any alternative names for the ‘place’ (APL). Further codes indicate the National Grid centroid of the area (NGR); the county in which it was placed in the mid-nineteenth century (CTY), which itself is based on sources across the years 1844–88; and the Ordnance Survey Sheet on which the place can be found (ONP). Another number comes next which is the unique number allocated to the place in the 1851 census (CEN), followed by the sources from which the boundary information was taken (SCE). These sources include the 1851 Census, drainage maps, enclosure maps, the Ordnance Survey first edition 1:10,560 series, rating maps, sanitary maps, and Tithe Maps of the mid-nineteenth century, followed by precise references for those sources (REF) when they were other than the OS and the 1851 Census. Finally there is a column of additional remarks (COM).
The CAT (category) column seems to classify the entries as B, C, D, EP, H, P, T, plus X and Y - Borough, Chapelry, Division, Hamlet, Parish or Township. EP is presumably ExtraParochial, while X and Y are unexplained but generally seem to relate to places where there is some confusion as to boundaries, location or county ownership.
As an indication of scale we have 8,545 parishes, 5,353 townships, 810 hamlets, 116 of the mysterious X (which look like mostly fells and commons of some description), 292 Y and 539 EPs, which includes the Oxbridge colleges, halls, abbeys, cathedral yards, some fens, and some less precisely designated ‘places’. If there is a pattern to this it is not obvious.
For an independent review of this material see "Historic parishes of England and Wales: an electronic map of boundaries before 1850 with a gazetteer and metadata", from which the excerpts above have been taken.
Various parish outlines in England have been created. Wales is already fully covered by a set of parish outlines.
Ireland (Northern Ireland and Republic of Ireland)
See a summary of progress in Ireland. see https://www.genuki.org.uk/org/maps/irelandparishes
Note: The procedure describes below takes much longer to read (and write) than it takes to execute in practice.
Suitable mapping geodata for Ireland is available at https://www.townlands.ie/page/download/, for a variety of features and in a variety of mapping formats, including KML and CSV.
The data can be downloaded from townlands.ie in various formats. The link above provides full details on the structure of this data.
The most useful file is the CSV one, of which there are two options, a normal ("geom") version, and a "no-geom" version. The normal CSV file has an additional column of a WKT data string representation of this area.
The "no-geom" CSV file does not have this, and has no geometry shape data at all (it only has the LATITUDE and LONGITUDE fields). This file is suitable to use for some analysis on the data in a spreadsheet programme. Some (most) spreadsheets have trouble with the long WKT field in the normal ("geom") CSV version, due to the field length exceeding the cell character limit. The files can however be opened in a suitable text editor.
This data all requires some "one-time" pre-processing before use in GENUKI. This and subsequent manipulation can be done in a basic text editor such as Windows Notepad, or TextEdit in MacOS (as long as it is configured to operate in basic "Notepad" mode), athough an extended editor such as Notepad++ (Windows) or BBEdit (MacOS) is required to make what may be described as one of the non-essential "convenience" edits. For some reason, BBEdit is unable to make one of the global (find-and-replace) edits required, although it can make the changes manually on an individual (per item) basis.
- Unzip the file obtained above containing the CSV data.
- Open the civil parish CSV file in a suitable text editor, as described above.
- The data is of the form
- A HEADER ROW describing the data structure and format : WKT String; blah, blah, blah, ......., Latitude, Longitude (the POINT location of the feature, which for later use the order needs to be reversed), and followed by a final term that can be ignored.
- ROWS containing the relevant location data, one row per location (may be wrapped by the text editor). Each row begins with "POLYGON or "MULTIPOLYGON (sic).
- The "civil-parishes" CSV contains data for all of Ireland, except for County Down, for which any such data is difficult to find.
- Close the file.
Pre-processing the WKT Data for Ireland
Open the CSV file in suitable text editor, as described above. At this point it may be considered worth saving it as a TXT file for more convenience in future use.
Make the following global changes to the text, using "Find and Replace".
- For convenience (but not essential) insert a new blank line to separate each data entry. This can be done in Notepad++ and BBedit using extended mode or grep edit "\nTextToBeReplaced", to change "MULTIPOLYGON and "POLYGON to insert a new blank line above each. (Ensure the initial quotation mark " is retained)
- Change "MULTIPOLYGON to "POLYGON (including the opening quotation mark ").
- Change "POLYGON to "GEOMETRYCOLLECTION (MARKER, POLYGON "- there should be one space left between POLYGON and the the succeeding two opening brackets ((. Include the opening quotation mark ".
- Change the multipolygon separator ")),((" to ")), POLYGON ((" exactly as written here, including spaces. (BBEdit may be unable to make this specific global (find-and-replace) edit, although it can make the changes manually on an individual (per item) basis).
- Change the numerisc longitude entry from ",-" to ", -". (Provides correct WKT formatting, and all Ireland entries are west of Greenwich.)
- Ensure that all occurrences of three opening brackets "(((" are amended to two opening brackets "((".
- Ensure that all occurrences of four closing brackets "))))" are amended to three closing brackets ")))". **
** Any remaining instances of "))))" may give a convenient warning that a MULTIPOLYGON is involved, and that detached portions will appear on the resultant map. If so, just copy the first three ")))", rather than all four "))))" to ensure that the WKT text is correct and viable within GENUKI.
As mentioned earlier, this only has to be done once. The file is now ready for use to create new parish outlines.
Create an outline for a parish or other feature in Ireland (do not prepare townland or other feature boundaries at this time, create parish outlines only; such other features are for Drupal 8/9 only).
Once the basic global pre-processing described has been done, open the civil parishes CSV or TXT file for Ireland Parishes.
- Find the data for a parish by searching for (e.g.) ",Antrim" (with the comma but without the quotation marks), to find an entry for the County of Antrim. Note the principal name of the place, a little to the left of the county name. (The text source is not arranged by county.)
- Copy the WKT geodata starting with "GEOMETRYCOLLECTION (MARKER, POLYGON" and ending with the final three closing brackets ")))". Paste this into a new text file and save with a name such as "CCCParishName".*
- In GENUKI, open the relevant parish page, then click the "Nearby Places" tab. On the page which opens, find the marker for the parish, and click to open it for edit.
- In the gazetteer editor view, select the marker location e.g. "POINT (longitude coordinate latitude coordinate)", and copy.
- Return to the open text file and paste this entry over the text "MARKER", then save the file. (The POINT location text should appear preceded by an opening round bracket and followed immediately by a comma, this then followed by a space and the word POLYGON, itself followed by a space before the opening brackets preceding the lat/lon coordinate set.)
- Click "Select All", then click "Copy" to copy the complete WKT string.
- Return to the gazetteer edit screen and paste the wicket string over the existing "POINT (longitude coordinate latitude coordinate)" text, then click Replace.* Then click Save to save the new gazetteer entry.
- For an entry consisting of only a single polygon, the completed WKT text string should look similar to the example shown above under "Using KML".
- For an entry consisting of multiple polygons, the completed WKT text string should look similar to the example shown above under "Using KML" , but will also contain the polygon separator ")), POLYGON ((" at appropriate points.
- If the original marker location is considered to be inaccurate, it can be replaced by the latitude/longitude coordinates contained as the third and second last entries of the data source (although they will need to be used in reverse in order, to be longitude/latitude).
- Obviously all temporary parish outline text files should be closed once no longer needed. Whether they are retained for possible future use is a decision for the individual maintainers.
- Some parishes contain detached portions (multiple polygons) for which there do not seem to be corresponding polygon holes in the other parishes affected. Requires attention.
Repeat all the above as necessary to create polygons for other parishes. Finally open the gazetteer-by-county view and click the "bounds" link near the appropriate county name. A view should appear showing the relevant parish outlines.
* Editor's Note : Experience has shown that is quickest and most effective just to create the appropriate WKT text within a new temporary text file, then "Select All", "Copy", and "Paste" into the appropriate gazetteer entry. Then discard (close) the temporary text file without saving just before starting on creation of the next entry (for which a new temporary text file will be created). In this way a good working rhythm can be developed, and the creation of possibly undesirable clutter can be avoided.
Scotland
Scotland is already fully covered by a set of parish outlines, but these are under continual review with regard to potential improvement or additions.
Channel Isles
There currently does not appear to be any such current mapping activity in the Channel Islands.
Isle of Man
There currently does not appear to be any such current mapping activity in the Isle of Man.
Tools
- QGIS and ArcGIS
QGIS (free) and ArcGIS (paid for) provide methods of harmonising boundaries, using tools such as "snap-to" and "no-intersection", but these have not been tried and are not further addressed here.
Neither QGIS nor ArcGIS are directly supported in GENUKI, if utilised at all they are essentially used off-line as general tools to assist in data preparation.
An attempt to load QGIS to a MacBook failed, as the software provider was unidentified. This seems to be a generally recognised problem regarding use of QGIS on MacOS. In addition, the most recent QGIS (Zanzibar) requires a small range of specific versions of Python 3,.6 but does not work is the most current one, 3.7 .QGIS has been loaded successfully to Windows, albeit working very slowly on a low-powered machine. Various QGIS tutorials are available on-line, You-Tube videos being the most educational.
- Leaflet - some personal investigation into "off-line" use of Leaflet and its javascipt library has been started.
-
OpenLayers - http://dev.openlayers.org/examples/vector-formats.html is useful for playing with WKT strings
-
WKT Lightweight - http://arthur-e.github.io/Wicket/sandbox-gmaps3.html is useful for playing with WKT strings and reading individual coordinates
Townlands in Ireland (For FutureApplication in Drupal 8/9)
Would be useful to be giving some thought as to how best we might import available WKT data on townlands' boundaries to our Drupal 8/9 implementation.
Cam-Pop
In December 2019, contact was made with Cam-Pop (The Cambridge Group for the History of Population and Social Structure), with a view to using some of their map datasets within GENUKI. Despite inital encouraging noises, as of mid- April 2020 this activity has not progressed.
External Work
In Suffolk, QGIS has been used by St Edmundsbury's Local History to create Parish and Hundred maps.
Useful Information
