Match, Compare, Classify, Recognise and Annotate: Computer Vision for the Digital Humanities. Visual Geometry Group Showcase, University of Manchester, 4 June 2018
Instructor: Giles Bergel, University of Oxford (Engineering Science).
Convenor: Guyda Armstrong.html), University of Manchester (Digital Humanities; Italian).
With the support of the John Rylands Research Institute.
This workshop introduces tools created by Oxford's Visual Geometry Group to address five common tasks in computer vision. It may be followed either through hands-on installation of the tools on your own laptop, or through online demos.
You will need a Windows, Mac or Linux machine to which you have administrative access rights (i.e. an admin password) to install the Matching (VISE), Comparison (Traherne) and Classification (VIC) tools.
Using VISE and VIC on Mac and Windows machines requires the use of Docker, which may additionally require modifying the BIOS settings of your laptop. See the instructions for VISE and VIC for guides on how to do this. Linux users may follow the instructions for each tool on how to compile from source code.
Traherne does not require Docker, but an admin password is still necessary.
VIA is an HTML application, and can be installed on any system by loading the file into a web-browser (Chrome and Firefox preferred).
A relatively recent Web browser is required, running on a laptop, tablet or phone.
Demo data for VISE, VIC, VIA and Traherne installations is available here and on USB sticks. This file is recommended for all attendees, as it also includes query data for users following the online demos.
#Task 1. Matching image elements with VISE
Overview Visual Geometry's Image Search Engine (VISE) provides instance-based image-matching. This means that it can match a specific object within an image when provided with a query of the same object, or one that is geometrically very similar.
If you have a laptop to which you have an admin password, follow the Installation Instructions here.
To set up the VISE environment and create your first index, see the User Guide.
Test data is provided on the Demo data zip file: it consists of illustrations from chapbooks held within the John Rylands Library Street Print Collection (with thanks to Julie Ramwell and the Rylands' Heritage Imaging Team!)
Bodleian Ballads ImageMatch.
This implementation of VISE indexes around 900 seventeenth-century English printed broadside ballads in Bodleian collections.
How to: Select a ballad from one of the thumbnails in the page at the link above. Click and drag a bounding-box around a region of interest, and then press 'SEARCH'.
Try: Selecting 'Detailed matches' to see a match and its context. From here, you can tick the boxes 'Regions' and 'Lines' to see the visual elements (known as 'visual words' that the tool is matching.
Try: Selecting 'Image Comparison' to see a flip-book view of the two matching woodcuts (move the mouse pointer across the images to flip).
Try: Uploading one of the images (a whole ballad sheet or woodcuts) inside the folder Bodleian Ballads Upload Data from the demo data zip (these are taken from items within Bodleian MS Wood E25).
British Library BL_Bindings. Includes images of several thousand binding ornaments and other features. Unlike Bodleian Ballads, this instance displays metadata for each item, and a rudimentary search interface for the filename.
Try: Querying the engine with an external image from the British Armorial Bindings database (NB not many of them match, but see e.g. this one [copy the Image Address by right-clicking and pasting it into the BL_Bindings URL search box]. NB This match does not work at the highest resolution available from British Armorial Bindings. Can you think why?
Ballads with metadata can be seen at Bodleian Ballads Online. Only a selection of the ballads in this larger resource are indexed with VISE - those for which we have high-resolution colour images. The shelfmarks for each item in ImageMatch provide a rudimentary crosswalk to Bodleian Ballads.
A catalogue of the woodcuts in this collection can be seen here. This catalogue sorts automatically-extracted woodcuts by Iconclass terms; by block (clustered by a distance-measurement); and by impressions. It further provides a comparison tool for similar blocks (this function is also provided in a separate tool called Traherne - see below), and indicates co-occurrence of woodcuts on each sheet.
Task 2. Comparing images with Traherne
Named after the Oxford Thomas Traherne project, this tool was originally built in order to detect resettings of type in early printed books. It can be used to help find small variations in pairs of otherwise-identical images.
Traherne is currently offline, but a basic comparison feature is built into the VISE demos as well as the Bodleian Ballads woodcut catalogue (see here for an example).
###Hands-on demo Traherne (admin password required) may be downloaded for Mac and Windows here. There is a folder of matched pairs of images in the demo data folder.
To use Traherne, click Load Base and select the first of a pair of fiiles, then do the same for Load Comp (more than one file can be selected in each operation). Select Base and Comp files in the main viewer. Then, hold down your trackpad button and drag the selection box around a region of the image that you would like to compare. Click Compare Base & Comp: Traherne will then register the two images. The comparison view can be an overlap (select Toggle to flip between the two images); a composite; or a colour-coded 'diff' view. Try the Zoom option.
#Task 3. Classifying images with VIC
Overview: VIC provides the ability to search images by categories, of high-level classifications of their content, using representations of those categories learnt from pre-tagged examples. At the leading edge of computer vision research, successful image classification depends on the size and uniformity of its training data.
VIC searches are typically initiated by a user entering a keyword: the system will then retrieve a training set of images tagged with that keyword from Google (mostly tagged manually). The system then creates a visual model of the images within that training set, then searches an unclassified collection with that model.
NB: Initially VIC results are often unsatisfactory, due to the lack of a sufficiently uniform training set, which may in turn be due to the variety of images that may be represented by a word). You can refine the search by selecting good matches and searching again.
VIC paintings. This provides a number of search modes. You can supply an image for similarity-based searching; a keyword for retrieving Google-tagged images for training purposes; or you can pull down a list of curated queries which act as browse points for onward visual searches. See 'Getting Started (bottom right of screen) for instructions. This tool is based on the ArtUK demo, which (also provides the ability to search by colour and texture.
BBC News also provides the ability to search by instance (using VISE), people and onscreen text.
To intall VIC on your own machine, follow the Installation Instructions here.
Test data from Microsoft's Common Objects in Context (COCO) dataset is on the demo data zip.
#Task 4. Recognising faces with VFF
Facial recognition represents many people's common experience of computer vision. Its accuracy is of considerable public interest.
A custom version of VFF has been provided for this workshop, consisting of the faces of three celebrities within a larger collection. See 'Getting Started (bottom right of screen) for instructions. Try searching for one of the three celebrities using an external URL, or an uploaded image.
Not supported in this workshop, but go to the VFF homepage for more information.
#Task 5. Annotating images with VIA
VGG Image Annotator is a simple tool for annotating images. It supports drawing arbitrary shapes on an image and tagging each image with arbitrary metadata, which can be exported as a csv or JSON file. Projects such as the 15C Booktrade project use VIA in conjunction with VISE to mark up matched regions of an image.
Online demo Pull down 'Loaded Images' to see the available images.
For Dr. Guyda Armstrong's Dante project, images of early printed editions of Dante have (thanks to Charlotte Alton) been marked up into a number of regions. The intention is to teach a machine-learning system to recognise page-elements and to infer their relationships. Some sample pages and a JSON file of annotations are provided in the demo data zip.