ONE FRAMEWORK TO RULE THEM ALL The Unifying Impact of IIIF on Teaching, Research, Museums, and Libraries at Harvard JEFF EMANUEL Associate Director of Academic Technology Harvard University Academic Technology ARTHUR BARRETT Senior Software Engineer Harvard University Academic Technology JEFF STEWARD Director of Digital Infrastructure and Emerging Technology Harvard Art Museums RANDY STERN Director of Systems Development Harvard Library (Library Technology Services) IMAGES fundamental information carriers for cultural heritage The rate at which library and museum holdings are being digitized and made available as pixels has increased significantly over the last quarter century, for both access and preservation purposes. This increases the potential for deeper interaction with digital visual material by the end user. But access to images has long been dominated by silos and duplication, And efforts to share content across institutions (and, in some cases, across repositories within an institution) continue to encounter obstacles. Aside from rights and permissions, the most significant obstacle is the differences in the handling of storage, management, and delivery of digital material across repositories, which frequently use separate frameworks, formats, and applications. Let’s say we have an image in a database that someone wants to see. Not too difficult...until that person wants to see another image from another database, at which point they may have to use an entirely different application, and have an entirely different experience while viewing that object This is a widely recognized issue, but efforts to resolve it have all too often fallen victim to what has been called “the not-invented-here syndrome: the conviction that ‘you and I will collaborate just fine if you adopt my system and abandon yours.’ Because of this, the easiest way for people to access, curate, and view images, in the viewer of their choice, has been to make a local copy, like you see here. But when someone else wants that same image, that creates another copy. And another and...you get the picture. This leads to tens, hundreds, thousands, or more copies of individual images, potentially living on hard drives all over the world. But how to solve this? Enter IIIF International Image Interoperability Framework Questions that guided the development of IIIF • What if deep zoom were standard and fast? • What if you could compare across sites, hosts, and institutions? • What if you could collect items that belong together physically – not just intellectually? • What if you could access images using the viewer of your choice, instead of a certain one tied to a particular repository? Opportunities afforded by IIIF • Dramatically reduce friction around access • Enable radical new opportunities and capabilities for interaction • Transcend silos by joining content communities in uniquely powerful ways You can think of the countless locally-stored copies of an image as one of a countless number of identical water bottles, filled and shipped to the end user, where it will potentially be isolated from the source and from others forever. In this vein, describing the impetus for IIIF, William Ying and James Shulman (ArtStor), wrote that “The community set out to create pathways for images to flow through systems from their source, rather than trying to get all repositories and service providers to build everything the same way. Digital image content generated and served up by a range of different image servers could be standardized as output (both the individual image and the structure of an image grouping) for re-use…and could flow, rather than be handed off like plastic water bottles.” Ying, W. and Shulman, J., 2015. “Bottled or tap? A map for integrating the International Image Framework (IIIF) into Shared Shelf and ArtStor.” D-Lib 21(7/8), 1-12. In other words, instead of handing off water bottles, IIIF allows users to fill the glass of their choice from the same tap. How does IIIF Work? Simple! By using shared APIs • An API (Application Programming Interface), is, at its most simple, a means for applications to communicate “An API is a software intermediary that allows two applications to talk to each other. Each time you use the Facebook app, send an instant message, or check the weather on your phone, you’re using an API. When you use one of these applications, the application on your phone connects to the Internet and sending data to a server. The server then retrieves that data, interprets it, performs the necessary actions and sends it back to your phone. The application then interprets that data and presents you with the information you wanted in a readable way. This is what an API is - this all happens via API.” https://www.mulesoft.com/resources/api/what-is-an-api IIIF APIs Image API: Gets pixels via simple, RESTful web service Presentation API: Just enough metadata to drive a remote viewing experience, including the table of contents, object name, and thumbnails I mentioned before that one of the advantages of the IIIF approach is that the user is not locked in to using a single particular viewer. The viewer you’ve seen here is called Mirador. Mirador is one example, out of multiple, of a IIIFcompliant viewer, used for this visual material by Harvard and several other institutions. You’ll see it more as this presentation continues, and learn more about it from my colleague, Arthur Barrett. OpenSeadragon allows for a deep zooming experience, even on mobile, while the data model also scales up to render beautifully on the Harvard Art Museums’ 9,000 pixel screen http://bit.ly/2tlHcy7 Multiple Views From Multiple Repositories: Side-by-side comparison of digitized statues from two repositories, the Harvard Art Museums (left) and the Yale Center for British Art (right), accessed via IIIF API and displayed in the Mirador viewer. Two-slotted workspace, featuring “book view” and page view A medieval Book of Hours (MS Richardson 7, Houghton Library, Harvard University) displayed in the left slot in Mirador’s “Book View,” with page thumbnails and table of contents included. In the slot at right is the denoted portion of the illumination on folio 15 recto, displayed at full zoom level (red emphasis added for clarity). Worldwide IIIF Consortium • >100 Institutions on 5 Continents • Sharing between 350 million and 1 billion images IIIF.HARVARD.EDU An institution of any size can find itself prone to “Cylinders of Excellence” (even Harvard) Together, though, Harvard has been able to leverage the IIIF/Mirador design pattern across multiple functional areas, sharing core Image API and Digital Repository Services. Sharing knowledge, expertise, digital content, and Mirador, multiple “heads” have sprouted: • An image viewer/annotation application for HarvardX courses • A new Harvard Library Viewer • Faculty image collections embedded in the Canvas course platform • Walls of images in the Harvard Art Museums. Harvard’s goal is relatively simple: interoperable digital content through common APIs. We have taken a university-wide approach. We are somewhat unusual among IIIF consortium members in that we have had significant faculty interest and input, and are deploying IIIF for teaching in addition to research. But it took communication between groups that don’t always speak with each other as often as they should. IIIF at Harvard Harvard Library Harvard Academic Technology Harvard Art Museums Faculty HarvardX Brief Chronology 2010-2012 • Focus group of faculty, academic technology, and library assess options for a new page turner for the digital repository • Harvard commits to edX and funds a IIIF developer 2014-2015 • Harvard and Stanford jointly develop Mirador 2.0 • Harvard Library releases IIIF services for 120,000 books and manuscripts (5M page images) plus 10M still images • Development of AnnotationsX for HarvardX • HarvardX launches “The Book” and AnnotationsX (LTI-Compliant), Harvard Art Museums deploys a IIIF manifest service for 250,000 art objects 2016 • University-wide IIIF working group formed, representing Academic Technology, HarvardX, Harvard Library, Harvard Art Museums, and Faculty • 3 Mirador-based apps deployed: Harvard Library Viewer, Image Media Management (LTI-Compliant) app, Art Museums digital tour builder 2017 • Planning a IIIF Research Workspace • Fostering research projects with School of Engineering and Applied Science At This Point: Harvard Library • >195,000 books and manuscripts exposed as IIIF manifests; >14,000,000 images exposed through the IIIF image API • Mirador and IIIF is the primary page viewer, and embedded in the Colonial North America Spotlight exhibit. HarvardX • “The Book” MOOC in 2015. • Mirador is embedded, with annotated images in HarvardX courses. • Navigate through a course using Mirador as the platform. Harvard Academic Technology • Mirador used with the ‘Beyond Words’ medieval manuscript exhibits • Mirador is used for several course projects • Image Media Management Canvas LTI-tool - Upload, display, and comparison of faculty-uploaded images in oncampus course web sites using IIIF and Mirador. Harvard Art Museums • Online exhibits, mashups, and digital tours of museum objects. • Over 250,000 art objects exposed as IIIF manifests. University–wide IIIF/Mirador Working Group • Planning an integrated IIIF workspace for research and teaching. • Planning collaboration with the School of Engineering and Applied Science IIIF IN THE CLASSROOM As Jeff has now shown, the Mirador image viewer figures prominently in our use of IIIF. We can take advantage of the fact that users may already be familiar with it from across the university and help provide a consistent user experience in Canvas. One of the reasons we like Mirador is because it is full-featured and flexible, allowing us to customize it depending on the application. In addition to the panning and zooming capabilities, it has tools for comparing images via the windowing system, displaying metadata and structures, and creating and displaying annotations that conform to the Open Annotation Format (among other features). One framework to rule them all, and in Mirador bind them. Client Server IIIF Image Viewer IIIF Presentation API IIIF Image API http://projectmirador.org/ Simple Use Case: Embedding IIIF Images in Canvas This is is a simple use case showing how we embedded Mirador in a Canvas course site along with other content on the page. In this course, we’re pulling IIIF resources from the Harvard Library and the Harvard Art Museums and embedding Mirador (via iFrame) on the page to display the images. Thanks to the IIIF APIs, we’re able to pull resources from either repository, mash it up with Mirador, and it just works. The challenging part is finding the resources themselves. Steps: 1. Search Harvard Library Catalog for an image (in the example, this is “Saint Jerome in his Study,” from Professor Jeffrey Hamburger’s course on the Illuminated Manuscript) 2. Add IIIF manifest to instance of Mirador 3. Embed Mirador in Canvas via iFrame Pros: • Ability to take advantage of IIIF resources from Harvard Library (or any other institution) that publishes IIIF-conforming images. • Consistent user experience, since Harvard Library uses the same viewer. • Relatively simple to set up. Cons: • Manual process. • Embedded Mirador has no knowledge about the course or student, which results in limited opportunities for interactivity. To take things a step beyond simply embedding and displaying IIIF resources, we introduce LTI into the mix. What is LTI? Like IIIF, it’s another standard or API that defines the interaction between an LMS and tools that provide services. By writing tools that are LTI-compliant, it’s possible to use them across more than one course platform such as Canvas and edX. Since LTI tools are able to communicate with the LMS, we can create more interactive applications that take advantage of IIIF. LTI Web Application Phase 1 Phase 2 Phase 3 IIIF + LTI ??? ...Profit! Learning Tools Interoperability Phase 1: collect existing standards for LMS integration (LTI) and image interoperability (IIIF) Phase 2: implement said standards using client/server libraries, e.g. Mirador+Loris for IIIF and django-auth-lti for LTI. Phase 3: use tools in the classroom via Canvas IIIF Image management tool 1. Faculty upload images to tool 2. Create and curate collections 3. Integrate into course modules Students view the collections in Mirador • Improves upon simple embedding by allowing faculty to upload images from their own personal image collections and then curate their own collections and integrate as modules within Canvas. • Automatically creates the IIIF manifests and serves images through a IIIF-compatible server. The user doesn’t have to know or care about the underlying technology (nor should they). Annotation Tool in Canvas IIIF AT HARVARD ART MUSEUMS Let’s pretend you have all of the technical bits of IIIF in place. Then what can you do with it? We’re a Teaching Museum at a Research University. We have 250,000 objects. Here is a sample. You get the idea. We’re an art museum. We have a variety of art stuff. Fun fact: All of the images in the previous slides were made on the fly by using the IIIF image API to grab random vertical slices from a random set of objects in our collection. Practical Stuff Comparative images demo in Mirador: IIIF on the museum’s website coupled with drag and drop from the Yale Center for British Art’s website (https://www.youtube.com/watch?v=OtiwyDg-Ht0) Virtual sketchbook to accompany an exhibition in the museums: We used IIIF to reassemble sketchbooks in a way that wasn’t possible in our cataloguing system. Slightly Less Practical Stuff IIIF manifests for each gallery in the museums Use Mirador and the comparative image viewing capabilities to ask visitors to find similar faces on view in the galleries Scaling it all to the big screen (9k!) Mirador on the big screen: Take advantage of high-resolution images through deep zoom on a large canvas. You can bring in objects from other collections for comparison. Fun Stuff IIIF + Machine Processing = Awesome Fun Google Vision Microsoft Cognitive Services Clarifai Imagga Untitled family portrait from the 1970s http://www.harvardartmuseums.org/ collections/object/140130 When Google Visions finds a face, the data include x,y coordinates, which tell us where the face is located in the image. We can then use those coordinate data to trace boxes around each. Not only do we get the location of the face, but Google Vision tries to tell how happy or angry or sad or surprised or blurred they are. With the data we can also do things like hide all of the faces. Because we have the data about all the location and size of each face, we can render just those portions of the image, and drop out the background. We can do this using a single image thanks to the IIIF Image API. All of the faces that you see are extracted from the full image non-destructively. We aren’t creating copies of the full image and cutting out the faces; we’re using Harvard’s IIIFcompatible services to fetch regions of interest, and they magically are derived from the single, full master image. Just to prove it, here are all of the URLs for the faces. If you can do this with faces, you can do this with text Google Vision finds all the text in the image We can use that info to black out the text. We can use that info to render images of just the regions of text. And there are all of the image regions with their URL in IIIF image API format. One master image, many tiny derivatives. Face Scramble: https://www.youtube.com/watch?v=xFRu2c0oG1E Perhaps you’d like to try your hand and matching faces to bodies. https://www.youtube.com/watch?v=MGLprhZlpWg If you can do these type of things with faces, you can do them with text: https://www.youtube.com/watch?v=dCYqzDw0oYA https://www.youtube.com/watch?v=3-5Hn3QwDnA Play at http://hvrd.art/iiif THANK YOU