Core Services Cumulative Monthly Report

Core Services Cumulative Monthly Report, November 2005

New and improved services

LIT on-call service.

May: Core Services agreed to take the first week of integrated LIT on-call, to begin in June.

Institutional Repository.

Dec: Marsh, in coordination with Blanco, installed Apache on sambuca to enable custom CGI development for the IR project.
Jan: Snavely installed the Java CoSign software on sambuca to prepare for integrating DSpace with CoSign for authentication to the institutional repository.
Mar: Snavely, with assistance from Blanco and Farber, coordinated the design of a development and production environment for DSpace that allows for multiple work areas for developers and/or mantainers and complete version control, including tracking and importing new releases of DSpace from MIT and preservation of local configuration.
Mar: Snavely configured Java CoSign on sambuca, establishing a proof-of-concept that logic within the DSpace application will be able to trigger CoSign authentication.
Apr: Snavely incorporated CoSign support into processes used by Blanco, and handed off work to Blanco to adapt the existing authentication servlet to honor CoSign authentication and properly initiate a DSpace session.
Jun: Snavely worked with Blanco to determine ways to get the proper logout behavior for DSpace using CoSign.
July: Snavely and Blanco discussed aspects of adapting a throttling filter Blanco discovered for use with DSpace.
Aug: Bonn, Ottaviani, Snavely, and Willett continued discussions on approaches for integration between DLXS and Deep Blue.
Aug: Snavely, working with Hall, resolved quality problems with OCR processing using Acrobat software, and developed a draft best practice for preparing submissions of scanned documents.
Sep: Snavely assisted Blanco with the first import of a new version of DSpace into the CVS repository on sambuca.

Shibboleth.

Mar: Snavely, working with Doster at ITCS, began the process of establishing a pilot service for access to ProQuest resources.
Aug: Snavely met with Luke Tracey, coordinator of the Enterprise Directory (ED) project at ITCS, to discuss ways that Shibboleth work could dovetail with the development of the ED.
Sep: Snavely began a series of monthly telephone calls with Doster to discuss the status of and next steps for Shibboleth work.
Nov: Snavely, in response to an interesting message from ProQuest to Internet2 and ITCS, proposed a timeline to ITCS for establishing a Shibboleth pilot in the spring, with a potential production rollout of Shibboleth access to ProQuest resources for the fall of 2006.

Infrastructure

Improvements to storage infrastructure, web server redundancy.

Nov: Marsh and Snavely selected a high-performance switch to underlie the first storage network, and Snavely placed the order.
Feb: Marsh installed and configured the new storage server, whisky, and the storage network switch in the Central Campus server environment.
Mar: Snavely synchronized content from dlps10 to whisky, reactivated mechanisms in the DLXS workflow to automatically synchronize both servers when content or middleware is updated and when page images are loaded, and automated a weekly synchronization as a backup to these routines.
Apr: Marsh installed and began setting up merlot, a new web server that will mirror current HTI services using network-attached storage. Snavely installed and configured Lib Auth and put access rules into place.
May: Marsh continued to configure merlot and install software required by DLXS, and Snavely made final tweaks to the Apache configuration to allow testing by DLPS staff.
June: Marsh and Snavely installed the new storage server and switch in the North Campus server environment, and Marsh began configuration work.
July: Marsh continued work on the new storage network, including installing the OS on the storage server and configuring the new storage array and switch.
July: Snavely and DLPS staff developed a strategy to use merlot as a production test environment for the new release of DLXS as a first step of putting the new server, and storage model, into production service.
Sep: Snavely, in coordination with DLPS staff, migrated HTI service from dlps10 to merlot, where DLPS staff had installed and tested the new release of DLXS and new XML/UTF8-encoded content.
Oct: Snavely, in coordination with DLPS staff, migrated HTI service from merlot to load-balanced virtual service running simultaneously on dlps10 and merlot.
Oct: Marsh began the process of installing new web servers in the North Campus server environment that will be used with the storage server already present there, but was delayed by rack space issues.
Nov: Marsh continued work to free rack space in the North Campus server environment to make room for new web servers.

CoSign.

Jan: Snavely installed and configured CoSign on bernie, the new Web Services development server. This will provide an environment for Web Services to migrate web applications that use Library Authentication to begin using CoSign.
Apr: Snavely began adapting the Library Authentication module for Apache to a new design that no longer handles password authentication, but still handles IP address authentication and general authorization, and which is compatible with CoSign.
May: Snavely achieved proof-of-concept with the new version of the Library Authentication module, demonstrating correct access control and authentication prompting behavior.
July: Snavely fixed a few old bugs in the new version of the Library Authentication module (mod_authz_umichlib); improved the linking to Oracle libraries in the module build process; achieved proof-of-concept on a new standard Apache build including CoSign, PHP, mod_authz_umichlib, mod_bwshare (a throttling module), mod_perl, and mod_ssl, which will be used on all web servers, and rolled out as a part of the transition to CoSign; and began the diplomatic process of transitioning non-uniqname accounts in the system to CoSign Friend accounts, working with Bonn, Willett, and others.
Aug: Snavely continued to weed out non-uniqname accounts in Library Authentication in preparation for the move to CoSign, working directly with hosting partners at the BAS, LLMC, UM Press, and others.
Sep: Snavely reached agreements for handling most non-uniqname accounts in Library Authentication, and began updating the database accordingly.
Oct: Snavely continued work to transition non-uniqname accounts to Friend accounts.
Nov: Snavely transitioned all non-uniqname accounts with the exception of one which is used for DLXS collection management to CoSign Friend accounts, and will continue working with DLPS staff to transition the final non-uniqname account; also, Snavely began discussions with DLPS staff on a single-hostname service model that will simplify the colocation of all DLPS and SPO collections, and which will be required for the SSL-based service that will accompany CoSign authentication.

Positioning DLPS as a trusted digital repository (TDR).

Dec: Snavely, working with Stephenson and Willett, reviewed options for persistent identifier schemes for the library and settled on a preference for CNRI handles. This preference will be reviewed with appropriate colleagues within the library.
Jan: Snavely installed and configured the Handle Server software on rum and registered the library with CNRI's global registry.
Mar: Snavely installed and configured a dedicated handle server for DSpace on sambuca with a subordinate naming authority.
Mar: Snavely, meeting with Stephenson and Willett, discussed approaches for introducing automation into workflows in the Digital Reformatting unit.
July: Snavely met again with Stephenson and Willett to discuss next steps in several areas of process improvement.
July: Stephenson interviewed Snavely on current backup procedures and redundancy of DLPS storage, and will formulate the information discussed into a document.
Sep: Snavely and Willett discussed current TDR efforts in light of Stephenson's departure, the new planning year, and the recent announcement of RLG's checklist for digital repositories, and decided to discuss a refactoring of tasks into soon-to-be-developed LIT goals for the next planning year.
Nov: Snavely, after discussions with Wilkin and Willett, drafted and incorporated new TDR-related goals for the next planning year.

Redundant MySQL.

Mar: March and Snavely discussed configuration plans for redundant MySQL servers, and Marsh placed the order.
June: Marsh and Snavely installed the new servers in the Central and North Campus computer rooms, and discussed using a different Linux distribution (Debian) for configuration.
Oct: Marsh installed one new MySQL server, cowmilk, in the Central Campus server environment and, with assistance from Snavely, installed its OS (our first production Debian installation).
Nov: Marsh installed soymilk, the other new MySQL server, in the North Campus server environment, and installed its OS (a Debian installation identical to that on cowmilk).

DHCP/DNS services.

Nov: Marsh, in coordination with DSS, rolled out temporary DHCP service on sake after the library's primary DHCP server, stella, failed. Plans are still being developed to move to ITCS-managed DHCP.

Develop an authentication and authorization management system.

Revisit logging practices.
Production server replacements/upgrades.

Nov: Marsh installed and configured scotch, a new file server for the Michigan Digitization Project.
Dec: Marsh and Snavely decided on a storage upgrade configuration for gracie to allow for more free space for Oracle archive logs and disk-to-disk backups, and Marsh placed the order.
Jan: When load scripts began to warn of decreasing free space, Marsh ordered a storage upgrade for dlps10.
Jan: Storage purchased for ARC arrived and Marsh and Snavely devised a new storage upgrade plan, diverting the new storage to gracie to accommodate a new, urgent need to do parallel indexing. Marsh began the process of installing and configuring the storage and ordered a new SCSI adapter for ethel.
Feb: Marsh installed the storage upgrade for Oracle archive logs and disk-to-disk backups on gracie and Snavely migrated the storage into place on the system, thus freeing storage that will be used on ethel. Marsh continued work on the parallel indexing storage upgrade for gracie, eventually involving Sun technical support due to a suspicion of hardware problems.
Mar: Marsh, working with Sun technical support, resolved all outstanding issues with the new storage, connected the new storage to gracie and the old repurposed storage to ethel, and conducted load testing on both.
Mar: Snavely rebuilt disk configurations on the new storage for ethel and gracie and put both into service.
Apr: Marsh, working with Steinberg, moved the library home page from myrna to belle, a new Linux web server.
Apr: Marsh upgraded the firmware on all Sun StorEdge 3310 RAID systems.
May: After load scripts began to report shrinking free space reserves on dlps10, Marsh installed and configured the spare disk array on dlps10 and ordered a new spare.
May: March installed and configured a replacement server for coffee by the same name, and Snavely put it online with no service interruption.
July: Marsh began the purchase process for servers to replace dlps8 and dlps9. The replacement process will be used as an opportunity to transition to networked storage at north campus.
Nov: Marsh and Snavely discussed plans for a major storage upgrade for the servers that host Mirlyn and their periphery, with the aim of building a fiber-based , managed storage area network (SAN) to simplify management, provisioning, and upgrades, to allow for better movement of data between servers, and to increase performance and reliability.

Formalize system administration infrastructure.
Replace DLXS statistics.
Centralize LIT backup.

Feb: Marsh arranged to acquire two large tape libraries from colleagues at ITCS for potential use in a centralized LIT backup service.
Apr: Marsh, with help from ITCS staff, moved the tape libraries into our Arbor Lakes rack area and began making arrangements for pickup.
Apr: Snavely began pursuing pricing information for backup software.
May: Snavely continued to have discussions with software vendors to explore options for a new backup system.
July: Snavely coordinated the move of the tape libraries acquired from ITCS from Arbor Lakes to Hatcher to begin testing them.

Development server replacements/upgrades.

Jan: Marsh, working with Steinberg, completed the transition of Web Services to a new Linux-based development server, bernie.
Jan: Snavely installed Oracle and Library Authentication on bernie.
Jan: Snavely dumped the journal management system database so Sheppard could migrate the service, one of the last fizzie dependencies, to production servers.
Apr: Working with Powell, Snavely captured the last remaining items from Bugzilla and stored them offline. All services on fizzie were shut down, and if no problems are discovered by the end of May, fizzie will be decommissioned.
May: Snavely decommissioned fizzie.
July: In preparation for the DLXS workshop, Marsh upgraded jolt, the workshop server, to a newer, but used, desktop system.
Sep: In follow-up from the DLXS workshop, Snavely and some DLXS staff discussed the idea of purchasing a larger server for the workshop that could be somehow put to use for staff during non-workshop time. This idea will be further explored after the DLXS release.

Security assessment.

Workflow, processes, and organizational work

LIT work requests and scheduling.

Nov: Snavely recommended the use of FootPrints to LIT managers for handling work requests and scheduling, and suggested a brief refresher session for FootPrints use for the group, to focus on reporting.
Dec: Snavely made basic configuration tweaks to the LIT Core Services project in FootPrints, and Marsh and Snavely began entering current work in process into the tracking database.

Clearly articulate LIT services and resources.

Dec: Snavely prepared an overview of Core Services, to be integrated with overviews from other LIT units.
May: Snavely reworked the topmost Core Services web page to be design-consistent with the LIT site, and incorporated content from the service overview into the new design.

Documentation and standardization of processes.

Dec: Marsh and Snavely moved several pockets of old documentation into the new structure on the LIT web site, and Snavely did some basic reorganization, simplification, and editing.
Apr: Snavely consolidated, simplified, and updated aged server inventory documentation to facilitate better tracking for server replacement.
Apr: Marsh began to configure all servers running various versions of RedHat to acquire updates from the Fedora Legacy project.
July: Marsh and Snavely added several new documents, including storage array configuration and a new Apache build process. Snavely updated the server overview with new retire dates.

Improve coordination of data loading.
Cost model.
Technology trends awareness.

Miscellaneous/Unplanned

Nov: Marsh continued to migrate more users from the local mail server, rum, to ITCS mail.
Dec: Marsh migrated all but the final few users to ITCS mail.
Dec: Snavely configured Samba file sharing on the MDP file server and, in coordination with Gietzen and Potter, finalized the design of scripts that will automate the quality review process.
Jan: Marsh completed the migration of mail users from the local mail server, run, to ITCS mail.
Jan: To work around a Solaris system limitation, Snavely compiled and installed a new 64-bit version of Perl on dlps10. After testing, Marsh distributed the new version of Perl to dlps8 and dlps9.
Jan: Snavely compiled and installed throttling software on dlps8, dlps9, and dlps10 to prevent overuse of the systems by automated downloaders.
Feb: After limited success and several attempts at tuning, Snavely concluded that the throttling software added an unacceptable amount of overhead, removed it, and began seeking other options.
Feb: Upon learning that the backup service currently used by DLPS servers on north campus will be discontinued, Snavely pursued a preliminary agreement with ITCS for backup services and began preparations for migrating.
Feb: Marsh investigated a potential problem with memory on gracie, and was advised by Sun that the error was harmless.
Mar: Snavely completed migrating all DLPS servers to ITCS backup services.
Mar: Snavely tested download and validation processes for a large batch of material from the Google return server, tweaking existing scripts as needed.
Apr: Snavely ordered three DVD drives to replace existing CD-ROM drives in Patterson's loading workstation.
Apr: Snavely researched and began compiling a new throttling package to prevent overuse of DLPS services.
Apr: Marsh upgraded MySQL on clamato for testing at the request of Weise, with plans to also upgrade MySQL on dlps8 if no problems are encountered.
Apr: March began the process of adjusting and renewing Sun support contracts for the next fiscal year.
Apr: Gray, Snavely, and Patterson agreed on a new scheme to manage user accounts for Web Services servers. Gray will continue as liaison to library staff, but will now dispatch requests to Patterson, who will take on responsibility for managing accounts using tools provided and supported by Marsh.
May: Snavely discussed file management issues with SPO and made several recommendations for simplifying file access for staff.
May: Snavely added functionality in load scripts to detect invalid directory and file naming.
May: Snavely participated in a conference call, along with Bonn, Sheppard, and Willett, to discuss implementation plans for supporting the Athens authentication model for several SPO-managed collections.
June: Snavely set up file service on ferment as part of a strategy to simplify file access for SPO.
June: Marsh and Snavely met with Map Library, Science Library, and Geospatial staff to discuss ArcIMS hosting options and strategies.
July: Snavely met with Miller and contacts at Onshore Development (via conference call) to discuss a belated OS upgrade for the WebCheckout server, gin.
July: Snavely organized a follow-up meeting with Sheppard to discuss Athens implementation plans in greater detail, and wrote up the strategy for hands-on work later in the fall.
July: In response to a request from Apex, Snavely created a diagram of the storage and web service model being put in place for DLPS.
Aug: Marsh began the process of transferring the journalofelectronicpublishing.com domain to our authority.
Aug: Marsh installed oXygen, an XML editor, on several development servers for evaluation.
Aug: Marsh began working with Syrigos to migrate FootPrints-related mailboxes from rum, which is being phased out as a mail server, to a new DSS-managed email server being installed specifically for use with FootPrints.
Aug: Marsh installed a utility to monitor and manage the internal hardware RAID controllers in our Dell PowerEdge servers, and will work with Snavely to develop and roll out monitoring scripts.
Aug: Snavely coordinated the planning of a very productive meeting of the DLF Aquifer Technology group in the library.
Sep: Marsh put more restrictive firewall rules in place for SSH access to all servers, to protect against future SSH vulnerabilities and reduce risk in general. Snavely will discuss more restrictive access via SSH to the library web server at the next library web managers' meeting.
Sep: Marsh, in coordination with Dueber and Lu, installed EZProxy on bernie to replace dlps5 as a warm spare proxy server.
Sep: Snavely, in coordination with Hagedorn, installed client software that works with Emory University's MetaCombine subject clustering project for Hagedorn's exploration.
Sep: Snavely, at the request of ITCS, began migrating servers using the ITCS TSM backup service to a newer, faster backup server.
Oct: Snavely completed the migration of backup services to the new backup server.
Oct: Marsh, in coordination with Dueber and Lu, switched gears and installed EZProxy on hiram rather than bernie to replace dlps5 as a warm spare proxy server, and began work to decommission dlps5.
Oct: Marsh upgraded the OS on sake and sangria.
Nov: Snavely transitioned load-balancing service from dlps5 to cowmilk.
Nov: Marsh upgraded the OS on lemonade.

System Performance

Arbor Lakes server environment

ethel.umdl (ARC server): No down time
ezra.umdl (Verde server): No down time
gracie.umdl (Aleph server): No down time
hiram.umdl (Zebra server): No down time
tequila.umdl (Metalib and SFX server): No down time

Central Campus server environment: All services were unavailable on Tuesday, November 29 from 12:00am to 8:00am due to a scheduled building-wide power outage.

belle.umdl (Library Web server): No additional down time
coffee.umdl (Oracle server): No additional down time
gin.umdl (WebCheckout server): No additional down time
sambuca.umdl (Deep Blue server): No additional down time
quik.umdl (Survivors of the Shoah cache server): No additional down time
ting.umdl (library proxy server): Also down on Friday, November 11 at 4:51pm for a brief period of time due to a configuration problem.

North Campus server environment: All text and image services were unavailable from Wednesday, November 9 at 10:45pm to Thursday, November 10 at 8:45am due to database problems.

dlps5.umdl (backup library proxy server): No down time
dlps6.umdl (Oracle server): No down time
dlps7.umdl (Numeric Data server): No down time
dlps8.umdl (non-public collections): No additional down time
dlps9.umdl (Image Services): No additional down time
dlps10.umdl (primary web server for DLPS and public collections): No additional down time

Data Loading and Archival Statistics

CD/DVDs loaded or reloaded (bitonal page images)	15
CD/DVDs loaded or reloaded and processed (contone images)	50
Removable drives loaded or reloaded	1
CD/DVDs burnt or duplicated (library)	123
CD/DVDs duplicated (ITCS)	0