Core Services Cumulative Monthly Report

Core Services Cumulative Monthly Report, August 2003

Overview

Several roadblocks were encountered having to do with storage: configuration problems are preventing a swift storage upgrade to dlps4, and persistent failures of quik's storage have essentially offlines that server until repair options can be more fully investigated.

In spite of these problems, we made progress on DNS service and OCR workflow.

Due to the fortunate positioning of Hatcher Library on the independent UM electrical power grid, and also in part to the redundancy of the Library Authentication system, critical services during the massive power outage on August 14 were essentially unaffected.

Refining and Improving Core Infrastructure

New storage model.

Nov/Dec: Marsh and Snavely continued to collect information on NAS and SAN related technologies for our upcoming storage needs and new storage model.
Apr: Snavely confirmed with Bonn that LLMC plans to fund two storage servers and two web servers to meet redundancy recommendations.
May: Snavely confirmed that NAS on high-bandwidth network is the preferable storage model for DLXS, although SAN may be used as a back end. Marsh coordinated the transfer of dlps3 to Hatcher for testing as a NAS server.
July: Snavely began research of NAS "head" devices as a potential low-cost option for NAS servers.

Redundant web services.

July: Snavely began rebuilding a disk subsystem on quik in preparation for putting quik online as a mirror of the HTI service.
Aug: Snavely concluded that the disk subsystem on quik was not functioning properly, and Marsh is currently seeking repair, upgrade, or replacement options.

Production server replacement.

Nov/Dec: Marsh replaced dlps1 with a new Sun 280R server, dlps8, and coordinated the transfer of services.
Nov/Dec: Marsh and Snavely coordinated the removal of failing storage equipment from dlps4.
Jan: Marsh replaced dlps3 with a new Sun 280R server, dlps9, and coordinated the transfer of all services with the exception of Pictures of Record pending a DNS registration change.
Feb: Snavely coordinated the configuration of backup software on dlps6, dlps7, dlps8, and dlps9.
Apr: Marsh coordinated the transfer of Pictures of Record from dlps3 to dlps9, working around persistent DNS registration problems.
June: Marsh and Snavely prepared and Marsh ordered configurations for replacement servers for hank, dlps4, and dlps5.

Development server replacement.

Nov/Dec: Marsh prepared and ordered configurations for replacement servers for fizzie and kukicha.
Apr: Marsh installed and deployed clamato to replace fizzie.
Apr: Marsh installed and deployed sangria to replace kukicha.
June: Marsh and Snavely prepared and Marsh ordered a configuration for a replacement server for alice.

DHCP services.

Mar: Snavely transferred the load-balancing DNS server from kombucha to sake, the new DNS/DHCP server.
Aug: Marsh, with assistance from Snavely, installed and configured name service on plumwine, a new virtual host on sake.

Image and OCR processing workflows.

Nov/Dec: Snavely began planning work for the LLMC project with Bonn and Stephenson.
Jan: Snavely, in coordination with Stephenson and Pagliere, made a small modification to the DLXS Pageview table to allow the handling of companion contoen page images.
Feb: Snavely revised existing load scripts to handle "continuing content" on image media.
Mar: Snavely, working with Hall, began investigation of the PrimeOCR "job server" functionality to better balance OCR load.
Apr: Snavely, working with Hall, successfully set up a proof-of-concept OCR "farm" using job server functionality and proposed a partial architecture for OCR workflow.
May: Marsh and Snavely prepared and Marsh ordered a configuration for an OCR file server.
June: Marsh began the setup of the OCR file server and a single replacement server for bill-ko (CD loading) and water (contone image processing).
July: Marsh completed the OS installation of lemonade, a new CD loading and contone image processing workstation, martini, a new OCR file server, and a new 19" cabinet to house martini and Hall's other rackmount OCR servers. Snavely ported CD loading and contone image processing scripts (eliminating our need for the use of rsh) and managed the transition of both workflows to the new loading workstation.
Aug: Snavely, working with Hall, continued the design of the new OCR workflow that will take advantage of the OCR file server and PrimeOCR job server functionality.

Digital object integrity.
Cost model.

Nov/Dec: Snavely developed a provisional model for tracking server administration costs and began to apply the model to generate several cost estimates.

Documentation.
Technology trends awareness.

Nov/Dec: Snavely began receiving subscriptions of several IT publications.
Mar: Marsh attended the Rubi-Con computer security conference in SE Michigan.

Support and Development for Access Systems

Library management system implementation.

Nov/Dec: Marsh, Snavely, and Prettyman researched server configurations for the LMS development server. Many different configurations were explored and many sales quotes generated. An eight-processor Sunfire 4800 with two attached 400GB disk arrays was chosen and ordered.
Jan: Marsh, working with Sun staff, installed and configured gracie, the LMS development server, in room 10.
Feb: Snavely coordinated the installation of SFX on the LMS development sevrer with Ex Libris.
Mar: Snavely coordinated the installation of ALEPH on the LMS development server with Ex Libris.
Apr: Snavely prepared a configuration for the production Metalib/SFX server according to Ex Libris recommendations, and placed the order.
Apr: Snavely, coordinating with Farber, Plantinga, and Steinberg, proposed and set up CVS on the LMS development server to facilitate version-tracking and release processes.
May: Marsh installed tequila, a new production SFX server. Snavely attended ALEPH System Admininistration training.
June: Marsh re-installed tequila with an older version of Red Hat Linux to satisfy late-breaking support requirements from Ex Libris.
July: Marsh configured hot spares for the OS disks on gracie and set up Sun Raid Manager storage status monitoring for the external storage arrays.
July: Snavely, in coordination with Steinberg and Plantinga, moved the installation of SFX from the LMS development server to tequila.
July: Snavely coordinated the installation of Metalib on tequila.
Aug: Snavely set up scripts to dump SFX databases for more effective backups.

DLXS.

Feb: Marsh successfully coordinated the use of the staging server, jolt, as a new environment for conducting the workshop.
Mar: Snavely met with Zieske to gather information on the workings of the stats system.
May: Snavely merged the CVS repository on the data prep server sangria into the repository on the primary development server clamato.

Production OAI harvesting.

Nov/Dec: Marsh and Snavely coordinated a meeting with data preparation staff and developed a plan to run data preparation and OAI harvesting on the same (new) server. Funds for the server will be supplied jointly by the OAI grant and Core Services.
May: Snavely, in coordination with Blanco, Burek, and Hagedorn, moved the OAI harvester from its temporary location on quik to the new data prep server sangria.

Institutional repository initiative.

June: Marsh and Snavely prepared and Marsh ordered a configuration for a DSpace server.
July: Marsh completed the OS installation of sambuca, a new DSpace server.

Collaborating with Other Areas of the Library and the University

Library/Core Services coordination.
IT Commons.

Nov/Dec: Snavely contributed heavily to the Authentication/Authorization and Directory working groups to help prepare final reports, due at year's end.
Jan: Snavely continued work at a slower pace in both working groups on various tasks in follow-up to the final reports.
Mar: Snavely contributed to sourcing and prioritization dicsussions in both working groups.
May: Snavely and Stephenson participated in an IT Commons meeting regarding disaster recovery.

CHEF/DLXS integration.

Nov/Dec: Snavely participated in a joint meeting with other DLXS staff and the CHEF group to help provide a better understanding of the DLXS system.
Feb: Snavely and Powell discussed strategies for exploring collaboration between DLXS and CHEF.

Integration with campus authentication.

Nov/Dec: Snavely joined the COSIGN co-design group and attended his first meeting to get an overview of the COSIGN system architecture and workings.
Feb: Snavely attended another COSIGN co-design group meeting to gather information on the proposed "Friend" system to handle authentication of non-UM affiliates.
Apr: Snavely began a discussion within the LMS group which will investigate the steps necessary to integrate COSIGN with ALEPH and identify issues associated with the rollout of COSIGN for DLS resources in general.

Miscellaneous/Unplanned

Nov/Dec: Marsh, working closely with DSS, reorganized space in room 10 to consolidate space usage and accommodate new servers.
Nov/Dec: Marsh applied OpenSSL and sendmail upgrades to resolve security vulnerabilities on several servers.
Nov/Dec: Marsh upgraded Perl from 5.6.1 to 5.8.0 on all systems not scheduled for near-term retirement.
Jan: Marsh, in coordination with Blanco and Simpson, began an iterative problem-solving process with ESRI to address several problems with the ArcIMS system.
Mar: Marsh began the installation of a LOCKSS server.
Mar: Snavely, working closely with technical staff at Oxford, set up rsync for synchronizing EEBO content from DLPS servers to the mirror at Oxford, and integrated the rsync configuration into existing local update scripts.
Apr: Snavely, working closely with Syrigos, set up rsync for synchronizing geospatial content from the geospatial lab to DLPS servers.
May: Snavely transitioned service of electronic University and Social Work reserves from dlps8 to hank.
May: Snavely coordinated the beginning stages of an improved design for production (on-call) support for DLS.
May: Marsh and Snavely worked with Blanco, Simpson, and Syrigos to refine directory conventions for automatic synchronization from the geospatial lab to DLPS servers.
May: Marsh did an initial install of the LOCKSS service on repurposed desktop hardware.
June: Marsh and Snavely did a second install of a new version of the LOCKSS service.
June: Marsh moved the backup server (nog) to new hardware (lambic) and developed a plan for upgrading our Arkeia backup sofware from version 4 to 5.
Aug: Marsh registered and set up virtual hosting for the new llmc-digital.org domain.

System Performance

Central Campus server environment

betty.umdl (MedSearch server): No down time
coffee.umdl (Oracle server for access control): No down time
gin.umdl (WebCheckout server): No down time
hank.umdl (Library Web server): No down time
myrna.umdl (MIRLYNWeb server): No down time
opal.umdl (SilverPlatter server): No down time
tequila.umdl (SFX server): No down time

North Campus server environment: All services were unreachable from Thursday, August 14 at approximately 4:00pm to Monday, August 18 at approximately 12:00pm due to the massive power outage in the Midwest and East Coast, and also on Thursday, August 21 from 7:00am to approximately 4:00pm due to major electrical upgrade work.

dlps4.umdl (primary web server for DLPS and public collections): No additional down time
dlps5.umdl (library proxy server): No additional down time
dlps6.umdl (Oracle server for access control and usage statistics): No additional down time
dlps7.umdl (Numeric Data server): No additional down time
dlps8.umdl (non-public collections): No additional down time
dlps9.umdl (Image Services): Also additional down on Tuesday, August 19 from 12:30pm to 1:45pm due to problems encountered while configuring a storage subsystem.