Core Services Cumulative Monthly Report

Core Services Cumulative Monthly Report, October 2004

Refining and Improving Core Infrastructure

New storage model.

Sep 2003: Snavely began researching the Linux Logical Volume Management system as a potential management suite for NAS storage. Marsh and Snavely are both maintaining lists of promising technologies for NAS service to seed more focused research.
Jan-Mar: Marsh set up a small out-of-band gigabit ethernet storage network for developing proof-of-concept for the NAS model.
Apr: Snavely did preliminary benchmarking on the test storage network and found promising results. Marsh and Snavely developed a benchmarking plan.
May: Marsh began to set up a NAS benchmarking environment using scripts provided by Snavely and with assistance from Farber.
May: Marsh identified a new storage vendor and RAID system that is being used successfully in other campus departments.
June: Marsh continued benchmarking setup work and revised some testing methods to ensure accuracy.
July: Marsh, with assistance from Farber and Snavely, finished the benchmarking setup and began running iterative performance tests.
Aug: Marsh completed benchmarking and prepared the results: running DLXS over NAS works. The performance cost, which we assume is partially due to the high CPU load of running the test itself, was less than 10%--small enough for us to move ahead with our plans.
Oct 2004: Marsh configured the server earlier earmarked for web service mirroring as the storage server, and installed and configured 3TB of new storage.
Oct 2004: Snavely began researching high-end managed switches to underlie the first storage network.

Redundant web services.

Oct 2003: Marsh and Snavely made plans to try to use the storage subsystem originally purchased for dlps10 on quik.
Jan-Mar: Marsh upgraded quik to Fedora Core 1 and installed on it the storage subsystem originally purchased for dlps10.
Jan-Mar: Snavely developed a process for synchronizing production data from dlps10 to quik and began network transfers, but began to discover problems with the storage subsystem, and started a series of tests.
Apr: Marsh and Snavely conducted additional tests on the storage subsystem, concluded it was faulty, and began the process of replacement with the vendor.
May: Marsh shipped the faulty unit back to the vendor for repair, but it was damaged in transit. The vendor was able to confirm the fault and supply a workaround--that we simply not use a particular drive bay. The unit was returned and will be used for a different project.
Aug: Marsh and Snavely re-strategized and targeted a brand-new server for use as a mirror server.
Sep 2004: Marsh installed the new server hardware in room 10.
Oct 2004: Marsh reconfigured the server as a storage server (above) and ordered a new, faster mirror server.

Integration of COSIGN single sign-on web authentication.

Jan-Mar: Snavely coordinated meetings with Prettyman and Steinberg to inventory existing web applications that use authentication and develop a transition plan for moving to COSIGN.
Apr: Snavely presented the transition plan to NISC, and it was approved.
June: Snavely met with representatives from NISC to adjust the implementation timeline and discuss the provision of self-service for non-umich patrons.
Aug: Snavely, with approval from NISC, adjusted Library Authentication processes to default to the use of Kerberos authentication and made several user interface tweaks consistent with that change.
Oct 2004: Snavely, working with Prettyman, adapted scripts in Library Authentication to allow guest patrons to authentication. The implementation is done in such a way that it will be consistent with COSIGN Friend.

Production server replacements/upgrades.

Sep 2003: Marsh installed and configured dlps10, a replacement server for dlps4. Marsh, with occasional assistance from Snavely, spent a substantial amount of time troubleshooting persistent problems with configuring the storage subsystem before putting the storage online.
Oct 2003: Marsh transitioned HTI services from dlps4 to dlps10, but after several days of full production load, Marsh and Snavely found problems with the storage subsystem and had to revert HTI service back to dlps4. Marsh began the purchase process for a Sun storage subsystem (at substantially higher cost) that should offer better support and allow the transition to dlps10 to happen as soon as possible.
Oct 2003: Marsh, in coordination with CAEN, formulated an upgrade plan for servers which are still running Solaris 2.6, which CAEN will no longer support starting in 2004.
Nov-Dec: Marsh transitioned HTI services from dlps4 to dlps10, using newly-installed Sun storage.
Nov-Dec: Marsh upgraded dlps5 to Solaris 8 and supported Steinberg in migrating proxy services to ting, a replacement server. dlps5 is online as a hot spare.
Jan-Mar: In coordination with Systems and Web Services, Marsh upgraded betty to Solaris 8, supported the transition of WebZ to betty, and then upgraded myrna to Solaris 8 and supported the transition of library web service to myrna. hank was then upgraded to Solaris 8 and left online as a hot spare.
Jan-Mar: Snavely and March upgraded dlps6 to Solaris 8.
June: Marsh ordered replacement servers for dlps7 and myrna and additional storage for dlps8 and quik.
Aug: March installed the new storage on dlps8.
Aug: Snavely diverted the replacement server for dlps7 to be used with ARC, a new reporting product that is a companion to Aleph, and the storage to be used for our first NAS configuration.

Development server replacements/upgrades.

Sep 2003: Marsh upgraded ferment to Red Hat 9.
Oct 2003: Marsh and Snavely spent a substantial amount of time pursuing what appeared to be a problem with a storage subsystem on sangria, but what turned out to be a bug in file system diagnostics. Marsh upgraded all potentially affected servers to newer versions of the diagnostic software.
Aug: Marsh upgraded ferment's storage from 70 GB to 140 GB.
Aug: Marsh repurposed the old library web server as vodka, a new Solaris development environment for DLPS.
Sep 2004: Marsh installed additional software on vodka for DLPS users.
Sep 2004: Snavely moved the development Oracle instance on dev to vodka, and Marsh retired dev.

DHCP/DNS services.

Sep 2003: Marsh, working with ITCS, transitioned DNS service from longjing to plumwine.
Oct 2003: Marsh, working with DSS, formulated a plan to cut over DHCP services from stella to sake during the upcoming holiday break.
Aug: Marsh, working with DSS, reformulated a plan to gradually move to ITCS-managed DHCP.

Image and OCR processing workflows.

Sep 2003: Snavely resolved a problem with DVD-R volume name recognition on lemonade, the new loading workstation, and signalled collection maintainers that we are ready to begin accepting tests of page image and continuous tone material submitted on DVD-R media.
Sep 2003: Snavely resolved reliability problems with data loading on lemonade related to the use of ssh, incorporated compression into the load process to marginally reduce bandwidth requirements, and added two additional CD-ROM drives for a total of two DVD-ROM drives and three CD-ROM drives.
Sep 2003: Snavely began configuring Samba-based file service on martini, but encountered persistent authentication problems.
Oct 2003: Snavely coordinated the purchase and installation of the library's first Gigabit Ethernet (GbE) switch for improved file service performance on martini. Initial testing confirmed substantial performance improvements, topping out at approximately 22MB/s sustained throughput.
Oct 2003: Snavely resolved the authentication problems with Samba and worked with Hall to begin live testing of file service on martini.
Oct 2003: Snavely took bill-ko, the old data loading workstation, offline.
Nov-Dec: Snavely began to develop revised OCR server configuration and scripting to support a load-balanced workflow.
Jan-Mar: Snavely fininshed setup work and scripting and, in coordination with Hall, activated the new load-balanced workflow.
Jan-Mar: Snavely developed scripts to load page images to the OCR server from Patterson's existing load processes.
Apr: Snavely revised scripts used for contone image processing to handle contone page images.
July: Snavely revised scripts used for contone image processing to increase reliability and accommodate collections that contain only contone page images.
July: Snavely revised loading scripts to automatically enqueue pages for OCR in the load-balanced OCR environment as they are loaded by Patterson's load processes.
Aug: Snavely reworked OCR loading scripts to generate logs and to output daily load summaries via email.

Digital object integrity.
Cost model.

Documentation.

Jan-Mar: Marsh and Snavely developed an outline of documentation to be gradually populated.
Jan-Mar: Marsh updated existing Linux and Perl installation documentation.
July: Snavely developed a server documentation template, using one of the LMS servers as an example.
Aug: Marsh added more information to the server documentation template.
Oct 2004: Patterson prepared draft documentation of her work processes.

Security assessment.
Technology trends awareness.

Support and Development for Access Systems

Library management system implementation.

Oct 2003: Marsh reinstalled Solaris 9 on gracie after a security incident. Marsh took this opportunity to implement several new security measures, including the Sunscreen host-based firewall, to substantially increase overall system security.
Dec: Marsh ordered the new development server, storage, and two CPU/memory boards to upgrade the current server to production capacity.
Jan-Mar: Marsh purchased a rack and coordinated the installation of power, networking, and a repurposed terminal server to the new ALDC space.
Jan-Mar: Marsh added two CPU/memory boards to gracie, bringing it to production capacity.
Jan-Mar: Marsh repurposed dlps4 as mabel, a dedicated training server.
Jan-Mar: Snavely configured CVS for version control of files related to OPAC customization.
Apr: Marsh installed and configured clyde, a dedicated development server. Snavely configured the storage subsystem.
May: Marsh coordinated moving gracie to the Arbor Lakes Data Center, with support from Prettyman and Rothman.
May: Marsh adjusted Sun hardware support agreements to include clyde and reflect hardware upgrades on gracie.
May: Marsh installed several Perl modules for use during conversion.
June: Snavely coordinated OPAC stress testing, with assistance from Goldberg, Marsh, and Steinberg.
June: Snavely installed and configured CVS on clyde, moved the central repository to clyde, and revised the CVS configuration on gracie and mabel to support an automated release process for changes from the central repository.
July: Snavely installed a small out-of-band management network at the Arbor Lakes Data Center to facilitate remote management and wrote monitoring scripts for one of the storage subsystems and for license usage.
July: Marsh set up additional monitoring software on gracie and removed the AFS configuration from all LMS servers to avoid potential impact from AFS outages.
July: Snavely resumed discussions with ITCS MAIS to decide on the details of a support agreement for the servers installed at Arbor Lakes and their Oracle instances, respectively.
Aug: Marsh removed AFS software from LMS servers to decrease complexity.
Aug: Marsh, with assistance from Plantinga, moved the SFX/Metalib server tequila to Arbor Lakes and upgraded its OS and storage to prepare for Metalib 3.0 installation. Snavely coordinated Metalib installation with Ex Libris.
Aug: Snavely adjusted backup scripts to avoid potential overflows of our tape backup service quota.
Sep 2004: Marsh installed ethel, a new Aleph Reporting Center (ARC) server and began pursuing quotes for storage.
Oct 2004: Marsh and Snavely decided on a configuration for ARC storage, with information from Ex Libris, and Marsh placed an order.
Oct 2004: Snavely set up automated Metalib backups.

DLXS.

Aug: Snavely, working with Stephenson and Willett, contributed to an analysis of our digitization and archival practices with respect to guidelines for trusted digital repositories.

Persistent URLs.
Institutional repository initiative.

Oct 2003: Snavely reviewed Ottaviani's draft of our institutional repository findings and contributed several comments for changes.
Oct 2003: Marsh researched requirements for installing DSpace on sambuca, discovering some difficulty and chains of dependency. Snavely and Ottaviani agreed to postpone DSpace installation until after the holidays, while in the meantime putting sambuca to use as a NAS test server.
Jan-Mar: Marsh installed a vanilla DSpace instance on sambuca for testing.
Apr: Snavely added several user accounts to the DSpace server.
Sep 2004: Marsh, in coordination with Blanco, upgraded the DSpace software on sambuca to the latest version.

Collaborating with Other Areas of the Library and the University

Library/Core Services coordination.
Integration with campus authentication.

Miscellaneous/Unplanned

Sep 2003: Marsh installed the Perl module Net::Z3950, the Yaz tool kit, and Zebra for the adjunct files project.
Jan-Mar: Marsh installed and configured a new terminal server in room 10, and moved the existing terminal server to ALDC.
Jan-Mar: Snavely and Blanco spent a significant amount of time reloading statistics and reworking statistics automation due to a problem with tabulation that occurred in the fall.
Apr: Snavely and Blanco resumed normal statistics processing.
May: Marsh began testing ITCS mail with new, higher storage quotas in preparation for shutting down CS-administered email service,
May: Marsh began testing mod_throttle configuration for potential use in production. If successful, this software would prevent the overuse/abuse of public text services we frequently observe.
June: Marsh reconfigured the subsystem that was originally purchased for dlps4 for use with the Shoah Visual History Foundation cache server.
June: Snavely began meeting with Stephenson and Willett to begin identifying areas of improvement for potential compliance with developing guidelines for trusted digital repositories.
July: Snavely, in consultation with Hagedorn and Willett, installed two new OAI harvesters, MyOAI and the Net::OAI::Harvester API, for evaluation by DLPS.
Aug: Snavely repurposed quik as a local cache server for the Shoah Visual History Foundation project, installing the storage array originally purchased for dlps4 and coordinating the cache software installation with the foundation.
Aug: Snavely reworked the patron load for Library Authentication to feed from Aleph rather than NOTIS.
Sep 2004: Marsh began transitioning users from CS mail services to new ITCS mail services in preparation for shutting down the former.
Sep 2004: Snavely began research options for providing secured network access to out-of-band management devices in our three server rooms.
Oct 2004: Marsh and Snavely met with Walt Reynolds from ITCS to seek recommendations on VPN hardware devices.
Oct 2004: Marsh helped 1/3 of the users of our email service to ITCS-provided email in preparation for retiring our email server.
Oct 2004: Marsh ordered and installed data communications adapters to restore console access to several servers that had been temporarily removed from the console server.
Oct 2004: Snavely made the final preparations to quik, the Shoah Visual History Foundation cache server, for production service: configuring the firewall and Samba to allow access from anywhere on campus.

System Performance

Arbor Lakes server environment

ethel.umdl (ARC server): No down time
gracie.umdl (Aleph server): No down time
hiram.umdl (Zebra server): No down time
tequila.umdl (Metalib and SFX server): No down time

Central Campus server environment

betty.umdl (MIRLYNWeb server): No down time
coffee.umdl (Oracle server for access control): No down time
gin.umdl (WebCheckout server): No down time
myrna.umdl (Library Web server): No down time
ting.umdl (library proxy server): No down time

North Campus server environment: All text and image services were down on Thursday, October 21 from 1:00pm to 1:40pm due to a brief network outage that triggered a database problem.

dlps5.umdl (backup library proxy server): No down time
dlps6.umdl (Oracle server for access control and usage statistics): No down time
dlps7.umdl (Numeric Data server): No down time
dlps8.umdl (non-public collections): No additional down time
dlps9.umdl (Image Services): No additional down time
dlps10.umdl (primary web server for DLPS and public collections): No additional down time

Data Loading and Archival Statistics

CD/DVDs loaded or reloaded (bitonal page images)	125
CD/DVDs loaded or reloaded and processed (contone images)	503
CD/DVDs created and catalogued (contone images)	0
CD/DVDs duplicated and stored (library)	43
CD/DVDs duplicated (ITCS)	0
CD/DVDs duplicated (other)	2