Core Services Cumulative Monthly Report

Core Services Cumulative Monthly Report, August 2004

Refining and Improving Core Infrastructure

New storage model.

Sep: Snavely began researching the Linux Logical Volume Management system as a potential management suite for NAS storage. Marsh and Snavely are both maintaining lists of promising technologies for NAS service to seed more focused research.
Jan-Mar: Marsh set up a small out-of-band gigabit ethernet storage network for developing proof-of-concept for the NAS model.
Apr: Snavely did preliminary benchmarking on the test storage network and found promising results. Marsh and Snavely developed a benchmarking plan.
May: Marsh began to set up a NAS benchmarking environment using scripts provided by Snavely and with assistance from Farber.
May: Marsh identified a new storage vendor and RAID system that is being used successfully in other campus departments.
June: Marsh continued benchmarking setup work and revised some testing methods to ensure accuracy.
July: Marsh, with assistance from Farber and Snavely, finished the benchmarking setup and began running iterative performance tests.
Aug: Marsh completed benchmarking and prepared the results: running DLXS over NAS works. The performance cost, which we assume is partially due to the high CPU load of running the test itself, was less than 10%--small enough for us to move ahead with our plans.

Redundant web services.

Oct: Marsh and Snavely made plans to try to use the storage subsystem originally purchased for dlps10 on quik.
Jan-Mar: Marsh upgraded quik to Fedora Core 1 and installed on it the storage subsystem originally purchased for dlps10.
Jan-Mar: Snavely developed a process for synchronizing production data from dlps10 to quik and began network transfers, but began to discover problems with the storage subsystem, and started a series of tests.
Apr: Marsh and Snavely conducted additional tests on the storage subsystem, concluded it was faulty, and began the process of replacement with the vendor.
May: Marsh shipped the faulty unit back to the vendor for repair, but it was damaged in transit. The vendor was able to confirm the fault and supply a workaround--that we simply not use a particular drive bay. The unit was returned and will be used for a different project.
Aug: Marsh and Snavely re-strategized and targeted a brand-new server for use as a mirror server.

Integration of COSIGN single sign-on web authentication.

Jan-Mar: Snavely coordinated meetings with Prettyman and Steinberg to inventory existing web applications that use authentication and develop a transition plan for moving to COSIGN.
Apr: Snavely presented the transition plan to NISC, and it was approved.
June: Snavely met with representatives from NISC to adjust the implementation timeline and discuss the provision of self-service for non-umich patrons.
Aug: Snavely, with approval from NISC, adjusted Library Authentication processes to default to the use of Kerberos authentication and made several user interface tweaks consistent with that change.

Production server replacements/upgrades.

Sep: Marsh installed and configured dlps10, a replacement server for dlps4. Marsh, with occasional assistance from Snavely, spent a substantial amount of time troubleshooting persistent problems with configuring the storage subsystem before putting the storage online.
Oct: Marsh transitioned HTI services from dlps4 to dlps10, but after several days of full production load, Marsh and Snavely found problems with the storage subsystem and had to revert HTI service back to dlps4. Marsh began the purchase process for a Sun storage subsystem (at substantially higher cost) that should offer better support and allow the transition to dlps10 to happen as soon as possible.
Oct: Marsh, in coordination with CAEN, formulated an upgrade plan for servers which are still running Solaris 2.6, which CAEN will no longer support starting in 2004.
Nov-Dec: Marsh transitioned HTI services from dlps4 to dlps10, using newly-installed Sun storage.
Nov-Dec: Marsh upgraded dlps5 to Solaris 8 and supported Steinberg in migrating proxy services to ting, a replacement server. dlps5 is online as a hot spare.
Jan-Mar: In coordination with Systems and Web Services, Marsh upgraded betty to Solaris 8, supported the transition of WebZ to betty, and then upgraded myrna to Solaris 8 and supported the transition of library web service to myrna. hank was then upgraded to Solaris 8 and left online as a hot spare.
Jan-Mar: Snavely and March upgraded dlps6 to Solaris 8.
June: Marsh ordered replacement servers for dlps7 and myrna and additional storage for dlps8 and quik.
Aug: March installed the new storage on dlps8.
Aug: Snavely diverted the replacement server for dlps7 to be used with ARC, a new reporting product that is a companion to Aleph, and the storage to be used for our first NAS configuration.

Development server replacements/upgrades.

Sep: Marsh upgraded ferment to Red Hat 9.
Oct: Marsh and Snavely spent a substantial amount of time pursuing what appeared to be a problem with a storage subsystem on sangria, but what turned out to be a bug in file system diagnostics. Marsh upgraded all potentially affected servers to newer versions of the diagnostic software.
Aug: Marsh upgraded ferment's storage from 70 GB to 140 GB.
Aug: Marsh repurposed the old library web server as vodka, a new Solaris development environment for DLPS.

DHCP/DNS services.

Sep: Marsh, working with ITCS, transitioned DNS service from longjing to plumwine.
Oct: Marsh, working with DSS, formulated a plan to cut over DHCP services from stella to sake during the upcoming holiday break.
Aug: Marsh, working with DSS, reformulated a plan to gradually move to ITCS-managed DHCP.

Image and OCR processing workflows.

Sep: Snavely resolved a problem with DVD-R volume name recognition on lemonade, the new loading workstation, and signalled collection maintainers that we are ready to begin accepting tests of page image and continuous tone material submitted on DVD-R media.
Sep: Snavely resolved reliability problems with data loading on lemonade related to the use of ssh, incorporated compression into the load process to marginally reduce bandwidth requirements, and added two additional CD-ROM drives for a total of two DVD-ROM drives and three CD-ROM drives.
Sep: Snavely began configuring Samba-based file service on martini, but encountered persistent authentication problems.
Oct: Snavely coordinated the purchase and installation of the library's first Gigabit Ethernet (GbE) switch for improved file service performance on martini. Initial testing confirmed substantial performance improvements, topping out at approximately 22MB/s sustained throughput.
Oct: Snavely resolved the authentication problems with Samba and worked with Hall to begin live testing of file service on martini.
Oct: Snavely took bill-ko, the old data loading workstation, offline.
Nov-Dec: Snavely began to develop revised OCR server configuration and scripting to support a load-balanced workflow.
Jan-Mar: Snavely fininshed setup work and scripting and, in coordination with Hall, activated the new load-balanced workflow.
Jan-Mar: Snavely developed scripts to load page images to the OCR server from Patterson's existing load processes.
Apr: Snavely revised scripts used for contone image processing to handle contone page images.
July: Snavely revised scripts used for contone image processing to increase reliability and accommodate collections that contain only contone page images.
July: Snavely revised loading scripts to automatically enqueue pages for OCR in the load-balanced OCR environment as they are loaded by Patterson's load processes.
Aug: Snavely reworked OCR loading scripts to generate logs and to output daily load summaries via email.

Digital object integrity.
Cost model.

Documentation.

Jan-Mar: Marsh and Snavely developed an outline of documentation to be gradually populated.
Jan-Mar: Marsh updated existing Linux and Perl installation documentation.
July: Snavely developed a server documentation template, using one of the LMS servers as an example.
Aug: Marsh added more information to the server documentation template.

Security assessment.
Technology trends awareness.

Support and Development for Access Systems

Library management system implementation.

Oct: Marsh reinstalled Solaris 9 on gracie after a security incident. Marsh took this opportunity to implement several new security measures, including the Sunscreen host-based firewall, to substantially increase overall system security.
Dec: Marsh ordered the new development server, storage, and two CPU/memory boards to upgrade the current server to production capacity.
Jan-Mar: Marsh purchased a rack and coordinated the installation of power, networking, and a repurposed terminal server to the new ALDC space.
Jan-Mar: Marsh added two CPU/memory boards to gracie, bringing it to production capacity.
Jan-Mar: Marsh repurposed dlps4 as mabel, a dedicated training server.
Jan-Mar: Snavely configured CVS for version control of files related to OPAC customization.
Apr: Marsh installed and configured clyde, a dedicated development server. Snavely configured the storage subsystem.
May: Marsh coordinated moving gracie to the Arbor Lakes Data Center, with support from Prettyman and Rothman.
May: Marsh adjusted Sun hardware support agreements to include clyde and reflect hardware upgrades on gracie.
May: Marsh installed several Perl modules for use during conversion.
June: Snavely coordinated OPAC stress testing, with assistance from Goldberg, Marsh, and Steinberg.
June: Snavely installed and configured CVS on clyde, moved the central repository to clyde, and revised the CVS configuration on gracie and mabel to support an automated release process for changes from the central repository.
July: Snavely installed a small out-of-band management network at the Arbor Lakes Data Center to facilitate remote management and wrote monitoring scripts for one of the storage subsystems and for license usage.
July: Marsh set up additional monitoring software on gracie and removed the AFS configuration from all LMS servers to avoid potential impact from AFS outages.
July: Snavely resumed discussions with ITCS MAIS to decide on the details of a support agreement for the servers installed at Arbor Lakes and their Oracle instances, respectively.
Aug: Marsh removed AFS software from LMS servers to decrease complexity.
Aug: Marsh, with assistance from Plantinga, moved the SFX/Metalib server tequila to Arbor Lakes and upgraded its OS and storage to prepare for Metalib 3.0 installation. Snavely coordinated Metalib installation with Ex Libris.
Aug: Snavely adjusted backup scripts to avoid potential overflows of our tape backup service quota.

DLXS.

Aug: Snavely, working with Stephenson and Willett, contributed to an analysis of our digitization and archival practices with respect to guidelines for trusted digital repositories.

Persistent URLs.
Institutional repository initiative.

Oct: Snavely reviewed Ottaviani's draft of our institutional repository findings and contributed several comments for changes.
Oct: Marsh researched requirements for installing DSpace on sambuca, discovering some difficulty and chains of dependency. Snavely and Ottaviani agreed to postpone DSpace installation until after the holidays, while in the meantime putting sambuca to use as a NAS test server.
Jan-Mar: Marsh installed a vanilla DSPace instance on sambuca for testing.
Apr: Snavely added several user accounts to the DSpace server.

Collaborating with Other Areas of the Library and the University

Library/Core Services coordination.
Integration with campus authentication.

Miscellaneous/Unplanned

Sep: Marsh installed the Perl module Net::Z3950, the Yaz tool kit, and Zebra for the adjunct files project.
Jan-Mar: Marsh installed and configured a new terminal server in room 10, and moved the existing terminal server to ALDC.
Jan-Mar: Snavely and Blanco spent a significant amount of time reloading statistics and reworking statistics automation due to a problem with tabulation that occurred in the fall.
Apr: Snavely and Blanco resumed normal statistics processing.
May: Marsh began testing ITCS mail with new, higher storage quotas in preparation for shutting down CS-administered email service,
May: Marsh began testing mod_throttle configuration for potential use in production. If successful, this software would prevent the overuse/abuse of public text services we frequently observe.
June: Marsh reconfigured the subsystem that was originally purchased for dlps4 for use with the Shoah Visual History Foundation cache server.
June: Snavely began meeting with Stephenson and Willett to begin identifying areas of improvement for potential compliance with developing guidelines for trusted digital repositories.
July: Snavely, in consultation with Hagedorn and Willett, installed two new OAI harvesters, MyOAI and the Net::OAI::Harvester API, for evaluation by DLPS.
Aug: Snavely repurposed quik as a local cache server for the Shoah Visual History Foundation project, installing the storage array originally purchased for dlps4 and coordinating the cache software installation with the foundation.
Aug: Snavely reworked the patron load for Library Authentication to feed from Aleph rather than NOTIS.

System Performance

Central Campus server environment

betty.umdl (MIRLYNWeb server): No down time
coffee.umdl (Oracle server for access control): No down time
gin.umdl (WebCheckout server): No down time
gracie.umdl (Aleph server): Down on Sunday, August 1 from 5:00am to 5:15am for database tuning; down on Monday, August 16 from 4:30pm to 6:15pm due to a cabling problem; service degraded on Thursday, August 26 from 11:40am to 1:47pm due to an accidental deletion of configuraton data; down on Monday, August 23 through Friday, August 27 and on Monday, August 30 through Friday, September 3 from 7:15am to 7:30am for routine maintenance.
myrna.umdl (Library Web server): No down time
tequila.umdl (Metalib and SFX server): Down Wednesday, August 25 from 7:00am to 5:30pm to be physically moved to Arbor Lakes and for a storage upgrade; down on Sunday, August 29 from 11:00am to 12:00pm for an operating system upgrade; down on Monday, August 30 from 2:00pm to 4:00pm to complete the operating system, upgrade.
ting.umdl (library proxy server): No down time

North Campus server environment: All text and image services were down on Sunday, August 1 from 1:00pm to 1:15pm as additional storage was added to a database server.

dlps5.umdl (backup library proxy server): No down time
dlps6.umdl (Oracle server for access control and usage statistics): Usage statistics were down from an unknown time on July 31 to Wednesday, August 4 at 1:45pm due to a database problem.
dlps7.umdl (Numeric Data server): No additional down time
dlps8.umdl (non-public collections): No additional down time
dlps9.umdl (Image Services): No additional down time
dlps10.umdl (primary web server for DLPS and public collections): Also down on Sunday, August 1 from 7:00pm to 9:30pm due to networking problems in our north campus server environment.

Data Loading and Archival Statistics

CD/DVDs loaded or reloaded (bitonal page images)	218
CD/DVDs loaded or reloaded and processed (contone images)	269
CD/DVDs created and catalogued (contone images)	76
CD/DVDs duplicated and stored (library)	4
CD/DVDs duplicated (ITCS)	6
CD/DVDs duplicated (other)	31