Core Services Cumulative Monthly Report, February 2008

Previous monthly reports

• Content/Web infrastructure
• Develop/support SDR hosting environment.
• Jan: Feeman, Korner, and Snavely continued software installation and configuration, including a new Apache build, numerous adaptations to GROOVE, MySQL (with replication), TSM, and usage statistics.
• Feb: Feeman, Korner, and Snavely finished all preparations to the new hosting environment, worked with DLPS staff to make a few small changes in the MBooks page turner to accommodate a few differences there, and set a tentative timeline for switching service to the new servers.
• Develop/support enhancements to Library Web Services hosting environment.
• Feb: Korner began planning discussions with Web Services on various software upgrades to be made to the library development web server.
• Finish transition to CoSign authentication.
• Replace web load-balancing system.
• Roll out new web-based authorization management system.
• Begin migration of Library Web Services to networked storage.
• Begin migration of DLPS development services to networked storage.
• Continue work with ITCS on Shibboleth origin pilot.
• Migrate all web service to Apache 2.
• Repository/Storage
• Finish migration to new Pillar Axiom SAN in ALDC.
• Jan: Snavely continued to monitor performance, and contacted Pillar about approaches for adding redundancy to the current storage configuration.
• Feb: Snavely repurposed a Sun StorEdge 3310 storage array freed by the move to the SAN to hold a redundant copy of Oracle redo logs.
• Implement snapshot-based database backups.
• Jan: Korner began testing snapshot features and mechanisms for creating and mounting snapshots from the operating system, and sent a query to Pillar for additional information.
• Feb: Korner continued experimentation with mounting snapshots.
• Install and configure new Isilon NAS systems and migrate SDR services.
• Jan: Korner upgraded the software on both Isilon clusters. Snavely finished the first pass of batch synchronization, and kept it running on a regular schedule to keep files in synchronization as items were added to the repository. Korner and Snavely investigated a problem of excessive space consumption which turned out to be related to the large number of very small files in the repository. Consulting with DLPS staff, an alternative file layout using a single ZIP file for each item was developed and planning began for using the new layout. Korner did initial experimentation with synchronizing between Isilon clusters, but postponed the work pending the change to the ZIP file layout.
• Feb: Feeman and Snavely stopped the ingest process and did a final synchronization of all data and several checks to verify that all was intact while DLPS staff finished development work to support the new file layout using ZIP files. Feeman moved the ingest servers to the MACC and adapted GROOVE to ingest new volumes using the new ZIP file layout, and adapted the module that creates the METS file to include a reference to the ZIP file and to create PREMIS markup (and update it on subsequent ingests). Feeman developed a script that uses the new functionality in GROOVE to convert volumes in the repository to the new ZIP layout with PREMIS markup. Korner, working with Syrigos, configured support for ActiveDirectory authentication for the file share that quality review staff use.
• Develop load process for SDR materials not submitted via GRIN.
• Feb: Feeman began adapting GROOVE to support multiple namespaces, which will be required for ingesting materials from other institutions.
• System administration
• Transition remaining servers to TSM backups; phase out Arkeia backups; perform testing and training on GUI-based recovery.
• Jan: This work was mostly completed in December; the Arkeia backup server will be retired in March.
• Improve remote access capabilities to server resources using serial console, network KVM (for blades), and VPN device(s).
• Jan: Korner experimented with serial port redirection and remote management functionality and will use the results as a model for new servers as well as for reconfiguring existing servers as opportunities arise.
• Standardize, document, and build mechanisms to centralize system administration processes.
• Feb: Korner did some initial research into available tools for centralized system administration.
• Replace production and development servers as scheduled, consolidating services and/or using virtualization as appropriate.
• Jan: Snavely continued work to upgrade the web authorization (Oracle) servers, focusing on backups, load balancing, and performance scaling configuration.
• Jan: Korner installed and configured a new SearchTools server and connected it to the SAN.
• Feb: Korner received and began installing and configuring pocarisweat, a new Deep Blue server.
• Feb: Feeman and Korner upgraded the Verde server to RHEL 4 and connected it to the SAN in preparation for an upgrade to the Verde software.
• Relocate servers as appropriate for availability, redundancy, and/or proximity to specific storage systems.
• Feb: Korner prepared an estimate of electrical usage at Arbor Lakes for data center management staff.
• Transition to ITCS-supported DHCP, and transition DNS service off plumwine using DHCP lease configuration.
• Feb: DSS indicated plans to move to a locally-run Windows-based DHCP service.
• Explore centralized storage for sharing among all LIT units.
• Security
• Complete security planning and perform TBD high-priority tasks.
• Feb: Munce and Snavely revisited the status of security planning and reviewed the results of a central security audit. A meeting with ITSS will be scheduled to determine additional required work on security planning.
• Investigate virtual firewall service and request/implement as appropriate.
• Feb: Korner scheduled meetings with ITCom staff to discuss the feasibility of using the virtual firewall service for ALDC and MACC server installations.
• Miscellaneous, routine, or unplanned activity
• Jan: Korner made changes to DNS server configuration to address a security vulnerability.
• Jan: Feeman generated some statistics for Kleinman and Santelli regarding SDR contents, customized GROOVE to intercept a specific set of volumes and generate QA samples differently (for the Parsons Rau project), and manually ingested several volumes into the SDR for demonstrations during the millionth book celebration.
• Feb: Korner and Snavely replaced two of four unmetered PDUs at ALDC with managed, metered PDUs to allow for better balancing and monitoring of electrical loads.
• Feb: Working with Longstreth and Green, Snavely copied data from the old Geospatial/Numeric Data server in preparation for its retirement.
• Feb: Munce and Snavely conferred on design ideas for upgrades to the Hatcher Server Room, and asked Hipkiss to arrange a meeting with campus engineers, as planned.
• Feb: Snavely discussed the security architecture of the Espresso Book Machine with a technology representative from the manufacturer, and reported the results of the conversation to Bonn and Wilkin.
• Feb: Feeman, working with Karle-Zenith, created samples of approximately 5000 volumes for a focused QA effort on large-format volumes.
• Feb: Feeman continued coordination between CAS, ITCS, and the Treasurer's office regarding PCI certification for the ILL e-commerce application.
• Service Availability
• Arbor Lakes Data Center (ALDC)
• ethel.umdl (ARC): No down time
• ezra.umdl (Verde): No down time
• gracie.umdl (Aleph): Down on Sunday, February 17 from 8:00am to 9:00am to apply security updates.
• hiram.umdl (Zebra): No down time
• tequila.umdl (Metalib and SFX): Down on Saturday, February 23 from 8:00am to 8:15am for scheduled maintenance.
• Duderstadt Center Machine Room (DCMR):
• dlps6.umdl (Oracle): No down time
• dlps7.umdl (Numeric and Geospatial Data): No down time
• dlps11.umdl (DLPS collections): No down time
• dlps12.umdl (DLPS collections): No down time
• soymilk.umdl (MySQL): No down time
• Hatcher Server Room
• belle.umdl (Library Web): No down time
• coffee.umdl (Oracle): No down time
• cowmilk.umdl (MySQL): No down time
• hefeweizen.umdl (DLPS collections and MBooks): MBooks service unavailable on Tuesday, February 12 from approximately 10:00pm to 11:45pm due to a problem with a storage system.
• merlot.umdl (DLPS collections and MBooks): See hefeweizen.umdl
• porter.umdl (DLPS collections and MBooks): See hefeweizen.umdl
• quik.umdl (Survivors of the Shoah cache): No down time
• ting.umdl (Library Proxy): No down time
• Michigan Advanced Computing Center (MACC):
• sambuca.umdl (Deep Blue): No down time
• Data Loading/Archival Statistics

  CD/DVDs loaded or reloaded (bitonal page images)	3
  CD/DVDs loaded or reloaded and processed (contone images)	31
  Removable drives loaded or reloaded	2
  CD/DVDs burnt or duplicated	39