Core Services Cumulative Monthly Report, March 2008

Previous monthly reports

• Content/Web infrastructure
• Develop/support SDR hosting environment.
• Jan: Feeman, Korner, and Snavely continued software installation and configuration, including a new Apache build, numerous adaptations to GROOVE, MySQL (with replication), TSM, and usage statistics.
• Feb: Feeman, Korner, and Snavely finished all preparations to the new hosting environment, worked with DLPS staff to make a few small changes in the MBooks page turner to accommodate a few differences there, and set a tentative timeline for switching service to the new servers.
• Mar: Feeman, Korner, and Snavely coordinated with DLPS staff to transition MBooks service to the new servers and to discuss the specifics of Solr infrastructure requirements, and Korner began researching Solr configuration options.
• Develop/support enhancements to Library Web Services hosting environment.
• Feb: Korner began planning discussions with Web Services on various software upgrades to be made to the library development web server.
• Mar: Snavely worked with Varnum to finalize the list of software upgrades and installations for the library development web server, and Korner worked with Web Services to set a time window for the work.
• Mar: Korner began attending meetings of the Web Team Technical Advisory Group, which will focus on selecting a CMS for the library.
• Finish transition to CoSign authentication.
• Mar: The web server configuration for the new SDR hosting infrastructure includes direct CoSign integration; in fact, the library authorization system is not currently in use in that environment. The configuration planned for the development library web server will use direct CoSign integration as well, including the new rewritten library authorization.
• Replace web load-balancing system.
• Roll out new web-based authorization management system.
• Begin migration of Library Web Services to networked storage.
• Begin migration of DLPS development services to networked storage.
• Continue work with ITCS on Shibboleth origin pilot.
• Mar: ITCS is re-assigning responsibility for Shibboleth work, and has posted a new position; ProQuest has announced readiness to begin Shibboleth work. Snavely will work with Montague to establish contacts with the new developer and continue work.
• Migrate all web service to Apache 2.
• Repository/Storage
• Finish migration to new Pillar Axiom SAN in ALDC.
• Jan: Snavely continued to monitor performance, and contacted Pillar about approaches for adding redundancy to the current storage configuration.
• Feb: Snavely repurposed a Sun StorEdge 3310 storage array freed by the move to the SAN to hold a redundant copy of Oracle redo logs.
• Implement snapshot-based database backups.
• Jan: Korner began testing snapshot features and mechanisms for creating and mounting snapshots from the operating system, and sent a query to Pillar for additional information.
• Feb: Korner continued experimentation with mounting snapshots.
• Mar: Korner continued discussions with Pillar on mounting snapshots.
• Install and configure new Isilon NAS systems and migrate SDR services.
• Jan: Korner upgraded the software on both Isilon clusters. Snavely finished the first pass of batch synchronization, and kept it running on a regular schedule to keep files in synchronization as items were added to the repository. Korner and Snavely investigated a problem of excessive space consumption which turned out to be related to the large number of very small files in the repository. Consulting with DLPS staff, an alternative file layout using a single ZIP file for each item was developed and planning began for using the new layout. Korner did initial experimentation with synchronizing between Isilon clusters, but postponed the work pending the change to the ZIP file layout.
• Feb: Feeman and Snavely stopped the ingest process and did a final synchronization of all data and several checks to verify that all was intact while DLPS staff finished development work to support the new file layout using ZIP files. Feeman moved the ingest servers to the MACC and adapted GROOVE to ingest new volumes using the new ZIP file layout, and adapted the module that creates the METS file to include a reference to the ZIP file and to create PREMIS markup (and update it on subsequent ingests). Feeman developed a script that uses the new functionality in GROOVE to convert volumes in the repository to the new ZIP layout with PREMIS markup. Korner, working with Syrigos, configured support for ActiveDirectory authentication for the file share that quality review staff use.
• Mar: Feeman continued converting volumes to the new ZIP file layout and reworked scripts and recreated some METS files to address problems with PREMIS markup caused by missing or inaccurate dates supplied by Google.
• Mar: Korner and Snavely worked with Lewis and Frazine from CAEN to test a 3-way NDMP backup of the Isilon system against their Veritas NetBackup system. NDMP backups between Isilon and TSM are not yet supported by both vendors, but the test was successful and is promising for future work.
• Develop load process for SDR materials not submitted via GRIN.
• Feb: Feeman began adapting GROOVE to support multiple namespaces, which will be required for ingesting materials from other institutions.
• Mar: Feeman continued work on namespace support in GROOVE, and began work on ingesting materials from the Historical Mathematics Collection.
• System administration
• Transition remaining servers to TSM backups; phase out Arkeia backups; perform testing and training on GUI-based recovery.
• Jan: This work was mostly completed in December; the Arkeia backup server will be retired in March.
• Mar: Snavely retired the Arkeia server, and Patterson began boxing tapes for shredding.
• Improve remote access capabilities to server resources using serial console, network KVM (for blades), and VPN device(s).
• Jan: Korner experimented with serial port redirection and remote management functionality and will use the results as a model for new servers as well as for reconfiguring existing servers as opportunities arise.
• Standardize, document, and build mechanisms to centralize system administration processes.
• Feb: Korner did some initial research into available tools for centralized system administration.
• Replace production and development servers as scheduled, consolidating services and/or using virtualization as appropriate.
• Jan: Snavely continued work to upgrade the web authorization (Oracle) servers, focusing on backups, load balancing, and performance scaling configuration.
• Jan: Korner installed and configured a new SearchTools server and connected it to the SAN.
• Feb: Korner received and began installing and configuring pocarisweat, a new Deep Blue server.
• Feb: Feeman and Korner upgraded the Verde server to RHEL 4 and connected it to the SAN in preparation for an upgrade to the Verde software.
• Mar: Korner and Snavely, working with Plantinga and Rothman, sought recommended configurations from Ex Libris for replacement Aleph and SFX servers.
• Mar: Snavely continued work to upgrade the web authorization (Oracle) servers, focusing on synchronization.
• Relocate servers as appropriate for availability, redundancy, and/or proximity to specific storage systems.
• Feb: Korner prepared an estimate of electrical usage at Arbor Lakes for data center management staff.
• Mar: Snavely and Wilkin met with Brooks, Inman, and Moynihan to discuss long-term plans and pricing for the ALDC. Based on these discussions, Snavely decided to reserve additional rack space at the MACC and move servers from ALDC and DCMR there; timing and detailed plans for these moves are being discussed.
• Transition to ITCS-supported DHCP, and transition DNS service off plumwine using DHCP lease configuration.
• Feb: DSS indicated plans to move to a locally-run Windows-based DHCP service.
• Explore centralized storage for sharing among all LIT units.
• Security
• Complete security planning and perform TBD high-priority tasks.
• Feb: Munce and Snavely revisited the status of security planning and reviewed the results of a central security audit. A meeting with ITSS will be scheduled to determine additional required work on security planning.
• Investigate virtual firewall service and request/implement as appropriate.
• Feb: Korner scheduled meetings with ITCom staff to discuss the feasibility of using the virtual firewall service for ALDC and MACC server installations.
• Mar: Korner met with ITCom staff regarding features of the virtual firewall service; at this time, the service does not appear to be sufficiently robust for our requirements.
• Miscellaneous, routine, or unplanned activity
• Jan: Korner made changes to DNS server configuration to address a security vulnerability.
• Jan: Feeman generated some statistics for Kleinman and Santelli regarding SDR contents, customized GROOVE to intercept a specific set of volumes and generate QA samples differently (for the Parsons Rau project), and manually ingested several volumes into the SDR for demonstrations during the millionth book celebration.
• Feb: Korner and Snavely replaced two of four unmetered PDUs at ALDC with managed, metered PDUs to allow for better balancing and monitoring of electrical loads.
• Feb: Working with Longstreth and Green, Snavely copied data from the old Geospatial/Numeric Data server in preparation for its retirement.
• Feb: Munce and Snavely conferred on design ideas for upgrades to the Hatcher Server Room, and asked Hipkiss to arrange a meeting with campus engineers, as planned.
• Feb: Snavely discussed the security architecture of the Espresso Book Machine with a technology representative from the manufacturer, and reported the results of the conversation to Bonn and Wilkin.
• Feb: Feeman, working with Karle-Zenith, created samples of approximately 5000 volumes for a focused QA effort on large-format volumes.
• Feb: Feeman continued coordination between CAS, ITCS, and the Treasurer's office regarding PCI certification for the ILL e-commerce application.
• Mar: Feeman submitted another PCI compliance assessment for the ILL e-commerce, and it was successfully granted PCI certification.
• Mar: Feeman, Korner, and Snavely upgraded the firmware and conducted redundancy checks on all JetStor storage arrays in active production.
• Mar: Korner and Snavely replaced the remainder of unmetered PDUs at ALDC with managed, metered PDUs and rerouted cabling for easier management and better airflow.
• Mar: Korner, Munce, and Snavely met with campus contruction engineers to discuss designs and priving for the Hatcher computer room upgrades. Major improvements will consist of 1) expansion into room 9, including security grating and window frosting, 2) a substantial upgrade of cooling capacity using available chilled water, 3) redundancy for the cooling system from the use of multiple smaller components and refrigerant backup, 4) a revised room layout using a hot/cold aisle arrangement to maximize efficiency, 5) simplified power distribution, 6) an enlarged entry door, and 7) as-minimal-as-possible renovation to floor and ceiling tile and other cosmetic elements.
• Service Availability All services may have been unreachable from off-campus on Monday, March 17 from 12:40pm to 1:05pm due to a problem with the campus interconnection to the Internet.
• Arbor Lakes Data Center (ALDC)
• ethel.umdl (ARC): Down on Sunday, March 16 for a software upgrade to a storage subsystem.
• ezra.umdl (Verde): Down on Sunday, March 16 for a software upgrade to a storage subsystem.
• gracie.umdl (Aleph): Down on Sunday, March 2 from 6:30am to 9:30am to detach two retired storage subsystems, reorganize power distribution, and apply security updates and on Sunday, March 16 for a software upgrade to a storage subsystem.
• hiram.umdl (Zebra): No down time
• tequila.umdl (Metalib and SFX): No down time
• Duderstadt Center Machine Room (DCMR):
• dlps6.umdl (Oracle): No down time
• dlps11.umdl (DLPS collections): No down time
• dlps12.umdl (DLPS collections): No down time
• soymilk.umdl (DLPS MySQL): No down time
• Hatcher Server Room
• belle.umdl (Library Web): No down time
• coffee.umdl (Oracle): No down time
• cowmilk.umdl (DLPS MySQL): No down time
• hefeweizen.umdl (DLPS collections): No down time
• merlot.umdl (DLPS collections): No down time
• porter.umdl (DLPS collections): No down time
• quik.umdl (Survivors of the Shoah cache): No down time
• ting.umdl (Library Proxy): No down time
• Michigan Advanced Computing Center (MACC):
• assam.umdl (MBooks/SDR): No down time
• bancha.umdl (MBooks/SDR): No down time
• dongding.umdl (MBooks/SDR MySQL): No down time
• jasmine.umdl (MBooks/SDR MySQL): No down time
• sambuca.umdl (Deep Blue): No down time
• Data Loading/Archival Statistics

  CD/DVDs loaded or reloaded (bitonal page images)	14
  CD/DVDs loaded or reloaded and processed (contone images)	24
  Removable drives loaded or reloaded	2
  CD/DVDs burnt or duplicated	27