Core Services Cumulative Monthly Report, April 2007
Content/Web infrastructure
Primary objectives
Finish migration of DLPS web services to redundant networked
storage.
Dec: Marsh installed new storage on the Hatcher storage
server, and Snavely began batch synchronizing ETS and Image Services
content to it.
Jan: Snavely completed batch synchronization of ETS and
Image Services content to the Hatcher storage server and put routine
synchronization routines in place to keep it up to date. Marsh upgraded
the storage array from dlps8 (which was retired) with new disks and
current firmware, installed it on on the Duderstadt Center storage
server, and Snavely began batch synchronizing HTI content to it.
Feb: Snavely completed batch synchonization of HTI content
to the Duderstadt storage server, put routine synchronization routines
in place to keep it up to date, and began discussions with DLPS about
workflow changes required to finish the consolidation.
Mar: Snavely worked with DLPS to transition the new
consolidated server name, quod.lib.umich.edu, into production for HTI
resources while maintaining the HTI website; Snavely continued
discussions with DLPS about the final steps of consolidation.
Apr: Snavely worked with DLPS to plan for consolidating
storage on the two storage servers, including determining authoritative
version of content where they were present in both environments, and
discussing solutions to workflow impacts.
Replace web load-balancing system.
Increase scalability of ingest/processing/validation
processes with multiple front-end servers and/or SSI clustering.
Dec: Marsh and Snavely installed a new blade server chassis
that will be used to house multiple new servers, some of which will be
dedicated to content ingest and processing.
Jan: Feeman and Marsh configured the new blade server
chassis and began to install operating systems on the blade servers.
Feb: Feeman and Marsh completed the blade server chassis
configuration and Marsh began an OS install on the first blade. Feeman
prepared detailed throughput metrics for different stages of GROOVE
that will be used to tune the aggregate throughput once blade servers
are available.
Mar: Marsh continued to work on OS installation on the
first blade server. Feeman revised throughput metrics for GROOVE.
Apply automation and processes created for MDP to local
Digital Conversion workflows.
Migrate to new web statistics system.
Dec: Feeman began running the new stats system side-by-side
with the old starting in December and ported all non-title-level legacy
statistics from the old system to the new. Because title-level
statistics data were stored in the old system in a non-portable way,
and because the new system provides this information in the form of
COUNTER-style reports, title-level data was not ported but instead the
web
pages for title-level stats were harvested and will be made
available alongside the new system.
Jan: Feeman continued to work out small problems in display
and tabulation in preparation for making the system viewable as a
release candidate.
Feb: Feeman added a totalling feature and made several
adjustments to COUNTER statistics for closer adherence to the
guidelines, and Feeman and Snavely helped Bonn prepare an announcement
about the new system for hosting partners.
Mar: The new stats system was well-received by hosting
partners and a tentative release date was scheduled for mid-April. The
review brought a problem to light, and Feeman reloaded and reprocessed
a large amount of data to correct it. Feeman and Snavely, prompted by
Bonn, began discussions with DLPS on mechanisms to re-introduce
session-based statistics, which were abandoned some time ago when DLXS
was re-engineered to store session information in web cookies.
Apr: The new stats system was released, and the interface
to the old stats system was shut down. Tabulation will continue in the
old system for at least another month.
Dec: Feeman and Snavely developed an ordered list of bug
fixes to address and new features to be developed for the
authentication
and authorization management system.
Jan: Feeman investigated several Perl modules that may be
useful for IP address range manipulation.
Feb: Feeman and Snavely worked together to create a
development environment for this system and Feeman began investigating
bugs and working on new features.
Mar: Feeman continued to make progress on the management
system, and she and Snavely worked together on approaches for
validating IP addresses input by the user and for handling ether
particulars of IP address editing.
Apr: Feeman finished bug fixes and enhancements to the
management system, and will review the fixes and new features with
Snavely at a future meeting.
Migrate Library Web Services to networked storage.
Continue work with ITCS on Shibboleth origin pilot.
Dec: Snavely contacted the people who expressed interest in
a Shibboleth pilot with ProQuest to determine their level of interest
and readiness, and received one or two enthusiastic responses; ProQuest
remains somewhat on the fence about implementation. Snavely sought
information on other service providers currently supporting Shibboleth
from Internet2 and consulted with Dennis and Folger on preferences.
Jan: Doster and Snavely discussed strategy, and decided to
keep contact with ProQuest but pursue work with other service providers
as identified by Folger. Doster and new manager Montague reiterated
support for the library's work with Shibboleth.
Feb: Snavely contacted JSTOR and Elsevier to determine
their interest level in a pilot; JSTOR has just announced Shibboleth
support and was responsive to initial questions.
Mar: Doster and Snavely discussed ITCS plans for building
out production Shibboleth infrastructure, and Snavely shared these
plans with Elsevier and JSTOR to begin to estimate when a pilot may
take place.
Apr: Snavely shared plans for Shibboleth development with
ProQuest on a quarterly conference call.
Migrate all web service to Apache 2.
Storage
Primary Objectives
Migrate to new Pillar Axiom SAN in ALDC.
Dec: Marsh discovered and resolved a Solaris problem
relating to the mapping of SAN LUNs to OS devices.
Jan: Marsh, Prettyman, and Snavely developed a plan to
migrate clyde from legacy storage to the SAN, and did the move a few
days later, collecting transfer rate statistics that will be used to
predict the downtime window for future production server migrations.
Feb: Marsh installed dual redundant host bus adapters in
gracie, connected it to the SAN, and began configuring and testing
them. Marsh ordered a host bus adapter for ethel.
Mar: After repeated problems with connectivity between
gracie and the SAN, Marsh involved Pillar and Sun technical support,
and began using ethel to troubleshoot the problems and try out
potential solutions.
Apr: Marsh migrated ethel from legacy storage to the SAN,
and scheduled two outages for gracie to troubleshoot persistent SAN
connectivity problems. The troubleshooting eliminated some potential
causes of the problems from consideration, but did not yet resolve them.
Finish MDP RFP process and design and install sustainable,
large-scale storage solution for MDP project.
Jan: Snavely developed an evaluation spreadsheet based on
criteria laid out in the RFP, received the RFP responses, and began
reviewing them. Snavely contacted Bridges and Lewis to seek their help
in evaluating options once the responses have been summarized.
Feb: Snavely continued preparing summaries of vendor
responses and scheduled a meeting for mid-March with Lewis and Marsh.
Mar: Snavely, consulting with Lewis and Marsh, selected the
top proposals from all responses and pursued presentations from those
bidders to be scheduled in April.
Apr: Snavely coordinated very productive presentations and
question-and-answer sessions with the remaining bidders, and began
drafting a best-and-final RFQ to be issues in May which will collect
pricing information on final system configurations.
System administration
Primary Objectives
Finish transition of production services to TSM backup
service and upgrade existing backup system for development servers.
Dec: Snavely upgraded the current development backup server
to new hardware to increase performance and reliability.
Improve remote access capabilities to server resources using
serial console servers, network KVM (for blades), and VPN device(s).
Jan: Feeman and Marsh configured the network KVM component
of the blade server chassis while setting up the chassis itself, as
reported above.
Replace production and development servers as scheduled.
Jan: Marsh upgraded the OS on sangria to Fedora Core 4.
Standardize and document system administration processes.
Transition to Gigabit Ethernet service in room 10.
Feb: DSS coordinated the installation of new switches in
room 10.
Secondary Objectives
Begin to develop centralized system administration mechanisms.
Transition to ITCS-supported DHCP.
Security
Primary Objectives
Complete security planning and perform TBD high-priority
tasks.
Dec: Marsh, Munce, Snavely, and Syrigos completed a first
draft of a security plan.
Jan: Marsh, Munce, Snavely, and Syrigos began discussing
high-priority tasks drawn from the security planning process, and
submitted to ITSS the first draft of the security plan and a first
draft of an incident response procedure.
Feb: Marsh, Munce, Snavely, and Syrigos continued to
develop ideas in three areas: a) staff guidelines for security
practices targeted toward distinct user types, b) establishing periodic
internal audits in areas such as user accounts and firewall
maintenance, and c) developing and testing a local process for security
incidents.
Secondary Objectives
Investigate virtual firewall service and request/implement as
appropriate.
Initiatives
Support integration/rearchitecture/massive scaling of
locally-developed content systems (Deep Blue, DLXS, MBooks).
Delve deeper into SSI clustering and server virtualization;
build proof-of-concept for virtual servers on an SSI cluster with an
eye toward centralized server resources for sharing among all LIT units.
Explore centralized storage for sharing among all LIT units.
Miscellaneous, routine, or unplanned
activity
Dec: Snavely met with CAS and ITCS staff to discuss the
dissolution of ITCS WATS and plans for transitioning application
support for a WATS-developed e-commerce system for ILLiad to Core
Services; Snavely discussed this work with Feeman, who began examining
the application and developing local documentation.
Jan: Feeman and Snavely met again with CAS staff to discuss the
progress on examining and documenting the e-commerce system as well
documenting ITCS support contacts we will need in order to take on
support. Snavely suggested a timeframe for taking on support, and
Feeman continued to work on documentation.
Feb: Feeman completed documentation of the ILLiad e-commerce
system and Core Services took on support for it.
Mar: Feeman, Snavely, and Syrigos met to discuss several
enhancements to the ILLiad e-commerce system, some proposed by CAS
staff, and some suggested by Core Services.
Mar: Marsh installed a new temporary storage server for MDP to
work alongside the existing one, adding 6 TB of capacity.
Apr: Feeman, working with Rennie, added support for JPEG 2000
validation and ingest to GROOVE, and Snavely began coordinating a date
to cut over to JPEG 2000 delivery from Google.
Apr: Marsh installed and configured buttermilk and cream, two
new servers that will be used for Oracle, replacing dlps6 and coffee.
Service Availability
Arbor Lakes Data Center
ethel.umdl (ARC): No down time
ezra.umdl (Verde): No down time
gracie.umdl (Aleph): Down on Sunday, April 8 from 8:00am to
11:00am and on Sunday, April 29 from 8:00am to 9:00am for scheduled
maintenance related to a new storage system.
hiram.umdl (Zebra): No down time
tequila.umdl (Metalib and SFX): No down time
Hatcher Server Room
belle.umdl (Library Web): No down time
coffee.umdl (Oracle): No down time
cowmilk.umdl (MySQL): No down time
merlot.umdl (DLPS public text collections and MBooks): No
down time
sambuca.umdl (Deep Blue): No down time
quik.umdl (Survivors of the Shoah cache): No down time
ting.umdl (library proxy): No down time
Duderstadt Center Machine Room:
dlps6.umdl (Oracle): No down time
dlps7.umdl (Numeric and Geospatial Data): No down time
dlps10.umdl (DLPS public text collections): No down time
dlps11.umdl (DLPS image and non-public text collections): No
down time
dlps12.umdl (DLPS image and non-public text collections): No
down time
soymilk.umdl (MySQL): No down time
Data Loading/Archival Statistics
CD/DVDs loaded or reloaded
(bitonal page images)
5
CD/DVDs loaded or reloaded and
processed (contone images)