Core Services Cumulative Monthly Report, February 2007
Content/Web infrastructure
Primary objectives
Finish migration of DLPS web services to redundant networked
storage.
Dec: Marsh installed new storage on the Hatcher storage
server, and Snavely began batch synchronizing ETS and Image Services
content to it.
Jan: Snavely completed batch synchronization of ETS and
Image Services content to the Hatcher storage server and put routine
synchronization routines in place to keep it up to date. Marsh upgraded
the storage array from dlps8 (which was retired) with new disks and
current firmware, installed it on on the Duderstadt Center storage
server, and Snavely began batch synchronizing HTI content to it.
Feb: Snavely completed batch synchonization of HTI content
to the Duderstadt storage server, put routine synchronization routines
in place to keep it up to date, and began discussions with DLPS about
workflow changes required to finish the consolidation.
Replace web load-balancing system.
Increase scalability of ingest/processing/validation
processes with multiple front-end servers and/or SSI clustering.
Dec: Marsh and Snavely installed a new blade server chassis
that will be used to house multiple new servers, some of which will be
dedicated to content ingest and processing.
Jan: Feeman and Marsh configured the new blade server
chassis and began to install operating systems on the blade servers.
Feb: Feeman and Marsh completed the blade server chassis
configuration and Marsh began an OS install on the first blade. Feeman
prepared detailed throughput metrics for different stages of GROOVE
that will be used to tune the aggregate throughput once blade servers
are available.
Apply automation and processes created for MDP to local
Digital Conversion workflows.
Migrate to new web statistics system.
Dec: Feeman began running the new stats system side-by-side
with the old starting in December and ported all non-title-level legacy
statistics from the old system to the new. Because title-level
statistics data were stored in the old system in a non-portable way,
and because the new system provides this information in the form of
COUNTER-style reports, title-level data was not ported but instead the
web
pages for title-level stats were harvested and will be made
available alongside the new system.
Jan: Feeman continued to work out small problems in display
and tabulation in preparation for making the system viewable as a
release candidate.
Feb: Feeman added a totalling feature and made several
adjustments to COUNTER statistics for closer adherence to the
guidelines, and Feeman and Snavely helped Bonn prepare an announcement
about the new system for hosting partners.
Dec: Feeman and Snavely developed an ordered list of bug
fixes to address and new features to be developed for the
authentication
and authorization management system.
Jan: Feeman investigated several Perl modules that may be
useful for IP address range manipulation.
Feb: Feeman and Snavely worked together to create a
development environment for this system and Feeman began investigating
bugs and working on new features.
Migrate Library Web Services to networked storage.
Continue work with ITCS on Shibboleth origin pilot.
Dec: Snavely contacted the people who expressed interest in
a Shibboleth pilot with ProQuest to determine their level of interest
and readiness, and received one or two enthusiastic responses; ProQuest
remains somewhat on the fence about implementation. Snavely sought
information on other service providers currently supporting Shibboleth
from Internet2 and consulted with Dennis and Folger on preferences.
Jan: Doster and Snavely discussed strategy, and decided to
keep contact with ProQuest but pursue work with other service providers
as identified by Folger. Doster and new manager Montague reiterated
support for the library's work with Shibboleth.
Feb: Snavely contacted JSTOR and Elsevier to determine
their interest level in a pilot; JSTOR has just announced Shibboleth
support and was responsive to initial questions.
Migrate all web service to Apache 2.
Storage
Primary Objectives
Migrate to new Pillar Axiom SAN in ALDC.
Dec: Marsh discovered and resolved a Solaris problem
relating to the mapping of SAN LUNs to OS devices.
Jan: Marsh, Prettyman, and Snavely developed a plan to
migrate clyde from legacy storage to the SAN, and did the move a few
days later, collecting transfer rate statistics that will be used to
predict the downtime window for future production server migrations.
Feb: Marsh installed dual redundant host bus adapters in
gracie, connected it to the SAN, and began configuring and testing
them. Marsh ordered a host bus adapter for ethel.
Finish MDP RFP process and design and install sustainable,
large-scale storage solution for MDP project.
Jan: Snavely developed an evaluation spreadsheet based on
criteria laid out in the RFP, received the RFP responses, and began
reviewing them. Snavely contacted Bridges and Lewis to seek their help
in evaluating options once the responses have been summarized.
Feb: Snavely continued preparing summaries of vendor
responses and scheduled a meeting for mid-March with Lewis and Marsh.
System administration
Primary Objectives
Finish transition of production services to TSM backup
service and upgrade existing backup system for development servers.
Dec: Snavely upgraded the current development backup server
to new hardware to increase performance and reliability.
Improve remote access capabilities to server resources using
serial console servers, network KVM (for blades), and VPN device(s).
Jan: Feeman and Marsh configured the network KVM component
of the blade server chassis while setting up the chassis itself, as
reported above.
Replace production and development servers as scheduled.
Jan: Marsh upgraded the OS on sangria to Fedora Core 4.
Standardize and document system administration processes.
Transition to Gigabit Ethernet service in room 10.
Feb: DSS coordinated the installation of new switches in
room 10.
Secondary Objectives
Begin to develop centralized system administration mechanisms.
Transition to ITCS-supported DHCP.
Security
Primary Objectives
Complete security planning and perform TBD high-priority
tasks.
Dec: Marsh, Munce, Snavely, and Syrigos completed a first
draft of a security plan.
Jan: Marsh, Munce, Snavely, and Syrigos began discussing
high-priority tasks drawn from the security planning process, and
submitted to ITSS the first draft of the security plan and a first
draft of an incident response procedure.
Feb: Marsh, Munce, Snavely, and Syrigos continued to
develop ideas in three areas: a) staff guidelines for security
practices targeted toward distinct user types, b) establishing periodic
internal audits in areas such as user accounts and firewall
maintenance, and c) developing and testing a local process for security
incidents.
Secondary Objectives
Investigate virtual firewall service and request/implement as
appropriate.
Initiatives
Support integration/rearchitecture/massive scaling of
locally-developed content systems (Deep Blue, DLXS, MBooks).
Delve deeper into SSI clustering and server virtualization;
build proof-of-concept for virtual servers on an SSI cluster with an
eye toward centralized server resources for sharing among all LIT units.
Explore centralized storage for sharing among all LIT units.
Miscellaneous, routine, or unplanned
activity
Dec: Snavely met with CAS and ITCS staff to discuss the
dissolution of ITCS WATS and plans for transitioning application
support for a WATS-developed e-commerce system for ILLiad to Core
Services; Snavely discussed this work with Feeman, who began examining
the application and developing local documentation.
Jan: Feeman and Snavely met again with CAS staff to discuss the
progress on examining and documenting the e-commerce system as well
documenting ITCS support contacts we will need in order to take on
support. Snavely suggested a timeframe for taking on support, and
Feeman continued to work on documentation.
Feb: Feeman completed documentation of the ILLiad e-commerce
system and Core Services took on support for it.
Service Availability
Arbor Lakes Data Center
ethel.umdl (ARC): No down time
ezra.umdl (Verde): No down time
gracie.umdl (Aleph): Down on Sunday, February 4 and on
Sunday, February 25 from 8:00am to 9:00am for hardware installation
required to connect to a new storage system.
hiram.umdl (Zebra): No down time
tequila.umdl (Metalib and SFX): No down time
Hatcher Server Room
belle.umdl (Library Web): Down on Sunday February 25 from
approximately 10:00am to 2:45pm due to an electrical equipment failure.
coffee.umdl (Oracle): Down on Sunday February 25 from
approximately 10:00am to 2:45pm due to an electrical equipment failure.
cowmilk.umdl (MySQL): Down on Sunday February 25 from
approximately 10:00am to 2:45pm due to an electrical equipment failure.
merlot.umdl (DLPS public text collections and MBooks): Down
on Sunday February 25 from approximately 10:00am to 2:45pm due to an
electrical equipment failure.
sambuca.umdl (Deep Blue): No down time
quik.umdl (Survivors of the Shoah cache): No down time
ting.umdl (library proxy): No down time
Duderstadt Center Machine Room:
All services were
unreachable on
Friday, February 9 from approximately 8:00am to 10:00am due to a
networking problem.
dlps6.umdl (Oracle): No down time
dlps7.umdl (Numeric and Geospatial Data): No down time
dlps10.umdl (DLPS public text collections): No down time
dlps11.umdl (DLPS image and non-public text collections):
Down on Wednesday, February 14 from 5:00pm to 6:00pm to resolve a
problem with a connected storage system.
dlps12.umdl (DLPS image and non-public text collections):
Down on Wednesday, February 14 from 5:00pm to 6:00pm to resolve a
problem with a connected storage system.
soymilk.umdl (MySQL): No down time
Data Loading/Archival Statistics
CD/DVDs loaded or reloaded
(bitonal page images)
6
CD/DVDs loaded or reloaded and
processed (contone images)