The NUS Centre for Bioimaging Sciences

CBIS Computational Facility

The Computational Core Facility provides infrastructure for computation, storage, networking and scientific software services and support for the NUS Centre for Bioimaging Science.

Core Mission

CBIS’s computational core facility was created to store, process, analyse and visualise experimental data and images as well as large scale computer modelling of cell processing and data mining of complex biological processes. As CBIS has moved to a new research space without any preexisting IT facilities, major efforts were taken to design, acquire, install and configure the IT infrastructure.

CBIS employs imaging technology to acquire biological data sets as a core component of their research. A single image data set may be several TB in size and is often acquired as individual files resulting in many thousands of files per experiment that must be stored, processed, analysed and visualised. CBIS houses multiple high-end cryoelectron and confocal microscopes that can each generate large continuous streams of new data. Thus a fast, efficient and reliable network, storage and computational infrastructure is critical to support our research activities. Having a local data centre to house this computational equipment is the optimal way to provide necessary services and support for our scientists.

Facilities

CBIS’s core IT infrastructure is currently composed of the following basic components.

Network

CBIS’ IT Core provides high-speed network connectivity between servers, storage, microscope and end users. Our network consists of:

• an InfiniBand network providing 56Gb/s (scalable to 100Gb/s) for storage communication between servers
• 1Gbps, 10Gbps transfer rate from end users to centralised storage
• 40GbE core network with plans to scale up to 100GbE
• 100GbE connectivity between CBIS, MBI, NUS IT, NSCC and other research centres

Storage

CBIS IT operates a high-performance parallel storage system providing large capacity and highly reliable storage. It is a GPFS based high-performance parallel storage providing 1.2PB of storage, scalable up to 3PB of capacity which providing 10Gbps of data transfer rate.

CBIS storage is centralised for collection of large datasets of optical and electron microscopes, and the next generation sequencer. Storage arrays are directly connected to the CBIS high-end compute cluster through an FDR 56Gb/s Infiniband switch.

High-performance computing cluster

CBIS operates a set of high-performance CPU and GPU servers managed by PBS job management system. All the compute cluster nodes are interconnected through FDR 56Gb/s Infiniband switches:

• 8 x CPU nodes with total 224 CPU Cores and 4TB memory
• 3 x GPU nodes with total 16 GPU cards
• 2 x SGI SMP nodes with total 180 CPU Cores and 2TB memory

In addition, the CBIS IT Core provides the CBIS community with a diverse set of both commercial and Open Source software applications for image processing, database mining, computer modelling, bioinformatics and genomics, archival storage, etc., along with a fully-trained staff of IT professionals with expertise in systems design, implementation and management and application development, and an independent web presence and network for inter- and intra- nets.

The CBIS data centre is connected to the Mechanobiology Institute (MBI) data centre through the NUS campus backbone. This connection utilises multiple high-speed 100 Gbps Ethernet circuits allowing us to share the computational and storage resources between the two facilities. This capability allows users the freedom to run experiments and collect data at either research centre and process their data without using local compute resources, minimising the need to move very large data sets across the network.