What is STREAMS ?

STREAMS is a high performance platform for continuous integration and testing of emerging technologies. It will address the challenges of massive scale data streams processing in the context of designing the next generation instrumentation for giant telescopes with a focus on SKA. This experimental facility is designed to combine multi-Tb/s data transport together with high performance computing and data analytics (including deep neural networks) and high-speed large capacity storage. Whilst this platform is extremely heterogeneous by nature, its exploitation relies on developing reusable and portable software tools to be deployed either on operational instruments or in other national / regional computing facilities for further optimization and integration. This platform will serve as a hub for close collaboration with industry, expanding to other application domains.

The long lifespan (10-20 years) contemplated for this kind of cyber-physical systems makes sustainability a key consideration for this project, including energy efficiency and cost containment across integration, operations and maintenance.

Hardware implementation

The platform is built around a backbone (high-speed, low-latency network switch), which is used to distribute data through the 3 main subsystems

• the data generation subsystem, enabling emulation of data production by sensors (radiotelescopes and radars), based on a high-performance computing server, making it possible to accommodate high-speed network (for sending) and FPGA cards (for sensors emulation)
• the ingestion-processing subsystem, composed of several high-performance servers, equipped with high-speed network cards (reception), a high density GPUs cluster (or future hardware accelerators) and responsible for real-time processing tasks. These servers will be designed to be scalable in order to test new computing technologies (GPU in the initial phase, then more heterogeneous configuration GPU + FPGA + other)
• the supervision and learning subsystem, composed of high-performance servers equipped with network cards for the capture of data transported through the platform, as well as a very dense integrated cluster of GPUs (for deep learning).

STREAMS collaboration

Several strategic science programs will benefit from the new capacities provided by STREAMS supporting teams from 4 laboratories (LESIA, GEPI, LERMA, USN) from the Paris Region.

A pilot study on next generation surface radar systems will be conducted by Thales Land & Air Systems, including hardware and software sub-systems co-design and performance validation.

REFLEX CES, a fast growing tech SME from the Paris region will be strongly involved in the co-design of core components and sub-systems, in particular the high throughput data ingestion solution.

Additionally, bidirectional links have been established with major national initiative (PEPR projects NumPEx and ORIGINS, as well as CLUSSTER) to turn STREAMS into a multi-disciplinary and trans-sectoral convergence point settled within the Paris region, with far reaching influence at national and international levels.

The project is sponsored by Region Ile-de-France through its DIM Origines program.