Key Takeaways
Upon completion of the course, participants will be able to:
- Understand the various facets of a real time data and stream processing pipeline.
- Design a reference architecture for a real time data processing system by determining the needful layers such as ingestion, collection, wrangling, message queues, analysis, and accessing new insights.
- Collect and design appropriate storage strategy for data originating from smaller devices such as IoT, Sensors and IoE.
- Integrate disparate data sources using unanimous ingestion layer that manages channels via MQTT, Flume, Kafka, Twitter, and a custom HTTP receiver.
- Build and optimise production-grade deployments of Streaming solutions via common algorithms, configuration recipes, and tuning of instrumentation API.
- Design robust message producers and consumers for writing and reading messages using Kafka.
- Evaluate and determine best stream processing framework suited for the given business needs.
Who Should Attend
This is an intermediate course, suitable for professionals with relevant experience, and with an interest or requirement to understand engineering for big data.
The target course participants are primarily software engineers, data engineers, team leads, and architects with seek to enhance their skills in the area of information architecture and design of data warehouse and data lake.
Prerequisites
- Participants are required to have completed the Big Data Engineering for Analytics course prior to attending this course.
- If participants have not completed the Big Data Engineering for Analytics course with NUS-ISS, they are required to produce experience or skills certification that benchmarks similar learning such as Cloudera Spark Developer Certificate.
What to Bring
No printed copies of course materials are issued.
Participants must bring their internet-enabled computing device (laptops, tablet etc) with power charger to access and download course materials.
If you are bringing a laptop, please see below for the tech specs:
|
|
Minimum
|
Recommended
|
|
Computer and processor
|
2 Cores or more. i7 (intel) or higher preferred.
|
4 Cores or more. i7 (intel) or higher preferred.
|
|
Memory
|
16 GB RAM (Minimum 4 GB Free memory while running the cluster)
|
32 GB RAM(Minimum 4 GB Free memory while running the cluster)
|
|
Hard Disk
|
1 TB (Minimum 20 GB free for use)
|
2 TB or more
|
|
Display
|
Minimum of 1280 x 768 screen resolution (32-bit requires hardware acceleration for 4K and higher)
|
|
Cluster Management Software
|
Run local Kubernetes clusters (example minikube or kind) on Windows, Linux, or Mac environments. |
|
|
Container or virtual machine manager
|
Kubernetes preferred. Other alternatives:
Docker, Hyperkit, Hyper-V, KVM, Parallels, Podman, VirtualBox, or VMware Fusion/Workstation
|
|
|
Others
|
An internet connection – broadband wired or wireless
Speakers and a microphone – built-in or USB plug-in or wireless Bluetooth
A webcam or HD webcam - built-in or USB plug-in
|
|
What Will Be Covered
This course will cover:
- Architecting for Real Time stream processing systems
- Design of Pipeline for Real Time Streaming Systems
- Ingestion Strategies for Data Streams
- Message Design and Queue Architecture
- Data Stream Analytics
- In-Memory vs Storage Strategies
- Streaming Visualisation Tools
- Case Study Discussion
Fees & Subsidies
Fees for 2024
| |
Full Fee |
Singaporeans & PRs
(self-sponsored) |
| Full course fee |
S$2700 |
S$2700 |
| ISS Subsidy |
- |
(S$270) |
| Nett course fee |
S$2700 |
S$2430 |
| 9% GST on nett course fee |
S$243 |
S$218.70 |
| Total nett course fee payable, including GST |
S$2943 |
S$2648.70 |
Note:
- All fees and subsidies are valid from January 2024, unless otherwise advised.
- All self-sponsored Singaporeans aged 25 and above can use their SkillsFuture Credit to pay for course fees. For more information about SkillsFuture Credit, click here.
- From 1st January 2024, the GST will be increased to 9%.
Certificate
Certificate of Completion
Participants have to meet a minimum attendance rate of 75% and are required to pass the assessment to be issued a Certificate of Completion.
Preparing for Your Course
NUS-ISS Course Registration Terms and Conditions
Find out more.
NUS-ISS and Learner’s Commitment and Responsibilities
Find out more.
WIFI Access
WIFI access will be made available to participants.
Venue
NUS-ISS
25 Heng Mui Keng Terrace
Singapore 119615
Click HERE for directions to NUS-ISS
In the event of a change of venue, participants are advised to refer to the acceptance email sent one week prior to the commencement date.
Course Confirmation
All classes are subject to confirmation and NUS-ISS will send an acceptance email to participants one week prior to the commencement date. Confirmed registrants are to attend and complete all lectures, class exercises, workshops and assessments (where applicable). Additionally, all responses to feedbacks and surveys conducted by NUS-ISS and its partners must be submitted. All training and assessments will be delivered as described in the course webpage.
General Enquiry
Please feel free to write to ask-iss@nus.edu.sg if you have any enquiry or feedback.