Sprint 2019_04-SystemSaturation

September 12, 2019

Ended: September 12, 2019

Objectives:

This print continued the theme of system scalability.  We looked at how the system degrades when resources are constrained.  

Results:

The team invested significant time in understanding how Kubernetes can be configured to impact how the system degrades under resource over-runs.  They examined both limitations and workarounds and a lot of this work will bleed over to the next sprint.  

More RCE Data collection and analysis was done during this sprint to help shape what the first production environment will look like.

The team was short-staffed throughout the sprint with operations work requiring input from multiple team members.

List of stories closed out:

Setup and configure Circle CI with Sid Repo #669    669
Retry/reconnect to Redis and Mongo upon timeout #531    531
Try to make service pods un-killable #606    606
Provide feedback when the cluster is full #361    361
vi text editor is hanging on /mnt/google-drive (suspect rclone) #653    653
Resize the EKS worker EBS root volumes #605    605
Set different CPU and RAM defaults for Sid apps depending on each app's attributes #592    592    
Document how to find out how much disk each container is using #573    573
Determine the types of ASGs we want for the cluster-autoscaler#638    638
Document process to upgrade, build Docker applications for Sid #192    192