Containers and Microservices make performance worse

Transcript

1. Containers and Microservices Puppet Labs Gareth Rushgrove Make everything performance

   worse

2. Gareth Rushgrove @garethr

3. Gareth Rushgrove

4. Gareth Rushgrove

5. Quiz Time

6. Gareth Rushgrove Apologies to the following. I love you really.

7. This Talk

8. Gareth Rushgrove

9. make snide and sharply critical comments Gareth Rushgrove snark /sna:k/

   noun

10. Fast in-process communication vs slow and unreliable HTTP Gareth Rushgrove

11. The performance cost of an overlay network Gareth Rushgrove

12. What happened to ps and friends? Gareth Rushgrove

13. Microservices and Performance It will get bad before it gets

    good

14. Monoliths are pretty easy to understand at a high level

    Gareth Rushgrove

15. Gareth Rushgrove Monolith

16. Even adding in the supporting infrastructure it’s not so bad

    Gareth Rushgrove

17. Gareth Rushgrove Monolith Database Load Balancer Network Network

18. I can optimise the network, the load balancer, the database,

    the application or the client. Easy Gareth Rushgrove

19. What about Microservices? Well, first start with a diagram like

    this… Gareth Rushgrove

20. Microservice Microservice Microservice Microservice Microservice Microservice Microservice Microservice Gareth Rushgrove

21. Whatever protocol you’re using lets assume you’re communicating over the

    network… Gareth Rushgrove

22. Microservice Microservice Microservice Microservice Microservice Microservice Microservice Microservice Gareth Rushgrove

    Network Network Network Network Network Network Network Network

23. And you probably want more than one instance of each

    service… Gareth Rushgrove

24. Gareth Rushgrove Microservice Load Balancer Network

25. Gareth Rushgrove Microservice Microservice Microservice Microservice Microservice Microservice Microservice Microservice

    Network Network Network Network Network Network Network Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network

26. And you probably have a few different databases now too…

    Gareth Rushgrove

27. Gareth Rushgrove Microservice Microservice Microservice Microservice Microservice Microservice Microservice Microservice

    Network Network Network Network Network Network Network Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Database Network Network Database

28. In my made-up 8 service architecture we went from 5

    things to optimise up to 32 Gareth Rushgrove The Bad

29. We went from 3 network hops to, er, more depending

    on the request Gareth Rushgrove The Bad

30. We ignored the cost of serialisation/deserialisation (JSON can be expensive)

    Gareth Rushgrove The Bad

31. The operational overhead just jumped considerably Gareth Rushgrove The Bad

32. Lots more network traffic. Watch out for latency in particular

    Gareth Rushgrove The Bad

33. Without request tracing you’re doomed Gareth Rushgrove The Bad

34. Granular services are easier to optimise individually Gareth Rushgrove The

    Good

35. Individual services can be scaled independently Gareth Rushgrove The Good

36. Debugging with Containers Is that process inside or outside a

    container?

37. Problems with free and top Gareth Rushgrove

38. Gareth Rushgrove $ free total used free shared buffers cached

    Mem: 1024444 864140 160304 5024 50008 637736 -/+ buffers/cache: 176396 848048 Swap: 473084 16 473068 $ docker exec test-container free total used free shared buffers cached Mem: 1024444 866440 158004 5024 50000 637732 -/+ buffers/cache: 178708 845736 Swap: 473084 16 473068 Can a container use that memory?

39. memory stats come from the proc filesystem: /proc/meminfo, / proc/vmstat,

    etc. Gareth Rushgrove

40. /proc/meminfo and /proc/vmstat are not aware of cgroups Gareth Rushgrove

41. Problems with ps Gareth Rushgrove

42. Gareth Rushgrove $ ps aux USER PID %CPU %MEM VSZ

    RSS TTY STAT START TIME COMMAND ... 999 1807 0.2 11.4 867624 464572 ? Ssl 09:38 0:21 mysqld Is this process in a container?

43. Gareth Rushgrove $ ps -eo ucmd,cgroup COMMAND CGROUP ... mysqld

    9:perf_event:/docker/61e76d2c39121282474ff895b9b3ba2addd775cdea6d2ba89ce76c28 Which container is that?

44. Gareth Rushgrove Sysdig

45. Provides a Kernel module, which hooks into cgroups and namespaces

    Gareth Rushgrove

46. Gareth Rushgrove $ sudo sysdig -c topcontainers_cpu CPU% container.name -----------------------------------------------------------------------

    90.13% mysql 15.93% wordpress1 7.27% haproxy 3.46% wordpress2 CPU usage across containers

47. Gareth Rushgrove $ sudo sysdig -pc -c topprocs_cpu container.name=client CPU%

    Process container.name ---------------------------------------------- 02.69% bash client 31.04% curl client 0.74% sleep client CPU usage in a single container

48. Gareth Rushgrove $ sudo sysdig -pc -c topprocs_net Bytes Process

    Host_pid Container_pid container.name --------------------------------------------------------------- 72.06KB haproxy 7385 13 haproxy 56.96KB docker.io 1775 7039 host 44.45KB mysqld 6995 91 mysql 44.45KB mysqld 6995 99 mysql 29.36KB apache2 7893 124 wordpress1 29.36KB apache2 26895 126 wordpress4 29.36KB apache2 26622 131 wordpress2 29.36KB apache2 27935 132 wordpress3 29.36KB apache2 27306 125 wordpress4 22.23KB mysqld 6995 90 mysqlclient Network bandwidth

49. Gareth Rushgrove $ sudo sysdig -pc -A -c echo_fds "fd.ip=172.17.0.3

    and fd.ip=172.17.0.7" ------ Write 103B to [haproxy] [d468ee81543a] 172.17.0.7:37557->172.17.0.3:80 (hapr GET / HTTP/1.1 User-Agent: curl/7.35.0 Host: 172.17.0.7 Accept: */* X-Forwarded-For: 172.17.0.8 ------ Read 103B from [wordpress1] [12b8c6a04031] 172.17.0.7:37557->172.17.0.3:80 ( GET / HTTP/1.1 User-Agent: curl/7.35.0 Host: 172.17.0.7 Accept: */* X-Forwarded-For: 172.17.0.8 ------ Write 346B to [wordpress1] [12b8c6a04031] 172.17.0.7:37557->172.17.0.3:80 (a HTTP/1.1 302 Found Date: Sat, 21 Feb 2015 22:19:18 GMT Traffic between containers

50. Don’t expect existing debugging tools to work Gareth Rushgrove The

    Bad

51. New tools are emerging. Often with better interfaces Gareth Rushgrove

    The Good

52. Container Overhead Count the performance penalties

53. Gareth Rushgrove

54. Containers add very little overhead Gareth Rushgrove The Good

55. Gareth Rushgrove

56. Gareth Rushgrove

57. Memory cgroups can be expensive Gareth Rushgrove The Bad

58. By default, the memory subsystem uses 40 bytes of memory

    per physical page on x86_64 systems. These resources are consumed even if memory is not used in any hierarchy Gareth Rushgrove https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html

59. Container networking is hard. Overlay networks make it easy. But

    slow. Gareth Rushgrove The Bad

60. Gareth Rushgrove http://www.generictestdomain.net/docker/weave/networking/stupidity/2015/04/05/weave-is-kinda-slow/

61. Gareth Rushgrove https://arjanschaaf.github.io/is-the-network-the-limit/

62. Gareth Rushgrove The Good

63. Gareth Rushgrove $ qperf between two EC2 c3.8xlarge with 10Gb/s

    tcp_bw: bw = 1.2 GB/sec udp_lat: latency = 48.1 us $ qperf over weave network using ODP/VXLAN tcp_bw: bw = 1.09 GB/sec udp_lat: latency = 61.9 us

64. Linux containers don’t really contain Gareth Rushgrove The Bad

65. Gareth Rushgrove

66. To get strict isolation guarantees you’re going to wrap them

    in virtual machines anyway Gareth Rushgrove The Bad

67. Projects like Clear Linux from Intel are innovating in this

    space Gareth Rushgrove The Good

68. User namespaces (host/container separation) and seccomp (limit syscalls) are coming

    to Docker Gareth Rushgrove The Good

69. Conclusions Snark free zone

70. Containers and microservices pose new performance challenges Gareth Rushgrove

71. Most problems aren’t performance problems Gareth Rushgrove

72. If you’re not already a networking expert start learning now

    Gareth Rushgrove

73. Lots of opportunities for new tooling to improve things Gareth

    Rushgrove

74. Questions? And thanks for listening Gareth Rushgrove