自分だけの仮想クラスタを高速かつ効率的に作る kubefork

Wantedly Tech Night 〜ウォンテッドリーで内製している開発用ツールの紹介〜

https://wantedly.connpass.com/event/341010/

Transcript

1. © 2025 Wantedly, Inc. ࣗ෼͚ͩͷԾ૝ΫϥελΛߴ଎͔ͭ ޮ཰తʹ࡞Δ kubefork Wantedly Tech Night #8 2025-02-25 Kazuki Obata
2. © 2025 Wantedly, Inc. ࣗݾ঺հ ڊി ࿨थ (Obata Kazuki) @donkomura_ Wantedly, Inc. (2024/09 ～ ) ΠϯϑϥΤϯδχΞ #k8s #ϘϧμϦϯά🧗
3. © 2025 Wantedly, Inc. ໨࣍ • Ϟνϕʔγϣϯ • ՝୊ͱ΍Γ͍ͨ͜ͱ • Ͳ͏΍࣮ͬͯݱ͢Δ͔ ◦ Wantedly ։ൃͷ৔߹ ◦ ΞΠσΟΞͱ޻෉ • ·ͱΊ
4. © 2025 Wantedly, Inc. Ϟνϕʔγϣϯɿ։ൃதͷϓϨϏϡʔ
5. © 2025 Wantedly, Inc. Ϟνϕʔγϣϯɿ։ൃதͷϓϨϏϡʔ • ؾܰʹϓϨϏϡʔͰ͖Δͱศར ◦ ຊ൪ͱಉ͡Α͏ͳ؀ڥͰಈ࡞֬ೝ͠ͳ͕Β։ൃͰ͖Δ ◦ ϨϏϡʔ͠΍͍͢ Πϝʔδɿhttps://github.com/marketplace/actions/deploy-pr-preview ։ൃதʹ ࣅͨΑ͏ͳ͜ ͱ͕͍ͨ͠
6. © 2025 Wantedly, Inc. ੔ཧɿݱঢ়ͷ։ൃϓϩηεͷτϨʔυΦϑ ͍͍ͱ͜औΓΛ͍ͨ͠ ݸਓͷ։ൃαʔόʔʢओʹϩʔΧϧʣ - ✅؀ڥ͕ଞͷӨڹʹΑͬͯԚ͞Εͳ͍ - ✅Docker ͳͲͰຊ൪ʹ͍ۙ؀ڥ͸࣮ݱͰ͖Δ - ❌Πϯϑϥ෦෼͢΂ͯΛ࠶ݱͰ͖ΔΘ͚Ͱ͸ ͳ͍ - ❌ґଘͨ͠αʔϏε͕͋Δͱ։ൃ͕೉͍͠ ڞ༗ͷ։ൃαʔόʔʢओʹϦϞʔτʣ - ✅ຊ൪ʹ͍ۙ؀ڥͰ։ൃͰ͖Δ - ❌ଞͷਓͷ࡞ۀ͔ΒӨڹΛड͚Δɾ༩͑Δ - ❌҆ఆͨ͠؀ڥΛอͪʹ͍͘ - յΕΔ͜ͱ͕͋Δ - ❌໰୊ͷಛఆ͕೉͘͠ͳΔ - σϓϩΠ͕িಥ͢Δ - ϩά͕ࠞࡏ͢Δ
7. © 2025 Wantedly, Inc. Ͳ͏࣮ݱ͢Δ͔ - Wantedly ։ൃͷ৔߹
8. © 2025 Wantedly, Inc. Ͳ͏࣮ݱ͢ΔɿWantedly ։ൃͷ৔߹ લఏ • ϚΠΫϩαʔϏεʢΞʔΩςΫνϟʣ • Πϯϑϥ͸ Kubernetes ◦ 1Ϋϥελ60ݸͷϚΠΫϩαʔϏεΛӡ༻ • ։ൃऀ͸໿50ਓ
9. © 2025 Wantedly, Inc. ࣮ݱͷͨΊͷΞΠσΟΞɿΫϥελʔؙ͝ͱίϐʔ͢Δ • શһʹಉ͡։ൃ؀ڥΛఏڙ͢Δͷ͕ཧ૝ • ͢΂ͯͷϦιʔεΛຊ൪͔Βίϐʔ͢Δͱ࣮ݱͰ͖Δ ◦ 1αʔϏε1αʔόʔͱͯ͠΋60 x 50 = 3000 αʔόʔ࡞੒͢Δ͜ͱʹͳΔ 🤔 ◦ Ϧιʔε͕५୔ʹ͋ΔΘ͚Ͱ͸ͳ͍
10. © 2025 Wantedly, Inc. ۪௚ʹ࣮ݱ͢Δͷ͸೉ͦ͠͏
11. © 2025 Wantedly, Inc. ޻෉͢Δ
12. © 2025 Wantedly, Inc. มߋ͍ͨ͠෦෼͚ͩઐ༻ ࢒ΓͷαʔϏε͸ڞ༻
13. © 2025 Wantedly, Inc. ޻෉ • deployment ͱ service ͚ͩมߋ͢Δ • ϧʔςΟϯά ◦ Ծ૝Ϋϥελ΁ͷΞΫηε৘ใΛ HTTP Header, gRPC Metadata Ͱ఻ൖ
14. © 2025 Wantedly, Inc. deployment ͱ service ͚ͩมߋ͢Δ
15. © 2025 Wantedly, Inc. ʮมߋ͍ͨ͠෦෼͚ͩʯઐ༻ • ΞϓϦέʔγϣϯ։ൃऀʹΑΔมߋ ◦ Stateless ͳϦιʔε͚ͩΛίϐʔ͢Ε͹े෼ ◦ Kubernetes ͩͱ Deployment • Deployment ͷίϐʔʹىҼ͢Δมߋ ◦ ίϐʔͨ͠ Deployment ʹ޲͚ͯϦΫΤετΛྲྀ͍ͨ͠ ◦ Kubernetes Ͱ͸ Service Λ࢖ͬͯτϥϑΟοΫΛసૹ͢Δ
16. © 2025 Wantedly, Inc. ʮมߋ͍ͨ͠෦෼͚ͩʯઐ༻ • deployment ͱ service ͷΈίϐʔ͢Δ ◦ มߋ͍ͨ͠ϚΠΫϩαʔϏεͷ Deployment ͱ Service Λίϐʔ ◦ มߋ͠ͳ͍ϚΠΫϩαʔϏε͸ڞ༻ͷ΋ͷΛ࢖͏Α͏ʹϧʔςΟϯά
17. © 2025 Wantedly, Inc. ϧʔςΟϯά
18. © 2025 Wantedly, Inc. • มߋ෦෼ʢService, DeploymentʣͷΈΛίϐʔ ◦ Ծ૝ΫϥελʹΞΫηε͢Δ৔߹ ▪ ଞͷϚΠΫϩαʔϏε͔ΒͷϦΫΤετΛ͜ΕΒʹྲྀ͍ͨ͠ ▪ ϓϨϏϡʔͰ͖Δঢ়ଶʹ͍ͨ͠ Ͳ͏΍ͬͯίϐʔͨ͠ Service, Deployment ʹ ϦΫΤετΛྲྀ͔͢ ͜͜·Ͱ A B B
19. © 2025 Wantedly, Inc. Istio ͷϦιʔεΛ࢖ͬͨϧʔςΟϯά • VirtualService ◦ ϚΠΫϩαʔϏε಺ͷαʔϏεؒ௨৴ͷϧʔςΟϯάΛઃఆ͢Δ resource ◦ header ͷ஋ΛجʹϦΫΤετͷϧʔςΟϯάઌΛมߋ͢Δ apiVersion: networking.istio.io/v1beta1 kind: VirtualService spec: hosts: - wantedly-x http: - match: - headers: x-fork-identifer: exact: donkomura route: - destination: host: kube-fork-donkomura x-fork-identifier
 ͱ͍͏ header Λݟͯ ࣗ෼ઐ༻ͷ host ʹ ϧʔςΟϯά͢Δ ίϐʔࡁΈͷ host
20. © 2025 Wantedly, Inc. ࣮૷͸Ͱ͖ͨ ӡ༻Ͱ͖Δʁ
21. © 2025 Wantedly, Inc. fork ʹ͓͚Δ VirtualService ӡ༻ͷ՝୊ • 1ͭͷ VirtualService ʹԾ૝Ϋϥελ͝ͱͷઃఆ͕ඞཁ • ྫ: 2ਓ͕ಉ͡αʔϏεΛมߋ͍ͨ͠ͱ͖ ◦ Ұͭͷ Virtual Service ʹ৚݅Λ·ͱΊΔ ◦ x-fork-identifer: cluster-A ʹϚον͢ΔͳΒ ▪ service-A ΁ϧʔςΟϯά ◦ x-fork-identifer: cluster-B ʹϚον͢ΔͳΒ ▪ service-B ΁ϧʔςΟϯ ◦ ͦΕҎ֎ͳΒ service-X ΁ϧʔςΟϯά
22. © 2025 Wantedly, Inc. fork ʹ͓͚Δ VirtualService ӡ༻ͷ՝୊ • 1ͭͷ VirtualService ʹԾ૝Ϋϥελ͝ͱͷઃఆ͕ඞཁ • ྫ: 2ਓ͕ಉ͡αʔϏεΛมߋ͍ͨ͠ͱ͖ ◦ Ұͭͷ Virtual Service ʹ৚݅Λ·ͱΊΔ ◦ x-fork-identifer: cluster-A ʹϚον͢ΔͳΒ ▪ service-A ΁ϧʔςΟϯά ◦ x-fork-identifer: cluster-B ʹϚον͢ΔͳΒ ▪ service-B ΁ϧʔςΟϯ ◦ ͦΕҎ֎ͳΒ service-X ΁ϧʔςΟϯά ධՁॱΛ੍ޚͰ͖ͳ͍ ίϯϑϦΫτ͢Δ
23. © 2025 Wantedly, Inc. VSConfig • VSConfig ◦ Ծ૝ΫϥελͷϧʔςΟϯάઃఆΛهड़͢ΔͨΊͷಠࣗϦιʔε ◦ ࣮૷͸ΧελϜίϯτϩʔϥʔ ▪ ઃఆΛू໿ͯ͠ VirtualService Λੜ੒͢Δ ◦ VirtualService ͷઃఆΛෳ਺ਓ͕৮Βͳͯ͘ྑ͍ ▪ ίϯϑϦΫτ͕ى͖ͳ͍ ◦ յΕͯ΋ࣗಈͰݩʹ໭Δ VirtualService ΁ͷҟͳΔมߋΛू໿͢Δ apiVersion: vsconfig.k8s.wantedly.com/v1beta1 kind: VSConfig spec: headerName: x-fork-identifer headerValue: donkomura host: # ϧʔςΟϯάର৅ͷ Service wantedly-x service: # ϦΫΤετΛྲྀ͢ઌͷ Service kube-fork-donkomura
24. © 2025 Wantedly, Inc. ։ൃதʹԾ૝ΫϥελʹϦΫΤετΛඈ͹͍ͨ͠ • ͜͜·Ͱ ◦ ಛఆͷ header ͷ͍ͭͨϦΫΤετ͸Ծ૝ΫϥελʹϧʔςΟϯά͞ΕΔ ◦ ϧʔςΟϯά͸ VirtualService ͰߦΘΕ͍ͯΔ ◦ VirtualService ಺ͷݸʑͷϧʔςΟϯάઃఆ͸ VSConfig ʹهड़͞Ε͍ͯΔ ͜ΕͰे෼͔ʁ
25. © 2025 Wantedly, Inc. ϚΠΫϩαʔϏεʹ͓͚ΔϦΫΤετॲཧͷྲྀΕ A B C E ௨ৗͷϦΫΤετ req B E A B C E req ؒ઀తʹϦΫΤετΛड͚ͯ ڍಈ͕มΘΔ ίϐʔͨ͠ Deployment Ծ૝Ϋϥελ΁ͷ ϦΫΤετ
26. © 2025 Wantedly, Inc. ʮԾ૝Ϋϥελ΁ͷϦΫΤετʯͱ͍͏ίϯςΩετ B E A B C E req Ծ૝Ϋϥελ΁ͷ ϦΫΤετ A B C E ௨ৗͷϦΫΤετ req ϦΫΤετΛ ίϐʔͨ͠ deploymen ΁޲͚͍ͨ
27. © 2025 Wantedly, Inc. ʮԾ૝Ϋϥελ΁ͷϦΫΤετʯͱ͍͏ίϯςΩετ B E A B C E req A B C E req ϦΫΤετΛ ίϐʔͨ͠ deploymen ΁޲͚͍ͨ Ծ૝Ϋϥελ΁ͷ ϦΫΤετ ௨ৗͷϦΫΤετ
28. © 2025 Wantedly, Inc. ʮԾ૝Ϋϥελ΁ͷϦΫΤετʯͱ͍͏ίϯςΩετ B E A B C E req A B C E req ϦΫΤετΛ ίϐʔͨ͠ deploymen ΁޲͚͍ͨ Ծ૝Ϋϥελ΁ͷ ϦΫΤετ ௨ৗͷϦΫΤετ ʮԾ૝Ϋϥελ΁ͷΞΫηεͰ͋Δʯ ͱ͍͏৘ใʢίϯςΩετʣΛ ޙଓͷϚΠΫϩαʔϏε΁ ఻ൖͤ͞Δඞཁ͕͋Δ
29. © 2025 Wantedly, Inc. Ծ૝ΫϥελͷϦΫΤετͰ͋Δ͜ͱΛ఻ൖ͢Δ • ͜͜·Ͱʢ࠶ܝʣ ◦ ಛఆͷ header ͷ͍ͭͨϦΫΤετ͸Ծ૝ΫϥελʹϧʔςΟϯά͞ΕΔ ◦ ϧʔςΟϯά͸ VirtualService ͰߦΘΕ͍ͯΔ ◦ VirtualService ͷ؅ཧ͸ VSConfig ͕ߦ͍ͬͯΔ Ծ૝Ϋϥελͷ ID Λ࣋ͭ header Λ͚ͭͯ ϚΠΫϩαʔϏε಺Ͱ࢖͏ɾ఻ൖͤ͞Δ → ϦΫΤετ͕ίϐʔͨ͠ Deployment ΁ → Ծ૝తͳΫϥελʹΞΫηε͢Δମݧ
30. © 2025 Wantedly, Inc. ୭͕ header Λ͚͍ͭͯΔʁ Ͳ͏΍ͬͯ header Λ఻ൖ͍ͯ͠Δʁ
31. © 2025 Wantedly, Inc. ୭͕ header Λ͚͍ͭͯΔʁ Ͳ͏΍ͬͯ header Λ఻ൖ͍ͯ͠Δʁ ຊ೔͸ͪ͜Β͚ͩઆ໌͠·͢
32. © 2025 Wantedly, Inc. ʮԾ૝ΫϥελʹΞΫηε͍ͯ͠Δʯ͜ͱΛ఻ൖ͢Δ
33. © 2025 Wantedly, Inc. • ಛఆͷ ID ͷ෇͍ͨϦΫΤετ (e.g. header) Λड͚Δ ◦ ड͚ͨϚΠΫϩαʔϏε͸ผͷϚΠΫϩαʔϏε΁ϦΫΤετ ◦ ͦͷࡍʹಉ͡ ID Λ͚ͭͯϦΫΤετΛൃߦ͢Δ Ծ૝ΫϥελͷIDΛ఻ൖ͢Δ A B C E req id: hoge E req id: hoge
34. © 2025 Wantedly, Inc. • ಛఆͷ ID ͷ෇͍ͨϦΫΤετ (e.g. header) Λड͚Δ ◦ ड͚ͨϚΠΫϩαʔϏε͸ผͷϚΠΫϩαʔϏε΁ϦΫΤετ ◦ ͦͷࡍʹಉ͡ ID Λ͚ͭͯϦΫΤετΛൃߦ͢Δ Ծ૝ΫϥελͷIDΛ఻ൖ͢Δ req id: hoge req id: hoge req id: hoge header Λ఻ൖ͞ ͓ͤͯ͘ A B C E req id: hoge E
35. © 2025 Wantedly, Inc. Context propagation • ʮԾ૝ΫϥελAʹདྷͨϦΫΤετͰ͋Δʯͱ͍͏ίϯςΩετΛ ϚΠΫϩαʔϏεʹ఻ൖ (propagate) ͢Δ ◦ ྫ ▪ X-Fork-Identifer: fork-a ͷΑ͏ͳ header ෇͖Ͱ request ͷॲཧத ▪ ผͷ microservice ʹ௨৴Λߦ͏৔߹͸ඞͣ X-Fork-Identifer: fork-a Λ෇͚Δ • servicex ͱ͍͏ϚΠΫϩαʔϏεڞ௨ϥΠϒϥϦͰ࣮ݱ ◦ wrap ͨ͠ΫϥΠΞϯτΛ࢖͏ ◦ ଞ΁ͷϦΫΤετ࣌ʹಉ͡ϔομɾϝλσʔλΛ࢖͏ ◦ ΋͏গ͠஌Γ͍ͨ > Wantedly Engineering Handbook
36. © 2025 Wantedly, Inc. ·ͱΊ • kubefork ͸Ծ૝ΫϥελΛ࡞੒͢Δ࢓૊Έ ◦ ։ൃऀ͕ࣗ෼ઐ༻ͷΫϥελΛ͍࣋ͬͯΔ͔ͷΑ͏ͳମݧΛఏڙ͢Δ • ΠϯϑϥͱΞϓϦέʔγϣϯͷٕज़Λ૊Έ߹Θͤͯ Ͱ͖͍ͯΔ ◦ Kubernetes (VirtualService, VSConfig etc.), Istio ◦ servicex