{"templateId":"markdown","sharedDataIds":{},"props":{"metadata":{"markdoc":{"tagList":["blog-author","blog-share"]},"type":"markdown"},"seo":{"title":"#TDTechTalk : 5 challenges in CDP","description":"The first TD in-person meet-up in three years.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]},"image":"/assets/tdtechtalk2022tokyo-cover.5f646daeefd567e57f821fdab4e94088d6903a5fd9d54ed6e147f0b5f931b076.978384e4.jpg"},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"tdtechtalk--5-challenges-in-cdp","__idx":0},"children":["#TDTechTalk : 5 challenges in CDP"]},{"$$mdtype":"Tag","name":"BlogAuthor","attributes":{"name":"TATSUNO Taz Yasuhiro","date":"2023-02-01","image":"./images/taz.jpg"},"children":[]},{"$$mdtype":"Tag","name":"BlogShare","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["During the November 2022 TDTechTalk Meetup, five developers from various engineering team presented challenges and solutions in their area."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This post is a summary of the meetup. We are very happy to see the return of in-person and hybrid meetups and conferences here in Japan!"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["I usually work from home in ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://en.visitkiso.com/"},"children":["the mountainous countryside in Kiso, Japan"]},". This time, I took a three-hour train ride to Tokyo to attend the meetup there as one of the speakers."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The cover picture captured the catered meals I ate after my presentations. Participating in offline events comes with such kinds of fun!"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"presentations","__idx":1},"children":["Presentations"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"1-embulk-in-td-and-in-the-future","__idx":2},"children":["1) Embulk in TD, and in the future"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.google.com/presentation/d/e/2PACX-1vTp0RdX_-sczYqoavK-VVnGLq2Dcrb4K2r5n7QV1BkbJqY2L1RfNSi5eLaT-CF7lsBRFlkdeCVLs-XR/pub"},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/embulk-in-td.7ad0aa5c151cda0aefdad297f020bb938028e9ab658c250132c80a842576a2e3.a3195d46.png","alt":""},"children":[]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["(Follow the image link above to see the original presentation slides, in Japanese.)"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/dmikurube"},"children":["Dai"]}," talked about ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://www.embulk.org/"},"children":["Embulk"]},", an open-source bulk data loader. Embulk was released as open source initially from Treasure Data. Many companies have adopted the open-source Embulk inside their business."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Embulk is also the key component of Treasure Data's ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://docs.treasuredata.com/int"},"children":["\"Data Connector\" to ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["import customer data into Treasure Data from varieties of data sources"]}]},", such as AWS, types of relational databases, file servers, and cloud services like Shopify, Zendesk, and more."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Dai is one of the core contributors of Embulk. Starting from how he addressed technical debts in Embulk and in Data Connector, he talked about his ambivalence about being a volunteer OSS maintainer, and an employee of the for-profit company behind OSS at the same time."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"2-journey-to-improve-stability-and-scalability-of-plazma","__idx":3},"children":["2) Journey to Improve Stability and Scalability of Plazma"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://speakerdeck.com/yajirobee/treasuredata-tech-talk-2022-journey-to-improve-stability-and-scalability-of-plazma"},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/keisuke.2e234d60ab9484802f580980fdfc8ff0dbcb49f10ceb53520ead4d8f8b49a876.a3195d46.jpeg","alt":""},"children":[]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/yajirobee"},"children":["Keisuke"]},", from the Storage team works on ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://www.slideshare.net/treasure-data/td-techplazma"},"children":["Plazma, the petabyte-scale storage system at Treasure Data"]},". When data is ingested into Treasure Data by the Data Connector (e.g. Embulk), it is stored in Plazma."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Keisuke's story was the scariest at this meetup. Because it was about the potential system meltdown which was about to happen if no countermeasures were implemented within 72 hours!!!"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["His retrospection in the presentation was very insightful."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"3-hive-distributed-profiling-system-in-treasure-data","__idx":4},"children":["3) Hive Distributed Profiling System in Treasure Data"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://speakerdeck.com/okumin/hive-distributed-profiling-system-in-treasure-data-english-version-number-tdtechtalk"},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/okumin.c9d0f3a1addad129876256d509d45e8355091f09668d71a26eac9def7aaf0702.a3195d46.jpeg","alt":""},"children":[]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/okumin"},"children":["Okumin"]},", from the Query Engine team, which is responsible for running queries (SQL) over Plazma in a fast and cost-efficient way. At Treasure Data, customers run a variety of queries in large volumes and sometimes suffer from slow queries."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["To tackle slow queries running inside ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://hive.apache.org/"},"children":["Hive, one of the query engines used at TD"]},", Okumin first collected tons of stack traces of JVM into Plazma, our own storage system. Then he created a profiler for the massive amount of traces. He identified several bottlenecks visually, and contributed to Hive by fixing those issues."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["He praised Treasure Data as an exciting place for query engineers, since incredibly large volumes of real-world queries bring interesting challenges."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"4-treasure-data-cdp-in-30-minutes-magical-technology-expo-by-applications-team","__idx":5},"children":["4) Treasure Data CDP in 30 Minutes: Magical Technology Expo by Applications Team"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://speakerdeck.com/aamine/treasure-data-techtalk-2022-td-cdp-in-30-minutes"},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/aoki.7577afd247a9d30d81e99f23a0e5f7ed634a473142dbb83f5ca437b9b0cfdb06.a3195d46.jpeg","alt":""},"children":[]},{"$$mdtype":"Tag","name":"br","attributes":{},"children":[]},"(In Japanese)"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If you know Treasure Data from the early period, you may still recognize Treasure Data as a \"kinda cloud-based data warehouse (DWH) or data lake\". Yes, Treasure Data offers a platform for that purpose, but not only that nowadays."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/aamine"},"children":["Aoki"]},", from the Application team, introduced Treasure Data CDP (Customer Data Platform) built on top of Plazma, Query Engine and many components. CDP is a flagship product of today's Treasure Data's product portfolio. CDP gives valuable insights about customers for many roles in enterprise: marketing, sales, customer service, operations, digital engineering, and finance."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Aoki also demonstrated some \"magical\" technologies (ab)used in the early days of CDP. He and his team are continuously modernizing the implementation to make our CDP more usable and robust."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Aoki's talk refreshed the attendee's impression of Treasure Data products in a very amusing fashion."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"5-empowering-app-dev-by-nicely-crafted-high-level-aws-components","__idx":6},"children":["5) Empowering App Dev by Nicely-crafted High-Level AWS Components"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://speakerdeck.com/exoego/empowering-app-dev-by-nicely-crafted-high-level-aws-components"},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/taz.241316e3d0f179ad1706016255c7bcaafa85742f508747e4537b832e702f4fed.a3195d46.jpeg","alt":""},"children":[]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://github.com/exoego"},"children":["I, Taz"]},", from the Operational DB team, introduced an in-house application development platform for application developers across Treasure Data and affiliated companies, for customers' custom applications."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This story showed a different area of Treasure Data compared to the other 4 stories. Since the in-house app development platform is quite new and is being incubated internally."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"summary","__idx":7},"children":["Summary"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/nahi.c96c63c36f7d5a1cbd129151c68cd174da8bc210d918c25d4ba5693ce8d07ed1.a3195d46.jpg","alt":"nahi"},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["We hope you find these five presentations interesting and insightful."," ","In case you missed the meetup, ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://www.youtube.com/watch?v=XMM08z9yG3Q"},"children":["check the video recording on YouTube"]}," (only in Japanese). I hope we will see you at other meetups and conferences coming in 2023!!"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"acknowledgement","__idx":8},"children":["Acknowledgement"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Many thanks to ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://wovn.io/"},"children":["Wovn Technologies"]}," for providing the venue for the meetup. At Treasure Data, we are empowered by Wovn's internationalization solution."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"bonus","__idx":9},"children":["Bonus:"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The below photos captured what I enjoyed during this \"business\" trip."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/ramen.ccd7607ac89106423207bf2c05d8fb7fce8cd15c7e27ff05531ae67156d37442.a3195d46.jpg","alt":"Ramen"},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/curry.4d3db307bf66b3fe83f536e95d1dcd4b938989aca89a1dcf5352a1a3bc64d114.a3195d46.jpg","alt":"Spicy Curry"},"children":[]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/unaju.9e999304df26e16e26528149368869e5d40eaf3f8e57f2f56a3d5b83defeaac8.a3195d46.jpg","alt":"Unagi (BBQ grilled eel) over rice"},"children":[]}]}]},"headings":[{"value":"#TDTechTalk : 5 challenges in CDP","id":"tdtechtalk--5-challenges-in-cdp","depth":1},{"value":"Presentations","id":"presentations","depth":1},{"value":"1) Embulk in TD, and in the future","id":"1-embulk-in-td-and-in-the-future","depth":2},{"value":"2) Journey to Improve Stability and Scalability of Plazma","id":"2-journey-to-improve-stability-and-scalability-of-plazma","depth":2},{"value":"3) Hive Distributed Profiling System in Treasure Data","id":"3-hive-distributed-profiling-system-in-treasure-data","depth":2},{"value":"4) Treasure Data CDP in 30 Minutes: Magical Technology Expo by Applications Team","id":"4-treasure-data-cdp-in-30-minutes-magical-technology-expo-by-applications-team","depth":2},{"value":"5) Empowering App Dev by Nicely-crafted High-Level AWS Components","id":"5-empowering-app-dev-by-nicely-crafted-high-level-aws-components","depth":2},{"value":"Summary","id":"summary","depth":1},{"value":"Acknowledgement","id":"acknowledgement","depth":2},{"value":"Bonus:","id":"bonus","depth":1}],"frontmatter":{"title":"#TDTechTalk : 5 challenges in CDP","author":"Taz","date":"2023-02-01T00:00:00.000Z","categories":["meetup"],"image":"tdtechtalk2022tokyo-cover.jpg","seo":{"title":"#TDTechTalk : 5 challenges in CDP","description":"The first TD in-person meet-up in three years.","image":"/assets/tdtechtalk2022tokyo-cover.5f646daeefd567e57f821fdab4e94088d6903a5fd9d54ed6e147f0b5f931b076.978384e4.jpg"}},"lastModified":"2026-06-04T10:17:23.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/ja/blog/tdtechtalk2022-tokyo","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}