Transcriptly

This Insane AI Video Search Technology Selected by NVIDIA and Snowflake | Twelve Labs, Jae Lee

EO21:25Added Aug 31, 2026

Access the ‘5 Essential Resources for Using ChatGPT at Work’ provided by HubSpot for Startups here: https://clickhubspot.com/ecbs Today's story is about Jae Lee, the CEO of Twelve Labs. Twelve Labs is developing a multimodal video understanding LLM that comprehends various

Watch on YouTube →
Contributed by 刘嘉琪

Transcript

Transcript format

  1. Intro

  2. 00:00If you try to accommodate all of the advices that you get from your mentors,
  3. 00:04your company will most likely become like an underwear company
  4. 00:07totally different from what you wanted to build.
  5. 00:09Having some fundamental like foundation for yourself and for the company,
  6. 00:14and be able to say thank you for your advice, but no thank you.
  7. 00:18So having that the gut to say no to someone that you you respect
  8. 00:22as a founder, I think I grew a lot.
  9. 00:25Hi yo my name is Jae.
  10. 00:26I'm one of the co-founders and CEO of Twelve Labs.
  11. 00:29Twelve Labs is an AI research and product company
  12. 00:32based here in San Francisco and Seoul.
  13. 00:34We are building video foundation models for developers and enterprises
  14. 00:38building video centric products.
  15. 00:39We basically build humongous AI models that can understand videos like humans,
  16. 00:43and we serve it to developers via APIs that are looking into building really
  17. 00:48powerful semantic search or classification or summarization into their products.
  18. 00:52We currently have a little over 20,000 developers
  19. 00:56that are actively using our search API.
  20. 00:59We have largest creators of the world adopting Twelve Labs as well as media,
  21. 01:03entertainment, and large sports organizations and law enforcement.

  22. Chapter 1. A Startup Founded during Military Service

  23. 01:10So I was born in Seoul. I spent about ten years in Seoul.
  24. 01:15I had a chance to move to the States.
  25. 01:17I moved when I was 11, so I went through elementary
  26. 01:19and middle school in Knoxville, Tennessee.
  27. 01:21I was able to pick up a lot of the culture and things like that.
  28. 01:26I was very interested in expanding my perspective and exploring the new world.
  29. 01:31My first experience with software engineering,
  30. 01:33or at least coding, was Matlab.
  31. 01:35My uncle was actually getting his PhD at the University of Tennessee,
  32. 01:40so I see him kind of like plotting this distribution graphs and things like that,
  33. 01:44and which made me curious what what he was doing.
  34. 01:47That's how I got into playing around with like, small data sets
  35. 01:51and doing the same thing that he was doing, because I wanted to be relevant and I wanted
  36. 01:55to talk to him about a bunch of things.
  37. 01:56He probably thought what I was doing was pretty cute,
  38. 01:59which sparked my interest in learning more about how do we capture all this data
  39. 02:04and be able to create a system that can really understand the
  40. 02:07distributions of the things of the world.
  41. 02:10And it just kind of felt like if you had the understanding of that,
  42. 02:13it gives you power to predict anything.
  43. 02:15And I went to Berkeley for college and studied computer science, so I really
  44. 02:19geeked out on AI and software engineering.
  45. 02:22So I spent about 15 years in the United States.
  46. 02:25So half of my life in Seoul and in the States,
  47. 02:28I was drafted into this organization called Korean Cyber Command, where
  48. 02:33like minded people are already there,
  49. 02:35armed with incredible knowledge in software engineering and AI.
  50. 02:39I joke about it I served the country with keyboards rather than a rifle.
  51. 02:44I was really fortunate to have met my chief architect, SJ,
  52. 02:49and then Aiden kind of also joined in.
  53. 02:51He was, you know, I still clearly remember Aiden with his
  54. 02:54buzzcut coming in from from boot camp.
  55. 02:56But from day one, we knew we had this common interest in AI.
  56. 03:01And what can we do as a as a young scientist, really like push the frontier of AI development.
  57. 03:08And we've spent a lot of times reading papers, discussing, arguing.
  58. 03:13And it turned out there were two clear paths.
  59. 03:16One was after military, we go pursue a career in academia, go become a professor,
  60. 03:21or start something of our own.
  61. 03:23And looking back, what we realized is that we were spending so much time together and
  62. 03:28military is this really special setting where, you know, you're basically jamming in
  63. 03:3350 to 100 of 20 year olds, right? ith Testosterones
  64. 03:38what we thought was, you know,
  65. 03:39if we're having this much fun in military, imagine what we can do when we go out.
  66. 03:42So it was pretty clear to us that we're going to start.
  67. 03:45I think we've spent about a year and a half really thinking about what is
  68. 03:49the next frontier for AI, and how can we contribute to pushing that boundary?
  69. 03:54There's this seminal paper called Attention is All You Need, which some
  70. 03:58people call it the transformer paper, which is making a lot of impact now.
  71. 04:02But when it was published in 2017, at least for text and image based,
  72. 04:07you know, foundation models, probably capital was going to be a mode
  73. 04:10whoever raises the most amount of money.
  74. 04:12What we realized was there's still a lot of unexplored research areas
  75. 04:17for multimodal video understanding
  76. 04:19there, rather than capital is probably going to be really passionate,
  77. 04:23smart, but also slightly dumb people.
  78. 04:26That's like dumb enough to start.
  79. 04:27It will have really good chance of succeeding.
  80. 04:30And also we've realized that with the explosion of that video and, and other
  81. 04:34complex multimedia data is going to be the infrastructural data for the internet.
  82. 04:38And we realized that probably developers and enterprises need something better
  83. 04:42than object detection or transcription to make sense of all, all this video data
  84. 04:47that the humanity is creating.
  85. 04:48So it was like a no brainer to start building for video Understanding
  86. 04:53the model that Twelve Labs is building,
  87. 04:55it's basically trying to map human language into whatever that's
  88. 04:59happening in video content.
  89. 05:00So if you can map precise human language to whatever that's happening
  90. 05:04within video content, that gives you this emergence capabilities,
  91. 05:08like being able to search for things really well
  92. 05:09or being able to classify things or summarize,
  93. 05:12you know, we didn't all join the Korean Cyber Command at the same time.
  94. 05:16So SJ was already like six months ahead of me, and then Aiden was six months behind.
  95. 05:21So like, okay, we decided we're gonna, we're gonna start this company.
  96. 05:26But then SJ is like leaving like next year and then I'm leaving the year after,
  97. 05:31and then Aiden is like six months after.
  98. 05:32So how do we do it? So it was genuinely very scary.
  99. 05:35So we had our ideas.
  100. 05:36I remember SJ was discharged on Thursday and he came back to the military base
  101. 05:41Saturday of that week with our laptops, and he took us out
  102. 05:44in front of our military base.
  103. 05:46There was a bagel shop called Last Bagel and that was like our office
  104. 05:50where SJ would bring all of our laptops and we would do our research and do
  105. 05:55some little bit of prototyping.
  106. 05:57And we did that for six months.
  107. 05:58And then I got this charge, and then I did the whole laptop
  108. 06:01carrying and and taking out Aiden.
  109. 06:03So we did that for like a good like year so that everyone is out.
  110. 06:07And then we had a bunch of friends that were working in AI and crypto
  111. 06:11and blockchain and Web3 was just booming.
  112. 06:14And we had a mutual friend, like the founders had a mutual friend that
  113. 06:17had like a really nice office in Seoul, and it was like, oh, you guys can come in
  114. 06:21and use our office space.
  115. 06:22So that's what we did.
  116. 06:23And then after like three weeks, that company went bankrupt.
  117. 06:26And then really scary, like people started coming in and we had like our desktops
  118. 06:31and our GPUs all set up there.
  119. 06:32And we got really scared.
  120. 06:34So we brought everything back out and we found like really tiny, tiny office,
  121. 06:38probably like a size of a dressing room.
  122. 06:41That's where, like, all five of us kind of spent the next six months
  123. 06:45before we raised our proper seed round.
  124. 06:48Looking back, if we were to do it again, I don't know if we'll be able to do it,
  125. 06:53but some people say ignorance is bliss.
  126. 06:55And I think we were just like, really naive and just really excited
  127. 06:58about building this company.
  128. 07:00And I guess not knowing what was ahead allowed us to kind of do what we did.

  129. Sponsor

  130. Chapter 2. What a startup with only $2000 can achieve

  131. 08:10When we first started the company and we hired our first employees,
  132. 08:13they had a hard time explaining what Twelve Labs does to their parents.
  133. 08:17Really new thing. What is foundation model?
  134. 08:19What is video understanding?
  135. 08:21It's a new concept that's hard to understand for I guess,
  136. 08:24people that are not in this space.
  137. 08:26Nowadays people talk about the foundation layer, the tooling layer
  138. 08:30and the application layer.
  139. 08:31It all. Everyone's very familiar.
  140. 08:32When we started Twelve Labs, I think technologically it made total sense.
  141. 08:37We knew it was going to happen, but what was uncertain about it
  142. 08:40was will market accept?
  143. 08:42But knowing that we are at the verge of breakthrough in building an AI that can
  144. 08:48at least, you know, get to a certain level of human understanding of videos.
  145. 08:52So we were betting on markets acceptance of foundation models.
  146. 08:56The founders are pretty pretty much broke, right?
  147. 08:58Because we spent two years at a military, and I think we had $2,000 to start with,
  148. 09:05so barely, barely enough to do anything but figuring out what was going
  149. 09:09to be impactful that we can do,
  150. 09:12given our current resources that will put us in the map,
  151. 09:14or at least, you know, let the world know that what we're doing is relevant.
  152. 09:18So our tactic here was, okay, we're going to talk to a bunch of customers.
  153. 09:22And there were early believers in Twelve Labs who took our APIs
  154. 09:26and built awesome things with us, but we needed more exposure basically.
  155. 09:31So as a team, we've decided to participate in ICCV
  156. 09:35it's International Conference in Computer Vision.
  157. 09:38They're putting this like awesome competition for video understanding.
  158. 09:42So we talked to Aiden. We have nothing to lose and only to gain.
  159. 09:46The team was extremely supportive of of Aiden spearheading
  160. 09:50that effort with the team.
  161. 09:51All I can do to support is.
  162. 09:53You know, there was some ideas and and directional kind of feedback
  163. 09:57that I gave to Aiden, but we needed compute and we needed determination
  164. 10:01to put some serious cash behind
  165. 10:03And back then for Twelve Labs like $200,000 in in compute wise,
  166. 10:07it was a lot of money for us.
  167. 10:08So and just thinking that, okay, we're going to blow through $200,000
  168. 10:13in ten days in compute was really scary.
  169. 10:17But the team was able to use that capital, that that precious capital
  170. 10:21and build something incredible helped us win the competition.
  171. 10:24So I think the important thing is, if you're building something
  172. 10:28really impactful and you think that it's going to significantly
  173. 10:31change the industry that you're in,
  174. 10:33there will always be someone that has very similar thesis.
  175. 10:37It's just a matter of how do you get yourself out there?
  176. 10:39How do you let the people know that you exist.
  177. 10:43For us, that was the competition.
  178. 10:45After winning the competition, companies like Index Ventures and Radical Ventures
  179. 10:49had very strong thesis around multimodal AI. The next idea would be what's next?
  180. 10:54And video happens to be the most relatable multimodal data.
  181. 10:59These amazing companies actually came in inbound, so they reached out to us
  182. 11:04and we started jamming.
  183. 11:06The conversations turned into next conversation,
  184. 11:09and then we talked about technology and and it just happened very serendipitously.
  185. 11:14The first pitch deck I was in Seoul, and my first call with Index Ventures
  186. 11:20was at like, it was like 3:30 a.m.
  187. 11:22Seoul time, and we didn't have pitch deck, and we just felt like
  188. 11:26this is our first meeting and we knew nothing about fundraising then.
  189. 11:32We didn't even know this was going to be like a friendly introduction,
  190. 11:35but I just kind of felt the need to like, oh, we don't have pitch deck,
  191. 11:39we need to build one.
  192. 11:40So I remember staying up till like 3 a.m. Building it.
  193. 11:43I think the storyline was like quite simple.
  194. 11:45We didn't have much too much to show for it.
  195. 11:47The idea was, hey, the problem that we're solving is massive.
  196. 11:5080% of the world's data is in video,
  197. 11:52and there's no adequate solution out there or technology out there for developers
  198. 11:56and enterprises to make sense of it all.
  199. 11:58That is the market that we're we're tackling.
  200. 12:00We want to index all of that, like 80% of the world's data.
  201. 12:04And this is the technology underlying research work that we've done.
  202. 12:07And then that was it. VCs asked a lot of hard questions.
  203. 12:10The most memorable one, if I bring you TikTok as a customer
  204. 12:14and they want to index like billion hours of content, how long does it take?
  205. 12:18And I think we were we were thinking about maybe like 1,000,000 hours.
  206. 12:22It's going to take like ten years, right?
  207. 12:24That's when we realized we should never be comfortable
  208. 12:27with what we've built, this whole new, incredibly large world out there.
  209. 12:32You know that maybe some people are impressed with our system being
  210. 12:36able to index like 1,000,000 hours.
  211. 12:37But there is others that are thinking about billion, 10 billion,
  212. 12:40100,000,000,000 hours.
  213. 12:41And that was like a really challenging question because like what we said
  214. 12:45during our pitch is we want to index all of the world's videos.
  215. 12:49And that question kind of made me stunned.
  216. 12:52And to think about all of the technical issues that that we had at the time,
  217. 12:55he probably find it like, funny, right?
  218. 12:57I was trying to give my best answer as a founder.
  219. 13:00It's like, so the company has raised about $30 million in seed funding.

  220. Chapter 3. Lessons from Acquiring Early Customers

  221. 13:08My name is Soyoung Lee.
  222. 13:09I'm one of the co-founders of Twelve Labs, and I currently lead our business
  223. 13:13development and go to market.
  224. 13:14We have a customer who was paying for our product,
  225. 13:18but they weren't actually using it.
  226. 13:20But we had gone through a lot of work to actually get them as a customer,
  227. 13:24through a lot of sales and kind of relationship building and so on.
  228. 13:27But I think they were, you know, they were extremely early and we had
  229. 13:30almost pushed the sales to happen.
  230. 13:32And I think what we learned from that experience was we have to optimize even
  231. 13:37early on, it might have been better for us to not actually make the sale because
  232. 13:41the customer probably wasn't ready.
  233. 13:43They didn't have the passion or the innovative drive
  234. 13:46that our other customers had had.
  235. 13:48But we were optimizing for kind of hearing the Yes
  236. 13:51We tried so hard to make that no into a yes.
  237. 13:54And we had succeeded.
  238. 13:56But at the at the end, I think it turned out that we probably should have kept it out No
  239. 14:00and focused on all the other customers where the yes was more clear,
  240. 14:05and they had a very clear vision of how they can build their
  241. 14:08new experiences with the technology,
  242. 14:11and especially for earlier products and earlier technologies
  243. 14:14where resources are limited and you want to build for your best customers
  244. 14:19and the innovators in every field,
  245. 14:22you should probably start to optimize for hearing the no than the yes,
  246. 14:26because that will help you find the right direction faster.
  247. 14:29We quickly.
  248. 14:30I think we learned through trial and error that the types of companies
  249. 14:33that we need to early customers that we need to find and work with are
  250. 14:37true innovators in their field,
  251. 14:39whether they come from like the content creation space, law enforcement space,
  252. 14:43e-learning and so on.
  253. 14:45You know, we've had instances of trying to oversell.
  254. 14:48We were going through sales, you know, sales 101 books
  255. 14:51or like sales methodologies.
  256. 14:53And we were trying to pitch to the customer, hey, here's the use case you
  257. 14:56could build out with our, you know, with our technology, you know, we'll improve
  258. 15:00your ROI by x percent with our technology.
  259. 15:03And we did make some sales from those, but I don't think it was something
  260. 15:06that we probably should have spent so much time on doing.
  261. 15:09Because if you find the right customer who's innovative, you don't need
  262. 15:12to explain anything to them.
  263. 15:14You show them a use case based demo of, you know, for us it
  264. 15:18would be we would index some videos that resemble the customers,
  265. 15:22and then we show them how you can search, or you can generate text very easily,
  266. 15:26like, like just like a person would.
  267. 15:28Had they been watching the video, they can draw out the full map of, okay,
  268. 15:32this is what I want to provide to my customers or my users.
  269. 15:36This is a technology that you have and I can they can fill
  270. 15:39that gap in pretty easily.
  271. 15:41And they know already what the return on that or what the opportunity
  272. 15:45of that experience would be for them, for the kind of really large, high profile
  273. 15:49customers that we have right now.
  274. 15:51It was all the same process.
  275. 15:53We actually learned all of these from the customers because they had, you know,
  276. 15:56seeing our demo and seeing the early kind of hints of the technology,
  277. 16:00they were able to teach us about how they could utilize the technology.
  278. 16:04And even today, it's not just a single use case that they want to power.
  279. 16:07They come to us with 4 or 5 different ideas of how the technology
  280. 16:11can impact different units, different business units and different,
  281. 16:15you know, optimize different workflows or build new experiences.
  282. 16:18I think we we make stupid mistakes probably every day.
  283. 16:21The one that I regret the most is we knew we had like this conviction
  284. 16:26around building a foundation model, but then we didn't have any data point
  285. 16:31as to how do you build that company.
  286. 16:32And I think I blindly kind of believed that the startup mantra like,
  287. 16:37identified a narrow problem and building a very narrow solution for it.
  288. 16:41I think we've spent a lot of early days thinking about, okay, if we have
  289. 16:45this really powerful AI that can understand videos, what do we do with it?
  290. 16:49And we know, we know from the from the get go that we wanted to serve it
  291. 16:52to developers and enterprises.
  292. 16:54But I think, you know, we've had mentors and, and other founders
  293. 16:58like, oh, you should build TikTok 2.0 or you should build like YouTube 2.0
  294. 17:02or you should build that.
  295. 17:03And I think we've spent a lot of time thinking, okay, maybe like TikTok
  296. 17:072.0 makes sense, or maybe like Gong 2.0 like sales call analysis.
  297. 17:12But that didn't really like excite us because we knew, like, we're good at
  298. 17:16building the infrastructure and helping developers build the next thing.
  299. 17:20And that's probably the stupidest thing that we've done.
  300. 17:23It's like spending time on thinking about things that we're not excited about.
  301. 17:27As a founder and CEO, it's really hard to get distracted.
  302. 17:31If you are a first time founder or a young founder,
  303. 17:34your mentor's advice means a lot to you.
  304. 17:36But having your own grounding, having your own, also really like relying a little bit
  305. 17:42on your own gut feeling is very important because what we've realized,
  306. 17:45if you try to accommodate all of the advices that you get from your mentors,
  307. 17:50your company will most likely become like an underwear company, totally different
  308. 17:54from what you wanted to build.
  309. 17:55So my key takeaway is having some fundamental like foundation
  310. 18:00for yourself and for the company, and be able to say thank you
  311. 18:04for your advice, but no thank you.
  312. 18:06So having that the gut to say no to someone that you respect as a founder
  313. 18:11I think I grew a lot right.
  314. 18:13Twelve Labs has multi-year compute partnership with Oracle Cloud
  315. 18:17Infrastructure, where we get all of the state of the art Nvidia chips.
  316. 18:21And Oracle have put together this small event for their key partners
  317. 18:26that are building foundation models for.
  318. 18:28And Aidan and I had chance to meet with Jenson because Jenson was at that event
  319. 18:33and we had, I think,
  320. 18:345 to 10 minutes to talk about Twelve Labs and I think it seems like he had he has a
  321. 18:39special place in his heart about computer vision and video understanding is,
  322. 18:43you know, that was one of the first use cases that Nvidia chips powered.
  323. 18:47So we got to meet with like Nvidia folks from that event.
  324. 18:50And then Twelve Labs was featured in Nvidia's 2023 GTC.
  325. 18:55And then I think that kind of sparked other people from Nvidia
  326. 18:58to be interested in Twelve Labs.
  327. 19:00And the Nvidia's venture team reached out to us.
  328. 19:03It was quite casual.
  329. 19:04We were talking about Twelve Labs and the future that we're drawing and the
  330. 19:08future of multimodal video understanding.
  331. 19:10And I think the venture team also had an idea of how Nvidia
  332. 19:15and Twelve Labs can partner up more than just financial investment,
  333. 19:18but also think about really robust product partnership from then on,
  334. 19:22what Twelve Labs is doing and what Nvidia wants in vision and video understanding
  335. 19:26was just like a perfect match.
  336. 19:28It happened quite naturally from from conversing.
  337. 19:31Our technology, our roadmap and Nvidia's future in terms
  338. 19:35of like producing really powerful chips for edge devices for smart cities, right?
  339. 19:41So there's that natural fit of two companies product really creating synergy.

  340. Chapter 4. The Best Engineer is not the Best Coder

  341. 19:50Nowadays I am focusing mostly on hiring.
  342. 19:54So I think Twelve Labs is a group of great people.
  343. 19:57I spend a lot of time meeting great people.
  344. 19:59I want to be able to recognize greatness when he or she comes in good engineers
  345. 20:04or even like just good people in general have this core values.
  346. 20:09I would go near their places and get together at a cafe,
  347. 20:13and we would speak for three four hours.
  348. 20:15My way of deciding whether this person is a good fit for Twelve Labs is
  349. 20:20I'm able to learn from their core values.
  350. 20:23Everyone's really good at coding nowadays, but great engineers can apply
  351. 20:26their core values and their skill sets,
  352. 20:28and is able to talk about the company that they're excited about
  353. 20:32and how they want to impact it.
  354. 20:34How do you see the product evolving? How do you see our interfaces evolving?
  355. 20:38Some of the best engineers are not the best coder, but having that perspective,
  356. 20:43really strong perspective and and groundedness is very important,
  357. 20:47and I try to look for that.
  358. 20:49Twelve Labs vision in the next two years is really becoming horizontal.
  359. 20:54Video understanding infrastructures for all of the businesses and developers
  360. 20:59that are working with video data, we want to enter into streaming as well.
  361. 21:05So real time video data and really become a visual cortex
  362. 21:11for modern video applications.
This Insane AI Video Search Technology Selected by NVIDIA and Snowflake | Twelve Labs, Jae Lee — Transcriptly