Full interview with Jim Blomo,
Head of Engineering at Outschool

Video Transcript

Andrew Yates:  Well, how is it working with Promoted?

Jim Blomo:  It's been amazing! We were looking for a partner to grow our ML stack, not just provide the specific service, but leave the team in a better spot than we started, and that's absolutely been the case, and then the results have continuously improved over time. And so that's the other main criteria and has also been awesome to see. So, a super responsive team and great results.

Andrew Yates: We forgot to start with who are you and what is your position and background?

Jim Blomo: I'm Jim Blomo. I am head of engineering at Outschool, and my background is in big data machine learning, and some of the search and recommendation technology at a few different companies.

Andrew Yates:  Awesome. And tell me a little bit about Outschool.

Jim Blomo: Yeah. Outschool is a marketplace for online classes for kids. So my son has enjoyed classes on everything from coding and Minecraft to social clubs around geology. My daughter loves classes like Moana dance and learning letters with animals, and we provide just a really wide range of online group and one-on-one classes for kids.

Andrew Yates:  Awesome. And when you said that it was impactful, can you talk a little bit more about that?

Jim Blomo:  Yeah, absolutely. So I guess there are two main ways that we wanted to see an improvement on search. One was an in conversion rate, and then relatedly, because we're a marketplace, those conversion rates can mean different things because teachers set their own prices and times on the platform.

And so, we saw conversion rate improvements pretty quickly after starting to send signals. And then recently we adopted a change to try to lift kind of the expected class value. So a combination of conversion rate and the expected bookings that we would get from that conversion. That factors in things like class value and expected time in class, and number of attendances.

Andrew Yates:  So something that happens substantially later after the original booking. There are all of these following purchase events that are part of your actual revenue.

Jim Blomo:  Yes, exactly. So on Outschool, we have different types of class formats. Sometimes it's a one-off class that you take once, and sometimes it's an ongoing class, like a book club or continuing lessons in math and those ongoing classes can have some sort of expectation of future booking events, just like you mentioned.

Andrew Yates: You mentioned that your background is in big data and machine learning, and you have a data science team, you have a search team at Outschool, and you're working with Promoted.

Can you tell me a little bit about some of your thinking coming into this?  Starting the engagement and then how your thinking has evolved over time and what you see in the future?

Jim Blomo:  Yeah, great question. So, we do have a data science team. We don't have any engineers specifically as data engineers.

Most of our engineers are on the infrastructure side or on the full stack side, and so when we started our journey looking at search, we had some support for looking at places where search rankings weren't ideal. But we certainly didn't have in-house the expertise of building a long-term platform for continually improving search through things like machine learning.

I think as most people know or will find out, machine learning might be 10% building models and 90% cleaning data, looking for signals, and building the ML operation structure around it. And that is all stuff that Outschool can and will build eventually, but we wanted to accelerate the impact of that.

And so working with a vendor like Promoted accelerates that. And as I mentioned in the beginning, that was one of the factors in picking a vendor. We wanted the team to end up in a stronger place than when we started. We wanted best practices and not vendor lock-in, and I think Promoted has provided that through things like pairing, through being really open about the types of signals, and how they're being used, about the results in the breakdown of new search rankings.

And so all of that has benefited the team and I think raised the level of the team so that we can use that information in other areas that still need ML investment.

Andrew Yates:  You said that search ranking has been one of the single biggest drivers of the revenue lift. Can you talk a little bit more about that?

Jim Blomo:  Yeah, absolutely. So, going through the search flow is one of the primary ways, especially for returning users, to book classes. So, looking either through topics or finding that perfect class for a learner that loves dragons and science, and finding a class that meets both.

Search is a really critical tool for that. So it's already a big lever that we have for impacting conversions and the ranking and bookings improvements that we saw through. The improved rankings that Promoted sends back, we saw a really large lift on that and I think it's fair to say it's been larger for many of the other product improvements that we've tested recently.

Andrew Yates:  Awesome. And you mentioned also that it wasn't a one-and-done black box. It's continuously improved and still improvin-we're actually launching things right now.

Jim Blomo: Yes!

Andrew Yates:   Tell me a little bit more about that process and how you expect that to continue to evolve.

Jim Blomo:   Yeah, totally. No, definitely not a one-and-done. We talked about some of the other operational things that need to go into machine learning, and Promoted has provided those including gradually launching new algorithms for rankings and being able to test those on a small subset of folks and then gradually roll that out as we get confidence in them.

All of that typically requires a big investment which is what we're getting through this partnership. So yeah, we've been really happy with that and I think it's enabled us to think more deeply about the problem of: Hey, wait, what are we trying to optimize for? What do we care about here? Instead of spending our day to day more granular signals or some of the technical incidental problems with search. So I think that's been really powerful as a way to keep the team's focus on taking big steps in our improvement of the user experience.

Andrew Yates:  Oh, awesome. When we started with Outschool, our impression was that you weren't starting from scratch. You had a good team, and you were following best practices, and we were able to improve on top of that and are continuing to improve. What were some of those best practices or other vendors and techniques that you’ve looked at and considered instead of Promoted?

Jim Blomo: Yeah, it wasn't too long ago, it was two or three years ago that we moved off of Postgres for Search and onto Elastic Search. So I think most companies go through this process. And so moving onto Elastic Search was already a pretty big step forward for us in terms of latency and relevancy and control over rankings and those sorts of things.

As you mentioned, we had started diving into those rankings and figuring out what aspects of the  results impacted the things like click-through rate and those sorts of things. So we'd already started playing with that, but the cycle time was really slow and it's because of all of these operational things that need to happen.

In terms of instrumenting your search and doing A/B testing on your rankings and seeing a lift and those sorts of things. So it was happening and we were taking small steps there, but moving to Promoted just accelerated that process dramatically. And as I keep mentioning, the data science, the ML investment hasn't gone away from search or recommendations, but it's allowed us to really focus it on the aspects of Outschool that are unique to us that we can really think through: Hey, what are the features that we think or have a hypothesis about personalizing search results better or just providing a better user experience, and we can just blast those to Promoted and have the machine learning infrastructure figure out the latent signals or features within those in order to convert that to better rankings. So it's just accelerated that cycle of learning.

Andrew Yates: When you're blasting these signals, what does that look like? You just send it and then it just works? Or how long does that take?

Jim Blomo:  Yeah. Again, one of the things that were important to me was that we develop a data infrastructure that I believed was best practice independent of vendors, and from my perspective, Promoted encouraged that, right? We didn't have to use some kind of hyper-specific API. We were able to use our, at the time, nascent Kafka data streaming setup to get signals piped through our data pipeline and then sent to Promoted, using a way that I believe could be impactful for lots of other systems.

And in terms of cycle time, from what I've seen, we're talking less than a week between getting signals. I believe there are some signals that require some extra massaging from the Promoted side. And I think Promoted and engineers there have been really responsive with getting those incorporated into the model in ways that will probably have the most impact.

But a lot of times I think the signals are just pretty straightforward. Things like the ranking of the class from prior reviews, like the star rating, the number of enrollments that have already happened, quality of the teacher, those sorts of things can be sent in a pretty straightforward way incorporated into the ranking model. And then we see the lift!

Andrew Yates: Including your data science team models. We did a launch with the expected class value, and that itself was a separate data science project, and we were able to get a synergy between both projects as opposed to having like one versus the other.

Jim Blomo:  That's exactly right. The data science side of that, I think was a really good step forward for us in part of what I was talking about in terms of leveraging some deeper thinking about data science and the particulars of Outschool versus some of the mechanics of signal creation and that sort of thing, and model building.

And so exactly, we started with a, currently a relatively simple model around the expected class value, but we've seen such a lift that I’ve talked to the team about: Okay, like what this means is we need to get our own model about customer LTV and the way that a booking impacts customer behavior.

And we take that to 10 because if we can get that at an amazing level, it helps both our own internal tools and understanding of users and can help Promoted do a better job at ranking.

Andrew Yates:   Oh, amazing. And any final thoughts for other top marketplaces or eCommerce apps that have a search and discovery challenge? Especially from the engineering management side.

Jim Blomo:  Yeah, I'll emphasize that going with a vendor generally for this was not a no-brainer. We did deeply considered what aspects of customer understanding and machine learning we wanted to keep in-house and I do believe that Promoted has allowed us to improve those things instead of giving them away.

And so I would recommend, considering what aspects you want to work with Promoted with, not only to being open about that, but trusting that in our case, working with Promoted has improved our overall stance in machine learning.h, amazing. And any final thoughts for other top marketplaces or eCommerce apps that have a search and discovery challenge? Especially from the engineering management side.

Andrew Yates:  Jim, thank you so much.

Jim Blomo: Thank you.