Upwork Interview Transcript

Intro - 0:00 My name is Sree Kamireddy. I lead the search and recommendations team at Upwork.

Launch Impact - 0:05 We are able to see positive improvements in contract starts. We also shipped a recent A/B test to deliver ads on newer surfaces and both of those we were able to do at a much quicker pace given Promoted is built from the bottom up to be able to deliver both ads and organic search results.

We all work together! - 0:52 So it’s very interesting, it’s very complex, and it’s high scale, and Promoted does this for you anyway, right?We all work together! Absolutely!

Using the Best - 1:01 Upwork takes a lot of attention to using the best out there to build that perfect match engine for the clients and talent.

How You Know It Works - 1:15And, how do you know that search, discovery, and your native ads are working? This is a common set of metrics that we clearly define —our success metrics. And, we look at a lot of variables. And in terms of the metrics, all the way from —are the contract starts increasing —that’s just one metric, and it can be a very short-term metric, so we also look at, are they hiring managers coming back again to do the hiring? Are the hiring managers coming the third time? And are those factors increasing? So not only contract starts are increasing, the retention is increasing. And, we’ll hopefully start doing more of also optimizing directly to increase the gross services volume in itself.

Measuring Unified Ad Impact - 2:03 And ads is another dimension, which is: we don’t want to, obviously, just get ad revenue at the cost of experience and the cost of the organic ranking. So that has its own set of metrics. So it’s a complex slew of metrics to make sure that we are delivering the best possible experience for the users at any point in time.

Promoted Increases All These Metrics - 2:26And is Promoted optimizing and increasing all of these metrics?

Multi-Objective Optimization - 2:29 I think that’s one of the good things about Promoted: the objective functions we can definitely work on figuring out. We can do our internal analysis to see, hey, which objective function should we optimize for. And Promoted, the way the stack is set up, enables different combinations of objective functions. We started with some click optimizations, we did hire optimizations, and all of that, and easily combined them. There are blender functions and all that —not to get too geeky, but all of this is definitely all possible.

Continuous Monitoring of Progress - 3:05 The way I find value is that there’s definitely a Slack channel where I constantly see new models being shipped and what are the different scores for the new models so that you can compare on a day-to-day basis. You can compare as this new model with maybe newer feature inputs —but maybe not— most of the time, it’s newer data. And that newer data is causing this newer model to be retrained and potentially be a little bit better on a day-to-day basis. That part is super valuable. Super valuable as a collaboration. Also, personally, it’s super valuable because I can kind of see the improvements happen. So that’s the offline experimentation part for sure.

Multiple A/B Testing - 3:52 On the A/B testing, we do definitely do A/B our test while you [Promoted] also A/B test within that A/B test. So if we launched Promoted to treatment, you [Promoted] are able to A/B test within that treatment to see if some of the models [are better] and you can move faster within that A/B test, I think that is great.

Continuous A/B Testing - 4:15 But we [Upwork] continue to do A/B test[ing] because that allows us to really validate whether we are doing the right things for the customers.

Recommending Promoted - 4:30 I would recommend Promoted because one, the team is really passionate about this problem space. Like, I definitely see in everyone that they care about the search and marketplaces and doing this matching process really effectively. The team is clearly, very experienced and skilled in this. Everybody has done this in their past life. The speed of iteration, I think is very, very important. And, Promoted has the infrastructure and the stack to go fast and iterate fast and build newer models and ship quickly and that allows for innovation.

The Best Possible Thing - 5:07 This combination of being passionate and having the skills and being able to innovate: that’s a deadly combination. It’s like a top notch professional team. Like, I don’t have second thoughts like, when we are collaborating. And, like, okay, your team is going to do the best possible thing at that point in time.

Promoted Really Gets It - 5:30 I feel like I can talk about some geeky stuff about search and recommendations and this whole match. And I feel like your team really gets it, and that’s pretty exciting.

Promoted Infra to Move Fast - 5:45 Promoted has the infrastructure to move fast. So essentially, all the goals that we were trying to trying to hit, using their infrastructure to move fast, we are able to hit them sooner and learn what works and what doesn’t work sooner.

Sree’s Deep Expertise in Search AI - 6:01 I started in the space long before machine learning was the cool deal. I was a research assistant in this lab called Group Lens. They created a lot of recommendation systems back in the day. I was at Microsoft after that, was part of the founding team of Bing. Then there’s been big companies and large companies primarily using data and AI to drive business outcomes.

The Joy of Solving Search and Ads - 6:26 It’s a complex problem, it’s a complex problem, has complex solutions. Changing all the time. Yeah, it’s a puzzle, it’s a puzzle that you have to solve for and that’s exciting! Interviewer: Solving employment worldwide. Yeah! Think think about it as a game you’re playing on a day to day basis at work. And like how cool is that. Like you come into work, you kind of solve a puzzle and make the clients and freelancers happy. You solve another puzzle another day. It’s very, very interesting. Not a single boring day.