Inside DuckDuckGo

In this episode, Jeffrey (Operations) and Anne (People Ops) discuss how performance reviews work in a distributed leadership model, and how the process has scaled as we’ve grown.
If you have feedback on Duck Tales, or episode ideas, email us at podcast@duckduckgo.com.
Disclaimers: (1) The audio, video (above), and transcript (below) are only lightly edited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.
Jeffrey: Hello everyone, and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo. We discuss the stories, technology, and the people that help build privacy tools for everyone. In each of these episodes you’ll hear from the employees about our vision, product updates, engineering, or things like approaches to AI.
Anne: There’s a lot of
Jeffrey: My name is Jeffrey and I’m on the internal efficiency team at DuckDuckGo.
Anne: Yeah,
Jeffrey: And today I’m joined with my colleague Anne. Anne, do you want to introduce yourself?
Anne: Hi, I’m Anne. I’m on the People Ops team at DuckDuckGo. It’s been about five years now, which I can’t believe, but lots of exciting stuff over that time.
Jeffrey: Great. Today we’re gonna be talking about our internal performance reviews and how we run them. And how would you describe our performance review process to someone who’s never heard of DuckDuckGo?
Anne: Well, I think the important thing to start with is sort of describing DuckDuckGo’s distributed leadership model, because with distributed leadership, that’s going to feed into performance reviews. And so the short answer I would give with that is that there’s no one decider of performance. There’s multiple different ways, multiple different people that give input, as well as a really strict requirement that it be evidence-based. And we’ll get into this a bit later, I’m sure. But we’ve done a lot of work on the evidence-based piece and being an asynchronous organization really helps with that because mostly everything is written down. So that’s that’s the high level that I would give.
Jeffrey: Who do we how do we decide what collaborators get involved with each person’s reviews?
Anne: Yeah, this has been a really interesting question over the years. And I would I would just love to brag a bit about how we used to do it when it was, you know, we were a smaller organization and then talk about how we’ve evolved that because well, I think way long ago it was the organization was small enough to just kind of know. You knew who should be giving input for who. And then the organization grew to the point where we needed help figuring out, right? You can’t one person couldn’t hold it all in their head.
And so you very generously helped us create an algorithm that put out a spreadsheet that basically it ingested information about people’s leadership relationships to one another, their work overlap with one another, and their experience giving feedback, among other things, calculated all of those relationships, also taking into account load.
Because you know, maybe one person should review 50 people, but that’s just too many, right? So balancing those trade-offs and then making a suggestion, which was really cool. And then those suggestions obviously had to be reviewed by humans and still, you know, load adjusted and everything. And there were several rounds of human review. So that was great. And then that also came to the point where we outgrew that because even those suggestions were gonna start fraying at a point, and the human review was cumbersome.
And so we’ve moved, I think quite naturally toward leaning on functional calibrators who are closer to the functional work and then pulling in the objective side of the house, which is sort of like where work is getting executed and having those be the default inputs for people. But yeah, that’s kind of how that’s changed over time. And it’s really, it’s been a really exciting problem, fun problem to be a part of.
Jeffrey: For the most part it’s scaled with as the company has grown and we’ve just had to make small changes here and there.
Anne: Yeah, I it’s as a people person, I also say, you know, people say how do we keep our culture as we grow? And this is similar to that where you say we don’t keep something, right? We s how do we scale something in accordance with our values as we grow? And then this I think would be a really good example of that in a really unique one too.
Jeffrey: Cool. What makes evaluating performance here different from a typical company?
Anne: Well, similar to what I said before, right? Distributed leadership means you have to figure out who it’s not like a managerial chain where the manager writes a performance review with peer feedback and then that gets approved by their manager and that gets approved. You know, it’s not a chain of approvals, it’s a collaborative process, which is also why that was one of the main reasons. There were several, but one of the bigger but bigger blockers for us purchasing performance review software from a company was that none of them had the functionality to have the level of collaboration that we have, like back and forth collaboration. And so there’s that. And then the real just like I said earlier, reliance on data. We are an asynchronous organization that really leverages processes in our project management tool, Asana, to fill the gaps of what a lot of live meetings would be. And I would imagine other Duck Tales have talked about no meeting Wednesdays, which is like for real. At other companies, it’s kind of in air quotes, but I would say here, not all of their companies to be clear, but oftentimes that can slip. But here it’s pretty, pretty legit. And so yeah, we’re generally meeting light, and so everything is in Asana, and you can pull that in to work summaries. And that’s also something you’ve been involved with, is really having a set of validated summary of work over the period of the year with links, right? So you can actually go and dig into the work over the course of the year as needed. And yeah, that’s the other thing I would really be I would say is is unique that the reliance on evidence and links and things you can actually see and look at and review is pretty high.
Jeffrey: Right. You touched on this a little bit, but can you go into how the process has scaled as we’ve grown? Or evolved.
Anne: Yeah, so yeah, that was one of the things is obviously sw figuring out how to fill those collaborative evaluator roles. Another is automation. We’ve really, you know, built our own system that integrates with where work gets done for us, which is Asana. And as a people operator, I feel blessed that I’m just given IT resources to be like, hey, you need this system to work. Here’s some really awesome skilled IT folks to help you make it a reality. And we have some really cool people on the team working to build our own performance review system that integrates exactly with where work happens for us. And so I’d say that is one that we’ve built that system and it’s evolved over time alongside other systems, right? So we have things like our daily asynchronous personal stand-up.
Right. So everybody does a stand-up for themselves every day and that gets recorded. And so then that can filter back into this whole performance review process because you have a record of what you were focused on every day. You have a record of the project you were focused on every week. You we have a whole summary of the relationships of your project advisory, career advisory, everything. You know, we’ve basically been automating more of these things to be centralized in a way that’s digestible for users. And those two things have really evolved together in in a way that’s pretty cool.
Jeffrey: I’m really proud of the our daily stand up we call our daily one must, but it’s been with us for a while now and it’s all asynchronous where people can fill it in in the morning and then everybody who needs to know can see what their stand up is. Cool.
Anne: Yeah, that’s the transparency part too, right? You can kinda go see what anyone’s working on at any time, which is pretty cool.
Jeffrey: How is this process suited for our remote asynchronous first culture?
Anne: Yeah, I mean I think similar to the things I’ve said before too, it’s it’s it fits the culture because for one, we have all of this substrate to work with, right? We have all of the the asynchronous culture is really reliant upon things being written down. And when things are written down, you can really shortcut a lot of the you know, forgetting what happened or whatever, or even just misremembering. And so you know, we have like bias checks that can happen on the data because the data are actually there, obviously all reviewed by people. But yeah, so it’s fit for us for that reason because of what you’re actually working with. And then the collaboration, too, is I think this is just part of asynchronous work in general. If you’re in a live meeting discussing something, it really just kind of depends on your processing speed in the moment for what you’re thinking and deciding on.
And we don’t make decisions as a general rule live. We can have discussions about them. Or maybe if it’s a really gnarly one, but generally decisions are happening in a really thought out way. And that’s the same thing with performance conversations, right? So it’s not just having a live debate about something in a calibration, right? You’re actually looking at, you’re having time to consider things and look at the actual evidence as opposed to just having a live conversation with all of its associated biases. So that’s that’s one thing I would I would say.
Jeffrey: Default to having things written down helps with the evidence. Yeah. What are the challenges with the process?
Anne: Yeah, I guess the flip side of having the everything written down is it’s not actually everything written down, right? Obviously, we still there’s a lot of that happens with like context and in a chat or in a live call. And we don’t want to over rotate on, well, if it wasn’t written down, then it didn’t happen. And that just relies on people’s judgment. And so that does rely on people actually taking the time to look at things and make those judgments. I think we have really smart capable people doing that. So it’s a challenge that just is something that they need to keep in mind. It’s not really something we’ve had to solve for necessarily. And then as we’ve grown, the question, you know, a lot of this is very time-intensive, right? There’s a lot of conversation around each of these pieces and a lot of consideration that goes into it. And then, you know, just the question organizationally is: how can we make this easier for everyone?
As much as you know, we all really value performance reviews and also it can be if you’re someone who has a lot to do, it’s a lot of work. We very much appreciate everyone who puts the work in because everyone cares about doing it well, and doing it well takes a lot of time. So that’s been a challenge for us is how can we keep the quality bar while, you know, making it easier and easier to do every time.
Jeffrey: Mm-hmm. It kinda leads into the last question of what our future plans are. Do you want to go into any of that?
Anne: Yeah, so high level. We’ve done a lot of work over the years to streamline, like I was talking about all of the data, and then it feeds into, you know, we have up to five different like categories of ratings. We call them professional dimensions. And we’ve done a lot of work to sort of categorize those things and how so to streamline the amount of work you have to do as someone who’s evaluating. And all of that work really we’re trying to take it to the next level of having that happen in a real focus time period rather than in combination with other work. And also, we used to give so well, I’ll finish that thought first, which is we’re going to move to a shorter cycle, which is going to be about three weeks. That’s the goal. Where everyone, we talked about our daily standup earlier, where ever if you have a task to do that day for the performance review, your daily stand-up will be write my self-reflection.
Everyone at the company writes it that day, and then they can move on to other things. So that it’s not this kind of organizational cost that’s dispersed across some time. And it also gives people the freedom to really focus on it that day because it is their one must. So you can really put all your attention there. And that follows through the rest of the process. The other piece is, you know, we are we used to do these annual, you know, shout-outs and focus areas, right? Which was the sort of written part of the review.
Sometimes they could be quite long, and that takes a lot of time to review, also. And as we thought about it scaling, and we’re just thinking also about scaling our career advisor function and coaching and you know, leveling up career advisors as we scale, which I think is a very normal company scaling thing, is our annual shout-outs and focus areas right for everyone? Like really, we shouldn’t just be doing this once a year. And of course, we should all be getting coaching conversations in an ongoing way. And so we’re actually building the process around ongoing coaching that happens in folks one-to-ones regularly as opposed to sort of having this heavy text written piece in the performance review. So that’s something we’re experimenting with. And I say experimenting with because everything at DuckDuckGo is an experiment and changes in response to evidence and how it’s going and the needs of the organization, but that’s what we’re looking at next.
Jeffrey: I’m really excited about the shorter time period for it and the focused week that it’ll be great.
Anne: Someone who was one of the highest loaded folks at the organization, I yeah, that you’d be very excited.
Jeffrey: Yeah. All right. I think that covers all of our topic today. Thanks for joining me, Anne. See y’all later.
Anne: Great to see you, Jeffrey. Bye.


This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Beah (Chief Product Officer) and John (Support) discuss our customer support process, why we read all tickets (good or bad), and how we actually use them to improve all aspects of the user experience.
If you have feedback on Duck Tales, or episode ideas, email us at podcast@duckduckgo.com.
Disclaimers: (1) The audio, video (above), and transcript (below) are only lightly edited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.
Beah: Hello, and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or approach to AI. Today you’re gonna hear about how we do support at DuckDuckGo, and my guest is John. John, do you wanna introduce yourself briefly?
John: Yeah, for sure. Yep, I’m John. I work within our marketing and communications team. And for the past few years, my main goal has been managing and improving how our support operates for the DuckDuckGo subscription and beyond.
Beah: Sweet. So if you’ve used DuckDuckGo support, perhaps you’ve gotten a message from John or one of the other teammates who we’ll learn more about in this call or in this pod. So I’m Beah. If you haven’t met me, I’m on the product team at DuckDuckGo. All right, let’s jump in. I’m gonna ask John some questions and we’ll go from there. So tell me John, just to kind of set the scene, what sort of things do we provide customer support for?
John: A lot. The bulk of the support that we offer is through the DuckDuckGo subscription. So for us who handle support, it means handling things like billing questions, assisting with activation across multiple devices, questions about the plans that we offer, and more technical troubleshooting related things. The subscription can be bought a few different ways, each with its own different path to activation. So a good bit of the support that we offer is helping our users navigate these paths to get more comfortable with our subscription and its features. So some of our great front-end team members built a support form where most of our tickets run through. It was built to be kind of as self-service as possible, but a category is picked and depending on what the selection is, a potential solution gets surfaced. This way, plenty of users can get an answer without really needing to submit anything or wait for us to respond to them. But basically, our help pages kind of sit behind this and there’s links to that support form on every one of our subscription help pages. It’s the first stop for most of these tickets. We’ve been piloting an AI agent on a few help pages and I’m sure we’ll get into that as well.
Beah: So like if we don’t, if we can answer, if we can get people an article right then and there that answers their questions, that’s the best case scenario for everybody because they get an immediate answer, but if not, then we have a process to kind of hand it off to a human or an in-between, which is like AI assisted answers.
John: Exactly. That’s right.
Beah: And maybe I should say like before we had the subscription, which is the first paid product that DuckDuckGo offered, and the only one we offer, we didn’t really have a support team or support process. I mean, we had emails for different things, but like effectively we didn’t have to worry about customer support. And it was the introduction of the paid subscription that sort of like brought this problem to our doorstep and to you. Is that fair to say? Yeah, yeah.
John: Yes. Absolutely. Yeah. Yeah. Perfect.
Beah: Okay, cool. So tell me a little bit more about like how this all works like team-wise, who’s working on this, how have we structured that?
John: Absolutely. We have a small but mighty team. So when that subscription first launched, there were a few of us that kind of rotated handling these support tickets. After a few months, we streamlined it significantly to where I was the only one who was handling things, which probably seemed like it was unfeasible and it really was. Early last year, we brought on an awesome specialist as...
Beah: It, I mean it, it worked, you just didn’t sleep great.
John: Yes, yeah, it worked. It worked. But there was a lot that was dependent on me and it caused a lot of lead time for some people to get responses. It was a lot to manage basically. So we brought on an awesome specialist as we continued to grow. She’s been awesome in the organization and the optimization of our entire end-to-end process. She’s got more support experience than I do by far and is a great Swiss Army knife of skill sets. So with her help, we, along with dozens of other teammates throughout the organization, we kind of overhauled the whole support structure at DuckDuckGo. And then recently, earlier this year, we brought on two more support rockstars to kind of help shoulder the ticket load to ensure that our users aren’t waiting too long before getting a real human response. So they’ve been a great addition. They’ve been working fantastically. And they’ve added an additional support layer to the mix. But yeah.
Beah: Got it. And you, you, you mentioned like lots of other people. I mean, so there’s like the people who are actually writing the emails back. And then I think part of what you’re getting at, tell me if this is right, is like, you know, they are likely in many, not in all cases, but in some cases, like need to go talk to an engineer to understand what’s going on, or like a product manager or somebody else who’s been working on the product directly day in and day out and like they’re just kind of interfacing between like this huge, you know, pool of resources who know different things about DuckDuckGo.
John: Yes, yeah, and that, the whole concept of working out in the open really helps make that a lot easier. So we have engineers and developers that really plug in whenever a support signal points to the need to escalate for a better response that we don’t know or we can’t handle.
Beah: It has been for you.
John: So that direct line to the actual builders that are behind the scenes ensures that we go straight to the source and we don’t have to wade through multiple layers of people to get an answer that’s accurate.
Beah: Gotcha. Cool. So tell me, like, kind of riffing off of that, how does this fit into like our broader approach to customer feedback loops? Like we have this email support system, but we also like interface with users or get feedback from users like on social media or, you know, all these various ways, which I would say for context for listeners, like we really, we like really read that stuff and really pay attention to it. Like we’re pretty, I’d say we’re pretty obsessed in fact with understanding. Like if you’re sending us, if you’re a user and you’re sending us signals about your experience, like we are pretty hungry for that. So yeah, tell me about like the boundary, the interface boundary between like that kind of feed, those customer feedback loops and the support that you’re talking about.
John: Yeah, so it’s a big circle. So essentially, the support piece to me, from what I’ve seen, I feel like I’ve been in every aspect of support, internal and external, that we offer. And this support for the subscription that we do is truly one of the most honest and raw feedback channels that we have. Most of the time, people write in when there’s a problem. Like, you’re not going to be contacting support to be like, hey, great job. Each ticket gets categorized in those categories.
Beah: Though feel free to.
John: Yeah, you’re more than welcome to, yes.
Beah: Yeah.
John: But we don’t even, we don’t handle things that are just a subscription, we handle any and all questions that come in, which I might get, we might get a deluge of tickets with me saying that. But, that said, we get a really wide range of feedback. It’s truthful. It’s unsanitized. It’s anonymous. So one thing that I’ve insisted upon since we launched this is really reading that rougher stuff on purpose. So I thrive on reading through our negative feedback. A good bit of it relates not just to the poor experience with the support team that happened from time to time, but also their experiences with the products that we offer. So both halves of those go to those who can action them. When we see things, multiple complaints about a specific piece of the product, we’ll let the people know who could actually work to fix it. So actions like these help drive rewrites to our help pages that might seem confusing. Clear responses to our users in the future. Updates on our products, bug fixes.
Beah: That is a problem that would not be using my C.
John: And this was also the catalyst to including suggested solutions on our help, our subscription support form. So sometimes when you select something, you’ll see a suggested solution that bumps you to a help page where you can clear things faster, like we mentioned prior. But yeah, things like social media posts. If you’re on Reddit and you see a response from someone that’s like, hey, it’s John at DuckDuckGo. That’s me. And app store reviews, those get watched by our team and others. And then they’re routed to us when they’re support related. So it ensures that those feedback loops are all connecting together.
Beah: Nice. All right. So tell me a little bit, like, as our support function has grown and needed to scale as the subscription has grown, like what changes have you had to make?
John: Yeah, absolutely. So for a little bit of inside baseball, in late 2022, DuckDuckGo acquired the company that I co-founded, bringing my friend and I on board to help build out the personal information removal piece of the subscription. But one of the things that drew DuckDuckGo to our company of Removaly was our stellar customer support. And that was something I really wanted to ingrain in this subscription. So early on, I made sure to take a leadership role in building that out, which starting from zero is pretty... It was pretty fun to go from zero to where we are now. But at first it was really, really bare bones. We actually just ran it through an Outlook email. I personally replied to over 15,000 emails in the first year and a half that the subscription existed. Recently we switched over to Zendesk to help kind of manage a growing user base and get analytics and things that we could use to help improve. So between that switch and the addition of these other fantastic support team members, it’s really allowed me to step back and focus on the more operational aspects of how we support our users, but I still try to sometimes get into the weeds.
Beah: Yeah. That’s the basic problem to be for.
John: And we use those Zendesk analytics to inform continued growth for support and ensures that we scale effectively in parallel with the subscription and even.
Beah: Nice. Fifteen thousand. That’s pretty good, John.
John: Yeah.
Beah: All right, what are some of the main challenges of this system?
John: Privacy. Privacy is, while it’s fantastic here, it can be the bane of existence for support.
Beah: Yeah.
John: We don’t hold any information about our users by design, so we troubleshoot with a lot less context than most companies have. It helps make us better at asking questions and writing clear documentation, but it’s a real constraint, honestly. There’s also the consistent balancing act between speed, self-service, and satisfaction. And in many cases, we get to pick two of those, which negatively affect the third. So every improvement that helps people find answers faster, we have to weigh it against the experience of the person who still needs a human to respond to them. We watch that side really closely. So for the many, fast can never mean worse for the few right now. Of course, there’s also the typical ordinary difficult stuff. We don’t close down support over the weekends. So we cover seven days a week with a really small team. We have volume spikes frequently when we launch new areas, when we introduce new features, when we go viral on social media, when we start large-scale advertising initiatives like we did a couple weeks ago. We’ve got multiple billing platforms that have different financial processes and platforms we build on shipping really quickly that we have to absorb to make sure that we’re not giving incorrect or outdated information. But despite this, our really small but mighty team has managed to continue operating at a really high level of customer satisfaction. So that’s pretty sweet.
Beah: Nice. So you’ve touched on using AI once or twice. Tell me a little bit more about how you think about the role of AI in customer support. Yeah.
John: Yeah, so first I was really hesitant. I mean, I’ve seen and I’ve witnessed firsthand with other companies that I’ve interacted with that AI can be great and helps with scaling, but it could also like really frustrate people when they know that they’re chatting with someone that’s not going to be able to help them. It’s a bot. So we’ve done so.
Beah: Yeah. Yeah, I kind of forgot. I’d kinda forgotten about those early conversations, but you’re, you’re bringing it back. I think I remember being like, like I know what side John’s on.
John: Yeah, I fought it a lot. But we’re working on it. We’re trying things out. So we piloted kind of a privacy preserving AI agent through Zendesk on a couple of our support pages. And it was built to only answer from our own help page content. It handled some real volume. But we’ve had to kind of be honest with ourselves about the results. They’ve been OK. But there’s a lot of nuance involved in the questions that people are asking us. So we’re still ideating on it. We’re tweaking it. We’re refining it more. But that’s something where instead of shipping quickly, we want to make sure we move slowly and do things the right way. And then AI and the customer support space in general is really touchy, as I mentioned. It’s got to reduce the user effort without touching the quality of the answers or negatively affecting the privacy that our users expect from us.
Beah: Yeah.
John: So we tool that to configure conservatively for privacy. Help content is really kept as consistent as possible. And we’ve used AI a bit to kind of make sure that we are keeping our help pages the latest and greatest.
Beah: Okay.
John: Internally, we use our own AI tools where they really actually will help. But we still make sure that a human owns every reply that goes out that’s not specifically driven through that AI agent.
Beah: Yeah, that’s interesting. I was thinking about like, I mean, I feel like for us to give everybody the information they need quickly, it’s kind of a no-brainer that we should be experimenting with getting AI help on that. It feels like it, it could be quite, quite powerful, but I think right now I’m on a very frustrating customer support thread with, I’m guessing, mostly AI driven responses for like a piece of furniture I purchased.
John: Mm-hmm.
Beah: And it’s responsive, like it’s sending me a lot of emails, but they’re just like such fluff, you know? Like it doesn’t have any good answers. It’s not really taking my like email all that seriously. It’s just kind of holding me off, you know? And like every time I get an email, and it’s like, and it’s also like super apologetic. It’s like, like we’re so sorry. That must be so frustrating. And it like, by the like third message like that, I’m like, what? This is like so disingenuous.
John: Exactly. Yeah. Yeah. Yes.
Beah: Like, and I’m like, are you just AI? Like I think you’re just AI, right? And yeah, it’s, it’s tricky. But I do, but I mean like I, yeah, I want a response fast and I want it to be meaningful and in theory, like there should be a good application of AI to do that.
John: Agree. Yeah, it’s really critical to find a good balance of that. So you’re not providing a really fast service that’s not answering the question effectively or that’s providing that fluff that’s making people like, obvious that maybe this company doesn’t care about me as much as I thought if they’re not willing to have someone on the other end actually having this back and forth.
Beah: Yeah. There’s also like a little bit of a sense of, I guess like worst case scenario, like deception, you know? Like I’m like, are you, are you a robot pretending to be a human? Like that’s weird. I d, that’s just weird.
John: Exactly. Yep. Exactly. Yes. Yeah. So I’m slowly coming around, but I’m still pretty dubious about it in its, in its current state. So.
Beah: Yeah. Yeah, nice. All right. Well, to be continued, maybe, yeah, we can do a V2 version of this episode a year from now and see if, see what kind of progress you guys have made.
John: Yeah.
Beah: But let’s close on a more positive note. What aspect of our support, our customer support are you most proud of, John?
John: Man, honestly, I feel like we’ve been able to keep or improve our set-by-ourselves quality bar while we’ve continued to scale. So we’ve been able to survive that increase in volume without negatively affecting the quality of service that we offer. At Removaly, I mentioned my co-founder Kyle and I, we did the customer support. It was a two man crew and we did live chat and it was difficult. But that closeness to our users shaped how I wanted that quality bar to be reflected at DuckDuckGo, which, it’s not easy pivoting from a very small startup to one of the most visited sites on the internet. Three plus years and a lot of growth later, every ticket here still gets a specific human answer. Negative feedback gets read by people who can actually drive the change to the products that people are saying that they’re having issues with.
Beah: But I won’t make that. Yeah.
John: And our support is consistently having a seat at the table in product decisions. We’re actually being listened to when it comes to like, well, what do our users actually think about this? Are we getting feedback? What do you guys recommend? So right now our support team is four people. But we’re a small team whose effectiveness is so strong that honestly I’d really put it up against support organizations with entire support structures 10 times the size of ours. And I think we’d be able to beat them.
Beah: Yeah.
John: So being able to keep that quality bar despite scaling quickly and rapidly over the course of the past few years has been wonderful to see.
Beah: Nice. Well yeah, I mean, DuckDuckGo I think has like some DNA in a time when we weren’t one of the most visited sites on the internet and were like, Gabriel was, you know, personally super focused on, you know, talking directly to, to customers, to users, and I think we still have a lot of that DNA. I mean, we need to on the one hand find ways to make it scale, but I think sometimes like caring inordinately about an e, a single angry individual can be a really powerful tool for improvement and motivation. And yeah, so thank, thanks John for like all the work that you and the team do on that front and also to listeners who provide us with feedback. Thank you guys. I, we really try to make good use of your time.
John: Absolutely, yeah. Thanks for having me, Beah.
Beah: All right, later.


This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Marc (Engineering) and Zach (Data Science) discuss how we use AI internally: our most common uses, how we use it privately, and why our principles matter more when using AI, not less.
If you have feedback on Duck Tales, or episode ideas, email us at podcast@duckduckgo.com.
Disclaimers: (1) The audio, video (above), and transcript (below) are only lightly edited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.
Marc: Hi, and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or approach to AI. My name is Marc and I’m on the engineering team. I have with me Zach. Zach, you want to introduce yourself?
Zach: hi, yeah. I’m Zach Deane-Mayer on our data science team here at DuckDuckGo. I am still a Duckling, so I’ve been here for about ten months. and I work on building our internal AI systems.
Marc: Great. yeah, so today we’re gonna talk even more about AI, which is Zach’s specialty. I think today we’re gonna talk specifically about how we use Duck or sorry, how we use AI internally at DuckDuckGo. so yeah, I guess to start out, as a company, you know, we’ve adopted AI pretty pretty heavily internally as a tool to help us move faster. can you talk a little bit, Zach, about the the approach that we’ve taken here? Given that I think this you were in on the ground floor with this, right?
Zach: Yeah, yeah. So DuckDuckGo as a company takes I I think a somewhat unique approach to AI as we take a unique approach to everything we do. We very much have a culture of of things like first principles thinking and and you know our own spin on things.
Marc: Right.
Zach: And a very important part of what we do are our privacy and security. so before we rolled out many of these tools, we spent quite a bit of time internally getting comfortable with their failure modes and the blast radius potential for for those failures. One of the first places we started using AI a lot was with coding tools. So right off the bat with Cursor and then Claude Code. And you know, one, the models are very, very good at writing code. But then two, if you are effectively using tools like git and GitHub, an agent that’s limited to writing code can only do so much damage if your code base is backed up in GitHub and you don’t let, for example, people force push to main or whatever like that. Like if your GitHub is properly configured. And so we had a lot of questions like that early on. And then, you know, once once we were confident, we really were able to kind of let open the floodgates and let people really start start using both Cursor, Claude Code and now to a lesser degree Codex to to write code and automate as much of engineering as they can.
Marc: Yeah, yeah, that’s great. you mentioned privacy, right? Like that is obviously it’s a core tenet of our products. It’s also a core tenet of how we operate here. and yeah, to your point, we do tend to do things a little bit differently. so I’m curious how the privacy kind of shaped our internal workflows with respect to AI, given that, you know, w we even though we don’t collect user data, we do still have things that we obviously don’t want to to share more broadly.
Zach: Yeah. Yeah, absolutely. so our privacy engineers very much think of like a an information hierarchy of stuff that is you know very I don’t know what the word is I want to use for the scale, but there’s a there’s a high to low scale. and so there’s certain information that we don’t even save to disk in our production servers. There’s certain information that is stored in memory and is never retained anywhere in our systems.
Marc: Right.
Zach: and that is fundamentally the safest form of data. If you don’t if you don’t store a piece of information, it’s it’s very hard to accidentally store it in the wrong place and in a way that you don’t you don’t want to do. and and that kind of principle thinking applies to AI too. Like if if an AI system doesn’t have access to certain data, it It can’t do the wrong thing with that data. and one of those ways to prevent access is just by by fundamentally not not storing the data in the first place. As you go through that hierarchy, you know, there there’s information that we’re more and more comfortable with working on internally, like for example, you know, time series traffic of you know how many users are on our various websites, like that sort of aggregated data that can’t be tied to an individual person. That’s a lot safer to start saying, hey, I’m gonna, you know, I’m on the data science team, I’m gonna do some data science work and look at, you know, whatever, whatever it is that the set of questions I have. and then I think the kind of the bottom piece that is incredibly important is we pay a lot of attention to our data processing agreements that we have with all of our vendors, but especially our AI vendors. and the contractual agreements you have with people about what they are and aren’t allowed to do with your data are are also very important. so so yeah, there’s there’s a spectrum of things and you go talk to any privacy engineer, I mean you can very quickly learn how just how deep the problem goes.
Marc: Yeah, yeah, it’s interesting you mentioned principled thinking, right? Like I I think that’s something that is a core part of the culture here. And I I think it’s fair to say that we have, you know, spent a lot of time talking about how to adapt AI to our principles as opposed to adapting our principles to AI, right? which I feel like you see a lot in the industry right now.
Zach: Yeah, yeah. Yeah. Yeah. And and and and that’s I mean that’s a that’s a great point. Like two of our principles that matter a lot to me personally are questioning assumptions and validating direction.
Marc: Absolutely, yeah.
Zach: and you know, I’m I’m like the the AI, you know, waving, you know, the flag and the the pom-poms and saying, let’s go AI internally.
Marc: Right.
Zach: but like I I find I have to like, you know, remind people, just because we’re using AI doesn’t mean we throw our principles out the window. Like,
Marc: Absolutely.
Zach: I am you know, again again as a leader here, I am more interested in like like I want to validate direction too. And so I am very interested to hear from people places where AI isn’t working. Like, like I want to see, you know, the PR that’s a 7,000 line of code wall of text where a team’s going, whoa, like we can’t we can’t review this, let alone ship it.
Marc: Right.
Zach: And those those kinds of failure modes, like AI is rife with them. And we don’t, you know, we don’t just look at an AI’s output and say, well, an AI made it, so let’s ship it. Like you you should use AI as as you know, a lever against, you know, with your principles, not not as a as a reason to throw your your principles out the window. But like again, like just questioning assumptions and validating direction, like you know, AIs are notoriously bad at questioning assumptions.
Marc: Right.
Zach: Like this will happen all the time, even with the latest models, where you’ll be like, hang on a sec. Like you said X and then you said Y, which implies not X, like, but you never connected the pieces together in in in in the chain of logic.
Marc: Sure, sure.
Zach: and so I I think I think your principles matter more when you’re using AI.
Marc: that’s interesting. Yeah. Yeah, that I think that’s really well put. actually leads us into our next question to a degree. there’s yeah there’s a lot of hype right now, obviously with AI. I mean to put it mildly, right? the hype cycle is has been super interesting. there’s a lot of concern about AI in the workforce today, right? And we’re certainly not immune to that, although we don’t really see it that way. as a company. So maybe can you talk a little bit about how we’re how we’re getting so many people to use it effectively and and you know where we’re trying to get to.
Zach: Yeah. Yeah, so AI is a very powerful tool. and it’s also in some ways a a slippery tool. Like it’s it’s changing so fast. It there’s not like a long history of like I don’t know, I I can’t I can’t think of an equivalent technology, but there’s there’s not like The field has changed so fast, everybody’s brand new to it.
Marc: Yeah.
Zach: And there’s a lot of surface area of of unknown unknowns, of things you don’t know that that can go go wrong.
Marc: Well.
Zach: I I heard a joke I really like that like working with AI, it’s like a cross between Amelia Bedelia and the guy from Memento.
Marc: That’s good.
Zach: and and it’s a great joke, but it’s like, you know, one, interprets everything you said too literally. and then two, it forgets tomorrow everything that you said to it today. And like, like those two problems, like, you know. It’s you know, I I’ve been playing around with Fable a lot recently. It’s an amazing model for one-shotting really difficult programming problems. But you still get that sometimes over literality, like like what you say wasn’t quite interpreted the way you meant it, and you have to be careful of that. And then the other side of like, you know, you come in tomorrow and you have to curate context that you take over between sessions. And I I think honestly that that latter part has been the part for me that has been the most difficult to adapt to is just what do you remember and how do you remember and how do you persist that memory?
Marc: Sure, sure.
Zach: And that’s very much something you have you have to build for yourself. and and you know from the individual level to the team level, like how are we storing information in, you know, when an AI goes wrong Where do we put that learning so that we don’t, you know, forget about it again in the future?
Marc: Right. Right, right. Yeah, I think from an engineering perspective, I mean, you know, we’re both engineers, so I I I think it’s safe to speak about first hand experience there. it’s definitely changing the field, right? Like everybody acknowledges that. I don’t think it’s obviating the field. I’ve heard so many analogies about how, you know, AI is like this or like, you know, whatever.
Zach: Yeah.
Marc: One I heard that resonates a little bit again from a software engineering perspective is like nobody hand compiles code anymore, right? Generally.
Zach: Right.
Marc: And I feel like, yeah, it maybe not a perfect analogy, but it didn’t make like that didn’t obviate the need for strong engineers, right? It just changed the nature of what they’re doing on a daily basis. And I think at at most that’s what we’re looking at here potentially, but obviously it’s a hot topic.
Zach: Yeah. Yeah. Yeah. No, so I I was literally just listening in a podcast, I’m on a podcast, I was listening to podcast, I love it, this morning with Fiona Fung, who’s the manager of the Claude Code team.
Marc: Mm-hmm. Okay.
Zach: And she had just like this wild and one-off anecdote. She’s like, my old boss, like my first engineering manager I ever had, you know, called me up the other day and and she said, This guy started his career programming in punch cards. Like
Marc: Yeah, right.
Zach: Programming in punch cards.
Marc: Right.
Zach: And the guy called her up, was hey, I’m using Claude Code and you run the Claude Code product team. I just want call you tell you how cool Claude Code is.
Marc: Ha ha ha. Yeah.
Zach: and I thought that was like a really great piece of perspective. Like just like imagining the transition between programming in punch cards to programming in Python. And like I’ve never, you know, I’ve never, you know, I didn’t experience that because I wasn’t I wasn’t around. but you you forget how long the history is of of computing and how much it already has changed.
Marc: Absolutely. Absolutely. To think about people. I mean, I think most of us have at least tried some assembly, right? Like it I don’t know if they still teach that in university today, but they certainly did. even if it was just to expose you to it. But yeah, that’s not something that, you know, the average person or the average engineer is gonna be doing. But anyway.
Zach: Mm. Yeah.
Marc: So yeah, I guess we could start wrapping up. I think what we’ve talked a lot about engineering examples. I’m I’m wondering if you know, we have some non engineering examples of how we’re using this at the company.
Zach: Yeah. one of my favorites is is spreadsheets. I spent a lot of time early in my career building models in Excel. and I definitely developed like an aesthetic for like what I want an Excel model to be like. and then I stopped using Excel and both the interface changed, but like my muscle memory, you know, I never got to like the keyboard shortcut level is always point and click. But like I I’ve been frustrated a lot recently with trying to build a model in Excel and I can’t find the commands I want or necessarily even remember them. and what’s really interesting to me is is a lot of Excel models, you can you can give a data set to Claude, you can describe the problem you want to solve, you can describe the shape of the model that you want. You know, I want one tab with my input parameters, one tab with my data, and one tab with the calculations and output or something like that. And it’ll do a r a remarkably good job of building things like that. and there’s you know, there’s a lot of work that still happens in spreadsheets. and it’s really nice to be able to go to Claude Code and give it a spreadsheet and say, Hey, can you change the formatting? I’ve got all these cells where the important information is bolded and I want to remove the bolding and replace the bolding with like a red background. And that’s really tedious to do with pointing and clicking. and Claude Code, like it’s just reading the XML. And so it’s just like it’s very good at, you know. A a spreadsheet is an XML file and it can just say, Cool, yeah, I’m just gonna change around the XML structure and now instead of bolding, you’ve got red highlight.
Marc: Yeah, yeah. Got it. Yeah. Yeah. No, that’s interesting. I th I feel like yeah, as much as I’ve embraced it in, you know, my daily workflow, I still have not yet gotten the nerve up to trust it with something like a spreadsheet. But I know that a lot of people do, so I probably need to get over that.
Zach: Yeah. I mean, if your spreadsheets are in version control, as mine all are, because I’m a huge nerd, like there’s there’s limited room for it to really, really screw screw something up. Yeah. The other one so I just
Marc: Yeah. Sure. Sure. All right, great. Well, you know, please go ahead.
Zach: w one more is just like summer camp forms. Like I’ve got these forms, like I look actually no, it was a school form. I got a form from my kids’ school and it was like it was like a scan. But it like a little bit sideways, so all the fields are a little bit off. And so, you know, it like filling out a PDF form is annoying enough when there’s like the little boxes that you can type in. But it’s like,
Marc: Right. Yeah.
Zach: okay, I don’t want to print this thing out, handwrite everything. My handwriting’s terrible. Scan it, send it back. And I was like, I actually did, I said, you know, Claude, I said to Claude Cowork, here’s the form. I dictated to it what I wanted in each field. And I said, can you just iterate? Actually, it was Claude Code, not Cowork. Can you iterate on you know, take my text, turn it to text box, put it in the PDF, and then, you know, line them all up correctly with the fields, which was not easy. And it built itself a little loop where it was like, you know, it got like my information to bullet points is pretty easy. Map them to the fields is pretty easy.
Marc: Yeah.
Zach: And then it just did an iteration of okay, write them out to the spreadsheet and then just like adjust the X and Y coordinates of the boxes until it lined up well enough that I could email it back to the school. Yeah. Yeah.
Marc: Wow, that’s awesome. Yeah, I have not I wouldn’t even have thought to do that with AI. So I feel like I need to be a little more ambitious.
Zach: Yeah. Yeah. It was the same where like I I spent five minutes talking to Claude, gave it the file, sent it off in the background and it came back and it was done and I sent it. I just, I checked the data, I was like, Yep, that is correct and you know, emailed it to the school and instead of, you know, I mean, who knows? It would have been probably twenty minutes of printing it out, filling it out, scanning it, sending it back, plus a bunch of steps and I didn’t want to do
Marc: Yeah, sure, sure. All right, great. I yeah, I feel like we could talk about this all day.
Zach: me too.
Marc: but we should probably wrap up. So yeah, thanks a lot for your time and thanks for everyone else for for listening in.
Zach: Yeah. Great, great talking with you as always, Marc.


This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Konrad (Engineering) and Shilpa (Engineering) discuss Personal Information Removal: what is a data broker, how we remove your records without ever sending your data to our servers, and the ‘whack-a-mole’ of constantly changing data broker sites.
If you have feedback on Duck Tales, or episode ideas, email us at podcast@duckduckgo.com.
Learn more about the DuckDuckGo Subscription here.
Disclaimers: (1) The audio, video (above), and transcript (below) are only lightly edited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.
Konrad: Hello everyone and welcome to Duck Tales. We will look under the hood of DuckDuckGo, talk about stories, technology, and people who are building the privacy tools that you all use. And in each of the episodes, you’ll hear from different employees about our vision, updates to our products, how we do engineering, or how we approach AI. So today I’m your host. My name is Konrad Zwinell. And I’m working at DuckDuckGo for seven years now. I work on the front end team at the very beginning. Then I run privacy engineering team. And now I’m back on the front end team working with Shilpa, who is my guest today. And Shilpa is running a personal information removal tool team. And she will tell us more about it today. Shilpa, why don’t you introduce yourself?
Shilpa: Yeah, hi. I’m Shilpa. I have been in DuckDuckGo for, I think, a year and a half now. I joined in on the Apple team working on iOS and macOS. And for the last few months, I’ve been driving the team that does works on the personal information and mobile product.
Konrad: Nice. OK, so first question that people may have after hearing this intro is, what exactly is personal information removal? Could you tell us, high level Shilpa, what’s the product and what it does? What’s the idea behind it?
Shilpa: Yeah, so in the US, this is predominantly a US-based product. In the US, there’s entities called data brokers that basically harvest your personal information from various different sources, your online activity, social media, what have you. And they collect that information, and they basically buy and sell it as a commodity for advertisers and different entities. And our product basically goes and looks at all of these data brokers, looks for your personal information. If it’s there, then it submits a request to remove it. And if the data broker is complying, they will go ahead and remove it and we will be able to verify the removals for you as well.
Konrad: Nice. So basically reducing your footprint on the web, reducing availability to your personal information, such as address, phone number in some cases, relatives, if I remember correctly.
Shilpa: Yeah.
Konrad: So they collect quite a bit of information about you and make it really easy for advertisers, but also other people to look it up. For FV,
Shilpa: Yeah.
Konrad: yeah, so that’s a bit scary. How does it work from the user perspective? Let’s say you want to use PIR. You want to try it out. How is the experience looking? Could you share with folks how it looks UI-wise and what you have to do to try it out?
Shilpa: Yeah, so today, actually in some platforms, you can even go try it out for free, part of the experience on macOS or Windows, where you can go ahead and put in your personal information, some basic stuff, like your name and date of birth and city and you will be able to go see, we will go ahead and do a search on the various different data brokers that we support. And it should be able to show you all of the different brokers that have your information, the information that they do have. And the first time, if you see this actually, it’s a little bit, it’s actually quite a bit scary to see how much of your information is out there. And literally anyone can do a search and pull up a lot of your information about you, your family and stuff. So yeah, once we do the search and then we’ll go ahead and go to those, once we know that your records are there on those specific brokers, we go ahead and submit requests to opt you out.
Konrad: Thank
Shilpa: And then there may or may not be some steps in between all of which we will take care of for you. And then usually there’s a delay from a couple of days to a few weeks maybe. Then if the record is removed, we will go ahead and do a scan again and make sure that your record is removed. And then we’ll report to you that that record’s been flagged as removed. And not just that, we also do periodic scans to make sure we do like maintenance scans to make sure that your record is removed and it continues to be removed. Because the way that data brokers work is once they remove it, they’re not very intentional or deliberate about these records. They’re just collecting and gathering from whatever sources they’re getting. So if it’s removed, doesn’t mean it stays removed. Sometimes it goes, sometimes it comes, sometimes it’s merged from other sources. So it’s important to continue to keep, keep removing it.
Konrad: Yeah. And we keep on adding new brokers, right? Meaning, you keep the PIR running, we will add new brokers and we will opt you out out of those brokers. So it’s not only like insurance that your records don’t pop up on the same brokers, but as we find new brokers starting business and appearing on the web, we will try to opt you out from those too, right?
Shilpa: Yeah, yeah. It’s the whole world of data brokers is very fluid. There’s new ones coming up. There’s mergers happening. There’s some of them are going defunct. Some of them are temporarily defunct. They come back. So we’re all constantly keeping an eye out for how things are changing, fixing it and adjusting for the changes they’re making, as well as constantly looking out for new ones that are coming up, seeing how we can support removals on those sites.
Konrad: That’s right. And I guess that’s our biggest challenge, right? Like fighting with this ever-changing websites and brokers kind of even fighting back on purpose, trying to kind of avoid our product, like make it not work. Would you say it’s the main challenge? Do we have like some other challenges?
Shilpa: I would say that is pretty much the main challenge. They are deliberately or not constantly changing things on their side. And since they don’t have a very clean, clear request that we can make to them, just to say, here’s my record, go remove it. But it involves usually going through and scraping the website. It’s not a very clean process, which means that we have any minor changes on their end leads to us breaking things on our side. So we have to constantly keep an eye out and monitor, make changes, fix. So this constant whack-a-mole, I think, is one of the biggest challenges. It’s also a good engineering challenge. So we are always thinking of various different ways to innovate and try to be as flexible as possible and see how we can actually get ahead of them. That actually is like the main engineering challenge related to this product for us.
Konrad: Yeah, we both like a challenge. That’s why we are on this team, right? OK, let’s see. So we are both engineers.
Shilpa: Absolutely.
Konrad: So maybe let’s do some shop talk and talk how this work under the hood. So yeah, can you tell us a bit about that?
Shilpa: Yeah, so doing a little bit of engineering talk, normally you would imagine that, you know, like we take the personal information from the users and then we go and make a nice clean API call directly to data brokers and they give us a response and we can say either success or failure. But that is actually not the case. Data brokers do not make our lives easy. What we actually have to do is basically what you would do if you were to manually go ahead and remove it. You would go to their website, type in your information, do a search, see if the record is found, and then go to another form. Again, type in an opt out request, and in between there may be various different stages. Depending on the data brokers, you may have to solve some captchas. You may have to get a confirmation code from the email or click on a link in the email. So we do all of that in the entire lifecycle of search and remove a record. So yeah, and I think one unique thing to the way we do versus our competitors is that when we take your personal information, it stays on your device. We do not send it off to our servers where it stays forever and potentially vulnerable to various other attacks. So we’re very particular about keeping all of your personal information within your device. So we just make the request from your browser within your device and keep it in your device so that it’s a lot less liable to be breached in any other way from server side.
Konrad: Right, because that’s our approach. Because the competition existed when we first started building this product, but we didn’t just copy the existing model. We thought it through and decided that what they are doing, it’s not going to work for us because we don’t want to receive personal information from our users. We don’t want to process that. We don’t want to deal with it. And so we came up with this model where everything happens on device, which has some pros. It has some cons. But yeah, we are making it work. And yeah, as you said, it’s challenging. But it’s the right thing to do to keep your data private. OK, so we talked a bit about what’s under the hood. We talked about the challenges, how those brokers are fighting back and by also going offline and changing their websites all the time and changing those forms. And yeah, so that has been our focus for a bit. You started talking, you mentioned like a bit about like flexibility, but we also do monitoring. We do other things like how we are adapting to this like well, complex environment.
Shilpa: Yeah, so constantly we have various different kinds of monitoring set up. So we’re always keeping track of how many of the brokers are responding correctly to the scans. We’re seeing what percentage of the scan attempts are going through, what percentage of the opt out request submissions are going through. Anytime we see more than a reasonable amount of variation in these, we always have one person that is on call watching out for any of these anomalies, any anomalies come up, we go do investigation to see what fixes need to be done. And also, we’re also trying to see how to reduce the amount of reactivity in terms of fixing this. We are trying to see how can we incorporate as much flexibility into our products, so there’s less reacting and more, we’d be staying on top of things when in case things do break. Currently, major part of our roadmap is actually focused on how to be more flexible as a product. And we’re also exploring to see how we can like incorporate AI, it’s fairly well suited to monitor things and maybe even try to attempt to fix it even before a human takes a look at it. So we’re pretty excited to kind of start working on all of that in the near future.
Konrad: Cool. Yeah, what’s next for PIR? What’s on our road?
Shilpa: I think one of the biggest requests from our customers has been to add the feature, make it available on mobile. We’ve had it on desktop for a while now on macOS and Windows. And there’s been a lot of requests to add it on iOS and Android because as expected, not everyone wants to run this on their desktop or has access to one all the time. So yeah, we’re currently working on that. We are excited to have it released on mobile, which should be coming up in the near future. There are challenges associated with making this work on mobile, especially since we don’t just send everything off to a server that will take care of things for us. On mobile, we need to make sure we have the connectivity. We don’t drain the battery. We are using the resources reasonably and efficiently. And we may not get a whole bunch of time from either the iOS or the Android operating system to run things as much as often as we want. On a desktop, we don’t have to worry about Wi-Fi connectivity or battery as much, but on mobile, there’s all of these additional challenges that we have to take care of. The team has been working for the last year almost to figure out ways around this and it’s a very good challenge to be working on.
Konrad: Yeah, the word ‘challenge’ comes up a lot when we discuss PIR. But thankfully, we have some great people on board and the work is moving smoothly forward. So yeah, look out for PIR on mobile. OK, and if anyone after hearing this discussion wants to try it out, even without becoming a subscriber, because this is like a subscriber feature. You mentioned they could do freemium, right? So how one does that? Where do you find it in our?
Shilpa: Yeah, so in the browser, if you’re not a subscriber, you should be able to find it in Settings, or sometimes you might just get a pop-up in a new tab when you open asking you to go try a free scan. So do try that. And if you live in the US, it’s going to be quite shocking. I will warn you of that to see how much of your personal information is out there. So it’ll ask you for a few things about your personal information, three to four fields. If you fill that in, it’ll do a quick search across the number of data brokers we support vary time to time because of the various reasons I said how they’re coming up and down all the time. But we support a maximum of about 60 currently. So we go do a search across all of these brokers. And you’ll be able to see how much of your information and even your family’s information is associated with you. And if that seems like a reasonable thing for you to invest in, then you can go ahead and subscribe in order to go for, allow us to go make the request to submit a request to remove your records from those sites.
Konrad: Perfect. I think we covered most of this stuff. Any kind of a closing thoughts, anything that we should have mentioned?
Shilpa: I think the only thing I would say is, yeah, it is a little bit scary to go see your personal information everywhere, but rest assured that we are constantly scanning and constantly monitoring. And a lot of the users also often of these products often mention how the number of scam calls that they get on their phones also goes down. So it’s really a very worthwhile product to be investing in. Check it out, check out the free scan version and we’d love to hear from you as to how things are working. And we’re also making a lot of improvements to our UX to make it more user-friendly and to make it more clear as to what’s going on behind the scenes and what stage of the various processes that we mentioned. End to end, each of your records is in with each of the brokers. So we’re excited to make all of those user facing improvements in the future.
Konrad: Perfect. Thanks so much, Shilpa. That was great. I really appreciate this discussion. And yeah, that’s all folks. See you in the next episode, I guess.
Shilpa: Thank you.


This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Gabriel (Founder) and Beah (Chief Product Officer) discuss our focus on delight, discovery, and dependability (the 3Ds), what they are, and how they shape our day-to-day work.
If you have feedback on Duck Tales, or episode ideas, email us at podcast@duckduckgo.com.
Show notes: Listen to our episode on Hack Days here.
Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.
Beah: Hello and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In this episode, we are going to be talking about what we internally call the 3Ds: discoverability, dependability, and delight. And if we If you haven’t met me, I’m Beah. I’m on the product team and I’m here today with Gabriel, founder and CEO of DuckDuckGo. Anything anything you want to add to your introduction, Gabriel?
Gabriel: No, I do not. If you’ve watched this at all, you’ve already seen me a bunch. So and they’ve seen you as well, so
Beah: Yes. Cool. Okay. So yeah, let’s just jump in. 3Ds. What what’s the deal? What I mean, I said what each D stood for, but like contextualize this whole thing.
Gabriel: Yeah, so part of the way we run DuckDuckGo as a company, just like internally, like how we operate, is we have a what we call our top priority, which is like the main thing that we think we need to focus on as a company. And that’s changed every few years. But like right now we’re in the middle of and we’ve been doing it for I think around 18 months, something like that, this 3Ds focus on discovery, dependability, and delight as you pointed out. And those are product things. So it’s really a focus on making our product better in those kind of three areas. And just to walk through them briefly, discovery is we actually have a lot of stuff in our product that people do not know about. So there’s the basic thing that a lot of people still don’t even know DuckDuckGo exists and we have a product and we’re a privacy company. But when you get a little beyond that, you know, we’re a search engine, we’re a browser, we have a subscription now. In our browser, you can sync your passwords and bookmarks. There’s all sorts of features, and like the holistic product is a lot bigger than even most of our users realize. And so it’s kind of discovery of those features. Dependability is a little more straightforward. That’s like, okay, you’re a user, the product should be very dependable. But I mean that means like reliable. You can switch to it easily. All the workflows work well. There’s no rough edges in it. Obviously, that’s a tall order when you run a search engine and a browser and a VPN and all these other things we do. But we’ve been nail we’ve been working down the list of all the little issues that people you know complain about, rightly so, that might have been issues with their bugs in our product and we’re trying to fix them. And then the third one is delight, which is, you know. I think most intuitive, I guess, but even harder to actually articulate how we get there.
Beah: Yeah.
Gabriel: But it’s like you have a delightful product experience. It’s those products and experiences that you are unexpected, really bring you joy, like you really want to share with your friends and family. Like we want to bring that delight into using DuckDuckGo.
Beah: Yeah.
Gabriel: So the focus has been on all three of those, because we think we need to do all three to make a great product. And so we have projects and people working on, you know, basically all three at all at all times at the moment under our top priority.
Beah: Yeah, just to add to the discovery point, I mean we get feature requests and feedback. Feature requests for features we have. Like people just straight up asking for features that exist. Yeah. Private browsing. Yeah. Yeah.
Gabriel: Yes. I would argue that might be our top request. Things like, yeah. Yeah. This is a core discovery issue. Yeah, and even more fundamental than that is like a lot of a lot of search engine users, because that was our first product, and we have the most search engine users, don’t even know we have a browser. And our most browser users on mobile don’t realize we have a desktop browser.
Beah: Yeah.
Gabriel: So regardless of features within all those things, like just at the very basic level, people don’t even know we offer a browser. And they should. Yeah.
Beah: Yeah. And like sometimes it’s as simple as like changing like making something more visible and then other times I think it’s about the language we use or like the way we characterize something so that people know that it’s what they’re looking for. So yeah, there’s like that one’s yeah.
Gabriel: Yeah, one more point on that too. Like that one’s subtly hard as well, because like you don’t discovery means that you have to get people to know about it to your point. And you maybe think of something, which is that you have to get in front of people then with a message, but that can be annoying. Like it look can look like an ad, right?
Beah: Mm.
Gabriel: And so that’s a delicate balance. It’s like, yeah, you’re a cer you’re a private search user, you probably want our private browser because it’ll actually make the benefit of privacy much more you know, better in your life if you have them paired together. However, you don’t want a thousand messages of, hey, we have a browser downloaded now. So there’s there’s like this delicate balance of like trying to get awareness to get the people who will who want it who want it to get it without annoying people who don’t want it.
Beah: Mm. Totally. Yeah. And there’s just like it’s the competition of many things. Like you probably want our browser. When in our browser, you probably want your passwords and your bookmarks. And then you probably want those synced across devices. And you probably want this thing that’s opt in turned on. And like, yeah, you can just we have to I think we spend a lot of time trying to make sure that we’re not like bombarding people with too much messaging or promos and it’s but then but then again we get feedback requests for things that are sitting there somewhere behind the scenes. So yeah.
Gabriel: So just before we move on to that one practically, then what that means is we’re constantly running experiments basically of like how best to do these in product kind of promos. And and and I think to your point, like it’s better to do them. The naive approach is to shove them all in onboarding, which people then whiz by. But the better approach is like very contextual, like maybe like when you do a bookmark for the first time, ask people if they want to sync. And then the other part of discovery, just to end that part, is is just our external marketing. So like not in talking a lot of in-product stuff, but we also spend a lot of time just doing general marketing, especially in the US, about educating people about our different experiences.
Beah: Yeah. Yep. So we ended up talking a lot about discovery, but just another like zooming out question. How how did we land on these besides the fact that they all share a D, why did we pick why did we pick these three things?
Gabriel: That’s that’s clearly part of it. Yeah. Yeah. Yeah, I mean if you think about the whole product kind of marketing funnel, which is you need awareness in a product, then you need to know a little bit more about its benefits and value proposition to consider using it. They call that consideration. Then if you kind of are interested in that, like in our case, like privacy benefits, you may trial and then that initial trial point on like is kind of an activation in the funnel. And then you want to retain users and then ultimately a referral. You want users to be happy enough that they’re referring you. So that’s the traditional like marketing funnel. And different companies have issues you know, at different parts of the marketing funnel where they would want to focus. I think it was pretty clear that we had some gaps in in in kind of all parts of the funnel almost equally. And the 3Ds from a product perspective map onto that pretty well. So, like the discovery part is just we keep measuring these surveys, but most people still don’t know about us, and most people don’t know about our different products. So there’s that’s an obvious gap. The dependability one, we’re constantly running user surveys.
Beah: Mm.
Gabriel: And so once people are activated, you know, our retention has gotten a lot better in the last three years because we’ve really improved the dependability of all our products. But we could tell it was a little bit of a leaky bucket, like people had come in and really wanting to love us and try us, but having an issue switching, or you know, we didn’t have, you know, sync or, you know, with their bookmarks. And so we felt that that was also deserved a lot of a focus because we could see that people weren’t always staying when they were activating. And then the third part is word of mouth. So we had grown mainly through word of mouth. The most most of our users have come through word of mouth. But a lot of that was on the back of privacy being one of the biggest stories in tech. And I I hope that’s coming back, and it should, because AI has a lot of privacy issues. But you know, since the pandemic that has been less so. And so to get word of mouth without a big news cycle all the time, you need delight.
Beah: Mm-hmm.
Gabriel: And so it’s really mapping these to a funnel and making it a product focus of that funnel.
Beah: Yeah.
Gabriel: And then something that has either if you’re gonna pick names, you either have to rhyme or they have to have some alliteration. So this is alliteration.
Beah: Yeah. On the delight thing, I also think like for a I think can because we are not the default in many senses. I I think delight pre plays a pretty important role in retention too, because like there’s sort of this FOMO factor to retention where if you run into a little bit of friction on DuckDuckGo, I think you’re more likely to churn than if you run into a little bit of friction on like what people what is actually literally the default or people perceive as the default, you know, and I mean literally the default. Like if you have an Android phone and go, you know, Google provides your default browser, your default search engine, like you people run into friction or, you know, bad dependability like in any product. But I think like their tolerance for it can be lower initially in DuckDuckGo than in other than in Google, basically. So I think delight is an important antidote to that as well. Like you have to have a reason to be willing to endure occasional friction. I mean, we’re gonna keep trying to reduce that friction and make the product better and more dependable, but like it’s just never gonna be perfect. Like we could be, you know, yeah, no no product is ever perfect. There’s always rough edges. So I think delight also just helps on the retention front too.
Gabriel: Yeah, I mean I I think you’re making a good point. I presented it as mapping on different parts of the funnel, but I think you’re absolutely right. Delight really can map onto all parts of the funnel, even the top where you can have delightful marketing and people, you know, we hope that might be wanna be associated with our brand and our product just because not just values driven on privacy, but also we’re presenting that to the world in a way that resonates with them, you know, like they want to be associated with us because they like the way that we make our product and present our marketing and present our, you know, our app store page and stuff like that. I know you thought more about delight from the product side as probably anyone else in the company. Do you have like any deeper thoughts on like from the product side, like how to even think about going to going to incorporate, you know, that kind of experience?
Beah: Yeah. Yeah. I mean, the way I think about delight usually is is basically addressing the emotional needs of people, not just the functional needs. So you can have a feature or an interface that just that like has that works in some basic sense, but it’s not actually pleasurable to use. And like, you know, we’re humans, and emotional beings and pleasure is worth something. So yeah, I mean, I think that can take so many forms, and that’s part of what’s difficult about it. And sometimes the most like sort of the lowest hanging fruit stabs at delight are not in fact delightful at all. Like you can bombard people with confetti, you know. And I don’t think that’s usually delightful. I mean there might be moments where some kind of celebration in the product is is good and appropriate, but like there’s a lot of ways to be annoying too that that if you just try to inject some personality into the product or some delight like you that can take you down some some bad paths. So I think one of the tricky things about delight is that it like requires maybe several ingredients. One is a point of view. Like I think there has to be some consistency. Like it I do think personality matters and brand matters. And if like the product’s just like throwing all these different sort of personalities at you, that like that’s actually like disconcerting, not delightful. To another ingredient that I think is really important, maybe most important, is just like the amount of like rigor and finesse and discretion that has to go into delight features. Like sometimes it means obsessing over a single word, like way more than is obviously worthwhile. Or I don’t know, just like fine tuning the details until it feels just just right. The good news is I think you can usually test for it qualitatively pretty well. And we have a lot of good qualitative testing tools, and we can kind of like you know, put it and and the other thing is once you’ve spent so much rigor internally on something, it it is like awfully hard to look at it the way a new user would. And so I think qualitative testing is is important. But the bottom line of all of that is I think it is actually like often pretty expensive to do well in terms of people’s time and effort. And so it it requires like a real commitment, which is I think, you know, speaks to why it is in this company top priority is it is a statement of commitment to putting in the time to do this, do this well. The other thing I’ll say is that I also just think it’s like more fun to build products that are delightful. Like personally, even though it is so hard and time consuming, it is like one of my favorite ways to spend my time at DuckDuckGo. And I think people who are good at it and like it generally agree. So yeah, maybe that’s kind of a good segue actually to another question, which is how we actually operationalize the 3Ds. Like we say this is the company top priority. Like beyond that, how does it work its way into our day to day or week to week or month to month operating rhythms?
Gabriel: Yeah, no, that that’s a good segue. I mean, I think to some extent every product would like these three things, right? They’re like mapped onto the funnel and they need to be thought of to some extent pretty much by everybody making any product. But to your point about delight, despite wanting it in our product and knowing how important it was, I don’t think we were getting it until we set the top priority. And so one answer to that is and when they get to how to operationalize it, but like just saying we wanted it was not working, you know, like and even with dependability, it’s like, yeah, we have lots of bugs and we would want to fix them all, but we weren’t systematically, I think, addressing dependability in some areas as well. As we needed to, and we were seeing that in different metrics, say like crash reports and stuff like that. So I think operationalizing it could look different in different companies. For us, you know, we often operate as objectives, so we have like maybe 25 different objectives, and so we’re thinking about what are the top discovery dependability delight issues that we have, and then making an objective around that. Which each objective will have its own then set of projects to go solve that objective. And so we’ve been doing that for about eighteen months. And then within each pro
Beah: And you and like we’ve started in that time we’ve seen objectives that have started stating their problem to solve in terms of the the 3Ds, which is cool.
Gabriel: Exactly. And that and and and I think you’re right that it takes that amount of time to like get it in people’s heads to start thinking that way. And then even one below on the project level, you know, we’re tagging projects with the 3Ds. So is this a project about one of them or multiple of them in some cases? And even the criteria, I think to your last point, of individual projects are often related to one of these. So like dependability has KPIs depending on different contexts, but like crash reports is a good example. Like let’s get this crash number down in this way. Delight we’ve to your measurement point earlier, we’ve developed our own internal like KPI survey methodologies to to like actual test delight. And discovery, we can run surveys as well on our users and the general public to understand like is this working? Are we raising awareness of these different features? But essentially we’re operationalizing it by literally tying the company objectives and projects directly to these 3Ds and making the success criteria about.
Beah: Yeah. One other place that it shows up it is we just recently f had Hack Days, which is a convention at DuckDuckGo where everybody there’s a few days in a week that basically people can work on any project that they think would be interesting to work on, dedicate three days to outside of our normal like prioritization rigor. And at the end everybody who wants to demos in a company meeting the the project that they worked on. And we give out some awards and special mentions. And those are grouped in terms of like, you know, best of in those three categories dependability, delight, and discovery. So yeah, it feels like over the last year it’s really kind of woven its way into the the fabric of the company. And I think that’s been I think people really like it too. Like it just, you know. People come to DuckDuckGo from different functional backgrounds and skills and it like gives us some shared language to reason about product decisions and prioritization. I think it’s I think if you surveyed people it it would be a you would find that like it’s been a really rewarding shift for the company.
Gabriel: Yeah, I think people in particular like the permission, like you were saying earlier, for delight, you know, to be able to spend a little extra time and on polish and trying to do something really cool that they’d like to their friends and family to see that they’d be proud of. You know, like I think that’s where you get to when you even go beyond dependability into delight. Also just wanna say that we had an earlier Hack Days episode, so if that is any interest, you can go look at that in the archive.
Beah: We’ll put that in the show notes. I get to say that because I don’t write the show notes. So let’s see.
Gabriel: Yeah, well they’re just a transcript of the whole thing, so effectively effectively you’re right.
Beah: Cool. Okay, well that sounds like a good place to wrap. All right.
Gabriel: Cool. Well thanks for you.
Beah: Thanks, Gabriel. Later.


This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com