# What’s Up With Mojo
This is a transcript of [What's Up With
That](https://www.youtube.com/playlist?list=PL9ioqAuyl6ULIdZQys3fwRxi3G3ns39Hq)
Episode 7, a 2023 video discussion between [Sharon ([email protected])
and Daniel ([email protected])](https://www.youtube.com/watch?v=zOr64ee7FV4).
The transcript was automatically generated by speech-to-text software. It may
contain minor errors.
---
Due to technical issues, timestamps were not available for this episode. The
transcript below uses 00:00 placeholders instead.
---
Mojo is used to communicate between processes. How does that happen? What can
go wrong? Is mojo the same as mojom? Today’s special guest telling us all about
it is Daniel. Daniel is an IPC reviewer and has written much of the guidance
and documentation around it. He’s also worked on cross-process synchronization,
navigation and hardening measures to mitigate security risks.
Notes:
- https://docs.google.com/document/d/15VD6WT-R3MN93gUmPAR_BXee5s0BfYL823Qtj9EHP9A/edit
Links:
- [Mojo - Chrome’s inter-process communication system](https://www.youtube.com/watch?v=o-nR7enXzII)
- [IPC 101](https://www.youtube.com/watch?v=ZdB5P88-w8s)
- [Life of a Navigation](https://www.youtube.com/watch?v=OFIvyc1y1ws)
- [Long IPC review doc](https://docs.google.com/document/d/1Kw4aTuISF7csHnjOpDJGc7JYIjlvOAKRprCTBVWw_E4/edit)
- [Mojo overview](https://chromium.googlesource.com/chromium/src/+/HEAD/mojo/README.md)
- [Intro to Mojo](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/mojo_and_services.md)
- [Mojo Style Guide](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/security/mojo.md)
---
00:00 SHARON: Hello. And welcome to "What's Up with That," the series that
demystifies all things Chrome. I'm your host Sharon. And today, we're talking
about Mojo. How do we communicate between processes? What can go wrong? What is
mojom? Today's special guest to answer all of that and more is Daniel. You know
him from the unparalleled volume of code reviews he does, including IPC Review.
For which, he wrote the documentation and guidelines. And in addition, he has
worked on navigation, cross-process synchronization, and hardening measures to
help mitigate security bugs. So hello, Daniel. Welcome to the program.
00:00 DANIEL: Thank you.
00:00 SHARON: Thank you for being here. First question, what is Mojo?
00:00 DANIEL: Mojo is basically Chrome's IPC system for talking between
processes.
00:00 SHARON: All right, that sounds pretty good. That sounds like what we're
here to talk about. So today, we're going to cover some questions around Mojo.
There are a couple of Chrome University talks and some documentation that are
really good to explain the basics of how Mojo works. So those will be linked
below. Check those out too. Today are questions you might have, if you've
watched those videos, maybe some followup questions that you might have. So you
mentioned IPC. Does that include RPC? Or is it just Inter Process
Communication?
00:00 DANIEL: So personally, I kind of think of them as the same thing. But I
guess RPC is probably more general. Because it could include calls over the
network, right? Mojo doesn't go over the network today.
00:00 SHARON: OK. So it mostly is between the processes we have in Chrome.
00:00 DANIEL: That's correct. Yeah. You also have things like gRPC, right,
Google for making network API calls. But yeah, that's not under the scope of
Mojo.
00:00 SHARON: OK. Cool. Very briefly, we have a thing called Legacy IPC that I
think is a long-term project in the works to get it removed. Anything briefly
there?
00:00 DANIEL: Yeah. Legacy IPC is what we used before Mojo. It was based on a
bunch of clever or horrible hacks, depending how you're looking at it, using C
preprocessor macros. We still have it around because NaCl and PPAPI actually
use a CIPC. So eventually, when we don't have NaCl support, we can get rid of
Legacy IPC altogether hopefully.
00:00 SHARON: Any day now.
00:00 DANIEL: Any day now.
00:00 SHARON: Any day now. OK. So what we'll do now is I think we'll just
rattle through some definitions because we'll come up with a bunch throughout
it. And they're words that probably you've heard before but have maybe a
special meaning in the context of Mojo. So the first of these is Mojo versus
.mojom. I've seen both of them. What is the difference?
00:00 DANIEL: So I think people kind of use them interchangeably in some
contexts. But usually, mojom is specifically the file that defines your
interfaces, structs, and other types that are going over Mojo IPC. Mojo is just
kind of the general name for this system, right? Mojom is specifically a file
that defines these kind of types.
00:00 SHARON: OK. That's cool. Next is pipes.
00:00 DANIEL: OK, yeah, so Mojo, basically, all the higher-level stuff that we
actually use, most of the time, is built on top of this primitive called a
message pipe. So Mojo message pipe always has two ends. It's actually
bidirectional. So basically, the idea is you can create a pipe. And then you
give the endpoints to whoever you want. And those two endpoints can talk to
each other.
00:00 SHARON: And that seems related to the next one, which is capabilities, in
terms of passing things around.
00:00 DANIEL: Yeah. So capabilities is kind of a pretty generic term. In Mojo,
I think we would kind of think of it as using interfaces to grant capabilities
to processes. So for example, if your renderer has permission to, say, use file
system stuff, right, we would give it an interface, like a message pipe with an
interface that's bound to an interface for accessing the file system. Or if it
can record audio for WebRTC, right, we would give it an interface for recording
audio, right? But the idea is we wouldn't just have this giant interface with
all these methods and then have to permission check, at each time like someone
calls a method, that they have permission, right? We would only give you the
interface if you have permission. And if you don't have permission, you don't
have the interface at all. And you can't use the capability.
00:00 SHARON: Can you have multiple capabilities and interfaces per pipe?
00:00 DANIEL: So that probably kind of gets into the associated stuff.
00:00 SHARON: OK. We'll get there. We'll get there. That's coming up. OK. Next
one on our list of words is bindings.
00:00 DANIEL: Yeah, so I think when most people think of Mojo and using Mojo,
the bindings layer is probably what they're thinking of. So this is stuff like
the remotes, receivers, and the glue that actually makes these calls between
processes. There's a lot of Mojo underneath that backing it all. In fact,
rockot actually rewrote the entire backend that Mojo is built on top of
recently to use something called IPCZ for efficiency and other reasons.
00:00 SHARON: OK. He's one of the ones that ones that gave one of those Chrome
University talks, which is very good. So go check that out. Cool. Moving along,
we have remotes, one of the things you just mentioned, I think.
00:00 DANIEL: Yeah. So earlier, I mentioned message pipes. Remotes, and
receivers - they kind of come as a pair - are kind of an abstraction on top of
message pipes to make it a bit easier to use. Because, with message pipes, it's
basically you stuff bytes in one end, and you get bytes out the other end,
right? And no one wants to deal with that. And basically, the idea with remotes
and receivers, remotes are basically a way of making a Mojo call. A receiver is
a way of handling a Mojo call. Yeah.
00:00 SHARON: OK. Neat. And then up next, we have pending.
00:00 DANIEL: OK, yeah. So to take a step back to get the broader picture, when
you use the bindings, you can create a remote. And that always comes with
another endpoint, right? Because a Mojo message pipe has two endpoints. So you
always get a remote and a receiver together. Pending is basically the form of
remotes and receivers that they are in when you can transfer them, right? So
something has to be pending if you want to, say, send it from one thread to
another. Because Mojo message pipe endpoints, they're all thread-bound - I
think sequence-bound, technically. But yeah, so if you want to move things
between threads or between processes, they have to be in pending form. Pending
just kind of means it's not handling - it's not reading things off the message
pipe or trying to send things. You can't use it in that form. You would have to
turn it from a pending into an actual remote or receiver to use it, right? And
we have pending forms of both remotes and receivers for type safety.
00:00 SHARON: Right. Can you briefly explain what sequence-bound means?
00:00 DANIEL: Yeah, so I think a few years ago now, we kind of rewrote the task
scheduling system in Chrome. And the idea was to abstract out some of the ideas
and make things a bit more flexible, right? Because, otherwise, a lot of people
in code was just creating threads, even though it didn't always need like a
dedicated OS thread, right? And so sequences are an abstraction on top of that.
And a sequence just promises that, when you PostTask to it, it runs tasks in
that order. But we could have multiple sequences on the same thread. That's
kind of an implementation detail. That same sequence could potentially even run
on different threads at times, right? So it's an abstraction. But in theory,
people shouldn't have to think about it.
00:00 SHARON: Right.
00:00 DANIEL: Not always true, but usually true.
00:00 SHARON: OK, so it's kind of like - in other places, it would be kind of a
thread. It's the thing you interact with. This is a unit of stuff happening.
00:00 DANIEL: Yeah. It's kind of Chrome's thread basically.
00:00 SHARON: OK. Cool. Another thing you mentioned already, associated.
00:00 DANIEL: Yeah. So the kind of tricky part sometimes with Mojo is message
ordering is only guaranteed on the same message pipe. So if you have a
remote-end receiver and you send stuff, it's a guarantee that the receiver will
get things in the order you sent it in, right? If you call ABC, it will get
ABC. But if you have two remote and receiver endpoints - if I call ABC on one
and then DEF on the other, assuming they both go through the same process,
there's actually no guarantee that ABC will happen before DEF, right? It could
be any kind of interleaving of those kind of things.
00:00 SHARON: Right.
00:00 DANIEL: So associated is basically a way for remotes and receivers to
share an underlying message pipe.
00:00 SHARON: Oh, OK.
00:00 DANIEL: Yeah. It's a bit tricky because the way it actually happens is,
when you create an associated remote and receiver, it kind of gets tied to the
message pipe. It's passed over, right? So when you have a remote, you pass a
pending associated receiver or a pending associated remote over it. It gets
tied to use that same underlying message pipe. It's kind of implicit. It
usually just works. But yeah, sometimes you have to think about the details,
and it gets complicated.
00:00 SHARON: OK, this sounds - this feels a bit like this strong ref counting
of, maybe we don't want to do this ourselves. But we can get into that more
later.
00:00 DANIEL: Yeah. Yeah. Yeah.
00:00 SHARON: OK. And the last thing on the list of definitions is entangled.
00:00 DANIEL: Yeah, so that's I think -
00:00 SHARON: Quantum Mojo.
00:00 DANIEL: Yes. Quantum Mojo. I think that's usually referring to the
receiver-remote pair that Mojo has. It's not a super precise term. And I don't
think we use it widely. But it does show up in a bunch of the comments, I
guess. But yeah, usually, when it means entangled, if you have a remote, the
entangled endpoint is the receiver on the other side or vice versa. If you have
the receiver, then it's the remote on the other end.
00:00 SHARON: Right. Yeah. OK. Probably all the other words that mean a similar
thing have been heavily overloaded already, like connected.
00:00 DANIEL: Yeah. Yeah. It's a bit hard to write comments for Mojo. We know
it could use improvements. But yeah, trying to find ways to write this sort of
information precisely without like writing novels is always a bit tricky.
00:00 SHARON: It is tough. OK. So let's briefly talk about how Mojo is used. So
I think the most typical case - the canonical case, I feel like, is between the
browser and the renderer.
00:00 DANIEL: Yeah.
00:00 SHARON: Right? Is that the case?
00:00 DANIEL: Yeah, I think that's fair to say that maybe that's where most of
the IPC in Chrome happens because Chrome is a web browser.
00:00 SHARON: Right. And I've heard it described as letting web pages get
things that they want from the browser. So Mojo is used in that process. Like a
web page wants maybe - I don't know - a file or something. And it uses Mojo to
get that. So apart from - what are all the kinds of things a web page might
want from the browser or want it to do that it would use Mojo for?
00:00 DANIEL: Yeah, so I think that's a pretty big question. So there's kind of
a set of core capabilities like a web page always has, right? So for example,
it can always navigate somewhere, kind of various things to manage the loading
state or to load some resources and that sort of stuff, right? So every web
page will probably have all URL-loader factories or the frame interface for
managing this sort of thing, right? And then there are additional capabilities
that aren't necessarily exposed to everything, right? Obviously, on the web,
you have all sorts of things gated by permissions, like file system access,
clipboard, audio recording, video recording, and that sort of thing, right? And
that's the thing where the renderer could go to the browser and be like, hey,
give me an interface for geolocation or something, right? And assuming it
passes the permission checks and other checks, we would give it back the
geolocation interface, right? We would grant it the capability by passing it
that interface.
00:00 SHARON: OK.
00:00 DANIEL: Yeah. That's the general sort of idea. It gets - as always, it
gets a bit messy, right? Because there are edge cases where things have to work
slightly differently. But in general, that's kind of the flow we try to follow.
00:00 SHARON: So basically, it sounds like the renderer wants something that is
kind of OS-level, right, like camera or audio. And because we don't trust
renderers, we have to do that through the browser. So this is how it gets to
the browser. And then, through whatever other magic happens -
00:00 DANIEL: Right. So yeah, there's some central places where we register
what interfaces are even exposed to a process, right? But that registration is
usually also - has other logic, like, should we even grant this thing, right?
Does the origin - does the document requesting this have a secure origin? Did
the user give it permissions potentially? It all kind of depends. There's a
wide gamut of things you might want to check. But yeah, that's the general
idea, this central point to kind of broker these sort of capabilities out.
00:00 SHARON: OK. Cool. So within the browser still, are there - what are other
examples of not browser-to-renderer or back uses of Mojo? Are there
render-to-render?
00:00 DANIEL: Yeah. So like any other kind of thing that evolves over time,
Chrome has gotten quite complicated. So there's, I think, a bunch of our things
actually running utility processes now. Like I think - but don't quote me on
this - like a lot of devices' code like can do this. And so what actually
happens is the renderer will talk to the browser, right? And the browser will
be like, you can use it, right? And it will actually maybe spin up the utility
even for the renderer and give it access. It can pass the message-type
endpoints. It can pass a remote back to the renderer and the receiver off to
the utility process. And then the renderer can talk to the utility directly.
And that actually kind of comes in for the other question about
renderer-to-renderer communication. We have these things called service
workers, which can do interesting things with page loads, like support offline
apps and that sort of thing. And the way that works is you can't necessarily,
from the renderer, go directly to another renderer. But the renderer, if we
know it's controlled by a service worker in that document, we can give it a
URL-loader factory that will actually go and talk to the service worker. In
that sense, there is renderer-to-renderer communication happening, but it's
brokered. It's not just a free for all.
00:00 SHARON: Why don't we want free for all, direct renderer-to-renderer
communication?
00:00 DANIEL: Well, it would probably complicate the kind of trying to - so the
thing with Mojo is it's very flexible. It's very easy to be - let any two
endpoints in Chrome talk to each other. But with that flexibility is also a
certain amount of danger, basically. We want to be able to - when things are
exposed to another process, we want to be able to audit them, from a security
perspective and just from a stability perspective as well. If we just kind of
made it a free-for-all, it would probably become pretty hard to figure out what
can talk to what? How is the permission checked? Where is it checked? So by
kind of centralizing these checks in the browser interface broker, for example,
the idea is we make it a bit easier to understand how the system - like, what
it's exposing, and what the attack surface is, and that sort of thing.
00:00 SHARON: Yeah. There's a lot of stuff that's very combinatorial explosion
to me, and this seems like it's trying to limit that a little bit.
00:00 DANIEL: Yeah. There's always going to be things that we can't catch,
obviously. But that is kind of the general idea. By kind of limiting it through
a central kind of broker area, we can figure out, if someone wants to audit it,
they can be like, OK, we are exposing these things to the renderer process. Oh,
no, we're exposing WebUI. Is that checked? It is, so we're OK. But that sort of
thing, yeah.
00:00 SHARON: OK. Can you explain a bit more about what service workers are?
For those of us who might not be familiar, it sounds like they're kind of
between a browser and a renderer process, maybe.
00:00 DANIEL: So I'm actually not the best person to talk about service
workers. But at a very high level, they're workers that aren't confined to the
lifetime of a page, of a document necessarily. And that's why they can
intercept network loads. They can also do some storage stuff. And I think some
notifications are tied to service workers and other capabilities. I'm not super
familiar with them. I just know how they work at a high level and that they can
be used to implement offline support for apps, as one example. But all sorts of
other things you could think.
00:00 SHARON: All right. That makes sense. Cool. So those are, within Chrome
browser, uses of Mojo. So let's talk about some adjacent Mojo use cases. So
before I used to work on Fuchsia, and they have something called FIDL. It
stands for Fuchsia Interface Definition Language. And to anyone who might have
seen it, it looks a lot like Mojo. So can you tell us a bit about that and how
that works?
00:00 DANIEL: So I wasn't actually super involved with Mojo at that point. But
my understanding is FIDL was basically forked from an earlier version of Mojo,
and then they evolved it in their own direction. And FIDL has kind a lot of
interesting things about it. And if we had infinite time in Chrome, it would be
nice to integrate some of those features back. But my understanding is FIDL is
very specific to Fuchsia. But they also have kind of this similar idea to
Chrome where I think you only expose a FIDL interface - if you give someone a
FIDL interface, you're granting them the capability to do that thing. So in
that sense, it's quite similar to Mojo. But yeah, because of the shared
heritage, I expect it probably looks pretty similar, but there are definitely
some differences.
00:00 SHARON: Yeah. Something I heard a lot was that Fuchsia was a
capabilities-based operating system. And it wasn't until I started seeing more
Mojo stuff that I was like, Oh, that's what that means!
00:00 DANIEL: Yeah, yeah, yeah.
00:00 SHARON: That's the same capabilities. And it looks a lot like Mojo. And I
think, from the case of using it, I think the only thing you might notice is
that they have more bindings in different languages. So in Chrome, it's mostly
C++. Are there any non-C++ Mojo usages, really?
00:00 DANIEL: There are, actually. So there's Java. That was one of the
motivations for doing this is to make it a bit easier to implement an endpoint
in Java. Because before people had to write a bunch of JNI boilerplate to jump
from the C++ IPC handling over to Javaland. Mojo kind of abstracts that away at
some cost. There's been some persistent concerns about binary size from the
Java bindings from the Android team. And they could probably be improved.
There's also the JavaScript and TypeScript bindings. I believe Chrome mostly
uses the TypeScript bindings these days for things like WebUI. I know some WPTs
also use the JavaScript endpoints for injecting test fakes or mocks and that
sort of thing.
00:00 SHARON: Oh, cool! I didn't know about that. Cool. So that's that. And
then another kind of OSey thing is LaCrOS. I'm not super familiar with this,
but I understand that Mojo is used in an interesting way in LaCrOS. So can you
tell us about that?
00:00 DANIEL: So LaCrOS is basically an effort to make it easier to update
Chrome on ChromeOS devices. Before, it was kind of this monolithic thing
because Chrome was also responsible for the Window environment Ash on ChromeOS.
And so it was sometimes a bit difficult to uprev Chrome if there is a critical
security fix or whatever. And LaCrOS is an effort to kind of decouple these. So
basically, it turns Chrome OS into more of an OS kind of environment. And
what's left on the LaCrOS Chrome - it's what it's called - is really just
browser related. So it's still kind of a work in progress. But in the future,
Ash the Chrome - right now we have Ash Chrome, which can show WebUI still. But
in the future, that would actually - WebUI would be displayed in LaCrOS Chrome.
And it would just be like an Ash backend without any blink renderer and that
sort of thing. And there's a bunch of Mojo to basically communicate between Ash
Chrome and LaCrOS Chrome. There's some constraints there. It uses versioned
interfaces, which is something you won't find too much of elsewhere in Chrome,
other than some ARC stuff.
00:00 SHARON: What are these interfaces?
00:00 DANIEL: So versioned just means that these interfaces have backwards
compatibility constraints because Ash Chrome and LaCrOS Chrome don't
necessarily ship together. We want to be able to update LaCrOS Chrome.
00:00 SHARON: That's the point.
00:00 DANIEL: Yeah, exactly. So we have to be able to tolerate some amount of
skew between the interfaces. But we have to do it in a way that's backwards
compatible. And so versioned interfaces are a way to more or less guarantee
that, assuming you follow the rules. And we have some checks to make sure you
don't break the rules, generally speaking. But yeah, there's some complexity
because of that. If you want to deprecate methods or remove fields, you can
deprecate methods and remove them eventually, but fields are a bit trickier,
and that sort of thing.
00:00 SHARON: It's like the whole Proto thing of you want them to optional
because they're never going away, or something.
00:00 DANIEL: Yeah. So Proto has an advantage over Mojo in this respect,
because they identify their fields with tag numbers. And so you can just omit
fields completely. Whereas, Mojo, we actually reserve space in the struct for
it. And that means, once you have a field there in a versioned interface, you
can never really get rid of it. You have to keep it there even if you're not
using it. In the future, maybe you might use it for something else if it's no
longer needed. But yeah, it becomes a bit tricky because of that sort of thing.
00:00 SHARON: Yeah. Because I guess with regular Mojo, it's meant to just work
within one monolith of the browser. So that, at least, has all the same
version, and is not - the version skew is not something that was initially
planned for.
00:00 DANIEL: Right. It all ships as kind of one monolithic block. You can kind
of refactor freely across the system. When you have versioned interfaces, it
becomes trickier. You have to follow a deprecation process. I think LaCrOS, at
one point, was kind of like a three-milestone, three-version thing before you
could remove old APIs. But don't quote me on that.
00:00 SHARON: Right. OK, interesting. Changing gears a bit here, so let's go
back to talking about receivers and remotes and the different states they can
be in. So some - these are all kind of words I've seen. I'm not that familiar
with Mojo. I haven't done too much cross-process stuff. But you see words like,
bound, connected, disconnected. I've seen all these words before. I know what
they mean, but I don't think I know what they mean in this context. So can you
explain?
00:00 DANIEL: Yeah. So I think maybe the simplest way to think of it is bound
is when a remote or receiver isn't null. Why would it be null? If you just
default construct a Mojo remote that's not bound to - you just default
construct on, it won't be bound to anything. It'll be null internally. If you
try to make a method call on it, it will crash. You actually have to create
that Mojo message pipe that's backing it to, quote, unquote, "bind" it. So when
you create that underlying Mojo message pipe, that's what it means to go from
unbound to bound. And this is kind of a bit tricky sometimes. I notice this
kind of mistake pretty often. Sometimes it's very easy to call
BindNewPipeAndPass, like, pending - I don't even know what the function is
called. We gave it a really long name to try to be descriptive, and now no one
can ever remember what the actual invocation is. But when you call that thing,
the remote or receiver that you're calling it on becomes bound synchronously at
that point. Even though there's no other side attached to the entangled
endpoint, it's still considered bound because it's no longer null. You could
create a Mojo remote. You could bind it. You could immediately start making
method calls on it, even though the other end hasn't been passed anywhere. And
what will happen is all that stuff would just be queued internally. And so when
it becomes connected is when the other endpoint basically goes from pending
to - actually, no, that's not true. Sorry. It's actually considered connected,
too.
00:00 SHARON: OK.
00:00 DANIEL: Yeah. When you bind it, it's considered both bound and connected.
00:00 SHARON: OK.
00:00 DANIEL: Yeah. The disconnection, if there is one, is always kind of
asynchronous. Internally, there's some control IPCs that do heartbeats and sort
of stuff to see what's alive and that sort of thing. I don't know those
details. You would have to ask rockot, who is probably the only person who
knows those details at this point.
00:00 SHARON: Oh, no!
00:00 DANIEL: So yes, let us all hope for rockot's continual safety. But yeah,
when you create a remote or receiver and you bind it, it's both bound and
connected. If you have a remote, you can start making method calls on it
immediately. You don't have to wait for the other side to turn from pending to
a receiver, for example. Everything would just get queued. And disconnected is
just when either endpoint is dropped. So if you drop the remote, the receiver
will become disconnected, if you destroy the remote. Or if you destroy the
receiver, the remote will become disconnected. But that's an asynchronous
process because it's always asynchronous, even if you're in process. But it
just happens at some point. And the tricky part here is if you have a bound
thing, it can be disconnected. You can still make method calls on it. And
that's OK. But your method calls will just disappear into thin air. Whether or
not that's desirable kind of depends on what you're doing.
00:00 SHARON: So going back to what you just said, can you have a case where
you have one of the ends of a pipe disconnect, and then reconnect it? Or is the
only way to disconnect one of the ends after you have connected it is to
destroy the object that represents one of those ends?
00:00 DANIEL: So disconnection is a permanent thing. You can't reconnect
something that was disconnected. There's some Mojo underlying system - I don't
know I would call it - but like low level Mojo APIs that you can use to fuse
message pipes together. But even those won't turn a disconnected message pipe
back into a connected one. The idea with the kind of endpoints is, once they're
entangled, they're always kind of that pair. So if either endpoint gets
destroyed, it becomes disconnected. And this could also happen if the other
process crashes. Your endpoint that's remaining alive, whether that's a remote
or receiver, will become disconnected at some point, but no guarantee when
exactly. There's no ordering guarantees there.
00:00 SHARON: OK. So whenever ordering and stuff comes up, like a concern - a
common concern is like deadlocks or all sorts of synchronizing issues. So what
are some of the concerns? Are deadlocks a common concern? How do we handle
this? Because this seems very fraught with all of the typical, distributed,
async problems that exist.
00:00 DANIEL: So if you're not using synchronous IPCs, you probably won't hit
deadlocks unless you're actually writing code that is blocking on receiving a
remote IPC. In general, I haven't seen code written like this in Chrome because
I think most developers are like, well, I probably shouldn't block waiting for
that reply because that's not a great thing. Obviously, you'll see this sort of
thing in tests because it's much more convenient in tests. But in actual
production code, I don't think this is a thing that happens. Where this could
run into problems more is with sync IPCs. So by default, Mojo methods are all
async. You have to actually give it a sync attribute if you want to be able to
make an async call in it. And what that means is, if you use the synchronous
version of the method, it will actually just wait until it gets - until the
remote process, or whatever, the other end calls the reply callback to let you
know that it's done. And there's a lot of trickiness involved there because,
when you're just waiting for the remote thing to reply, there were concerns
because - before Mojo IPC, with legacy IPC, you could also have sync calls. But
the way we tried to ensure safety was to make sure that the sync IPCs only ever
went in one direction. So they only go renderer to browser, and not browser to
renderer as well.
00:00 SHARON: Because we don't want to block the browser ever.
00:00 DANIEL: I mean, we don't want to block the browser. But we also don't
want to end up with sync call cycles where the browser process is waiting for a
sync reply from the renderer, and the renderer is waiting for a sync reply from
the browser. That would be bad.
00:00 SHARON: That would be bad.
00:00 DANIEL: Mojo tries to avoid this problem by saying, if I'm waiting for a
reply to my message, to that sync call I made, and someone else makes a sync
call to me, I better let that through and handle it and let them know just to
avoid deadlocks. But this is also problematic in another way, because it means
the messages you're getting sent may be reordered, basically. So what this
means is, say, I make a sync call from the renderer to the browser. The browser
sends us some async IPCs, like A and B. And we see those. And we're like, OK,
we're in the middle of a sync call. We're not going to handle them right now.
And then, for some reason, someone added a sync call from the browser to the
renderer. And so the browser goes to the renderer. And the renderer is like,
hey, I better handle that sync - that incoming sync IPC. And it handles C. But
at this point, you haven't handled A or B yet. And if you were kind of assuming
that A and B would happen before C, that's no longer the case. It's pretty
messy, which is why we've actually considered switching the behavior of sync
IPCs to no interrupt by default rather than allowing sync interrupts,
basically, is how it currently works. We actually had some security bugs kind
of around this sort of message reordering thing. Really, the whole takeaway
from this is don't use sync IPCs if you can avoid it in any way. They do add a
lot of complexity, just for the considerations. Obviously, they aren't great
performance-wise because they are blocking - if you don't need it, please,
please, don't use them.
00:00 SHARON: Is that the main takeaway of today is don't use sync IPCs, if at
all possible.
00:00 DANIEL: I mean, that is definitely one thing I would like people to
remember just because, yeah, if you can avoid it, it will make things - it will
make life much easier down the road, most likely.
00:00 SHARON: So to make your life and Daniel's life easier down the road, try
to minimize use of sync IPCs. So of course, what are some cases where they are
used now and cases where they are currently used, and we would hope to
transition away from them also.
00:00 DANIEL: Hmm. That's a hard question, mostly because I don't have Code
Search pulled up right now.
00:00 SHARON: Right, fair enough.
00:00 DANIEL: I know there's some sync stuff around GPU and render stuff. A lot
of the older web APIs weren't written with promises in mind. So for example, I
think document.cookie involves a sync IPC to go get whatever the latest cookie
is from the cookie jar. We've added some caching there to make it better, but
fundamentally, those sorts of things need to happen synchronously. So we don't
have much of a choice. Interestingly enough, I think Android WebView actually
has some sync IPCs from the browser to the GPU, I want to say. Don't quote me
on that. I don't understand that code at all, despite having reviewed a lot of
those CLs. But I'm given to understand that it's necessary. So yeah, I mean, I
don't know that we're actively migrating anything away from sync IPC at this
point. I know people have worked on optimizing cookie access. And so we will
reduce the amount of sync IPCs, but never completely eliminate, I think.
Luckily, I think a lot of the new web APIs are using promises, so they can be
async. They don't need to be synced. And end life is great.
00:00 SHARON: OK. That's good.
00:00 DANIEL: Yeah. There is also some, I think, additional kind of Google
integrations with Chrome. I think previously they were pretty complex because
it was just trying to translate a Java code base into C++. There was a bunch of
assumptions around sync calls. So they wrote sync IPCs kind of to wrap all that
in their helper utility process. And that definitely led to some problems with
deadlocks because we would make a Mojo sync IPC. And then to simulate the
environment Java would have had, it would have - it spun a run loop internally.
But it got into deadlocks. So don't write sync IPCs. Do yourself a favor.
00:00 SHARON: Do yourself a favor. That's right. So when it comes to all of
this async/sync, mostly the async stuff - and you mentioned binding earlier.
Something we see a lot in Chrome is callbacks. So these are used for async
stuff. And you also see them bound. Is that the same binding as Mojo binding or
is that - no.
00:00 DANIEL: No, it's completely different.
00:00 SHARON: It's completely different. Is there much intersection between
callbacks and Mojo? These are both heavily used in async situations. Do they
intersect?
00:00 DANIEL: Yeah. So it's actually kind of a known - I guess I would call it
a wart at this point that our way of writing async code leads to kind of
hard-to-follow code. If you want to make a Mojo message call and do something
after it replies, you bind a reply callback. And that's kind of the case of how
async code in Chrome often works. You create callbacks, and then you wait for
this other thing to be done, and call your async callback. But it kind of means
that trying to read the control flow of the program can be pretty tricky
sometimes. You have to be like, oh, this thing has an async callback. Let me
see what it's bound to. So you go in Code Search. You look at the caller.
You're like, oh, it bounded to this onFooDone thing. Let me go look it
onFooDone. And then if onFooDone has more async work, you're just kind of
chasing these chains all over the place. And that's kind of the case with Mojo.
I think Mojo used callback just because that's kind of our language for it in
Chrome. It would be nice to do better. There was a bunch of exploration around
some sort of promise-based idea a while back. Ultimately, we didn't implement
that because it was felt it would be hard to migrate everything. And it was
kind hard to justify prioritizing that. But we've played with a lot of other
ideas since then to try to make these sorts of things a bit easier to write. If
you're chaining two callbacks, you can use a callback helper called then.
There's also something called a sequence bound which can help you if you have
two objects that live on different sequences. You don't have to post task
yourself. Sequence bound can happen - handles that under the hood for you and
binds the callbacks and whatever.
00:00 SHARON: Right, right. Yeah, we're still migrating off of legacy IPC. So
to introduce another migration at this point seems ambitious.
00:00 DANIEL: There's kind of varying opinions on this, obviously.
00:00 SHARON: Well, they're not here right now. So what are your opinions, if
you want to share them.
00:00 DANIEL: I mean, it would be really nice if we could improve on this. I
know that now that we're slowly getting C++20, thanks to Peter Kasting's work.
I think there will probably be some exploration around co-routines and if
that's something that we could use to help us migrate to simpler patterns for
async code. It is kind of a very open-ended question now because there's also
things like Rust that are up and coming, and figuring how to do async Rust and
async in Chrome, in C++, and making that all mesh together is probably going to
be a pretty complex problem.
00:00 SHARON: Probably.
00:00 DANIEL: Yeah.
00:00 SHARON: Probably.
00:00 DANIEL: Yeah.
00:00 SHARON: So kind of transitioning a bit to more security things, and also
as it ties into callbacks and async, is when you bind a thing - because memory
safety and use-after-free and whatnot are a major problem that we have from a
security perspective, especially because C++ and all of that. So when it comes
to passing around these things that are async, you don't know when they'll be
done, if you're passing in things that you're calling from - like in the
callbacks, how do you make sure that they're still around when you need them
and that call doesn't become either a crash, like null dereference, or worse, a
use-after-free? Is this a big concern we have? How are we dealing with it?
00:00 DANIEL: Yeah. So if you're using Mojo, quote, unquote, "the normal way",
you're probably safe-ish. So when I mean the normal way is, you have a class.
It needs to make Mojo calls. And it owns the Mojo remote. And the way that
works is if you make calls on the remote, but then your class is destroyed, it
will kind of cancel any reply callbacks. You will never get them. So you don't
have to worry about that case. And that's kind of nice. But there's, obviously
a lot of other ways for things to go wrong. In particular, if the lifetime of
the class is tied to the lifetime of the Mojo message pipe, like, if it gets
disconnected, you destroy this. That's kind of an area that's a bit fraught
with peril. We've had this problem with self-owned receivers. A self-owned
receiver is basically a shorthand way of creating an implementation for
handling Mojo messages that deletes itself as soon as the message pipe is
disconnected. And at first glance, this kind of seems a very natural pattern.
If I'm disconnected, I don't need to be there. Just delete this. But it becomes
problematic if other people are holding pointers to you. We had this problem, I
think, a lot with - so a common kind of scope - for IPCs between browser and
renderer, a common kind of anchoring point is the RenderFrame(Host) or
RenderFrame rate. And what would happen is we -
00:00 SHARON: What is a RenderFrame or RenderFrame(Host)?
00:00 DANIEL: Yeah. So it kind of corresponds to, basically, either the main
frame or an iframe. And it's just kind of responsible for dealing with all the
fun logic of navigating, loading the page, and if the page wants to do other
stuff, figuring out how to get it to the code that actually knows how to do the
extra stuff, like the capabilities thing. So a common problem we had was the
RenderFrame host could be destroyed, like if you remove an iframe from the
document. The RenderFrame(Host) could be destroyed. But what would happen is
people would grant capabilities using interfaces, but these interfaces would be
self-owned receivers. And what would happen is the self-owned receiver would
have a raw pointer to the RenderFrame(Host), but it wouldn't destroyed with the
RenderFrame(Host) because it's a self-owned receiver. And the thing controlling
its lifetime is whoever holds the other endpoint. In this case, that's a
renderer that might be malicious or compromised. And so without any way to
guarantee that the RenderFrame(Host) will outlive the self-owned receiver, it
becomes dangerous. We had a lot of use-after-free bugs from this, actually. And
that's why we added something called Document Service. And if you're writing
web APIs and you need to implement IPCs, and your thing is kind of roughly
scoped to the lifetime of the document, it's highly encouraged to use something
like Document Service rather than a self-owned receiver. That way you don't
need to hold a raw pointer to RenderFrame(Host) yourself. We guarantee the
lifetimes are more or less correct. Obviously, kind of with anything of this
nature, if other people hold pointers to you, you still need to be sure that
you're clearing them, or your ref counted or something. It's hard to give a
one-size-fits-all fix for this sort of thing. Document Service is kind of the
closest we have. There's a couple other helpers along those lines. And if your
code can fit within that framework, it will probably make your code a bit more
robust against those kind of problems.
00:00 SHARON: It sounds like, yeah, avoiding ref counting, or strong ref
counting, we want to generally do that because that's easy to get wrong. And
probably just general good advice or good practices to not use a `T*` to use a
global pointer.
00:00 DANIEL: Well -
00:00 SHARON: `raw_ptr` instead.
00:00 DANIEL: Ref counting has its place. But it's a bit tricky to use
correctly. And in Chrome, we've traditionally tried to discourage it if it's
not needed. And then, also, with the `T*` thing, with the MiraclePtr and
BackupRefPtr work, I think we've actually turned on some enforcement that you
can't actually have `T*` fields anymore.
00:00 SHARON: Oh, cool.
00:00 DANIEL: So that's an additional layer of safety, which is nice.
00:00 SHARON: Things that have changed since the first episode. Wow!
00:00 DANIEL: Yes. It's great. You can use `raw_ptr` or `raw_ref`. And you
should be doing that where possible, just because that way, if you mess up, or
you forget about an edge case, it turns into, hopefully, a mostly
nonexploitable kind of stability bug, rather than an, oh my gosh. It's a
critical-severity security bug. We must ship a fix out ASAP.
00:00 SHARON: So that's how lifetimes can cause problems. So in the case of
this - so it sounds like the bad thing that will happen in this case is a
general memory safety, use-after-free problem. So there's nothing necessarily
Mojo-specific about what can go wrong in this case where the problems are being
sync and async.
00:00 DANIEL: So yeah, it's not so much about async and sync but just
remembering that the thing - like if you're implementing an interface, the
other thing calling into you, whether it's a remote process or not, may be
malicious, especially if it's from the renderer. We have to assume that the
renderer is compromised. And that means it's better to try to structure things
in a way that either Mojo will enforce invariants, or that impossible things
can't happen. So one common area where we have these sort of issues is maybe
something will pass like two arrays of stuff. And I don't know - say instead of
passing a bunch of pixels, it passes all the reds in one array, all the greens
in one array, and all the blues in one array. And then it just assumes those
are the same length. That's not a safe assumption if it's coming from the
renderer, so you would have to check that. But it would be better to structure
a code in ways that didn't require checking all these assumptions. So in this
contrived case, it would be better to have a pixel type, and then have an array
of pixels, because then you have to specify RGB. And it's guaranteed that you
won't have an array mismatch because you won't be passing multiples of them. So
just stuff like that. It's really hard to go over all the ways things can go
wrong. We did try to do that. And I think the document is 20-plus pages. It's a
doc of guidelines for IPCs, like what reviewers and reviewees could, in theory,
look for. But it is massive. It'd be nice if it could be more compact, but I
think that's kind of the nature of people can write whatever they want. And
there are all sorts of creative ways to get into trouble with these sort of
things.
00:00 SHARON: Yeah. As an IPC reviewer, when you look when someone is making a
change, adding, removing - maybe not removing, but adding things, what are the
first things you check for when you are reviewing a new or updated IPC?
00:00 DANIEL: So the first things I will look at are the CL description and the
comments in the module. And if I can't really figure out what the change is
about from there, if I have extra time on my hands, I will go look at the bug.
I will go read any design docs that were linked and try to kind of reverse
engineer. But in general, that is the first thing I look for because I want to
understand what they want to do at a high level. There's no point in trying to
nitpick like things here and there in the implementation details if the
operation that's being exposed is fundamentally unsafe. If someone's writing a
file system interface, and it provides the capability to read any file, and
they want to pass that to the renderer, that is fundamentally unsafe. And
there's no point in reviewing the implementation. So you want to review the
overall high-level ideas, and make sure you understand those. That's what I
personally go for because sometimes I think it's very easy, if you're writing a
CL, to be, like, I know the context behind it. I'm fixing X bug or fixing Y
bug. But it's easy to forget that someone else coming in reading it - the IPC
reviewer is not going to know every feature like the back of their hands. And
so giving them the context to be, like, oh, this is a fix for Y, and we need it
because Z, really helps the review. And also having these comments in the
mojom, can help document constraints, or what is this going to be used for, or
how will it be used, what is it expected to do, if you implement it? If you
call it with - if something is nullable, you can pass nothing for it. What does
that mean? Is that just a I didn't feel like figuring out the test, kind of
thing, or it actually has some significance? Like documenting those sort of
things.
00:00 SHARON: Who would do something like that and not have figured out the
tests first?
00:00 DANIEL: I have never done anything like that.
00:00 SHARON: Yeah.
00:00 DANIEL: Yeah. But once those kind of high-level things are more out of
the way, then it's easier to review the rest of the CL in the context of that.
But without that background context, it can be quite tricky to do IPC reviews
sometimes. And the other thing I would say is I would encourage people to send
out reviews to IPC Reviewer Center. I kind of understand that people don't want
the spam, like the people that are asking to review. But people, if they don't
feel like they don't need to review it, they can ignore the CL until it is
ready to review. But sometimes it's useful to peek in and glance and be like,
yeah, this is about the right shape. I have no concerns that require immediate
action. Because what's really unfortunate is if you're at the end of - I don't
know - a three-week review, and you're like, oh, you shouldn't do it this way.
You actually need to re-engineer this entire thing and hook it up this other
different way over here. That's just not fun for anyone. It's not fun for the
reviewer to give that kind of feedback. And it's not fun to get that kind of
feedback either.
00:00 SHARON: Yeah. I'm sure we've all been on at least one end of this kind of
interaction before, so for sure. So would you say IPC review is basically a
security review for IPC? Or are you reviewing for additional stuff beyond that?
00:00 DANIEL: That's the minimal scope. Some people, depending on how they're
familiar with the area, may have ideas beyond that. But the kind of expected
scope - it's expected the cover is, basically, does this IPC make sense to add?
Is it safe? What are some additional things we need to consider if the sender
or the receiver is malicious? And this extra layer of scrutiny is just because,
historically, before we had IPC review, we actually had a lot of security bugs
due to - it's really easy to write this code because day to day, you're like,
oh, I'm just working the same process. Everything is fine. I can assume that
people won't violate my invariants. If I say this thing must always be called
with at least one item in the array, I can assume there will always be one item
in the array. But that all goes out the window if you have to assume a
malicious attacker in the renderer. And so the IPC reviewer is usually just
coming in more with a hostile mindset, like ways things could go wrong,
basically. In that sense, very much a security review. But to be clear, it's
very different from the security review for launches. That's an entirely
different thing. Sometimes there might be times when an IPC review is like, I
don't know. This seems a bit potentially dangerous. Has this gone through any
sort of launch review yet? And at that point, you might punt it to a security
review. It's not super common, though.
00:00 SHARON: OK.
00:00 DANIEL: Yeah.
00:00 SHARON: OK. Yeah. Lots of reviews of all kinds. And I think what you said
about the reviewer not having all the context applies to lots of reviews. In a
launch review, you have so many fields you need to get approved. All of these
people don't have the same context as you. And the same is true for IPC
reviews. So are there any cases where something about the actual design of the
Mojo interface itself went wrong that caused a problem that you can tell us
about?
00:00 DANIEL: I don't think I have a prepared example.
00:00 SHARON: That's fine. It's cool.
00:00 DANIEL: We can edit one in in post-production.
00:00 SHARON: We can edit one in in post-production. So you're going to sort
out an example very shortly.
00:00 DANIEL: Sure. Let's go with that.
00:00 SHARON: Yeah, let's go with that. And then moving - so best practices,
any - when it comes to introducing new IPCs? So you mentioned getting review
early, just a quick kind of sanity-check situation. Do you have any other tips
for best reviews for best practices for IPC reviews?
00:00 DANIEL: Well, you could go read the 20-plus page IPC guidelines doc and
try to memorize it. I don't recommend that, though. I would say, in general, it
probably comes down just to several things. It's better not to have stateful
interfaces. And so what I mean by that is an interface where it's like, hey,
you must call the init method before you do anything else, or else it will
explode. We don't want that because that means all your other methods have to
check that init has been called. And otherwise, they'll explode. Depending on
who your caller is, they may or may not be trustworthy, and that sort of thing.
They kind of - sorry.
00:00 SHARON: Do we want a lot of Mojo calls to generally be idempotent, too?
00:00 DANIEL: They don't need to be idempotent, necessarily. But when it's a
very complex set of state transitions, that is where things can get into
trouble. And obviously, there are some situations where this is unavoidable.
And you'll just have to deal with it. But if you can avoid it, like if you have
an init method, it might be worth it to create a factory interface. This is
what I usually recommend. Obviously, it's a bit more boilerplate, and it's not
the nicest always. But it can also save some headache down the road. We
definitely had some IPCs in the past where this was a problem, just because
malicious code could not call the init method. Or it could call it twice and
cause a use-after-free. So if you can factor these out into separate
interfaces, that can be a very helpful thing. And the other thing is - and I
mean, it really goes along with the first - try to structure things in a way
that a malicious - if the other end, if they're malicious, they can't violate
the invariants. So the contrived pixel example, but also using things like
struct traits, rather than having each thing be like, hey, let me validate all
the data, or call a function to validate all the data, try to write struct
traits if you have this sort of validation logic. And so that validation kind
of happens centrally in one place. And everyone using the type, does it need to
go, I don't know - data is valid, or something. Because if someone forgets,
then, boom, potential security bug. So yeah, that sort of thing. It's very
general. But if we wanted to get into specifics, we would be here for a couple
of days.
00:00 SHARON: OK, OK, a couple of days, all right. I think we might have lost
people after at least the second day. I think we might.
00:00 DANIEL: Yeah.
00:00 SHARON: Yeah. And then moving on from that now, mostly a personal
question, sometimes you have a function. It's a Mojo call. You click it, and
there are no callers, like in Code Search, I mean. So why are there no callers?
Why are they not shown? Does it mean I can just delete this interface? OpenURL,
who needs that?
00:00 DANIEL: OK. Yeah. So if you want to find out what's calling a Mojo
method, the most reliable way is to go to the mojom definition first, and then
click - get the cross references from there. And the reason for this is
because, I guess, it's a quirk. I don't know what you want to call it.
00:00 SHARON: A feature.
00:00 DANIEL: A feature, yeah, we'll go with that. It sounds nicer. When we
generate the C++ definitions for a mojom-like interface or struct, we actually
generate two, what's called, variants. So one is - I call it the regular
variant. It uses STL types as `std::string`, `std::map`, all the fun things
that you're normally - sorry - `base::flat_map`. It doesn't use `std::map`. But
you get the idea. It's all the kind of regular container types. And the other
variant is what's called the Blink variant. And Blink uses `WTF::String`. It
has its own hash map type, its own vector type, et cetera. And so if you have a
Blink variant of an interface, when you pass arrays, it'll be passed as
`WTF::Vector`. And you're probably like, why did we do this? Why are we hurting
ourselves?
00:00 SHARON: [INAUDIBLE] like WTF Mojo.
00:00 DANIEL: Yeah, something like that. And the idea behind this is we already
had to do a conversion in the past. The way things worked is we handled IPCs in
the content layer, like in content render, or if you have Chrome render, or
whatever. But then we had to pass the data across what's called the Blink
public API. And the Blink public API would take all these STL types and marshal
it into the WTF types. And that means copying a bunch of string data or copying
a bunch of vectors or maps or whatever. And so it's not great from an
efficiency perspective. So we were like, well, we have to deserialize this data
already for Mojo. So why don't we just turn it into the right type to begin
with? So that's kind of what that's all about. So the problem with this is,
especially if you're in Blink, or in Content Browser, or something, if you
click on a Mojo - like on a call that you know is a Mojo call, it will find the
callers to that variant. So if you're on the browser side, there might - sorry
- that wasn't [INAUDIBLE]. So if you're in the renderer, you're like, who calls
this method? It's a Mojo - I want to know who is calling it from the browser
side. I click on it. Because it's a Blink variant, Code Search actually won't
go find the regular variant's caller. But if you go from the mojom definition,
it will. So that's the most reliable way to do it. It can also help if you
filter out generated files. Because, otherwise, it shows all the boilerplate
from the generated files. But usually, if you do that, it should work. If it
doesn't work, that's probably a bug. Please, file one, and we will try to fix
it.
00:00 SHARON: OK. When you say the Mojo file, there are - typically, there's
the .mojom file, and there's like .mojom.h. So you mean the first?
00:00 DANIEL: Yeah, I mean the first. Don't look at the generated files for
Code Search.
00:00 SHARON: In general.
00:00 DANIEL: It's because of this feature with variants that sometimes you'll
kind of get zero callers. But actually, your caller's in content, but you're
handling it in Blink - yeah, it's a mess.
00:00 SHARON: Yeah, all right. Because I've done that before, where I click a
function. I don't realize it's a Mojo call because it's overriding something.
And it's not immediately obvious. And you're like, oh, no one's calling it. We
should just remove it. But it's something that's very long and very clearly
important looking.
00:00 DANIEL: Yeah, yeah, yeah.
00:00 SHARON: And you're like, why are there no callers? Good tip! All right, I
think that is all of our questions. If someone watched this and was like, wow,
Mojo, this is so cool. Where can they go to learn more? We'll link the long
20-page doc and some other documentation. But beyond that, what can people do
if they're just like, I love me some IPC?
00:00 DANIEL: Well, I think one thing that's in pretty shabby shape perpetually
is the documentation for Mojo. We have tried to sort of incrementally improve
it. We did sit down and try to write docs for it a while back. But over time, I
think people have questions. And we haven't always had the time to go back and
update the documentation to reflect the questions people are having. But if you
do have questions, please, always ask them. There's a chromium-mojo mailing
list for public questions. There's a chrome-mojo one for internal questions.
And there's also the Mojo channel on the Slack. If you have questions, if
you're hitting weird compile errors with struct traits, I know that's always
kind of a big mess. Please, please, do ask questions. There's usually someone
lurking on there who's happy to help with -
00:00 SHARON: They're all very helpful.
00:00 DANIEL: But don't be silent. Because if you're silent, we don't know
things are a problem. And if we don't know it's a problem, it's kind of hard to
fix. But in general, we do try. Reach out. Mojo is not supposed to be
intentionally hard to use. And if you do find that's the case, please, ask us,
because people who work on Mojo don't always understand the tricky parts.
They're like, oh, this all make sense. But they already have that entire
framework in their mind. Whereas, someone kind of coming into, it's kind of
like, this makes no sense. This is dumb. We should - why doesn't it work like
X? And then we might change it to work like X, or we might update the
documentation to be like, it can't work like X because some reason. And that's
just helpful for everyone in the long run.
00:00 SHARON: I mean, as people often say, if you're new, you have perspective,
which is you are seeing this. You're not just used to how it works, including
the good and the bad parts. So yeah, it's a good time to ask questions. All
right, well, that sounds great. Thank you very much, Daniel. Thank you for
being here on the show. And we will see you all -
00:00 DANIEL: Thank you!
00:00 SHARON: next time. Cool, cool. We're relatively centered. No.
Codebase Browser
chromium