What are the guarantees about the networking? I saw this in the blog post:
> The server state is automatically synchronized to clients, and the data model is the same on the server and client. This means that in the future you can move your code between backend and frontend, or even run it on both seamlessly.
Different types of games need different types of networking. It seems like it's going to be hard to reason about how to program the game without knowing the constraints and guarantees on the network code.
Not necessarily, emulators has been doing this a while I think and with WASM this is the second impl I'm seeing of this.
Basically, keep a "full" state backlog and ship user control events, if control events arrive late you simulate forward to "now" from the backlog with corrected input.
I've been working on something very similar, gonna comment on the top-level.
Unless they are cooperative games giving copies of the server data model to the client inevitably leads to wild cheating. I give you the example of Escape for tarkov, where not only the data model is shared, but the client is authoritative over the server. A beautiful game ruined by the incompetence of the network-side development team.
Tarkov like most games uses client side prediction with some server side checks on actions, it can be hard to see the difference between a predicted action and client authority, they've been slowly increasing checks over the years as cheat developers finds new things to exploit. Since checks can be costly for performance you don't want to check things you don't have to, so doing it incrementally as you need is ok, for a game that is in a heavy development stage.
Unfortunately they are not doing lazy loading of loose loot and containers (including player/scav), so ESP is extra powerful in this game. But since Tarkov has complex maps there's little point in trying to hide players that are not visible, CSGO and Valorant are the only FPS games that does this but they also have simple and small maps with fewer players. So player-ESP and aimbot will most likely always be possible as long as you can bypass Battleye, so instead of cheaters rushing for loot they can see you might just go and kill you.
I agree on most of it. I think the way to do server side check is not in real time during the game, but after the game session ended. You should unload every gamestate on a secondary server that process everything and rate what could be manipulated with a degree of confidence. After this rating drop a lot on a player it should trigger a manual/automatic ban.
This went pretty terribly for Fall Guys[0] because the moment a false positive slips through, and then that player posts their clip of a game with close to 100% cheaters on Twitter or Reddit, it exacerbates their image of a game with a serious cheating problem. Everyone saw a game taking place jam packed with cheaters and it became a PR problem they had to overcome.
Giving full state is still a big problem. Aimbot, ESP, wall hacking are possibile only due to giving full state to the client. Most competitive games avoid this as much as possibile, eg. Dota2, Valorant, etc.
Right, I agree there but from a development perspective it's a problem that can/should be tackled later if you know you have a hit, and the Ambient design should have no issue in fixing it later.
- Aimbots is a problem regardless of state it, as long as you can see/shoot at it then aimbots can exist, stopping that is a separate issue.
- On ESP/Wall hacking you're right, it can be an issue
- Full snapshot/heap shipping as emulators and Kettlewells WASM sample is hard to patch here.
- an DoD ECS grounded design like Ambient could selectively withhold entities (ie out of sight), however I think for starting development it shouldn't be an initial worry as long as you don't design yourself into a corner in such a way that it's impossible to implement later.
At the theoretical level I agree with you about creating an MVP first and then solve these kinds of problems the moment you hit the jackpot at the ROI level. The problem is that in 20 years I have never seen a single game do that. Because revisiting the whole network/data stack in a game that has come of age is a nightmare and the PM probably doesn't see value in it. By now you are a hit. The problem is that instead of milking the cow for 10 years because you have a cheat proof product, you make money for 3-4 and then you close. (possibily damaging your brand as a cheater infested game)
Right, and that's why I like this approach (that I'm doing and it seems the Ambient lib is built for from the examples I looked at) because there is an actual architecture that is designed for networking from day one and visiting netcode isn't as bad an issue because it's mostly just regular game-code as long as the core principles are followed.
We've had generations of games that was built as webs-of-objects (and popular engines that still default to it), and yes, adding and maintaining netcode with that is like pulling tooth because you need to manually keep track of so much.
This puts netcode mostly as an automatable task with clear default rules, stuff like hiding players/entities out of sight becomes filters (in the same way as not updating far-away stuff).
Only a couple of years ago I played an incredibly fun game with an open multiplayer aspect to it. You didn’t really know who was friend or foe. Alliances and rivalries formed ad-hoc and betrayal was always around tge corner.
It was very well made but had a couple of power balance issues, which is it bo expected.
The problem was that it didn’t get its network code right from the very start. After a fun honeymoon period that was way too short, we would see cheaters all over the place. Not even “just” wallhacking but also teleporting around and dramatically increasing gun RPM.
This sucked out the whole fun in just a few days. My friends and I never played again even though they might have fixed it down the line.
Yup it relies on two things though, the simulation being deterministic which is easier said than done and the simulation being cheap enough you can replay enough frames to correct the missed inputs without derailing the current frame.
I didn't know this was a restriction before! Interesting. I would have assumed old headless had a profile, that typical command-line efforts[1] would let one load extensions. Are we sure that your question is valid? Are we sure that previous headless Chrome didn't have profiles or couldn't load extensions? I'm not sure this question is valid. I think maybe the assumptions here are incorrect.
The new Chrome headless certainly purports to be "just Chrome" "without actually rendering." One of the notable differences in the new headless mode is that it at least shows the stock/built-in extensions. From the submission:
> Similarly, when it comes to plugins, the old headless Chrome used to return no plugins with navigator.plugins, which is a technique that used to be exploited for detection when Headless Chrome got released 6 years ago, cf this blog post. The new headless Chrome returns the same plugins as a headful Chrome, and that’s the same for the mimeTypes obtained with navigator.mimeTypes:
Maybe perhaps the new headless is faking it, but my impression is that extensions definitely work as normal in the new headless Chrome. How or whether they worked before is another very very interesting question I'd like answers to.
I do wish the AMA dev had actually replied to this. My hope is that this wasn't an issue before (but default plugins just weren't installed, and now they are, just to alter fingerprinting), and that now the situation is unchanged but default plugins are installed.
This is definitely a valid concern. OpenAI did extensive data grooming to ensure high-quality inputs were used as training data. They went out of their way, for instance, to attempt to remove auto-translated content.
One interesting thing is that the situation you describe provides a huge moat for the first successful system. For instance, OpenAI can store and fingerprint all of the output from GPT-3 and ChatGPT. They can use these fingerprints to prevent training newer versions of GPT on old outputs. Less popular systems won't be able to sanitize the training data as well.
CRDTs have been on HN a lot recently. I'm working on a database that deals in events rather than raw data. Application developers specify event handlers in JavaScript. The database layer then takes the event handlers and distributes them to the clients. Clients receive the raw stream of events and reconcile the final data state independently. The key aspect is all the event handlers are reversible. This allows clients to insert locally generated events immediately. If any remote events are received out-of-order, the client can undo events, insert the new events, and reapply everything on top of the new state.
I'm curious how many people have a need for solving the real-time multi-player use case. The database I'm working on was inspired by the networking code used in fighting games (rollback netcode), but I'm always curious to learn about additional use cases.
I want most software on my computer to be transparently collaboratively editable. Like, I want to open a draft blog post on my desktop, go out, open my phone, keep editing the post, share it with susan, then get home and see her proposed changes and merge them like I would in a code editor.
And I want to be able to do that same workflow when editing video. Or when drawing. Or recording music. Or when configuring a server on AWS.
I agree. Everything should be collaborative. Feel free to reach out if you're interested in collaborating (email in my HN profile). I'm trying to get a v0 of this database out now, and building desirable use-cases is super important.
I'm working on a CRDT to solve this problem too[1]. How do you plan on implementing collaborative text editing on top of your event-reordering system? Off the top of my head I can't think of a way to implement text on your proposed system which would be performant and simple.
Basically, the data model is a string of text and a set of cursors, one for each user. The cursors contain an offset into the text where each user is editing.
When a user connects, the `user-connected` event is handled by setting the cursor position for that user to offset 0. When a user presses a key, an `insert-text` event is sent. The event handler for `insert-text` looks at the user's cursor position and inserts the text at that position and increases the user's cursor. Then, the event handler looks at all the cursors. Each cursor that has an offset greater than the location of the inserted text gets incremented by the length of the inserted text. Removal works the same way, but it removes text and decrements cursor offsets instead. When a user disconnects, the `user-disconnected` event handler removes the cursor to keep the total data size bounded.
As an example, let's say the text was "cat boy" and User A has cursor offset 0 and User B has cursor offset 4. Imagine User A inserts "dog" just a moment before User B inserts "girl", e.g. User B sends `insert-text: "girl"`. The first time this event gets applied, "girl" gets inserted at offset 4. When User A's insertion of "dog" arrives, the insertion of "girl" gets undone and then "dog" is inserted. After "dog" is inserted, User B's cursor is now at offset 7. So when `insert-text: "girl"` is re-applied, "girl" gets inserted at offset 7 and the final text is the same.
One thing I like about this system is that it's very simple. My prototype is like a collaborative notepad. The logic for state updates is only 100 lines long, including a lot of boiler plate for the event handlers (each is an interface implementation).
What happens if User B moves his cursor to position 0 at the same time that User A types at position 0?
Depending on the order that those events arrive, and how ordering is disambiguated, it's possible that different peers could see User B's cursor at different positions.
I believe that the system you're describing is akin to an OT system, but where the "operations" to transform are the "cursor move" actions that occur simultaneously with insertions and deletions. Most OT systems, on the other hand, are trying to transform inserts and deletes against one another.
Edit: Yes, in fact this is equivalent to OT if you replace every OT insert or delete with a tuple [move cursor, ins/del] in your system.
The case where User B navigates to cursor offset 0 as User A inserts at position 0 is actually easy because User B's cursor can just stay at 0. A harder case would be where User B moves to cursor offset X while User A removes text between (Y, Z) where 0 < Y < Z < X. How that situation gets handled is up to the application developer.
I'm not familiar with OT specifically, but the text editor I'm working on is just a proof-of-concept for the database. The database I'm building is designed to make it very easy to program very complicated multi-player scenarios, although it requires a little work to think of conflict resolution.
I'm developing a new type (AFAIK) of database that makes many types of real-time multi-player use-cases really easy to implement. The original idea came from implementing something called "rollback netcode" in multi-player video games, but the technique works for all types of applications.
Traditional databases (e.g. sql databases and key-value stores) treat the database as a blob of data. This makes multi-player updates very difficult, as reconciling differences in real-time is a hard problem, hence the invention of CRDTs. My new database, however, treats the data as a series of events. In the example of a text editor, the stream of events might be something like: User 1 connected, User 1 pressed key a, User 2 connected, User 2 pressed b, User 1 disconnected, etc.
In addition to the stream of events, application developers also provide an "engine" that converts the stream of events into a blob of data. The engine runs locally on each connected client. When a client connects to the database, they fetch the engine from the database server. The client then receives the stream of events from the database server in real-time. When a client generates an event, it gets inserted into the local engine immediately and also gets broadcasted to the server. The server then broadcasts the event to other connected clients.
The most important part that makes this work comes from a technique called rollback netcode. Events in the stream are ordered and undo-able. When a connected client generates an event, they insert it into their local engine immediately, but other clients are also generating events simultaneously. If all clients optimistically inserted events locally, the event order would differ across clients. To solve this, the engine automatically will undo events when new events come in from the server. The engine then applies the received event in its proper place and reapplies all the undone events on top. The result is that all users have a consistent view of the data.
The logic of an engine looks something like this:
1) When User X connected, set X's cursor position to offset 0.
2) When User X types a letter, insert text at User X's cursor offset and increase User X's cursor offset by 1. Also increment the cursor offset of all other connected users if their cursor offset is > User X's.
And that's it for a basic insert-only text editor. You can see that applying all the events in order yields the correct state. Adding a moveable cursor and the ability to remove text is also trivial. Basically just the opposite of 1 and 2 above. The whole engine can be implemented in like 100 lines of code and is very easy to reason about.
Our Aper (https://aper.dev) implements a number of similar concepts (state machine replication with optimistic local transitions + rollback). I 100% agree that it’s an easier model to reason about.
Your approach with cursors is clever, that part I haven’t seen elsewhere.
That looks cool. Thanks for sharing. Rust's enums and match syntax make the API look pretty slick. What types of applications are you seeing people use Aper for?
> The server state is automatically synchronized to clients, and the data model is the same on the server and client. This means that in the future you can move your code between backend and frontend, or even run it on both seamlessly.
Different types of games need different types of networking. It seems like it's going to be hard to reason about how to program the game without knowing the constraints and guarantees on the network code.