First: Day 0

Breakfast for day 2 was had at The Laundromat - a busy cafe specialising in breakfast and lunch foods. We met with a few members of Star Frontiers, a fellow TEST corporation and interesting people. 

The opening speech / keynote [video 1 / video 2]

However, the main focus of the day was Fanfest, and it was just about to begin. We gradually shuffled into the main auditorium as the lights dimmed. After the introductions were over, there was some talk of EVE’s release schedule. For around ten years, EVE expansions were released on a regular basis every six months. Over the past year or so, however, EVE has started releasing every six weeks instead. This has been a massively positive change for the game’s development - it allows both work shorter than six months to be released early, and long-term work to go on longer without having to squeeze into a six-month timeslot. It also means less thrashing since there’s much less competition for different teams to get their code into a release. The new announcement this time is that EVE would move away from named releases altogether - there are still planned release dates spread over the next year, but smaller features might be delivered in between those releases whenever they’re ready. Release notes are now tracked on a new website that shows off the continuous delivery model better than having individual pages for each release. 

At work we’re trying to improve the frequency of our product releases, and I can confirm that it really increases the flow of development and reduces fear when developing features - things can be released once they’re done, and if something goes wrong, another release is always around the corner. It also forces you to streamline the release process as much as possible. We’re working on automating releases at the moment, and having frequent releases highlights how important it is to have a well-defined (if not fully automated) process.

EVE does have an advantage here - since they’re deploying releases to a single game instance, they can be sure that everyone is always updated to the latest release. Our updates feature isn’t slick enough that it would become quite annoying if you were forced to update every week, and we have customers that prefer to stay on a particular version since it’s working for them. Nevertheless, this is something we could look into improving in the future.

Next up was Valkyrie, a VR dogfighting game that was spun off from EVE after a tech demo two years turned out to be super popular. The focus here was on polish and improvement - Valkyrie doesn’t have a release date yet (although we know that it will launch alongside the Oculus Rift and Sony’s Project Morpheus), but it’s obviously progressing along nicely. The most interesting part of the presentation from a development perspective was the razor used for choosing features - “to be the best competitive multiplayer VR game.” There’s a lot of interesting things that could be done with Valkyrie - including a strong singleplayer campaign - but choosing early what to focus on should hopefully make it more likely to be a successful product. This means that a lot of the backstory will be told though a set of comics instead, freeing up development time to hone the multiplayer component. There will be a short singleplayer introduction, acting as a tutorial for the multiplayer game, and a stunning trailer was shown off to much excitement.

Also announced during the keynote was a wide-ranging rewrite of player-built structures in EVE. ‘Structures’ are one of the main ways players interact with space - either as stations to fight over, dock in and trade through, mini fortresses to stage assaults from and fight over, or industrial complexes to gather materials and build EVE items from rounds of ammo to giant spaceships.  For a while, the main unit of EVE structures has been the POS, or ‘player-owned starbase’ - basically a round bubble about 60km wide, anchored to a moon, that can be filled with anything from hundreds of guns to a set of moon mining modules. The POS code is notoriously hard to change and some of the more legacy aspects of POSs have come to really annoy players in the last few years, although they’re still super useful overall.

The main thrust of the structures changes is introducing a whole new set of small, modular structures instead of trying to replace POSs directly. The new structures should be much better from a player perspective - being able to pick and choose the right things to deploy - and from a development perspective - being able to implement and iterate on individual structures in a piecemeal way. Instead of having to rewrite POS code, POSs will slowly be phased out and replaced with the new structures. 

Also shown off were some of the metrics that are collected around EVE’s playerbase. As well as showing off some pretty animated graphs, a split of players into different personas, or ‘stereotypes,’ was demonstrated, and some player-built visualisations were demonstrated based on CCP making the same data available through EVE’s extensive API.

Once the opening was over, Fanfest begun properly. Each day there are a range of presentations and roundtables related to many different aspects of EVE and Valkyrie. I ended up mostly investigating dev-related topics, but a lot of the other sessions were also recorded and should be uploaded by CCP sometime this week. I may have other reactions to post once that happens.

‘Politics by a different name’ [video]

The first presentation was about Fozziesov (or Sovereignty 5.0, to give it it’s proper name). CCP Fozzie took the stage to explain the current proposals for a new sov system and reveal a couple of extra new details. Firstly, a quick explanation: Sovereignty is one of the main game mechanics by which EVE players alter the universe they live in. A large proportion of the star systems in the EVE universe are not owned by the ingame Empire factions, but instead can be owned and fought over by different player alliances. The current sov system is notoriously complicated, particularly since system ownership, station ownership and system upgrades were all fought over in one long process. The new system is particularly notable for decomposing these three processes into three separate systems with much simpler mechanics.

What’s also unusual is that the new sov proposal was actually announced a couple of weeks before Fanfest. Normally, big announcements like this would be reserved for the fanfest keynote, but CCP are now saying that they’d much rather have some of the basic details hashed out first before talking with players over the roundtables. This means that CCP is sacrificing some significant media exposure for the sake of better discussion - it means a slightly less exciting fanfest, but is better for the game overall. Fozzie did have a couple of new annoucements, though, mostly tied to some quality-of-life issues that will be improved before the main sov changes happen. These were minor but well-appreciated by the audience.

‘Bro, do you even test?’ [video]

The next talk I went to was about EVE’s QA department and the steps they go through when working on a new feature. They talked a bit about their ‘whole team quality’ approach to QA and how they would start testing features as soon as the devs had something ready. Most of the testing from the QA side happens as detailed test plans for manual tests - testing an interactive game like EVE is hard to do in a more automated way, although some progress is being made on that front with custom tools such as thin clients and ‘EVE Probe’. The QA department also gets players involved on its test server instance in occasional mass tests which test how new features or optimizations affect server load one way or the other. Having well-defined test plans means that CCP can also delegate testing to one of their ‘outsource partners.’ 

So what happens when you can’t rely on automated tests for debugging and regression testing? Firstly, devs and testers have a strong reliance on high-quality logging from both the client and the server. All exceptions are logged with stack traces which can often be enough to pinpoint a bug. Interactions between the client, player and server are also heavily logged, which means CCP often has enough information to debug an issue without a direct reproduction. Dxdiag output is also captured when reporting a bug in order to track down hardware-related issues. All in all, EVE generates about 52GB of logs a day (and rising).

EVE also has a whole load of ‘dev hacks’ that can be added to the client. These will teleport the player around, spawn different ships in space, create some enemies to fight, and so on. This can massively speed up running through a test script, although care has to be taken not to skip too much of the game for fear of missing bugs.

When a feature eventually gets released, the server is sometimes brought up in ‘VIP mode,’ where only devs are able to play, for last-minute checks and tests. This is possible because EVE players are used to a daily downtime period where star systems are re-seeded and the database is cleaned up. CCP have been working to reduce the downtime period, and are now able to release a few changes (particularly client-only fixes) without causing a downtime to happen.

The presentation went on to talk a bit about bug reporting in EVE, along with a bit of help on writing useful bug reports. Of note was the fact that QA tend to get about 2500 bug reports per release, with about 80% of them being addressed before the next one.

After all that I want to look at adding devhax to some of the programs I’m working on - being able to quickly get the UI into a certain state or inspect some live variables while the program is running would be very useful. I imagine the initial setup could be troublesome, though, but just adding a simple UI that would let you evaluate arbitrary C# code within the context of the program could be a good first step. The other potential improvement is allowing users to report bugs from within the program — not only is this easier for the user, but it would also allow additional information about the program state (and the user’s setup) to be captured and inspected by the user before being sent off to support.

I've historically been very much against any form of manual testing - but that might just be because we’re pretty bad at it compared to CCP. Make no mistake, I still prefer automated testing where possible, but on current systems were it’s hard to get automated tests in place easily, getting better at manual testing could well make the transition easier.

One of the common problems we have with large manual or automated test suites is having to run the whole thing before each release. It seems like the QA team here has a much better idea (with documentation) of how to separate out different features and which test plans are applicable to the features being changed in each release. I’m guessing that this makes the manual test burden a lot more manageable than it is for some of the products we’re working on at the moment.

The other factor is having better test scripts with easy setup - currently the best test script we have is a worked example in our support site. It’s good because it has a script that sets up the environment in one step, and it provides a set of steps to run through that are easy to follow while still allowing a bit of exploration.

The 'American Bar'

Dinner was taken at the more traditional 'American Bar'. Nothing to particularly write home about, but the burgers were tasty and the beer was light and refreshing. I headed home early, since tomorrow was the eclipse ...