September 15th – Buncha Crunch

September 15, 2014 in Dev Blog

This week is off to a very good start, with lots of small (and not so small) progress made on several different fronts. My main accomplishment was adding functionality for quests to be turned in to quest giving NPCs rather than completing immediately once the requirements are met. Kyren’s continuing work on the status system and Lua APIs. We improved the gate placement and starting system selection, cleaned up a bunch of quest text, improved the layout of the AI interface a bit and fixed some GUI issues and other bugs. We’ve also started adding in tiered guns and ship upgrades for the Novakids!

novakidship

September 12th: Missions and Stuff.

September 12, 2014 in Dev Blog

Hello there!

So we’re going to have a few missions in Starbound and here’s just a glimpse of another one. This is a Floran mission where… bad things happen. Staying true to myself I’ll barely say anything and let you wonder…

pretty

pretty2

September 11ish – Post Lag

September 12, 2014 in Dev Blog

So as Kyren mentioned, yesterday I arrived in the UK! I said I’d write a post, but then immediately got distracted by a bunch of other discussions… sorry!

My main activities yesterday were discussing plans and priorities, getting my desk set up, and being a zombie from lack of sleep! Kyren and I talked a lot about her progress on the new status system, which is sounding awesome. The biggest feature is the conversion of all complex status effects to Lua scripts. Aside from massively improving the engine code and cleaning out a ton of unused hard coded stuff, this will allow us to easily implement some very interesting status effects, so we spent awhile coming up with ideas for possible scripted statuses. I also talked with GeorgeV and Armagon about how to implement features they need for the missions they’re designing. More traps, logic/wiring bits, and scripted/triggered events for NPCs are all in the works. More on that later; time for me to write some code!

Oh, and a small note for modders: Kyren added Lua bindings for getting and setting arbitrary world properties, so we now have access to world-level storage to communicate between script contexts and store persistent world data. I’m sure people will come up with some great uses for this!

September 9th – Pulse like you mean it!

September 10, 2014 in Dev Blog

Howdy guys!

Since I”m waiting on Metadept’s arrival at the office later this week before proceeding further on the missions, I spent my time today continuing one of my side tasks, which is to convert all existing lights over to the new flickering light code I discussed a few weeks ago. There’s a lot of lights that used the old flickering code, but I’m almost done with the conversion, and many lights (particularly fire-based ones) are being adjusted to look a bit more natural.

Working on this stuff isn’t terribly exciting in and of itself, but it makes for a welcome change of pace from all the mission work, and really it’s so gosh-darn nice to be able to make the lights pulse in time with their object’s animations!

Oooooh, shiny!

September 8, 2014

September 9, 2014 in Dev Blog

Hello again, since I spoke last, I’ve been working on a couple of different things.

First, there was the new status system for players, npcs, and monsters which has been written and integrated into Player, but for right now it sits in its own branch awaiting a couple more things.  The new system is basically ready, but there is a lot to the old status system which is needs conceptual reworking because it just wasn’t working out or wasn’t fun.  The biggest example of this is the heat / warmth system, which will probably be replaced by something much simpler.  The system is ready, but merging it to master is going to happen once metadept arrives in the UK and we can sit down together and make some design decisions and implement the scripts to drive them.  In any case, the system is cool, it works very much like tech that you don’t actively control (player effects that involve movement == tech, player effects that don’t involve movement == status).

Oh and by the way!  Metadept is moving to the UK to work with us in our physical actual corporeal office, and be a real life employee.  He arrives on thursday and I am super thrilled, because you just can’t beat working together in person.  Right now starbound programmers in the UK are pretty thin on the ground (me alone) so having me + metadept will be amazin’.

Other than status, I think you heard a little about the whole Star::Root threading debacle.. the important bit is that it’s fixed, and the atomic_store / atomic_load thing with shared_ptrs turned out not to even be needed.  Now *every* database in Star::Root is updated through Read Copy Update, not just the few with the problems, and Root locking is avoided entirely.  This fixes the problem without causing rendering slowdown, and actually speeds things up in a lot of places.

Other than THAT, I’m working on a few items in the starbound 1.0 todo, mission things and some boring technical ones involving Drawables and Particles.  When I’m done, rain will NOT slow down your game to a crawl, which is nice.  Next week, once metadept and I are both in the office, we’re going to start focusing very very hard on getting a new stable patch out, with all the save file compatibility fun that will go along with that.

Aaaaand in case you haven’t heard, Chucklefish is making a new game!  It’s called Wayward Tide and, you should check out the Chucklefish blog post about it here.  Development on Wayward Tide is *entirely* parallel with Starbound, a whole separate team in fact with Palf as the programmer.

In a technical sense, Wayward Tide is kind of cool for Chucklefish, because it represents a radical departure from the way starbound was developed, and if it works out it may very well be the way that Chucklefish makes games from now on.  It’s entirely written in Haskell, primarily based on Functional Reactive Programming, and it’s really really cool.  I would tell you more about it but I’m sure Palf will want to tell everyone more about it at some point in the near future.

September 5, 2014 – “Nightlies weren’t updating, here’s why” edition.

September 6, 2014 in Dev Blog

Another technical post from me. (yay?)

Some of you may have noticed that over the past few days the nightlies weren’t updating. There were two seperate issues and we managed to resolve both of them. Both of them took a bit of research and in the second case going through going back a long time (over a month!) to find the commit where stuff stopped working, (and serves as sort of an example case of how particularly pernicious software bugs form.)

If you rewind your brains to the September 1st you’ll remember reading this passage:

Kyren’s been working on nitty gritty technical stuff, addressing some threading issues we’re running into with C++11 pointers.

Specifically, what this was referring to was the fact that, while shared_ptr assignment is thread safe in the control section, it isn’t in the pointer section. We had been operating under the assumption that is was thread safe in this form and the bug affected two areas of code (leading to very intermittent problems).

The specific areas of code affected were the MaterialDatabase and the LiquidsDatabase, we hunted down the affected areas, replaced the code with std::atomic_load and std::atomic_store and all was well with the universe.

Or all was well, specifically on Kyren’s computer, running OSX 10.9 and Clang 3.4. After pushing she found out via our automated build system that gcc didn’t want to compile it. After some researching, she found that gcc, in fact, did not support atomic_store or atomic_load. So cue adding a new define and a short header file to our core library to handle the case of a compiler not having that feature implemented.

Fast forward a day or two, someone poked us about the nightly not updating. Quite a few people actually. We thought, “Well, that’s weird.” Because our automated build system automatically tells us when a git commit fails an automatic compile and test, and we didn’t get anything about that.

So we check the status of the test builders, and sure enough, everything is green. BUT, the OSX Release builder failed. Why? Link errors according to the log. It can’t find the atomic operations.

Long story short, we were targetting 10.7 on the release builder, and not the test builder. The test builder was using defaults and targetting the installed OS, 10.9. The version of libc++ that comes with the latest version of XCode that can target 10.7 also does not support atomic_store and atomic_load. Easy enough fix, but obnoxious. While we’re in the area, we clean the asset database to trigger a full rebuild, because it’s been a few months for that. Turns out that triggered the second problem.

The code that generated the asset pak file was silently failing. Actually, the failure wasn’t completely silent, but because the file was created and because our script that controls it did not check the return value of the program, the build system thought everything was AOK. And we wound up creating and uploading a 30kb asset pak file (instead of an 800MB one, how embarrassing.)

We, of course, didn’t notice at all, until someone else alerted us to it, actually quite a few people (again.)

The exception the asset packer is throwing is:

Exception caught: (EofException) Error, unexpected end of file found 1 0 0

Which seems super weird, because when we have an existing pak file, everything works, the only thing that’s wrong is when we’re creating a new one. And why would we be reading from the file in that case? Just weird all around.

Why is this suddenly failing? This is more or less how I approached the problem, step by step.

First stop, let’s take a look at our backtrace:

$ gdb ./asset_packer
(gdb) set args -d -i ../assets/pak_ignore.config ../assets/packed assets.pak
(gdb) catch throw
Catchpoint 1 (throw)
(gdb) r
Starting program: /home/omnipotententity/work/starbound/dist/asset_packer -d -i ../assets/pak_ignore.config ../assets/packed assets.pak
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Catchpoint 1 (exception thrown), 0x00007ffff7916a30 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0 0x00007ffff7916a30 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1 0x000000000044a2a8 in Star::Buffer::readPartial (this=0x7fffffffd6b8, data=0x7fffffffd57f "", len=1) at /home/omnipotententity/work/starbound/source/core/StarBuffer.cpp:76
#2 0x0000000000451a58 in Star::IODevice::readFull (this=0x7fffffffd6b8, data=0x7fffffffd57f "", len=1) at /home/omnipotententity/work/starbound/source/core/StarIODevice.cpp:125
#3 0x000000000044e083 in readData (len=1, data=0x7fffffffd57f "", this=0x7fffffffd690) at /home/omnipotententity/work/starbound/source/core/StarDataStream.cpp:71
#4 operator>> (d=@0x7fffffffd57f: 0 '\000', this=0x7fffffffd690) at /home/omnipotententity/work/starbound/source/core/StarDataStream.cpp:165
#5 read<unsigned char> (this=0x7fffffffd690) at /home/omnipotententity/work/starbound/source/core/StarDataStream.hpp:339
#6 operator() (__closure=<optimized out>) at /home/omnipotententity/work/starbound/source/core/StarDataStream.cpp:241
#7 operator* (this=<optimized out>) at /home/omnipotententity/work/starbound/source/core/StarAlgorithm.hpp:381
#8 readVlqU<Star::FunctionInputIterator<Star::DataStream::readVlqU(uint64_t&)::__lambda18> > (in=..., x=@0x7fffffffd5b8: 0) at /home/omnipotententity/work/starbound/source/core/StarVlqEncoding.hpp:37
#9 Star::DataStream::readVlqU (this=0x7fffffffd690, i=@0x7fffffffd5b8: 0) at /home/omnipotententity/work/starbound/source/core/StarDataStream.cpp:241
#10 0x000000000044e211 in Star::DataStream::readVlqU (this=this@entry=0x7fffffffd690) at /home/omnipotententity/work/starbound/source/core/StarDataStream.cpp:272
#11 0x0000000000441707 in Star::DataStream::readMapContainer<Star::Map<Star::String, Star::ByteArray, std::unordered_map<Star::String, Star::ByteArray, Star::hash<Star::String, void>, std::equal_to<Star::String>, std::allocator<std::pair<Star::String const, Star::ByteArray> > > >, void Star::DataStream::readMapContainer<Star::Map<Star::String, Star::ByteArray, std::unordered_map<Star::String, Star::ByteArray, Star::hash<Star::String, void>, std::equal_to<Star::String>, std::allocator<std::pair<Star::String const, Star::ByteArray> > > > >(Star::Map<Star::String, Star::ByteArray, std::unordered_map<Star::String, Star::ByteArray, Star::hash<Star::String, void>, std::equal_to<Star::String>, std::allocator<std::pair<Star::String const, Star::ByteArray> > > >&)::{lambda(Star::DataStream&, Star::String&, Star::ByteArray&)#1}>(Star::Map<Star::String, Star::ByteArray, std::unordered_map<Star::String, Star::ByteArray, Star::hash<Star::String, void>, std::equal_to<Star::String>, std::allocator<std::pair<Star::String const, Star::ByteArray> > > >&, void Star::DataStream::readMapContainer<Star::Map<Star::String, Star::ByteArray, std::unordered_map<Star::String, Star::ByteArray, Star::hash<Star::String, void>, std::equal_to<Star::String>, std::allocator<std::pair<Star::String const, Star::ByteArray> > > > >(Star::Map<Star::String, Star::ByteArray, std::unordered_map<Star::String, Star::ByteArray, Star::hash<Star::String, void>, std::equal_to<Star::String>, std::allocator<std::pair<Star::String const, Star::ByteArray> > > >&)::{lambda(Star::DataStream&, Star::String&, Star::ByteArray&)#1}) (this=this@entry=0x7fffffffd690, map=..., function=function@entry=...) at /home/omnipotententity/work/starbound/source/core/StarDataStream.hpp:490
#12 0x0000000000441942 in readMapContainer<Star::Map<Star::String, Star::ByteArray, std::unordered_map<Star::String, Star::ByteArray, Star::hash<Star::String, void>, std::equal_to<Star::String>, std::allocator<std::pair<Star::String const, Star::ByteArray> > > > > (container=..., this=0x7fffffffd690) at /home/omnipotententity/work/starbound/source/core/StarDataStream.hpp:523
#13 operator>><Star::ByteArray> (map=..., ds=...) at /home/omnipotententity/work/starbound/source/core/StarDataStreamExtra.hpp:155
#14 read<Star::Map<Star::String, Star::ByteArray, std::unordered_map<Star::String, Star::ByteArray, Star::hash<Star::String, void>, std::equal_to<Star::String>, std::allocator<std::pair<Star::String const, Star::ByteArray> > > > > (data=...,
this=0x7fffffffd690) at /home/omnipotententity/work/starbound/source/core/StarDataStream.hpp:345
#15 Star::DataStreamBuffer::deserialize<Star::Map<Star::String, Star::ByteArray, std::unordered_map<Star::String, Star::ByteArray, Star::hash<Star::String, void>, std::equal_to<Star::String>, std::allocator<std::pair<Star::String const, Star::ByteArray> > > > > (t=..., data=...) at /home/omnipotententity/work/starbound/source/core/StarDataStream.hpp:567
#16 0x0000000000441b05 in Star::DataStreamBuffer::deserialize<Star::Map<Star::String, Star::ByteArray, std::unordered_map<Star::String, Star::ByteArray, Star::hash<Star::String, void>, std::equal_to<Star::String>, std::allocator<std::pair<Star::String const, Star::ByteArray> > > > > (data=...) at /home/omnipotententity/work/starbound/source/core/StarDataStream.hpp:598
#17 0x000000000043efba in Star::AssetsDatabaseBackend::build (assetsFolder=..., databaseFilename=..., ignoreFiles=..., skipFiles=..., digest=digest@entry=true) at /home/omnipotententity/work/starbound/source/game/StarAssetsDatabaseBackend.cpp:17
#18 0x0000000000437cce in main (argc=<optimized out>, argv=<optimized out>) at /home/omnipotententity/work/starbound/source/utility/asset_packer.cpp:90
(gdb)

So walking backwards through the frames, let’s try to find out what’s going wrong. The top is just the internal throw, it’s not going to be that. After that we have the buffer readPartial, which is what actually threw the exception. But what could have caused it? Well, frame 10 has readVlqU, which is called from readMapContainer. The readMapContainer source looks like this:

template<typename Container, typename ReadFunction>
void DataStream::readMapContainer(Container& map, ReadFunction function) {
  map.clear();
  size_t size = readVlqU();
  for (size_t i = 0; i < size; ++i) {
    typename Container::key_type key;
    typename Container::mapped_type mapped;
    function(*this, key, mapped);
    map[key] = mapped;
  }
}

That seems reasonable. We read in the size and iterate through the entries. So it’s failing on the very first read, almost as if the buffer is empty.

So, if it’s failing the read, then maybe there’s something wrong with the write function?

template<typename Container, typename WriteFunction>
void DataStream::writeMapContainer(Container& map, WriteFunction function) {
  writeVlqU(map.size());
  for (auto const& elem : map)
    function(*this, elem.first, elem.second);
}

Well, no, this seems perfectly fine, it always writes the size.

Well, this seems like a dead end… next port of call is “How long has this been a problem, what triggered it?” So I start going through commits, roughly doubling and then bisecting. Each of these steps takes roughly 5 minutes, some more, some less.

$ git checkout '@{Sep 1}' 2>&1 >/dev/null && ../scripts/linux/setup.sh 20 2>&1 >/dev/null && rm assets.pak && ./asset_packer -d -i ../assets/pak_ignore.config ../assets/packed assets.pak 2>&1 >/dev/null && echo "Success!" || echo "Fail!"
Fail!
$ ^Sep 1^Aug 23
Fail!
$ ^Aug 23^Aug 15
Fail!
$ ^Aug 15^Aug 1
Fail!
$ ^Aug 1^Jul 1
Success!
$ ^Jul 1^Jul 15
Success!
$ ^Jul 15^Jul 23
Success!
$ ^Jul 23^Jul 27
Success!
$ ^Jul 27^Jul 30
Fail!
$ ^Jul 30^Jul 29
Fail!
$ ^Jul 29^Jul 28
Success!

So now… we know that the issue was caused sometime between Jul 28 and Jul 29, so let’s look at that diff.

$ git diff '@{Jul 28}' '@{Jul 29}'

It turns out that in one of these commits, some code that calls readMapContainer and writeMapContainer was changed in our DataStreamBuffer class, but only on the deserialization or reading side:

template<typename T, typename ReadFunction>
void DataStreamBuffer::deserializeMapContainer(T& t, ByteArray data, ReadFunction readFunction) {
  if (!data.empty()) {
    DataStreamBuffer ds(move(data));
    ds.readMapContainer(t, readFunction);
  }
}

Was changed to:

template<typename T, typename ReadFunction>
void DataStreamBuffer::deserializeMapContainer(T& t, ByteArray data, ReadFunction readFunction) {
  DataStreamBuffer ds(move(data));
  ds.readMapContainer(t, readFunction);
}

And if we double check our gdb backtrace way up there we do, indeed, go through this function. So this is what broke it. Why was it changed?

Well, if we look at it, it looks like it’s a check for an empty buffer, and an empty buffer used to equal an empty map, rather than a buffer that contained a encoded 0 (for the size). We had judged an empty buffer as actual data to be kludgy and not really fitting in the spirit of good programming, so we made that a fail condition. So now we know how it broke. But still there’s the question of why? Why was it working and now suddenly it’s not?

Well, let’s follow the backtrace farther up. At frame 17 we have where the call originated, let’s take a look:

void AssetsDatabaseBackend::build(String const& assetsFolder, String const& databaseFilename, StringList const& ignoreFiles, StringList const& skipFiles, bool digest) {
  SimpleSha256Database db("Assets2");
  db.setBlockSize(2048);
  db.setAutoCommit(false);
  db.setIODevice(File::open(databaseFilename, File::ReadWrite));
  db.open();
  // First, load the existing index for the database, if it exists.
  IndexMap oldIndex = DataStreamBuffer::deserialize<IndexMap>(db.find("_index").value());

So it looks like, we initialize the database, and then we check to see if already has an existing index of keys to pull from, and if it does, farther down the function we add stuff directly to the old index, rather than regenerating from scratch. This is how we initially fixed the issue where every update, not matter how small, was several hundred megabytes, by reusing the index key. So what happens if the index key is missing?

Well, db.find seems to return a Maybe<ByteArray> which is essentially in C parlance a char** where the internal char* is to an array of char and the external pointer is either to the pointer, or to nullptr, to represent a missing value. Calling .value() on a Maybe will return either what’s in the Maybe, or if nothing is in the Maybe, then it returns the parameter we passed to it. Or if no parameter is passed, it returns a default constructed value. So in this case when it’s a new database db.find returns an empty Maybe, which has value() called on it, which returns an empty ByteArray (eureka!)

So that’s where our empty ByteArray was coming from, and why the change broke this code.

So to fix, we simply just apply some more explicit coding in this area:

// First, load the existing index for the database, if it exists.
Maybe<ByteArray> oldIndexByteArray = db.find("_index");
IndexMap oldIndex;
if (oldIndexByteArray) {
  oldIndex = DataStreamBuffer::deserialize<IndexMap>(*oldIndexByteArray);
}

And we deftly avoid having an empty byte array sent through deserialize. And the nightlies work again.

September 4th – And in today’s news…

September 4, 2014 in Dev Blog

Evening folks!

I returned to my efforts on the lunar base today, as I’ve begun my detailing pass to make it all look nice and fill it with loot. There’s still a couple of elements I’m going to need before I can totally wrap up on the mission, like scripted NPC events (among other things). The good news is that we will have Metadept officially joining us in the office next week so we should see some really big progress on this front soon! I’m quite looking forward to having him in the office for the long haul, since he does a lot of stellar work and is a great guy to collaborate with.

George has continued working on his own mission and has been making excellent progress, with very little support required from yours truly. He’s already got a solid foundation down to build upon, and even though it’s rough around the edges it’s already looking great, I fear I may soon be eclipsed by his talent! That said, the fun bit will be when I get to teach him how to configure NPCs.

I’m afraid I don’t personally have much more to report, again for the wish to keep missions relatively spoiler-free. The programmers have continued to toil away on their respective tasks, which continue to sail over my non-programmer head, but what I was able to grasp is that they’re intensely focused on crushing a number of insidious bugs that crept their way in some time ago. Kyren in particular seems to be gearing up to assist Metadept when he arrives by tackling the systems he’ll need to work with.

Lastly, just in case you missed the announcement; the Chucklefish Blog has just gone live and Tiy kicked it off with a fairly comprehensive post on our plans for what will feature in Starbound’s 1.0 release. I wouldn’t advise reading it if you want a spoiler-free experience, but for those of you who absolutely must know what’s coming next you can read it . On this new blog we’ll be posting Chucklefish-related news, along with updates from the developers of games that we’re publishing, so you guys should definitely check it out from time to time!

Good night everybody!

3rd September – Chucklefish Blog

September 4, 2014 in Dev Blog, News

Hey guys!

In lieu of the daily post, yesterday Tiy took to our newly-launched Chucklefish Blog to post about what’s left to do before Starbound reaches version 1.0. Fair warning, though– the post and charts within contain spoilers aplenty.

You can check out the post here.

RE: The Chucklefish Blog – we’ll be posting Chucklefish-related updates there, and updates from developers of games we’re publishing. So keep an eye out! The site’s still not quite finished, either, and we’ll be filling it out a bit more in the coming days.

Regular daily progress posts will resume today!

September 2nd – New Mission Prep

September 2, 2014 in Dev Blog

Hey y’all!

I spent my day helping George by setting up a dungeon file and key for the mission he’s about to start working on. He had already knocked up a detailed plan of the mission’s structure and layout, so he spent most of today roughing it out visually in-game while I worked on getting a functional setup together. George seems keen to tell a story with the environment, so he’s been creating a bunch of cool new objects as he’s progressed.

It looks for the most part like he has everything necessary to get started turning it into a functional location, so most likely he’ll be spending the next week focusing on this mission. I’ll probably get called in from time to time to help him with the finer points of the dungeon system, but I’m really looking forward to seeing how he does with it, since his mockups are generally very good and having someone else to work on this stuff with me will be most welcome.

Sorry I don’t have anything fancy to show off, it’s just been that kind of work lately and we’d like to keep at least a few surprises, if possible.

Good night!

September 1st – WIPs

September 2, 2014 in Dev Blog

Not much exciting to post about today. Kyren’s been working on nitty gritty technical stuff, addressing some threading issues we’re running into with C++11 pointers. Armagon and the artists are continuing work on missions that I’m not allowed to post about yet.

I’m still playing with the terrain generation, seeing what its capabilities and limitations are. Here’s a screenshot of some WIP desert canyons (click for big):

canyonswipThe terrain generator works by combining multiple layers of Perlin noise. In this shot, I’m using one source to define the primary surface terrain (low dunes), another source to define the rough bottom of the canyons, and a third to mix between them, with bias toward the primary source to make canyons less frequent. There are several problems with the current configuration, however. The most obvious problems are the sharp corners at the canyon edge where the mix shifts from 100% primary surface and rapidly begins switching to the lower canyon terrain. These can be addressed with some smoothing in the mixer (which needs engine changes to implement). Another issue without an obvious solution is what to do with the background. Because of the dimensionality and limited number of layers, it’s difficult to visually distinguish an open valley from a long, narrow canyon from a deep hole. I’ll have to experiment further to find a satisfactory solution.

 

spiele