Degredation, Line Wrapping & Cache Performance
Made bunch of fixes, some content additions and fallen down a bunch of rabbit holes over the winter break.
Degradation
Items that have charges and degrade is one of the final core systems that was missing, it's implementation wasn't too complex, storing charges either per player (varbit), alongside inventories slots or per item id e.g. Black mask (3).
I implemented all of the charge values and degrading and some bits of the recharging to:
- Barrows
- Torva, Virtus, Pernix
- Statius, Morrigans & Vestas
- Corrupt equipment
- Crystal bow/shield
- Games necklace
- Ring of duelling
- Dig site pendant
- Skills necklace
- Combat bracelet
- Amulet of glory
- Pharaohs Sceptre
- Teleport scrolls
- Ectophial
Which led onto my first rabbit hole...
Dialogue
Adding some of the dialogues used in item recharging I re-discovered https://chisel.weirdgloop.org/ and the dialogue data that was gathered by the Wiki Crowdsourcing Project.
I scraped all the data, which unfortunately is harder since the wiki has started using cloudflare, and wrote a little command line script to take a dialogue chain and print it out as code.
It's not perfect, method names need manually refactoring and it doesn't include chat head animations as it unfortunately wasn't included in the data collection but it's a 60% start when writing new dialogues.
Which naturally led onto adding support for dialogue text line wrapping, easy enough but a nice quality of life feature for content development.
Font's are used in line wrapping and some but not all of the font names are known, queue rabbit hole 3...
Hashes
I thought it would be a bit of fun to try brute force some of the hashed file names stored in the cache to see if I could discover some more of the font's names, a few approaches later making a dictionary of common runescape words didn't work out to badly plus cross referencing some of the known osrs hashes and leaked hashes and I discovered not only most of the font names but names across all of the cache's index, you can find out more about that on my config thread.
Cache Library
One of the goals of this year is to make the setup experience smoother for people who want to pick up the server and play, part of this I want to add different run modes both for development and theoretical live gameplay, as well as a low-memory mode. Part of doing this I want to bring my file-server back into the codebase to offer a single complete solution.
The file-server currently while performant, does required a file lock on the cache, it's memory usage varies over time and it's read speed isn't as fast because the data is being read from file.
So, why not load the whole cache into memory?
Well, because it's slow. Up until now I've been using Displee cache library which has all the features I need but takes 200ms to start up and 5.4s to load the entire cache into memory, which if you've been following along doesn't meet my < 3s server boot time target. That is also one of the reasons why I created an "active cache" to re-write cache data into a binary format for faster load times last year.
So I thought, can I write a cache library to load it any faster?
Profiling the library I found half of the time was spent decompressing BZIP2 files which is surprising given there are only 1702 files compressed with BZIP2 out of 630k total files. Replacing those files compression with GZIP reduced load times to 2.7s!
But there's still more to be done. I replaced the LMZA decompression with a newer implementation, removed some unnecessary array creations, then I stripped the code down completely; removing classes and slower data structures until nothing but a multi-dimensional byte array was remaining.
This knocked off hundreds of milliseconds here and there and finally I parallized loading across multiple threads. Results will vary by machine and threads used but with mine I have gotten it down to a startup time of 700ms, and read time is essentially nothing (10-25ms) as everything is held in memory.
Having the cache now fully in memory also removes the need for this "active cache", which removed another 50-80ms for some file checksums but the map loading isn't quite as fast at 300-450ms. So server boot times are 3.1s vs 2.4s previously which is acceptable for a live server and a worthwhile trade off given the amount of complexity and code it removes. There won't always be the need for the 500MB overhead of storing the cache in memory however, so I also wrote a cache implementation using random access files as well. This has a 30-50ms startup time and 2.5s full cache load time, fractionally faster than Displee's cache and few less MBs of memory usage, much more suitable for development.
That brings the total server boot time to 3.1s with the MemoryCache, 2.3s with the FileCache, 4.3s with Displee's cache, and 2.5s with my previous active-cache implementation.
Cache files
With the removal of Bzip2 compression the number of cache modifications is now starting to grow and making new caches manually is error prone so I spent some time creating a script to take a blank 634 cache and apply all of the modifications to it automatically.
While I was at it I wrote a script to take the middle mouse button camera control cs2 which makes the full list of cache modifications currently:
- Copying inventory definitions from higher revision cache
- Copying item parameters from higher revision cache
- Copying middle mouse camera control scripts and interface components
- Finding missing maps from other revision caches
- Removing map xteas
- Removing bzip2 compression
For those asking the project doesn't have a discord as I there's no current plan to host and all updates are posted here anyway. Testers also aren't needed currently, but you of course are able to give it a play for yourselves instructions are on the github wiki page and I'll be making it easier setup with release builds coming soon.
That is all for now but there's plenty more on the way.
