I’m sure by now most of you have noticed one of the changes we’ve implemented. In this post, I’ll explain what this update means and how it will change the database moving forward.
For the first change, I added a column to the database for “Oldest Record” where you will see a date. Here’s a couple examples of it looks.
Let me remind you how we obtain data. Once a week, each of the guilds’ sales history is recorded for the previous 7 days. We do a bulk scan on Tuesday evening after trader flip, so tomorrow I’ll have a scan for sales dated 12/28/2020 through 01/05/2021. When the data is uploaded to the database, it will include 12/28/2020 in the date column. This isn’t necessarily the date you will see associate with all of the items though. Keep reading and you’ll find out why.
The next change involves the monthly data dump. With the previous method of purging, all data older than 60 days and all crafting materials older than 30 days were removed at the end of each month. The latter method was also applied to popular items that had a decent number of sales during the month. Moving forward, purging will occur weekly while the time frame data can remain in the database will be infinite. That doesn’t make a whole lot of sense, but let me explain. By adding in the date column, I am able to easily identify how old each record is (on the back end there’s always been a date column, but I used months, not dates).
High volume items like Rubedite Ingots, will be purged every other week (I’ll be watching closely to determine if I can purge these weekly), while moderate volume items will be kept for up to a month. Anything older than a month will be removed unless it is the only remaining record for that item. (take a look at “Alinor Amphora, Portrait” under Furnishings for an example). With these rare items, once a new sale comes in, the old data will immediately be removed.
The last change you’ll see is in the data archives. If you look at any of the files for previous months, you’ll see it includes up to 3 months’ worth of data (all data that was live in the database at the end of each month). Moving forward, each month’s archive file will show only the data obtained during that month. Why am I doing this? I don’t know, it just seems to make sense.
If you have any concerns regarding this update, I urge you to open a dialog with me. I’m always happy to discuss how things are done!