----------------------------------------------------------------------------------------------------------------------------
Help, my index file grows like crazy, its even bigger than my data file, what's going on?
Explanation
Whether or not your index file grows like crazy when inserting thousands of files in your filestorage depends on how random your GUIDs are. The IndexFile is currently everything BUT designed to be small. The reason is that NFileStorage is simply not optimized (at least not currently) on keeping the index small. For my current needs I don't mind my index file is around 3 GB (yes, no kidding :) for storing around 130.000 items. If I would have picked a different alghoritm to create GUIDs the index file could have been way smaller though. The 'smarter' algorithm would not generate a random GUID (like Guid.NewGuid() would), but make a GUID based upon for example a timestamp. Each byte (of the 16 bytes of the GUID) will allocate a 256 x 64bit allocation to point to the next structure, up and until the 16th byte itself, which points to the offset in the data file.
The index file is just a big dictionary that will allow quick lookups from a GUID to the location where the data is stored in the data file.
The index file can be restored (re-generated) from the data file, so if you want to deploy a new FileStorage to your production environment, or if you want to make a backup, it will be sufficient to only backup the data file, and use the command tool to generate the index file. Regenerating an index file goes pretty fast for smaller files, if you have > 10.000 items you do need to allocate some time for the restoring of the index file to finish (like an hour or so). But then again.
Solutions to reduce the size of the index file
Road map regarding the index file
Maybe in the future the logic (strategy) of the index and data file can become pluggable (I think it should be fairly simple to make it pluggable), which could mean depending on your situation the NFileStorage will behave differently (for example; drastically reducing the size of the index file), or it could or should even be possible to migrate one storage to another one which behaves differently when this would be more efficient. Anyways, that's the roadmap I am talking about now.
For a detailed version of the technical design of the index file, see
index file details
----------------------------------------------------------------------------------------------------------------------------