By default the cache is kept below 42MB and 20 files; to change the defaults change "FFmpegSource max cache size" and "FFmpegSource max cache files" in config.dat. Either or both may be set to 0, but the cleaner will not delete the last index file in the directory, regardless of cache settings and of the file's size.
Originally committed to SVN as r2893.