Sparse Readdirs are a warning condition that occurs on our current storage system. Under certain circumstances, the number of entries (directory or file entries) directly below a particular directory, can cause Sparse Readdirs, and can cause further problems if not dealt with.

What causes Sparse Readdirs?

Sparse Readdirs occur when the number of directory or file entries immediately within a directory is very large, then reduces in size. Basically, the size of the directory entry itself on the file system increases as the number of entries underneath it increases. However, when those entries are removed, the directory entry's size does not shrink back down, but rather holes are left in the directory entry for future use. If the number of entries in a directory is very large (eg. many thousands or more), and then most of them are deleted, the directory entry is considered sparse, much like a Sparse Matrix.

What problems do Sparse Readdirs cause?

Sparse directory entries take significantly longer to read, especially during the readdir operation. This operation occurs in a number of situations, as it is the general system operation used to list all the entries underneath a directory. For example, if you use the ls command to list all the files and directories underneath a particular directory, it uses the readdir operation to do so.

In addition to slowing down the operations, if a readdir takes too long, the storage system sends us an email, notifying us of the problem. If we get more than a few of these, we will usually contact you. Most likely, if you're reading this page, that's what has happened.

What do I do about Sparse Readdirs?

When we send you an email about the sparse readdir problems, we should identify which directory in particular is causing the problem. What to do about it will depend on whether or not you have full control of the directory.

If you do not have full control of the directory (eg. it's your home directory or your compute directory, etc., and you can't re-create it on your own), then we'll have to implement the fix, and we will have to coordinate with you.

If you do have full control of the directory (eg. if you created it as a subdirectory somewhere, etc.,), then you can fix it using the following process:

1. Make sure that the directory and any files or subdirectories underneath it are not being used. This includes any jobs you might have running on the system.
2. Create a new directory, setting the same ownership and permissions as the original directory
3. Move all entries from the original directory to the new directory. Make sure to include any hidden files as well.
4. Remove the original directory
5. Rename the new directory to take the place of the old directory

Here is one example of the commands to create the process:

[lbrown@m6int02 sparse_readdir_example]$ ls -l
total 4
drwxrwx--- 5 lbrown lbrown 2048 Nov 29 09:47 datadir
[lbrown@m6int02 sparse_readdir_example]$ #Create a new directory, setting the same ownership and permissions as the original directory
[lbrown@m6int02 sparse_readdir_example]$ mkdir newdatadir
[lbrown@m6int02 sparse_readdir_example]$ chmod 770 newdatadir
[lbrown@m6int02 sparse_readdir_example]$ chown lbrown:lbrown newdatadir
[lbrown@m6int02 sparse_readdir_example]$ ls -l
total 8
drwxrwx--- 5 lbrown lbrown 2048 Nov 29 09:47 datadir
drwxrwx--- 2 lbrown lbrown 2048 Nov 29 09:49 newdatadir
[lbrown@m6int02 sparse_readdir_example]$ #Move all entries from the original directory to the new directory.  Make sure to include any hidden files as well.
[lbrown@m6int02 sparse_readdir_example]$ mv -v datadir/* newdatadir/
`datadir/1' -> `newdatadir/1'
`datadir/2' -> `newdatadir/2'
`datadir/3' -> `newdatadir/3'
`datadir/text.1' -> `newdatadir/text.1'
`datadir/text.2' -> `newdatadir/text.2'
`datadir/text.3' -> `newdatadir/text.3'
[lbrown@m6int02 sparse_readdir_example]$ ls -l datadir/
total 0
[lbrown@m6int02 sparse_readdir_example]$ ls -la datadir/
total 8
drwxrwx--- 2 lbrown lbrown 2048 Nov 29 09:50 .
drwxrwx--- 4 lbrown lbrown 2048 Nov 29 09:49 ..
-rw-rw---- 1 lbrown lbrown    0 Nov 29 09:47 .hidden.text.1
-rw-rw---- 1 lbrown lbrown    0 Nov 29 09:47 .hidden.text.2
-rw-rw---- 1 lbrown lbrown    0 Nov 29 09:47 .hidden.text.3
[lbrown@m6int02 sparse_readdir_example]$ mv -v datadir/.hidden.text.* newdatadir/
`datadir/.hidden.text.1' -> `newdatadir/.hidden.text.1'
`datadir/.hidden.text.2' -> `newdatadir/.hidden.text.2'
`datadir/.hidden.text.3' -> `newdatadir/.hidden.text.3'
[lbrown@m6int02 sparse_readdir_example]$ ls -la datadir/
total 8
drwxrwx--- 2 lbrown lbrown 2048 Nov 29 09:50 .
drwxrwx--- 4 lbrown lbrown 2048 Nov 29 09:49 ..
[lbrown@m6int02 sparse_readdir_example]$ 
[lbrown@m6int02 sparse_readdir_example]$ rmdir datadir
[lbrown@m6int02 sparse_readdir_example]$ mv -v newdatadir datadir
`newdatadir' -> `datadir'
[lbrown@m6int02 sparse_readdir_example]$ ls -a
.  ..  datadir
[lbrown@m6int02 sparse_readdir_example]$ ls -al
total 16
drwxrwx---  3 lbrown lbrown 2048 Nov 29 09:51 .
drwxrwx--- 51 lbrown lbrown 8192 Nov 29 09:46 ..
drwxrwx---  5 lbrown lbrown 2048 Nov 29 09:50 datadir
[lbrown@m6int02 sparse_readdir_example]$ ls -al datadir/
total 20
drwxrwx--- 5 lbrown lbrown 2048 Nov 29 09:50 .
drwxrwx--- 3 lbrown lbrown 2048 Nov 29 09:51 ..
drwxrwx--- 2 lbrown lbrown 2048 Nov 29 09:46 1
drwxrwx--- 2 lbrown lbrown 2048 Nov 29 09:46 2
drwxrwx--- 2 lbrown lbrown 2048 Nov 29 09:46 3
-rw-rw---- 1 lbrown lbrown    0 Nov 29 09:47 .hidden.text.1
-rw-rw---- 1 lbrown lbrown    0 Nov 29 09:47 .hidden.text.2
-rw-rw---- 1 lbrown lbrown    0 Nov 29 09:47 .hidden.text.3
-rw-rw---- 1 lbrown lbrown    0 Nov 29 09:46 text.1
-rw-rw---- 1 lbrown lbrown    0 Nov 29 09:46 text.2
-rw-rw---- 1 lbrown lbrown    0 Nov 29 09:46 text.3
[lbrown@m6int02 sparse_readdir_example]$ 

Preventing Sparse Readdirs in the future

The best thing you can do to keep the problem from happening in the future, is to avoid having too many entries within a specific directory. Since the problem occurs when the list of subentries for a directory gets too large, we recommend using subdirectories to organize your data, keeping the number of entries within a particular directory, to a few hundred at most.