Package org.apache.cassandra.db

Class Directories

  • public class Directories
extends java.lang.Object
    Encapsulate handling of paths to the data files. 
     
   /<path_to_data_dir>/ks/<cf dir>/ks-cf1-jb-1-Data.db
                         /<cf dir>/la-2-Data.db
                         /<cf dir>/.<index name>/ks-cf1.idx-jb-1-Data.db
                         /<cf dir>/.<index name>/la-1-Data.db
                         ...
    Until v2.0, <cf dir> is just column family name. Since v2.1, <cf dir> has column family ID(tableId) added to its end. SSTables from secondary indexes were put in the same directory as their parent. Since v2.2, they have their own directory under the parent directory whose name is index name. Upon startup, those secondary index files are moved to new directory when upgrading. For backward compatibility, Directories can use directory without tableId if exists. In addition, more that one 'root' data directory can be specified so that <path_to_data_dir> potentially represents multiple locations. Note that in the case of multiple locations, the manifest for the leveled compaction is only in one of the location. Snapshots (resp. backups) are always created along the sstables there are snapshotted (resp. backuped) but inside a subdirectory named 'snapshots' (resp. backups) (and snapshots are further inside a subdirectory of the name of the snapshot). For secondary indexes, snapshots (backups) are not created in their own directory, but are in their parent's snapshot (backup) directory. This class abstracts all those details from the rest of the code.

    • Constructor Detail

      • Directories

        public Directories​(TableMetadata metadata,
                   Directories.DataDirectory[] paths)
        Create Directories of given ColumnFamily. SSTable directories are created under data_directories defined in cassandra.yaml if not exist at this time.
        Parameters:
        metadata - metadata of ColumnFamily

    • Method Detail

      • verifyFullPermissions

        public static boolean verifyFullPermissions​(File dir,
                                            java.lang.String dataDir)
        Checks whether Cassandra has RWX permissions to the specified directory. Logs an error with the details if it does not.
        Parameters:
        dir - File object of the directory.
        dataDir - String representation of the directory's location
        Returns:
        status representing Cassandra's RWX permissions to the supplied folder location.

      • getLocationForDisk

        public File getLocationForDisk​(Directories.DataDirectory dataDirectory)
        Returns SSTable location which is inside given data directory.
        Parameters:
        dataDirectory -
        Returns:
        SSTable location

      • find

        public Descriptor find​(java.lang.String filename)

      • getDirectoryForNewSSTables

        public File getDirectoryForNewSSTables()
        Basically the same as calling getWriteableLocationAsFile(long) with an unknown size (-1L), which may return any allowed directory - even a data directory that has no usable space. Do not use this method in production code.
        Throws:
        FSWriteError - if all directories are disallowed.

      • getWriteableLocationAsFile

        public File getWriteableLocationAsFile​(long writeSize)
        Returns an allowed directory that _currently_ has writeSize bytes as usable space.
        Throws:
        FSWriteError - if all directories are disallowed.

      • getWriteableLocationToLoadFile

        public File getWriteableLocationToLoadFile​(File sourceFile)
        Returns a data directory to load the file sourceFile. If the sourceFile is on same disk partition as any data directory then use that one as data directory otherwise use getWriteableLocationAsFile(long) to find suitable data directory. Also makes sure returned directory is not disallowed.
        Throws:
        FSWriteError - if all directories are disallowed.

      • getTemporaryWriteableDirectoryAsFile

        public File getTemporaryWriteableDirectoryAsFile​(long writeSize)
        Returns a temporary subdirectory on allowed data directory that _currently_ has writeSize bytes as usable space. This method does not create the temporary directory.
        Throws:
        java.io.IOError - if all directories are disallowed.

      • removeTemporaryDirectories

        public void removeTemporaryDirectories()

      • getWriteableLocation

        public Directories.DataDirectory getWriteableLocation​(long writeSize)
        Returns an allowed data directory that _currently_ has writeSize bytes as usable space.
        Throws:
        FSWriteError - if all directories are disallowed.

      • hasDiskSpaceForCompactionsAndStreams

        public boolean hasDiskSpaceForCompactionsAndStreams​(java.util.Map<File,​java.lang.Long> expectedNewWriteSizes,
                                                    java.util.Map<File,​java.lang.Long> totalCompactionWriteRemaining)
        Sums up the space required for ongoing streams + compactions + expected new write size per FileStore and checks if there is enough space available.
        Parameters:
        expectedNewWriteSizes - where we expect to write the new compactions
        totalCompactionWriteRemaining - approximate amount of data current compactions are writing - keyed by the file store they are writing to (or, reading from actually, but since CASSANDRA-6696 we expect compactions to read and written from the same dir)
        Returns:
        true if we expect to be able to write expectedNewWriteSizes to the available file stores

      • hasDiskSpaceForCompactionsAndStreams

        public static boolean hasDiskSpaceForCompactionsAndStreams​(java.util.Map<File,​java.lang.Long> expectedNewWriteSizes,
                                                           java.util.Map<File,​java.lang.Long> totalCompactionWriteRemaining,
                                                           java.util.function.Function<File,​java.nio.file.FileStore> filestoreMapper)

      • hasDiskSpaceForCompactionsAndStreams

        public static boolean hasDiskSpaceForCompactionsAndStreams​(java.util.Map<java.nio.file.FileStore,​java.lang.Long> totalToWrite)
        Checks if there is enough space on all file stores to write the given amount of data. The data to write should be the total amount, ongoing writes + new writes.

      • getAvailableSpaceForCompactions

        public static long getAvailableSpaceForCompactions​(java.nio.file.FileStore fileStore)

      • perFileStore

        public static java.util.Map<java.nio.file.FileStore,​java.lang.Long> perFileStore​(java.util.Map<File,​java.lang.Long> perDirectory,
                                                                                       java.util.function.Function<File,​java.nio.file.FileStore> filestoreMapper)

      • allFileStores

        public java.util.Set<java.nio.file.FileStore> allFileStores​(java.util.function.Function<File,​java.nio.file.FileStore> filestoreMapper)

      • getFileStore

        public static java.nio.file.FileStore getFileStore​(File directory)
        Gets the filestore for the actual directory where the sstables are stored. Handles the fact that an operator can symlink a table directory to a different filestore.

      • getSnapshotDirectory

        public static File getSnapshotDirectory​(Descriptor desc,
                                        java.lang.String snapshotName)

      • getSnapshotDirectory

        public static File getSnapshotDirectory​(File location,
                                        java.lang.String snapshotName)
        Returns directory to write snapshot. If directory does not exist, then one is created. If given location indicates secondary index, this will return <cf dir>/snapshots/<snapshot name>/.<index name>. Otherwise, this will return <cf dir>/snapshots/<snapshot name>.
        Parameters:
        location - base directory
        snapshotName - snapshot name
        Returns:
        directory to write snapshot

      • getSnapshotDirectoryIfExists

        public static java.util.Optional<File> getSnapshotDirectoryIfExists​(File location,
                                                                    java.lang.String snapshotName)
        Returns directory to write a snapshot to. If directory does not exist, then it is NOT created. If given location indicates secondary index, this will return <cf dir>/snapshots/<snapshot name>/.<index name>. Otherwise, this will return <cf dir>/snapshots/<snapshot name>.
        Parameters:
        location - base directory
        snapshotName - snapshot name
        Returns:
        directory to write a snapshot

      • getSnapshotManifestFile

        public File getSnapshotManifestFile​(java.lang.String snapshotName)

      • getSnapshotManifestFile

        public static File getSnapshotManifestFile​(File snapshotDir)

      • getSnapshotSchemaFile

        public File getSnapshotSchemaFile​(java.lang.String snapshotName)

      • getSnapshotSchemaFile

        public static File getSnapshotSchemaFile​(File snapshotDir)

      • getBackupsDirectory

        public static File getBackupsDirectory​(Descriptor desc)

      • getBackupsDirectory

        public static File getBackupsDirectory​(File location)
        Returns directory to write a backup to. If directory does not exist, then one is created. If given location indicates secondary index, this will return <cf dir>/backups/.<index name>. Otherwise, this will return <cf dir>/backups/.
        Parameters:
        location - base directory
        Returns:
        directory to write a backup

      • getBackupsDirectoryIfExists

        public static java.util.Optional<File> getBackupsDirectoryIfExists​(File location)
        Returns directory to write a backup to. If directory does not exist, then it is NOT created. If given location indicates secondary index, this will return <cf dir>/backups/.<index name>. Otherwise, this will return <cf dir>/backups/.
        Parameters:
        location - base directory
        Returns:
        directory to write a backup

      • isStoredInLocalSystemKeyspacesDataLocation

        public static boolean isStoredInLocalSystemKeyspacesDataLocation​(java.lang.String keyspace,
                                                                 java.lang.String table)
        Checks if the specified table should be stored with local system data.

        To minimize the risk of failures, SSTables for local system keyspaces must be stored in a single data directory. The only exception to this are some of the system table as the server can continue operating even if those tables loose some data.

        Parameters:
        keyspace - the keyspace name
        table - the table name
        Returns:
        true if the specified table should be stored with local system data, false otherwise.

      • listSnapshots

        public java.util.Map<java.lang.String,​TableSnapshot> listSnapshots()

      • maybeLoadManifest

        protected static SnapshotManifest maybeLoadManifest​(java.lang.String keyspace,
                                                    java.lang.String table,
                                                    java.lang.String tag,
                                                    java.util.Set<File> snapshotDirs)

      • listSnapshotDirsByTag

        protected java.util.Map<java.lang.String,​java.util.Set<File>> listSnapshotDirsByTag()

      • snapshotExists

        public boolean snapshotExists​(java.lang.String snapshotName)

      • clearSnapshot

        public static void clearSnapshot​(java.lang.String snapshotName,
                                 java.util.List<File> tableDirectories,
                                 com.google.common.util.concurrent.RateLimiter snapshotRateLimiter)

      • removeSnapshotDirectory

        public static void removeSnapshotDirectory​(com.google.common.util.concurrent.RateLimiter snapshotRateLimiter,
                                           File snapshotDir)

      • trueSnapshotsSize

        public long trueSnapshotsSize()
        Returns:
        total snapshot size in byte for all snapshots.

      • getRawDiretoriesSize

        public long getRawDiretoriesSize()
        Returns:
        Raw size on disk for all directories

      • getTrueAllocatedSizeIn

        public long getTrueAllocatedSizeIn​(File snapshotDir)

      • getKSChildDirectories

        public static java.util.List<File> getKSChildDirectories​(java.lang.String ksName)

      • isSecondaryIndexFolder

        public static boolean isSecondaryIndexFolder​(File dir)

      • isSecondaryIndexFolder

        public static boolean isSecondaryIndexFolder​(java.nio.file.Path dir)

      • getCFDirectories

        public java.util.List<File> getCFDirectories()

      • getUIDGenerator

        public <T extends SSTableId> java.util.function.Supplier<T> getUIDGenerator​(SSTableId.Builder<T> builder)
        Initializes the sstable unique identifier generator using a provided builder for this instance of directories. If the id builder needs that, sstables in these directories are listed to provide the existing identifiers to the builder. The listing is done lazily so if the builder does not require that, listing is skipped.

      • get

        public static java.util.Optional<File> get​(File base,
                                           java.lang.String... subdirs)