includes/filebackend/README

   1 /*!
   2 \ingroup FileBackend
   3 \page file_backend_design File backend design
   4
   5 Some notes on the FileBackend architecture.
   6
   7 \section intro Introduction
   8
   9 To abstract away the differences among different types of storage media,
  10 MediaWiki is providing an interface known as FileBackend. Any MediaWiki
  11 interaction with stored files should thus use a FileBackend object.
  12
  13 Different types of backing storage media are supported (ranging from local
  14 filesystem to distributed object stores). The types include:
  15
  16 * FSFileBackend (used for mounted filesystems)
  17 * SwiftFileBackend (used for Swift or Ceph Rados+RGW object stores)
  18 * FileBackendMultiWrite (useful for transitioning from one backend to another)
  19
  20 Configuration documentation for each type of backend is to be found in their
  21 __construct() inline documentation.
  22
  23
  24 \section setup Setup
  25
  26 File backends are registered in LocalSettings.php via the global variable
  27 $wgFileBackends. To access one of those defined backend, one would use
  28 FileBackendStore::get( <name> ) which will bring back a FileBackend object
  29 handle. Such handles are reused for any subsequent get() call (singleton
  30 paradigm). The FileBackends objects are caching request calls such as file stats,
  31 SHA1 requests or TCP connection handles.
  32
  33 \par Note:
  34 Some backends may require additional PHP extensions to be enabled or can rely on a
  35 MediaWiki extension. This is often the case when a FileBackend subclass makes use of an
  36 upstream client API for communicating with the backing store.
  37
  38
  39 \section fileoperations File operations
  40
  41 The MediaWiki FileBackend API supports various operations on either files or
  42 directories. See FileBackend.php for full documentation for each function.
  43
  44
  45 \subsection reading Reading
  46
  47 The following operations are supported for reading from a backend:
  48
  49 On files:
  50 * state a file for basic information (timestamp, size)
  51 * read a file into a string or  several files into a map of path names to strings
  52 * download a file or set of files to a temporary file (on a mounted file system)
  53 * get the SHA1 hash of a file
  54 * get various properties of a file (stat information, content time, mime information, ...)
  55
  56 On directories:
  57 * get a list of files directly under a directory
  58 * get a recursive list of files under a directory
  59 * get a list of directories directly under a directory
  60 * get a recursive list of directories under a directory
  61
  62 \par Note:
  63 Backend handles should return directory listings as iterators, all though in some cases
  64 they may just be simple arrays (which can still be iterated over). Iterators allow for callers to
  65 traverse a large number of file listings without consuming excessive RAM in the process. Either the
  66 memory consumed is flatly bounded (if the iterator does paging) or it is proportional to the depth
  67 of the portion of the directory tree being traversed (if the iterator works via recursion).
  68
  69
  70 \subsection writing Writing
  71
  72 The following operations are supported for writing or changing in the backend:
  73
  74 On files:
  75 * store (copying a mounted filesystem file into storage)
  76 * create (creating a file within storage from a string)
  77 * copy (within storage)
  78 * move (within storage)
  79 * delete (within storage)
  80 * lock/unlock (lock or unlock a file in storage)
  81
  82 The following operations are supported for writing directories in the backend:
  83 * prepare (create parent container and directories for a path)
  84 * secure (try to lock-down access to a container)
  85 * publish (try to reverse the effects of secure)
  86 * clean (remove empty containers or directories)
  87
  88
  89 \subsection invokingoperation Invoking an operation
  90
  91 Generally, callers should use doOperations() or doQuickOperations() when doing
  92 batches of changes, rather than making a suite of single operation calls. This
  93 makes the system tolerate high latency much better by pipelining operations
  94 when possible.
  95
  96 doOperations() should be used for working on important original data, i.e. when
  97 consistency is important. The former will only pipeline operations that do not
  98 depend on each other. It is best if the operations that do not depend on each
  99 other occur in consecutive groups. This function can also log file changes to
 100 a journal (see FileJournal), which can be used to sync two backend instances.
 101 One might use this function for user uploads of file for example.
 102
 103 doQuickOperations() is more geared toward ephemeral items that can be easily
 104 regenerated from original data. It will always pipeline without checking for
 105 dependencies within the operation batch. One might use this function for
 106 creating and purging generated thumbnails of original files for example.
 107
 108
 109 \section consistency Consistency
 110
 111 Not all backing stores are sequentially consistent by default. Various FileBackend functions
 112 offer a "latest" option that can be passed in to assure (or try to assure) that the latest
 113 version of the file is read. Some backing stores are consistent by default, but callers should
 114 always assume that without this option, stale data may be read. This is actually true for stores
 115 that have eventual consistency.
 116
 117 Note that file listing functions have no "latest" flag, and thus some systems may return stale
 118 data. Thus callers should avoid assuming that listings contain changes made my the current client
 119 or any other client from a very short time ago. For example, creating a file under a directory
 120 and then immediately doing a file listing operation on that directory may result in a listing
 121 that does not include that file.
 122
 123
 124 \section locking Locking
 125
 126 Locking is effective if and only if a proper lock manager is registered and is
 127 actually being used by the backend. Lock managers can be registered in LocalSettings.php
 128 using the $wgLockManagers global configuration variable.
 129
 130 For object stores, locking is not generally useful for avoiding partially
 131 written or read objects, since most stores use Multi Version Concurrency
 132 Control (MVCC) to avoid this. However, locking can be important when:
 133 * One or more operations must be done without objects changing in the meantime.
 134 * It can also be useful when a file read is used to determine a file write or DB change.
 135   For example, doOperations() first checks that there will be no "file already exists"
 136   or "file does not exist" type errors before attempted a given operation batch. This works
 137   by stating the files first, and is only safe if the files are locked in the meantime.
 138
 139 When locking, callers also should use the latest available file data for reads.
 140 Also, one should always lock the file *before* reading it, not after. If stale data is used
 141 to determine a write, there will be some data corruption, even when reads of the original file
 142 finally start returning the updated data without using the "latest" option (eventual consistency).
 143
 144 Since acquiring locks can fail, and lock managers can be non-blocking, callers should:
 145 * Acquire all required locks up font
 146 * Be prepared for the case where locks fail to be acquired
 147 * Possible retry acquiring certain locks
 148
 149 MVCC is also a useful pattern to use on top of the backend interface, because operations
 150 are not atomic, even with doOperations(), so doing complex batch file changes or changing files
 151 and updating a database row can result in partially written "transactions". One should avoid
 152 changing files once they have been stored, except perhaps with ephemeral data that are tolerant
 153 of some inconsistency.
 154
 155 Callers can use their own locking (e.g. SELECT FOR UPDATE) if it is more convenient, but note
 156 that all callers that change any of the files should then go through functions that acquire these
 157 locks. For example, if a caller just directly uses the file backend store() function, it will
 158 ignore any custom "FOR UPDATE" locks, which can cause problems.
 159
 160 \section objectstore Object stores
 161
 162 Support for object stores (like Amazon S3/Swift) drive much of the API and design
 163 decisions of FileBackend, but using any POSIX compliant file systems works fine.
 164 The system essentially stores "files" in "containers". For a mounted file
 165 system as a backing store, these will just be "files" under "directories". For
 166 an object store as a backing store, the "files" will be "objects" stored in
 167 "containers".
 168
 169
 170 \section file_obj_diffs File and Object store differences
 171
 172 An advantage of objects stores is the reduced Round-Trip Times. This is
 173 achieved by avoiding the need to create each parent directory before placing a
 174 file somewhere. It gets worse the deeper the directory hierarchy is. For both
 175 object stores and file systems, using "/" in filenames will allow for the
 176 intuitive use of directory functions. For example, creating a file in Swift
 177 called "container/a/b/file1" will mean that:
 178 - a "directory listing" of "container/a" will contain "b",
 179 - and a "file listing" of "b" will contain "file1"
 180
 181 This means that switching from an object store to a file system and vise versa
 182 using the FileBackend interface will generally be harmless. However, one must be
 183 aware of some important differences:
 184
 185 * In a filesystem, you cannot have a file and a directory within the same path
 186   whereas it is possible in an object stores. Calling code should avoid any layouts
 187   which allow files and directories at the same path.
 188 * Some file systems have file name length restrictions or overall path length
 189   restrictions that others do not. The same goes with object stores which might
 190   have a maximum object length or a limitation regarding the number of files
 191   under a container or volume.
 192 * Latency varies among systems, certain access patterns may not be tolerable for
 193   certain backends but may hold up for others. Some backend subclasses use
 194   MediaWiki's object caching for serving stat requests, which can greatly
 195   reduce latency. Making sure that the backend has pipelining (see the
 196   "parallelize" and "concurrency" settings) enabled can also mask latency in
 197   batch operation scenarios.
 198
 199 */