Store Namespace Reference

Classes

class  Controlled
 
class  Controller
 
class  Disk
 manages a single cache_dir More...
 
class  DiskConfig
 
class  Disks
 summary view of all disk caches (cache_dirs) combined More...
 
class  EntryGuard
 
class  LocalSearch
 iterates local store_table More...
 
class  ParsingBuffer
 
class  Storage
 
class  SwapMetaIterator
 iterates serialized swap meta fields loaded into a given buffer More...
 
class  SwapMetaUnpacker
 Store entry metadata view providing a for-range loop meta field iterator API. More...
 
class  SwapMetaView
 a swap metadata field inside the buffer given to SwapMetaUnpacker More...
 

Typedefs

typedef ::StoreEntry Entry
 
typedef ::MemStore Memory
 
typedef ::Transients Transients
 
using RawSwapMetaType = char
 
using RawSwapMetaLength = int
 
using RawSwapMetaPrefixLength = int
 

Enumerations

enum  IoStatus {
  ioUndecided ,
  ioWriting ,
  ioReading ,
  ioDone
}
 cache "I/O" direction and status More...
 
enum  SwapMetaType {
  STORE_META_VOID = 0 ,
  STORE_META_KEY_MD5 = 3 ,
  STORE_META_URL = 4 ,
  STORE_META_STD = 5 ,
  STORE_META_VARY_HEADERS = 8 ,
  STORE_META_STD_LFS = 9 ,
  STORE_META_OBJSIZE = 10
}
 

Functions

void Stats (StoreEntry *output)
 
void Maintain (void *unused)
 
ControllerRoot ()
 safely access controller singleton More...
 
void Init (Controller *root=nullptr)
 initialize the storage module; a custom root is used by unit tests only More...
 
void FreeMemory ()
 undo Init() More...
 
StoreSearchNewLocalSearch ()
 
std::ostream & operator<< (std::ostream &os, const ParsingBuffer &b)
 
constexpr RawSwapMetaType RawSwapMetaTypeTop ()
 
constexpr bool DeprecatedSwapMetaType (const RawSwapMetaType type)
 
constexpr bool ReservedSwapMetaType (const RawSwapMetaType type)
 
constexpr bool HonoredSwapMetaType (const RawSwapMetaType type)
 
constexpr bool IgnoredSwapMetaType (const RawSwapMetaType type)
 
void CheckSwapMetaSerialization (RawSwapMetaType, RawSwapMetaLength, const void *)
 
static void CheckSwapMetaKey (const SwapMetaView &meta, const StoreEntry &entry)
 validates serialized STORE_META_KEY_MD5 swap meta field More...
 
static void UnpackSwapMetaKey (const SwapMetaView &meta, cache_key *key)
 deserializes STORE_META_KEY_MD5 swap meta field More...
 
static void CheckSwapMetaUrl (const SwapMetaView &meta, const StoreEntry &entry)
 validates serialized STORE_META_URL swap meta field More...
 
static SBuf UnpackNewSwapMetaVaryHeaders (const SwapMetaView &meta, const StoreEntry &entry)
 deserializes STORE_META_VARY_HEADERS swap meta field More...
 
static size_t UnpackPrefix (const char *const buf, const size_t size)
 
size_t UnpackSwapMetaSize (const SBuf &)
 
size_t UnpackIndexSwapMeta (const MemBuf &, StoreEntry &, cache_key *)
 
void UnpackHitSwapMeta (char const *, ssize_t, StoreEntry &)
 deserializes entry metadata from the given buffer into the cache hit entry More...
 
static void PackField (std::ostream &os, const SwapMetaType type, const size_t length, const void *value)
 writes a single swap meta field to the given stream More...
 
static void PackFields (const StoreEntry &entry, std::ostream &os)
 writes all swap meta fields of the given Store entry to the given stream More...
 
AllocedBuf PackSwapMeta (const StoreEntry &, size_t &size)
 
static void HandleBadRawType (const RawSwapMetaType type)
 properly reports or rejects a problematic raw swap meta field type More...
 
template<typename Item >
void SwapMetaExtract (Item &item, const char *&input, const void *end)
 
std::ostream & operator<< (std::ostream &, const SwapMetaView &)
 writes a short human-readable summary of the given SwapMetaView object More...
 
static void checkIgnorableSwapMetaRawType (const RawSwapMetaType rawType)
 check rawType that may be ignored More...
 
static void checkTooSmallSwapMetaRawType (const RawSwapMetaType rawType)
 check a raw swap meta field type below SwapMetaType range or STORE_META_VOID More...
 
static void checkKnownSwapMetaRawType (const RawSwapMetaType rawType)
 check a raw swap meta field type within SwapMetaType range, excluding STORE_META_VOID More...
 
static void checkTooBigSwapMetaRawType (const RawSwapMetaType rawType)
 check a raw swap meta field type exceeding RawSwapMetaTypeTop() More...
 
static void checkSwapMetaRawType (const RawSwapMetaType rawType)
 check a given raw swap meta field type More...
 

Variables

static RefCount< ControllerTheRoot
 
const size_t SwapMetaFieldValueLengthMax = 64*1024
 
const char SwapMetaMagic = 0x03
 the start of the swap meta section More...
 
const auto SwapMetaPrefixSize = sizeof(SwapMetaMagic) + sizeof(RawSwapMetaPrefixLength)
 The size of the initial (and required) portion of any swap metadata. More...
 
const RawSwapMetaType RawSwapMetaTypeBottom = 0
 
const auto STORE_HDR_METASIZE
 
const auto STORE_HDR_METASIZE_OLD
 
void Controller::create() STUB void Controller Controller nil
 

Typedef Documentation

◆ Entry

Definition at line 51 of file forward.h.

◆ Memory

Definition at line 52 of file forward.h.

◆ RawSwapMetaLength

using Store::RawSwapMetaLength = typedef int

The type of a serialized swap meta field part called "length" (i.e. L in TLV). Valid values of this type do not include the size of T and L components. Low-level serialization code aside, we use size_t for swap meta field sizes.

Definition at line 100 of file SwapMeta.h.

◆ RawSwapMetaPrefixLength

The type of the serialized "metadata size" field that follows SwapMetaMagic. Together, the two fields form a swap metadata "prefix". The meaning of this size field is different from RawSwapMetaLength! Valid values of this field include the prefix size itself.

Definition at line 119 of file SwapMeta.h.

◆ RawSwapMetaType

using Store::RawSwapMetaType = typedef char

The type of a serialized swap meta field part called "type" (i.e. T in TLV). Meant for storing the serialized version of SwapMetaType.

Definition at line 95 of file SwapMeta.h.

◆ Transients

Definition at line 53 of file forward.h.

Enumeration Type Documentation

◆ IoStatus

Enumerator
ioUndecided 
ioWriting 
ioReading 
ioDone 

Definition at line 40 of file forward.h.

◆ SwapMetaType

"Swap meta" (a.k.a. "swap header") is Store entry metadata stored at the beginning of each cache_dir entry. This entry-dependent metadata typically includes entry cache key, URL, size, and various timestamps. Copies of this information may also be present in Squid process memory, readily accessible via StoreEntry and MemObject pointers, but swap metadata is about cached entry information stored in serialized form meant to cross process and instance boundaries.

Layout of swap metadata (for a single Store entry) in pseudo code: struct SwapMeta { struct Prefix { char magic; // see SwapMetaMagic int swap_hdr_sz; // total SwapMeta size: prefix and all fields }; Prefix prefix;

struct TLV { char type; // value meaning (and format); see SwapMetaType int length; // value length in octets char value[length]; // type-specific storage; exactly length bytes }; TLV fields[]; // as many swap meta fields as swap_hdr_sz accommodates

// XXX: int fields above should have been using a fixed-size type. };

Stored response (e.g., HTTP headers and body) follows swap metadata. Identifies the meaning (and associated format) of a single swap meta field. This enumeration only contains identifiers used by the current code. The gaps in the enum item values below represent deprecated/reserved IDs.

See also
DeprecatedSwapMetaType(), ReservedSwapMetaType()
Enumerator
STORE_META_VOID 

An invalid swap metadata field or field with an unknown meaning. Never used by swapout code.

STORE_META_KEY_MD5 

This represents the MD5 cache key that Squid currently uses. When Squid opens a disk file for reading, it can check that this MD5 matches the MD5 of the user's request. If not, then something went wrong and this is probably the wrong object.

STORE_META_URL 

The object's URL. This also may be matched against a user's request for cache hits to make sure we got the right object.

STORE_META_STD 

This is the "standard metadata" for an object. Really its just this middle chunk of the StoreEntry structure:

time_t timestamp;
time_t lastref;
time_t expires;
time_t lastmod;
uint64_t swap_file_sz;
uint16_t refcount;
uint16_t flags;
STORE_META_VARY_HEADERS 

Stores Vary request headers.

STORE_META_STD_LFS 

Modern STORE_META_STD version, with 64-bit swap_file_sz supporting objects larger than 2GB.

STORE_META_OBJSIZE 

Definition at line 52 of file SwapMeta.h.

Function Documentation

◆ checkIgnorableSwapMetaRawType()

static void Store::checkIgnorableSwapMetaRawType ( const RawSwapMetaType  rawType)
static

◆ checkKnownSwapMetaRawType()

static void Store::checkKnownSwapMetaRawType ( const RawSwapMetaType  rawType)
static

◆ CheckSwapMetaKey()

◆ checkSwapMetaRawType()

static void Store::checkSwapMetaRawType ( const RawSwapMetaType  rawType)
static

◆ CheckSwapMetaSerialization()

void Store::CheckSwapMetaSerialization ( RawSwapMetaType  type,
RawSwapMetaLength  length,
const void *  value 
)

Ensures that the given serialized swap meta field is valid and can be subsequently de-serialized (by the same code). Also detects some failures to update one of the classification functions above when editing SwapMetaType.

Definition at line 13 of file SwapMeta.cc.

References assert, HonoredSwapMetaType(), and SwapMetaFieldValueLengthMax.

Referenced by PackField().

◆ CheckSwapMetaUrl()

static void Store::CheckSwapMetaUrl ( const SwapMetaView meta,
const StoreEntry entry 
)
static

◆ checkTooBigSwapMetaRawType()

static void Store::checkTooBigSwapMetaRawType ( const RawSwapMetaType  rawType)
static

Definition at line 169 of file testStore.cc.

References checkIgnorableSwapMetaRawType(), and HonoredSwapMetaType().

Referenced by checkSwapMetaRawType().

◆ checkTooSmallSwapMetaRawType()

static void Store::checkTooSmallSwapMetaRawType ( const RawSwapMetaType  rawType)
static

◆ DeprecatedSwapMetaType()

constexpr bool Store::DeprecatedSwapMetaType ( const RawSwapMetaType  type)
inlineconstexpr

Whether the given raw swap meta field type represents a type that we should inform the admin about (if found in a store) but can otherwise ignore.

Deprecated:
Using URL as the cache key, as in Squid-1.1.
Deprecated:
Using SHA (secure hash algorithm) as a cache key
Deprecated:
hit-metering (RFC 2227)
Deprecated:

Definition at line 158 of file SwapMeta.h.

Referenced by checkIgnorableSwapMetaRawType(), checkTooSmallSwapMetaRawType(), HandleBadRawType(), and IgnoredSwapMetaType().

◆ FreeMemory()

◆ HandleBadRawType()

static void Store::HandleBadRawType ( const RawSwapMetaType  type)
static

◆ HonoredSwapMetaType()

constexpr bool Store::HonoredSwapMetaType ( const RawSwapMetaType  type)
inlineconstexpr

Whether we store the given swap meta field type (and also interpret the corresponding swap meta field when the Store loads it). Matches all SwapMetaType enum values except for the never-stored STORE_META_VOID.

Definition at line 197 of file SwapMeta.h.

References STORE_META_KEY_MD5, STORE_META_OBJSIZE, STORE_META_STD, STORE_META_STD_LFS, STORE_META_URL, STORE_META_VARY_HEADERS, and STORE_META_VOID.

Referenced by Store::SwapMetaView::SwapMetaView(), checkKnownSwapMetaRawType(), CheckSwapMetaSerialization(), checkTooBigSwapMetaRawType(), checkTooSmallSwapMetaRawType(), and TestStore::testSwapMetaTypeClassification().

◆ IgnoredSwapMetaType()

constexpr bool Store::IgnoredSwapMetaType ( const RawSwapMetaType  type)
inlineconstexpr

Whether the given raw swap meta field type can be safely ignored.

See also
HonoredSwapMetaType()

Definition at line 219 of file SwapMeta.h.

References DeprecatedSwapMetaType(), and ReservedSwapMetaType().

Referenced by checkIgnorableSwapMetaRawType(), checkKnownSwapMetaRawType(), and checkTooSmallSwapMetaRawType().

◆ Init()

◆ Maintain()

void Store::Maintain ( void *  unused)

Definition at line 1118 of file store.cc.

References eventAdd(), Store::Controller::maintain(), Maintain(), and Root().

Referenced by mainInitialize(), and Maintain().

◆ NewLocalSearch()

StoreSearch * Store::NewLocalSearch ( )

Definition at line 44 of file LocalSearch.cc.

Referenced by Store::Controller::search(), and storeDirSwapLog().

◆ operator<<() [1/2]

std::ostream & Store::operator<< ( std::ostream &  os,
const SwapMetaView meta 
)

◆ operator<<() [2/2]

std::ostream & Store::operator<< ( std::ostream &  os,
const ParsingBuffer b 
)
inline

Definition at line 118 of file ParsingBuffer.h.

References Store::ParsingBuffer::print().

◆ PackField()

static void Store::PackField ( std::ostream &  os,
const SwapMetaType  type,
const size_t  length,
const void *  value 
)
static

Definition at line 22 of file SwapMetaOut.cc.

References CheckSwapMetaSerialization(), Here, and SwapMetaFieldValueLengthMax.

Referenced by PackFields().

◆ PackFields()

◆ PackSwapMeta()

AllocedBuf Store::PackSwapMeta ( const StoreEntry entry,
size_t size 
)

swap metadata prefix and all swap metadata fields of the given entry

Parameters
sizegets filled with the total swap metadata size

Definition at line 77 of file SwapMetaOut.cc.

References Assure, SBufStream::buf(), PackFields(), SwapMetaMagic, and xmalloc.

Referenced by CtoCpp1(), and StoreEntry::getSerialisedMetaData().

◆ RawSwapMetaTypeTop()

constexpr RawSwapMetaType Store::RawSwapMetaTypeTop ( )
inlineconstexpr

Maximum value of a serialized SwapMetaType ID. This is not the largest RawSwapMetaType value (that is usually +127).

Definition at line 131 of file SwapMeta.h.

References STORE_META_KEY_MD5, STORE_META_OBJSIZE, STORE_META_STD, STORE_META_STD_LFS, STORE_META_URL, STORE_META_VARY_HEADERS, and STORE_META_VOID.

Referenced by checkSwapMetaRawType(), HandleBadRawType(), and TestStore::testSwapMetaTypeClassification().

◆ ReservedSwapMetaType()

constexpr bool Store::ReservedSwapMetaType ( const RawSwapMetaType  type)
inlineconstexpr

Whether the given raw swap meta field type represents a type that we should ignore without informing the admin.

the Store-ID url, if different from the normal URL

unique ID linking variants

Definition at line 180 of file SwapMeta.h.

Referenced by checkIgnorableSwapMetaRawType(), checkTooSmallSwapMetaRawType(), HandleBadRawType(), and IgnoredSwapMetaType().

◆ Root()

◆ Stats()

◆ SwapMetaExtract()

template<typename Item >
void Store::SwapMetaExtract ( Item &  item,
const char *&  input,
const void *  end 
)

a helper function to safely copy raw end-bounded serialized input into the given item and advance that input to the next item

Definition at line 68 of file SwapMetaView.h.

References Here.

Referenced by Store::SwapMetaView::SwapMetaView(), and UnpackPrefix().

◆ UnpackHitSwapMeta()

◆ UnpackIndexSwapMeta()

size_t Store::UnpackIndexSwapMeta ( const MemBuf buf,
StoreEntry tmpe,
cache_key key 
)

◆ UnpackNewSwapMetaVaryHeaders()

static SBuf Store::UnpackNewSwapMetaVaryHeaders ( const SwapMetaView meta,
const StoreEntry entry 
)
static

◆ UnpackPrefix()

static size_t Store::UnpackPrefix ( const char *const  buf,
const size_t  size 
)
static

deserializes entry metadata size from the given buffer

Return values
totalswap metadata size (a.k.a. swap_hdr_sz)

Definition at line 151 of file SwapMetaIn.cc.

References Assure, Here, Less(), size, SwapMetaExtract(), SwapMetaMagic, and SwapMetaPrefixSize.

Referenced by Store::SwapMetaUnpacker::SwapMetaUnpacker(), and UnpackSwapMetaSize().

◆ UnpackSwapMetaKey()

static void Store::UnpackSwapMetaKey ( const SwapMetaView meta,
cache_key key 
)
static

◆ UnpackSwapMetaSize()

size_t Store::UnpackSwapMetaSize ( const SBuf buf)

deserializes entry metadata size from the given buffer

Return values
totalswap metadata size (a.k.a. swap_hdr_len)

Definition at line 237 of file SwapMetaIn.cc.

References SBuf::length(), SBuf::rawContent(), and UnpackPrefix().

Referenced by Rock::HeaderUpdater::parseReadBytes().

Variable Documentation

◆ nil

◆ RawSwapMetaTypeBottom

const RawSwapMetaType Store::RawSwapMetaTypeBottom = 0

SwapMetaType IDs will never have this or smaller serialized value. This is not the smallest RawSwapMetaType value (that is usually -128).

Definition at line 126 of file SwapMeta.h.

Referenced by checkSwapMetaRawType(), and HandleBadRawType().

◆ STORE_HDR_METASIZE

const auto Store::STORE_HDR_METASIZE
Initial value:
=
4*sizeof(time_t) + sizeof(uint64_t) + 2*sizeof(uint16_t)

Expected size of a STORE_META_STD_LFS swap meta field. XXX: The actual size is environment-specific due to 4 parts that do not use fixed-size types.

Definition at line 226 of file SwapMeta.h.

Referenced by PackFields(), and UnpackIndexSwapMeta().

◆ STORE_HDR_METASIZE_OLD

const auto Store::STORE_HDR_METASIZE_OLD
Initial value:
=
4*sizeof(time_t) + sizeof(size_t) + 2*sizeof(uint16_t)

Expected size of a STORE_META_STD swap meta field. XXX: The actual size is environment-specific due to 5 parts that do not use fixed-size types.

Definition at line 231 of file SwapMeta.h.

Referenced by UnpackIndexSwapMeta().

◆ SwapMetaFieldValueLengthMax

const size_t Store::SwapMetaFieldValueLengthMax = 64*1024

Store entries with larger (serialized) swap metadata field values are not swapped out and are considered invalid when validating being-loaded entries. This arbitrary limit protects code that adds individual swap metadata field sizes from overflowing and prevents allocation of huge buffers when loading variable-length fields. Reevaluate this limit when increasing MAX_URL.

Definition at line 107 of file SwapMeta.h.

Referenced by Store::SwapMetaView::SwapMetaView(), CheckSwapMetaSerialization(), and PackField().

◆ SwapMetaMagic

const char Store::SwapMetaMagic = 0x03

Definition at line 113 of file SwapMeta.h.

Referenced by PackSwapMeta(), and UnpackPrefix().

◆ SwapMetaPrefixSize

const auto Store::SwapMetaPrefixSize = sizeof(SwapMetaMagic) + sizeof(RawSwapMetaPrefixLength)

Definition at line 122 of file SwapMeta.h.

Referenced by Store::SwapMetaUnpacker::SwapMetaUnpacker(), and UnpackPrefix().

◆ TheRoot

RefCount<Controller> Store::TheRoot
static

Definition at line 934 of file Controller.cc.

Referenced by FreeMemory(), Init(), and Root().

 

Introduction

Documentation

Support

Miscellaneous

Web Site Translations

Mirrors