Chunked allocator evaluation

Ok guys,

Here's the latest from Andre's chunked allocator.
Id like to add this to the list of things to evaulate
(like we've all got time. heh.)

I'm going to be busy for the next week (until the 5th)
preparing to move back home so I'll be out of commissions
until then.

Adrian

----- Forwarded message from Andres Kroonmaa <andre@online.ee> -----

From: "Andres Kroonmaa" <andre@online.ee>
To: Adrian Chadd <adrian@squid-cache.org>
Date: Mon, 27 Aug 2001 10:44:14 +0200
Subject: Re: Release Schedule
X-mailer: Pegasus Mail for Win32 (v3.12c)

On 24 Aug 2001, at 16:18, Adrian Chadd <adrian@squid-cache.org> wrote:

> > Ok, see: http://www.online.ee/~andre/squid/chunked/
> >
> > There are 3 caches: cache1-80 and cache3-80 are production boxes I
> > mentioned. cache1-3000 is a stub cache feeded by Cidera. It has pretty
> > constant and small load, so it represents totally different usage
> > pattern. But its big.
> >
> > Basically I'd say that new malloc uses roughly 2-4 times less cpu,
> > and chunking avoids most of "unaccounted" ram used, which is mostly
> > in libmalloc overhead. About 20-30% of ram savings it seems.
>
> Right. How about publishing this information to the squid-dev list?

 no prob. Feel free to forward relevant parts. We've been through it
 once I think. But I believe that what we need is more dev'ers trying
 this code on their boxes. Then we'll have more to discuss.

> The Memory Allocation Statistics on the info pages are what..
> the non-mempools interface? There's an awful lot of allocations of
> small-sized chunks going on. Can't they be turned into mempools?

 On info pages its all together. All mempool allocs are also counted
 there. When mempools are created, stats there don't increase that fast
 any further. With chunked_mempools, only large allocs should be seen.

 What is heavily used after populating mempools is a problem.
 When I last tried to find them all, I noticed that they are mostly str
 ops, I think related to headerparser. Also there are few that do very
 shortliving allocs. But mostly this is variable-sized allocs, like URLs.

 I was thinking about moving str ops onto pools. But that isn't easy.
 There is so much assumptions built into squid that string can be
 freed with xfree at any time, even when creator is already unknown.
 This makes it very hard to use mempools... Probably we'd need to
 create our own strings library, imho. I also recall someone saying
 that future headerparsing code eliminates many of those allocs.

 Also, imo it would help alot if we could parse stack from xmalloc(),
 create a list of caller funcs, say upto 4 pointers deep, and count
 them as a single entity, perhaps even see on cachemgr. This would
 make it easy finding most often used mallocers. But thats not trivial.

------------------------------------
 Andres Kroonmaa <andre@online.ee>
 CTO, Microlink Online
 Tel: 6501 731, Fax: 6501 725
 Pärnu mnt. 158, Tallinn,
 11317 Estonia

----- End forwarded message -----
Received on Mon Aug 27 2001 - 06:15:35 MDT