Re: unlink bottleneck

map@iphil.net writes:

>Hello,
>
>Duane Wessels wrote:
>> #3 is easy and should work well. The store_clean.c routines
>> will clean up unused files so the swap directories dont grow to
>> huge lengths. truncate() may take just as long as unlink()
>> but it should win on later open() calls.
>
>When do we clean up the directories?

storeDirClean() gets called every 15 sec or so. If you have
the default 16*256 directories, it takes 17 hours to go through
all of them. Each call to storeDirClean() unlinks up to 10 of
the highest unused filenumbers.

>
>> #2 is pretty easy too. I have mostly working code from Mark Treacy
>> to already do it. I would probably add a FIFO of unused file numbers
>> as well so we can pass files to unlinkd without worry they will be
>> reused again quickly.
>
>FIFO the data structure or FIFO the IPC construct?

data structure. A stack not unlike the {get,put}_free_8k_page()
stuff. unused filenumbers would be placed on the stack.
We wouldn't actually unlink anything until the stack became "full."

>> #4 Stew's patch has been incorporated into Squid 1.2.x. His
>> implementation requires linking with a POSIX threads library. Maybe
>> not ultra portable?
>
>What's the schedule on 1.2? Is the code base diverging into
>1.1-stable and 1.2-beta?
>
>No, I think 1.1 should just be bug fixes and new stuff get into 1.2.
>Your thoughts?

I've been spending most of my time hacking on 1.1.9 recently. At this
point the diff file is quite large, so I'm considering NOT calling it
1.1.9 and instead putting all those changes into 1.2.alpha.

My guess is that it will be 3-4 months before something called 1.2.0 were
released.

Duane W.

FYI, here's my changelog since 1.1.8:

        - Fixed aclIpNetworkCompare for USE_SPLAY_TREE; was applying
          netmask to non-local variable through pointer. (Ansgar
          Hockmann).
        - Eliminated timeout/lifetime duality. Now we have only a
          single timeout parameter for network connections. The
          following heuristics are used:
          * for client connections, the timeout is set to
            'read_timeout' until the request headers are recieved, then
            the timeout is set to 'client_lifetime'.
          * For HTTP/FTP/Gopher/Wais/SSL/passthrough server
            connections, the timeout is extended by the 'read_timeout'
            after each read().
          * When the server side gets ahead of the client and we
            defer the read, the timeout is set to 'defer_timeout'.
        - Fixed ftpget to implement CWD1,CWD2,... as per section 3.2.2
          of RFC1738.
        - Fixed FTP to use the Authorization header only when the
          usernames match.
        - Removed redundant file:// --> ftp:// check (Willy TARREAU).
        - Added config option to disable 'client_db' statistics
          collection.
        - Added support for using ICMP data stuffed into ICP replies
          for peer selection algorithm.
        - Fixed situation where all requests would be handled by
          "single parent" and other peers would never be queried
          again.
        - Removed check for reply->code == 200 in icpGetHeadersForIMS().
        - Removed storeCheckExpired() from storeGetSwapSpace() as
          likely cause of slowing down cache when exceeding the high
          water mark.
        - Deny access to dangerous TCP ports (7, 9, 19).
        - Miscellaneous code cleanup, including removing a lot of
          unnecessary casts, and removing 'fd' args from proto.c
          functions.
        - Switched ping timeout to an "event" instead of an
          FD timeout.
        - Changed checkTimeouts() to get called every 15 seconds.
        - Fixed multicast ICP bug causing unnecessary timeouts.
        - Lowered 'store_objects_per_bucket' to 20 so that
          storeMaintainSwapSpace() runs frequently enough to
          keep up with incoming objects.
        - Added non-posix tempnam() replacement for NeXTStep
          (Gerben Wierda).
        - Fixed reconfigure (SIGHUP) to wait lifetime_shutdown
          seconds instead of happening immediately (Ron Gomes).
        - Fixed another continuation-lines-not-starting-with-NNN bug
          in ftpget.
        - Switched to table of regular expressions for choosing
          FTP and Gopher MIME types.
        - Fixed IP cache bug which caused 'dns_children 0' to break;
        - Fixed net_db design bug where _net_db_peer structures
          pointed to 'peer' structures that get freed and reallocated
          during a reconfigure.
Received on Tue Jul 29 2003 - 13:15:40 MDT