run "free -m" in shell and paste the output here
On Mon, Jul 7, 2014 at 5:46 AM, Martin Sperl <Martin.Sperl_at_amdocs.com> wrote:
> Hi!
>
> We have found out that since we moved from squid 3.3 to 3.4.3 (and the corresponding Vary) the memory utilization of squid has increased.
> It is 10GB right now 9.6G in memory, but we have only configured 4GB for in Memory caching.
>
> Here the ps line:
> squid 17695 10.9 61.2 10629864 10001520 ? Rl Apr21 12131:59 (squid-1) -f /opt/squid/squid.conf
>
> Any Ideas what that could be? See below the fine details from mgr:...
>
> BTW: I know it is not the latest version, but I am unsure if there have been any fixes for this behavior in a later version...
>
> Thanks,
> Martin
>
> P.s: the relevant config-section for memory settings:
>
> # define the memory cache as 4GB
> cache_mem 4096 MB
>
> # and in memory sizes for single objects:
> maximum_object_size 32 KB
> maximum_object_size_in_memory 32 KB
>
> # define a disk cache (preferably on ram-disk to avoid IOs)
> cache_dir aufs /var/spool/squid/ 8192 16 256
>
> here the infos from:
>
> Mgr:info:
> Squid Object Cache: Version 3.4.3
> Build Info:
> Start Time: Mon, 21 Apr 2014 08:31:27 GMT
> Current Time: Mon, 07 Jul 2014 08:21:22 GMT
> Connection information for squid:
> Number of clients accessing cache: 0
> Number of HTTP requests received: 998061101
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 9002.1
> Average ICP messages per minute since start: 0.0
> Select loop called: 7233613675 times, 0.920 ms avg
> Cache information for squid:
> Hits as % of all requests: 5min: 40.8%, 60min: 45.5%
> Hits as % of bytes sent: 5min: 50.7%, 60min: 51.6%
> Memory hits as % of hit requests: 5min: 72.6%, 60min: 74.9%
> Disk hits as % of hit requests: 5min: 1.8%, 60min: 2.0%
> Storage Swap size: 7549744 KB
> Storage Swap capacity: 90.0% used, 10.0% free
> Storage Mem size: 4153612 KB
> Storage Mem capacity: 99.0% used, 1.0% free
> Mean Object Size: 5.14 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.00179 0.00091
> Cache Misses: 0.27332 0.23230
> Cache Hits: 0.00000 0.00000
> Near Hits: 0.00286 0.00286
> Not-Modified Replies: 0.00000 0.00000
> DNS Lookups: 0.00094 0.00094
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 6652194.459 seconds
> CPU Time: 727888.282 seconds
> CPU Usage: 10.94%
> CPU Usage, 5 minute avg: 2.76%
> CPU Usage, 60 minute avg: 2.93%
> Process Data Segment Size via sbrk(): 10517816 KB
> Maximum Resident Size: 40036704 KB
> Page faults with physical i/o: 2100678
> Memory usage for squid via mallinfo():
> Total space in arena: -2064964 KB
> Ordinary blocks: 2047465 KB 161900 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 38432 KB 9 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 81875 KB
> Total in use: 81875 KB -4%
> Total free: 81875 KB -4%
> Total size: -2026532 KB
> Memory accounted for:
> Total accounted: 1269447 KB -63%
> memPool accounted: 9658055 KB -477%
> memPool unaccounted: -11684587 KB -0%
> memPoolAlloc calls: 507567155336
> memPoolFree calls: 508713005568
> File descriptor usage for squid:
> Maximum number of file descriptors: 65536
> Largest file desc currently in use: 1351
> Number of file desc currently in use: 249
> Files queued for open: 0
> Available number of file descriptors: 65287
> Reserved number of file descriptors: 100
> Store Disk files open: 0
> Internal Data Structures:
> 1477933 StoreEntries
> 851676 StoreEntries with MemObjects
> 851675 Hot Object Cache Items
> 1470120 on-disk objects
>
> Mgr:mem:
> Current memory usage:
> Pool Obj Size Chunks Allocated In UseIdle Allocations Saved Rate
> (bytes) KB/ch obj/ch (#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB) high (KB) high (hrs) %alloc (#) (KB) high (KB) (#) %cnt %vol (#)/sec
> mem_node 4136 1038636 4195116 4195359538.01 43.434 1038434 4194300 4195359 538.01 99.981 202 816 1564 695940913 0.137 1.2200.499
> 4K Buffer 4096 471762 1887048 1916096 61.2219.538 471750 1887000 1916096 61.22 99.997 12 48 3120 986312276 0.194 1.713 0.707
> Short Strings 40 20241644 790690 794615 61.01 8.18620241331 790677 794615 61.01 99.998 313 13 841 -2147483648 66.374 5.726 240.914
> HttpHeaderEntry 56 9903143 541579 544664 61.01 5.6079902997 541571 544664 61.01 99.999 146 8 551 -2147483648 16.522 1.996 60.070
> 2K Buffer 2048 238817 477634 724132 967.29 4.945 238800 477600 724132 967.29 99.993 17 34 546 -2147483648 1.566 6.917 5.673
> 16K Buffer 16384 25372 405952 1106704 1357.164.203 25355 405680 1106704 1357.16 99.933 17 272 4048 268150503 0.053 1.863 0.193
> HttpReply 280 851758 232903 233312 67.89 2.411 851732 232896 233312 67.89 99.997 26 8 138 -2147483648 0.688 0.415 2.495
> 8K Buffer 8192 27681 221448 275280 1357.54 2.29327679 221432 275280 1357.54 99.993 2 16 312 235405729 0.046 0.818 0.168
> MemObject 240 851740 199627 199973 67.89 2.067 851731 199625 199973 67.89 99.999 9 3 93 492884384 0.097 0.050 0.354
> StoreEntry 120 1477969 173200 173405 69.33 1.7931477949 173198 173405 69.33 99.999 20 3 32 448740915 0.088 0.023 0.322
> Medium Strings 128 1262162 157771 160057 61.01 1.6331262100 157763 160057 61.01 99.995 62 8 279 -2147483648 4.731 1.306 17.148
> cbdata BodyPipe (43) 152 765008 113556 114424 68.08 1.176 765004 113556 114424 68.08 99.999 4 1 34 358799947 0.071 0.023 0.257
> HttpHdrCc 96 769579 72149 72667 59.52 0.747 769566 72147 72667 59.52 99.998 13 2 34 1594117469 0.313 0.065 1.138
> LRU policy node 24 2321816 54418 54496 68.09 0.5632321811 54418 54496 68.09 100.000 5 1 12 159292188 0.031 0.002 0.117
> cbdata MemBuf (13) 64 851759 53235 53330 67.89 0.551 851734 53234 53330 67.89 99.997 25 2 33 -2147483648 0.923 0.127 3.346
> 32K Buffer 32768 1603 51296 111520 979.75 0.531 160051200 111520 979.75 99.813 3 96 736 20665690 0.004 0.287 0.015
> MD5 digest 16 1477969 23094 23121 69.33 0.2391477949 23093 23121 69.33 99.999 20 1 5 595307879 0.117 0.004 0.426
> Long Strings 512 6736 3368 7858 1063.01 0.0356699 3350 7858 1063.01 99.451 37 19 234 -2147483648 1.034 1.142 3.750
> 64K Buffer 65536 46 2944 33792 1711.70 0.0300 0 33792 1711.70 0.000 46 2944 32768 -2147483648 0.466 65.894 1.691
> acl_ip_data 96 5725 537 537 1848.01 0.0065725 537 537 1848.01 100.000 0 0 1 276 0.000 0.000 0.000
> cbdata IdleConnList (16) 4160 51 208 281 1757.200.002 45 183 281 1757.20 88.235 6 25 86 36686446 0.007 0.065 0.026
> cbdata clientReplyContext (23) 4320 31 131 1730 1293.490.001 4 17 1730 1293.49 12.903 27 114 1321 995516255 0.196 1.823 0.711
> cbdata ClientSocketContext (22) 4256 31 129 1705 1293.490.001 4 17 1705 1293.49 12.903 27 113 1301 995516208 0.196 1.796 0.711
> HttpRequest 1768 63 109 1428 1293.49 0.0017 13 1428 1293.49 11.111 56 97 992 -2147483648 0.827 3.153 2.998
> cbdata ConnStateData (20) 440 201 87 1259 390.81 0.001171 74 1259 390.81 85.075 30 13 343 528103233 0.104 0.099 0.381
> Comm::Connection 192 247 47 591 390.81 0.000 21741 591 390.81 87.854 30 6 171 829565512 0.163 0.068 0.596
> ipcache_entry 128 289 37 82 1252.79 0.000288 36 82 1252.79 99.654 1 1 49 821666 0.000 0.000 0.001
> cbdata idns_query (9) 8696 3 26 3661 1252.79 0.0000 0 3661 1252.79 0.000 3 26 3346 1134547 0.000 0.004 0.001
> Acl::AndNode 136 184 25 25 1848.01 0.000184 25 25 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> MimeEntry 128 177 23 23 1848.01 0.000177 23 23 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> fqdncache_entry 160 99 16 16 11.53 0.000 9916 16 11.53 100.000 0 0 1 607 0.000 0.000 0.000
> cbdata ClientHttpRequest (21) 408 31 13 164 1293.490.000 4 2 164 1293.49 12.903 27 11 125 995516208 0.196 0.172 0.711
> 4KB Strings 4096 3 12 100 136.37 0.000 00 100 136.37 0.000 3 12 100 4568442 0.001 0.008 0.003
> cbdata ServiceRep (15) 344 34 12 12 1848.01 0.00034 12 12 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata Tree (6) 192 52 10 10 1848.01 0.00052 10 10 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata ModXact (30) 496 17 9 127 1711.70 0.0000 0 127 1711.70 0.000 17 9 126 1177041017 0.231 0.247 0.839
> squidaio_small_bufs 4096 2 8 120 1065.11 0.0000 0 120 1065.11 0.000 2 8 120 60491040 0.012 0.105 0.043
> cbdata clientStreamNode (24) 128 62 8 103 1293.490.000 8 1 103 1293.49 12.903 54 7 79 1991032510 0.391 0.108 1.421
> cbdata OptXact (32) 312 19 6 11 885.88 0.000 00 11 885.88 0.000 19 6 11 7038032 0.001 0.001 0.005
> RegexList 88 59 6 6 1848.01 0.00059 6 6 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> ACLStrategised 136 38 6 6 1848.01 0.00038 6 6 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> ACLStrategised 136 37 5 5 1848.01 0.00037 5 5 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> HttpHdrScTarget 88 55 5 6 343.75 0.000 555 6 343.75 100.000 0 0 2 149 0.000 0.000 0.000
> cbdata Logfile (11) 4192 1 5 5 1848.01 0.0001 5 5 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata store_client (34) 160 23 4 117 1293.490.000 4 1 117 1293.49 17.391 19 3 76 1290240791 0.254 0.088 0.920
> 1KB Strings 1024 3 3 55 412.45 0.000 00 55 412.45 0.000 3 3 38 50912098 0.010 0.022 0.037
> cbdata HttpStateData (38) 304 9 3 115 1293.490.000 2 1 115 1293.49 22.222 7 3 76 190925179 0.038 0.025 0.137
> cbdata Iterator (28) 160 17 3 40 1711.70 0.0000 0 40 1711.70 0.000 17 3 40 1177188448 0.231 0.080 0.839
> ACLSourceIP 120 22 3 3 1848.01 0.00022 3 3 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata ModXactLauncher (29) 152 17 3 38 1711.700.000 0 0 38 1711.70 0.000 17 3 38 1177189213 0.231 0.076 0.839
> Acl::NotNode 136 18 3 3 1848.01 0.00018 3 3 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata AccessCheck (27) 144 17 3 41 1711.69 0.0000 0 41 1711.69 0.000 17 3 41 1177394228 0.231 0.072 0.839
> cbdata OptXactLauncher (31) 128 19 3 5 885.88 0.0000 0 5 885.88 0.000 19 3 5 7037587 0.001 0.000 0.005
> ev_entry 48 47 3 3 1831.15 0.00045 3 3 1831.15 95.745 2 1 2 33129146 0.007 0.001 0.024
> ACLHTTPHeaderData 48 40 2 2 1848.01 0.00040 2 2 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata ClientRequestContext (25) 104 18 2 30 1711.70 0.000 0 0 30 1711.70 0.000 18 2 30 988402315 0.194 0.044 0.705
> ACLRegexData 16 110 2 2 1848.01 0.000110 2 2 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata FwdState (36) 168 9 2 65 1293.49 0.0002 1 65 1293.49 22.222 7 2 42 191697095 0.038 0.014 0.137
> cbdata ACLFilledChecklist (26) 440 3 2 93 1252.790.000 1 1 93 1252.79 33.333 2 1 85 -2147483648 2.155 2.045 7.805
> cbdata ConnOpener (33) 136 6 1 27 1711.70 0.0000 0 27 1711.70 0.000 6 1 27 66249937 0.013 0.004 0.048
> cbdata PortCfg (2) 344 2 1 1 1848.01 0.0002 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> ACLStrategised 136 5 1 1 1848.01 0.0005 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> aio_thread 40 16 1 1 1847.93 0.00016 1 1 1847.93 100.000 0 0 0 0 0.000 0.000 0.000
> squidaio_micro_bufs 128 5 1 14 1065.11 0.0000 0 14 1065.11 0.000 5 1 14 176721246 0.035 0.010 0.126
> aio_ctrl 112 5 1 13 1065.11 0.0000 0 13 1065.11 0.000 5 1 13 237358432 0.047 0.011 0.169
> ACLStrategised 136 4 1 1 1848.01 0.0004 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata CbDataList (3) 40 13 1 1 1848.01 0.00013 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> aio_request 104 5 1 12 1065.11 0.0000 0 12 1065.11 0.000 5 1 12 237365801 0.047 0.010 0.169
> cbdata ErrorState (46) 240 2 1 3 1497.43 0.0000 0 3 1497.43 0.000 2 1 3 10391751 0.002 0.001 0.007
> ACLStrategised 136 3 1 1 1848.01 0.0003 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata CbDataList (1) 40 10 1 1 1848.01 0.00010 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata CbDataList (47) 96 4 1 4 1664.29 0.0000 0 4 1664.29 0.000 4 1 4 27953666 0.005 0.001 0.020
> acl_name_list 72 5 1 1 1848.01 0.0005 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> ACLDestinationIP 120 3 1 1 1848.01 0.0003 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata UFSStoreState (39) 176 2 1 13 1065.110.000 0 0 13 1065.11 0.000 2 1 13 112646684 0.022 0.008 0.080
> ACLDomainData 16 19 1 1 1848.01 0.00019 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata CbDataList (4) 64 4 1 1 1848.01 0.0004 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata ps_state (37) 248 1 1 1 1835.63 0.0000 0 1 1835.63 0.000 1 1 1 192773573 0.038 0.020 0.137
> ACLCertificateData 72 3 1 1 1848.01 0.0003 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata RemovalPolicy (7) 104 2 1 1 1848.010.000 2 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata StoreSearchHashIndex (18) 104 2 1 1 1848.01 0.000 1 1 1 1848.01 50.000 1 1 1 20 0.000 0.000 0.000
> cbdata TcpAcceptor (17) 88 2 1 1 1848.01 0.0002 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata DiskThreadsDiskFile (40) 88 2 1 7 1065.110.000 0 0 7 1065.11 0.000 2 1 7 112646684 0.022 0.004 0.080
> AclDenyInfoList 32 5 1 1 1848.01 0.0005 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> HttpHdrRange 32 5 1 3 339.96 0.000 21 3 339.96 40.000 3 1 1 1563142 0.000 0.000 0.001
> cbdata ReadRequest (41) 72 2 1 3 1065.11 0.0000 0 3 1065.11 0.000 2 1 3 60491040 0.012 0.002 0.043
> ACLStrategised 136 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> UFSStoreState::_queued_write 32 4 1 3 1428.470.000 0 0 3 1428.47 0.000 4 1 3 134435143 0.026 0.002 0.096
> HttpHdrContRange 24 5 1 2 1664.29 0.0000 0 2 1664.29 0.000 5 1 2 792627 0.000 0.000 0.001
> ACLStringData 16 6 1 1 1848.01 0.0006 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> ACLSslErrorData 16 6 1 1 1848.01 0.0006 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata generic_cbdata (35) 32 3 1 14 1252.790.000 0 0 14 1252.79 0.000 3 1 13 65234012 0.013 0.001 0.047
> UFSStoreState::_queued_read 40 2 1 2 1065.110.000 0 0 2 1065.11 0.000 2 1 2 46057286 0.009 0.001 0.033
> cbdata IoResult (42) 40 2 1 2 1065.11 0.0000 0 2 1065.11 0.000 2 1 2 60491040 0.012 0.001 0.043
> cbdata WriteRequest (44) 80 1 1 1 1847.930.000 0 0 1 1847.93 0.000 1 1 1 134753572 0.026 0.005 0.096
> HttpHdrRangeSpec 16 5 1 2 339.96 0.000 21 2 339.96 40.000 3 1 1 1563133 0.000 0.000 0.001
> ACLUserData 24 3 1 1 1848.01 0.0003 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> StoreSwapLogData 72 1 1 1 1848.01 0.0000 0 1 1848.01 0.000 1 1 1 129809816 0.026 0.004 0.092
> cbdata RemovalPurgeWalker (19) 72 1 1 1 1848.010.000 0 0 1 1848.01 0.000 1 1 1 7712019 0.002 0.000 0.006
> ACLMethodData 16 4 1 1 1848.01 0.0004 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> link_list 16 4 1 2 1065.11 0.0000 0 2 1065.11 0.000 4 1 2 180627075 0.036 0.001 0.129
> dwrite_q 48 1 1 1 1848.01 0.0000 0 1 1848.01 0.000 1 1 1 264625573 0.052 0.005 0.188
> ACLNoteData 40 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata CbDataList (5) 40 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> cbdata IoResult (45) 40 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 134753572 0.026 0.002 0.096
> ACLASN 16 2 1 1 1848.01 0.0002 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> ACLHierCodeData 32 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> ACLProtocolData 16 2 1 1 1848.01 0.0002 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> ACLTimeData 32 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> CacheDigest 32 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000
> wordlist 16 2 1 1 1848.01 0.0002 1 1 1848.01 100.000 0 0 1 169 0.000 0.000 0.000
> StoreMetaMD5 32 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 110536338 0.022 0.001 0.079
> StoreMetaSTDLFS 32 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 110536338 0.022 0.001 0.079
> StoreMetaURL 32 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 110536338 0.022 0.001 0.079
> StoreMetaObjSize 32 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 110535723 0.022 0.001 0.079
> StoreMetaVary 32 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 6413952 0.001 0.000 0.005
> FwdServer 24 1 1 1 1835.63 0.0000 0 1 1835.63 0.000 1 1 1 192773573 0.038 0.002 0.137
> 16KB Strings 16384 0 0 32 853.17 0.000 00 32 853.17 -1.000 0 0 32 0 0.000 0.000 0.000
> cbdata nsvc (8) 72 0 0 1 1819.31 0.0000 0 1 1819.31 -1.000 0 0 1 0 0.000 0.000 0.000
> cbdata RebuildState (12) 688 0 0 1 1848.010.000 0 0 1 1848.01 -1.000 0 0 1 0 0.000 0.000 0.000
> cbdata RemovalPolicyWalker (48) 56 0 0 1 1846.340.000 0 0 1 1846.34 -1.000 0 0 1 0 0.000 0.000 0.000
> cbdata BodySink (49) 80 0 0 1 1844.65 0.0000 0 1 1844.65 -1.000 0 0 1 549 0.000 0.000 0.000
> HttpHdrSc 16 0 0 1 343.78 0.000 00 1 343.78 -1.000 0 0 1 82 0.000 0.000 0.000
> Total 1 42593287 9658537 971909061.01 100.000 42591806 9653779 9713153 61.04 99.951 1481 4758 37935 -2147483648 99.766 99.807 362.183
> Cumulative allocated volume: 235.8852 TB
> Current overhead: 34522 bytes (0.000%)
> Idle pool limit: 5.00 MB
> Total Pools created: 134
> Pools ever used: 123 (shown above)
> Currently in use: 78
> String Pool Impact
> (%strings) (%volume)
> Short Strings 94 83
> Medium Strings 6 17
> Long Strings 0 0
> 1KB Strings 0 0
> 4KB Strings 0 0
> 16KB Strings 0 0
> Other Strings 0 0
>
> Large buffers: 0 (0 KB)
>
>
>
> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
> you may review at http://www.amdocs.com/email_disclaimer.asp
Received on Mon Jul 07 2014 - 11:54:13 MDT
This archive was generated by hypermail 2.2.0 : Mon Jul 07 2014 - 12:00:04 MDT