Re: [squid-users] anyone knows some info about youtube "range" parameter?

well a nice list.
all the filesharing sites are not something i really want to cache
because the effort to understand most of the sites is too much then the
benefit.

megavideo got shutdown.

what i'm and others are very interested is microsoft windows updates.
they are using a 206 range request that make it's impossible to cache
with squid as far as i understand.

Eliezer

On 01/05/2012 13:41, Ghassan Gharabli wrote:
> Cached Contents are so many ....
>
> -Youtube
>
> -GoogleSyndication
>
> -ADS and it will always give me a strong headache to follow every ADby
> caching its content not to block it because blocking any ADS URL would
> present a javascript error at the browser's side!.
>
> -VideoZer
>
> - android Google ( MARKET .. etc )
>
> - Ziddu
>
> - xHamster
>
> - SoundCloud
>
> - Some websites that have a cdn folder which generates a key or long
> key in the middle of the URL which everytime you refresh the website
> then it will generate a dynamic folder incase if it has an extension
> jpg or any multimedia extension and sometimes you see URL which has an
> end like that abc.jpg;blablabla or abc.jpg?blablabla so I try to
> remove that CDN Folder and at the same time i also remove everything
> which comes after the extension jpg if has ";" or "?" .
>
> - wlxrs.com
>
> - reverbnation , megavideo , xaxtube
>
> - NOGOMI/ AHLASOOT .. for example to save bandwidth !!! . usually
> before you download the MP3 file they allow you to listen online
> through their web player which has the same file size and same
> file_name but if you download the file then you get a different domain
> name which made me think just to rewrite the URL which matches and
> save the same URL into cache if a client wanted to listen or download
> and also alot of websites has the same idea.
>
> - vkontakte , depositfiles , eporner , 4shared , letitbit , sendspace
> , filesonic , uploaded , uploading , turbobit, wupload , redtubefiles
> , filehippo , oron , rapishare , tube8 , pornhub , xvideos , telstra ,
> scribdassets , przeklej , hardsextube , fucktube , imageshack , beeg ,
> yahoo videos , youjizz , gcdn
>
> for example , look at this URL :
> #http://b2htks6oia9cm5m4vthd6hhulo.gcdn.biz/d/r/*/FileName WIthout
> Extension
> (*) DYNAMIC CONTENT and just look into the sub domain how its like!
>
> and so many websites but unfotunately EASY-SHARE is using POST
> response and I cant cache it.
>
> Lately I was monitoring NOKIA Phones and I added OVI STORE.
>
> Do you have any website that its not cacheable or using CDN or
> something because Im really interested to look at :) .
>
>
>
> Ghassan
>
> On Mon, Apr 30, 2012 at 2:53 AM, Eliezer Croitoru<eliezer_at_ngtech.co.il> wrote:
>> On 30/04/2012 02:18, Ghassan Gharabli wrote:
>>>
>>> Hello Eliezer,
>>>
>>> Are you trying to save all video chunks into same parts or capture /
>>> download the whole video object through CURL or whatever! but i dont
>>> think it should work since it will occure an error with the new
>>> Youtube Player.
>>>
>>> What I have reached lately is saving same youtube video chunks that
>>> has youtube 360p itag=34 ,480p itag=35 without saving its itag since
>>> i want to save more bandwidth (thats why i only wrote scripts to you
>>> as an example) which means if someone wants to watch 480p then he
>>> would get the cached 360p contents thats why i didnt add the itag but
>>> if he thought of watching 720p and above then another script would
>>> catch it matching ITAG 37 , 22 ... I know that is not the best
>>> solution but at least its working pretty well with no erros at all as
>>> long as the client can always fast forward .
>>>
>>> Im using Squid 2.7 Stable9 compiled on windows 64-bit with PERL x64.
>>>
>>> Regarding the 302 Redirection .. I have made sure to update the
>>> source file client_side.c to fix the loop 302 Redirection but really I
>>> dont have to worry now about anything so what is your target regarding
>>> Youtube with argument Range and whats the problem till now ?
>>>
>>> I have RAID with 5 HDD and the average HTTP Requests per minute :
>>> 2732.6 and because I want to save more bandwidth I try to analyze HTTP
>>> Requests so i can always update my perl script to match most wanted
>>> websites targetting Videos , Mp3 etc.
>>>
>>> For a second I thought of maybe someone would help to compile an
>>> intelligent external helper script that would capture the whole
>>> byte-range and I know it is really hard to do that since we are
>>> dealing with byte-range.
>>>
>>> I only have one question that is always teasing me .. what are the
>>> comnparison between SQUID and BLUE COAT so is it because it is a
>>> hardware perfromance or just it has more tricks to cache everything
>>> and reach a maximum ratio ?
>>>
>>>
>>>
>>> Ghassan
>>
>> i was messing with store_url_rewrite and url_rewrite quite some time just
>> for knowledge.
>>
>> i was researching every concept exists with squid until now.
>> a while back (year or more) i wrote store_url_rewrite using java and posted
>> the code somewhere.
>> the reason i was using java was because it's the fastest and simples from
>> all other languages i know (ruby perl python).
>> i was saving bandwidth using nginx because it was simple to setup.
>> i dont really like the idea of faking my users about the quality and also it
>> can make a very problematic state that the user can get partial HQ content
>> what will break the user from watching videos.
>>
>> i dont really have any problems with the ranges.
>> i just noticed that there are providers in my country that are not using the
>> "range" parameter but the "begin" parameter...
>>
>> i will be very happy to get the client_side.c patch to fix the 302 loop.
>>
>> the problem with external_helper is that it's not really needed.
>> if you need something you can use ICAP for that but you still there is a lot
>> of work to be dont so as far it seems that store_url_rewrite is the best
>> option.
>>
>> BLUECOAT has option to relate objects also using ETAG and other objects
>> parameters so it makes it a very robust caching system.
>>
>> nginx does the work just fine for youtube videos but it seems like some
>> headers are messing and should be added.
>>
>> by the way,
>> What video\mp3 sites are you caching using your scripts?
>>
>>
>> Eliezer
>>
>>
>>>
>>>
>>>
>>> On Mon, Apr 30, 2012 at 1:29 AM, Eliezer Croitoru<eliezer_at_ngtech.co.il>
>>> wrote:
>>>>
>>>> On 24/04/2012 21:02, Eliezer Croitoru wrote:
>>>>>
>>>>>
>>>>> as for some people asking me recently about youtube cache i have checked
>>>>> again and found that youtube changed their video uris and added an
>>>>> argument called "range" that is managed by the youtube player.
>>>>> the original url\uri dosnt include range but the youtube player is using
>>>>> this argument to save bandwidth.
>>>>>
>>>>> i can implement the cahing with ranges on nginx but i dont know yet the
>>>>> way that range works.
>>>>> it can be based on user bandwidth or "fixed" size of chunkes.
>>>>>
>>>>> if someone up to the mission of analyzing it a bit more to understand it
>>>>> so the "range" cache will be implemented i will be happy to get some
>>>>> help with it.
>>>>>
>>>>> Thanks,
>>>>> Eliezer
>>>>>
>>>>>
>>>> as for now the "minimum_object_size 512 bytes" wont do the trick for 302
>>>> redirection on squid2.7 because the 302 response is 963 big size.
>>>> so i have used:
>>>> minimum_object_size 1024 bytes
>>>> just to make sure it will work.
>>>> and also this is a youtube videos dedicated server so it's on with this
>>>> limit.
>>>>
>>>> Regards,
>>>>
>>>> Eliezer
>>>>
>>>> --
>>>> Eliezer Croitoru
>>>> https://www1.ngtech.co.il
>>>> IT consulting for Nonprofit organizations
>>>> eliezer<at> ngtech.co.il
>>
>>
>>
>> --
>> Eliezer Croitoru
>> https://www1.ngtech.co.il
>> IT consulting for Nonprofit organizations
>> eliezer<at> ngtech.co.il

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il