Re: [squid-users] anyone knows some info about youtube "range" parameter?

From: Ghassan Gharabli <sounarose_at_googlemail.com>
Date: Tue, 1 May 2012 13:41:00 +0300

Cached Contents are so many ....

-Youtube

-GoogleSyndication

-ADS and it will always give me a strong headache to follow every ADby
caching its content not to block it because blocking any ADS URL would
present a javascript error at the browser's side!.

-VideoZer

- android Google ( MARKET .. etc )

- Ziddu

- xHamster

- SoundCloud

- Some websites that have a cdn folder which generates a key or long
key in the middle of the URL which everytime you refresh the website
then it will generate a dynamic folder incase if it has an extension
jpg or any multimedia extension and sometimes you see URL which has an
end like that abc.jpg;blablabla or abc.jpg?blablabla so I try to
remove that CDN Folder and at the same time i also remove everything
which comes after the extension jpg if has ";" or "?" .

- wlxrs.com

- reverbnation , megavideo , xaxtube

- NOGOMI/ AHLASOOT .. for example to save bandwidth !!! . usually
before you download the MP3 file they allow you to listen online
through their web player which has the same file size and same
file_name but if you download the file then you get a different domain
name which made me think just to rewrite the URL which matches and
save the same URL into cache if a client wanted to listen or download
and also alot of websites has the same idea.

- vkontakte , depositfiles , eporner , 4shared , letitbit , sendspace
, filesonic , uploaded , uploading , turbobit, wupload , redtubefiles
, filehippo , oron , rapishare , tube8 , pornhub , xvideos , telstra ,
scribdassets , przeklej , hardsextube , fucktube , imageshack , beeg ,
yahoo videos , youjizz , gcdn

for example , look at this URL :
#http://b2htks6oia9cm5m4vthd6hhulo.gcdn.biz/d/r/*/FileName WIthout
Extension
(*) DYNAMIC CONTENT and just look into the sub domain how its like!

 and so many websites but unfotunately EASY-SHARE is using POST
response and I cant cache it.

Lately I was monitoring NOKIA Phones and I added OVI STORE.

Do you have any website that its not cacheable or using CDN or
something because Im really interested to look at :) .

Ghassan

On Mon, Apr 30, 2012 at 2:53 AM, Eliezer Croitoru <eliezer_at_ngtech.co.il> wrote:
> On 30/04/2012 02:18, Ghassan Gharabli wrote:
>>
>> Hello Eliezer,
>>
>> Are you trying to save all video chunks into same parts or capture /
>> download the whole video object through CURL or whatever! but i dont
>> think it should work since it will occure an error with the new
>> Youtube Player.
>>
>> What I have reached lately is saving same youtube video chunks  that
>> has  youtube 360p itag=34 ,480p itag=35 without saving its itag since
>> i want to save more bandwidth (thats why i only wrote scripts to you
>> as an example) which means if someone wants to watch 480p then he
>> would get the cached 360p contents thats why i didnt add the itag but
>> if he thought of watching 720p and above then another script would
>> catch it matching ITAG 37 , 22 ... I know that is not the best
>> solution but at least its working pretty well with no erros at all as
>> long as the client can always fast forward .
>>
>> Im using Squid 2.7 Stable9 compiled on windows 64-bit with PERL x64.
>>
>> Regarding the 302 Redirection  .. I have made sure to update the
>> source file client_side.c to fix the loop 302 Redirection but really I
>> dont have to worry now about anything so what is your target regarding
>> Youtube with argument Range and whats the problem till now ?
>>
>> I have RAID with 5 HDD and the average HTTP Requests per minute :
>> 2732.6 and because I want to save more bandwidth I try to analyze HTTP
>> Requests so i can always update my perl script to match most wanted
>> websites targetting Videos , Mp3 etc.
>>
>> For a second I thought of maybe someone would help to compile an
>> intelligent external helper script that would capture  the whole
>> byte-range and I know it is really hard to do that since we are
>> dealing with byte-range.
>>
>> I only have one question that is always teasing me .. what are the
>> comnparison between SQUID and BLUE COAT so is it because it is a
>> hardware perfromance or just it has more tricks to cache everything
>> and reach a maximum ratio ?
>>
>>
>>
>> Ghassan
>
> i was messing with store_url_rewrite and url_rewrite quite some time just
> for knowledge.
>
> i was researching every concept exists with squid until now.
> a while back (year or more)  i wrote store_url_rewrite using java and posted
> the code somewhere.
> the reason i was using java was because it's the fastest and simples from
> all other languages i know (ruby perl python).
> i was saving bandwidth using nginx because it was simple to setup.
> i dont really like the idea of faking my users about the quality and also it
> can make a very problematic state that the user can get partial HQ content
> what will break the user from watching videos.
>
> i dont really have any problems with the ranges.
> i just noticed that there are providers in my country that are not using the
> "range" parameter but the "begin" parameter...
>
> i will be very happy to get the client_side.c patch to fix the 302 loop.
>
> the problem with external_helper is that it's not really needed.
> if you need something you can use ICAP for that but you still there is a lot
> of work to be dont so as far it seems that store_url_rewrite is the best
> option.
>
> BLUECOAT has option to relate objects also using ETAG and other objects
> parameters so it makes it a very robust caching system.
>
> nginx does the work just fine for youtube videos but it seems like some
> headers are messing and should be added.
>
> by the way,
> What video\mp3 sites are you caching using your scripts?
>
>
> Eliezer
>
>
>>
>>
>>
>> On Mon, Apr 30, 2012 at 1:29 AM, Eliezer Croitoru<eliezer_at_ngtech.co.il>
>>  wrote:
>>>
>>> On 24/04/2012 21:02, Eliezer Croitoru wrote:
>>>>
>>>>
>>>> as for some people asking me recently about youtube cache i have checked
>>>> again and found that youtube changed their video uris and added an
>>>> argument called "range" that is managed by the youtube player.
>>>> the original url\uri dosnt include range but the youtube player is using
>>>> this argument to save bandwidth.
>>>>
>>>> i can implement the cahing with ranges on nginx but i dont know yet the
>>>> way that range works.
>>>> it can be based on user bandwidth or "fixed" size of chunkes.
>>>>
>>>> if someone up to the mission of analyzing it a bit more to understand it
>>>> so the "range" cache will be implemented i will be happy to get some
>>>> help with it.
>>>>
>>>> Thanks,
>>>> Eliezer
>>>>
>>>>
>>> as for now the "minimum_object_size 512 bytes" wont do the trick for 302
>>> redirection on squid2.7 because the 302 response is 963 big size.
>>> so i have used:
>>> minimum_object_size 1024 bytes
>>> just to make sure it will work.
>>> and also this is a youtube videos dedicated server so it's on with this
>>> limit.
>>>
>>> Regards,
>>>
>>> Eliezer
>>>
>>> --
>>> Eliezer Croitoru
>>> https://www1.ngtech.co.il
>>> IT consulting for Nonprofit organizations
>>> eliezer<at>  ngtech.co.il
>
>
>
> --
> Eliezer Croitoru
> https://www1.ngtech.co.il
> IT consulting for Nonprofit organizations
> eliezer <at> ngtech.co.il
Received on Tue May 01 2012 - 10:41:07 MDT

This archive was generated by hypermail 2.2.0 : Sun May 06 2012 - 12:00:03 MDT