Caching Dynamic Content dan Youtube Menggunakan Squid

29 Mei 2012 Fajar R. Squid Cache 103

Terkadang beberapa clients secara bersamaan menonton video youtube yang sama – dan hal ini berimbas pada penggunaan bandwidth yang besar. Dengan menggunakan squid, kita dapat melakukan efisiensi penggunaan bandwidth dengan melakukan dynamic content caching – termasuk didalamnya melakukan caching video youtube.

Pertanyaan yang sering muncul : pilih lusca atau squid?

Lusca di kembangkan berdasarkan squid 2.7, dan memiliki opsi tuning lebih banyak. Lusca dapat di implementasikan lebih baik dalam hal melakukan video caching jika dibandingkan dengan squid versi lainnya.

Kekurangannya adalah, lusca di kembangkan secara pribadi (oleh Adrian Chadd) dan menggunakan dasar squid 2.7. Squid 2.7 sendiri sudah “agak” lama tidak didevelop oleh developer squid. Squid 2.7 dinyatakan obsolete dan tidak didukung lagi jika squid 3.2 secara resmi dirilis.

Apa saja yang bisa di cache oleh lusca?

Ada banyak, seperti windows update, antivirus updates, video streaming seperti pada Youtube, Tube8 dan tube-tube lainnya, Facebook, mp3, mp4 dan sebagainya.

Lusca dapat dengan mudah di instal pada mesin berbasis *NIX, seperti Linux (Ubuntu dan CentOS sangat saya rekomendasikan), FreeBSD, OpenBSD, dan lain-lainnya.

Walaupun demikian, tutorial ini dapat di implementasikan pada squid2.7STABLE9 dan LUSCA

Catatan – Read me!! 

Pada posting ini youtube yang akan di rewrite adalah youtube range (video youtube yang di split menjadi beberapa bagian yang ukurannya bervariasi antara 1 – 15 Mb).

Jika anda ingin me-rewrite youtube non-range (alias video youtube di cache secara full / bukan di split atau dipisah menjadi beberapa bagian), silakan membaca posting saya yang berjudul “Squid Caching Youtube Non-Range“.

Pada update youtube terbaru, tampaknya mustahil untuk “memaksa” agar video yang di play menjadi non-range :(

Bagaimana melakukan caching youtube dengan Lusca / squid2.7STABLE9?

Saya berasumsi Lusca/Squid sudah terinstal dengan baik dimesin anda. Jika anda belum menginstal Lusca/Squid atau kebingungan melakukan instalasinya, silakan bertanya kepada mbah Google. :)

Pertama, carilah tag konfigurasi seperti dibawah ini pada squid.conf, kemudian hapus:

acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY

Kemudian, tambahkan konfigurasi berikut ini (sesuaikan dengan konfigurasi anda!) :

# allow cache untuk youtube
acl youtube dstdomain .youtube.com
cache allow youtube
cache allow all

# max. object 128 Mb / bisa disesuaikan
minimum_object_size 512 bytes
maximum_object_size 128 Mb

# acl untuk URL youtube yang dikirim ke storeurl
acl youtube url_regex -i youtube.*(ptracking|stream_204|player_204|gen_204).*$
acl youtube url_regex -i \.c\.(youtube|google)\.com\/(get_video|videoplayback|videoplay).*$
storeurl_access allow youtube

# acl tambahan untuk storeurl - 1 domain perbaris :)
# bisa di jadiin 1 baris, tapi kata temen ane yg bule, biar squid lebih
# efisien parsing config, sebaiknya dijadiin beberapa baris.. (bener nggak ya?)
acl storeurl_domain dstdomain .fbcdn.net
acl storeurl_domain dstdomain .dailymotion.com 
acl storeurl_domain dstdomain .metacafe.com 
acl storeurl_domain dstdomain .filehippo.com 
acl storeurl_domain dstdomain .redtube.com 
acl storeurl_domain dstdomain .tube8.com 
acl storeurl_domain dstdomain .phncdn.com 
acl storeurl_domain dstdomain .download.youporn.com
acl storeurl_domain dstdomain .static.blip.tv
storeurl_access allow storeurl_domain

# catch all... deny all...
acl store_regex url_regex -i cdn(.?)/[0-9a-zA-Z_-]*.?\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4)
acl store_regex url_regex ^http:\/\/([a-zA-Z-]+[0-9-]+)\.[A-Za-z]*\.[A-Za-z]*
acl store_regex url_regex (([a-z]{1,2}[0-9]{1,3})|([0-9]{1,3}[a-z]{1,2}))\.[a-z]*[0-9]?\.[a-z]{3}
storeurl_access allow store_regex
storeurl_access deny all

# setting storeurl-nya disini nih...
# kalo user yg dilayani squid jumlahnya banyak, maka children bisa ditambah
storeurl_rewrite_program /etc/squid/storeurl.pl
storeurl_rewrite_children 7
storeurl_rewrite_concurrency 50

# refresh_pattern

refresh_pattern (get_video\?|videoplayback\?|videodownload\?) 5259487 99999999% 5259487 override-expire ignore-reload ignore-private 
refresh_pattern -i \.(flv|mp4)$ 10080 90% 999999 ignore-no-cache override-expire ignore-private

##### jika ada refresh pattern lain, letakkan dibawah baris ini #####

refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|mp3) 10080 80% 10080 ignore-reload override-expire ignore-no-cache

##### default refresh_pattern, letakkan paling bawah! #####
refresh_pattern -i (/cgi-bin/|\?)   0   0%      0
refresh_pattern .                   0   0%   4320

 Selanjutnya, bikin storeurl nya (credits mas ucok_karnadi)

# vim /etc/squid/storeurl.pl

Isi storeurl nya :

#!/usr/bin/perl

$|=1;
while (<>) {
@X = split;

if ( $X[0] =~ m/^http\:\/\/.*/) { 
$x = $X[0]; 
$_ = $X[0];
$u = $X[0];
} else { 
$x = $X[1]; 
$_ = $X[1];
$u = $X[1];
}

if ($x =~ m/^http\:\/\/.*(profile|photo|creative).*\.ak\.fbcdn\.net\/((h|)(profile|photos)-ak-)(snc|ash|prn|frc)[0-9]?(.*)/) {
        $out="http://fbcdn.net.squid.internal/" . $2  . "fb" .  $6  ;

} elsif ($X[1] =~ m/^http(|s)\:\/\/.*youtube.*(ptracking|stream_204|player_204|gen_204).*(video_id|docid|v)\=([^\&\s]*).*/){
		$vid = $4 ;
		@cpn = m/[&?]cpn\=([^\&\s]*)/;
		$fn = "/var/log/squid/@cpn";
		unless (-e $fn) {
		open FH,">".$fn ;
		print FH "$vid\n";
		close FH;
		}
		print $x . $X[1] . "\n";

} elsif ($X[1] =~ m/^http\:\/\/.*(youtube|google).*videoplayback.*/){
		@itag = m/[&?](itag=[0-9]*)/;
		@ids = m/[&?]id\=([^\&\s]*)/;
		@mime = m/[&?](mime\=[^\&\s]*)/;
		@cpn = m/[&?]cpn\=([^\&\s]*)/;
		if (defined($cpn[0])) {
		$fn = "/tmp/@cpn";
		if (-e $fn) {
		open FH,"<".$fn ;
		$id  = <FH>;
		chomp $id ;
		close FH ;
		} else {
		$id = $ids[0] ;
		}
		} else {
		$id = $ids[0] ;
		}
		@range = m/[&?](range=[^\&\s]*)/;
		print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/id=" . $id . "&@itag@range@mime\n";

} elsif ($x =~ m/^http:\/\/i[1-4]\.ytimg\.com\/(.*)/) {
        $out="http://ytimg.com.squid.internal/" . $1 ;

} elsif ($x =~ m/^http:\/\/.*\.dl\.sourceforge\.net\/(.*)/) {
          $out="http://dl.sourceforge.net.squid.internal/" . $1 ;

		#Speedtest
} elsif ($x =~ m/^http\:\/\/.*\/speedtest\/(.*\.(jpg|txt)).*/) {
        $out="http://speedtest.squid.internal/" . $1 ;

		#BLOGSPOT
} elsif ($x =~ m/^http:\/\/[1-4]\.bp\.(blogspot\.com.*)/) {
        $out="http://blog-cdn." . $1  ;

		#AVAST
} elsif ($x =~ m/^http:\/\/download[0-9]{3}.(avast.com.*)/) {
          $out="http://avast-cdn." . $1  ;

	      #AVAST
} elsif ($x =~ m/^http:\/\/[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\/(iavs.*)/) {
        $out="http://avast-cdn.avast.com/" . $1  ;

  	#KAV
} elsif ($x =~ m/^http:\/\/dnl-[0-9]{2}.(geo.kaspersky.com.*)/) {
          $out="http://kav-cdn." . $1  ;

		#AVG
} elsif ($x =~ m/^http:\/\/update.avg.com/) {
          $out="http://avg-cdn." . $1  ;

		#maps.google.com
} elsif ($x =~ m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) {
        $out="http://" . $1  . $2 ;

		#gstatic and/or wikimapia
} elsif ($x =~ m/^http:\/\/([a-z])[0-9]?(\.gstatic\.com.*|\.wikimapia\.org.*)/) {
        $out="http://" . $1  . $2 ;

		#maps.google.com
} elsif ($x =~ m/^http:\/\/(khm|mt)[0-9]?(.google.com.*)/) {
        $out="http://" . $1  . $2 ;

		#Google
} elsif ($x =~ m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
        $out="http://www.google-analytics.com/__utm.gif";

} elsif ($x =~ m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
        $out="http://" . $1 ;

		#cdn, varialble 1st path
} elsif (($x =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
        @y = ($1,$2,$4,$5);
        $y[0] =~ s/[a-z0-9]{2,5}/cdn./;
        $out="http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] ;

		#rapidshare
} elsif (($x =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
        $out="http://cdn." . $3 . "/squid.internal/" . $5 ;

		#for yimg.com video
} elsif ($x =~ m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
        $out="http://cdn.yimg.com/" . $3 ;

		#for yimg.com doubled
} elsif ($x =~ m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
        $out="http://cdn.yimg.com/"  . $3 ;

		#for yimg.com with &sig=
} elsif ($x =~ m/^http:\/\/([^\.]*)\.yimg\.com\/(.*)/) {
        @y = ($1,$2);
        $y[0] =~ s/[a-z]+([0-9]+)?/cdn/;
        $y[1] =~ s/&sig=.*//;
        $out="http://" . $y[0] . ".yimg.com/"  . $y[1] ;

} else {
        $out=$x;

}
if ( $X[0] =~ m/^http\:\/\/.*/) { 
	print "$out\n" ;
	} else { 
	print $X[0] . "$out\n" ;
	}
}

Lakukan chmod pada storeurl agar dapat di eksekusi :

# chmod +x /etc/squid/storeurl.pl

Jalankan lusca/squid anda, dan monggo cek Youtube dan access.log untuk memastikan video cachingnya berjalan baik.

Catatan :

  • Youtube biasanya suka gonta ganti coding, yang mengakibatkan storeurl.pl tidak dapat melakukan redirect dengan benar. Oleh karena itu, biasakan mengupdate storeurl.pl dengan versi terbaru.
  • Untuk konfigurasi dasar squid2.7STABLE9 atau Lusca, bisa dilihat disini.

Last update : 17 Desember 2013

Catatan :

  • Jangan asal copas… perhatikan kode / perintah yang anda copas.
  • Saya tidak menjamin panduan ini bisa berhasil melakukan cache Youtube atau tidak. Just give a try! :)

_____________________________

Credits :

  1. http://aacable.wordpress.com/2012/01/11/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/
  2. http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube
  3. http://www.forummikrotik.com/linux-support/21089-new-storeurl-untuk-youtube-file-range.html (thanks mas ucok_karnadi)

About Fajar R. 297 Articles
Seseorang yang antusias terhadap dunia internet, IT, networking dan komputer. Tipikal introvert tapi suka berorganisasi dan adventuring ke tempat-tempat baru.

20 Komentar

  1. wkwkw wkwkw, mudah mudahan, saya juga baca baca TCP_REFRESH_UNMODIFIED artinya ter cache/HIT

    karena dah lama menggunakan squid di lognya belum nemu

    revalidated=*_REFRESH_*

    kapan “REFRESH” nya? squid 1/2 tahun, apa ada yang salah dengan pengaturan saya?

    Balas

    • squid kan ngecek/validasi object yg di cache mengikuti mekanisme dalam refresh_pattern. mungkin mas ada naroh refresh_pattern dengan setting stale/lm-factor/dsb dengan nilai yg terlalu tinggi. contoh refresh_pattern -i \.jpg$ 999999 999% 99999 .. atau sejenisnya.

      Balas

  2. pada saat “TCP_HIT” ada “NONE” berarti data diambil dari cache

    sedangkan “TCP_REFRESH_UNMODIFIED” ada “DIRECT” apakah data tidak diambil dari cache?

    dengan kata lain “TCP_REFRESH_UNMODIFIED” tidak ter cache/cache tidak digunakan?

    Balas

    • direct karena squid memvalidasi object tsb langsung ke webservernya. objectnya masih ter-cache kok (sepengetahuan ane sih begitu.. hehehe).

      Balas

  3. mas fazar,

    saya udah hampir 1/2 tahun pakai squid, semakin lama malah banyak
    “TCP_REFRESH_UNMODIFIED”
    apa berarti cache tidak digunakan?
    mohon bnatuannya.terimakasih

    Balas

    • don’t worry.. be happy. :)
      ini kemungkinan disebabkan karena penggunaan reload-into-ims. squid melakukan validasi dan mendapatkan respon not modified dari objek yg di akses klien.

      Balas

  6. mas fazar, ane dapat squid.conf dan storeurl.pl yang mantap nih buat lusca FMI….. maaf tapi lupa sumbernya dimana, soalnya keasikan otak atik sih.. :) … ini saya berbagi aja ya, mudah2an berkenan… kalo ada yang kurang pas tolong dilengkapi ya, karna saya masih nebie :(

    pertama :

    saya pakai ubuntu 11.10 server x64
    masuk ke proxy sebagai root
    #cd lusca_FMI
    LUSCA_FMI# /etc/init.d/squid stop
    LUSCA_FMI# make distclean ( kalo pakai ubuntu server x64)
    truss compile ulang lusca nya dengan tambahan –enable-referer-log
    jadi compilenya ane begini :
    ./configure –prefix=/usr –exec_prefix=/usr –bindir=/usr/sbin –sbindir=/usr/sbin –libexecdir=/usr/lib/squid –sysconfdir=/etc/squid –localstatedir=/var/spool/squid –datadir=/usr/share/squid –enable-http-gzip –enable-async-io=24 –with-aufs-threads=24 –with-pthreads –enable-storeio=aufs –enable-referer-log –enable-linux-netfilter –enable-arp-acl –enable-epoll –enable-removal-policies=heap –with-aio –with-dl –enable-snmp –enable-delay-pools –enable-htcp –enable-cache-digests –disable-unlinkd –enable-large-cache-files –with-large-files –enable-err-languages=English –enable-default-err-language=English –with-maxfd=65536 && make && make install

    setelah siap compile kita edit squid.conf kita dengan tambahan ini:

    referer_log /var/log/squid/referer.log dan
    acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*redirect_counter=1.*$
    acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&cms_redirect=yes.*$
    acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&ir=1.*$
    acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&rr=12.*$
    storeurl_access deny dontrewrite

    jadinya begini deh :

    http_port 3128 transparent
    server_http11 on

    pid_filename /var/run/squid.pid
    coredump_dir /var/spool/squid/
    error_directory /usr/share/squid/errors/English
    icon_directory /usr/share/squid/icons
    mime_table /usr/share/squid/mime.conf

    cache_mem 8 MB
    maximum_object_size_in_memory 512 bytes
    memory_replacement_policy heap GDSF
    cache_replacement_policy heap LFUDA

    minimum_object_size 0 KB
    maximum_object_size 1024 MB
    cache_swap_low 97
    cache_swap_high 99

    cache_dir aufs /cache1 50000 38 256
    cache_dir aufs /cache2 50000 38 256
    cache_dir aufs /cache3 50000 38 256
    cache_dir aufs /cache4 50000 38 256
    cache_dir aufs /cache5 50000 38 256
    cache_dir aufs /cache6 50000 38 256
    cache_dir aufs /cache7 50000 38 256
    cache_dir aufs /cache8 50000 38 256

    access_log daemon:/var/log/squid/access.log squid
    cache_log /var/log/squid/cache.log
    referer_log /var/log/squid/referer.log <<<<< tambahan
    cache_store_log none
    cache_peer_access httpsAccel allow all
    store_dir_select_algorithm round-robin
    logfile_daemon /usr/lib/squid/logfile-daemon
    logfile_rotate 1

    acl all src all
    acl manager proto cache_object
    acl localhost src 127.0.0.1/32
    acl to_localhost dst 127.0.0.0/8
    acl localnet src 192.168.7.0/24

    acl SSL_ports port 443
    acl Safe_ports port 80 # http
    acl Safe_ports port 21 # ftp
    acl Safe_ports port 443 # https
    acl Safe_ports port 70 # gopher
    acl Safe_ports port 210 # wais
    acl Safe_ports port 1025-65535 # unregistered ports
    acl Safe_ports port 280 # http-mgmt
    acl Safe_ports port 488 # gss-http
    acl Safe_ports port 591 # filemaker
    acl Safe_ports port 777 # multiling http
    acl CONNECT method CONNECT
    acl purge method PURGE
    acl snmppublic snmp_community public

    http_access allow manager localhost
    http_access deny manager
    http_access allow purge localhost
    http_access deny !Safe_ports
    http_access deny CONNECT !SSL_ports

    http_access allow localnet
    http_access allow all
    http_access deny all

    icp_access allow localnet
    icp_access deny all
    icp_port 0

    buffered_logs on

    acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
    upgrade_http0.9 deny shoutcast

    acl apache rep_header Server ^Apache
    broken_vary_encoding allow apache

    forwarded_for off
    header_access From deny all
    header_access Server deny all
    header_access Link deny all
    header_access Via deny all
    header_access X-Forwarded-For deny all
    httpd_suppress_version_string on

    acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*redirect_counter=1.*$
    acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&cms_redirect=yes.*$
    acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&ir=1.*$
    acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&rr=12.*$
    storeurl_access deny dontrewrite <<<<<< tambahan

    acl store_rewrite_list urlpath_regex \/(get_video|video\?v|videoplayback\?id|videoplayback.*id)
    acl store_rewrite_list urlpath_regex \/(get_video\?|videodownload\?|videoplayback.*id|watch\?)
    acl store_rewrite_list urlpath_regex \.(3gp|mp(3|4)|flv|(m|f)4v|on2|fid|avi|mov|wm(a|v)|(mp(e?g|a|e|1|2))|mk(a|v)|jp(e?g|e|2)|gif|png|tiff?|bmp|tga|svg|ico|swf|exe|ms(i|u|p)|cab|psf|mar|bin|z(ip|[0-9]{2})|r(ar|[0-9]{2})|7z)\?
    acl store_rewrite_list_domain url_regex ^http:\/\/(([a-z-]+[0-9-]+)|([0-9-]+[a-z-]+))\.[a-z0-9-]*\.[a-z]{2,4}
    acl store_rewrite_list_domain url_regex ^http:\/\/([a-zA-Z-]+[0-9-]+)\.[A-Za-z]*\.[A-Za-z]*
    acl store_rewrite_list_domain url_regex (([a-z]{1,2}[0-9]{1,3})|([0-9]{1,3}[a-z]{1,2}))\.[a-z]*[0-9]?\.[a-z]{3}
    acl store_rewrite_list_path urlpath_regex \.(3gp|mp(3|4)|flv|(m|f)4v|on2|fid|avi|mov|wm(a|v)|(mp(e?g|a|e|1|2))|mk(a|v)|jp(e?g|e|2)|gif|png|tiff?|bmp|tga|svg|ico|swf|exe|ms(i|u|p)|cab|psf|mar|bin|z(ip|[0-9]{2})|r(ar|[0-9]{2})|7z)$
    acl store_rewrite_list_domain_CDN url_regex (cbk|mt|khm|mlt|tbn)[0-9]?.google\.co(m|\.uk|\.id)
    acl store_rewrite_list_domain_CDN url_regex photos-[a-z].ak.fbcdn.net
    acl store_rewrite_list_domain_CDN url_regex ^http:\/\/([a-z])[0-9]?(\.gstatic\.com|\.wikimapia\.org)
    acl store_rewrite_list_domain_CDN url_regex ^http:\/\/[.a-z0-9]*\.photobucket\.com.*\.[a-z]{3}$
    acl store_rewrite_list_domain_CDN url_regex ^http:\/\/.*speedtest.*
    acl store_rewrite_list_domain_CDN url_regex streamate.doublepimp.com.*\.js\? \.doubleclick\.net.* yieldmanager cpxinteractive quantserve\.com
    acl speedtest_allow_url url_regex -i \.speedtest\.net\/ speedtest
    acl speedtest_allow_url url_regex ^http:\/\/speedtest\.*
    acl speedtest_allow_dom dstdomain .speedtest.net

    acl dontrewrite url_regex (get_video|video\?v=|videoplayback\?id|videoplayback.*id).*begin\=[1-9][0-9]* \.php\? \.asp\? \.aspx\? threadless.*\.jpg\?r=
    acl getmethod method GET

    storeurl_access deny dontrewrite
    storeurl_access deny !getmethod

    storeurl_access allow store_rewrite_list_domain_CDN
    storeurl_access allow store_rewrite_list
    storeurl_access allow store_rewrite_list_domain store_rewrite_list_path
    storeurl_access allow speedtest_allow_url
    storeurl_access allow speedtest_allow_dom
    storeurl_access deny all
    storeurl_rewrite_program /etc/squid/storeurl.pl
    storeurl_rewrite_children 1
    storeurl_rewrite_concurrency 99

    dan storeurl.pl nya :

    #!/usr/bin/perl
    # $Rev$
    # by chudy_fernandez@yahoo.com
    # rev by ucok_karnadi@yahoo.com
    # Youtube updates at http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion

    # http://www.facebook.com/stressss

    $|=1;
    while () {
    @X = split;
    $x = $X[0] . ” “;

    #facebook
    if (m/^http\:\/\/.*(profile|photo).*\.ak\.fbcdn\.net(\/h(profile|photos)-ak-)(snc|ash|prn)[0-9]?(.*)/) {
    print $x . “http://facebook.SQUIDINTERNAL” . $2 . “fb” . $5 . “\n”;

    #Speedtest
    } elsif (m/^http\:\/\/.*\/speedtest\/(.*)\?.*/) {
    print $x . “http://speedtest.SQUIDINTERNAL/speedtest/” . $1 . “\n”;

    #reverbnation
    } elsif (m/^http:\/\/[a-z0-9]{4}\.reverbnation\.com\/.*\/([0-9]*).*/) {
    print $x . “http://reverbnation.com.SQUIDINTERNAL/” . “$1” . “\n”;

    #BLOGSPOT
    } elsif (m/^http:\/\/[1-4].bp.(blogspot.com.*)/) {
    print $x . “http://blog-cdn.” . $1 . “\n”;

    #ytimg
    } elsif (m/^http:\/\/i[1-4]\.ytimg\.com(.*)/) {
    print $x . “http://cdn.ytimg.com” . $1 . “\n”;

    #AVAST
    } elsif (m/^http:\/\/download[0-9]{3}.(avast.com.*)/) {
    print $x . “http://avast-cdn.” . $1 . “\n”;

    #KAV
    } elsif (m/^http:\/\/dnl-[0-9]{2}.(geo.kaspersky.com.*)/) {
    print $x . “http://kav-cdn.” . $1 . “\n”;

    #AVG
    } elsif (m/^http:\/\/update.avg.com/) {
    print $x . “http://avg-cdn.” . $1 . “\n”;

    #maps.google.com
    } elsif (m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) {
    print $x . “http://” . $1 . $2 . “\n”;

    #gstatic and/or wikimapia
    } elsif (m/^http:\/\/([a-z])[0-9]?(\.gstatic\.com.*|\.wikimapia\.org.*)/) {
    print $x . “http://” . $1 . $2 . “\n”;

    #maps.google.com
    } elsif (m/^http:\/\/(khm|mt)[0-9]?(.google.com.*)/) {
    print $x . “http://” . $1 . $2 . “\n”;

    # for ALL Youtube ( range & non range )
    # first you need do this
    # build your squid with “–enable-referer-log”
    # install package dependencies “apt-get install libfile-readbackwards-perl”
    # add this “referer_log /var/log/squid/referer.log” to your squid config

    } elsif ($X[1] =~ m/^http\:\/\/.*(youtube|google).*videoplayback.*/){
    @itag = m/[&?](itag=[0-9]*)/;
    @CPN = m/[&?]cpn\=([a-zA-Z0-9\-\_]*)/;
    @IDS = m/[&?]id\=([a-zA-Z0-9\-\_]*)/;
    $id = &GetID($CPN[0], $IDS[0]);
    @range = m/[&?](range=[^\&\s]*)/;
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/id=” . $id . “&@itag@range\n”;

    #Google
    } elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
    print $x . “http://www.google-analytics.com/__utm.gif\n”;

    #Cache High Latency Ads
    } elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
    $y = $3;$z = $2;
    for ($y) {
    s/pixel;.*/pixel/;
    s/activity;.*/activity/;
    s/(imgad[^&]*).*/\1/;
    s/;ord=[?0-9]*//;
    s/;&timestamp=[0-9]*//;
    s/[&?]correlator=[0-9]*//;
    s/&cookie=[^&]*//;
    s/&ga_hid=[^&]*//;
    s/&ga_vid=[^&]*//;
    s/&ga_sid=[^&]*//;
    # s/&prev_slotnames=[^&]*//
    # s/&u_his=[^&]*//;
    s/&dt=[^&]*//;
    s/&dtd=[^&]*//;
    s/&lmt=[^&]*//;
    s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/[;&?]ord=[?0-9]*//;
    s/[;&]mpvid=[^&;]*//;
    s/&xpc=[^&]*//;
    # yieldmanager
    s/\?clickTag=[^&]*//;
    s/&u=[^&]*//;
    s/&slotname=[^&]*//;
    s/&page_slots=[^&]*//;
    }
    print $x . “http://” . $1 . $2 . $y . “\n”;

    #cache high latency ads
    } elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
    print $x . “http://” . $1 . “/” . $2 . “\n”;

    } elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
    print $x . “http://” . $1 . “\n”;

    #cdn, varialble 1st path
    } elsif (($X[1] =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
    @y = ($1,$2,$4,$5);
    $y[0] =~ s/[a-z0-9]{2,5}/cdn./;
    print $x . “http://” . $y[0] . $y[1] . “/” . $y[2] . “.” . $y[3] . “\n”;

    #rapidshare
    } elsif (($X[1] =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
    print $x . “http://cdn.” . $3 . “/SQUIDINTERNAL/” . $5 . “\n”;

    } elsif (($X[1] =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
    print $x . “http://” . $1 . “/SQUIDINTERNAL/” . $3 . “\n”;

    #domain/path/.*/path/filename
    } elsif (($X[1] =~ /fucktube/) && (m/^http:\/\/(.*?)(\.[^\.\-]*?[^\/]*\/[^\/]*)\/(.*)\/([^\/]*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})(\?.*?)$/)) {
    @y = ($1,$2,$4,$5,$6);
    $y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|([^\.]*cdn[^\.]*)|([^\.]*cache[^\.]*))/cdn/;
    print $x . “http://” . $y[0] . $y[1] . “/” . $y[2] . “/” . $y[3] . “.” . $y[4] . “\n”;

    #like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
    } elsif (($X[1] =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
    print $x . “http://cdn.” . $4 . $6 . “\n”;

    #for yimg.com video
    } elsif (m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
    print $x . “http://cdn.yimg.com/” . $3 . “\n”;

    #for yimg.com doubled
    } elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
    print $x . “http://cdn.yimg.com/” . $3 . “\n”;

    #for yimg.com with &sig=
    } elsif (m/^http:\/\/([^\.]*)\.yimg\.com\/(.*)/) {
    @y = ($1,$2);
    $y[0] =~ s/[a-z]+([0-9]+)?/cdn/;
    $y[1] =~ s/&sig=.*//;
    print $x . “http://” . $y[0] . “.yimg.com/” . $y[1] . “\n”;

    #youjizz. We use only domain and filename
    } elsif (($X[1] =~ /media[0-9]{1,5}\.youjizz/) && (m/^http:\/\/(.*?)(\.[^\.\-]*?\.[^\/]*)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})(\?.*?)$/)) {
    @y = ($1,$2,$4,$5);
    $y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|([^\.]*cdn[^\.]*)|([^\.]*cache[^\.]*))/cdn/;
    print $x . “http://” . $y[0] . $y[1] . “/” . $y[2] . “.” . $y[3] . “\n”;

    #general purpose for cdn servers. add above your specific servers.
    } elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
    print $x . “http://squid-cdn-url/” . $2 . “.” . $3 . “\n”;

    # spicific extention
    # } elsif (m/^http:\/\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|wmv|3gp|mp(4|3)|exe|msi|zip|on2|mar|swf).*?/) {
    # @y = ($1,$2);
    # $y[0] =~ s/((cache|cdn)[-\d]*)|([a-zA-A]+-?[0-9]+(-[a-zA-Z]*)?)/cdn/;
    # print $x . “http://” . $y[0] . “.” . $y[1] . “\n”;

    #generic http://variable.domain.com/path/filename.”ex”, “ext” or “exte”
    #http://cdn1-28.projectplaylist.com
    #http://s1sdlod041.bcst.cdn.s1s.yimg.com
    } elsif (m/^http:\/\/(.*?)(\.[^\.\-]*?\..*?)\/([^\?\&\=]*)\.([\w\d]{2,4})\??.*$/) {
    @y = ($1,$2,$3,$4);
    $y[0] =~ s/([a-z][0-9][a-z]dlod[\d]{3})|((cache|cdn)[-\d]*)|([a-zA-A]+-?[0-9]+(-[a-zA-Z]*)?)/cdn/;
    print $x . “storeurl://” . $y[0] . $y[1] . “/” . $y[2] . “.” . $y[3] . “\n”;

    # all that ends with ;
    } elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
    print $x . “http://” . $1 . “/” . $2 . “\n”;

    } else {
    print $x . $X[1] . “\n”;
    }
    }

    sub GetID
    {
    use File::ReadBackwards;
    my $ref_log = File::ReadBackwards->new(‘/var/log/squid/referer.log’);
    while (defined($line = $ref_log->readline))
    {
    if ($line =~ m/.*youtube.*\/watch\?.*v=([a-zA-Z0-9\-\_]*).*\s.*id=$IDS[0].*/){
    $id = $1;
    last;
    }
    if ($line =~ m/.*youtube.*\/ptracking.*cpn=$CPN[0].*video\_id=([a-zA-Z0-9\-\_]*).*/){
    $id = $1;
    last;
    }
    if ($line =~ m/.*youtube.*\/ptracking.*video\_id=([a-zA-Z0-9\-\_]*).*cpn=$CPN[0].*/){
    $id = $1;
    last;
    }
    }
    $ref_log->close();
    return $id;
    }

    kalo sudah siap edit tinggal :

    LUSCA_FMI# squid -f /etc/squid/squid.conf -z
    LUSCA_FMI# /etc/init.d/squid restart

    trusss tinggal di test aja youtube nya dan liat acces.log nya deh

    dijamin makyosss deh :)

    Balas

    • trims mas udah share. oh iya… untuk storeurl ada juga kok di share masbro ucok_karnadi di postingan sebelah. it works! :)

      Balas

  7. mau nanya mas, kalo konfig di squid nya yng ada hubungannya dng storeurl di non aktif, nanti apakah ndak ganggu fungsi squidnya untuk cache dynamic content selain video youtube.

    Balas

  8. iya nih mas bro, youtubenya, salah putar,
    mau nonton film yg ini, malah keluar film yg itu,,

    mau tanya nih mas, kalo mau mem bypass kan youtube dari squid gmn ya,,?

    selagi nunggu update yg terbaru keluar hehehe

    makasih mas fazar.

    Balas

  10. salam kenal mas fazar, nice & usefull post, setelah dibikin pusing proxy berminggu2, ternyata ada rekan lain yng punya masalah sama, ternyata masalahnya ada di youtube nya, ditunggu share solusinya  ya.maturnuwun

    Balas

Leave a Reply

Alamat email Anda tidak akan dipublikasikan.


*