Caching Dynamic Content dan Youtube Menggunakan Squid

wkwkw wkwkw, mudah mudahan, saya juga baca baca TCP_REFRESH_UNMODIFIED artinya ter cache/HIT

karena dah lama menggunakan squid di lognya belum nemu

revalidated=*_REFRESH_*

kapan “REFRESH” nya? squid 1/2 tahun, apa ada yang salah dengan pengaturan saya?

pada saat “TCP_HIT” ada “NONE” berarti data diambil dari cache

sedangkan “TCP_REFRESH_UNMODIFIED” ada “DIRECT” apakah data tidak diambil dari cache?

dengan kata lain “TCP_REFRESH_UNMODIFIED” tidak ter cache/cache tidak digunakan?

mas fazar,

saya udah hampir 1/2 tahun pakai squid, semakin lama malah banyak
“TCP_REFRESH_UNMODIFIED”
apa berarti cache tidak digunakan?
mohon bnatuannya.terimakasih

mas mau tnya klu storeurl.pl nya dibuat sendiri (offline) berjalan gx mas?

mudah2an youtube jangan menganti codenya lagi deh, bisa bikin stresss dih….. :) wkwwkkw

mas fazar, ane dapat squid.conf dan storeurl.pl yang mantap nih buat lusca FMI….. maaf tapi lupa sumbernya dimana, soalnya keasikan otak atik sih.. :) … ini saya berbagi aja ya, mudah2an berkenan… kalo ada yang kurang pas tolong dilengkapi ya, karna saya masih nebie :(

pertama :

saya pakai ubuntu 11.10 server x64
masuk ke proxy sebagai root
#cd lusca_FMI
LUSCA_FMI# /etc/init.d/squid stop
LUSCA_FMI# make distclean ( kalo pakai ubuntu server x64)
truss compile ulang lusca nya dengan tambahan –enable-referer-log
jadi compilenya ane begini :
./configure –prefix=/usr –exec_prefix=/usr –bindir=/usr/sbin –sbindir=/usr/sbin –libexecdir=/usr/lib/squid –sysconfdir=/etc/squid –localstatedir=/var/spool/squid –datadir=/usr/share/squid –enable-http-gzip –enable-async-io=24 –with-aufs-threads=24 –with-pthreads –enable-storeio=aufs –enable-referer-log –enable-linux-netfilter –enable-arp-acl –enable-epoll –enable-removal-policies=heap –with-aio –with-dl –enable-snmp –enable-delay-pools –enable-htcp –enable-cache-digests –disable-unlinkd –enable-large-cache-files –with-large-files –enable-err-languages=English –enable-default-err-language=English –with-maxfd=65536 && make && make install

setelah siap compile kita edit squid.conf kita dengan tambahan ini:

referer_log /var/log/squid/referer.log dan
acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*redirect_counter=1.*$
acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&cms_redirect=yes.*$
acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&ir=1.*$
acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&rr=12.*$
storeurl_access deny dontrewrite

jadinya begini deh :

http_port 3128 transparent
server_http11 on

pid_filename /var/run/squid.pid
coredump_dir /var/spool/squid/
error_directory /usr/share/squid/errors/English
icon_directory /usr/share/squid/icons
mime_table /usr/share/squid/mime.conf

cache_mem 8 MB
maximum_object_size_in_memory 512 bytes
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA

minimum_object_size 0 KB
maximum_object_size 1024 MB
cache_swap_low 97
cache_swap_high 99

cache_dir aufs /cache1 50000 38 256
cache_dir aufs /cache2 50000 38 256
cache_dir aufs /cache3 50000 38 256
cache_dir aufs /cache4 50000 38 256
cache_dir aufs /cache5 50000 38 256
cache_dir aufs /cache6 50000 38 256
cache_dir aufs /cache7 50000 38 256
cache_dir aufs /cache8 50000 38 256

access_log daemon:/var/log/squid/access.log squid
cache_log /var/log/squid/cache.log
referer_log /var/log/squid/referer.log <<<<< tambahan
cache_store_log none
cache_peer_access httpsAccel allow all
store_dir_select_algorithm round-robin
logfile_daemon /usr/lib/squid/logfile-daemon
logfile_rotate 1

acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 192.168.7.0/24

acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
acl purge method PURGE
acl snmppublic snmp_community public

http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

http_access allow localnet
http_access allow all
http_access deny all

icp_access allow localnet
icp_access deny all
icp_port 0

buffered_logs on

acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
upgrade_http0.9 deny shoutcast

acl apache rep_header Server ^Apache
broken_vary_encoding allow apache

forwarded_for off
header_access From deny all
header_access Server deny all
header_access Link deny all
header_access Via deny all
header_access X-Forwarded-For deny all
httpd_suppress_version_string on

acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*redirect_counter=1.*$
acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&cms_redirect=yes.*$
acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&ir=1.*$
acl dontrewrite url_regex -i c\.youtube\.com\/videoplayback.*\&rr=12.*$
storeurl_access deny dontrewrite <<<<<< tambahan

acl store_rewrite_list urlpath_regex \/(get_video|video\?v|videoplayback\?id|videoplayback.*id)
acl store_rewrite_list urlpath_regex \/(get_video\?|videodownload\?|videoplayback.*id|watch\?)
acl store_rewrite_list urlpath_regex \.(3gp|mp(3|4)|flv|(m|f)4v|on2|fid|avi|mov|wm(a|v)|(mp(e?g|a|e|1|2))|mk(a|v)|jp(e?g|e|2)|gif|png|tiff?|bmp|tga|svg|ico|swf|exe|ms(i|u|p)|cab|psf|mar|bin|z(ip|[0-9]{2})|r(ar|[0-9]{2})|7z)\?
acl store_rewrite_list_domain url_regex ^http:\/\/(([a-z-]+[0-9-]+)|([0-9-]+[a-z-]+))\.[a-z0-9-]*\.[a-z]{2,4}
acl store_rewrite_list_domain url_regex ^http:\/\/([a-zA-Z-]+[0-9-]+)\.[A-Za-z]*\.[A-Za-z]*
acl store_rewrite_list_domain url_regex (([a-z]{1,2}[0-9]{1,3})|([0-9]{1,3}[a-z]{1,2}))\.[a-z]*[0-9]?\.[a-z]{3}
acl store_rewrite_list_path urlpath_regex \.(3gp|mp(3|4)|flv|(m|f)4v|on2|fid|avi|mov|wm(a|v)|(mp(e?g|a|e|1|2))|mk(a|v)|jp(e?g|e|2)|gif|png|tiff?|bmp|tga|svg|ico|swf|exe|ms(i|u|p)|cab|psf|mar|bin|z(ip|[0-9]{2})|r(ar|[0-9]{2})|7z)$
acl store_rewrite_list_domain_CDN url_regex (cbk|mt|khm|mlt|tbn)[0-9]?.google\.co(m|\.uk|\.id)
acl store_rewrite_list_domain_CDN url_regex photos-[a-z].ak.fbcdn.net
acl store_rewrite_list_domain_CDN url_regex ^http:\/\/([a-z])[0-9]?(\.gstatic\.com|\.wikimapia\.org)
acl store_rewrite_list_domain_CDN url_regex ^http:\/\/[.a-z0-9]*\.photobucket\.com.*\.[a-z]{3}$
acl store_rewrite_list_domain_CDN url_regex ^http:\/\/.*speedtest.*
acl store_rewrite_list_domain_CDN url_regex streamate.doublepimp.com.*\.js\? \.doubleclick\.net.* yieldmanager cpxinteractive quantserve\.com
acl speedtest_allow_url url_regex -i \.speedtest\.net\/ speedtest
acl speedtest_allow_url url_regex ^http:\/\/speedtest\.*
acl speedtest_allow_dom dstdomain .speedtest.net

acl dontrewrite url_regex (get_video|video\?v=|videoplayback\?id|videoplayback.*id).*begin\=[1-9][0-9]* \.php\? \.asp\? \.aspx\? threadless.*\.jpg\?r=
acl getmethod method GET

storeurl_access deny dontrewrite
storeurl_access deny !getmethod

storeurl_access allow store_rewrite_list_domain_CDN
storeurl_access allow store_rewrite_list
storeurl_access allow store_rewrite_list_domain store_rewrite_list_path
storeurl_access allow speedtest_allow_url
storeurl_access allow speedtest_allow_dom
storeurl_access deny all
storeurl_rewrite_program /etc/squid/storeurl.pl
storeurl_rewrite_children 1
storeurl_rewrite_concurrency 99

dan storeurl.pl nya :

#!/usr/bin/perl
# $Rev$
# by chudy_fernandez@yahoo.com
# rev by ucok_karnadi@yahoo.com
# Youtube updates at http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion

# http://www.facebook.com/stressss

$|=1;
while () {
@X = split;
$x = $X[0] . ” “;

#facebook
if (m/^http\:\/\/.*(profile|photo).*\.ak\.fbcdn\.net(\/h(profile|photos)-ak-)(snc|ash|prn)[0-9]?(.*)/) {
print $x . “http://facebook.SQUIDINTERNAL” . $2 . “fb” . $5 . “\n”;

#Speedtest
} elsif (m/^http\:\/\/.*\/speedtest\/(.*)\?.*/) {
print $x . “http://speedtest.SQUIDINTERNAL/speedtest/” . $1 . “\n”;

#reverbnation
} elsif (m/^http:\/\/[a-z0-9]{4}\.reverbnation\.com\/.*\/([0-9]*).*/) {
print $x . “http://reverbnation.com.SQUIDINTERNAL/” . “$1” . “\n”;

#BLOGSPOT
} elsif (m/^http:\/\/[1-4].bp.(blogspot.com.*)/) {
print $x . “http://blog-cdn.” . $1 . “\n”;

#ytimg
} elsif (m/^http:\/\/i[1-4]\.ytimg\.com(.*)/) {
print $x . “http://cdn.ytimg.com” . $1 . “\n”;

#AVAST
} elsif (m/^http:\/\/download[0-9]{3}.(avast.com.*)/) {
print $x . “http://avast-cdn.” . $1 . “\n”;

#KAV
} elsif (m/^http:\/\/dnl-[0-9]{2}.(geo.kaspersky.com.*)/) {
print $x . “http://kav-cdn.” . $1 . “\n”;

#AVG
} elsif (m/^http:\/\/update.avg.com/) {
print $x . “http://avg-cdn.” . $1 . “\n”;

#maps.google.com
} elsif (m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) {
print $x . “http://” . $1 . $2 . “\n”;

#gstatic and/or wikimapia
} elsif (m/^http:\/\/([a-z])[0-9]?(\.gstatic\.com.*|\.wikimapia\.org.*)/) {
print $x . “http://” . $1 . $2 . “\n”;

#maps.google.com
} elsif (m/^http:\/\/(khm|mt)[0-9]?(.google.com.*)/) {
print $x . “http://” . $1 . $2 . “\n”;

# for ALL Youtube ( range & non range )
# first you need do this
# build your squid with “–enable-referer-log”
# install package dependencies “apt-get install libfile-readbackwards-perl”
# add this “referer_log /var/log/squid/referer.log” to your squid config

} elsif ($X[1] =~ m/^http\:\/\/.*(youtube|google).*videoplayback.*/){
@itag = m/[&?](itag=[0-9]*)/;
@CPN = m/[&?]cpn\=([a-zA-Z0-9\-\_]*)/;
@IDS = m/[&?]id\=([a-zA-Z0-9\-\_]*)/;
$id = &GetID($CPN[0], $IDS[0]);
@range = m/[&?](range=[^\&\s]*)/;
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/id=” . $id . “&@itag@range\n”;

#Google
} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
print $x . “http://www.google-analytics.com/__utm.gif\n”;

#Cache High Latency Ads
} elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
$y = $3;$z = $2;
for ($y) {
s/pixel;.*/pixel/;
s/activity;.*/activity/;
s/(imgad[^&]*).*/\1/;
s/;ord=[?0-9]*//;
s/;&timestamp=[0-9]*//;
s/[&?]correlator=[0-9]*//;
s/&cookie=[^&]*//;
s/&ga_hid=[^&]*//;
s/&ga_vid=[^&]*//;
s/&ga_sid=[^&]*//;
# s/&prev_slotnames=[^&]*//
# s/&u_his=[^&]*//;
s/&dt=[^&]*//;
s/&dtd=[^&]*//;
s/&lmt=[^&]*//;
s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/[;&?]ord=[?0-9]*//;
s/[;&]mpvid=[^&;]*//;
s/&xpc=[^&]*//;
# yieldmanager
s/\?clickTag=[^&]*//;
s/&u=[^&]*//;
s/&slotname=[^&]*//;
s/&page_slots=[^&]*//;
}
print $x . “http://” . $1 . $2 . $y . “\n”;

#cache high latency ads
} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
print $x . “http://” . $1 . “/” . $2 . “\n”;

} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
print $x . “http://” . $1 . “\n”;

#cdn, varialble 1st path
} elsif (($X[1] =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
print $x . “http://” . $y[0] . $y[1] . “/” . $y[2] . “.” . $y[3] . “\n”;

#rapidshare
} elsif (($X[1] =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
print $x . “http://cdn.” . $3 . “/SQUIDINTERNAL/” . $5 . “\n”;

} elsif (($X[1] =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
print $x . “http://” . $1 . “/SQUIDINTERNAL/” . $3 . “\n”;

#domain/path/.*/path/filename
} elsif (($X[1] =~ /fucktube/) && (m/^http:\/\/(.*?)(\.[^\.\-]*?[^\/]*\/[^\/]*)\/(.*)\/([^\/]*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})(\?.*?)$/)) {
@y = ($1,$2,$4,$5,$6);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|([^\.]*cdn[^\.]*)|([^\.]*cache[^\.]*))/cdn/;
print $x . “http://” . $y[0] . $y[1] . “/” . $y[2] . “/” . $y[3] . “.” . $y[4] . “\n”;

#like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
} elsif (($X[1] =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
print $x . “http://cdn.” . $4 . $6 . “\n”;

#for yimg.com video
} elsif (m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
print $x . “http://cdn.yimg.com/” . $3 . “\n”;

#for yimg.com doubled
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
print $x . “http://cdn.yimg.com/” . $3 . “\n”;

#for yimg.com with &sig=
} elsif (m/^http:\/\/([^\.]*)\.yimg\.com\/(.*)/) {
@y = ($1,$2);
$y[0] =~ s/[a-z]+([0-9]+)?/cdn/;
$y[1] =~ s/&sig=.*//;
print $x . “http://” . $y[0] . “.yimg.com/” . $y[1] . “\n”;

#youjizz. We use only domain and filename
} elsif (($X[1] =~ /media[0-9]{1,5}\.youjizz/) && (m/^http:\/\/(.*?)(\.[^\.\-]*?\.[^\/]*)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})(\?.*?)$/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|([^\.]*cdn[^\.]*)|([^\.]*cache[^\.]*))/cdn/;
print $x . “http://” . $y[0] . $y[1] . “/” . $y[2] . “.” . $y[3] . “\n”;

#general purpose for cdn servers. add above your specific servers.
} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
print $x . “http://squid-cdn-url/” . $2 . “.” . $3 . “\n”;

# spicific extention
# } elsif (m/^http:\/\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|wmv|3gp|mp(4|3)|exe|msi|zip|on2|mar|swf).*?/) {
# @y = ($1,$2);
# $y[0] =~ s/((cache|cdn)[-\d]*)|([a-zA-A]+-?[0-9]+(-[a-zA-Z]*)?)/cdn/;
# print $x . “http://” . $y[0] . “.” . $y[1] . “\n”;

#generic http://variable.domain.com/path/filename.”ex”, “ext” or “exte”
#http://cdn1-28.projectplaylist.com
#http://s1sdlod041.bcst.cdn.s1s.yimg.com
} elsif (m/^http:\/\/(.*?)(\.[^\.\-]*?\..*?)\/([^\?\&\=]*)\.([\w\d]{2,4})\??.*$/) {
@y = ($1,$2,$3,$4);
$y[0] =~ s/([a-z][0-9][a-z]dlod[\d]{3})|((cache|cdn)[-\d]*)|([a-zA-A]+-?[0-9]+(-[a-zA-Z]*)?)/cdn/;
print $x . “storeurl://” . $y[0] . $y[1] . “/” . $y[2] . “.” . $y[3] . “\n”;

# all that ends with ;
} elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
print $x . “http://” . $1 . “/” . $2 . “\n”;

} else {
print $x . $X[1] . “\n”;
}
}

sub GetID
{
use File::ReadBackwards;
my $ref_log = File::ReadBackwards->new(‘/var/log/squid/referer.log’);
while (defined($line = $ref_log->readline))
{
if ($line =~ m/.*youtube.*\/watch\?.*v=([a-zA-Z0-9\-\_]*).*\s.*id=$IDS[0].*/){
$id = $1;
last;
}
if ($line =~ m/.*youtube.*\/ptracking.*cpn=$CPN[0].*video\_id=([a-zA-Z0-9\-\_]*).*/){
$id = $1;
last;
}
if ($line =~ m/.*youtube.*\/ptracking.*video\_id=([a-zA-Z0-9\-\_]*).*cpn=$CPN[0].*/){
$id = $1;
last;
}
}
$ref_log->close();
return $id;
}

kalo sudah siap edit tinggal :

LUSCA_FMI# squid -f /etc/squid/squid.conf -z
LUSCA_FMI# /etc/init.d/squid restart

trusss tinggal di test aja youtube nya dan liat acces.log nya deh

dijamin makyosss deh :)

mau nanya mas, kalo konfig di squid nya yng ada hubungannya dng storeurl di non aktif, nanti apakah ndak ganggu fungsi squidnya untuk cache dynamic content selain video youtube.

iya nih mas bro, youtubenya, salah putar,
mau nonton film yg ini, malah keluar film yg itu,,

mau tanya nih mas, kalo mau mem bypass kan youtube dari squid gmn ya,,?

selagi nunggu update yg terbaru keluar hehehe

makasih mas fazar.

salam kenal mas fazar, nice & usefull post, setelah dibikin pusing proxy berminggu2, ternyata ada rekan lain yng punya masalah sama, ternyata masalahnya ada di youtube nya, ditunggu share solusinya  ya.maturnuwun