7 CONTENT="Modular DocBook HTML Stylesheet Version 1.76b+
10 TITLE="Privoxy 3.0.1 User Manual"
11 HREF="index.html"><LINK
14 HREF="seealso.html"><LINK
17 HREF="../p_doc.css"></HEAD
28 SUMMARY="Header navigation table"
37 >Privoxy 3.0.1 User Manual</TH
79 >14.1. Regular Expressions</H2
84 > uses Perl-style <SPAN
89 HREF="actions-file.html"
93 HREF="filter-file.html"
97 HREF="http://www.pcre.org/"
102 HREF="http://www.oesterhelt.org/pcrs/"
107 > If you are reading this, you probably don't understand what <SPAN
111 > are, or what they can do. So this will be a very brief
112 introduction only. A full explanation would require a <A
113 HREF="http://www.oreilly.com/catalog/regex/"
118 > Regular expressions provide a language to describe patterns that can be
119 run against strings of characters (letter, numbers, etc), to see if they
120 match the string or not. The patterns are themselves (sometimes complex)
121 strings of literal characters, combined with wild-cards, and other special
122 characters, called meta-characters. The <SPAN
124 >"meta-characters"</SPAN
126 special meanings and are used to build complex patterns to be matched against.
127 Perl Compatible Regular Expressions are an especially convenient
131 > of the regular expression language.</P
133 > To make a simple analogy, we do something similar when we use wild-card
134 characters when listing files with the <B
141 > matches all filenames. The <SPAN
145 character here is the asterisk which matches any and all characters. We can be
146 more specific and use <TT
149 > to match just individual
152 >"dir file?.text"</SPAN
160 >, etc. We are pattern
161 matching, using a similar technique to <SPAN
163 >"regular expressions"</SPAN
166 > Regular expressions do essentially the same thing, but are much, much more
167 powerful. There are many more <SPAN
169 >"special characters"</SPAN
171 building complex patterns however. Let's look at a few of the common ones,
172 and then some examples:</P
187 > - Matches any single character, e.g. <SPAN
225 > - The preceding character or expression is matched ZERO or ONE
248 > - The preceding character or expression is matched ONE or MORE
271 > - The preceding character or expression is matched ZERO or MORE
297 > character denotes that
298 the following character should be taken literally. This is used where one of the
299 special characters (e.g. <SPAN
302 >) needs to be taken literally and
303 not as a special meta-character. Example: <SPAN
305 >"example\.com"</SPAN
307 sure the period is recognized only as a period (and not expanded to its
308 meta-character meaning of any single character).
330 > - Characters enclosed in brackets will be matched if
331 any of the enclosed characters are encountered. For instance, <SPAN
335 matches any numeric digit (zero through nine). As an example, we can combine
339 > to match any digit one of more times: <SPAN
364 > - parentheses are used to group a sub-expression,
365 or multiple sub-expressions.
390 > character works like an
394 > conditional statement. A match is successful if the
395 sub-expression on either side of <SPAN
398 > matches. As an example:
401 >"/(this|that) example/"</SPAN
402 > uses grouping and the bar character
403 and would match either <SPAN
405 >"this example"</SPAN
419 > These are just some of the ones you are likely to use when matching URLs with
423 >, and is a long way from a definitive
424 list. This is enough to get us started with a few simple examples which may
425 be more illuminating:</P
437 that uses the common combination of <SPAN
444 denote any character, zero or more times. In other words, any string at all.
445 So we start with a literal forward slash, then our regular expression pattern
449 >) another literal forward slash, the string
453 >, another forward slash, and lastly another
458 a directory path here. This will match any file with the path that has a
459 directory named <SPAN
466 any characters, and this could conceivably be more forward slashes, so it
467 might expand into a much longer looking path. For example, this could match:
470 >"/eye/hate/spammers/banners/annoy_me_please.gif"</SPAN
474 >"/banners/annoying.html"</SPAN
475 >, or almost an infinite number of other
476 possible combinations, just so it has <SPAN
482 > A now something a little more complex:</P
490 >/.*/adv((er)?ts?|ertis(ing|ements?))?/</TT
494 We have several literal forward slashes again (<SPAN
498 building another expression that is a file path statement. We have another
502 >, so we are matching against any conceivable sub-path, just so
503 it matches our expression. The only true literal that <SPAN
510 > our pattern is <SPAN
514 the forward slashes. What comes after the <SPAN
518 interesting part. </P
523 > means the preceding expression (either a
524 literal character or anything grouped with <SPAN
528 can exist or not, since this means either zero or one match. So
531 >"((er)?ts?|ertis(ing|ements?))"</SPAN
532 > is optional, as are the
533 individual sub-expressions: <SPAN
539 >"(ing|ements?)"</SPAN
550 >. We have two of those. For instance,
553 >"(ing|ements?)"</SPAN
554 >, can expand to match either <SPAN
567 >. What is being done here, is an
568 attempt at matching as many variations of <SPAN
570 >"advertisement"</SPAN
572 similar, as possible. So this would expand to match just <SPAN
588 >"advertisement"</SPAN
592 >"advertisements"</SPAN
593 >. You get the idea. But it would not match
596 >"advertizements"</SPAN
600 >). We could fix that by
601 changing our regular expression to:
604 >"/.*/adv((er)?ts?|erti(s|z)(ing|ements?))?/"</SPAN
605 >, which would then match
614 >/.*/advert[0-9]+\.(gif|jpe?g)</TT
618 another path statement with forward slashes. Anything in the square brackets
622 > can be matched. This is using <SPAN
626 shorthand expression to mean any digit one through nine. It is the same as
630 >. So any digit matches. The <SPAN
634 means one or more of the preceding expression must be included. The preceding
635 expression here is what is in the square brackets -- in this case, any digit
636 one through nine. Then, at the end, we have a grouping: <SPAN
640 This includes a <SPAN
643 >, so this needs to match the expression on
644 either side of that bar character also. A simple <SPAN
647 > on one side, and the other
648 side will in turn match either <SPAN
658 > means the letter <SPAN
662 can be matched once or not at all. So we are building an expression here to
663 match image GIF or JPEG type image file. It must include the literal
667 >, then one or more digits, and a <SPAN
671 (which is now a literal, and not a special character, since it is escaped
675 >), and lastly either <SPAN
685 >. Some possible matches would
688 >"//advert1.jpg"</SPAN
692 >"/nasty/ads/advert1234.gif"</SPAN
696 >"/banners/from/hell/advert99.jpg"</SPAN
697 >. It would not match
701 > (no leading slash), or
704 >"/adverts232.jpg"</SPAN
705 > (the expression does not include an
711 >"/advert1.jsp"</SPAN
716 in the expression anywhere).</P
718 > We are barely scratching the surface of regular expressions here so that you
719 can understand the default <SPAN
723 configuration files, and maybe use this knowledge to customize your own
724 installation. There is much, much more that can be done with regular
725 expressions. Now that you know enough to get started, you can learn more on
728 > More reading on Perl Compatible Regular expressions:
730 HREF="http://www.perldoc.com/perl5.6/pod/perlre.html"
732 >http://www.perldoc.com/perl5.6/pod/perlre.html</A
735 > For information on regular expression based substitutions and their applications
736 in filters, please see the <A
737 HREF="filter-file.html"
738 >filter file tutorial</A
752 >'s Internal Pages</H2
757 > proxies each requested
758 web page, it is easy for <SPAN
762 trap certain special URLs. In this way, we can talk directly to
767 configured, see how our rules are being applied, change these
768 rules and other configuration options, and even turn
772 > filtering off, all with
773 a web browser. </P
775 > The URLs listed below are the special ones that allow direct access
783 > must be running to access these. If
784 not, you will get a friendly error message. Internet access is not
803 HREF="http://config.privoxy.org/"
805 >http://config.privoxy.org/</A
810 > There is a shortcut: <A
815 doesn't provide a fall-back to a real page, in case the request is not
825 Show information about the current configuration, including viewing and
826 editing of actions files:
836 HREF="http://config.privoxy.org/show-status"
838 >http://config.privoxy.org/show-status</A
846 Show the source code version numbers:
856 HREF="http://config.privoxy.org/show-version"
858 >http://config.privoxy.org/show-version</A
866 Show the browser's request headers:
876 HREF="http://config.privoxy.org/show-request"
878 >http://config.privoxy.org/show-request</A
886 Show which actions apply to a URL and why:
896 HREF="http://config.privoxy.org/show-url-info"
898 >http://config.privoxy.org/show-url-info</A
906 Toggle Privoxy on or off. In this case, <SPAN
910 to run, but only as a pass-through proxy, with no actions taking place:
920 HREF="http://config.privoxy.org/toggle"
922 >http://config.privoxy.org/toggle</A
927 > Short cuts. Turn off, then on:
937 HREF="http://config.privoxy.org/toggle?set=disable"
939 >http://config.privoxy.org/toggle?set=disable</A
951 HREF="http://config.privoxy.org/toggle?set=enable"
953 >http://config.privoxy.org/toggle?set=enable</A
961 > These may be bookmarked for quick reference. See next. </P
969 >14.2.1. Bookmarklets</H3
971 > Below are some <SPAN
973 >"bookmarklets"</SPAN
974 > to allow you to easily access a
978 > version of some of <SPAN
982 special pages. They are designed for MS Internet Explorer, but should work
983 equally well in Netscape, Mozilla, and other browsers which support
984 JavaScript. They are designed to run directly from your bookmarks - not by
985 clicking the links below (although that should work for testing).</P
987 > To save them, right-click the link and choose <SPAN
989 >"Add to Favorites"</SPAN
993 >"Add Bookmark"</SPAN
994 > (Netscape). You will get a warning that
997 >"may not be safe"</SPAN
998 > - just click OK. Then you can run the
999 Bookmarklet directly from your favorites/bookmarks. For even faster access,
1000 you can put them on the <SPAN
1003 > bar (IE) or the <SPAN
1007 > (Netscape), and run them with a single click. </P
1015 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=enabled','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1017 >Privoxy - Enable</A
1024 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=disabled','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1026 >Privoxy - Disable</A
1033 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=toggle','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1035 >Privoxy - Toggle Privoxy</A
1036 > (Toggles between enabled and disabled)
1042 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y','ijbstatus','width=250,height=2,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1044 >Privoxy- View Status</A
1051 HREF="javascript:w=Math.floor(screen.width/2);h=Math.floor(screen.height*0.9);void(window.open('http://www.privoxy.org/actions/index.php?url='+escape(location.href),'Feedback','screenx='+w+',width='+w+',height='+h+',scrollbars=yes,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1053 >Privoxy - Submit Actions File Feedback</A
1060 HREF="javascript:void(window.open('http://config.privoxy.org/show-url-info?url='+escape(location.href),'Why').focus());"
1069 > Credit: The site which gave us the general idea for these bookmarklets is
1071 HREF="http://www.bookmarklets.com/"
1073 >www.bookmarklets.com</A
1075 have more information about bookmarklets. </P
1085 >14.3. Chain of Events</H2
1087 > Let's take a quick look at the basic sequence of events when a web page is
1088 requested by your browser and <SPAN
1098 > First, your web browser requests a web page. The browser knows to send
1099 the request to <SPAN
1102 >, which will in turn,
1103 relay the request to the remote web server after passing the following
1112 > traps any request for its own internal CGI
1113 pages (e.g http://p.p/) and sends the CGI page back to the browser.
1121 > checks to see if the URL
1123 HREF="actions-file.html#BLOCK"
1129 so, the URL is then blocked, and the remote web server will not be contacted.
1131 HREF="actions-file.html#HANDLE-AS-IMAGE"
1134 >"+handle-as-image"</SPAN
1137 is then checked and if it does not match, an
1141 > page is sent back. Otherwise, if it does match,
1142 an image is returned. The type of image depends on the setting of <A
1143 HREF="actions-file.html#SET-IMAGE-BLOCKER"
1146 >"+set-image-blocker"</SPAN
1149 (blank, checkerboard pattern, or an HTTP redirect to an image elsewhere).
1154 > Untrusted URLs are blocked. If URLs are being added to the
1158 > file, then that is done.
1163 > If the URL pattern matches the <A
1164 HREF="actions-file.html#FAST-REDIRECTS"
1167 >"+fast-redirects"</SPAN
1170 it is then processed. Unwanted parts of the requested URL are stripped.
1175 > Now the rest of the client browser's request headers are processed. If any
1176 of these match any of the relevant actions (e.g. <A
1177 HREF="actions-file.html#HIDE-USER-AGENT"
1180 >"+hide-user-agent"</SPAN
1183 etc.), headers are suppressed or forged as determined by these actions and
1189 > Now the web server starts sending its response back (i.e. typically a web page and related
1195 > First, the server headers are read and processed to determine, among other
1196 things, the MIME type (document type) and encoding. The headers are then
1197 filtered as determined by the
1199 HREF="actions-file.html#CRUNCH-INCOMING-COOKIES"
1202 >"+crunch-incoming-cookies"</SPAN
1206 HREF="actions-file.html#SESSION-COOKIES-ONLY"
1209 >"+session-cookies-only"</SPAN
1213 HREF="actions-file.html#DOWNGRADE-HTTP-VERSION"
1216 >"+downgrade-http-version"</SPAN
1225 HREF="actions-file.html#KILL-POPUPS"
1228 >"+kill-popups"</SPAN
1231 action applies, and it is an HTML or JavaScript document, the popup-code in the
1232 response is filtered on-the-fly as it is received.
1238 HREF="actions-file.html#FILTER"
1245 HREF="actions-file.html#DEANIMATE-GIFS"
1248 >"+deanimate-gifs"</SPAN
1251 action applies (and the document type fits the action), the rest of the page is
1252 read into memory (up to a configurable limit). Then the filter rules (from
1256 >) are processed against the buffered
1257 content. Filters are applied in the order they are specified in the
1261 > file. Animated GIFs, if present, are
1262 reduced to either the first or last frame, depending on the action
1263 setting.The entire page, which is now filtered, is then sent by
1267 > back to your browser.
1271 HREF="actions-file.html#FILTER"
1278 HREF="actions-file.html#DEANIMATE-GIFS"
1281 >"+deanimate-gifs"</SPAN
1287 > passes the raw data through
1288 to the client browser as it becomes available.
1293 > As the browser receives the now (probably filtered) page content, it
1294 reads and then requests any URLs that may be embedded within the page
1295 source, e.g. ad images, stylesheets, JavaScript, other HTML documents (e.g.
1296 frames), sounds, etc. For each of these objects, the browser issues a new
1297 request. And each such request is in turn processed as above. Note that a
1298 complex web page may have many such embedded URLs.
1311 >14.4. Anatomy of an Action</H2
1318 HREF="actions-file.html#ACTIONS"
1321 HREF="actions-file.html#FILTER"
1324 to any given URL can be complex, and not always so
1325 easy to understand what is happening. And sometimes we need to be able to
1336 doing. Especially, if something <SPAN
1340 is causing us a problem inadvertently. It can be a little daunting to look at
1341 the actions and filters files themselves, since they tend to be filled with
1343 HREF="appendix.html#REGEX"
1344 >regular expressions</A
1345 > whose consequences are not
1346 always so obvious. </P
1348 > One quick test to see if <SPAN
1351 > is causing a problem
1352 or not, is to disable it temporarily. This should be the first troubleshooting
1354 HREF="appendix.html#BOOKMARKLETS"
1355 >the Bookmarklets</A
1356 > section on a quick
1357 and easy way to do this (be sure to flush caches afterward!). Looking at the
1358 logs is a good idea too.</P
1365 HREF="http://config.privoxy.org/show-url-info"
1367 >http://config.privoxy.org/show-url-info</A
1369 page that can show us very specifically how <SPAN
1373 are being applied to any given URL. This is a big help for troubleshooting.</P
1375 > First, enter one URL (or partial URL) at the prompt, and then
1380 how the current configuration will handle it. This will not
1381 help with filtering effects (i.e. the <A
1382 HREF="actions-file.html#FILTER"
1391 > file since this is handled very
1392 differently and not so easy to trap! It also will not tell you about any other
1393 URLs that may be embedded within the URL you are testing. For instance, images
1394 such as ads are expressed as URLs within the raw page source of HTML pages. So
1395 you will only get info for the actual URL that is pasted into the prompt area
1396 -- not any sub-URLs. If you want to know about embedded URLs like ads, you
1397 will have to dig those out of the HTML source. Use your browser's <SPAN
1401 > option for this. Or right click on the ad, and grab the
1404 > Let's try an example, <A
1405 HREF="http://google.com"
1409 and look at it one section at a time:</P
1419 > Matches for http://google.com:
1421 In file: default.action <SPAN
1431 -crunch-outgoing-cookies
1432 -crunch-incoming-cookies
1433 +deanimate-gifs{last}
1434 -downgrade-http-version
1438 -filter{shockwave-flash}
1439 -filter{crude-parental}
1440 +filter{html-annoyances}
1441 +filter{js-annoyances}
1442 +filter{content-cookies}
1444 +filter{refresh-tags}
1446 +filter{banners-by-size}
1447 +hide-forwarded-for-headers
1448 +hide-from-header{block}
1449 +hide-referer{forge}
1454 +prevent-compression
1457 +session-cookies-only
1458 +set-image-blocker{pattern} }
1461 { -session-cookies-only }
1467 In file: user.action <SPAN
1474 (no matches in this file) </PRE
1480 > This tells us how we have defined our
1482 HREF="actions-file.html#ACTIONS"
1488 which ones match for our example, <SPAN
1491 >. The first listing
1492 is any matches for the <TT
1494 >standard.action</TT
1499 >. Then next is <SPAN
1506 > file. The large, multi-line listing,
1507 is how the actions are set to match for all URLs, i.e. our default settings.
1508 If you look at your <SPAN
1511 > file, this would be the section
1512 just below the <SPAN
1515 > section near the top. This will apply to
1516 all URLs as signified by the single forward slash at the end of the listing
1522 > But we can define additional actions that would be exceptions to these general
1523 rules, and then list specific URLs (or patterns) that these exceptions would
1524 apply to. Last match wins. Just below this then are two explicit matches for
1527 >".google.com"</SPAN
1528 >. The first is negating our previous cookie setting,
1530 HREF="actions-file.html#SESSION-COOKIES-ONLY"
1533 >"+session-cookies-only"</SPAN
1536 (i.e. not persistent). So we will allow persistent cookies for google. The
1545 HREF="actions-file.html#FAST-REDIRECTS"
1548 >"+fast-redirects"</SPAN
1551 action, allowing this to take place unmolested. Note that there is a leading
1554 >".google.com"</SPAN
1555 >. This will match any hosts and
1556 sub-domains, in the google.com domain also, such as
1559 >"www.google.com"</SPAN
1560 >. So, apparently, we have these two actions
1561 defined somewhere in the lower part of our <TT
1568 > is referenced somewhere in these latter
1574 > file, we again have no hits.</P
1576 > And finally we pull it all together in the bottom section and summarize how
1580 > is applying all its <SPAN
1597 > Final results:
1601 -crunch-outgoing-cookies
1602 -crunch-incoming-cookies
1603 +deanimate-gifs{last}
1604 -downgrade-http-version
1608 -filter{shockwave-flash}
1609 -filter{crude-parental}
1610 +filter{html-annoyances}
1611 +filter{js-annoyances}
1612 +filter{content-cookies}
1614 +filter{refresh-tags}
1616 +filter{banners-by-size}
1617 +hide-forwarded-for-headers
1618 +hide-from-header{block}
1619 +hide-referer{forge}
1624 +prevent-compression
1627 -session-cookies-only
1628 +set-image-blocker{pattern} </PRE
1634 > Notice the only difference here to the previous listing, is to
1637 >"fast-redirects"</SPAN
1640 >"session-cookies-only"</SPAN
1643 > Now another example, <SPAN
1645 >"ad.doubleclick.net"</SPAN
1656 > { +block +handle-as-image }
1659 { +block +handle-as-image }
1662 { +block +handle-as-image }
1663 .doubleclick.net</PRE
1669 > We'll just show the interesting part here, the explicit matches. It is
1670 matched three different times. Each as an <SPAN
1672 >"+block +handle-as-image"</SPAN
1674 which is the expanded form of one of our aliases that had been defined as:
1677 >"+imageblock"</SPAN
1679 HREF="actions-file.html#ALIASES"
1685 the first section of the actions file and typically used to combine more
1686 than one action.)</P
1688 > Any one of these would have done the trick and blocked this as an unwanted
1689 image. This is unnecessarily redundant since the last case effectively
1690 would also cover the first. No point in taking chances with these guys
1691 though ;-) Note that if you want an ad or obnoxious
1692 URL to be invisible, it should be defined as <SPAN
1694 >"ad.doubleclick.net"</SPAN
1696 is done here -- as both a <A
1697 HREF="actions-file.html#BLOCK"
1711 HREF="actions-file.html#HANDLE-AS-IMAGE"
1714 >"+handle-as-image"</SPAN
1717 The custom alias <SPAN
1719 >"+imageblock"</SPAN
1720 > just simplifies the process and make
1721 it more readable.</P
1723 > One last example. Let's try <SPAN
1725 >"http://www.rhapsodyk.net/adsl/HOWTO/"</SPAN
1727 This one is giving us problems. We are getting a blank page. Hmmm ...</P
1737 > Matches for http://www.rhapsodyk.net/adsl/HOWTO/:
1739 In file: default.action <SPAN
1749 -crunch-incoming-cookies
1750 -crunch-outgoing-cookies
1752 -downgrade-http-version
1754 +filter{html-annoyances}
1755 +filter{js-annoyances}
1756 +filter{kill-popups}
1759 +filter{banners-by-size}
1762 +hide-forwarded-for-headers
1763 +hide-from-header{block}
1764 +hide-referer{forge}
1768 +prevent-compression
1771 +session-cookies-only
1772 +set-image-blocker{blank} }
1775 { +block +handle-as-image }
1789 we did not want this at all! Now we see why we get the blank page. We could
1790 now add a new action below this that explicitly does <SPAN
1804 various ways to handle such exceptions. Example:</P
1821 > Now the page displays ;-) Be sure to flush your browser's caches when
1822 making such changes. Or, try using <TT
1827 > But now what about a situation where we get no explicit matches like
1838 > { +block +handle-as-image }
1845 > That actually was very telling and pointed us quickly to where the problem
1846 was. If you don't get this kind of match, then it means one of the default
1847 rules in the first section is causing the problem. This would require some
1848 guesswork, and maybe a little trial and error to isolate the offending rule.
1849 One likely cause would be one of the <SPAN
1853 tend to be harder to troubleshoot. Try adding the URL for the site to one of
1854 aliases that turn off <SPAN
1869 .worldpay.com # for quietpc.com
1887 >"{ -filter -session-cookies-only }"</SPAN
1889 Or you could do your own exception to negate filtering: </P
1906 > This would turn off all filtering for that site. This would probably be most
1907 appropriately put in <TT
1913 > Images that are inexplicably being blocked, may well be hitting the
1916 >"+filter{banners-by-size}"</SPAN
1917 > rule, which assumes
1918 that images of certain sizes are ad banners (works well most of the time
1919 since these tend to be standardized).</P
1924 > is an alias that disables most actions. This can be
1925 used as a last resort for problem sites. Remember to flush caches! If this
1926 still does not work, you will have to go through the remaining actions one by
1927 one to find which one(s) is causing the problem.</P
1935 SUMMARY="Footer navigation table"