7 CONTENT="Modular DocBook HTML Stylesheet Version 1.64
10 TITLE="Privoxy User Manual"
11 HREF="index.html"><LINK
14 HREF="seealso.html"><LINK
17 HREF="../p_doc.css"></HEAD
36 >Privoxy User Manual</TH
76 >14.1. Regular Expressions</A
82 > uses Perl-style <SPAN
87 HREF="actions-file.html"
91 HREF="filter-file.html"
95 HREF="http://www.pcre.org/"
100 HREF="http://www.oesterhelt.org/pcrs/"
105 > If you are reading this, you probably don't understand what <SPAN
109 > are, or what they can do. So this will be a very brief
110 introduction only. A full explanation would require a <A
111 HREF="http://www.oreilly.com/catalog/regex/"
116 > Regular expressions provide a language to describe patterns that can be
117 run against strings of characters (letter, numbers, etc), to see if they
118 match the string or not. The patterns are themselves (sometimes complex)
119 strings of literal characters, combined with wild-cards, and other special
120 characters, called meta-characters. The <SPAN
122 >"meta-characters"</SPAN
124 special meanings and are used to build complex patterns to be matched against.
125 Perl Compatible Regular Expressions are an especially convenient
129 > of the regular expression language.</P
131 > To make a simple analogy, we do something similar when we use wild-card
132 characters when listing files with the <B
139 > matches all filenames. The <SPAN
143 character here is the asterisk which matches any and all characters. We can be
144 more specific and use <TT
147 > to match just individual
150 >"dir file?.text"</SPAN
158 >, etc. We are pattern
159 matching, using a similar technique to <SPAN
161 >"regular expressions"</SPAN
164 > Regular expressions do essentially the same thing, but are much, much more
165 powerful. There are many more <SPAN
167 >"special characters"</SPAN
169 building complex patterns however. Let's look at a few of the common ones,
170 and then some examples:</P
182 > - Matches any single character, e.g. <SPAN
217 > - The preceding character or expression is matched ZERO or ONE
237 > - The preceding character or expression is matched ONE or MORE
257 > - The preceding character or expression is matched ZERO or MORE
280 > character denotes that
281 the following character should be taken literally. This is used where one of the
282 special characters (e.g. <SPAN
285 >) needs to be taken literally and
286 not as a special meta-character. Example: <SPAN
288 >"example\.com"</SPAN
290 sure the period is recognized only as a period (and not expanded to its
291 meta-character meaning of any single character).
310 > - Characters enclosed in brackets will be matched if
311 any of the enclosed characters are encountered. For instance, <SPAN
315 matches any numeric digit (zero through nine). As an example, we can combine
319 > to match any digit one of more times: <SPAN
341 > - parentheses are used to group a sub-expression,
342 or multiple sub-expressions.
364 > character works like an
368 > conditional statement. A match is successful if the
369 sub-expression on either side of <SPAN
372 > matches. As an example:
375 >"/(this|that) example/"</SPAN
376 > uses grouping and the bar character
377 and would match either <SPAN
379 >"this example"</SPAN
393 > These are just some of the ones you are likely to use when matching URLs with
397 >, and is a long way from a definitive
398 list. This is enough to get us started with a few simple examples which may
399 be more illuminating:</P
408 that uses the common combination of <SPAN
415 denote any character, zero or more times. In other words, any string at all.
416 So we start with a literal forward slash, then our regular expression pattern
420 >) another literal forward slash, the string
424 >, another forward slash, and lastly another
429 a directory path here. This will match any file with the path that has a
430 directory named <SPAN
437 any characters, and this could conceivably be more forward slashes, so it
438 might expand into a much longer looking path. For example, this could match:
441 >"/eye/hate/spammers/banners/annoy_me_please.gif"</SPAN
445 >"/banners/annoying.html"</SPAN
446 >, or almost an infinite number of other
447 possible combinations, just so it has <SPAN
453 > A now something a little more complex:</P
459 >/.*/adv((er)?ts?|ertis(ing|ements?))?/</TT
462 We have several literal forward slashes again (<SPAN
466 building another expression that is a file path statement. We have another
470 >, so we are matching against any conceivable sub-path, just so
471 it matches our expression. The only true literal that <I
475 > our pattern is <SPAN
479 the forward slashes. What comes after the <SPAN
483 interesting part. </P
488 > means the preceding expression (either a
489 literal character or anything grouped with <SPAN
493 can exist or not, since this means either zero or one match. So
496 >"((er)?ts?|ertis(ing|ements?))"</SPAN
497 > is optional, as are the
498 individual sub-expressions: <SPAN
504 >"(ing|ements?)"</SPAN
515 >. We have two of those. For instance,
518 >"(ing|ements?)"</SPAN
519 >, can expand to match either <SPAN
529 >. What is being done here, is an
530 attempt at matching as many variations of <SPAN
532 >"advertisement"</SPAN
534 similar, as possible. So this would expand to match just <SPAN
550 >"advertisement"</SPAN
554 >"advertisements"</SPAN
555 >. You get the idea. But it would not match
558 >"advertizements"</SPAN
562 >). We could fix that by
563 changing our regular expression to:
566 >"/.*/adv((er)?ts?|erti(s|z)(ing|ements?))?/"</SPAN
567 >, which would then match
574 >/.*/advert[0-9]+\.(gif|jpe?g)</TT
577 another path statement with forward slashes. Anything in the square brackets
581 > can be matched. This is using <SPAN
585 shorthand expression to mean any digit one through nine. It is the same as
589 >. So any digit matches. The <SPAN
593 means one or more of the preceding expression must be included. The preceding
594 expression here is what is in the square brackets -- in this case, any digit
595 one through nine. Then, at the end, we have a grouping: <SPAN
599 This includes a <SPAN
602 >, so this needs to match the expression on
603 either side of that bar character also. A simple <SPAN
606 > on one side, and the other
607 side will in turn match either <SPAN
617 > means the letter <SPAN
621 can be matched once or not at all. So we are building an expression here to
622 match image GIF or JPEG type image file. It must include the literal
626 >, then one or more digits, and a <SPAN
630 (which is now a literal, and not a special character, since it is escaped
634 >), and lastly either <SPAN
644 >. Some possible matches would
647 >"//advert1.jpg"</SPAN
651 >"/nasty/ads/advert1234.gif"</SPAN
655 >"/banners/from/hell/advert99.jpg"</SPAN
656 >. It would not match
660 > (no leading slash), or
663 >"/adverts232.jpg"</SPAN
664 > (the expression does not include an
670 >"/advert1.jsp"</SPAN
675 in the expression anywhere).</P
677 > We are barely scratching the surface of regular expressions here so that you
678 can understand the default <SPAN
682 configuration files, and maybe use this knowledge to customize your own
683 installation. There is much, much more that can be done with regular
684 expressions. Now that you know enough to get started, you can learn more on
687 > More reading on Perl Compatible Regular expressions:
689 HREF="http://www.perldoc.com/perl5.6/pod/perlre.html"
691 >http://www.perldoc.com/perl5.6/pod/perlre.html</A
694 > For information on regular expression based substititions and their applications
695 in filters, please see the <A
696 HREF="filter-file.html"
697 >filter file tutorial</A
710 >'s Internal Pages</A
716 > proxies each requested
717 web page, it is easy for <SPAN
721 trap certain special URLs. In this way, we can talk directly to
726 configured, see how our rules are being applied, change these
727 rules and other configuration options, and even turn
731 > filtering off, all with
732 a web browser. </P
734 > The URLs listed below are the special ones that allow direct access
742 > must be running to access these. If
743 not, you will get a friendly error message. Internet access is not
762 HREF="http://config.privoxy.org/"
764 >http://config.privoxy.org/</A
769 > There is a shortcut: <A
774 doesn't provide a fallback to a real page, in case the request is not
784 Show information about the current configuration, including viewing and
785 editing of actions files:
795 HREF="http://config.privoxy.org/show-status"
797 >http://config.privoxy.org/show-status</A
805 Show the source code version numbers:
815 HREF="http://config.privoxy.org/show-version"
817 >http://config.privoxy.org/show-version</A
825 Show the browser's request headers:
835 HREF="http://config.privoxy.org/show-request"
837 >http://config.privoxy.org/show-request</A
845 Show which actions apply to a URL and why:
855 HREF="http://config.privoxy.org/show-url-info"
857 >http://config.privoxy.org/show-url-info</A
865 Toggle Privoxy on or off. In this case, <SPAN
869 to run, but only as a pass-through proxy, with no actions taking place:
879 HREF="http://config.privoxy.org/toggle"
881 >http://config.privoxy.org/toggle</A
886 > Short cuts. Turn off, then on:
896 HREF="http://config.privoxy.org/toggle?set=disable"
898 >http://config.privoxy.org/toggle?set=disable</A
910 HREF="http://config.privoxy.org/toggle?set=enable"
912 >http://config.privoxy.org/toggle?set=enable</A
920 > These may be bookmarked for quick reference. See next. </P
927 >14.2.1. Bookmarklets</A
930 > Below are some <SPAN
932 >"bookmarklets"</SPAN
933 > to allow you to easily access a
937 > version of some of <SPAN
941 special pages. They are designed for MS Internet Explorer, but should work
942 equally well in Netscape, Mozilla, and other browsers which support
943 JavaScript. They are designed to run directly from your bookmarks - not by
944 clicking the links below (although that should work for testing).</P
946 > To save them, right-click the link and choose <SPAN
948 >"Add to Favorites"</SPAN
952 >"Add Bookmark"</SPAN
953 > (Netscape). You will get a warning that
956 >"may not be safe"</SPAN
957 > - just click OK. Then you can run the
958 Bookmarklet directly from your favorites/bookmarks. For even faster access,
959 you can put them on the <SPAN
962 > bar (IE) or the <SPAN
966 > (Netscape), and run them with a single click. </P
974 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=enabled','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
983 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=disabled','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
985 >Privoxy - Disable</A
992 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y&set=toggle','ijbstatus','width=250,height=100,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
994 >Privoxy - Toggle Privoxy</A
995 > (Toggles between enabled and disabled)
1001 HREF="javascript:void(window.open('http://config.privoxy.org/toggle?mini=y','ijbstatus','width=250,height=2,resizable=yes,scrollbars=no,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1003 >Privoxy- View Status</A
1010 HREF="javascript:w=Math.floor(screen.width/2);h=Math.floor(screen.height*0.9);void(window.open('http://www.privoxy.org/actions/index.php?url='+escape(location.href),'Feedback','screenx='+w+',width='+w+',height='+h+',scrollbars=yes,toolbar=no,location=no,directories=no,status=no,menubar=no,copyhistory=no').focus());"
1012 >Privoxy - Submit Actions File Feedback</A
1019 > Credit: The site which gave us the general idea for these bookmarklets is
1021 HREF="http://www.bookmarklets.com"
1023 >www.bookmarklets.com</A
1025 have more information about bookmarklets. </P
1034 >14.3. Chain of Events</A
1037 > Let's take a quick look at the basic sequence of events when a web page is
1038 requested by your browser and <SPAN
1048 > First, your web browser requests a web page. The browser knows to send
1049 the request to <SPAN
1052 >, which will in turn,
1053 relay the request to the remote web server after passing the following
1062 > traps any request for its own internal CGI
1063 pages (e.g http://p.p/) and sends the CGI page back to the browser.
1071 > checks to see if the URL
1073 HREF="actions-file.html#BLOCK"
1079 so, the URL is then blocked, and the remote web server will not be contacted.
1081 HREF="actions-file.html#HANDLE-AS-IMAGE"
1084 >"+handle-as-image"</SPAN
1087 is then checked and if it does not match, an
1091 > page is sent back. Otherwise, if it does match,
1092 an image is returned. The type of image depends on the setting of <A
1093 HREF="actions-file.html#SET-IMAGE-BLOCKER"
1096 >"+set-image-blocker"</SPAN
1099 (blank, checkerboard pattern, or an HTTP redirect to an image elsewhere).
1104 > Untrusted URLs are blocked. If URLs are being added to the
1108 > file, then that is done.
1113 > If the URL pattern matches the <A
1114 HREF="actions-file.html#FAST-REDIRECTS"
1117 >"+fast-redirects"</SPAN
1120 it is then processed. Unwanted parts of the requested URL are stripped.
1125 > Now the rest of the client browser's request headers are processed. If any
1126 of these match any of the relevant actions (e.g. <A
1127 HREF="actions-file.html#HIDE-USER-AGENT"
1130 >"+hide-user-agent"</SPAN
1133 etc.), headers are suppressed or forged as determined by these actions and
1139 > Now the web server starts sending its response back (i.e. typically a web page and related
1145 > First, the server headers are read and processed to determine, among other
1146 things, the MIME type (document type) and encoding. The headers are then
1147 filtered as deterimed by the
1149 HREF="actions-file.html#CRUNCH-INCOMING-COOKIES"
1152 >"+crunch-incoming-cookies"</SPAN
1156 HREF="actions-file.html#SESSION-COOKIES-ONLY"
1159 >"+session-cookies-only"</SPAN
1163 HREF="actions-file.html#DOWNGRADE-HTTP-VERSION"
1166 >"+downgrade-http-version"</SPAN
1175 HREF="actions-file.html#KILL-POPUPS"
1178 >"+kill-popups"</SPAN
1181 action applies, and it is an HTML or JavaScript document, the popup-code in the
1182 response is filtered on-the-fly as it is received.
1188 HREF="actions-file.html#FILTER"
1195 HREF="actions-file.html#DEANIMATE-GIFS"
1198 >"+deanimate-gifs"</SPAN
1201 action applies (and the document type fits the action), the rest of the page is
1202 read into memory (up to a configurable limit). Then the filter rules (from
1206 >) are processed against the buffered
1207 content. Filters are applied in the order they are specified in the
1211 > file. Animated GIFs, if present, are
1212 reduced to either the first or last frame, depending on the action
1213 setting.The entire page, which is now filtered, is then sent by
1217 > back to your browser.
1221 HREF="actions-file.html#FILTER"
1228 HREF="actions-file.html#DEANIMATE-GIFS"
1231 >"+deanimate-gifs"</SPAN
1237 > passes the raw data through
1238 to the client browser as it becomes available.
1243 > As the browser receives the now (probably filtered) page content, it
1244 reads and then requests any URLs that may be embedded within the page
1245 source, e.g. ad images, stylesheets, JavaScript, other HTML documents (e.g.
1246 frames), sounds, etc. For each of these objects, the browser issues a new
1247 request. And each such request is in turn processed as above. Note that a
1248 complex web page may have many such embedded URLs.
1260 >14.4. Anatomy of an Action</A
1268 HREF="actions-file.html#ACTIONS"
1271 HREF="actions-file.html#FILTER"
1274 to any given URL can be complex, and not always so
1275 easy to understand what is happening. And sometimes we need to be able to
1283 doing. Especially, if something <SPAN
1287 is causing us a problem inadvertently. It can be a little daunting to look at
1288 the actions and filters files themselves, since they tend to be filled with
1290 HREF="appendix.html#REGEX"
1291 >regular expressions</A
1292 > whose consequences are not
1293 always so obvious. </P
1295 > One quick test to see if <SPAN
1298 > is causing a problem
1299 or not, is to disable it temporarily. This should be the first troubleshooting
1301 HREF="appendix.html#BOOKMARKLETS"
1302 >the Bookmarklets</A
1303 > section on a quick
1304 and easy way to do this (be sure to flush caches afterward!).</P
1311 HREF="http://config.privoxy.org/show-url-info"
1313 >http://config.privoxy.org/show-url-info</A
1315 page that can show us very specifically how <SPAN
1319 are being applied to any given URL. This is a big help for troubleshooting.</P
1321 > First, enter one URL (or partial URL) at the prompt, and then
1326 how the current configuration will handle it. This will not
1327 help with filtering effects (i.e. the <A
1328 HREF="actions-file.html#FILTER"
1337 > file since this is handled very
1338 differently and not so easy to trap! It also will not tell you about any other
1339 URLs that may be embedded within the URL you are testing. For instance, images
1340 such as ads are expressed as URLs within the raw page source of HTML pages. So
1341 you will only get info for the actual URL that is pasted into the prompt area
1342 -- not any sub-URLs. If you want to know about embedded URLs like ads, you
1343 will have to dig those out of the HTML source. Use your browser's <SPAN
1347 > option for this. Or right click on the ad, and grab the
1350 > Let's try an example, <A
1351 HREF="http://google.com"
1355 and look at it one section at a time:</P
1365 > Matches for http://google.com:
1367 In file: default.action <SPAN
1377 -crunch-outgoing-cookies
1378 -crunch-incoming-cookies
1379 +deanimate-gifs{last}
1380 -downgrade-http-version
1384 -filter{shockwave-flash}
1385 -filter{crude-parental}
1386 +filter{html-annoyances}
1387 +filter{js-annoyances}
1388 +filter{content-cookies}
1390 +filter{refresh-tags}
1392 +filter{banners-by-size}
1393 +hide-forwarded-for-headers
1394 +hide-from-header{block}
1395 +hide-referer{forge}
1400 +prevent-compression
1403 +session-cookies-only
1404 +set-image-blocker{pattern} }
1407 { -session-cookies-only }
1413 In file: user.action <SPAN
1420 (no matches in this file) </PRE
1426 > This tells us how we have defined our
1428 HREF="actions-file.html#ACTIONS"
1434 which ones match for our example, <SPAN
1437 >. The first listing
1438 is any matches for the <TT
1440 >standard.action</TT
1445 >. Then next is <SPAN
1452 > file. The large, multi-line listing,
1453 is how the actions are set to match for all URLs, i.e. our default settings.
1454 If you look at your <SPAN
1457 > file, this would be the section
1458 just below the <SPAN
1461 > section near the top. This will apply to
1462 all URLs as signified by the single forward slash at the end of the listing
1468 > But we can define additional actions that would be exceptions to these general
1469 rules, and then list specific URLs (or patterns) that these exceptions would
1470 apply to. Last match wins. Just below this then are two explicit matches for
1473 >".google.com"</SPAN
1474 >. The first is negating our previous cookie setting,
1476 HREF="actions-file.html#SESSION-COOKIES-ONLY"
1479 >"+session-cookies-only"</SPAN
1482 (i.e. not persistent). So we will allow persistent cookies for google. The
1488 HREF="actions-file.html#FAST-REDIRECTS"
1491 >"+fast-redirects"</SPAN
1494 action, allowing this to take place unmolested. Note that there is a leading
1497 >".google.com"</SPAN
1498 >. This will match any hosts and
1499 sub-domains, in the google.com domain also, such as
1502 >"www.google.com"</SPAN
1503 >. So, apparently, we have these two actions
1504 defined somewhere in the lower part of our <TT
1511 > is referenced somewhere in these latter
1517 > file, we again have no hits.</P
1519 > And finally we pull it all together in the bottom section and summarize how
1523 > is applying all its <SPAN
1540 > Final results:
1544 -crunch-outgoing-cookies
1545 -crunch-incoming-cookies
1546 +deanimate-gifs{last}
1547 -downgrade-http-version
1551 -filter{shockwave-flash}
1552 -filter{crude-parental}
1553 +filter{html-annoyances}
1554 +filter{js-annoyances}
1555 +filter{content-cookies}
1557 +filter{refresh-tags}
1559 +filter{banners-by-size}
1560 +hide-forwarded-for-headers
1561 +hide-from-header{block}
1562 +hide-referer{forge}
1567 +prevent-compression
1570 -session-cookies-only
1571 +set-image-blocker{pattern} </PRE
1577 > Notice the only difference here to the previous listing, is to
1580 >"fast-redirects"</SPAN
1583 >"session-cookies-only"</SPAN
1586 > Now another example, <SPAN
1588 >"ad.doubleclick.net"</SPAN
1599 > { +block +handle-as-image }
1602 { +block +handle-as-image }
1605 { +block +handle-as-image }
1606 .doubleclick.net</PRE
1612 > We'll just show the interesting part here, the explicit matches. It is
1613 matched three different times. Each as an <SPAN
1615 >"+block +handle-as-image"</SPAN
1617 which is the expanded form of one of our aliases that had been defined as:
1620 >"+imageblock"</SPAN
1622 HREF="actions-file.html#ALIASES"
1628 the first section of the actions file and typically used to combine more
1629 than one action.)</P
1631 > Any one of these would have done the trick and blocked this as an unwanted
1632 image. This is unnecessarily redundant since the last case effectively
1633 would also cover the first. No point in taking chances with these guys
1634 though ;-) Note that if you want an ad or obnoxious
1635 URL to be invisible, it should be defined as <SPAN
1637 >"ad.doubleclick.net"</SPAN
1639 is done here -- as both a <A
1640 HREF="actions-file.html#BLOCK"
1651 HREF="actions-file.html#HANDLE-AS-IMAGE"
1654 >"+handle-as-image"</SPAN
1657 The custom alias <SPAN
1659 >"+imageblock"</SPAN
1660 > just simplifies the process and make
1661 it more readable.</P
1663 > One last example. Let's try <SPAN
1665 >"http://www.rhapsodyk.net/adsl/HOWTO/"</SPAN
1667 This one is giving us problems. We are getting a blank page. Hmmm...</P
1677 > Matches for http://www.rhapsodyk.net/adsl/HOWTO/:
1679 In file: default.action <SPAN
1689 -crunch-incoming-cookies
1690 -crunch-outgoing-cookies
1692 -downgrade-http-version
1694 +filter{html-annoyances}
1695 +filter{js-annoyances}
1696 +filter{kill-popups}
1699 +filter{banners-by-size}
1702 +hide-forwarded-for-headers
1703 +hide-from-header{block}
1704 +hide-referer{forge}
1708 +prevent-compression
1711 +session-cookies-only
1712 +set-image-blocker{blank} }
1715 { +block +handle-as-image }
1729 we did not want this at all! Now we see why we get the blank page. We could
1730 now add a new action below this that explicitly does <I
1741 various ways to handle such exceptions. Example:</P
1758 > Now the page displays ;-) Be sure to flush your browser's caches when
1759 making such changes. Or, try using <TT
1764 > But now what about a situation where we get no explicit matches like
1775 > { +block +handle-as-image }
1782 > That actually was very telling and pointed us quickly to where the problem
1783 was. If you don't get this kind of match, then it means one of the default
1784 rules in the first section is causing the problem. This would require some
1785 guesswork, and maybe a little trial and error to isolate the offending rule.
1786 One likely cause would be one of the <SPAN
1790 adding the URL for the site to one of aliases that turn off <SPAN
1805 .worldpay.com # for quietpc.com
1823 >"{ -filter -session-cookies-only }"</SPAN
1825 Or you could do your own exception to negate filtering: </P
1842 > This would probably be most appropriately put in <TT
1846 for local site exceptions.</P
1851 > is an alias that disables most actions. This can be
1852 used as a last resort for problem sites. Remember to flush caches! If this
1853 still does not work, you will have to go through the remaining actions one by
1854 one to find which one(s) is causing the problem.</P