Contents

• How it works
• How ASProxy works
• Background of process
• Tracing Background Working
• A Review on the Main Functions
• Implementation

How it works

Data is usually filtered in three ways: The first and the simplest way is through controlling the incoming and outgoing addresses. These addresses are stored in a black list. If one of these addresses is detected when requested or responded the filtering software will block the data transfer. The second way is like the almost like the first, but there is a list containing the unallowed words. The filtering software will automatically detect these words in the address and block the data. In the third way, in addition to the first and second ways, a search is done into the received data. The search engine will search for the certain links, addresses and words and if the number of these items reaches the certain limit, the page will be filtered. This way of filtering is more reliable but lessens the network speed, thus it is usually used in non-public centers where there is a need to have more control over the users' activities. All these three ways work based on one a list of certain words and addresses. And all the filter crossing tools let you cross the filters that work with the fist and second ways. Notice that the third way is defeated in a few cases.

How ASProxy works

The users must enter the site on which ASProxy is installed. The URL is entered in a text box provided and the request is sent correctly depending on how much the site has been popular and recognized by the filtering system. The ASProxy engine receives the URL and processes the request. After several-stage processes, the whole addresses of the requested page are converted and sent to the ASProxy-installed site. The result is magnificent, because the site will not be detected as an anti-filtering one and it will not be blocked. Below is the diagram of the process:

Background

ASProxy tool consists of the following processing classes taking the responsibilities as explained for each:

"DataManagement" section:

"DataManagement" section includes classes which process raw data from web.

• "ASProxyEngine" class: This class receives the requests and sends them for processing and returns the result to the requester. This class encapsulates all necessary activities with its properties and methods.
• "WebDataCore" class: Sends and receives the requests from web. This class does all necessary configurations to get actual data from web. This class is ASProxys core.
• "CookieManager" class: Saves and restores cookies among several requests. "WebDataCore" class always use this class.
• "HttpCompressor" class: Compresses the pages to increases the transfer speed. Compression can be enabled from the checkbox in "default" page. (Disabled by default for compatibility reasons)
• "ResumableDownload" class: Sends any data to the user (which we call "Download"). This class makes the resume-support downloads possible.

"DataProcessor" section:

"DataProcessor" section includes classes to process common MIME content types that are used in HTML pages.

• "Processors" class: This class especially focuses on encoding type conversions.

This section includes some subsections for better classification. The class names in this section follow these rules for these suffixes:

Parser: Classes defined with this suffix, find positions of tags in html or changes their properties.
Processor: Classes defined with this suffix, perform all their required operations. The "ASProxyEngine" class uses these classes directly.
Replacer: Classes defined with this suffix, change or replace a part of text or data (usually Html). To do this, these classes use their implemented methods.

The "DataProcessor" section operations are implemented in three subsections.

• HTML Classes: The classes of this part perform all necessary operations to remove or replace HTML codes in a requested page. This is the most important place in ASProxy to get desired data in correct format.
• JavaScript Classes: The classes of this part perform operations on JavaScript codes. This part is the hardest in implementation in ASProxy.
• CSS Classes: Perform necessary operations to replace addresses in CSS files. The main goal of this part is to replace "background" attributes.

"Configurations" section:

This section includes classes which save or restore saved data.

• "ASProxyConfig" class: Saves and restore ASProxy settings among requests. The default settings are retrieved from "Web.config" and user changes are retrieved from the cookie.
• "SiteExceptions" class: Stores unhandled errors list for future debugging. Error logging is stopped by default and can be changed in "Web.config" file.

"General" section:

Contains common and useful classes.

• "Performance" class: contains some replacement functions for simple dotNET Framework string functions that considerably increases ASProxy engine speed.
• "UrlProvider" and "UrlBuilder" classes: contains some functions for working with URLs.
• "HtmlTags" class: contains some functions for working with HTML.
• "ASProxyFunctions" classes: contains some generally useful functions used in ASProxy.

Tracing Background Working

Here we look over the ASProxy working steps.

Below we request to visit a site (for example www.softprojects.org ), so expect that the response data type should be HTML.

Execution traceRequest a site: Initialize an ASProxyEngine Test content type by PreExecute { IMAGE: Link to images.aspx page and display it in page as img tag. END. Unknown: Redirect to download page. END. HTML: Continue! } in ASProxyEngine { Initialize a WebDataCore Execute the request by WebDataCore instance { Initialize "PostBack" or "Get" methods information. Add cookies. Get the response. Restore returned cookies and save them. } if html process requested { Test content mime type { HTML: Initialize HtmlProcessor and process data. JavaScript: Initialize JSProcessor and process data. CSS: Initialize CSSProcessor and process data. } } return processed data. } Display response in a label!

A Review on the Main Functions

For the user to request the URL, first of all, an instance of the main class needs to be created for processing the request (gethtml.aspx):

engine = new ASProxyEngine(ProcessTypeForData.HTML,true); engine.RequestInfo.ContentType = MimeContentType.text_html; engine.Initialize(Request); engine.Execute(Response);

In this code, we specify that our data type is HTML. Then we call "initialize" method of ASProxyEngine to specify the request information such as request URL.

At the end we call "Execute" method to confirm the operations. This method will apply the result to the response directly.

How to implement the operation in a new page:

To do this add a new page to the ASProxy project and add a textbox, a button and a label. To save some resources set the "EnableViewState" property of Label to false; we do not need its state. Then write these codes in button click event:

protected void Button1_Click(object sender, EventArgs e) {   SalarSoft.ASProxy.ASProxyEngine engine = new SalarSoft.ASProxy.ASProxyEngine(SalarSoft.ASProxy.ProcessTypeForData.HTML);   engine.RequestInfo.ContentType = SalarSoft.ASProxy.MimeContentType.text_html;   engine.Initialize(TextBox1.Text);   string myResponse;   engine.Execute(out myResponse);   Label1.Text = myResponse; }

Run the new page and enter an URL and click the button; the URL will be shown.

Nothing will work on that page. It's just HTML code and it is not useful by itself. So to get better results we change the code.

protected void Button1_Click(object sender, EventArgs e) {   SalarSoft.ASProxy.ASProxyEngine engine = new SalarSoft.ASProxy.ASProxyEngine(SalarSoft.ASProxy.ProcessTypeForData.HTML, true);   engine.RequestInfo.ContentType = SalarSoft.ASProxy.MimeContentType.text_html;   engine.Initialize(TextBox1.Text);   string myResponse;   engine.Execute(out myResponse);   Label1.Text = myResponse; }

A "true" at the constructor of ASProxyEngine is added. This "true" specifies that the options should be restored automatically from the cookie or default options.

If you want more control over the settings you can set them manually. To do this the OptionsType class is needed. And if you want to access to the stored settings you should use ASProxyConfig class.

Look at this example:

protected void Button1_Click(object sender, EventArgs e) {   SalarSoft.ASProxy.ASProxyEngine engine = new SalarSoft.ASProxy.ASProxyEngine(SalarSoft.ASProxy.ProcessTypeForData.HTML, false);   engine.RequestInfo.ContentType = SalarSoft.ASProxy.MimeContentType.text_html; SalarSoft.ASProxy.OptionsType opt = SalarSoft.ASProxy.ASProxyConfig.GetCookieOptions(); opt.Images = false; opt.AcceptCookies = false; engine.Options = opt;   engine.Initialize(TextBox1.Text);   string myResponse;   engine.Execute(out myResponse);   Label1.Text = myResponse; }

In this example, the options is restored by "GetCookieOptions" method and then Images and AcceptCookies are disabled.

Now everything seems OK, but the links do not work. The current page should detect the URLs in query and display them. To do this we have to write the same codes in Page_Load event:

protected void Page_Load(object sender, EventArgs e) { SalarSoft.ASProxy.FilesConsts.DefaultPage = System.IO.Path.GetFileName(Request.Url.AbsolutePath).ToLower(); bool isClicked=(Request.Form[Button1.ID]!=null); if(isClicked==false && SalarSoft.ASProxy.UrlProvider.IsASProxyAddressUrlIncluded(Request.QueryString)) { SalarSoft.ASProxy.ASProxyEngine engine = new SalarSoft.ASProxy.ASProxyEngine(SalarSoft.ASProxy.ProcessTypeForData.HTML,false); engine.RequestInfo.ContentType = SalarSoft.ASProxy.MimeContentType.text_html; SalarSoft.ASProxy.OptionsType opt = SalarSoft.ASProxy.ASProxyConfig.GetCookieOptions(); opt.Images = false; opt.AcceptCookies = false; engine.Options = opt; engine.Initialize(TextBox1.Text); string myResponse; engine.Execute(out myResponse); Label1.Text = myResponse; } }

In this code, we set the "DefaultPage" value that specifies the default page for proxy and if it changes all link destinations will change. So be careful in using this property; it may cause ambiguity.

The other important method used in this code is "IsASProxyAddressUrlIncluded". This method checks the current query for URL existence.

The important point of this code is detecting the type of post back. The code has to detect if the post back is done through pressing button1 or not. The result of this detection is set to "isClicked" variable. In "Page_Load" event the operations will not run when the button1 is not clicked and the query has URL address or the post back  has occurs from a submit form.

Implementation

Below you see some important method implementation.

Here is implementation of "Initialize" method that has 2 overloads. And you will see important related methods such as "InitializeRequestQueriesByHttpRequest" and "DoAutoDetection".

/// <summary> /// Initialize ASProxy engine with as HttpRequest /// </summary> /// <param name="httpRequest">HttpRequest instance</param> public void Initialize(HttpRequest httpRequest) { InitializeRequestQueriesByHttpRequest(httpRequest); DoAutoDetection(httpRequest); } /// <summary> /// Initialize ASProxy engine with specified url address /// </summary> /// <param name="requestUrl">Requested url address</param> public void Initialize(string requestUrl) { RequestInfo.RequestUrl = requestUrl; RequestInfo.RequestedQueries = requestUrl; DoAutoDetection(null); } private void DoAutoDetection(HttpRequest httpRequest) { if (httpRequest != null) { RequestInfo.RequestedQueries = GetQueryCollectionString(httpRequest.QueryString); // When a post back event occurred "httpRequest.ContentType" contains ContentType of request // In other cases the "httpRequest.ContentType" is empty and we shouldn't use this property if(!string.IsNullOrEmpty(httpRequest.ContentType)) RequestInfo.ContentTypeString = httpRequest.ContentType;// Added again in version 3.7 RequestInfo.InputStream = httpRequest.InputStream; } // Get posted form data string if (RequestInfo.HttpRequestMethod == WebRequestMethod.POST) { if (httpRequest != null) { RequestInfo.PostedFormData = httpRequest.Form.ToString(); // Some web sites encodes the url, and we need to decode it to be useful //RequestInfo.PostedFormData = HttpUtility.UrlDecode(RequestInfo.PostedFormData); RequestInfo.PostedFormData = HttpUtility.HtmlDecode(RequestInfo.PostedFormData); } } else RequestInfo.PostedFormData = ""; if(RequestInfo.RequestInQueryMethod== WebRequestMethod.GET) { // Apply filter for ASP.NET pages RequestInfo.PostedFormData = RequestInfo.PostedFormData; if (fAutoDetection) { // Change requested url by posted data RequestInfo.RequestUrl = UrlBuilder.AppendAntoherQueries(RequestInfo.RequestUrl, RequestInfo.PostedFormData); } } } /// <summary> /// Auto detect the request information from HttpRequest /// </summary> /// <param name="httpRequest">HttpRequest instance</param> private void InitializeRequestQueriesByHttpRequest(HttpRequest httpRequest) { if (fAutoDetection) { // Get http request method RequestInfo.HttpRequestMethod = ASProxyFunctions.StringToRequestMethod(httpRequest.HttpMethod); bool tmpBool = false; // Get requested url string url = httpRequest.QueryString[Consts.qUrlAddress]; if (!string.IsNullOrEmpty(url)) { string decode = httpRequest.QueryString[Consts.qDecode]; if (decode != null) { try { tmpBool = Convert.ToBoolean(Convert.ToInt32(decode)); } catch { tmpBool = false; } } // If url is encoded, decode it if (tmpBool) { url = UrlProvider.DecodeUrl(url); } RequestInfo.RequestUrl = url; } tmpBool = false; // Get request post method state string postForm = httpRequest.QueryString[Consts.qIsPostForm]; if (postForm != null && postForm != "") { try { tmpBool = Convert.ToBoolean(Convert.ToInt32(postForm)); } catch { tmpBool = false; } } // if request is post method set the options if (tmpBool) RequestInfo.RequestInQueryMethod = WebRequestMethod.POST; else RequestInfo.RequestInQueryMethod = WebRequestMethod.GET; } }

Here is the implementation of "PreExecution" method. This method will perform the request but doesn't return any results. This is useful to check the response headers and determine to continue the operations with calling "Execute" method or not.

Note that after calling "PreExecution" method we can call "Execute" method or we can call "Execute" method directly without calling "PreExecute" method.

/// <summary> /// Executes the request and gets the responses but don't process the results /// </summary> public void PreExecution() { if (fAutoDetection) { // If this is an image request, we should provide orginal link as referer if (RequestInfo.ContentType == MimeContentType.image_gif || RequestInfo.ContentType == MimeContentType.image_jpeg) this.fUseRequestUrlAsReferer = true; } // Data executin should be one time! if (fWebData == null) { do { // If this is auto recection request send request again to new location if (fWebData != null && fWebData.ResponseInfo.AutoRedirect) { Uri autoRedirectLocation = fWebData.ResponseInfo.AutoRedirectLocation; CookieCollection cookies = fWebData.ResponseInfo.Cookies; bool redirectIsInternal = fWebData.ResponseInfo.AutoRedirectIsInternal; RequestInfo.RequestUrl = autoRedirectLocation.ToString(); fWebData.Dispose(); fWebData = null; fWebData = new WebDataCore(RequestInfo.RequestUrl); // In auto redirection it previous writed cookies needed and we set it here! if (redirectIsInternal) { fWebData.RequestInfo.Cookies = cookies; } } else fWebData = new WebDataCore(RequestInfo.RequestUrl); fWebData.UseRequestUrlAsReferer = fUseRequestUrlAsReferer; fWebData.RequestInfo.RequestMethod = RequestInfo.RequestInQueryMethod; fWebData.RequestInfo.PostDataString = RequestInfo.PostedFormData; fWebData.RequestInfo.ContentType = RequestInfo.ContentTypeString; fWebData.RequestInfo.InputStream = RequestInfo.InputStream; fWebData.AcceptCookies = fOptions.AcceptCookies; fWebData.DisplayErrorPageAsResult = false; // Run the request fWebData.Execute(); // If execution returned an error if (fWebData.Status == LastActivityStatus.Error) { this.fLastErrorMessage = fWebData.ErrorMessage; this.fLastStatus = LastActivityStatus.Error; this.fLastException = fWebData.LastException; return; } } while (fWebData.ResponseInfo.AutoRedirect); SetResponseInformation(); } }

Note that the loop is used to execute the requests to implement Redirect behavior.