Handling HTTP Redirection in Ruby

I have a Ruby project where I'm dumping a bunch of bookmarks from delicious.com, then fetching each bookmarked page for analysis.

One of the problems I encountered early on is that the some of the web pages bookmarked would redirect to some other location. Simply checking for HTTP response code 200 was insufficient. I needed to check for redirection as well.

A quick Google search for "ruby follow http redirect" yields lots of results. Unfortunately, they're all very similar, and not quite right. In general, the examples you come across (even the one in the official Ruby documentation) don't handle the case when the redirected location is path relative to the original location. So you end up doing a get on a URL that looks like "../../redirected/location/index.html," which clearly won't work.

It turns out that detecting relative redirection is fairly straightforward:

 until( found || attempts>=@@MAX_ATTEMPTS)
     attempts+=1
     http=Net::HTTP.new(url.host,url.port)
     http.open_timeout = 10
     http.read_timeout = 10
     path=url.path
     path="/" if path==""

     req=Net::HTTP::Get.new(path,{'User-Agent'=>@@AGENT})
     if url.instance_of? URI::HTTPS
       http.use_ssl=true
       http.verify_mode = OpenSSL::SSL::VERIFY_NONE
     end
     resp=http.request(req)
     if resp.code=="200"
       break
     end
     if (resp.header['location']!=nil)
       newurl=URI.parse(resp.header['location'])
       if(newurl.relative?)
           puts "url was relative"
           newurl=url+resp.header['location']
       end
       url=newurl

     else
       found=true #resp was 404, etc
     end #end if location
   end #until

The trick here is to ask the redirected url object if it is relative. If it is, then add the redirected path onto the old url object. the URI class overrides the '+' operator (what is this, C++?) so that you can concatenate the new path onto the old URL, by doing:
newurl=url+resp.header['location']

Tweet

How to sudoedit non-interactively

Okay, this one's a bit esoteric, but I think it's pretty cool. How do you use 'sudoedit' non-interactively such as from a script? Just a brief background about sudo: sudo is an authentication mechanism in Unix & Linux that allows unprivileged users to run specific commands (as defined by the system administrator) with root privileges without having the root password. This has several advantages over logging in as root:

• Users can have specific, limited set of root privileges without having the entire set of root privileges.
• Users use their own password, so the root password doesn't have to be shared. If a user's sudo privileges are revoked, the root password doesn't have to be reset.
  
• Each use of sudo is audited per user, so that each time sudo privileges are invoked, there is an event in the system logs that identifies the specific user and the command they ran.

Sudoedit is a command that is related to sudo. It lets users edit files that normally only root can edit, such as system configuration files. However instead of using "sudo" followed by the editing command, the user runs "sudoedit filename" and sudo invokes the user's default editor, letting them edit the file.

So, what if you want to make changes to a system file via a script, and the only access you have to the file is via sudoedit? It isn't useful to have the script call sudoedit and then bring up vi for you to manually make the changes.

Well, the way sudoedit works, is that sudo makes a temporary copy of the file you want to edit, then calls your default editor, giving it the name of the temp file as the first argument. So, say your default editor is "fooedit". You run "sudoedit /etc/systemfile" and sudo makes a copy of /etc/systemfile to /tmp then runs "fooedit /tmp/tempfile". Your changes are saved to the temp file. When your editor exits, sudo copies the temp file over the original, and then removes the temp file.

From your main script, stage the pre-edited version of the system file, then set your default editor to a custom script that will copy that staged file over the temp file. When it exits, sudo will copy the temp file into place as normal.

Your script would go like this:

#!/bin/sh

export EDITOR="/bin/sh ./editor.sh";
PRE_STAGE=/tmp/stage.tmp;

echo "my new config"> $PRE_STAGE;

sudoedit /etc/config;

/bin/rm $PRE_STAGE;

exit;

Your custom script would look like:

#!/bin/sh
CAT=/bin/cat
PRE_STAGE="/tmp/stage.tmp"

$CAT $PRE_STAGE > "$1";
exit;

This script just cats your pre-staged file over whatever sudo passed in as the first argument ($1).

The only interaction you'll have to do is type your password for sudo, if you haven't already done so in the last few minutes (sudo temporarily remembers your authentication).

Tweet