Merry Christmas! New law gives your home computer and phone to government

Has your computer ever been infected with malware or spyware? Well, if it has, a new law that will go into effect December 1, says that if you’re computer has been hacked into into, then the federal government can legally hack into it without your knowing about it, too!

That’s right! Anyone acting on behalf of the federal government can go to any judge in any district to get a warrant to search your computer just because some jerk on the Internet infected your computer with some bogus scamware.

That means those bank statements and other private documents you keep on your home computer are up for grabs. Pictures of your friends, of your family, of you—including that selfie stash of yours! they’re all fair game. Run a company with sensitive customer data? Well, if any of your computers gets hacked, KA-BOOM! The feds can legally hack it, too. They can peek around and copy all of your company records. Who cares if it’s confidential? Now it’s “evidence”.

Starting three years ago, the Department of Justice drafted changes to the Search and Seizure rules of Federal Criminal Procedure law. The DOJ sent its changes to the Federal Courts. The courts okayed the changes to hack into American computers and passed them on to the Supreme Court. SCOTUS approved the changes on April 28. Now Congress has them for review. If not blocked, they go into effect on December 1. Christmas comes early for the FBI!

I’ve highlighted the relevant parts of the proposed amendment to the Search and Seizure Rule 41 that give the feds license over your data if your computer has been infected by malware and is part of a botnet.

10 FEDERAL RULES OF CRIMINAL PROCEDURE
Rule 41. Search and Seizure
***
(b) Authority to Issue a Warrant.
At the request of a federal law enforcement officer or an attorney for the government:
* * *
(6) a magistrate judge with authority in any district where activities related to a crime may have occurred has authority to issue a warrant to use remote access to search electronic storage media and to seize or copy electronically stored information located within or outside that district if:

(A) the district where the media or information is located has been concealed through technological means; or
(B) in an investigation of a violation of U.S.C. ยง1030(a)(5), the media are protected computers that have been damaged without authorization and are located in five or more districts

But there is hope! You are the hope. You can call Congress. Here’s what to say.

Congress can change the law. There is a bill in the House and one in the Senate to repeal this odious big government overreach of power and invasion of privacy.

Here’s what I said when I called Rep. Katherine Clark’s office earlier (district office: (617) 354-0292):

Hi, my name is Joshua Reyes and I live in Cambridge, MA. I’m calling to ask Representative Clark to cosponsor H.R.5321 Stopping Mass Hacking Act. Personal privacy is important to me. And I do not expect to give up my personal privacy just because my computer or phone is connected to the internet. Does Representative Clark agree with me?

Thank you and have a nice day.

And to my senators Warren (Boston office: (617) 565-3170) and Markey (Boston office: (617) 565-8519):

Hi, my name is Joshua Reyes and I live in Cambridge, MA. I’m calling to ask Senator Markey/Warren to cosponsor S.3475 Review the Rule Act of 2016. Personal privacy is important to me. And I do not expect to give up my personal privacy just because my computer or phone is connected to the internet. Does Senator Markey/Warren agree with me?

Thank you and have a nice day.

Many representatives and senators don’t know about this obscure amendment. So it’s important to call now and let them know how you feel! December 1 is just a day away. I believe in you!


Don’t live in Massachusetts? No problem! Look up your representative. Look up your senators.

Render D3.js-driven SVG server-side!

Recently I’ve been working on a congressional tweet aggregator to get a handle on what our legislators are saying. To make that easier to see, I figured I’d start adding some charts and lists and other snazzy dataviz gizmos that are so hot these days.

I like D3.js as a graphing library. It makes clean, interactive, data-driven charts a snap to make in just a few lines of Javascript. Then it does its magic to render the data in crisp SVG, which I am quite fond of. On my site, I wanted to turn the crank on the back-end for charts that don’t update all that frequently, inject them into my templates, and spare the viewers of my site the heavy-lifting required for multiple charts—not to mention my poor server that has to execute several complicated queries to do the appropriate counting to generate the data to back the charts.

After a little poking around on the internet, I stumbled on to PhantomJS, which bills itself as a full-stack headless WebKit. Perfect. It can ping my website periodically, load the chart pages, and extract the SVG, I thought.

Not so fast. The Phantom is excellent at reading SVG, and it’s even good at rendering it to PDF or PNG. But that’s not what I wanted! I just wanted it to spit out the SVG for me after D3 was finished making it, untouched. And since SVG elements don’t have an innerHTML property, I needed to think harder to find a solution; i.e., ask Google. But Google didn’t seem to know, either. So I wrote a tiny script to extract page elements by ID. Maybe one of you will find it useful, too.

var system = require('system');

if (system.args.length != 3) {
    console.log("Usage: extract.js  ");
    phantom.exit(1);
}

var address = system.args[1];
var elementID = system.args[2];
var page = require('webpage').create();

function serialize(elementID) {
    var serializer = new XMLSerializer();
    var element = document.getElementById(elementID);
    return serializer.serializeToString(element);
}

function extract(elementID) {
  return function(status) {
    if (status != 'success') {
      console.log("Failed to open the page.");
    } else {
      var output = page.evaluate(serialize, elementID);
      console.log(output);
    }
  phantom.exit();
  };
}

page.open(address, extract(elementID));

A little birdie told Congress

Since I couldn’t find a list of twitter feeds from the US Congress, I made one today.

Now you can get a snap-shot of our legislators highest priorities, as captured in 140 characters at a time, at twitter.com/CongressBirdie/legislators.

In case you’re curious how I did it without painstakingly searching each congressperson’s name and username to add to my list by hand, I’ll let you in on my little my secret: I relied heavily on a few open source projects to automate the process. To find the Twitter IDs, I simply looked them up from the very excellent Github project unitedstates/congress-legislators. Then I used the Python Twitter Tools module to chat with the Twitter API to create the list and add all the legislators in bulk.

Life wasn’t exactly as easy as all that, though. I had to make a little tweaks in order to gather all the tweets. First, there is an easy-to-fix bug in the Python Twitter Tools package. You need to make sure it knows how to POST to lists/members/create_all command. Right now employs a GET request—and that doesn’t work. It looks like at least one other person has run into the same problem. If you run into the problem, you can read how I fixed it.

But Twitter didn’t handle my create_all request as they promised. The documentation claims you can add up to 100 users to your list at a time, but that wasn’t my experience. Instead, I could only get the API to add legislators 25 at a time. But that’s a small price to pay for democracy.

And this list is active! In the time it took me to write this post, the list reported 13 new tweets. Your tax dollars hard at work.

Populate URLs in Your Javascript with Django

All this Olympics has inspired me. I’ve decided to train for the summer games in Rio and to pursue my hobbies in full force. The former requires me to learn to kayak; the latter, to program, program, program.

I’m building a fun web-app game on top of Django. And I want to make it dance with a little Javascript. I really like that I can look up the URLs of my pages by an internal name in templates using the {% url %} tag, but unfortunately that functionality isn’t available to me inside my static media—particularly, I’d like to be able to reverse look-up URLs in my site’s javascript, but I don’t want to have to render the file each time it gets served.

After hunting around for a while, I figured, “Why not just render to Javascript once, fill in all the links, and be done with it?” So that’s what I did. And it doesn’t take much.

I wrote a custom command called populate_urls that consumes a template (in this case, my Javascript) and spits out the finished product with all the holes filled in by Django and its handy URL resolvers. Then I stash its output in my static directory and pretend the whole thing never happened.

from __future__ import print_function

import sys

from django.core.management.base import BaseCommand
from django.template import Template, Context

class Command(BaseCommand):
from __future__ import print_function

import sys

from django.core.management.base import BaseCommand, CommandError
from django.template import Template, Context

class Command(BaseCommand):
args = ‘
help = ‘Populates the URLs in the template. If no output filename is supplied, renders the template to standard out.’
traceback = True

def handle(self, *args, **options):
if len(args) < 1: raise CommandError("You must supply a template.") with open(args[0], 'r') as templatefile: template = Template(templatefile.read()) context = Context() with sys.stdout if len(args) < 2 else open(args[1], 'w') as output: print(template.render(context), file=output) [/sourcecode]

Keep the Internet Open: Update

Last week, the Senate voted on SJ Res. 6, that piece of legislation that stated plainly: Congress disapproves of openness on the internet. That is, that companies should be able to block your traffic on the internet unless you paid for it. Fortunately, the resolution was rejected (by a slim margin of four votes). See the tally here.

Imagine how this sort of deregulation would work in the phone industry. Phone companies could monitor who were calling, listen in to what you were talking about, and then decide how much to charge you accordingly. Would you want the phone companies listening in to your phone conversations and then deciding how much to charge you based on what you said? “Oh, he’s calling his sister again—I bet he’d pay more to talk to his family.” “Her doctor is about to say something important, that phone call’ll cost extra.” Eek.

The FCC has been charged with the responsibility to make sure the internet remains open, transparent, and is free from blocking and unreasonable discrimination. According to the FCC, if it’s legal, you can do it and the internet providers should respect that. Since this policy seems like a good idea to me as it promotes technological innovation and rightful consumer protection, I was shocked that Scott Brown voted to dismantle openness on the internet. I’ve written to his office for an explanation for his vote. I’ll let you know the reasons if they respond.

I’m thankful to Kerry for supporting net neutrality. And I’m surprised and pleased to see that the White House had the backs of the American people, even if a near majority of the Senate didn’t. In an official statement, the President openly opposed SJ Res. 6 because he is in favor of job creation. And this resolution would have stifled technological innovation. Good work.

As an aside: in the end Olympia Snowe, Susan Collins, and Kelly Ayotte did vote for big business. If anyone can tell me how this resolution would have helped families, consumers, the poor, and/or children in Massachusetts, Maine, or New Hampshire, I’d like to know!

Keep the Internet Open

The internet is this country’s greatest, most used, and largest public library. Nearly a year ago, the FCC adopted FC 10-201 to keep the doors of the internet open to American citizens. In this regulation, the FCC cites evidence that broadband providers had been covertly blocking or degrading Internet traffic, and that cable companies have financial incentive and ability to shut things down even more.

Here’s a summary of values stated in the report:

PRESERVING THE FREE AND OPEN INTERNET

1. Today the Commission takes an important step to preserve the Internet as an open platform for innovation, investment, job creation, economic growth, competition, and free expression. To provide greater clarity and certainty regarding the continued freedom and openness of the Internet, we adopt three basic rules that are grounded in broadly accepted Internet norms, as well as our own prior decisions:

i. Transparency.
Fixed and mobile broadband providers must disclose the network
management practices, performance characteristics, and terms and conditions of their
broadband services;
ii. No blocking.
Fixed broadband providers may not block lawful content, applications, services, or non-harmful devices; mobile broadband providers may not block lawful websites, or block applications that compete with their voice or video telephony services; and
iii. No unreasonable discrimination.
Fixed broadband providers may not unreasonably discriminate in transmitting lawful network traffic.

We believe these rules, applied with the complementary principle of reasonable network management, will empower and protect consumers and innovators while helping ensure that the Internet continues to flourish, with robust private investment and rapid innovation at both the core and the edge of the network. This is consistent with the National Broadband Plan goal of broadband access that is ubiquitous and fast, promoting the global competitiveness of the United States.

But NO! Senator Kay Hutichson (R-TX) introduced S.J. Resolution 6 to the Senate floor to strike down openness and transparency on the internet. The resolution states in plain English that Congress is against openness on the internet! If passed, this resolution will make the FCC unable to ensure the doors of the world’s greatest public library stay open to the public. This resolution is job-killing, innovation-stalling, and education-thwarting.

The resolution is so short, I’ll post its contents in their entirety for you to see for yourself:

Disapproving the rule submitted by the Federal Communications Commission with respect to regulating the Internet and broadband industry practices.

Resolved by the Senate and House of Representatives of the United States of America in Congress assembled, That Congress disapproves the rule submitted by the Federal Communications Commission relating to the matter of preserving the open Internet and broadband industry practices (Report and Order FCC 10-201, adopted by the Commission on December 21, 2010), and such rule shall have no force or effect.

Today I called Scott Brown’s office in DC to tell him to vote NO on S.J. Res. 6. The number is (202) 224-4543. Do you use the internet? Do you plan to use it in the future? If so, then please call your senators to tell them to vote for American innovation and against S.J. Res. 6.

p.s. — If you live in Maine, ask Olympia Snowe (202) 224-5344 and Susan Collins (202) 224-2523 how this resolution will create jobs in Maine. If you live in New Hampshire, ask Kelly Ayotte (202) 224-3324 whether this resolution will help New Hampshire families make ends meet. They must believe it will. They co-sponsored the resolution, after all.

Geotools, WFS-T Update Request

A pet project of mine has flung me into the exciting though less-than-firm territory of web-backed geographical information systems. Since I don’t have the thousands of dollars it costs to get a commercial server like those provided by ESRI, I’ve had to check out the open-source alternatives. And there are some out there. I’m using GeoServer, and it works great! I can send all the web-feature service transactions (WFS-T) in XML I want and it works every time. Not bad if you want to make a GoogleMap of your house on your own—so long as you’re content to hard-code everything by hand. Should you want to jazz things up a bit (i.e., make minimally useful dynamic maps), like me, then you have to do a little more work. Actually, you need to do a lot more work.

GeoServer is built on top of a gargantuan set of Java libraries, collectively packaged under the name GeoTools. Now I appreciate that this thing exists, all two hundred and fifty megs of source code and all the functionality that comes with it. However, navigating the mountain of documentation for this thing is, at least for me, a little daunting. It took me a few days (and some serious help from my friend Matt) to figure out how to write a simple update transaction using their API. (Compare that to the forty-two seconds it takes me to type up the XML.)

Since other people might want to know what they have to do update an attribute field using WFS with GeoTools, and since I couldn’t easily find out how to do it elsewhere, I’ve decided to post a short snippet of code right here on my blog. That’s right: my charity knows no bounds.

In this example I’m going to update the value of all the features (polygons, lines, points, whatever) that match a simple filter. Here I’m going to change the value of propertyToUpdate to updatedValue using a filter to get all the features with the attribute called constraintProperty with a value of constraintValue. I’ve marked them in red, so that it’s as easy as possible to customize this example to fit your needs. Let’s start with the XML that the Open Geospatial Consortium standards expects to see.

<wfs:Transaction service=”WFS” version=”1.0.0″
     xmlns:myns=”http://www.domain.com/ns
     xmlns:ogc=”http://www.opengis.net/ogc”
     xmlns:wfs=”http://www.opengis.net/wfs”>
  <wfs:Update typeName=”myns:LayerToUpdate“>
    <wfs:Property>
        <wfs:Name>propertyToUpdate</wfs:Name>
        <wfs:Value>updatedValue</wfs:Value>
    </wfs:Property>
    <ogc:Filter>
        <ogc:PropertyIsEqualTo>
            <ogc:PropertyName>constraintProperty</ogc:PropertyName>
            <ogc:Literal>constraintValue</ogc:Literal>
        </ogc:PropertyIsEqualTo>
    </ogc:Filter>
  </wfs:Update>
</wfs:Transaction>

Now let’s rock out the Java.

Like I said, GeoTools is mammoth. To make life easy, we’re going to import a whole bunch of classes for this example. So many, in fact, that their number really warrants my displaying them here in their own list. What’s more, the names of some of classes (like Filter) show up in more than one package, and you need to keep track of which is used where. So keep an eye out for things from org.opengis.

import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;
import java.util.logging.Level;

import org.geotools.data.DataStore;
import org.geotools.data.DefaultTransaction;
import org.geotools.data.FeatureStore;
import org.geotools.data.Transaction;
import org.geotools.data.wfs.WFSDataStoreFactory;
import org.geotools.feature.AttributeType;
import org.geotools.feature.FeatureType;
import org.geotools.filter.FilterFactoryFinder;
import org.geotools.xml.XMLSAXHandler;

import org.opengis.filter.Filter;
import org.opengis.filter.FilterFactory;
import org.opengis.filter.expression.Expression;

In our constructor we’ll set up a connection to the WFS server using a URL. If you’re tinkering with GeoServer, then that URL you’re looking for probably looks something like http://localhost:8080/geoserver/wfs. Since we know that we’ll want to filter our responses, it’s not a terrible idea to make a filter factory now and save it for later. In GeoTools everything is made using a factory. For filters, we need to make a factory using the new keyword, though. Here goes.

public class WFSUpdater {

    private DataStore wfs;
    private FilterFactory filterFactory;

    public WFSUpdater(String url) {
      try {
        URL endPoint = new URL(url);

        XMLSAXHandler.setLogLevel(Level.OFF); // turns off logging for XML parsing.

        // Parameters to connect to the WFS server, namely the URL.
        // You could have others, say if you had to authenticate your connection with a username and password.

        Map params = new HashMap();
        params.put(WFSDataStoreFactory.URL.key, endPoint);

        wfs = (new WFSDataStoreFactory()).createNewDataStore(params);
        filterFactory = FilterFactoryFinder.createFilterFactory();

      } catch (MalformedURLException e) {
        e.printStackTrace();
      } catch (IOException e) {
        e.printStackTrace();
      }
    }

Now that we have a connection, it’s time to make the transaction. As a first timer to GeoTools, I found it difficult to crawl through the documentation. Lots of their classes and methods have been deprecated with no clear hints about what to use instead. I had a hard time finding the right constructors. In what follows everything is hard-wired into the code, but it shouldn’t be all that bad to tweak things so that it works the way you like.

public void updateProperty(String propertyToUpdate, String updatedValue) {
    // This is the layer name on the server.
    String layer = “myns:LayerToUpdate“;
    Transaction update = new DefaultTransaction(“update”); // The handle/ID of this transaction is called “update.” It’s required.

    try {
      // Make the filter.
      Expression property = filterFactory.property(“constraintProperty“);
      Expression value = filterFactory.literal(“constraintValue“);
      Filter filter = filterFactory.equals(property, value); // This is an org.opengis.filter.Filter.

      FeatureStore features = (FeatureStore) wfs.getFeatureSource(layer);

      // Set the transaction. Otherwise, we can’t commit our changes later on.
      features.setTransaction(update);

      // Fetch the property from the FeatureType schema so that we can update it with the new value.
      FeatureType schema = features.getSchema();
      AttributeType atrributeToUpdate = schema.getAttributeType(propertyToUpdate);

      features.modifyFeatures(atrributeToUpdate, updatedValue, (org.geotools.filter.Filter) filter); // There’s that casting again.

      // Record the modifications.
      update.commit();

      } catch (IOException e) {
        e.printStackTrace();
      }
    }
}

Anyway, I hope this saves some people the hassle of tearing through the Javadocs for GeoTools. Also, if there’s a better way to do what I did, please let me know. Happy GIS-ing.

Technorati Tags: , , , , , , , , , , , , ,

Genetics by the Poolside

Happy Independence Day! To celebrate our nation’s founding, my family and I often hit up the Cape. This year was no exception, there’s little to report. The weather has been spotty: a little rain here, a few showers there, but nothing substantial. Someone was playing bagpipes the other night. And I witnessed a gruesome car accident a few feet from my balcony during the fireworks spectacular. A mass of people immediately sprang up to help the man, direct traffic, and call 911 repeatedly until emergency vehicles could make their way here. I was genuinely impressed by the response, professional and make-shift alike. Within seconds the response team had the guy off to the hospital in no time flat. I think it prudent not to speculate on the cyclist’s health. I don’t want to jinx anything, you know.

And since it’s vacation time, I’m here, at the kitchen table, on my laptop, implementing genetic algorithms. Maybe later I’ll describe what I did. Maybe if I do, someone will be able to tell me if my results make any sense. Whether or not my programs reproduce the classical results isn’t really the point, though. Look at the evolution of strategies for playing the iterated prisoner’s dilemma: they make perfect modern art tile mosaics! I bet someone’d love to have this pattern on their pool floor or garden wall. (Don’t be alarmed that they don’t appear all that related. Each row in black represents the fittest individual from one of a number of independent runs. That is, they probably never had the chance to meet each other.) Imagine the graphic tastefully obscured by flowering vines. (Click on it for a larger image.)

I can see an upside-down raccoon in it. What can you find?

Crab Canon

This week we had to create a sound collage for my computational media class. I didn’t set aside a lot of time to work on it, so it became something of a last-minute project. Today I spent most of my day meeting with bioinformatics folk to discuss herberia and taxa and distributed architectures. I may end up working in a CS research group playing with this (or other) stuff. Anyway, by the time I got home, I only had a few hours to start and finish my project. Luckily, I’ve been toying around with MIDI on my own. So I took a line from J.S. Bach and tried to reconstruct part of his crab canon. (This amounts to reversing, compositing, and normalizing a small bit of data.)

Here’s what I started with.

Two hours later, here’s what I ended up with. True it’s not precisely a crab canon—I prefixed a short introduction before the canon starts proper. But if you played from then on backwards, it would sound exactly the same. That’s right: I one-upped Bach. He thought he was writing a musical palindrome. Unfortunately, he couldn’t reverse the attack and decay of each note. He needed me to come along and help him out with the minor, technical details. There’s no shame in that.

You can even download my project in spiffy MP3 format if you like. I’m just that sort of guy. Giving, courteous, clean.

Crab Canon.mp3

Technorati Tags:, , , , , , ,