Showing posts with label geek. Show all posts
Showing posts with label geek. Show all posts

Tuesday, August 31, 2021

blargh.

The wildfire smoke finally turned around and hit us for a few days. I bought a couple PurpleAir sensors, which was useful, but awkward: they have built-in webservers, but the outdoor sensor actually has two sensors for some reason. And it was a pain to look at the two side-by-side, when that's actually an important comparison.



One of the many nice things about being a computer programmer is that I know how to find the data made available for computer programs to read and process. And one thing that makes me an expert is knowing that while I could go through the work of creating an authorization key to fetch the data from PurpleAir's service, that's silly, because the sensors are right here on the house network.

   "Adc" : 0,

   "DateTime" : "2021/09/01T06:26:33z",

   "Geo" : "PurpleAir-c882",

   "Mem" : 14200,

   "SensorId" : "10:52:1c:44:c0:87",

   "current_dewpoint_f" : 53,

   "current_humidity" : 35,

   "current_temp_f" : 84,

This is quite civilized: the sensor provides this in plain text, and the fields aren't rocket science to figure out. I don't know what Adc is, but I also don't care, because I want temperature, humidity, and the Air Quality Index, which the sensors label pm2.5_aqi. (PM 2.5 overwhelmingly dominates the AQI calculation for our purposes.)

We have data! But data is not information; information comes when you take your data and apply your questions. You can see that just in the screenshots up there: because I'm not an expert, 759 particles/deciliter is not a number that tells me if I should go outside or turn on another air filter. I need the user-friendly AQI for that. Next technical challenge, is that I would like to know...

  • when the AQI goes above one or more threshold values,
  • when it goes back down,
  • the relationship between inside and outside AQI,
  • and temperature/humidity/atmospheric pressure, just for fun.
If you've ever done any data processing or visualization—for which you probably used Excel and/or Google Sheets—you have a hint of the challenges, if you want to collect 10 data points, every 2 minutes, store them somewhere, do calculations on them, make them available via a web server, graph them, show the graphs next to each other...it's a pain in the ass. For businesses, it can be thousands or millions of data points, every few seconds. It's so hard that there are many companies you can pay to do it for you. DataDog is one of those. I wrote a 37-line script to send the sensor data out to DataDog, which also lets me set up alerts on it.


If you look at the live version of the charts, you can change the time window, and moving the mouse pointer along any of the charts shows a cursor at the same moment on each other chart, so you can correlate events.

So that's been useful. The wind has turned back for the moment, sending the smoke back to the rest of North America. But it will be back.

Monday, October 13, 2014

true story.

Once when I worked in Palo Alto, a bunch of us were walking back from afternoon coffee. One of the schools would get out around that time, and our office's street is also a bike thoroughfare, so you'd see teenagers of all ages biking to and from school or other places.

I was wearing my (now-deceased) t-shirt with the capsaicin molecule: the primary chemical that makes peppers spicy. The t-shirt doesn't say what the molecule is. It's just the diagram.

We're walking along, and a few kids are riding towards us on the sidewalk. They turned off before they got to us, but as the second kid rode off the curb, he said:
"Hey, look! Capsaicin!"

Thursday, October 20, 2011

polyglot

Since I've been teaching J how to program in Scratch off and on, I had the rare moment of talking to my parents about programming languages the other day. There's not usually much to say about specific programming languages with someone who doesn't write code--though there's lots to say about the nature of programming, and I have a blog post in the works about that--but I mentioned I was reading some horrible C++ code and it was a little tricky because I don't really know C++. I used it a couple semester in college and that was it. Mom said, "But you can tell it's horrible anyway?", which I can, and that got me thinking about why, and what I understand about programming languages and how they're related, and all kinds of stuff.

Programming languages are not mathematically related to human languages, but they're related inasfar as they're made by and for humans. Much like human linguists, we talk about "families" and "dialects", and languages influence each other's development, often with cycles of back-and-forth borrowing of ideas. To see this profusion, take a look at the canonical Programming Languages Family Tree, or O'Reilly Media's much prettier version (now somewhat out of date).

Making a programming language is now well-enough understood that college students do it in one semester. Making a programming language that's useful is a whole other project, and highly likely to fail. Solving a real-life problem isn't just about the language, but about its associated libraries, which need to:
  • manipulate files on disk and otherwise talk to the computing environment.
  • deal with networking.
  • deal with multiprocessing (doing more than one thing at a time).
  • provide utilities for common programmer needs, like finding the length of a string of text.
This is a lot of work, and programmers have limited time to learn new things and even less time to actually adopt them, so your language has to have something special to offer to get off the ground. Java, for instance, looks a lot like C++ but doesn't suck quite as much. Ruby, which I work in, lets you accomplish a lot of work with very little code. Python is similar to Ruby, but a bit easier to read. And so on.

Enough describing. Why can I be judgemental about C++ code when I don't know C++? Let's start with some languages I'm more familiar with. Here's a class in Java. A class is technically defined as "a thinger that combines data with the operations on that data". Like so:

public class MessageCounter
{
int count;
MessageList messages;

public void receive(Message msg) {
MessageList.add( msg );
count++;
}
}


(None of these examples are complete, by the way. I'm stripping out a bunch of stuff that's needed in real life but would just obscure my point.)

That receive() thing is a function (or sometimes subroutine), but for irrelevant historical reasons, when a function is part of a class, we call it a method. The ++ operator comes from C, and increments a variable by 1.

My second language was Perl:

module MessageCounter;

$counter = 0;
@messages = [];

sub receive {
my $msg = shift;
push @messages, $msg;
$counter++;
}


Okay, that's a bit out there, I'll grant you: Perl is not for the faint of heart. But see how there's still a block marked MessageCounter, we're clearly doing something with a structure named messages and the message we received, and there's that ++ again. sub is short for "subroutine".

Here's Ruby, invented some years after Perl by a guy who couldn't take Perl's punctuation any more.

class MessageCounter
def receive(msg)
@messages.push msg
@counter += 1
end
end


See? It's like Perl, but easier to read. Ruby lacks the ++ operator, so we say += 1: in this language family, = means "assign the right side to the left side", and @counter +=1 is shorthand for @counter = @counter + 1.

After all that, here's C++, which predates all three.

class MessageCounter
{
int counter;
MessageList messages;

public void receive( Message msg ) {
messages.add( msg );
counter++;
}
}


That's right. There's a one-word difference from the Java. C++ is a huge, awful language, and even professional C++ programmers often don't know all of it, but much like seeing Catalán when you know Spanish, you can usually get the gist.

As for judging quality, because all these languages belong in a very, very large category called imperative languages, and furthermore (because they have classes) they're object-oriented,
(Pedants: Yes, Perl is not object-oriented. Hush, it's not important right now.)
there are common principles, the subject of many many books, of how to organize programs in a way that is comprehensible. The code I was reading violated many of those principles.

See? Simple!

Thursday, October 13, 2011

Dennis M. Ritchie, 1941-2011

Everyone knows Steve Jobs died, because everyone knows who Steve Jobs was. Steve Jobs made sure everyone knew who Steve Jobs was, by being the public visionary of Apple, then Pixar, then NeXT (you're easily forgiven for not recognizing that one), which Apple bought as the foundation for Mac OS X, making him the public visionary of Apple again. Everyone knew him because he personally masterminded and introduced dozens of things that changed our everyday life.

If you're a non-Microsoft computer geek, though, you probably owe your career to Dennis Ritchie, who died on October 9th. With Brian Kernighan, he created the C programming language, which was mostly invented so that Ritchie and Ken Thompson could write the Unix operating system. Steve Jobs' NeXT was based on Unix1, and Linux was a complete re-implementation of the Unix design.


(Those cabinets behind Ritchie and Thompson is the computer, and there's paper in that computer terminal.)


It's hard to overestimate Ritchie's impact. My entire career has been programming software in Unix-like environments, and that looks like it will be true for a long time. Even DOS, the hoary undercarriage of Windows, was actually a half-assed braindead Unix rip-off. You may recognize
c:\\Documents\Your Mom\Friday Night.doc
but that started out as
/home/chris/writeups/your_mom/friday_night.txt
If you're an advanced DOS user, you may have done something like
list | more
where the output of list gets "piped" through the pager more so you can actually read it if it's more than one page. You wouldn't know it from DOS, but pipes were Unix's gift to the world, enabling people to write small, general-purpose tools and string them together to create extremely complex effects (also known as the Unix philosophy). This sort of thing is routine in Unix:
grep apple some_file.txt | sort | uniq -c
Which, in order, finds all lines in some_file.txt with the word "apple", sorts them, and simultaneously removes duplicates and counts the occurrences of each line.

The C programming language and its Frankenstein descendant C++ have been so popular that any new programming language needs to either have C-like syntax, or have a really compelling reason not to. Languages that kept C's lower-level syntax (like 'x=5' for "assign the value 5 to the variable x") include C++, Objective-C, Java, Microsoft's C#, Perl, Python, and Ruby.

I don't know when this was made, but they're already greybeards.




In his quieter, harder-to-understand way, Dennis Ritchie changed the world every bit as widely as Steve Jobs did. Well done.

1 Well, kind of. The history of "Unix" and its derivatives and copies is a tangled mess: here is a "simplified diagram" of the Unix timeline.

Monday, June 13, 2011

my job: a vignette

Last week, one of our server applications, which has had a long list of troubles, suddenly developed a massive memory leak, seizing up and crashing over time. We set it to restart every 30 minutes, which is hardly sustainable, because it reduces our capacity and reliability.

I volunteered to go sit with the engineering team while they decided what to do. The options were:
  1. spend some time with the debugger and profiler and such and try to figure it out, or
  2. start upgrading the code to Ruby 1.9 and hope that fixes it.
My sage advice:
"That's not exactly an afternoon's work, right?"
"Right."
"And it will probably introduce additional bugs?"
"Yep."
"That sounds like very expensive voodoo, that might not even work."
"So you think we should do the profiling thing?"
"Yeah."
"Okay, we'll do that."
From any angle, it's difficult to believe I actually do this for a living.

Saturday, March 5, 2011

preliminary verdict

After Week 1, the new job is a thumbs-up. Nice, smart people, in a culture dedicated to having fun and shipping working software. And by "dedicated," I mean "willing to make serious changes in pursuit of the goal," rather than "spending hours in aimless meetings moaning about how we should make serious changes and then refusing to do so." It's pretty awesome.

I'm having fun working with the Ruby programming language. I learned the fundamentals before I started--the syntax, which is basically how to read and write it--but the standard library, which you can think of as the vocabulary to make useful programs, is large and I'll be learning that for some time.
(One example is how to make a directory on a hard drive: this turns out to be a function called mkdir in the module Dir. But to check if a file, including a directory, exists, that's the function exists? in the module File. It's hard to remember these things without using them at least a couple times.)
Ruby is quite a bit like Perl, which you could think of as my "native" programming language, the one I've spent the most intense and interesting time with. They're both called "scripting" languages, which isn't rigorously defined, but generally means "a bit slower, but lets you get more work done with less programming."

The people are really cool, I'm enjoying taking the train, and the food is good. Let's hope it keeps up.

Tuesday, February 8, 2011

export your Sidekick contacts

One of the various flaws of the Danger Sidekick service--besides its lack of database backups--was its lack of interaction with the outside world. To my memory, like so many things, this was T-Mobile's fault, but however it happened, exporting your data required buying a third-party sync application, which didn't work as well as one might hope, and was eventually not an option for Mac OS X. It's also pretty lame to have to pay $25 to get your data off the service because you plan to stop using it.

In the run-up to Chile, I explored ways of getting my address book off the backend service, and discovered that through a combination of functionality (likely not on purpose), you can actually export your contacts.
  1. Go to the Mail section of the Desktop Interface.
  2. Start a new message to yourself.
  3. Click "Attach vCard." This brings you to the Contacts screen, where it will show your "A" contacts.
  4. Under the "Attach Checked vCards" button, there's a "View" selection. Select "All."
  5. Click the checkbox column header to select all your contacts.
  6. Click "Attach Checked vCards." This brings you back to your email in progress.
  7. Send the email to yourself (give it some time to send).

Some bullet points about the result:
  • The email will have one vCard for every contact.
  • Assuming you don't want to import them by hand into anything, GMail will let you download them as a .zip file.
  • I then merged them (using cat in the command-line shell) into one giant .vcf file.
  • The only thing on Linux or OS X I found that will sensibly parse a multi-contact vCard is Google Contacts.
  • Google can then export the list in Outlook or other formats.
  • I only exported 130 contacts. I don't know what happens if you have the limit of 2,000. Should be fine. Will definitely be slow.
  • Plan to do some hand-editing. There's all kinds of crazy "CLASS: PUBLIC" notes and weird wrong birthdays and stuff in the vCards.
Enjoy!

Wednesday, February 2, 2011

job hunt: the home stretch

I have a job offer! It's from the San Francisco company, and it's an excellent offer. But, I'm going to finish up interviewing with the 2 remaining companies, and decide between them. Companies don't like that, particularly--quite reasonably, they want you to be gung-ho to work there. All these jobs have trade-offs, though, and I've been beaten up by enough jobs, and lost job opportunities, that "gung-ho" is not where I'm at right now. I was pretty excited and optimistic about working for the Palo Alto company, and we saw how that turned out.

A friend did point out that if I say "I'm considering all the options" rather than "I'd like to finish the process with a couple other companies," it might annoy them less.

A second Mountain View company has come into play very suddenly; call them MV #2. I had a quick sounding-out talk with a sysadmin and an engineer today, for both sides to see if we got along and which of two very different jobs I might be more interested in and better suited for: Big Data Software Engineer (SWE) or Site Reliability Engineer (SRE). The SRE job is supposed to be "half sysadmin [SA], half engineering," and since I don't really like SA work, I said I'd prefer to apply for the SWE job.

The sysadmin, the engineer, and the VP of Engineering (who I didn't meet but everyone says is awesome) all think I, or my skillset, would be a better fit for the SRE role. In some ways this isn't surprising: at Google, which seems to have originated the SRE idea, there are Software Engineer SREs (SWE-SRE) and Systems Administration SREs (SA-SRE). The SWE-SRE focuses on making things faster and more scalable, and the SA-SRE focuses on automation and monitoring. For both, the geek in question has to be good at working with complex systems on a broader scale than just a single project, and that sort of systemic understanding is one of the things I'm good at. Honestly, if you need someone to go heads-down on a software project, there are plenty of people who are better than me at the line-by-line process of writing code. My better talents are a bit fuzzier: working with larger systems of many moving parts, figuring out what's wrong or what might go wrong. My friend who referred me to the Palo Alto company said, "He fixes shit."

Anyway, I'll be talking to the VP to get some more details about what exactly he has in mind and why they think I'd be good for it.

I'm in for a second round with MV #1 on Friday: 5 hours, talking to 6 people, one of whom is in London, so I assume there's a phone call or a laptop involved or something. Then I think they'll have a roundtable on Monday, and if they decide to make me an offer, I have to clear up some things about the role with the team director before I'd consider taking it.

I should be employed somewhere by the end of next week. Almost done!

Saturday, September 4, 2010

when I look up, I just trip over things

New Zealand had an earthquake. Here's some pics.

I've been getting my geek on by learning about discrete-event simulation. That's a fancy technical term for "making a computer model things like banks and fast-food restaurants," with independent events (customer arrivals) that take time and require limited resources (like cashiers). There's actually a large and important branch of mathematics for this, called queueing theory, but beyond me having a hard time and no interest in learning math on my own, the kinds of systems I'm interested in actually get too complicated for formal analysis. We saw all kinds of nutty stuff in Danger's backend system, for example, which (so far as I know) has too many moving parts to divulge its secrets in formal analysis. So you simulate it and see what happens, and it's faster and cheaper because you don't need 3 weeks and $8 million worth of computers.

We desperately wanted a simulation at the time, and one guy started writing one, but writing your own discrete-event simulator isn't a small project. Like so many things in software, it's not that it's hard, exactly...not simple, but mostly tedious, and a pain to get it right. Fortunately, other people have already done this, many times, and I'm playing with a thing called SimPy. It's pretty cool, and I'm not sure if it wasn't up to snuff in 2005-2006, or what. I want to use it to model something like the Danger system and see if I can produce the screwy behavior we saw, so I can think about what caused it and how to do things differently in my future projects.

Programming is unfortunately more fun than teaching, and way more fun than lesson planning. I'm trying to work on the planning anyway, though, since then I can enjoy my Sunday.

We had a lovely aikido class this afternoon. Jorge says he may add 2 more classes, in the evenings at a different location, which would be pretty awesome. It's a fun group for me to train with, because they're all really nice and Jorge studies aspects of aikido that interest me. It's also fascinating, though, because I'm far and away the most experienced student there, and it turns out that my training has been really good. Although we don't talk about it much, the standards at Aikido West are really high compared to many other places: partly because of the impossibly high level of accumulated experience in the students there, but also on purpose, so that when we go out into the world, we can acquit ourselves well and represent our teacher and our association. We've all had the experience of training with someone and thinking, "Wow, you're a second-degree black belt? Really? Because you're not very good." Aikido West intends that one of our students should never be that person.

From what people say and how they respond, it seems like they're pretty happy to have another experienced person around, someone who's maybe a bit more willing to show details of techniques than Jorge is. It seems like they're interested in how a more experienced person trains with Jorge.--he's bigger than me (about 6'1", 200lbs), and while he doesn't use any strength in throwing you, it's quite another thing for you or me, at our level, to throw him. Many of the students are smaller women, so it may be helpful to see that scrawny little me can throw him, without breaking a blood vessel with the effort. And I think Jorge likes having someone he can fling around a bit more fearlessly.

Odd side effect: despite the lack of training, I think my aikido has improved, and will continue to. Manuel (the teacher in Santiago) and Jorge both teach styles where relaxation is important, and I just don't have the physical strength and endurance I'm used to having, so I'm finding out to do things the relaxed way.

Short version of everything: I'm excited to be home, which makes it hard to focus on being present here and focusing on what's needed in each moment. Just like normal, only more so.

(The title is from Ani diFranco's excellent song "As Is": "When I look down, I miss all the good stuff / When I look up, I just trip over things.")