Sunday, November 02, 2014

Synology hacked by bitcoin miners?

More Fun with the Diskstation

After I'd made several posts in the past about my adventures hacking the Synology Diskstation, I got away from messing with the device due to several factors. First, I've been dealing with family issues that were more urgent and important than the device. Second, the client I was working with developed a different method for backing up their files so I didn't need to perform the remote backups anymore. 

A year later, I logged into the Diskstation to check on updates, as I'd not run any in a while. Synology updates the DSM software frequently and the update option in the control panel usually finds a new version. This time, there was no indication there was a new version available. This was very strange since I left my version, 4.2-3211 out of date on purpose due to bugs encountered in subsequent releases.

I then went to the Synology website and found that I was indeed several versions behind that the DSM software was up to version 5. I figured that perhaps my tinkering with the Diskstation caused a problem with the update. Or perhaps the Synology guys had read some of my posts and said, "Ok, we'll make sure your device is dropped from notifications. No updates for you!" 

Not a problem! The Synology site has a download page where you can get the update file and then manually update your device. But when I tried to perform the manual update I got an error saying "Field value is invalid". Those Synology guys must have really hated my posts, right?

The reality is much worse. The Internet is filled with pinheads. Crafty tech-savvy pinheads. There was an exploit from some hackers that found Diskstations and put processes on them to turn the servers into zombies doing mining/farming work for bitcoins. Older versions of the DSM software are vulnerable, and one of the signs of this is that it breaks the automatic update capability. 

Thank goodness the Internet is also filled with crafty good people. I found a post [Arinium Blog] at Arinium Blog that discussed the same issue I had. The fellow there had the same version of the DSM and was having trouble upgrading. He did not identify the hack as being the issue, but he successfully identified there was a problem with 4.2-3211 and that upgrading manually to 5 wasn't working. His solution was to tinker a bit and go to 4.3 before going to 5.

The bitcoin hack is referenced in the comments section of the post. One of the responses references a dialog with Synology where the exploit is noted. The Synology support team suggests two options:

  1. Shut down the Diskstation. Pull out the hard drive, replace with a single spare hard drive, and then update the DSM. When finished, shut down the Diskstation again, reinstall the original hard drives, then start up. 
  2. The other way is to reinstall the DSM software. There is a link in the post to instructions on how to do this. 
I didn't like Option 1 since it was a bit of hassle, getting a spare hard drive and messing with taking the existing drives out. Option 2 involves a mildly arcane exercise of pressing the reset button on the back of the Diskstation, then doing it again within 10 seconds and then doing some checks after logging into the station.

I instead did something else:
  • Download the oldest version of the DSM 4.3 (in my case 4.3-3776)
  • Download the latest version of DSM 5 
  • Manually update to 4.3
  • In my case, the Diskstation behaved a little oddly, like it wasn't taking the update, but then it rebooted on its own and came back with DSM 4.3 so it must have worked
  • I then manually updated the Diskstation to DSM 5
This all seemed to work at successfully updating the DSM software, but it did reset a number of settings. I had to reinstall several of the applications like Audio Station and I had to reapply DDNS settings. I'm still working to configure some things. And I'm noticing so far that my old nemesis, the Diskstation's refusal to sleep, has returned. But reviewing the process monitor shows the new version of Audio Station is running the indexing process on the media files, so perhaps this will pass when it is done.

Your mileage may vary depending on how badly your system was compromised. Some of the posts indicate people had to do more tinkering to get things straight again, but I'm glad I was able to fix it without having to fiddle with hard drives.

In any event, a big "thank you" goes out to Ari, of the Arinium Blog for his post. 

Monday, May 26, 2014

Annual Memorial Day Post 2014: When the Truth is the Casualty, it Hurts Everyone

This is a day to remember our veterans and fallen heroes. The one I mourn the most for is a warrior we often forget, not just in the military or in IT but in all life. This warrior's name is Truth.

The truth is a simple but beautiful thing, if you allow yourself to accept it. It is the understanding of something with complete clarity, totally free of bias. It's something that is not subject to argument, it does not take sides, it brings us answers and as the old adage says, it sets us free.

Such a wonderful thing should be revered, even cherished. We humans instead fear the truth. We bury it under our weighty bureaucracies of politics and pettiness. We worry about the burden of individual accountability it brings and we spent more energy deflecting the truth than it would take to accept blame and issue a reparation for a mistake. We've turned lying into an art, an art that in some professions is lucrative.

There's an affliction of lying that's pervasive in human culture. It's probably older than prostitution, but I often use that convenient scapegoat of the Vietnam Conflict as a recognizable symbol to describe it. It was common in Vietnam for American "leaders" to only want to pass good news up the chain rather than the truth. No one wanted to lose a job, so they kept telling their bosses, "Everything is good." Tangible things like body counts became the superficial manifestations of managerial dog treats.

Does that sound familiar? If you work in any modern company, it probably does. In Vietnam, the cost of such institutionalized lying was a meager sixty thousand American and countless more Vietnamese lives. In Corporate America, the cost is a numbing level of inefficiency. I see it in every company I've ever worked for or dealt with. It's not that companies can't be profitable even with the inefficiency. Many are. They have to be to survive. But they could be so much better. Responding sensibly to the truth would improve many lives and jobs.

It appears however that we will be unable to overcome our fear of the truth. Our politicians continue to come across like a bumbling litany of clowns and in our companies I rarely see "leadership" serious about identifying real opportunities to improve and engage trans-formative measures. Serving clients and workers becomes less important than protecting management. What a shame. This is how we use the freedom our veterans died for?

I can understand why people fear the truth. Another old adage says, "The truth hurts." But it hurts because it scrubs away the fester left by lies. We can give lip service to our fallen troops until the end of time, but when do we make moves to be better than we are, to make a society that would truly honor them?

Sunday, January 05, 2014

Second Thoughts on Having a Personal NAS

A year ago I finally took the plunge and joined Amazon Prime. What a happy prison it is. Good discounts, fast shipping, and lots of incentives to buy an Amazon Kindle tablet. But that's not really what I wanted to write about. It's fallout from being in the happy prison that has caused me to question whether my approach to having a personal NAS is a good idea.

So here's what's happening: I'm now buying a lot of ebooks from Amazon. I've got a Nook HD+ tablet so I also buy them from Barnes and Noble. I also have found my way on to some nice free ebook mailings. And I have digital magazines on Zinio. And comics on Comixology. And more ebooks on Steam, and some on Humble Bundle, and some more from Groupees and still more on BundleHunt. I also have a few loose ebooks on my local drive, managed by Calibre.

Do you begin to see the problem? In a world where technology is supposed to make life easier, I now have several more accounts and passwords to remember, and the sad truth is I'm probably not going to read but half of those ebooks, and that's being optimistic.

"So wait," you ask, "isn't this exactly why you got the NAS? To put all that content in one place and be able to access it from any device?" Well, sort of. The effort involved in transformation of that data from the commercial cloud to my personal cloud is sort of a pain in the ass. It's more effort than memorizing ten passwords.

When I use Amazon's cloud service for storing my MP3's or Microsoft's SkyDrive or DropBox for a commercially provided network storage, it's really convenient. Security, infrastructure, capacity and maintenance are all someone else's problem. I do get the point of the personal NAS: I have full control of my content and if Amazon goes out of business (unlikely) or Microsoft decides to pull the plug on SkyDrive or change it into something else (less unlikely) then my content is still safe on my own hardware. Not to mention that if any of the data is sensitive such as client information, it's better on my own device than on someone else's.

But for non-sensitive materials, I'm not sure having a personal NAS is really that big a deal. I love the Synology Diskstation I have, but it wasn't free. And it's not free to maintain, although as you've learned from my last several entries, harnessing additional functionality was really cool.

I think what I need is for someone to write a consolidation app that pulls all of this together. In the meantime, I've got a Frankenstein of a storage approach. And you know what? Even with all their problems, the happy prisons that Amazon and Steam give me for all those books, music, and games are awfully comfortable and I'm glad to have them.

The Best Bonus I Ever Got

I know I complain a lot on the blog about IT management. Well, in my opinion, IT management asks for it. Just like lawyers do when they send our society on a downward spiral to hell on riptides of lies and blame deflection.

But this post will be different. I promise. Today, I'll talk about the extra bits of cash compensation employees get outside of their base salaries. These have been far and few between in my career, so maybe this will also get to be a blessedly short post. Apologies again in advance, for some salty language that might follow.

The first bonus I got didn't come until about five years into my career. A lot of that had to do with the crappy company I worked for, but a lot also had to do with me being an inexperienced and poorly managed resource. Anyway, it was a day cruise given to my team for working hard. I appreciate the gesture, but it was on the lame side as rewards go. And I didn't like how only half the team got to go and in retrospect consider this a managerial mistake. Some of the newer team members weren't included (in what I would bet was a cost-cutting move). I felt that was not a smart way to handle team morale in an effort about raising team morale. But it's the thought that counts, so I count it as a bonus even though it sucked in more ways than one. Shit, sorry, I was supposed to be positive in this post. I'll try harder on the next paragraph.

The next bonus was much better. It came in my sixth year with my first company. I had moved to a new, smaller, team and I was doing a much better job of being useful as I'd become more experienced. I also had a more laid-back supervisor and a pretty reasonable manager. My team received an end-of-year bonus of about three thousand dollars. Not enough to buy and island and retire, but nothing to sneeze at either. What is so damn goofy is that I worked less hard for that bonus than I did for the day cruise.

I switched to contracting for a while and bonuses are typically not part of the compensation structure for hourly employees, so there's nothing to report until I switched back to full-time work about 1999. Then I got a variety of bonuses. An annual performance bonus could be between two thousand to five thousand dollars. A spot performance bonus I got was three hundred dollars.

I bounced back and forth between full-time and contracting for a while after that but didn't get another bonus until I was again full time and had a manager that appreciated my work. I killed myself for more than a year straight of overtime and got a spot bonus of a thousand dollars.

I think it's fair at this point to note some lessons I've learned about bonuses. Your experience may be different. In fact, I hope it is. I hope you've done significantly better.
  • Bonuses are usually but not necessarily tied to company profitability
  • Bonuses are highly dependent on your immediate superiors and their superiors
  • Bonuses are a very subjective thing.
    • At one company I got almost no bonuses until the end, and I was working less hard than I did in the earlier years. Some employees told me of bonuses they got for putting in a mere hour of overtime. Now that's the kind of consistency that earns employee trust!
    • At another company, bonuses sometimes came with formal recognition in the form of "President's Awards" or "Outstanding Performance Awards". These were REALLY ridiculous. It's not that some of the people getting them didn't deserve them. The problem was that the significance of the achievements earning these awards were all over the map. Some people got them for working hard on a specific important project, even though the teams on that project might have had several deserving people. Or two people might get awards for working on different projects, even though one was a multiple month or multiple year effort and the other was a one week commitment. It all came down to who had the manager that liked them, and in the end, I think this hurts morale more than it helps. Getting no recognition really hurts when you give your heart and soul for a long time and when you really make a difference. I'm not sure what the answer is for this bullshit though because for the people that deserve it, it is nice to see them get something.
  • Don't depend on bonuses. They're not guaranteed. Hold their feet to the fire in salary negotiations. If you get a bonus, great, but either way you will get the salary.
  • IT shops are pretty barren when it comes to bonuses especially when the company treats IT like a cost center. For sales and a few other divisions, bonuses may be a more legitimate part of the compensation structure.
  • If you want to work in IT and get bonuses, find IT shops in companies where an annual bonus is universal to the pay structure. For example, one of my clients was a trading firm, and everyone, even IT, got significant bonuses (like 20-40% of the salary, a concept that is completely alien to me!).
So which of the bonuses above was the best one? I am thankful for them all, but the answer is, "none". The best one didn't come from management, it came from my users. One of my clients had a legacy system that had (and still has) a terrible user interface. They were suffering greatly on having to enter data one row at a time, spending multiple man-days of effort each month. I added a simple import capability so they could massage their data in Excel and then import it through cutting and pasting. Did it work? A few weeks after the feature went live, I got this from them:

That's right: a modest $25 gift card, for a place that makes food that's mostly not on my diet. It's the best bonus I ever got. Why? Because as Jeff Atwood would say, it showed that people were using my software. It showed that my work improved lives. What makes this bonus great is not even the $25, but the kind comments from my users on the card it came with. 

Now I'm sure there are managers looking at this and saying, "Gee what an asshole that Bernard is. How could that meaningless shit be worth more to him than a thousand dollars?" Man, if you're a manager saying that right now, I pity you. You have completely missed the boat on how to do your job and how to be a leader. And I pity even more your subordinates.

Oh shit, I'm supposed to be positive! Ah, ok, well, I took the card and had a nice date with my wife, eating wings before a movie.

And for any overly literal pinheads reading this, no, this doesn't mean I don't appreciate monetary bonuses. But really, this kind of recognition is truly special and particular to software developers in the same way that a compliment to a chef or an artist means as much emotionally as the money. The chef gets a paycheck either way, but if he knows his clients were enriched by his cooking, he has a sense of purpose fulfilled. And this really is where IT management really needs to get a better understanding of how technical people respond to feedback.

We really don't give a shit if you praise us for good attendance or being on time to meetings. We do like pizza, but throwing a pizza party isn't really doing much for morale. When you use metrics like how many SOX audits we passed or how little we were penalized for dress code violations, you're just drawing attention to the parts of the job that suck. 

Wednesday, December 04, 2013

Craftsmanship is Dead

Sorry in advance for the negativity but I just encountered something that has really pissed me off. Also, apologies in advance for salty language, but what I'm about to show you doesn't deserve professionalism. Again: WARNING, FOUL LANGUAGE FOLLOWS.

I complain about software developers misunderstanding their job as "easy" mostly because they're lazy bums that don't want to do a complete job. But developers aren't the only half-assing bastards out there.

I'm still living in my first home and I am regularly appalled at the shortcuts I've seen the builders take in making this house. There are places where the most fundamental of building constructs, the 90 DEGREE RIGHT ANGLE, were built incorrectly. Good grief, how stupid do you have to be to mess up a right angle?

When I look in the attic, it's just a head-smacking collection of hacks and patches. And I paid money for this?

Yesterday I replaced a garbage disposal unit that went bad after 14 years. This Whirlaway 191 1/3 horsepower unit is not what pissed me off. It was dirty but 14 years in a day when everything is disposable is not bad, and really it was just one of the blades that went bad, the rest of the unit probably could have soldiered on a couple more years.

But the part that got me was when I went to remove the existing power cable so I could transition it to a replacement Whirlaway 291. The power cable's ground wire had not been fastened to the ground screw on the old unit. Wow. Really? The guy that installed this was that lazy? He just cut the ground wire so it wouldn't be in the way, connected the other two wires, and plugged it in. Not a care in the world about any possible power surge or electrocution. Fucking asshole!

And it gets better! When I took a look at the plug to verify the larger prong so I could trace it back to the exposed wire and connect it to the proper wire PER INSTRUCTIONS, this is what I find:

Yep. Take a close look at those prongs. Just look at it. Yes, that's right, the lazy fucker filed them down so he wouldn't have to worry about whether he had the right wires connected. FUCK ME. I thought this kind of bullshit was supposed to exist only in the realm of Tim Allen jokes.

Yeah, I know, there are dozens of professional electricians out there that will say, "Oh, you're making too much of a big deal about it. This is low voltage bullshit that won't hurt anyone."

Except...THIS ISN'T ABOUT VOLTAGE. This is about doing a COMPLETE, THOROUGH and CORRECT job! This is about the simple task of following instructions handed down by the professionals that made the garbage disposal unit. Was it really that hard that the contractors couldn't do it right? Would it really have taken that much longer to do it to specifications? To be, oh, I don't know...SAFE?

Here's the kicker...the guy that did this is not just lazy, he's stupid too. Because he didn't file down the ground prong, filing down the other prongs doesn't make a difference...the ground prong forces you to put the plug into the socket correctly. So I'm left to think this guy filed down the prongs out of habit.

It turns out the wires were connected correctly, but the evidence points to this being a stroke of luck rather than the product of professionalism and preparedness.

Holy shit. I paid for this shit. This entire thing is just fucking embarrassing. It brings me such comfort to know this house was at least partially built by zombies.

You know, in the past I've withheld names to protect the not-so-innocent. But I can't take it anymore. The only thing that will make me feel better is the truth. I bought this house from PULTE, owned by BILL PULTE, a so-called MASTER BUILDER. The fact is that Bill Pulte never touched my house, it was one of the thousands of contractors he's got working for him that he's never even met (decision-consequence gap, bitches!). So there you go. When you're looking at houses, remember what I told you. But I wouldn't be surprised if other builders were cutting corners too.


Saturday, September 21, 2013

The Linux Adventure Part 5: Everything is Broken

Ye gods.

You know all the stuff I wrote in the last five blog posts or so about the NAS unit? Well, it turns out that while it worked, side effects broke some the unit's functionality. First, the unit's network light would flicker and then it would cease to go to sleep, a critical function for energy savings on a 24/7 appliance. Then I also noticed the Diskstation would not shut down or restart when given the manual command to do so in the software.

After much gnashing of teeth and many face palms, I learned that bootstrapping the unit caused the issues. Synology generally is quite liberal about bootstrapping, even including information on how to do it in its official support resources. However, the operating system software, Disk Station Manager (DSM), gets updated regularly. Usually, this is a good sign that shows a company hasn't abandoned its product and is actively supporting and improving it. However, it also means that the environment running the station is a moving target. And by also allowing bootstrapping, Synology has led me into the very bear traps I see corporate IT shops sticking their balls into every day.

Young Profession, Old Debate

In the corporate IT world, there's an established debate about "build vs buy". Do you build the software you need from scratch or do you buy an off-the-shelf package? The debate usually has these points:

  • PRO: Software is customized to your needs and way of doing business
  • PRO: You have complete control of the code
  • PRO and CON: You have complete responsibility for the code
  • PRO: Proprietary business knowledge is institutionalized in the code
  • PRO: Enterprise processes are enforced by the software globally (barring proper implementation)
  • CON: In-house development is expensive
  • CON: In-house development often requires non-software shops to have proficiency in software development (bigger con: in reality most IT shops have at best mediocre competency).
  • PRO: Properly implemented, a custom software team can more rapidly change the software than a major vendor can or will
  • PRO: Buying someone else's software is less expensive because you don't have to have in-house development resources and licensing is cheaper than development. That's the theory reality I'm not convinced it's less expensive, but I agree that writing a check monthly is probably less work than having to account for a staff of developers and all the extra HR overhead.
  • PRO: Buying means you just use the software and the vendor handles development and support. Those of you that know better can laugh now.
  • CON: Your business now does business the way the vendor's software wants you to, not necessarily the way your business people want to.
  • CON: There may be limited capacity for customization of the vendor software to implement proprietary strategic processes
  • CON: Reality shows that for core business (not commodity items like word processing or spreadsheets) buying is still expensive and implementation missteps often negate any cost advantage
  • PRO: Certain industry standard processes, if implemented well, can be a part of the vendor software
  • CON: It is rare that all companies do business exactly the same way
  • PRO: Vendors may be able to capitalize on integration with common third party packages for things like accounting
  • CON: You may think that a vendor is more dedicated to the principles of good software design and best practices and therefore will deliver stable, efficient and intuitive high-quality software just like the kind you find on Apple computers. The truth is that some shops may be very good, but most are made up the same knuckleheads that soured you on the "build" approach.

We Want to Have our Cake and Eat it Too

In the 80's and 90's it was common for companies to take a "build" approach. Software was new and exciting and there weren't many vendor packages available, so the typical IT guy said, "Oh, it's easy, all we have to do is..." and much spaghetti code was born.

Later companies started realizing what a mess it was to maintain crappy systems. Prodded along by drinks on the golf course and rides on corporate yachts, they decided to buy instead. But they made a critical mistake. They still wanted to do business "their" way and that required changing the software they bought. They wanted the best of both worlds but ended up with the worst of both worlds by buying and then heavily customizing.  Oops.

Now they had vendors that wouldn't or couldn't respond quickly to changing the software to fix or add desired functionality. Their in-house customizations did address some of the core software's gaps but built by inexperienced developers, proved cantankerous and bug-ridden and the users hated using them. And there was another issue: when the vendor had a new version, upgrades were that much harder because they could break the customizations. Many would suffer nervous breakdowns during this time, but consulting companies would happily offer help in exchange for a chunk of the company's life savings.

Holy Crap I did it Too!

That brings me back to the NAS unit. I got it initially so I could store my photos, documents and music in one location instead of having them dispersed on four different PCs and dozens of other flash cards and portable drives and memory sticks. I could have rolled my own by putting together a simple Linux server in a low profile case, but I wanted the off-the-shelf solution that would let me plug-and-play. I wanted the benefits of commodity.

But like the corporate idiots before me, I got greedy and wanted this wonderful Linux server to do more. So I bootstrapped. I customized. And I encountered an incompatibility with the latest version of the NAS software that doesn't work when you also have the NAS change the default shell to bash from ash. It caused other commands in the script to fail, so several services such as the sleep function and the audio server and photo server ceased to work. The unit would not even heed the manual shutdown or restart commands. Ugh.

I suppose I'm not quite as stupid as the so called "leadership" which commits to decisions that really create hassles for thousands of people and ultimately cost billions and billions of dollars. My suffering is confined to just me; honest men wouldn't have it any other way. But I do feel some embarrassment at having made a similar mistake.

Sometimes It Takes Two to Mess Things Up

I had help though. For the first year I owned the Synology unit I was pretty happy with it. I still am, when it comes to the basic functionality of the unit. However, when I spend money on something I expect it to work, not to be brittle like the rest of the software out there. The support forums for the Diskstations are filled with people that have similar problems as mine, and some even from folks that did not bootstrap. It appears that the regular updates to the DSM software can be risky, which shouldn't surprise me, but the level of dramatic errors and functionality loss that can occur do. This is Synology's hardware, not mine, so they ought to be able to release a beta that doesn't crush functionality. This excellent thread [] shows though that apparently Synology's developers took several short cuts and made some sloppy moves in building their software and products.

In my case, I had to remove the lines I added to profile config to launch the bash shell and allow it to remain in the default ash shell. Now the Diskstation again responds to the manual shutdown and restart commands. After reindexing, Audiostation is back to working status. However, the sleep behavior is still broken, and now the next thing for me to try is downgrading the DSM software (currently in version 4.3 beta) back to perhaps version 4.2. The DSM front-end software warns before doing DSM updates that the DSM software cannot be rolled back, but thanks to the enterprising user community, there are ways to do it.

I'll be working on downgrading the software. I will probably lose the GUI-based task scheduler since that was part of the 4.3 beta, but I may still be able to access crontab in ash, and create the scheduled job that way. And having the Diskstation off for a bit isn't so bad; it'll be nice not to worry about those Internet pinheads that keep probing the machine. In the meantime I'll continue to use my laptop for some of my Linux needs.

Monday, September 09, 2013

The Linux Adventure Part 4: Putting the machine to work

All right. Moving the vpnc command to the main script has indeed worked, though I do need to quickly get onto obfuscating the password so it doesn't sit visible in the config file.

Once I did that, I confirmed it ran ok, connecting to the VPN, running the database copy, and then disconnecting from the VPN. All ran well.

To automate, I utilized a new feature of the Synology Diskstation's DSM software, version 4.2. There's now an integrated Task Scheduler. Interestingly, it doesn't appear to be a simple GUI interface to crontab. Instead it has some proprietary commands and probably data structures. It's also poorly documented. The help file explains what the various parts of the Task Scheduler screen are but doesn't have Synology's usually good tutorials on operating the feature. DSM 4.2 is in beta I believe, so perhaps the documentation will improve when the final product is out.

In any event, it's fairly self-explanatory to set the custom user job up. You give it a name and enter the command exactly as you would enter it at the command line interface, and then set a frequency. You can also run on demand at any time from the GUI.

I don't like that there's no record of tasks in the logs when you run them. Perhaps that will also be improved in the final release. But as it stands, when you run a task it doesn't look like anything is happening and none of the task's output is displayed anywhere.

You have to run this line from the terminal session and it will give you information on the scheduled tasks and what their last run status was:

/tmp/synoschedtask --get

Aside from the /tmp directory being a weird place to put a main feature, this line will return a list of the scheduled tasks and their configurations, along with a last run time and status.

At this point I have a boat load of error handling and feedback features to add but can now start taking advantage of automation to have this thing run automatically every day.