Saturday, January 23, 2010

RSS and Computer Security Intellegence Gathering

A short time ago, I created an OPML of computer security related RSS feeds. I hand this file out to all of my students and show them how to use google reader as an intelligence aggregation and threat response tool.

Simply put: using this OPML file (included below) and the search function within google reader (or any RSS aggregator) you can gather everything from current exploits, blog punditry, snort rules, and academic research on any computer security topic.

Try it out. Load the OPML (linked below) and the search for "Internet Explorer" (our favorite browser) and see what turns up. You won't be disappointed.

I've included a listing of sites below, and would appreciate it if you left a comment with links that I missed. Feel free to link against the OPML file, I'll keep the URL static for as long as I have this domain.

Computer Security OPML File

Here's an expanded list of the current Computer Security OPML File:

Tuesday, December 29, 2009

A Well Tempered Sysadmin

There is no single path to becoming a good sysadmin, this process takes time, talent, and dedication. There is no easy place to begin this discussion, and there's an amazing amount of controversy around this subject. So let me begin with what I know about myself. I will tell you two different stories of my life; I promise to lie only once.

The First Story:

I've been a professional sysadmin for almost 7 years. In that time, I have worked in many different countries, working alone or with teams of other sysadmins. I have built global networks, been personally responsible for millions of dollars worth of equipment, and debugged mission critical infrastructure. I also teach night classes at a local University. I've done all this without a High School Diploma.

The Second Story:

I have attended 4+ years of college learning about Information Systems and Computer Security. Last week I was awarded a dual Masters degree in Information Assurance and IT Management. I have two certifications, and will be studying for more now that I've reached a major milestone in my formal education. I cannot imagine my life without the benefit of education and the wealth of knowledge that it has brought me.

My lie is one of omission. Each story is true, but each story only represents part of the truth. My life, in fact, is a combination of both.

The Truth:

I've been a professional sysadmin for almost 7 years. In that time, I have worked in many different countries, received a few certifications , and attended 4+ years of college. I have built global networks, debugged mission critical infrastructure, and managed millions in equipment, all while finishing a dual Masters degree. I also currently teach night classes at a local University. I've done all of this without a High School Diploma, but I cannot imagine my life without the benefit of a formal education and the wealth of knowledge that it has brought me.

What did I learn in the past 7 years? Education and experience must work together, or they will only be incomplete halves of a story.

Mine is not an example to follow, working full time and attending college full time is mentally and physically challenging, and not for the faint of heart. This works for me because I have a passion for my work (and a very supportive wife). But I would try to leave you with a little practical advice:

  1. Find something that you are passionate about (networks, operating systems, enterprise architecture, whatever). If you're like me, and like a little of everything, find something that is convenient or important to your current work.

  2. Find a certification track or school that offers what you want to learn. Then jump in, rip apart the coursework, and make it your own.

  3. Find a mentor that can help you when you are stuck or motivate you when you are slacking. I've had many different mentors (some didn't even know the roll they played), each one brought something different to my life, and I value all of their contributions.

  4. Once you have reached an understanding and an internal level of mastery, you must complete the training. You must study for the test, or complete your degree. The time you spent following your passion can now be converted into better leverage during the hiring process. Here is the true alchemy to a happy life: turning passion into profit.

Education is a racket, you will not learn something through the mere awarding of a degree or certification. You only get what you take from your education. Treat each class like an outline and your coursework like a single bullet in the outline. Don't waste your time; attack your education with passion and determination. Build upon success.

Experience is overrated, knowledge of a sysadmin is hard won and quickly outdated. You must search desperately for the underlying truth in all your actions. The experience you gain must be fundamental, and must be applicable to many situations. Failures are costly, do not waste them.

Education stands on the shoulders of success, building over time. Experience flows from failure, sharp reminders that help us predict the future. A well tempered sysadmin will seek both success and failure with equal delight. Success expands our horizons and creates new frontiers to conquer. Failure keeps our head in the game and builds an understanding of the future.

It is within this combination of education and experience, of success and failure, that we may find true understanding and fulfillment. Finding work that fulfills our needs and pays the bills, is the beginning of a life well lived.

A New Year is upon us, and I personally want you to be passionate about your work. Pursue your passion, and turn your experience into education. If you don't know where to begin, start with the links below. Good luck, and have a passionate New Year!

Further Reading:

500 Word Summary of Dewey’s “Experience & Education”

The World Needs You to Do What You Love

Are There Blue Collar and White Collar Admins?

SAGE Sysadmin Core Job Descriptions

108 Things a Sysadmin Might Do

How To Become a Hacker

Learning From Failure

Wednesday, December 23, 2009

How To Become A Hacker for the New Year

I'm getting a jump on my New Year Resolutions by reading How to Become a Hacker by Eric Raymond. From this beautiful essay comes a list of things every hacker (or sysadmin) should know how to do well:

  1. Learn to write your native language well. Though it's a common stereotype that programmers can't write, a surprising number of hackers (including all the most accomplished ones I know of) are very able writers.
  1. Read science fiction. Go to science fiction conventions (a good way to meet hackers and proto-hackers).
  1. Train in a martial-arts form. The kind of mental discipline required for martial arts seems to be similar in important ways to what hackers do. The most popular forms among hackers are definitely Asian empty-hand arts such as Tae Kwon Do, various forms of Karate, Kung Fu, Aikido, or Ju Jitsu. Western fencing and Asian sword arts also have visible followings. In places where it's legal, pistol shooting has been rising in popularity since the late 1990s. The most hackerly martial arts are those which emphasize mental discipline, relaxed awareness, and control, rather than raw strength, athleticism, or physical toughness.
  1. Study an actual meditation discipline. The perennial favorite among hackers is Zen (importantly, it is possible to benefit from Zen without acquiring a religion or discarding one you already have). Other styles may work as well, but be careful to choose one that doesn't require you to believe crazy things.
  1. Develop an analytical ear for music. Learn to appreciate peculiar kinds of music. Learn to play some musical instrument well, or how to sing.
  1. Develop your appreciation of puns and wordplay.

Eric's vision of building a better hacker (or sysadmin) is an excellent way to start your list of New Year Resolutions. Personally I would love to see more hacker/warrior monks on the street in 2010.

Wednesday, November 18, 2009


I switched my RSS feed to feedburner, Enjoy! Please resubscribe, if I decide to change blog platforms, there will be no need for you to change RSS feeds again.

Monday, November 16, 2009

Sysadmins And The Turbulent Waters of PEBKAC

Cory Doctrow's recent story Epoch (commissioned by Mark Shuttleworth), has a brilliant passage about sysadmins:

I will tell you a secret of the sysadmin trade: PEBKAC. Problem Exists Between Keyboard and Chair. Every technical problem is the result of a human being mis-predicting what another human being will do.

Surprised? You shouldn't be. Think of how many bad love affairs, wars, con jobs, traffic wrecks, and bar fights are the result of mis-predicting what another human being is likely to do. We humans are supremely confident the we know how others will react. We are supremely, tragically, wrong about this. We don't even know how we will react.

Sysadmins live in the turbulent waters of PEBKAC. Programmers think that PEBKAC is just civilians, just users. Sysadmins know better; sysadmins know that programmers are as much of the problem between chair and keyboard as any user is.

They write the code that gets users into so much trouble.

I've met more than a few sysadmins who don't like dealing with people. This point of view is a tragic mistake. People design these systems, people give value to information they hold, and people create the need for sysadmins in the first place.

Sysadmins above all: manage and troubleshoot the relationship between people and services.

Cory's done an excellent job distilling the many facets of sysadmin work while still making it accessible to the average person (ie. non-sysadmins). Epoch is Cory's second story about sysadmins, his first was When Sysadmins Ruled The Earth.

Saturday, November 14, 2009

108 Things a Systems Administrator Might Do

When I meet new people, and they ask me what I do for a living, I usually just respond with "computer stuff". This is about as far as most non-technical people want to take it. But its always bothered me that, as a SysAdmin, I have no elevator pitch.

An elevator pitch is a 30 second description of a service or a product that should ignite the interest of the audience. Being able to describe yourself or your work in an exciting manner for a general audience is an important rhetorical skill. Ultimately it may save your job or help you get a new one.

I recently found a list of 108 Tasks that a Systems Administrator Might Do, it appears to be from a SAGE article or document entitled: Analysis of the System Administrator Occupation. I dumped the entire list into wordle and created a weighted list. I was hoping that this visualization would help me build a narrative about System Administration, and help me create an elevator pitch.

While I'm still working on my elevator pitch. I thought both of these lists were too useful to keep to myself any longer than necessary. If you have your own SysAdmin elevator pitch, or would like to add anything to this list, leave a comment.

108 Things a Systems Administrator Might Do.

Hardware Installation and Maintenance

  1. Install/configure mother boards and memory cards/chips into systems (e.g., NICs, CPU cards, I/O cards).
  2. Modify operating system to recognize new hardware.
  3. Install and maintain cabling and device hardware (e.g., peripheral cabling, power cabling).
  4. Debug cable problems to resolve issues of connectivity (e.g., breakout box, protocol analyzer).
  5. Assemble components into working systems (e.g., plug components together, replace controller).
  6. Fix/repair computer system to the field replaceable unit level (e.g., disk failure, network or memory card failure).
  7. Dispose of old equipment and sensitive material (e.g., completely erase disk) factoring in relevant security and environmental considerations.

    Peripheral and Device Management

  8. Install/configure peripherals and devices (e.g., jukebox, modems, printers)
  9. Configure device drivers and ports (e.g., serial ports).
  10. Control access to network resources (e.g., printers, modems).
  11. Maintain and configure local and remote printing capabilities.
  12. Fix/repair printing function failures and problems (e.g., queues, spooling).
  13. Manage dial-up modem banks to maintain incoming/outgoing remote access capabilities.

    Data Integrity Management

  14. Devise system administration scheme and plans to mitigate common system failures, disasters, or emergencies (e.g., file corruption, hardware failures, power surges, fire, theft).
  15. Prepare/maintain backup media tracking system (e.g., tapes, CD ROM, floppy disks).
  16. Backup necessary system files on appropriate device/media (e.g., magnetic tape, disks).
  17. Restore files and system from backup device/media.
  18. Reinstall/repair operating system (e.g., corrupt kernel image, volume header).
  19. Maintain/reinitialize or repair disk drives.
  20. Verify/ensure integrity of backups.

    Data Storage Management

  21. Prepare disk and layout for data (e.g., RAID management, format/label/partition disks).
  22. Connect and/or configure new storage devices.
  23. Monitor, verify, and correct file systems (e.g., fsck, Checkdisk, Scandisk).
  24. Create, modify, and organize directory structures.
  25. Monitor, set, and change file permissions to control user access.
  26. Monitor and correct corrupted files
  27. Monitor file system usage (e.g., disk space remaining, disk usage over time).
  28. Reevaluate/redesign file systems layout (e.g., add/shrink/enlarge file systems).

    Network Configuration and Management

  29. Coordinate network topology and design with network administrators (e.g., new installation, upgrade).
  30. Plan, obtain, assign, and manage Internet names (e.g., DNS, domain name registration).
  31. Plan, obtain, assign, and manage Internet addresses (e.g., DHCP, AS numbers, OSPF areas).
  32. Configure and manage network file/data synchronization and/or distribution (e.g., rdist, SMS).
  33. Configure and manage network time sychronization in servers (e.g., ntpd).
  34. Configure and manage network file systems and servers (e.g., NFS, RFS, AFS, SAMBA).
  35. Monitor connectivity to detect network faults and measure network performance (e.g., ping, traceroute).
  36. Troubleshoot and correct network failures (e.g., cables, hubs, routing).
  37. Configure network interfaces (e.g., netmask, broadcast, speed, mode, ppp modem).

    Internet Services and Electronic Mail Systems

  38. Configure mail systems (e.g., MTA, anti-spam).
  39. Create, configure, and manage mail aliases and distribution lists.
  40. Install, configure, and manage mail reading applications (e.g., Eudora, Elm, Pine).
  41. Manage the web server and server-related programs (e.g., Apache, IIS).
  42. Install and configure non-web host services (e.g., FTP, archives).
  43. Install, configure, and manage network news, bulletin board, and chat services.

    Software System Development, Configuration, and Management

  44. Locate/download software packages and patches from the Internet or computer vendors.
  45. Build, install, and configure operating systems (e.g., NT, Linux).
  46. Install upgrades and operating system patches and service packs.
  47. Build, install, and configure application software and tools (e.g., third-party, public domain, or shareware).
  48. Debug application software problems (e.g., business-specific software such as Adobe software such as Adobe Acrobat or Netscape).
  49. Port system utilities to other operating system environments (e.g., convert script from Perl4 to Perl5, convert script from Unix to NT).
  50. Resolve compatibility and inter-operability issues (i.e., resolving machine-to-machine problems).
  51. Audit/evaluate existing source code for problems (e.g., for buffer overflows, Y2K related issues).

    User Support and Help Desk

  52. Configure/create templates for user interfaces and user environment (e.g. CDE, browser, windows, log in scripts, shell rc files).
  53. Identify and translate potential or actual user needs into technical requirements.
  54. Verify, remove, and disable user accounts (e.g., logins, passwords, shells, account validation).
  55. Manage user privileges (e.g., security levels in groups, file server access).
  56. Train and orient new and existing users.
  57. Respond to user requests, trouble reports, and questions.
  58. Triage and dispatch user requests to appropriate personnel.
  59. Communicate system status (e.g., planned outages, cause of network crashes) to users.
  60. Write local environment documentation to support users (e.g., FAQ).


  61. Evaluate potential problems, liabilities, and costs of potential or actual security attacks (i.e., risk analysis).
  62. Identify/evaluate/implement security mechanisms and tools (e.g., IDS, tripwire utilities, intrusion prevention software, firewalls, TCP wrappers).
  63. Formulate security procedures to prevent, detect, and respond to internal and external security threats (e.g., passwords).
  64. Evaluate and create site security plans.
  65. Monitor and detect security threats, holes, and attacks (e.g., viruses, detecting users with no passwords, unlocked administrative systems).
  66. Analyze internal/external security attacks (e.g., scan system logs for incidents, analyze network packets, implement intrusion detection software).
  67. Deploy and manage authentication systems (e.g., tokens, one-time passwords, Kerberos, NIS).
  68. Manage cryptographic facilities to protect sensitive information in network applications (e.g., PGP encryption in electronic mail).
  69. Respond, resolve, and report security incidents (e.g., unauthorized access to system).
  70. Monitor emerging security threats/tools/issues (e.g., via security news groups, CERT).
  71. Perform periodic security audits to ensure security has not been breached or compromised.

    System Resource Management and Performance Tuning

  72. Create/specify service-level agreements for site primary services.
  73. Debug and/or optimize network performance and performance issues.
  74. Manage system resources (e.g., monitor user disk and print quotas, CPU usage, swap usage).
  75. Evaluate and optimize system resources (e.g., organize disk space and memory).
  76. Manage system processes (e.g., signaling, changing priorities).
  77. Modify operating system configuration (e.g., add or modify services, configure/rebuild kernel).
  78. Perform housekeeping and clean-up activities (e.g., remove files, log rotation, archive, delete old users).
  79. Develop or enhance software tools to automate tasks (e.g., write scripts).
  80. Plan and build high-availability systems for critical services (e.g., business critical environments such as banking, real-time systems

    Technical Record Keeping and Procedural Documentation

  81. Develop/maintain operational instructions and procedures (e.g., How Tos, runtime procedures, runbook).
  82. Develop/maintain records and technical documentation (e.g., software version numbers, user logins, system architecture, licenses, descriptions).
  83. Develop/maintain daily operation logs to track problems and to establish an audit trail to debug and isolate potential problems (e.g., track mean time between failures and uptimes).
  84. Audit and inventory user licenses to ensure legal compliance.
  85. Maintain data in work request and tracking systems (e.g., Remedy, clarify, Tkrep, MHQ).

    Procurement and Vendor Relations

  86. Evaluate needs and develop system design and upgrade proposals/justification.
  87. Research and evaluate hardware/software/equipment to satisfy requirements (e.g., user needs, budgetary, legal, technical specifications).
  88. Write software/hardware specifications to meet user needs (e.g., RFI, RFP).
  89. Evaluate and recommend third- party products and services.
  90. Develop/write purchase justification (e.g., based on growth and needs).
  91. Negotiate/renegotiate service- level agreements and terms with provider to optimize costs and/or services (e.g., technical support, equipment, maintenance).
  92. Establish and cultivate relationship with vendor for problem resolution, technical support, etc.
  93. Monitor vendor contract performance (e.g., track vendor response time).
  94. Place, manage, and track equipment orders.
  95. Establish/update equipment inventory.
  96. Provide/solicit information to/from vendor to fix software_ to/from vendor to fix software bugs and problems.

    Technical Management

  97. Train system administration staff.
  98. Supervise and manage technical staff.
  99. Anticipate and plan computer system resources for future needs (i.e., system capacity planning).
  100. Anticipate and plan network resources for future needs (e.g., bandwidth, redundancy).
  101. Anticipate and plan human resources for future technical needs (i.e., hiring and staffing).
  102. Manage relations between the technical staff and the user community.
  103. Audit system and equipment to ensure readiness and compliance with industry standards (e.g., ISO 9000, Y2K).
  104. Formulate and enforce information technology-related policies, procedures, and guidelines.
  105. Recommend resource allocation policies, privacy policies, and user policies (e.g., use of email and Internet, disk allocation).

    Facilities Management

  106. Anticipate and plan computer operation center resources to meet future needs (e.g., air conditioning, electrical capacity).
  107. Coordinate with facilities manager to secure power, space, and environmental resources (e.g., power-UPS, fire suppression, HVAC, equipment, lighting, safety, shelving) for computer operation center(s).
  108. Plan for and evaluate physical security of computer operation center(s) (e.g., install cable locks on desktops).

(From Analysis of the System Administrator Occupation Copyright © 2000 by SAGE, The System Administrators Guild.)
(Edited for spelling and clarity, WIP, Joseph Kern)

Thursday, November 12, 2009

Why SysAdmins Should Use Git: Reason 1

Git is so simple, you can tweet tutorials.