Adventures in Compiling - Notepad2

I've been using Notepad2 for awhile now at work since it is an approved tool, and it's somewhat useful.  I like the single document interface and the syntax highlighting, but it has a few drawbacks; it doesn't support Bash or Cygwin scripting.

Long story short I decided to download a copy and make a few modifications.

1. Download a copy of all the source code.
The source can be found on the Notepad2 page, after unpacking it into a project directory you also have to download a copy of the Scintilla source and place it in the top level project directory (ex. Notepad2\scintilla).
Notepad2 Source Code
Scintilla Source Code

2. Choose your development environment.
Pleasant surprise - everything is in C++.  Since I'm using a Windows XP machine, and I'm cheap I opted for the free Microsoft Visual Studio 2010.  Download, Install, Register.  According to the instructions a few things need to happen first though, we need to locate the "lexlink.js" script and run it (double click).  This modifies the Catologue.cxx file to remove syntax files from the Scintilla build that we don't care about.

Open up the solution and convert it and we should be good to go.

Now before we make any modifications we need to make sure that we can get a clean compile, otherwise we'll be chasing false errors as we make changes.  It's about now that I discovered that my computer doesn't have a copy of winres.h on my system.  This isn't such a big deal, a few web searches later and I locate my Microsoft SDK - Include directory (for me it's in Program Files) and added a new file named "winres.h" with these contents:

   #include <winresrc.h>
   #ifdef IDC_STATIC
   #undef IDC_STATIC
   #endif
   #define IDC_STATIC (-1)

That was easy, now I'm getting clean compiles without too much trouble.

3. Making some modifications.
Basically what I want to do is add in Syntax Highlighting for Cygwin/Bash.  I want comments Gray, Strings Green, Executed Commands Orange, Numbers Red, Keywords Blue, Variables light blue, and External Commands (from Cygwin) with a gray background.  Kind of like this:

First we need to add support for our Bash files.  That Catologue.cxx file gives us a pretty good starting point - we want to uncomment out this line: ("//LINK_LEXER(lmBash);") and while we are at it we want to add in lmBash to lexlink.js so re-running it won't mess it up again.  And then we add in the lexBash.cxx file to our Scintilla\Lexers in our project.

Now we need to add in support for our new type... searching around we run across the Styles.cxx file, this is where all of our keywords and format defaults are created.  We need to do a few things here:
- increase the NUMLEXERS variables in the header to support one more type.
- Add in our new KEYWORDLIST.
- Add in our new EDITLEXER.
- Add our new lexer to pLexArray.

I created a script in BASH to generate a list of shell built-ins and common commands.  It was easy enough to do these steps, but I quickly discovered that the Bash lexer doesn't support External Commands... only Keywords.  I added them in as a new type anyway which involved adding a new ENUM type (SCE_SH_EXTERNAL) in SciLexer.h and tossing it in as an additional case to LexBash.cxx.  We'll get back to this in a minute.

A KEYWORDLIST is an array of strings.  Our Bash Lexer reads these into wordlists for internal use.
    WordList &keywords = *keywordlists[0];
    WordList &keywords2 = *keywordlists[2]; // we added this one.

Pretty straightforward, we add the keywords to the appropriate list.  The list is handled in LexBash.cxx.  We are using the first string for BASH keywords (there are a few special cases in the lexer, so they don't need to be added).

KEYWORDLIST KeyWords_SH = {
// "if elif fi while until else then do done esac eval for case select "
"alias bg break builtin cd command compgen complete compopt continue declare dirs disown echo enable exec exit export "
"false fg getopts hash help history jobs kill let local logout mapfile popd printf pushd pwd read readarray readonly "
"return set shift shopt source suspend test time times trap true type typeset ulimit umask unalias unset wait fc", "", 
"awk banner clear df diff dirname du egrep env expr fmt fold free ftp g++ gcc grep groups gzip head hostname identify "
"import integer install ipcs join ln login look ls make man man2html mkdir mkgroup more mount mv nice od perl print "
"ps rm rmdir script sed setenv sh since size sleep sort strings stty su tac tail tar telnet tidy top touch tput tr "
"umount uname uniq unix2dos unzip uptime users vmstat watch wc whereis which who whoami xargs yacc yes zip basename "
"bash bc c++ cal cat chgrp chmod chown chroot cksum cpp crontab cut date factor file find flip flock",
"", "", "", "", "", "" };







Next we add in our EDITLEXER, the first line matches the ENUM token SCLEX_BASH, a string ID from Notepad2.rc (we added: 63022   "Bash Script"), the fourth field indicates the file types that will default to this type, and then we finally get to our styles.

Each style line gives us the ENUM cases (from LexBash.cxx) that we are applying the given style to, a string ID that describes the rule (press cntrl-F12 to change rules for Bash and this is the identifier that will show up), and our Coloring Rules in field 4.  If you want to combine multiple ENUM types to the same rules you can combine up to four of them using MULTI_STYLE.  The last line is an empty rule.
 

EDITLEXER lexSH = { SCLEX_BASH, 63022, L"Bash Script", L"sh; bash", L"", &KeyWords_SH, {
{ STYLE_DEFAULT, 63126, L"Default", L"", L"" },
{ SCE_SH_COMMENTLINE, 63127, L"Comment", L"fore:#808080", L""},
{ SCE_SH_WORD, 63128, L"Keyword", L"bold; fore:#0000C0", L"" },
{ SCE_SH_EXTERNAL, 63236, L"External", L"bold; fore:#4040C0; back:#C0C0C0", L"" },
{ SCE_SH_NUMBER, 63130, L"Number", L"fore:#FF0000", L"" },
{ MULTI_STYLE(SCE_SH_STRING,SCE_SH_CHARACTER,0,0), 63131, L"String", L"fore:#008000", L"" },
{ SCE_SH_OPERATOR, 63132, L"Operator", L"fore:#0000C0", L"" },
{ SCE_SH_BACKTICKS, 63229, L"Backtick", L"fore:#FF8000", L"" },
{ SCE_SH_PARAM, 63249, L"Variable", L"fore:#0080C0", L"" },
{ -1, 00000, L"", L"", L"" } } };
OK, so now we need to make some changes to LexBash.cxx to recognize External Commands.
In ColouriseBashDoc() we tell it to save off a copy of our External Keywords:
   WordList &keywords2 = *keywordlists[2];

And we Added our new SCE_SH_EXTERNAL as an additional case in the same place we handled SCE_SH_WORD.
   case SCE_SH_WORD:
   case SCE_SH_EXTERNAL:

At this point we can compile and test... all we need to do now is seperate the behavior for EXTERNAL and WORD.  We use the "sc.ChangeState(_ENUM_VALUE_)" to apply the rule, so we need to handle the two types.  Scrolling down to the very bottom of this case we make a minor modification - when we were going to apply the INTERNAL identifier check to see if this is an external command, if so change to SCE_SH_EXTERNAL, otherwise do what we were going to do originally.

   else if (cmdState != BASH_CMD_START || !(keywords.InList(s) && keywordEnds)) {
      if (keywords2.InList(s) && keywordEnds) {
         sc.ChangeState(SCE_SH_EXTERNAL);
      } else {
         sc.ChangeState(SCE_SH_IDENTIFIER);
      }
   }

Compile, Run, Test... everything looked good.  This was surprisingly easy... I got the whole thing done in about two hours.  I think I've documented all of the stumbling blocks, and I feel comfortable modifying the code now.  I might try to tackle some other things in the code, we'll see.  I decided to throw this together as a tutorial in case anyone wanted to add support for their own favorite languages.  Enjoy.

Thanks to Florian Balmer for his excellent software, and for making it available to the world.

Complete CMQA Access Dump

I've been doing a lot of work on Access Database Migration lately and a question came up about how we could provide CM/QA level auditing of the current database.  And that is what led me to coming up with the following bit of code attached to a little button in our new database.


Private Sub CMQA_Audit_Button_Click()
   On Error GoTo Err_DocDatabase
   Dim dbs As DAO.Database
   Dim cnt As DAO.Container
   Dim doc As DAO.Document

   Set dbs = CurrentDB()
   Dim OutDir As String
   OutDir = CurrentProject.Path & "\" & Format(Now(), "ddmmmyy-hhmmss")
   MkDir OutDir

   Dim Tbl As TableDef
   For Each Tbl In dbs.TableDefs\
      If Tbl.Attributes = 0 Then ' Ignore System Tables
         Application.ExportXML acExportTable, Tbl.Name, , OutDir & "\tbl_" & Tbl.Name & ".xsd"
      End If
   Next

   Set cnt = dbs.Containers("Forms")
   For Each doc In cnt.Documents
      Application.SaveAsText acForm, doc.Name, OutDir & "\form_" & doc.Name & ".txt"
   Next doc

   Set cnt = dbs.Containers("Reports")
   For Each doc In cnt.Documents
      Application.SaveAsText acReport, doc.Name, OutDir & "\rep_" & doc.Name & ".txt"
   Next doc

   Set cnt = dbs.Containers("Scripts")
   For Each doc In cnt.Documents
      Application.SaveAsText acMacro, doc.Name, OutDir & "\scr_" & doc.Name & ".txt"
   Next doc

   Set cnt = dbs.Containers("Modules")
   For Each doc In cnt.Documents
      Application.SaveAsText acModule, doc.Name, OutDir & "\mod_" & doc.Name & ".txt"
   Next doc

   Dim QryAs QueryDef
   For Each Qry In dbs.QueryDefs
      If Not (Qry.Name Like "~sq_*") Then
         Application.SaveAsText acQuery, Qry.Name, OutDir & "\qry_" & Qry.Name & ".txt"
      End If
   Next

   Set doc = Nothing
   Set cnt = Nothing
   Set dbs = Nothing

Exit_DocDatabase:
   Exit Sub

Err_DocDatabase:
   Select Case Err
      Case Else
         MsgBox Err.Description
         Resume Exit_DocDatabase
   End Select
End Sub         
     

I'm still not 100% happy with the output of the raw SQL code, but it does allow us to run Beyond Compare on the output and gives us all of the components (of our rather complex database) including VB Code, SQL and Form changes.

Automated Storytelling

One of the things that always fascinated me about AI was Automated Storytelling, or Emergent Narrative.  This is the idea that a computer can tell a compelling story.  There are a couple of approaches to this:

1. Fill in the blanks on a template-story; Think of this option as the Mad-Libs approach.  A lot of webpages use a simplified version of this for automatic plot generators, etc... Very hit-or-miss and the computer has no real understanding of the story it is creating.

2. Using grammar-type rules construct a legitimate story the same way you would construct a sentence; pick a hero, pick a compatible mission, pick a compatible obstacle, etc...

2. Create a world and characters to inhabit the world, set things loose and look for an interesting story to develop.  Think of this option as a Soap-Opera, or a Reality TV Show.  Things are very open-ended.

3. Create a world and characters, pick compelling plot-arcs and then force the characters into situations to fulfill the plot requirements.  This option can use sub-plots and back-seeding (changing previous portions of the story to support changes in the plot).  An AI construct known as fate can be used to keep the story on track, while another AI construct works on making the story dramatic, or funny, or whatever the theme of the story is.  I like to think of this option as a card game between two players (at least that's the way I'd implement it).

I've thought a lot about this topic over the years and I think I am ready to start trying a few things out.  I have started by looking over my notes and simplifying the ideas, one thing I've learned from working on Decision Aid Systems, Expert Systems, Natural Language Recognition, and doing Automatic Report Generation is that complexity rarely leads to a better result.

So, I am consolidating my 46 traits in seven spheres:
MIND: crazy romantic sensitive clever creative funny logical critical
ACTS: lazy trusting careful perceptive secretive suspicious controlling sleazy BODY: weak klutz lazy strong tense abusive
TALK: quiet follower gossip charismatic 
WORK: reckless selfish ambitious honest careful sacrificing  
LOVE: unselfish-love friends-love logical-love game-love posessive-love romantic-love 
LOOK: overweight dowdy plain athletic cute sexy glamorous exotic
into this:
MIND: ambitious, honest, romantic, player, possessive, planning
BODY: lazy, sexy, sleazy, reckless, abusive, addict
SOUL: crazy, leader, follower, gossip

I think that this offers sufficient variety, this also means Actors will have a limit of (up to) three traits each.

I'm narrowing in on a theme too, I'm going to go with deserted island (think Lost or Gilligans' Island) since it provides an easy sand-box (limited number of locations, don't have to worry about outside actors if we don't want to).

There are a number of other simplifications I'm making as well.  My hope is that this will simplify the rules engine that guides behavior.

PS - Here is an old-style set of Actors, there are some Stats that are hidden behind them as well:
Name: ____________   Sex: female
DOB: 21 JUN    AGE: YOUNG   JOB: judge (second-job)   Income: comfortable
     lucky sensitive perceptive tense
     healthy     dowdy        LOVE-STYLE: logical-love

Name: ____________   Sex: female
DOB: 21 JAN    AGE: OLD   JOB: accountant (cold-as-ice)   Income: average
      logical secretive strong quiet
     healthy     cute        LOVE-STYLE: logical-love

Name: ____________   Sex: male
DOB: 21 AUG    AGE: YOUNG   JOB: bar-tender (ugly-betty)   Income: comfortable
      sensitive careful  quiet
     healthy     plain        LOVE-STYLE: friends-love

And, here is the new version:
Name: _____________  Sex: female   Job: Engineer
Planning, Gossip

I'll post when I get a little further.
David

The Art of the Meeting

In any business there are a number of soft skills that are incredibly useful; a lot of people have talked about communication and technical writing, but I wanted to talk for a few minutes about another area that is often overlooked: Meetings.

A long time ago (when I was at Lockheed) everyone in our group was required to participate in Facilitator training (not just those running meetings), they believed that if everyone facilitates meetings they will run smoother.  My experience was very positive with meetings when everyone had the training.

The idea was that the facilitator should do everything they could to prepare the meeting to run smoothly:
- scrub the list of required vs. optional attendees. (don't waste peoples time)
- ensure all required attendees are available for the meeting. (avoid rescheduling)
- send out meeting notices well in advance (1 week+).
- include the agenda for the meeting prior to the meeting (eg. with the meeting notice). (keep the meeting focused and on-track)

Everyone in the meeting was expected to work on keeping the meeting on-track:
- fill-in for any role as needed (eg. Note taker, facilitator if you have the expertise and the facilitator isn't available at the start of the meeting)
- keep good thorough notes for the meeting, the note taker should share the minutes to all attendees after the meeting (by the next business day).
- during the meeting keep the topic on the agenda items, other issues should immediately be relegated to off-line discussion.
- once an item has been decided by group consensus be prepared to move on.

General:
- Be on time for the meeting.
- Action Items should be re-capped at the end of the meeting to ensure they are accurate.
- Action Items should be descriptive enough such that someone not at the meeting will immediately understand what needs to be done.
- Minutes should be descriptive enough such that someone not at the meeting will understand what was discussed.
- Keep the meeting on-track, distractions such as side-bars or phone calls should not take place (or at the very least be moved outside of the room).

Facilitator:
- ensure all resources are available and you know how to use them prior to the meeting (projectors, podium, etc...)
- don't schedule meetings if an alternative is available and appropriate. (don't waste peoples time)
   Phone Call (no documentation quick)
   In-Person Talk (no documentation, quick)
   Email Chain (provides documentation)
   Scheduled Meeting (slowest option, meeting invite and distributed minutes/action items provide documentation)

I don't mean to champion etiquette, but I've been at companies where some of these items weren't followed and it makes it very difficult to be productive.  Imagine discussing an Action Item from a month ago that says "Have Jake talk to the EP Group." or having a meeting scheduled and everyone preparing for it and then in the first five minutes of the meeting being told "we aren't going to be discussing that".

I've had it both ways; Facilitator training gets my vote.

A look at the Destroyers

Just a real quick post to a few links... many of you know that I have been working on Destroyers for a little over a decade now; first with Aegis and now with the MCS components.  Here are a couple of links showing off these ships:

DDG52
DDG112

They focus more on Aegis (acquiring targets and shooting missiles) and the big guns than MCS (electric plant , propulsion, damage control, fuel control, etc...), but I'm still proud of the work I do.

DETERMINING TIME DELTAS IN DOS SHELL

I ran into a bit of a tricky problem the other day and I wanted to share it with you.  I am using a program called ATRT (Automated Test and Re-Test) to test our software remotely, and we ran into a problem where two OCR (optical character recognition) extracted time stamp needs to be compared to find a time delta.  Unfortunately, in this version of software we are unable to convert the text time stamp back into a time object, and the software does not currently have a full math library (ex. Modulus function) or token splitting capabilities to determine the delta in-house.  Since it will be a at least a few weeks before we could get these features we decided to find a work-a-round; and, since ATRT can spawn child processes with custom arguments and read in the results we figured a simple Dos Shell Script was the way to go (note - powershell, unix tools, etc... will not be available on the target systems).

Simple idea, of course it's been a few years since I've been forced to use the Windows Command Prompt and the idiosyncracies can definitely lead to some frustrations.  It took about a minute to double check some syntax and write up the script and then to do the testing... first let's look at some code:

   REM > time_delta.bat <start> <stop>
   REM > time_delta.bat 2:03:04 2:05:06
   REM 122

   set MY_START=%1%
   set MY_STOP=%2%

   REM grab the hours, minutes and seconds from the start time fields 1 through 3
   REM grab the hours, minutes and seconds from the stop time  fields 4 through 6
   FOR /f "tokens=1-6 delims=:" %%a IN ( "%MY_START%:%MY_STOP%" ) DO (
     set H1=%%a
     set M1=%%b
     set S1=%%c
     set H2=%%d
     set M2=%%e
     set S2=%%f
     echo "%H2% %M2% %S2% - %H1% %M1% %S1%"
   )

I'm going to stop right here... I ran this from the command line about a dozen times with different arguments, and it didn't work.  The idea is really simple, place the timestamp into a string of colon delimited tokens and extract each of the fields and assign them sequentially to variables starting with %a% through %f% (six fields).  If I gave it the same arguments twice in a row it would be wrong the first time, and right each subsequent time.  If I ran from the command line the arguments would not be set.

Once I understood what was going on it was easy enough to fix.  Moving my echo outside of the loop caused it to behave properly... it took about an hour to finish up the script that I thought would take five minutes because of this behavior.  Convinced that there was a pointless bug in Dosshell I went off to consult the inter-webs to find out the logic behind this, what I discovered is a bit disturbing.

Dos Shell has something called "Delayed vs. Immediate Variable Expansion".  The default behavior is for the Shell interpreter to read in all of the lines between "(" and ")" at once and apply any variable expansion prior to executing any of these lines.  So if you are doing an IF/ELSE construct and have a series of assignments they will be expanded prior to evaluating the first expression:

   IF %H1% GTR %H2% (
      REM we have had a clock rollover since we started, add 24 hours
      set /a TEMP=%H2% + 24
      set /a HOUR2=%TEMP% * 3600
   )

In this example the TEMP variable will not be equal to "%H2% + 24", it will either be set to a previous value or not at all.  The disturbing thing is that Microsoft considers this a "feature" and not a "bug"; the documentation gives an example of how you could use a value stored in a variable and reset it to the original value all in one fell swoop... but since the value would be the same for both actions I don't see how this could ever be useful, or even do what they are saying cleanly.  The good news is that there are work-a-rounds... the bad news is that they are ugly.  The simplest method is to enable delayed expansion by setting an environment variable and then use alternate variable syntax (!VARIABLE!) in the instances where you want delayed expansion.  For me, I decided to leave my code as-is, and took a silent vow to avoid using Dos Shell in the future if any other alternative existed.

For completeness the code for the time delta is included below:

   @ECHO OFF
   REM > time_delta.bat <start> <stop>
   REM > time_delta.bat 2:03:04 2:05:06
   REM 122

   set MY_START=%1%
   set MY_STOP=%2%

   REM grab the hours, minutes and seconds from the start time fields 1 through 3
   REM grab the hours, minutes and seconds from the stop time  fields 4 through 6
   FOR /f "tokens=1-6 delims=:" %%a IN ( "%MY_START%:%MY_STOP%" ) DO (
     set H1=%%a
     set M1=%%b
     set S1=%%c
     set H2=%%d
     set M2=%%e
     set S2=%%f
   )

   REM convert the start time into raw seconds
   set /a MIN1=%M1% * 60
   set /a HOUR1=%H1% * 3600
   set /a TOTAL_1=%HOUR1% + %MIN1% + %S1%

   REM convert the stop time into raw seconds
   REM NOTE: the next line needs to be outside of the IF block in order to work due to
   REM       immediate variable expansion - variables are expanded when read, but the
   REM       commands are not executed until the ending brace is reached ")".  There
   REM       are other work-arounds for this, but none of them are clean.
   set /a TEMP=%H2% + 24

   IF %H1% GTR %H2% (
      REM we have had a clock rollover since we started, add 24 hours
      set /a MIN2=%M2% * 60
      set /a HOUR2=%TEMP% * 3600
   ) ELSE (
      REM no clock rollover, so convert directly into seconds
      set /a MIN2=%M2% * 60
      set /a HOUR2=%H2% * 3600
   )

   set /a TOTAL_2=%HOUR2% + %MIN2% + %S2%
   set /a DELTA=%TOTAL_2% - %TOTAL_1%
   echo %DELTA%

WHY I COMMENT MY CODE

I was reading an article (http://ardalis.com/when-to-comment-your-code) and it got me thinking about the subject, which is near and dear to my heart.

Over the years I've worked on a lot of projects, some for only a few days others spread out over years.  I've produced a lot of helper programs, implemented huge honking scripts and programs in a wide variety of languages over a wide range of platforms.  I've even managed large chunks of code in legacy systems spanning thousands of files.  I've worked in C, Objective-C, C++, Java, SQL, Bash, Ksh, Shell, Batch, Visual Basic, Tcl/Tk, Clearcase triggers, etc...  I've worked on Windows NT through Windows 7, Several Flavors of Linux including ones with Real-Time, Solaris, HP-UX, Powermax, Powerhawk, Lynx, some older flavors of Macintosh, VXWorks, etc...

I remember a day when a coworker was modifying a script to run on a new platform and asked me for some help.  It took a few minutes to figure out what the script was doing, I saw a few places for improvement, figured out the bug she was running into  and work out a solution.  As I was reading through the code I commented that it was a good script and she should consider sharing it with the department to which she responded "You do know you wrote this script, don't you?"  It occured to me that if it wasn't already in my coding style it would have been very difficult for me to figure out what the script was doing.  It was truly unfair to have created this huge program and then expect someone else to maintain it after I had lost interest (usually within a day or so).  I made sure that I left the program well commented before handing it back.

The idea that well-written code doesn't need comments is laughable.  This might fly if you are writing a tiny program, but once you start a project of any real scale - a few hundred thousand lines of code, written in various languages, various coding styles, maintained over a few decades by different companies... Even if you are writing a small program that someone else is going to use, and probably modify a few years down the line when your favorite language is no longer the defacto standard around the office and an intern needs to make a change... It is your duty to make sure that the code is well documented with sprinkled comments.

I consider comments to be bread crumbs that will aid a developer in understanding my code.  I like to strictly follow a coding standard; it doesn't really matter too much which one, but you should follow a consistent coding standard.  If you don't have one I like to do the following:
   - each file should have a header section that describes what the file is doing at a high level.
      //    - provide a revision history section, each entry is on one line
      // REVISION HISTORY
      // DSM  3Dec2012 added support for new signal types including COUNTS and 2-Byte floats
      // DSM  7Dec2012 moved signal information to a seperate class
   - provide a list of features you would like to add at some point, along with priority for implementation
      // TO-DO LIST
      // HIGH   move initialization data to an external file
      // LOW    move printing functionality to a seperate class
   - functions should contain header information that describe what the function is going to do
   - code should frequently comment what you are trying to do (at a high level)
   - if you are not using a tool to do versioning of your code you should provide change tracking in your code (which lines were touched by change number 5?)

If your compile environment supports any kind of documentation you should be using that as well... so for Visual Studio you should be using the XML style documentation that will allow for tool-tips when you mouse over a class or variable.

A lot of people dislike comments because they feel they can lie.  This is a maintenance issue, and when discovered the comments should be updated appropriately.  This is not a reason to do away with the comments entirely.  If anything it's a call for following better coding standards on updating comments.

It doesn't matter how clean you think your code is, some day an intern will attempt to make a change and if you don't give him a few breadcrumbs to follow he will fail.  When he fails it will not be his fault for not understanding your idiosyncratic coding style.  It will be your fault for not living up to the unspoken contract all paid developers have with their employers to write good and maintainable code.

Look at my coding samples for further examples of the minimum level of coding that should be acceptable.  Your code needs to be maintainable.