Continuous Integration Supporting Software
anime
he_the_great
I like to start out by saying that I'm not very happy with Jenkins. Yep, that is what has prompted this post, and while I'm not very well verse in the different software options out there I wanted to write down a list of things I'd like to see from a CI system and maybe it will help me find the one I want.

The CI system needs to handle a few basic elements:


  • Obtain source code from source repository

  • Monitor source repository for changes

  • Execute programs


Jenkins provides a framework for plugins to add functionality to the CI system. It provides concepts such as:


  • Build machine

  • Jobs

  • Build artifacts

  • Dependencies

  • Credentials


This are also important concepts to have for a CI system. It also provides a Web UI where plugins can add UI elements to specific organized sections. It doesn't do a very good job of pipelining, they through out all this UI configuration work and force you to use the Groovy scripting language to define out all the stuff already provided by the UI configuration scheme.

So I'd like to see a system that it structured more like a library then a framework. Provide a way to stand up a web server, provide resources for show builds and their status, provide the algorithms for managing build machines, jobs, and their relationships. Expose those relationships for UI visuals to be created. Make these concepts a core part of the library:


  • Job relation

  • Product

  • Product relation

  • Environment

  • Product tests

  • Product release

  • Environment management


I'm having a hard time identifying the best way to put this list into words. Jenkins appears to have been fairly close to what I'm describing, but it lacked these concepts so the pipeline plugin came along and through out the UI for the groovy script. Maybe had I instead learn the groovy script for Jenkins I wouldn't be so annoyed with Jenkins now.

I've also done some work with the GitLab CI system and don't believe it would be any better for what I'm trying to achieve.

MSI Custom Action DLL Written in D
anime
he_the_great
I'm tasked with testing software. I've implemented some ActiveX controls to make sure we integrate into software we don't have access to. That was a pretty exciting accomplishment but I've been haunted by the installers. Our installers include a DLL we have little insight on and calls a couple functions as a Custom Action, so I haven't been able to verify our installs work until we hand it over.

Well today I had some time to look at mocking something up for this and my success has come through utilizing a DLL written in D. To make this happen I used some help from "MSI Custom Action DLL" which provided the signature for the Custom Action function and what the primary header file was.

extern(Windows):
pragma(lib, "msi.lib");
UINT MsiSetPropertyW(MSIHANDLE hInstall,
	LPCWSTR   szName,       // property identifier, case-sensitive
	LPCWSTR   szValue);     // property value, null to undefine property

alias MSIHANDLE = uint;

export uint CustomActionFunction(MSIHANDLE hModule) { return ... }


Compilation was done from a Developer Command Prompt for AMD64 build. The generated DLL was then used to replace the black box DLL, and things were great.

dmd -m64 -shared -ofmain.dll main.d

Unable to access ecryptfs home folder
anime
he_the_great
Update: I feel I have verified that my access issues with my encrypted home were with utilizing an older version of ecryptfs. Yesterday I installed the latest testing and was able to mount my private home.

I started using an encrypted home Private folder as outlined in some many posts. This is a cautionary tail probably due to a bug I do not wish to track down.

Recently I had my partition table corrupted and I was unable to access my data. Supposedly all you need to recover your data is to know your login information or the encryption passphrase.

To get to my data (while I was trying to figure out what was wrong with the system hard drive) I had installed a fresh Debian Linux and mounted my home directory as I've done so many times before. Now that I have my ~/.ecryptfs folder and my Private/.Private folders back in their proper location, it should have been a simple $ ecryptfs-mount-private and provide my old login password. No such luck, claims the passphrass is wrong.

Ok, ok. Maybe I did something wrong there. Now I'll just mount private manually. I already know the passphrase my login password was used to wrap so it should be simple enough. Nope. Still no luck mounting even though I know the encoding passphrase.

I think that there may be a version miss-match that has caused this issue, but at this time I have not verified this. This is a cautionary tail that recovery may not be as simple as it has been made out to be.

Parsing CSV Files that exceed Available Memory
anime
he_the_great
It is not to unusual to read very large files and D does have a few options one is std.stdio.File. But using std.csv requires a input range of dchar which isn't available in the standard library (that I know of) for files.

I'd done some previous work in providing a file as range in my OpenStreetMap project. I had to make a few modifications so that it would provide an input range of dchar. While I tried to make it able to read files encode in UTF-8, UTF-16, or UTF-32, my plans were not completed since FileRange would need to read multiple bytes for those not UTF-8.

If someone is more interested then it should be possible to remove the cast on line 70 and make a conditional which correctly builds the character size desired before returning it.
Tags:

Compatibility Settings and Internet Explorer
anime
he_the_great

Recently I've been having to deal with IE and its compatibility modes. This has been a blast! Consider all of these factors:

  • Webpage is loaded within a WebBrowser control (as provided by Windows Forms or the like)
  • Page should support IE 9, 11, with emulation option for 7
  • Support ActiveX object events for IE 7 and IE 11 style hookups

There are a number of ways to dictate how IE should be rendering a page, here are a few.

  • Meta tag
  • Doc Type
  • IE Intranet compatibility setting
  • Registry Key

There are other ways of telling IE how it should render a page, but are the main ones when dealing with a browser control. I've ordered them by precedence, but the Registry Key can override the Doc Type.

  • The browser version IIS was told to load the page for.
  • The document mode the browser actually loaded the page for.

It would be really nice if you could render your page in IE 11 Edge if you're in a version 11 browser. You can, but any logic which depends on the browser version must be placed within Javascript and IIS version checks must be thrown out. Here are some things to keep in mind about these configurations.

The browser control is "perpetually stuck in IE 7 rendering mode." This means IIS will always be told to render in IE 7 unless otherwise specified. And this is specified in the Windows Registry. For debugging, be sure to follow the instructions about add the vshost executable when running with Visual Studio debugging (if you don't match the executable name exactly it won't use that entry, and for some reason I kept wanting to use vhost). Which this does not matter if IIS is not the one producing code for IE 7 or IE 11 (where that code is unsupported on the other version).

For some reason all of these different places make it so complicated to understand where the bug resides. Hopefully this helps others or my future self.


String Replacement Driven by Enum Data
anime
he_the_great

Similar to my previous post "String Replacement Driven by CSV Data" I'd like to give another example which utilizes an enumeration. This approach is limited to replacing valid symbol names, but could be combined with other approaches.

dlang
import scriptlike;

enum Data : string {
REPLACEME = "ReplaceValue",
REPLACETHISTOO = "ValueReplaced",
}

void main() {
    auto file = "MyFile.txt";
    std.file.write(file, file
        .readText()
        .replaceWithData());
}


string replaceWithData(string str) {
    foreach(enumMember; EnumMembers!Data) {
        mixin("str = str.replace((cast(Data)r\""
          ~enumMember~"\").to!string, r\""~enumMember~"\");");
    }
    return str;
}

Scripting With Dub

Unless I've got this wrong, Dub isn't very well setup for running short scripts. RDMD does a good job of taking a file and building it, but it lacks the ability to grab some nice helper libraries that exist outside the standard library, in my case "scriptlike." This has lead me to, assuming dub to be installed, utilizes a very basic dub.json file to run a script.

dub.json
{
    "name": "postchanges",
    "sourcePaths": ["."],
    "targetType": "executable",
    "dependencies": {
        "scriptlike": ">=0.7.0",
    }
}

Now running 'dub' in the directory will execute the script and include the specified dependencies.


Freedom's Just Another Word for
anime
he_the_great
I came across this little article, "More Government Does Not Mean Less Freedom" which demonstrates the change in meaning when someone talks about freedom.

"Freedom's just another word for nothing left to lose" come from an American song, "Me and Bobby McGee." It comes from the idea that freedom means, "the ability to act without concern of others." We have certain conditions that need to be met before we act. Some of these conditions are very basic, like only purchasing a new gallon of milk when the current one is empty or low. This doesn't mean you aren't free to purchase a second gallon, some people find freezing milk to be a viable option.

The problem is by viewing you're personal conditions as the universal conditions, you are preventing those who have different conditions from exercising their freedom. In essence with this view of freedom, it is only possible to trade freedoms. With this thinking it becomes, I'm trading your freedom to kill me for my freedom to live. This makes freedom the ability to prevent people from doing what you don't want, and is why we see the conclusion found in the article.

But what happens when we remove the freedom to control people?

Can I then prevent you from breathing clean air? No, if you are able to obtain clean air then I would be unable to stop you without employing control. I could make it prohibitively difficult to obtain clean air, this gives us a basis discuss at what point something is prohibiting. But it should be something measurable that has measurable damage.

Can I prevent you from crossing the street? No, but I can make it prohibitively dangerous to do so. So we discuss what it means for someone to be placed in unreasonable danger.

Don't be fooled though, these are not acts of increasing freedom; instead we are looking at where the boundary of control resides. It is asking the question, at what point are you threatening another person's life that you are effectively controlling that person.
Tags:

Government Spending
anime
he_the_great
The government spends money, lots of money. I was watching a John Stossel episode recently, "Trust" and the subject of Bitcoin came up. The context was about how Bitcoin was not transparent and allowed the sale of illegal items without tracking. The other guest mentioned how the Bitcoin ledger is publicly available and would allow auditing government spending.

"It is not an open ledger system." -- E.J. Fagan

What does he mean, how is it not open (how would the system work without it being public)? Well I don't have his answer, assuming he is familiar with how Bitcoin actually works, I suspect that his thought of an open ledger is one that documents who the two parties in the transaction were.

Bitcoin utilizes an anonymous wallet, anyone can generate and can generate any number of wallets. The ledger records the transaction between the two wallets, and other than making associations or other publicly made available data there is no association to any specific party.

What if the government was required to obtain funding through Bitcoin. What if every government wallet was required to be made publicly recorded. Now the exact amount of money government possess and the exact transactions made will be known to the world. Every coin spent by the government will be accounted for. Every politicians salary will be known.

Don't forget to read my other posts on Bitcoin.
Tags:

2014 Tax Season
anime
he_the_great
Taxes are simple. It is just a bunch of information gathering and following very specific instructions. Ok, those instructions span hundreds of pages and any action has like 10 exceptions and sometimes you can doing things EZ. Gathering the needed information and placing that information in to the correct boxes, isn't simple and gets more complicated the more exceptions you want to take advantage of.

Free File Fillable Forms is a great place to do taxes yourself. Even if TaxAct and H&R provide even simpler options which are also free. I don't know why but the Free File forms changed their provided service so you no longer get to include 1099-DIV/1099-INT and other similar forms. Yes they do allow the W2, but originally all of these forms were used to "calculate" into the appropriate location.

Anyway, the point of this post is to draw attention to the worksheets that also need to be filled out. For the "2014 Qualified Dividend and Capital Gain Tax Worksheet" I throw this program together which asks direct questions and does the specified math.

I have not used this myself, I still need to compare it with my personal taxes. Please review the logic for yourself when running your own calculations.

https://gist.github.com/JesseKPhillips/6f15d9e35f989060adb6

There is a similar, and possible more complete tax preparation software over on sourceforge. I haven't used it, but I think there needs to be a different approach for FOSS tax software. The software needs to be broken into solvers per form/sheet. Each program should be readable to review based on the form. The programs are given a "review count" that indicates how many people have compared the logic to the form/worksheet. The sheets need a way to communicate, save progress, and reset. This makes for uniform way of adding sheets, provides indications to suggest personal review, and keeps the sheets simple to review their logic.

String Replacement Driven by CSV Data
anime
he_the_great

Sometimes it is worth writing a up a script that does something of use only once; there are times these this could be a need to generate a bunch of scripts which actually only differ by some predefined set of data. Usually you'll want to write such a program so that the data drives the execution. However this can be fairly challenging if the script language isn't really meant for such a task, an example of this might be SQL. At which point it may be desirable to have a script connect to the database and build a transaction; this may not be an option if security prevents running/installing the needed script, allowing only for someone to run SQL.

Due to the feature set of D building a simple replacement script generator isn't too challenging. And if you need to do more complicated text processing to get at specific data, I personally find D to decent processing options even if they tend to be a little more complicated since the tools for text processing tend to emphasize a single pass causing multi-pass processing to be a little more work.

Here is a program that does a basic generation driven by CSV data for a simple string template.

dlang
import std.algorithm;
import std.csv;
import std.exception;
import std.file;
import std.path;
import std.range;
import std.stdio;

immutable replacementString =
`Some data [ReplaceThis] and [ReplaceThat]`;

auto replacementData =
`23,"fishes that eat"
42,eagles`;

struct Data {
    string number;
    string comment;
}

void main(string[] args) {
    version(FileInput) {
        enforce(args.length == 2,
          "Usage: ./" ~ args[0].baseName ~ " FileName");
        auto replacementData = readText(args[1]);
    }
    foreach(record; csvReader!Data(replacementData))
        writeln(replacementString.replaceWithData(record));
}

string replaceWithData(string str, Data d) {
    return str.replace("[ReplaceThis]", d.number)
        .replace("[ReplaceThat]", d.comment);
}

This script can then be run with rdmd scriptName.d, and I provided a version to show loading data from a file which is rdmd -version=FileInput scriptName.d dataFile.csv.

Conclusion

This is not a statement of D being the most concise means to accomplish this particular task, or that another language like Python, Ruby, PHP, etc. aren't fully capable of performing the same task just as or more easily. What I'd like to get across is that such a simple task is pretty straight forward in a language providing machine accesses such as pointers.


?

Log in